I got this file with this info pattern:
# Query 1: 204.60k QPS, 230.79x concurrency, ID XXXXXXXXXX at byte 19XXX9318
# This item is included in the report because it matches --limit.
# Scores: V/M = 0.00
# Time range: 2020-01-29 18:18:59.073995 to 18:18:59.074005
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 7 2
# Exec time 10 2ms 1ms 1ms 1ms 1ms 12us 1ms
# Rows affecte 0 0 0 0 0 0 0 0
# Query size 7 74 37 37 37 37 0 37
# Warning coun 0 0 0 0 0 0 0 0
# String:
# Hosts 10.1.1.5 (1/50%), 10.8.0.2 (1/50%)
# Query_time distribution
# 1us
# 10us
# 100us
# 1ms ################################################################
# 10ms
# 100ms
# 1s
# 10s+
SHOW SESSION STATUS LIKE 'XXXXX'\G
\n break line
repeat
I want to run a script in python to get only some information from that file. There will be multiple querys.
Currently Im trying something like this:
#!/usr/bin/python
file = open("/etc/openvpn/logs/log-2020_01_29_06_20_PM.txt", "r")
read = file.read()
removeChar = read.replace("#", "")
for item in removeChar.split("\n"):
if "Hosts" and "Time range" in item:
print item.strip()
The output is:
Time range: 2020-01-29 18:18:59.073995 to 18:18:59.074005
Time range: 2020-01-29 18:18:58.489162 to 18:18:59.188582
Time range: 2020-01-29 18:18:58.666020 to 18:18:58.666028
I want it to be something like this:
['Query 1, 2020-01-29 18:18:59, 10.1.1.5, 10.8.0.2, SHOW SESSION STATUS LIKE 'XXXXX'\G']
['Query 2, 2020-01-29 18:19:59, 10.1.1.5, 10.8.0.2, SHOW FROM BLA * LIKE 'BLA'\G']
Im tired of trying to find how to do this, also I'm learning python because it is a good language to learn! :)
Thanks.
You could try without regex, using string manipulation only:
data = '''# Query 1: 204.60k QPS, 230.79x concurrency, ID XXXXXXXXXX at byte 19XXX9318
# This item is included in the report because it matches --limit.
# Scores: V/M = 0.00
# Time range: 2020-01-29 18:18:59.073995 to 18:18:59.074005
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 7 2
# Exec time 10 2ms 1ms 1ms 1ms 1ms 12us 1ms
# Rows affecte 0 0 0 0 0 0 0 0
# Query size 7 74 37 37 37 37 0 37
# Warning coun 0 0 0 0 0 0 0 0
# String:
# Hosts 10.1.1.5 (1/50%), 10.8.0.2 (1/50%)
# Query_time distribution
# 1us
# 10us
# 100us
# 1ms ################################################################
# 10ms
# 100ms
# 1s
# 10s+
SHOW SESSION STATUS LIKE 'XXXXX'\G
\n break line
repeat
'''
data = data.split('\n')
all_results = []
result = []
for row in data:
if row.startswith('# Query ') and not row.startswith('# Query size'):
row = row.split(':')[0].split('# ')[1]
result.append(row)
elif row.startswith('# Hosts'):
row = row.replace('# Hosts', '').replace(' ', '').split(',')
result.append(row[0].split('(')[0])
result.append(row[1].split('(')[0])
elif row.startswith('# Time range:'):
row = row.replace('# Time range:', '').split('.')[0].strip()
result.append(row)
elif row.startswith('SHOW') and row.endswith('\G'):
result.append(row)
result = ', '.join(result)
all_results.append(result)
result = []
print(all_results)
# output: "Query 1, 2020-01-29 18:18:59, 10.1.1.5, 10.8.0.2, SHOW SESSION STATUS LIKE 'XXXXX'\\G"
Hey! Thanks for that buddy! When I try to get a file and read it and use it says: IndexError: list index out of range. How can I overcome this? I want this to open a file and read it
At which line the error occurs? Or post if you can one of these files to test it.
I add a << [data = open("/path/to/file.txt", "r")] >> then I changed data split to << [data = data.read().split('\n')] >> and the error its on line 10 << [ result = [data[0].split(':')[0].split('# ')[1]] IndexError: list index out of range]] >>
Does each file contains more than one query?
Yes. Its a file with weekly querys... and I want to format that all like that you did because the next step its to archive it all on database to see who did what :)