Python parse web access logs

Question

I'm trying to parse a specific data from access logs, the logs format vary, it could be from nginx or apache. I need to get the following data:

• Remote host IP
• Request datetime
• Request type {GET|POST|PUT|..etc}
• Request Path {/main/index.html etc}
• HTTP Version {HTTP 1.1 | HTTP 1.0}
• HTTP Response code {200|400|403 ...etc}

I tried using split but it does not always work because of the log formats not always the same:

sample = """
::1 - - [03/Jan/2018:21:28:49 +0100] "GET /moodle/course/view.php?id=19 HTTP/1.1" 200 78325 "http://localhost/moodle/login/index.php" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0"
83.198.250.175 - - [22/Mar/2009:07:40:06 +0100] "GET /style.css HTTP/1.1" 200 1692 "http://www.example.org/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Wanadoo 6.7; Orange 8.0)" "-"
212.31.110.34 0.597 - [16/May/2018:12:30:44 +0000] safefin.example.com "GET / HTTP/1.1" 200 18193 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36"
151.227.152.48 - - [02/Jul/2014:14:35:55 +0100] "GET /css/main.css HTTP/1.1" 200 4658 "http://example.org/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
109.169.248.247 - - [12/Dec/2015:18:25:11 +0100] "POST /administrator/index.php HTTP/1.1" 200 4494 "http://example.net/administrator/" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" "-"
80.91.33.133 - - [17/May/2015:08:05:24 +0000] "GET /downloads/product_1 HTTP/1.1" 304 0 "-" "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.17)"
217.168.17.5 - - [17/May/2015:08:05:34 +0000] "GET /downloads/product_1 HTTP/1.1" 200 490 "-" "Debian APT-HTTP/1.3 (0.8.10.3)"
192.168.0.11 - - [27/Jun/2016:18:36:14 -0500] "GET / HTTP/1.1" 302 - "-" "Mozilla/5.0 (Linux; Android 5.1.1; SM-N910T Build/LMY47X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.81 Mobile Safari/537.36"
51.68.152.26 - - [09/Apr/2019:01:37:30 +0400] "GET / HTTP/1.1" 302 0 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
71.169.154.24 - - [01/Mar/2015:20:58:55 -0500] "GET /BarHarborcemeteries/Burns-RichardsonCemeteryimages/general%20view%20(2008).jpg HTTP/1.1" 200 165457 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.3.18 (KHTML, like Gecko) Version/7.1.3 Safari/537.85.12"
94.90.115.82 - - [02/Apr/2012:04:56:17 +0900] "GET /manager/html HTTP/1.1" 404 77 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"
172.20.32.1 - - [25/Feb/2015:10:42:29 +0300] "PUT /putfile?partNumber=5&uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt HTTP/1.1" 200 - "-" "-"
172.20.32.1 - - [25/Feb/2015:10:42:32 +0300] "POST /putfile?uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt HTTP/1.1" 200 279 "-" "-"
172.20.32.1 - - [25/Feb/2015:10:43:04 +0300] "DELETE /putfile HTTP/1.1" 400 81 "-" "-"
172.20.32.1 - - [25/Feb/2015:10:43:04 +0300] "DELETE /putfile HTTP/1.1" 204 - "-" "-"
172.20.32.1 - - [25/Feb/2015:10:41:02 +0300] "POST /putfile?uploads HTTP/1.1" 200 242 "-" "-"
151.227.152.48 - - [02/Jul/2014:14:35:56 +0100] "GET /img/Customers/Absolute-Steel-Framing.gif HTTP/1.1" 200 10123 "http://example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"
159.226.202.17 - - [31/Aug/2010:23:45:30 +0100] "GET / HTTP/1.1" 403 323 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; iCafeMedia; .NET CLR 2.0.50727; CIBA)"
65.55.3.169 - - [01/Sep/2010:08:03:47 +0100] "GET /robots.txt HTTP/1.1" 403 272 "-" "msnbot/2.0b (+http://search.example.com/msnbot.htm)._"
66.187.104.20 - - [24/Apr/2009:19:15:52 +1100] "GET /misc/arrow-desc.png HTTP/1.1" 404 217
77.35.168.108 - - [28/Apr/2009:10:38:09 +1100] "GET / HTTP/1.1" 200 85
77.35.172.105 - - [28/Apr/2009:12:49:27 +1100] "GET / HTTP/1.1" 304 -
79.137.201.45 - - [02/May/2009:12:17:26 +1100] "GET /robots.txt HTTP/1.0" 404 208
151.21.4.47 - - [17/Feb/2018:16:06:48 +0100] "GET /noindex/css/open-sans.css HTTP/1.1" 200 5081 "http://94.177.222.96/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0"
151.21.4.47 - - [17/Feb/2018:16:06:48 +0100] "GET /images/apache_pb.gif HTTP/1.1" 200 2326 "http://94.177.222.96/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0"
"""

lines = sample.split('\n')
structured_data = []
for line in lines:
    if line == '':
        continue

    parts = line.split(' ')

    remote_host = parts[0]    # IP
    time_date = parts[3]      # Log datetime
    request_method = parts[5] # GET OR POST..
    request_path = parts[6]   # Requested resource
    http_version = parts[7]
    response_status_code = parts[8]

    structured_dict = {'remote_host': remote_host,
                       'datetime': time_date,
                       'method': request_method,
                       'path': request_path,
                       'http_version': http_version,
                       'response_code': response_status_code
                       }
    structured_data.append(structured_dict)

for dict in structured_data:
    print(dict)

sample Output:

{'remote_host': '::1', 'datetime': '[03/Jan/2018:21:28:49', 'method': '"GET', 'path': '/moodle/course/view.php?id=19', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '83.198.250.175', 'datetime': '[22/Mar/2009:07:40:06', 'method': '"GET', 'path': '/style.css', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '212.31.110.34', 'datetime': '[16/May/2018:12:30:44', 'method': 'safefin.example.com', 'path': '"GET', 'http_version': '/', 'response_code': 'HTTP/1.1"'}
{'remote_host': '151.227.152.48', 'datetime': '[02/Jul/2014:14:35:55', 'method': '"GET', 'path': '/css/main.css', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '109.169.248.247', 'datetime': '[12/Dec/2015:18:25:11', 'method': '"POST', 'path': '/administrator/index.php', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '80.91.33.133', 'datetime': '[17/May/2015:08:05:24', 'method': '"GET', 'path': '/downloads/product_1', 'http_version': 'HTTP/1.1"', 'response_code': '304'}
{'remote_host': '217.168.17.5', 'datetime': '[17/May/2015:08:05:34', 'method': '"GET', 'path': '/downloads/product_1', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '192.168.0.11', 'datetime': '[27/Jun/2016:18:36:14', 'method': '"GET', 'path': '/', 'http_version': 'HTTP/1.1"', 'response_code': '302'}
{'remote_host': '51.68.152.26', 'datetime': '[09/Apr/2019:01:37:30', 'method': '"GET', 'path': '/', 'http_version': 'HTTP/1.1"', 'response_code': '302'}
{'remote_host': '71.169.154.24', 'datetime': '[01/Mar/2015:20:58:55', 'method': '"GET', 'path': '/BarHarborcemeteries/Burns-RichardsonCemeteryimages/general%20view%20(2008).jpg', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '94.90.115.82', 'datetime': '[02/Apr/2012:04:56:17', 'method': '"GET', 'path': '/manager/html', 'http_version': 'HTTP/1.1"', 'response_code': '404'}
{'remote_host': '172.20.32.1', 'datetime': '[25/Feb/2015:10:42:29', 'method': '"PUT', 'path': '/putfile?partNumber=5&uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '172.20.32.1', 'datetime': '[25/Feb/2015:10:42:32', 'method': '"POST', 'path': '/putfile?uploadId=2/fFEtO5aTFYNO7tjxbbmw6QkGOmeeOFt', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '172.20.32.1', 'datetime': '[25/Feb/2015:10:43:04', 'method': '"DELETE', 'path': '/putfile', 'http_version': 'HTTP/1.1"', 'response_code': '400'}
{'remote_host': '172.20.32.1', 'datetime': '[25/Feb/2015:10:43:04', 'method': '"DELETE', 'path': '/putfile', 'http_version': 'HTTP/1.1"', 'response_code': '204'}
{'remote_host': '172.20.32.1', 'datetime': '[25/Feb/2015:10:41:02', 'method': '"POST', 'path': '/putfile?uploads', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '151.227.152.48', 'datetime': '[02/Jul/2014:14:35:56', 'method': '"GET', 'path': '/img/Customers/Absolute-Steel-Framing.gif', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '159.226.202.17', 'datetime': '[31/Aug/2010:23:45:30', 'method': '"GET', 'path': '/', 'http_version': 'HTTP/1.1"', 'response_code': '403'}
{'remote_host': '65.55.3.169', 'datetime': '[01/Sep/2010:08:03:47', 'method': '"GET', 'path': '/robots.txt', 'http_version': 'HTTP/1.1"', 'response_code': '403'}
{'remote_host': '66.187.104.20', 'datetime': '[24/Apr/2009:19:15:52', 'method': '"GET', 'path': '/misc/arrow-desc.png', 'http_version': 'HTTP/1.1"', 'response_code': '404'}
{'remote_host': '77.35.168.108', 'datetime': '[28/Apr/2009:10:38:09', 'method': '"GET', 'path': '/', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '77.35.172.105', 'datetime': '[28/Apr/2009:12:49:27', 'method': '"GET', 'path': '/', 'http_version': 'HTTP/1.1"', 'response_code': '304'}
{'remote_host': '79.137.201.45', 'datetime': '[02/May/2009:12:17:26', 'method': '"GET', 'path': '/robots.txt', 'http_version': 'HTTP/1.0"', 'response_code': '404'}
{'remote_host': '151.21.4.47', 'datetime': '[17/Feb/2018:16:06:48', 'method': '"GET', 'path': '/noindex/css/open-sans.css', 'http_version': 'HTTP/1.1"', 'response_code': '200'}
{'remote_host': '151.21.4.47', 'datetime': '[17/Feb/2018:16:06:48', 'method': '"GET', 'path': '/images/apache_pb.gif', 'http_version': 'HTTP/1.1"', 'response_code': '200'}

I collected samples from various of places, Most of formats looks same the samples above.

Answer 1

If you want to use regexes to parse the log, here are a few that could be helpful:

Capturing IP addresses is a little tougher. If you want to check it is a valid ip address, try this. Otherwise, if you want 4 groups of numbers that are at most 3 digits long separated by dots:

\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}

For the datetime, it looks like you can grab the first occurrence of characters surrounded by square brackets

\[([^\]]+)\]

For the method, path, and response, it looks like you can grab the first occurrence of characters surrounded by quotes and then the number directly after it

"([^"]+)"\s+(\d{1,3})

because multiple things are matched here, you can make use of groups to grab the individual pieces. Using this regex, you would take the first group and simply take off the "GET, POST, DELETE, etc" and what's remaining would be the path.

Use python's re library and apply each regex to a line from your input and see what you get

#!/usr/bin/env python
import re

bad_ip_regex = re.compile("\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}")
datetime_regex = re.compile("\[([^\]]+)\]")
other_regex = re.compile('"([^"]+)"\s+(\d{1,3})')
with open("input.log", "r") as f:
  for line in f:
    item = {}

    # attempt to grab IP
    ip = bad_ip_regex.search(line)
    if ip:
      item["remote_host"] = ip.group(0)
    else:
      # no ip, just skip?
      continue

    # attempt to grab datetime
    datetime = datetime_regex.search(line)
    if datetime:
      item["datetime"] = datetime.group(1)
    else:
      continue

    # attempt to grab other
    other = other_regex.search(line)
    if other:
      item["method"] = other.group(1).split()[0]
      item["path"] = other.group(1).split()[1]
      item["response"] = other.group(2)
    else:
      continue

    print(item)

Because you can't guarantee the order of these items, it doesn't make sense to try to grab all the fields at once using regular expressions. Just try one at a time on each line.