Log lines and line formats

Each line format to parse is encapsulated by a LogLine class, which defines a regular expression used to parse the line.

There is also the facility to use a “quick” regular expression to parse the date and time of the log line, in order to more quickly filter log lines by date.

Log lines can be used to parse a line:

>>> line = LogLine(l)
>>> line.verb
'GET'
>>> line.req
'/'

Log lines are also mutable, though the output line format is cooerced to combined format:

>>> line.verb = 'POST':
>>> str(line)
'127.0.0.1 - [28/Jan/2012 15:37:03 +00:00] "POST / HTTP/1.0" 200 597 "-" "-"'

New log line formats can be defined by extending CombinedLogLine; such a line format needs only override these three properties:

LogLine.name

A human-readable name for the line format.

LogLine.stamp_pattern

A compiled regular expression object that will match the date part of the line.

LogLine.full_pattern

A regular expression string that will match the line. The regular expression should use named groups; these groups become properties of a parsed line instance.

Of course, other properties can be overriden for performance when analysing large log files.

LogLine classes

class loglab.lineformats.CombinedLogLine(line, line_number=None)

Parser/wrapper for a log line in Apache combined log format.

http://httpd.apache.org/docs/2.2/logs.html#combined

as_combined_line()

Output this log line again in combined format

time()

Return date as timestamp

loglab.lineformats.LogLine

alias of CombinedLogLine

class loglab.lineformats.ApacheLogLine(line, line_number=None)

This is a parser for the Apache log format that includes an additional cookie field after the User-Agent field.

Because the UA field can contain quotes so we cannot unambiguously determine whether the extra field exists. This class attempts to parse cookie field and falls back to the CombinedLogLine parser.

class loglab.lineformats.S3LogLine(line, line_number=None)

Implements a parser for the S3 log format as documented at

http://docs.amazonwebservices.com/AmazonS3/latest/index.html?LogFormat.html