File Sources

Webserver logs are usually retrieved from compressed or uncompressed log files on disk.

Loglab provides these classes for accessing log files on disk as iterables.

class loglab.file_sources.LogFile(fname, window_size=1000, line_class=<class 'loglab.lineformats.CombinedLogLine'>, ignore_invalid=True)

class loglab.file_sources.DayLogFile(filename, date)

Wrapper around the above for outputting a filtered, sorted logfile

class loglab.file_sources.GZipLogFile(filename, window_size=1000, line_class=<class 'loglab.lineformats.CombinedLogLine'>, ignore_invalid=True)

Wrapper to construct a LogBuffer from a gzipped file.

Tailing a logfile

Loglab also includes a source that can retrieve log lines from the end of a file.

class loglab.tail.TailSource(logfile, from_start=False)

Tails a logfile, yielding lines in real time.

If from_start is True, the source will start emitting lines from the beginning of the file. Otherwise, it will emit the first full line added after the current end of the file.

__iter__()

Iterate over lines added to the file since it was opened.

Blocks until new lines are available to read, so must be wrapped as a thread if non-blocking behaviour is required.

This method takes care not to output a partial line by only outputting lines that are terminated by a newline. It also skips the first line as it does not know whether this is the start of a line.