Log Line Sources

Sources take log lines as strings from some iterable and wrap them with LogLine classes allowing them to be parsed on demand.

class loglab.sources.LogLineSource(iterable, line_class=<class 'loglab.lineformats.CombinedLogLine'>, ignore_invalid=True)

Reads log lines from an iterable and wraps it in LogLine

Some services may fail to log in an explicitly chronological order, perhaps because the timestamp marks the start of the request while the log line is only written when the request finishes.

Because it can be preferable to process log lines in chronological order, a source can be wrapped with LogBuffer, which uses a heap to output lines in ascending order of timestamp, assuming logs are mostly sorted already.

class loglab.sources.LogBuffer(iterable, window_size=1000)

Buffers and sorts a log within a sliding window, to ensure strict chronological ordering.

There is a utility class to set up a LogLineSource and a LogBuffer on the same object.

class loglab.sources.OrderedSource(iterable, window_size=1000, line_class=<class 'loglab.lineformats.CombinedLogLine'>, ignore_invalid=True)

Wrapper to construct a LogBuffer/LineSource from an iterable