Command Line Tools


Tool for performing brewery framework functionality from command line.


brewery command [command_options]

Commands are:

Commands are:

Command Description
run Run a stream
graph Generate graphviz structure from stream



brewery run stream.json

The json file should contain a dictionary with nodes and connections.


Generate a graphviz graph structure.


brewery run stream.json >
dot -o graph.png -T png


List available nodes. If a node name is specified, then node information, including list of node attributes is displayed.


brewery nodes
brewery nodes csv_source


Create and run non-branched pipe stream. Each argument is either a node or a node attribute. The attribute has form attribute_name=value. There should be at least one node defined. If there is no source node, then CSV on standard input is assumed. if there is no target node, then CSV on standard output is assumed.

Example - audit a CSV:

cat data.csv | brewery pipe audit

Make output nicer:

cat data.csv | brewery pipe audit pretty_printer

Read CSV from a file and store in newly created SQLite database table:

brewery pipe csv_source resource=data.csv \
             sql_table_target \
                url=sqlite:///data.sqlite \
                table=data  \
                create=1 \


This command is not fully working. There is no type conversion of values, which might cause problems. There is no way to specify non-scalar values (arrays, dictionaries). Some nodes might not have properely implemented attributes, therefore you might get error of non-existing attribute even if the attribute is there.


Audit mongo database collections from data quality perspective.


mongoaudit [-h] [-H HOST] [-p PORT] [-t THRESHOLD] [-f {text,json}] database collection

Here is a foo:

Argument Description
-h, --help show this help message and exit
-H HOST, --host HOST host with MongoDB server
-p PORT, --port PORT port where MongoDB server is listening
-t THRESHOLD, --threshold THRESHOLD threshold for number of distinct values (default is 10)
-f {text,json}, --format {text,json} output format (default is text)

The threshold is number of distict values to collect, if distinct values is greather than threshold, no more values are being collected and distinct_overflow will be set. Set to 0 to get all values. Default is 10.

Measured values

Probe Description
field name of a field which statistics are being presented
record_count total count of records in dataset
value_count number of records in which the field exist. In RDB table this is equal to record_count, in document based databse, such as MongoDB it is number of documents that have a key present (being null or not)
value_ratio ratio of value count to record count, 1 for relational databases
null_record_ratio ratio of null value count to record count
null_value_ratio ratio of null value count to present value count (same as null_record_ration for relational databases)
null_count number of records where field is null
null_value_ratio ratio of records
unique_storage_type if there is only one storage type, then this is set to that type

Example output

Text output:

    storage type: unicode
    present values: 1257 (10.09%)
    null: 0 (0.00% of records, 0.00% of values)
    empty strings: 0
    distinct values:
    storage type: unicode
    present values: 22 (95.65%)
    null: 0 (0.00% of records, 0.00% of values)
    empty strings: 0

JSon output:

{ ...
    "pdf_link" : {
       "unique_storage_type" : "unicode",
       "value_ratio" : 0.956521739130435,
       "distinct_overflow" : [
       "key" : "pdf_link",
       "null_value_ratio" : 0,
       "null_record_ratio" : 0,
       "record_count" : 23,
       "storage_types" : [
       "distinct_values" : [],
       "empty_string_count" : 0,
       "null_count" : 0,
       "value_count" : 22


This tool will change into generic data source auditing tool and will support all datastores that brewery will support, such as relational databases or plain structured files.

Table Of Contents

Previous topic

Data Quality

Next topic


This Page