python-anyconfig [1] is a MIT licensed python library provides generic access to configuration files in various formats with configuration merge along with config template and schema validation/generation support.
I, Satoru SATOH <ssato@redhat.com>, originally developed and keep maintain it with others’ help [2] .
[1] | This name took an example from the ‘anydbm’ python standard library. |
[2] | see the output of git log –pretty=format:”%an %ae” | grep -vE “Satoru SATOH” | sort | uniq. |
python-anyconfig provides very simple and unified APIs to process configuration files in various formats:
It enables to load configuration file[s] in various formats in the same manner, and in some cases, even there is no need to take care of the actual format of configuration file[s] like the followings:
import anyconfig
# Config type (format) is automatically detected by filename (file
# extension) in some cases.
conf1 = anyconfig.load("/path/to/foo/conf.d/a.yml")
# Loaded config data is a dict-like object, for example:
#
# conf1["a"] => 1
# conf1["b"]["b1"] => "xyz"
# conf1["c"]["c1"]["c13"] => [1, 2, 3]
# Or you can specify the format (config type) explicitly if automatic
# detection may not work.
conf2 = anyconfig.load("/path/to/foo/conf.d/b.conf", "yaml")
# Specify multiple config files by the list of paths. Configurations of each
# files are merged.
conf3 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"])
# Similar to the above but all or one of config file[s] is/are missing:
conf4 = anyconfig.load(["/etc/foo.d/a.json", "/etc/foo.d/b.json"],
ignore_missing=True)
# Specify config files by glob path pattern:
conf5 = anyconfig.load("/etc/foo.d/*.json")
# Similar to the above, but parameters in the former config file will be simply
# overwritten by the later ones:
conf6 = anyconfig.load("/etc/foo.d/*.json", ac_merge=anyconfig.MS_REPLACE)
Also, it can process configuration files which are actually jinja2-based template files:
In [1]: import anyconfig
In [2]: open("/tmp/a.yml", 'w').write("a: {{ a|default('aaa') }}\n")
In [3]: anyconfig.load("/tmp/a.yml", ac_template=True)
Out[3]: {'a': 'aaa'}
In [4]: anyconfig.load("/tmp/a.yml", ac_template=True, ac_context=dict(a='bbb'))
Out[4]: {'a': 'bbb'}
In [5]: open("/tmp/b.yml", 'w').write("{% include 'a.yml' %}\n") # 'include'
In [6]: anyconfig.load("/tmp/b.yml", ac_template=True, ac_context=dict(a='ccc'))
Out[6]: {'a': 'ccc'}
And python-anyconfig enables to validate configuration files in various format with using JSON schema like the followings:
# Validate a JSON config file (conf.json) with JSON schema (schema.yaml).
# If validatation suceeds, `rc` -> True, `err` -> ''.
conf1 = anyconfig.load("/path/to/conf.json")
schema1 = anyconfig.load("/path/to/schema.yaml")
(rc, err) = anyconfig.validate(conf1, schema1) # err is empty if success, rc == 0
# Validate a config file (conf.yml) with JSON schema (schema.yml) while
# loading the config file.
conf2 = anyconfig.load("/a/b/c/conf.yml", ac_schema="/c/d/e/schema.yml")
# Validate config loaded from multiple config files with JSON schema
# (schema.json) while loading them.
conf3 = anyconfig.load("conf.d/*.yml", ac_schema="/c/d/e/schema.json")
# Generate jsonschema object from config files loaded and get string
# representation.
conf4 = anyconfig.load("conf.d/*.yml")
scm4 = anyconfig.gen_schema(conf4)
scm4_s = anyconfig.dumps(scm4, "json")
And in the last place, python-anyconfig provides a CLI tool called anyconfig_cli to process configuration files and:
[3] | http://json-schema.org |
python-anyconfig supports various (configuration) file formats if the required module is available and the corresponding backend is ready to use:
Format | Type | Requirement | Notes |
---|---|---|---|
JSON | json | json (standard lib) or simplejson [4] | Enabled by default. |
Ini-like | ini | configparser (standard lib) | Ditto. |
Java properties [5] | properties | None (native implementation with standard lib) | Ditto. |
XML | xml | lxml [6] or ElementTree | Ditto. |
YAML | yaml | PyYAML [7] | Enabled automatically if the left requirement is satisfied. |
ConifgObj | configobj | configobj [8] | Ditto. |
MessagePack | msgpack | msgpack-python [9] | Ditto. |
TOML | toml | toml [10] | Ditto. |
BSON | bson | bson in pymongo [11] | Ditto. |
The supported formats of python-anyconfig on your system is able to be listed by ‘anyconfig_cli -L’ like this:
$ anyconfig_cli -L
Supported config types: bson, configobj, ini, json, msgpack, toml, xml, yaml
$
or with the API ‘anyconfig.list_types()’ will show them:
In [8]: anyconfig.list_types()
Out[8]: ['bson', 'configobj', 'ini', 'json', 'msgpack', 'toml', 'xml', 'yaml']
In [9]:
It utilizes plugin mechanism provided by setuptools [12] and other formats may be supported by corresponding pluggale backends like the following:
See also How to write backend plugin modules section about writing plugins.
[4] | https://pypi.python.org/pypi/simplejson |
[5] | ex. https://docs.oracle.com/javase/7/docs/api/java/util/Properties.html |
[6] | https://pypi.python.org/pypi/PyYAML |
[7] | https://pypi.python.org/pypi/lxml |
[8] | https://pypi.python.org/pypi/configobj |
[9] | https://pypi.python.org/pypi/msgpack-python |
[10] | https://pypi.python.org/pypi/toml |
[11] | https://pypi.python.org/pypi/pymongo |
[12] | http://peak.telecommunity.com/DevCenter/setuptools#dynamic-discovery-of-services-and-plugins |
[13] | https://pypi.python.org/pypi/pyjavaproperties |
Many runtime dependencies are resolved dynamically and python-anyconfig just disables specific features if required dependencies are not satisfied. Therefore, only python standard library is required to install and use python-anyconfig at minimum.
The following packages need to be installed along with python-anycofig to enable the features.
Feature | Requirements | Notes |
---|---|---|
YAML load/dump | PyYAML | none |
ConifgObj load/dump | configobj | none |
MessagePack load/dump | msgpack-python | none |
TOML load/dump | toml | none |
BSON load/dump | bson | bson from pymongo package may work and bson [14] does not |
Template config | Jinja2 [15] | none |
Validation with JSON schema | jsonschema [16] | Not required to generate JSON schema. |
[14] | https://pypi.python.org/pypi/bson/ |
[15] | https://pypi.python.org/pypi/Jinja2/ |
[16] | https://pypi.python.org/pypi/jsonschema/ |
There is a couple of ways to install python-anyconfig:
Binary RPMs:
If you’re Fedora or Red Hat Enterprise Linux user, you can install RPMs from the copr repository, http://copr.fedoraproject.org/coprs/ssato/python-anyconfig/.
PyPI: You can install python-anyconfig from PyPI with using pip:
$ pip install anyconfig
Build RPMs from source: It’s easy to build python-anyconfig with using rpm-build and mock:
$ python setup.py srpm && mock dist/python-anyconfig-<ver_dist>.src.rpm
or:
$ python setup.py rpm
and install built RPMs.
Build from source: Of course you can build and/or install python modules in usual way such like ‘python setup.py bdist’, ‘pip install git+https://github.com/ssato/python-anyconfig/’ and so on.
If you have any issues / feature request / bug reports with python-anyconfig, please open an issue ticket on github.com (https://github.com/ssato/python-anyconfig/issues).
The following areas are still insufficient, I think.
Any feedbacks, helps, suggestions are welcome! Please open github issues for these kind of problems also!
Run ‘[WITH_COVERAGE=1] ./pkg/runtest.sh [path_to_python_code]’ or ‘tox’ for tests.
About test-time requirements, please take a look at pkg/test_requirements.txt.
Backend class must inherit anyconfig.backend.base.Parser or its children in anyconfig.backend.base module and need some members and methods such as load_from_string(), load_from_path(), load_from_stream(), dump_to_string(), dump_to_path() and dump_to_stream(). And anyconfig.backend.tests.ini.Test10 and anyconfig.backend.tests.ini.Test20 may help to write test cases of these methods.
JSON and YAML backend modules (anyconfig.backend.{json,yaml}_) should be good examples to write backend modules and its test cases, I think.
Also, please take a look at some example backend plugin modules mentioned in the Supported configuration formats section.