Shove extension

A storage/query backend for shove which is bundled with Python.

status:beta database:any supported by shove: storage — Amazon S3 Web Service, Berkeley Source Database, Filesystem, Firebird, FTP, DBM, Durus, Memory, Microsoft SQL Server, MySQL, Oracle, PostgreSQL, SQLite, Subversion, Zope Object Database (ZODB); caching — Filesystem, Firebird, memcached, Memory, Microsoft SQL Server, MySQL, Oracle, PostgreSQL, SQLite dependencies:shove suitable for:“smart” interface to a key/value store; temporary memory storage

This extension wraps the shove library and provides the uniform query API along with support for Document API.

Note

Regardless of the underlying storage, Shove serializes the records and only offers access by primary key. This means that efficient queries are impossible even with RDBMS; moreover, such databases are more likely to perform slower than simple key/value stores. The Docu queries with Shove involve iterating over the full set of records on client side and making per-row comparison without proper indexing.

That said, the backend is considered not suitable for applications that depend on queries and require decent speed of lookups by value. However, it can be very useful as a memory storage (e.g. to analyze a JSON dump or calculate some data on the fly) or as an improved interface to an existing pure key/value storage which is mostly used without advanced queries.

class doqu.ext.shove_db.StorageAdapter(**kw)

All parametres are optional. Here are the most common:

Parameters:

• store_uri – URI for the data store
• cache_uri – URI for the caching instance

The URI format for a backend is documented in its module (see the shove documentation). The URI form is the same as SQLAlchemy’s.

clear()

Clears the whole storage from data.

connect()

Connects to the database. Raises RuntimeError if the connection is not closed yet. Use StorageAdapter.reconnect() to explicitly close the connection and open it again.

delete(primary_key)

Permanently deletes the record with given primary key from the database.

disconnect()

Writes the data into the file, closes the file and deletes the connection.

get(model, primary_key)

Returns model instance for given model and primary key.

get_many(doc_class, primary_keys)

Returns a list of documents with primary keys from given list. Basically this is just a simple wrapper around get() but some backends can reimplement the method in a much more efficient way.

get_or_create(doc_class, **kwargs)

Queries the database for records associated with given document class and conforming to given extra condtions. If such records exist, picks the first one (the order may be random depending on the database). If there are no such records, creates one.

Returns the document instance and a boolean value “created”.

reconnect()

Gracefully closes current connection (if it’s not broken) and connects again to the database (e.g. reopens the file).

save(data, primary_key=None, sync=False)

Saves given model instance into the storage. Returns primary key.

Parameters:

• data – dict containing all properties to be saved
• primary_key – the key for given object; if undefined, will be generated
• sync – if True, the storage is synchronized to disk immediately. This slows down bulk operations but ensures that the data is stored no matter what happens. Normally the data is synchronized on exit.

Note that you must provide current primary key for a model instance which is already in the database in order to update it instead of copying it.

class doqu.ext.shove_db.QueryAdapter(*args, **kw)

The Query class.

count()

Same as __len__ but a bit faster.

delete()

Deletes all records that match current query. Iterates the whole set of records.

order_by(names, reverse=False)

Defines order in which results should be retrieved.

Parameters:

• names – the names of columns by which the ordering should be done. Can be an iterable with strings or a single string.
• reverse – If True, direction changes from ascending (default) to descending.

Examples:

q.order_by('name')                  # ascending
q.order_by('name', reverse=True)    # descending

If multiple names are provided, grouping is done from left to right.

Note

while you can specify the direction of sorting, it is not possible to do it on per-name basis due to backend limitations.

Warning

ordering implementation for this database is currently inefficient.

values(name)

Returns an iterator that yields distinct values for given column name.

Supports date parts (i.e. date__month=7).

Note

this is currently highly inefficient because the underlying library does not support columns mode (tctdbiternext3). Moreover, even current implementation can be optimized by removing the overhead of creating full-blown document objects.

Note

unhashable values (like lists) are silently ignored.

where(**conditions)

Returns Query instance filtered by given conditions. The conditions are defined exactly as in Pyrant’s high-level query API. See pyrant.query.Query.filter documentation for details.

where_not(**conditions)

Returns Query instance. Inverted version of where().