simple key-value storage api

simplekv is an API for key-value store of binary data. Due to its basic interface, it is easy to implemented a large number of backends. simplekv‘s origins are in storing user-uploaded files on websites, but its low overhead and design should make it applicable for numerous other problems, an example is a session backend for the Flask framework.

Built upon the solid foundation are a few optional bells and whistles, such as automatic ID generation/hashing (in simplekv.idgen). A number of backends are available, ranging from FilesystemStore to support for Amazon S3 and Google Storage through BotoStore.

Faster in-memory stores suitable for session management are supported through the likes of RedisStore or MemcacheStore.

Example

Here’s a simple example:

from simplekv.fs import FilesystemStore

store = FilesystemStore('./data')

store.put('key1', 'hello')

# will print "hello"
print store.get('key1')

# move the contents of a file to "key2" as efficiently as possible
store.put_file('key2', '/path/to/data')

Note that by changing the first two lines to:

from simplekv.memory.redisstore import RedisStore
import redis

store = RedisStore(redis.StrictRedis())

you could use the code exactly the same way, this time storing data inside a Redis database.

Why you should use simplekv

no server dependencies
simplekv does only depend on python and possibly a few libraries easily fetchable from PyPI, if you want to use extra features. You do not have to run and install any server software to use simplekv (but can at any point later on).
specializes in (even large!) blobs
The fastest, most basic simplekv backend implementation stores files on your harddrive and is just as fast. This underlines the focus on storing big blobs without overhead or metadata. A typical usecase is starting out small with local files and then migrating all your binary data to something like Amazon’s S3.

The core API

class simplekv.KeyValueStore

The smallest API supported by all backends.

Keys are ascii-strings with certain restrictions, guaranteed to be properly handled up to a length of at least 250 characters. Any function that takes a key as an argument raises a ValueError if the key is incorrect.

The regular expression for what constitutes a valid key is available as simplekv.VALID_KEY_REGEXP.

__contains__(key)

Checks if a key is present

Parameters:

key – The key whose existence should be verified.
Raises:
Returns:

True if the key exists, False otherwise.
__iter__()

Iterate over keys

Raises exceptions.IOError:
 If there was an error accessing the store.
delete(key)

Delete key and data associated with it.

If the key does not exist, no error is reported.

Raises:
get(key)

Returns the key data as a string.

Parameters:

key – Key to get
Raises:
get_file(key, file)

Write contents of key to file

Like KeyValueStore.put_file(), this method allows backends to implement a specialized function if data needs to be written to disk or streamed.

If file is a string, contents of key are written to a newly created file with the filename file. Otherwise, the data will be written using the write method of file.

Parameters:
  • key – The key to be read
  • file – Output filename or an object with a write method.
Raises:
iter_keys()

Return an Iterator over all keys currently in the store, in any order.

Raises exceptions.IOError:
 If there was an error accessing the store.
keys()

Return a list of keys currently in store, in any order

Raises exceptions.IOError:
 If there was an error accessing the store.
open(key)

Open key for reading.

Returns a read-only file-like object for reading a key.

Parameters:

key – Key to open
Raises:
put(key, data)

Store into key from file

Stores string data in key.

Parameters:
  • key – The key under which the data is to be stored
  • data – Data to be stored into key
Returns:

The key under which data was stored
Raises:
put_file(key, file)

Store into key from file on disk

Stores data from a source into key. file can either be a string, which will be interpretet as a filename, or an object with a read() method.

If the passed object has a fileno() method, it may be used to speed up the operation.

The file specified by file, if it is a filename, may be removed in the process, to avoid copying if possible. If you need to make a copy, pass the opened file instead.

Parameters:
  • key – The key under which the data is to be stored
  • file – A filename or an object with a read method. If a filename, may be removed
Returns:

The key under which data was stored
Raises:

In addition to that, a mixin class is available for backends that provide a method to support URL generation:

class simplekv.UrlMixin

Supports getting a download URL for keys.

url_for(key)

Returns a full external URL that can be used to retrieve key.

Does not perform any checks (such as if a key exists), other than whether or not key is a valid key.

Parameters:key – The key for which the url is to be generated
Raises exceptions.ValueError:
 If the key is not valid.
Returns:A string containing a URL to access key
class simplekv.UrlKeyValueStore

Deprecated since version 0.9: Use the UrlMixin instead.

Some backends support setting a time-to-live on keys for automatic expiration, this is represented by the TimeToLiveMixin:

class simplekv.TimeToLiveMixin

Allows keys to expire after a certain amount of time.

This mixin overrides some of the signatures of the api of KeyValueStore, albeit in a backwards compatible way.

Any value given for a time-to-live parameter must be one of the following:

  • A positive int, representing seconds,
  • simplekv.FOREVER, meaning no expiration
  • simplekv.NOT_SET, meaning that no TTL configuration will be done at all or
  • None representing the default (see TimeToLiveMixin‘s default_ttl_secs).

Note

When deriving from TimeToLiveMixin, the same default implementations for _put, _put_file and _put_filename are provided, except that they all take an additional ttl_secs argument. For more information on how to implement backends, see Implementing a new backend.

put(key, data, ttl_secs=None)

Like put(), but with an additional parameter:

Parameters:ttl_secs – Number of seconds until the key expires. See above for valid values.
Raises exceptions.ValueError:
 If ttl_secs is invalid.
put_file(key, file, ttl_secs=None)

Like put_file(), but with an additional parameter:

Parameters:ttl_secs – Number of seconds until the key expires. See above for valid values.
Raises exceptions.ValueError:
 If ttl_secs is invalid.
default_ttl_secs = simplekv.NOT_SET

Passing None for any time-to-live parameter will cause this value to be used.

ttl_support = True

Indicates that a key-value store supports time-to-live features. This allows users of stores to test for support using:

getattr(store, 'ttl_support', False)
simplekv.VALID_KEY_REGEXP = '^[\\\\\\`\\\\\\!\\"\\#\\$\\%\\&\\\'\\(\\)\\+\\,\\-\\.\\<\\=\\>\\?\\@\\[\\]\\^\\_\\{\\}\\~0-9a-zA-Z]+$'

This regular expression tests if a key is valid. Allowed are all alphanumeric characters, as well as !"`#$%&'()+,-.<=>?@[]^_{}~.

simplekv.VALID_KEY_RE = <_sre.SRE_Pattern object>

A compiled version of VALID_KEY_REGEXP.

Implementing a new backend

Subclassing KeyValueStore is the fastest way to implement a new backend. It suffices to override the _delete(), iter_keys(), _open() and _put_file() methods, as all the other methods have default implementations that call these.

After that, you can override any number of underscore-prefixed methods with more specialized implementations to gain speed improvements.

Default implementation

Classes derived from KeyValueStore inherit a number of default implementations for the core API mehthods. Specifically, the delete(), get(), get_file(), keys(), open(), put(), put_file(), methods will each call the _check_valid_key() method if a key has been provided and then call one of the following protected methods:

KeyValueStore._check_valid_key(key)

Checks if a key is valid and raises a ValueError if its not.

When in need of checking a key for validity, always use this method if possible.

Parameters:key – The key to be checked
KeyValueStore._delete(key)

Implementation for delete(). The default implementation will simply raise a NotImplementedError.

KeyValueStore._get(key)

Implementation for get(). The default implementation will create a io.BytesIO-buffer and then call _get_file().

Parameters:key – Key to be retrieved
KeyValueStore._get_file(key, file)

Write key to file-like object file. Either this method or _get_filename() will be called by get_file(). Note that this method does not accept strings.

Parameters:
  • key – Key to be retrieved
  • file – File-like object to write to
KeyValueStore._get_filename(key, filename)

Write key to file. Either this method or _get_file() will be called by get_file(). This method only accepts filenames and will open the file with a mode of wb, then call _get_file().

Parameters:
  • key – Key to be retrieved
  • filename – Filename to write to
KeyValueStore._has_key(key)

Default implementation for __contains__().

Determines whether or not a key exists by calling keys().

Parameters:key – Key to check existance of
KeyValueStore._open(key)

Open key for reading. Default implementation simply raises a NotImplementedError.

Parameters:key – Key to open
KeyValueStore._put(key, data)

Implementation for put(). The default implementation will create a io.BytesIO-buffer and then call _put_file().

Parameters:
  • key – Key under which data should be stored
  • data – Data to be stored
KeyValueStore._put_file(key, file)

Store data from file-like object in key. Either this method or _put_filename() will be called by put_file(). Note that this method does not accept strings.

The default implementation will simply raise a NotImplementedError.

Parameters:
  • key – Key under which data should be stored
  • file – File-like object to store data from
KeyValueStore._put_filename(key, filename)

Store data from file in key. Either this method or _put_file() will be called by put_file(). Note that this method does not accept strings.

The default implementation will open the file in rb mode, then call _put_file().

Parameters:
  • key – Key under which data should be stored
  • file – Filename of file to store

Atomicity

Every call to a method on a KeyValueStore results in a single operation on the underlying backend. No guarantees are made above that, if you check if a key exists and then try to retrieve it, it may have already been deleted in between (instead, retrieve and catch the exception).

Python 3

All of the examples are written in Python 2. However, Python 3 is fully supported and tested. When using simplekv in a Python 3 environment, the only important thing to remember is that keys are always strings and values are always byte-objects.

Indices and tables