Redis

Redis is an advanced key-value store where each key is associated with a value. What makes Redis different from many other key-value databases, is that values can be of different types:

• Strings
• Lists
• Sets
• Sorted Sets
• Hash tables

In other words, you can look at redis as a data structure server, the networked equivalent of the standard template library in C++.

Redis loads and maintains the whole data-set into memory, but the data-set is persistent, since at the same time it is saved on disk, so that when the server is restarted data can be loaded back in memory. If you need speed, Redis is great solution.

Note

stdnet is a made up word from std for Standard Template Library and net for networked.

Requirements

• redis-py, provides the standard redis client.
• pulsar optional. It is required by the asynchronous connection and the publish/subscribe redis application.

Connection String

The connection string is a way to specify the various parameters of the backend to use. Redis supports the following parameters:

• db, the database number.
• namespace, the namespace for all the keys used by the backend.
• password, database password.
• timeout, connection timeout (0 is an asynchronous connection).

A full connection string could be:

redis://127.0.0.1:6379?db=3&password=bla&namespace=test.&timeout=5

Model data

Each stdnet.odm.StdModel class has an associated base key which specifies the namespace for all keys associated with it:

>>> from stdnet import getdb
>>> from stdnet.apps.searchengine import WordItem
>>> rdb = getdb('redis://localhost:6379?db=7&namespace=bla.')
>>> rdb.basekey(WordItem._meta)
'bla.searchengine.worditem'

Instances

Each stdnet.odm.StdModel instance is mapped into a redis Hash table. The hash table key is uniquely evaluated by the model hash and the id of the model instance and it is stored at:

<<basekey>>:obj:<<id>>

For example, a WordItem with id 1 is mapped by the database handler in the code snipped above, into a redis hash table at key bla.searchengine.worditem:obj:1. The hash fields and values are given by the field name and values of the model instance.

Indexes

Indexes are obtained by using sets with keys obtained using the following form:

<<basekey>>:idx:<<field name>>:<<field value>>

If the model specify an implicit ordering via the stdnet.odm.Metaclass.ordering attribute, indexes are stored in sorted sets rather than sets.

Unique Constratins

For some models you may need to specify certain field to be unique across the Model. For example the following User model:

class User(odm.StdModel):
    username = odm.SymbolField(unique=True)
    emauil = odm.SymbolField(unique=True)
    password = odm.CharField(required=True)

specifies two constrains. In redis these constraints are stored into two separate hash tables with key given by:

<<basekey>>:uni:<<field name>>

Therefore our User model will have two additional hash tables at:

<<basekey>>:uni:username
<<basekey>>:uni:email

Each hash table map a field value to the id containing that value

Redis Session

Redis stdnet.odm.Session and Query are handled by lua scripts which perform them in a single atomic operation.

Redis Query

A stdnet.odm.Query is handled in redis by two different lua scripts:

• the first is script performs the aggregation of which results in a temporary redis key holding the ids resulting from the query operations.
• The second script is used to load the data from redis into the client.

Aggregation

Loading

The list of arguments passed to the stdnet.lib.lua.load_query script:

• query_key, the redis key holding the ids from the aggregation step.
• basekey the prefix to apply to all keys in the model to aggregate.
• List of field to loads as [num_fields, field1, ...]. if num_fields is 0, all model fields will load.
• List of related model to load as [num_rel_models, rel_models1, ...].

Asynchronous Connection

The stdnet.backends.redisb.async module implements an asynchronous connector for redis-py. It uses pulsar asynchronous framework. To use this connector, add timeout=0 to redis connection string:

'redis://127.0.0.1:6378?password=bla&timeout=0'

Usage:

from stdnet import getdb

db = getdb('redis://127.0.0.1:6378?password=bla&timeout=0')

Asynchronous Publish/Subscribe

class stdnet.backends.redisb.async.PubSub(connection_pool, shard_hint)

Asynchronous Publish/Subscriber handler for redis.

To listen for messages you can bind to the on_message event:

from stdnet import getdb

def handle_messages(channel_message):
    ...
    
redis = getdb('redis://122.0.0.1:6379?timeout=0').client
pubsub = redis.pubsub()
pubsub.bind_event('on_message', handle_messages)
pubsub.subscribe('mychannel')

channels

The set of channels this handler is subscribed to.

patterns

The set of patterns this handler is subscribed to.

publish(channel, message)

Publish a new message to a channel.

This method return a pulsar Deferred which results in the number of Subscribers that will receive the message.

subscribe(*args, **kwargs)

Subscribe to a list of channels or channel patterns.

It returns an asynchronous component which results in the number of channels this handler is subscribed to. If this is the first time the method is called by this handler, than the PubSub starts listening for messages which are fired via the on_message event.

unsubscribe(*args, **kwargs)

Un-subscribe from a list of channels or channel patterns.

It returns an asynchronous component which results in the number of channels this handler is subscribed to.

close(*args, **kwargs)

Stop listening for messages.

unsubscribe() from all channels and patterns and close the subscriber connection with redis.

parse_response(connection, command_name, **params)

Callback from the AsyncRedisRequest.

This method is invoked multiple times when new results are available.

Client Extensions

The stdnet.backends.redisb.client implements several extensions to the standard redis client in redis-py

Client

class stdnet.backends.redisb.client.Redis(host='localhost', port=6379, db=0, password=None, socket_timeout=None, connection_pool=None, charset='utf-8', errors='strict', decode_responses=False, unix_socket_path=None)

execute_command(*args, **options)

Execute a command and return a parsed response

parse_response(connection, command_name, **options)

Override redis-py parse_response method to deal with scripts.

prefixed(prefix)

Return a new PrefixedRedis client

execute_script(name, keys, *args, **options)

Execute a registered lua script at name. The script must be implemented via subclassing RedisScript.

Parameters:

• name – the name of the registered script.
• keys – tuple/list of keys pased to the script.
• args – argument passed to the script.
• options – key-value parameters passed to the RedisScript.callback() method once the script has finished execution.

countpattern(pattern)

delete all keys matching pattern.

delpattern(pattern)

delete all keys matching pattern.

zdiffstore(dest, keys, withscores=False)

Compute the difference of multiple sorted sets specified by keys into a new sorted set, dest.

zpopbyrank(name, start, stop=None, withscores=False, desc=False)

Pop a range by rank

zpopbyscore(name, start, stop=None, withscores=False, desc=False)

Pop a range by score

Prefixed Client

class stdnet.backends.redisb.client.PrefixedRedis(client, prefix)

A class for a prefixed redis client. It append a prefix to all keys.

prefix

The prefix to append to all keys

execute_command(cmnd, *args, **options)

Execute a command and return a parsed response

RedisScript

class stdnet.backends.redisb.client.RedisScript(script, name)

RedisScriptBase is a class which helps the sending and receiving lua scripts to redis via the evalsha command.

script

The lua script to run

required_scripts

A list/tuple of other RedisScript names required by this script to properly execute.

sha1

The SHA-1 hexadecimal representation of script required by the EVALSHA redis command. This attribute is evaluated by the library, it is not set by the user.

callback(response, **options)

This is the only method user should override when writing a new RedisScript. By default it returns response.

Parameters:

• response – the response obtained from the script execution.
• options – Additional options for the callback.