September 06, 2010

shabti_couchdbkit – FormAlchemy with CouchDBKit

Warning

Incoherent documentation! The content has been copied wholesale from shabti_auth_couchdb — couchdbkit is not yet documented.

In this template, the now-familiar basic identity model pattern is expressed in CouchDB, a document-oriented, non-relational database recently released from Apache incubator status. The pattern is taken from kai, the Pylons code that currently runs the Pylons HQ web site (as at 2009-02-3), the source of which is available from the kai bitbucket repository.

Sometimes the rows-and-columns architecture of a relational store is not well-suited for the modelling job in hand. Sometimes a hierachically-organised architecture is a better fit — and sometimes the best fit is a schema-less ad hoc store. CouchDB is such a store and in kai, has proved an effective solution for document storage. retrieval and manipulation.

Using CouchDB with Pylons is quite straightforward but depends on easy_installing both python-couchdb and the supporting httplib2 plus the availability of a locally running CouchDB server. It is possible to connect to a remote CouchDB server if you propitiate the gods of ssh tunnelling. I haven’t tried it with Python but it is possible to connect Futon to a remote instance of CouchDB via an ssh tunnel.

Note

shabti_couchdbkit source code is in the bitbucket code repository

Built on top of shabti_auth, shabti_auth_repozewhat and shabti_formalchemy templates, the shabti_couchdbkit template integrates repoze auth’n’auth, FormAlchemy WebUI and CouchDBKit “NoSQL” modelling toolset.

About CouchDB

“Instead of storing data in rows and columns, the database manages a collection of JSON documents” - wikipedia

Note

What CouchDB is

• A document database server, accessible via a RESTful JSON API.
• Ad-hoc and schema-free with a flat address space.
• Distributed, featuring robust, incremental replication with bi-directional conflict detection and management.
• Query-able and index-able, featuring a table oriented reporting engine that uses Javascript as a query language.

What it is Not

• A relational database.
• A replacement for relational databases.
• An object-oriented database. Or more specifically, meant to function as a seamless persistence layer for an OO programming language.

(the above text was lifted from the Introduction page on the couchdb web site.)

See also

Getting started with Python on the Apache site.

Some understanding of the basics of couchdb is necessary in order to use it effectively. Futon is a web-based administration tool which gives immediate access for experimentation.

The following short series of screen shots showing a demonstration database (available with the PylonsHQ source code) is not just a great saving on an otherwise total of three thousand words but it illustates the degree to which couchdb’s schema-less, document-oriented model differs from the probably more familiar relational model ...

Futon view on a collection of databases

Futon view of a collection of documents

Futon view of a collection of attributes

Using the template

After successfully installing Shabti, additional paster templates will be available. Simply create a Shabti-configured project by specifying that paster should use the shabti_couchdbkit template:

$ paster create -t shabti_couchdbkit myproj

These are the option dialogue choices appropriate for the Shabti auth couchdb template — which uses mako templates and for this template, SQLAlchemy is optional, use just CouchDB or SQLAlchemy as well if you wish ...

(mako/genshi/jinja/etc: Template language) ['mako']:
(True/False: Include SQLAlchemy 0.5 configuration) [False]: True
(True/False: Setup default appropriate for Google App Engine) [False]:

Once the project has been created, navigate to the project directory and run the (brief) test suite:

$ nosetests

7 tests should be executed successfully. If the tests succeed, the next step is to initialise the couchdb store by running the project setup script:

$ paster setup-app development.ini

Once the store has been inialised, start the Pylons web app with:

$ paster serve --reload development.ini

The Shabti couchdb auth’n’auth template’s variant on the standard Pylons welcome screen is browsable at at http://localhost:5000/ ...

Welcome screen

Uauthenticated model view (via browse-db link)

sign-in page

Authenticated model view (via sign-in link)

Futon view of the User entity in the identity model

Lastly, for completeness, the User identity model as presented by couchdb’s GUI Futon:

Notes on the template

Auth’n’auth is not the best use of CouchDB. It would be far more sensible to adopt a standard relational approach to auth’n’auth whilst reserving CouchDB for handling documents. However, (ab)using CouchDB in the context of the standard Shabti auth’n’auth design pattern does allow some of the basic flavour to filter through.

CouchDB is used extensively behind the scenes on the new PylonsHQ web site (2009-01-04 :: source code is currently available on bitbucket - http://www.bitbucket.org/bbangert/kai/). A significantly useful approach to using CouchDB in the context of developing web apps is described in this introduction to using couchdb with django. The example code that is provided can be readily adapted for use with Pylons.

The shabti_couchdbkit template follows the same approach as the shabti_auth – boilerplate authentication template and re-uses the majority of the code in that template. After the user performs a successful authentication, the request.environ dict holds the standard AUTH_USER key, the value of which is the couchdb id key, usable for retrieving the corresponding modelled User object, this binding persists until the user explicitly signs off.

The @authorize decorator is inherited from shabti_auth – boilerplate authentication and is immediately available for use in applying authorization constraints on access to controllers and/or actions. Persistence of user, group and permission data is handled in the application’s couchdb-mediated model and is amenable to manipulation via the usual REST controller approach.

Pylons adjustments for CouchDB

Note, the examples shown below all assume that myproj was the project name supplied as arg to paster create.

All conversations with CouchDB take place over HTTP and development.ini specifies the host on which CouchDB is presenting a RESTful API on port 5984

development.ini

couchdb_server = http://localhost:5984/
couchdb_uri = http://localhost:5984/myproj

config/environment.py

The bindings made in development.ini are instantiated into the config dictionary, as per usual.

# Setup the database connection
config['myproj.server'] = Server(config['couchdb_server'])
config['myproj.db'] = Database(config['couchdb_uri'])

model/user.py

The scheme of the declaration of the user model properties runs along reasonably familiar lines but the access methods will be unfamiliar to many. This is an illustrative fragment of the modelling which captures most of the significant points:

class User(Document):
    """Represents a human user"""
    type = TextField(default='User')
    displayname = TextField()
    email = TextField()
    password = TextField()
    password_check = TextField()
    created = DateTimeField(default=datetime.utcnow)
    active = TextField()
    session_id = TextField()
    groups = ListField(TextField())

    by_displayname = View('user', '''
        function(doc) {
          if (doc.type == 'User') { emit(doc.displayname,doc) }
        }''')

    by_email = View('user', '''
        function(doc) {
          if (doc.type == 'User') { emit(doc.email, doc) }
        }''')

Clearly, the approach shows a slight departure from the standard identity model structure used in shabti templates that deal with relational models but, pace the View methods, the difference is less salient than the similarity. (It’s just a standing jump to get from here to there, not a running jump and grab).

lib/base.py

All communication with couchdb is via HTTP on the net, so the standard persistence concepts of server/client or exclusive access to the local filesystem do not apply, nor do issues of database “sessions”. Each interaction with the database is RESTful and is therefore less reliable than a firmly-socketed connection to an RDBMS (even if the latter does have a bad habit of “going away”). The read/write function is necessarily GET/PUT respectively, with all of the attendant issues of leading a RESTful life.

Pylons vs Shabti lib/base.py

--- pylons/lib/base.py
+++ shabti/lib/base.py
@@ -1,11 +1,27 @@
 from {{package}}.model import meta
+from couchdb import Database

 class BaseController(WSGIController):

+    def _setup(self):
+        """Do basic request setup"""
+        if pylons.session.get('logged_in', False):
+            user = User.load(self.db, pylons.session.get('user_id'))
+            if not user or user.session_id != pylons.session.id:
+                user = None
+                pylons.session['logged_in'] = False
+                pylons.session.delete()
+                pylons.session.save()
+        else:
+            user = None
+
     def __call__(self, environ, start_response):
         """Invoke the Controller"""
         # WSGIController.__call__ dispatches to the Controller method
         # the request is routed to. This routing information is
         # available in environ['pylons.routes_dict']

+        pylons.c.db = self.db = Database(pylons.config['couchdb_uri'])
+        self._setup()
+
         return WSGIController.__call__(self, environ, start_response){{endif}}

websetup.py

The included websetup.py gives an example of using Python to set up a database and add an initial entry.

server = pylons.config['myproj.server']

# Change teardown to True if the database
# exists and is to be torn down and re-created
teardown = True
if teardown and 'myproj' in server:
    del server['myproj']
    db = server.create('myproj')
elif not 'myproj' in server:
    db = server.create('myproj')
else:
    db = pylons.config['myproj.db']

gadmin = model.user.Group(
        name = "Administrators",
        description = u"Administration group",
        created = datetime.datetime.utcnow(),
        active = True)
gadmin.store(db)

gadmin_retrieved = Group.load(db, gadmin.id)
# Check the status
assert gadmin_retrieved.name == gadmin.name

Controller usage

Documents stored in CouchDB are typically retrieved with a previously defined and stored View (of the document collection). In the example shown below, the index action requests 5 user documents to be retrieved and ordered by the by_displayname view, the latter being accessed as a property of the User entity. The retrieval of Group and Permission data takes the same approach.

# controllers/user.py

def index(self):
    c.users = User.by_displayname(c.db, descending=True, count=5)
    c.groups = Group.by_name(c.db, descending=True, count=5)
    c.permissions = Permission.by_name(c.db, descending=True, count=5)
    c.title = 'Test'
    authusr = req.environ.get('AUTH_USER', False)
    c.identity = c.db[authusr] if authusr else False
    return render('test.mak')

Listed below is the more familiar SQLAlchemy access approach, included here for ease of comparison.

# controllers/user.py

def index(self):
    c.users = Session.query(User).all()
    c.groups = Session.query(Group).all()
    c.permissions = Session.query(Permission).all()
    c.title = 'Test'
    return render('test.mak')

Once the database connection parameters have been bound, interactions with couchdb are conducted over the net as per the REST GET/PUT/POST/DELETE architecture, content is expressed in standard JSON.

Authentication of the application to the database can be dispensed with if access is purely via the loopback, otherwise the model back-end handles per-request authentication of the application to a remote couchdb server.

Couchdb principles

(The following is lifted directly from Getting started with python-couchdb and shows fragments of the API for a Document class)

Getting Started

This tutorial exposes key features of this library through mainly code examples. For in-depth description of the modules, you’ll want to read the API documentation.

Create a CouchDB document:

from simplecouchdb import schema

class Greeting(schema.Document):
    author = schema.StringPropertty()
    content = schema.StringProperty()
    date = schema.DateTimePropertty()

Store the submitted Greetings

import datetime
from simplecouchdb import Server

db = Server()

greet = Greeting(
    author="Benoit",
    content="Welcome to simplecouchdb world",
    date=datetime.datetime.utcnow()
)

greet.save(db)

Updating documents in the database

greet.content = "Welcome to CouchDB"
greet.save(db)

Adding a property to a document

greet.homepage = "http://www.e-engura.org"
greet.save(db)

Get all instances

Create a view and save it in the db, add a simplecouchdb.schema.ViewProperty to the Document schema:

class Greeting(schema.Document):
    ...
    all = schema.ViewProperty("greeting", map_fun="""function(doc)
        { if (doc.doc_type == "Greeting") emit(doc._id, doc); }""")

greets = Greeting.all.get(db)

That’s it you now get all greets.

author:Graham Higgins <gjh@bel-epa.com>

September 06, 2010