QAM - A python RPC Framework using AMQP

Introduction

qam is a framework for remote-procedure-calls. It uses the carrot messaging framework. The RPC specific code is based on Python XML-RPC.

Note

qam is not actively maintened anymore by the qamteam. The successor of qam is callme <http://pypi.python.org/pypi/callme>. callme uses internally a more simpler approach as qam due to the improvements the AMQP standard has made since QAM was released. Therefore we made a complete re-design and wrote callme which has less overhead and is much faster than qam.

Key Features:

• uses AMQP as Transport Protocol
• supports synchronous and asynchrouns remote method calls
• supports timeouts in synchronous and asynchronous mode
• JSON marshaling for high interoperatbility (see Serialization: JSON vs. Pickle for details)
• Pickle marshaling for Object Transport Support
• Supports Remote Exception Transfer in JSON/Pickle/Synchronous/Asynchrounus mode
• Fully Threaded
• Easy to Use
• OpenSource BSD-licensed

The AMQP messaging system manages the remote-procedure-calls for a client and a server. The client sends a amqp message to the server, where the method is specified which should be called on server. After executing the function the result is packed into a amqp message and sent back to the client.

The aim of qam is to offer a simple RPC framework, which is reliable and secure.

You have to install a AMQP message broker. The most popular are:

• RabbitMQ
• ZeroMQ
• Apache Qpid

Personally we have used RabbitMQ, it is easy to install and to configure.

Before you start using the qam framework you should know a little bit about AMQP. Therefor we advice you to read Rabbits and warrens a very good article which introduces the basic ideas of AMQP. Further you can look at the carrot documentation. There you can get also good overview of AMQP.

Documentation

The full Documentation including Reference can be found at pypi documentation

Mailing list

Join the QAM Users mailing list: QAM-Users

If you are developing inside QAM join: QAM-Developers

Bug Tracker:

If you find any issues please report them on http://bitbucket.org/qamteam/qam/issues/

Getting QAM

You can get the python package on the Python Package Index

The Mercurial Repository is available at bitbucket.org qam

Installation

qam can be installed via the Python Package Index of from source.

Using easy_install to install qam:

$ easy_install qam

If you have downloaded a source tarball you can install it by doing the following:

$ python setup.py build
# python setup.py install # as root

Tutorial

Starting the QAMServer

First it is necessary, to tell the QAMServer which functions are available. Therefore you have to register the functions on the QAMServer. After registering the functions, the QAMServer waits for method-calls from the QAMProxy.

Here is how you register a function on QAMServer and switch into the serving mode:

>>> from qam.qam_server import QAMServer
>>> qam_server = QAMServer(hostname="localhost",
...                    port=5672,
...                    username='guest',
...                    password='guest',
...                    vhost='/',
...                    server_id='qamserver')
...
>>> def adder_function(x, y):
...    return x + y
...
>>> qam_server.register_function(adder_function, 'add')
...
... # it is also possible to register the adder_function as follows:
... # qam_server.register_function(adder_function)
... # the method-name for registering in this case is adder_function.__name__
...
>>> qam_server.serve()

It is also possible to register whole classes on QAMServer. Therefore you only have to register the class instance on the QAMServer. It’s not necessary to register all functions of the class, it’s enough when you register the class instance.

IMPORTANT: When you register an instance you must specify a name as second argument in the register_class() method which selects the instance when calling it. In the example below you would call the remote method proxy.my_instance.adder_function(1,2)

Here is how you register a class instance on QAMServer and switch into the serving mode:

>>> from qam.qam_server import QAMServer
>>> qam_server = QAMServer(hostname="localhost",
...                    port=5672,
...                    username='guest',
...                    password='guest',
...                    vhost='/',
...                    server_id='qamserver')
...
>>> class TestClass():
...    def __init__(self):
...        pass
...    def adder_function(self,a,b):
...        return a+b
...
>>> instance = TestClass()
>>> qam_server.register_class(instance,'my_instance')
>>> qam_server.serve()

Managing RPC with QAMProxy

The QAMProxy sends the RPC-requests to the QAMServer and receives the result. It acts like a client which can call Server Methods and receive their results.

There are two different ways to receive the result:

• synchronous call: the QAMProxy blocks until a result arrives
• asynchronous call: it is possible to register a callback-function which will be called when the result arrives. In the meantime the QAMProxy can execute other functions or you can go in with your programm to execute.

Synchronous RPC

This example shows you how to call a simple method registered on the QAMServer. You have to wait for the result, because no callback-function is registered. So it is a synchronous RPC.

>>> from qam.qam_proxy import QAMProxy, QAMMethodNotFoundException, QAMException
... # create a new QAMProxy-instance
>>> qam_proxy = QAMProxy(hostname="localhost",
...                    port=5672,
...                    username='guest',
...                    password='guest',
...                    vhost='/',
...                    server_id='qamserver',
...                    client_id='qamproxy')
...
>>> result = qam_proxy.add(2,3) # call method on QAMServer and wait for a result
... # close all open AMQP connections and cleanup
>>> qam_proxy.close()

In case you have registered a class instance on the QAMServer and you want to call a method from this instance you can simple call this instance on QAMProxy. You can call the instance with the name you specified with qam.qam_server.QAMServer.register_class(instance,name). In this example it is a synchronous RPC again. You have to wait for the result.

>>> from qam.qam_proxy import QAMProxy, QAMMethodNotFoundException, QAMException
>>> qam_proxy = QAMProxy(hostname="localhost",
...                    port=5672,
...                    username='guest',
...                    password='guest',
...                    vhost='/',
...                    server_id='qamserver',
...                    client_id='qamproxy')
...
>>> result = qam_proxy.my_instance.adder_function(2,4)
>>> qam_proxy.close()

Asynchronous RPC

If you don’t want to wait for the result it is possible to register a callback-function. You can do this by calling QAMProxy.callback(callback_function, error_function).method_to_call_on_server(params). The callback-function takes two parameters as arguments. The first is the callback-function. The second is optional and is only called if an error occourd on QAMServer.

SIDENOTE: It is highly recomended that you alway set a error_function as well, as you can never know if the remote method will succeed or will throw an exception or if an internal exception will happen. Especially in asynchronous calls the only way you will be notified in case of an error WITHOUT an error_function is the qam logging.

After receiving the result from QAMServer the callback-function or the error-function is executed, with the result as parameter. You can monitor the state of the callback with qam.qam_proxy.get_callback_state(uid). Possible States are:

• 0: waiting on result
• 1: processing (result arrived and callback/error function is currently executing)
• 2: callback finished (callback/error function have finished executing)

In the following example you can see how to use callback-functions. In this example a simple method-call on the Server should be executed. No class instance is registered on the QAMServer, only the adder_function is registered.

>>> from qam.qam_proxy import QAMProxy, QAMMethodNotFoundException, QAMException
>>> qam_proxy = QAMProxy(hostname="localhost",
...                    port=5672,
...                    username='guest',
...                    password='guest',
...                    vhost='/',
...                    server_id='qamserver',
...                    client_id='qamproxy')
...
... # defining functions for callback
>>> def success(arg):
...    print arg
>>> def error(arg):
...    # if an error occours on QAMServer
...    print arg
>>> uid = qam_proxy.callback(success, error).add(2,4)
>>> while True:
...    state = qam_proxy.get_callback_state(uid)
...    if state == 2 :
...        # execution of callback finished
...        break
...
>>> qam_proxy.close()

The function success and error are registered as callback and error function. If everything succeeds the success-function will be called. If an error occoured on the QAMServer the error-function will be called. If no error-function is defined and an error occourd a log-message is written into the logging system.

It is also possible to execute asynchronous class-instance-method calls on the QAMServer. In the following example you can see how you can manage that. You can call the instance with the name you specified with qam.qam_server.QAMServer.register_class(instance,name).

>>> from qam.qam_proxy import QAMProxy, QAMMethodNotFoundException, QAMException
>>> qam_proxy = QAMProxy(hostname="localhost",
...                    port=5672,
...                    username='guest',
...                    password='guest',
...                    vhost='/',
...                    server_id='qamserver',
...                    client_id='qamproxy')
...
... # defining functions for callback
>>> def success(arg):
...    print arg
>>> def error(arg):
...    # if an error occours on QAMServer
...    print arg
>>> uid = qam_proxy.callback(success, error).my_instance.adder_function(2,4)
>>> while True:
...    state = qam_proxy.get_callback_state(uid)
...    if state == 2 :
...        # execution of callback finished
...        break
...
>>> qam_proxy.close()

Timeouts

It is also possible to set timeouts for remote functions. E.g. you might use timeouts if you don’t want to wait longer than for example 10 seconds for a function to return because after 10 seconds the result isn’t important for you anymore.

A simple client side synchronous code would look like this:

>>> from qam.qam_proxy import QAMProxy, QAMMethodNotFoundException, QAMException, QAMTimeoutException
... # create a new QAMProxy-instance
>>> qam_proxy = QAMProxy(hostname="localhost",
...                    port=5672,
...                    username='guest',
...                    password='guest',
...                    vhost='/',
...                    server_id='qamserver',
...                    client_id='qamproxy')
...
>>> try:
>>>     result = qam_proxy.set_timeout(10).add(2,3) # call method on QAMServer and wait for a result
>>> except QAMTimeoutException:
>>>     print 'Remote function is too slow, timeout occoured.'
...# close all open AMQP connections and cleanup
>>> qam_proxy.close()

But we also can set timeouts in asynchronous mode. The Callback/Error function will then only get called if it gets executed before the timeout occours. If the timeout occours before the callback/error function gets executed, the error function gets called with a qam.qam_proxy.QAMTimeoutException as argument. Let’s have a look at some sample code, again we assume that after 10 seconds result is not anymore important to us.

A simple client side asynchronous code would look like this:

>>> from qam.qam_proxy import QAMProxy, QAMMethodNotFoundException, QAMException, QAMTimeoutException
>>> qam_proxy = QAMProxy(hostname="localhost",
...                    port=5672,
...                    username='guest',
...                    password='guest',
...                    vhost='/',
...                    server_id='qamserver',
...                    client_id='qamproxy')
...
... # defining functions for callback
>>> def success(arg):
...    print arg
>>> def error(arg):
...    if isinstance(arg, QAMTimeoutException):
...      #timeout occoured
...      print 'Timeout occoured'
...    else:
...      print 'Other error happened'
>>> uid = qam_proxy.callback(success, error).set_timeout(10).add(2,4)
>>> while True:
...    state = qam_proxy.get_callback_state(uid)
...    if state == 2 :
...        # execution of callback finished
...        break
...
>>> qam_proxy.close()

Internally, if an timeout occours it doesn’t matter if the actual function will return some day or if it will never return. In case the remote function returns after the timeout exception has occoured the message will be correctly processed (so that everything stays clean in the AMQP Subsystem) but the result will be thrown away.

Serialization: JSON vs. Pickle

When working with QAM all the remote methods and results will get serialized in the background for you that you don’t have to bother about that. But for flexibility you can choose between two serializer: JSON and Pickle. Both have their benefits and drawbacks. In most cases you will do fine with the default Pickle serializer. But if you have special requirements you might choose the JSON serializer.

IMPORTANT: Anyway which serializer you choose, you must specify the same serializer on the QAMServer and on the QAMProxy, otherwise they can’t communicate correctly.

SIDENOTE: The default serializer is Pickle.

To set the serializer on the proxy, here e.g. json:

>>> from qam.qam_proxy import QAMProxy
... # create a new QAMProxy-instance
>>> qam_proxy = QAMProxy(hostname="localhost",
...                    port=5672,
...                    username='guest',
...                    password='guest',
...                    vhost='/',
...                    server_id='qamserver',
...                    serializer='json')

To set the serializer on the server, again json:

>>> from qam.qam_server import QAMServer
>>> qam_server = QAMServer(hostname="localhost",
...                    port=5672,
...                    username='guest',
...                    password='guest',
...                    vhost='/',
...                    server_id='qamserver',
...                    serializer='json')

To get an slight overview which serializer fits your need best here is a small comparison. This comparision is specially trimmed for the use in QAM. In other environments there might be other things to be aware of.

Property	Pickle	JSON
Compression	good	no compression
Complex Object Transport	yes	only json encodable objects are supported (dict, list, scalar)
Support for Custom Exception Inheritance	yes	no, but you can use Exceptions as well. The only drawback is that you cannot create custom Exceptions with inheritance
Interoperability with other languages	worse	good, as our transport format is quite easy to implement in other languages with json support
Needs Complex Type Definitions on both sides (Proxy and Server)	yes, because proxy and the server need to know which type they will get. You have to import your custom Argument Classes or custom Exceptions you want to raise on both proxy side and server side.	no, because we can only transfer dict list, and scalars we don’t need to define them seperately.

Architecture

Supported by

Wingware - The Python IDE (http://wingware.com)

Contributing

We are welcome everyone who wants to contribute to QAM. Development of QAM happens at http://bitbucket.org/qamteam/qam/

License

QAM is released under the BSD License. The full license text is in the root folder of the QAM Package.