Internal API Documentation

These Python modules are only useful to you if you’re working directly on or with the MeteorPi server and database code.

Server: meteorpi_server

Logic to build the Flask based WSGI app, and optionally run it as a local Tornado based server for testing. The app is modular, initially not populated with any routes - these are provided by the xxx_api modules, allowing for modular assembly of an app with the functions you need for your application.

class meteorpi_server.MeteorApp(db)[source]

Common functionality for MeteorPi WSGI apps. This won’t contain any routes by default, these must be added using the functionality in admin_api, importer_api, query_api etc. This allows you to customise your application to suit, rather than forcing all the functionality into every instance. It might be sensible, for example, to only import the query API for a node which will never need external administration or configuration.

Variables:app – A WSGI compliant application, this can be referenced from e.g. a fastcgi WSGI container and used to connect an external server such as LigHTTPD or Apache to the application logic.
__init__(db)[source]

Create a new MeteorApp, setting up the internal DB

Parameters:db (MeteorDatabase) – An instance of meteorpi_fdb.MeteorDatabase to use when accessing the data and file stores.
static authentication_failure(message='Authorization required')[source]

Returns an authentication required response, including an optional message.

Parameters:message (string) – An optional message, can be used to specify additional information
Returns:flask error code, can be used as a return type from service methods
static not_found(entity_id=None, message='Entity not found')[source]

Build a response to indicate that the requested entity was not found.

Parameters:
  • message (string) – An optional message, defaults to ‘Entity not found’
  • entity_id (string) – An option ID of the entity requested and which was not found
Returns:

A flask Response object, can be used as a return type from service methods
requires_auth(roles=None)[source]

Used to impose auth constraints on requests which require a logged in user with particular roles.

Parameters:roles (list[string]) – A list of string representing roles the logged in user must have to perform this action. The user and password are passed in each request in the authorization header obtained from request.authorization, the user and password are checked against the user database and roles obtained. The user must match an existing user (including the password, obviously) and must have every role specified in this parameter.
Returns:The result of the wrapped function if everything is okay, or a flask.abort(403) error code if authentication fails, either because the user isn’t properly authenticated or because the user doesn’t have the required role or roles.
static success(message='Okay')[source]

Build a response with an object containing a single field ‘message’ with the supplied content

Parameters:message (string) – message, defaults to ‘Okay’
Returns:A flask Response object, can be used as a return type from service methods
class meteorpi_server.MeteorServer(db_path, file_store_path, port, add_routes=True)[source]

Tornado based server, exposes an instance of meteorpi_server.MeteorApp on localhost, can either be configured to expose all available APIs or can have such APIs added individually.

The database and application are exposed as instance properties to aid testing.

Variables:
  • db (MeteorDatabase) – The db used for this server
  • meteor_app (MeteorApp) – The application this server exposes
  • server (HTTPServer) – The tornado HTTPServer instance
  • port (int) – The port on which we’re listening for HTTP requests
class IOLoopThread[source]

A thread used to run the Tornado IOLoop in a non-blocking fashion, mostly for testing

__init__()[source]
MeteorServer.__init__(db_path, file_store_path, port, add_routes=True)[source]

Create a new instance, does not start the server.

Parameters:
  • db_path (string) – Path to the database, i.e. ‘localhost:/var/lib/firebird/2.5/data/meteorpi.fdb’
  • file_store_path (string) – File path to a directory on disk where the data store can store and retrieve its file data, i.e. path.expanduser(“~/meteorpi_files”)
  • port (int) – Port on which the HTTP server should run
  • add_routes (boolean) – Optional, defaults to True. If True then all routes from admin_api, importer_api and query_api will be added to the application, otherwise no routes will be added and you’ll have to do so explicitly.
static MeteorServer.add_all_routes(meteor_app)[source]

Add routes from admin_api, importer_api and query_api to the specified application

Parameters:meteor_app (MeteorApp) – The application to which routes should be added
MeteorServer.base_url()[source]
Returns:The full URL of this server
MeteorServer.start()[source]

Start an IOLoop and server in the current thread, this will block until killed

MeteorServer.start_non_blocking()[source]

Start a non-blocking server, returning a function which can be called to stop the server.

Returns:A zero-argument function which, when called, will stop the server.
meteorpi_server.query_api.add_routes(meteor_app, url_path='')[source]

Adds search and retrieval routes to a meteorpi_server.MeteorServer instance

Parameters:

Also provides the import API - this is an extensible system which can be used to receive data from an export task on the database side. The code includes an implementation of this system which handles event and file record replication, but the system can also be used for other purposes such as social media gateways, notification etc.

class meteorpi_server.importer_api.BaseImportReceiver[source]

Base class for entities which should be able to receive imports, whether that’s to handle database to database replication or otherwise. This base implementation is functional, but simply replies to both event and file record imports with ‘complete’ messages, thus never triggering either status imports or binary data.

static get_importing_user_id()[source]

Retrieve the importing user ID from the request context, this user will have already authenticated correctly by the point the import receiver is called.

Returns:The string user_id for the importing user
receive_event(import_request)[source]

Handle an Event import

Parameters:import_request (ImportRequest) – An instance of meteorpi_server.importer_api.ImportRequest which contains the parsed request, including the meteorpi_model.Event object along with methods that can be used to continue the import process. The Event is available as import_request.entity.
Returns:A Flask response, typically generated using the response_xxx() methods on the import_request. If no return is made or None is returned then treated as equivalent to returning response_complete() to terminate import of this Event
receive_file_data(file_id, file_data, md5_hex)[source]

Handle the reception of uploaded file data for a given ID. There is no return mechanism for this method, as the import protocol specifies that after a binary file is uploaded the corresponding file record or event should be sent again. This allows us to implement the import in a stateless fashion, caching aside, at the cost of an additional very small HTTP request. The request is small because all the exporter has to do, typically, is send the ID of the event or file record as the import infrastructure in this module caches the full record locally.

Parameters:
  • file_id (uuid.UUID) – The ID of the FileRecord for this file data
  • file_data – A file upload response from Flask’s upload handler, acquired with file_data = request.files[‘file’]
  • md5_hex – The hex representation of the MD5 hash of this file from the exporting party. This should be checked against the result of meteorpi_model.get_md5_hash(path) to ensure integrity of the file reception, and the file discarded or otherwise ignored if it does not match.
Returns:

None, continues the import irrespective of whether the file was received or not.
receive_file_record(import_request)[source]

Handle a FileRecord import

Parameters:import_request (ImportRequest) – An instance of meteorpi_server.importer_api.ImportRequest which contains the parsed request, including the meteorpi_model.FileRecord object along with methods that can be used to continue the import process. The FileRecord is available as import_request.entity.
Returns:A Flask response, typically generated using the response_xxx() methods on the import_request. If no return is made or None is returned then treated as equivalent to returning response_complete() to terminate import of this FileRecord
receive_status(import_request)[source]

Handle a meteorpi_model.CameraStatus import in response to a previous return of response_need_status() from one of the meteorpi_model.Event or meteorpi_model.FileRecord import methods.

Parameters:import_request (ImportRequest) – An instance of meteorpi_server.importer_api.ImportRequest which contains the parsed request, including the meteorpi_model.CameraStatus object along with methods that can be used to continue the import process. The CameraStatus is available as import_request.entity.
Returns:A Flask response, typically generated using the response_xxx() methods on the import_request. If no return is made or None is returned then treated as equivalent to returning response_continue() to return to the import of the enclosing entity. This is different to receive_file_record() and receive_event() because we only ever see a status imported in response to an attempt to import an event or file record which has a status we don’t have on the importing side.
class meteorpi_server.importer_api.ImportRequest(entity, entity_id)[source]

Helper used when importing, makes use of the ‘cached_request’ request transparent to the importing party.

Variables:
  • entity_cache – LRU cache used to stash entities being imported.
  • logger – Logs to ‘meteorpi.server.import’
  • entity_type – The type of the ID being imported, which will be one of ‘file’, ‘status’, ‘event’ or ‘none’.
__init__(entity, entity_id)[source]

Constructor, don’t use this from your own code, instead use process_request() to create one from the Flask request context.

Parameters:
  • entity – The entity being imported, either pulled from the request directly or from the cache. This can be None under error conditions, in which case the only legitimate response is to send a ‘continue’ message back to the exporter, at which point it will re-send the necessary information to rebuild the cache.
  • entity_id – The ID of the entity being imported, this will always be defined.
static process_request()[source]

Retrieve a CameraStatus, Event or FileRecord from the request, based on the supplied type and ID. If the type is ‘cached_request’ then the ID must be specified in ‘cached_request_id’ - if this ID is not for an entity in the cache this method will return None and clear the cache (this should only happen under conditions where we’ve failed to correctly handle caching, such as a server restart or under extreme load, but will result in the server having to re-request a previous value from the exporting party).

Returns:A dict containing ‘entity’ - the entity for this request or None if there was an issue causing an unexpected cache miss, and ‘entity-id’ which will be the UUID of the entity requested. The entity corresponding to this request, or None if we had an issue and there was an unexpected cache miss.
response_complete()[source]

Signal that this particular entity has been fully processed. The exporter will not send it to this target again under this particular export configuration (there is no guarantee another export configuration on the same server won’t send it, or that it won’t be received from another server though, so you must always check whether you have an entity and return this status as early as possible if so)

Returns:A response that can be returned from a Flask service method
response_continue()[source]

Signals that a partial reception of data has occurred and that the exporter should continue to send data for this entity. This should also be used if import-side caching has missed, in which case the response will direct the exporter to re-send the full data for the entity (otherwise it will send back the entity ID and rely on the import party’s caching to resolve it). Use this for generic cases where we need to be messaged again about this entity - currently used after requesting and receiving a status block, and in its cache-refresh form if we have a cache miss during import.

Returns:A response that can be returned from a Flask service method
static response_continue_after_file()[source]

As with response_continue, but static to allow it to be called from context where we don’t have a populated ImportRequest object. Always uses cached IDs, with the expectation that a subsequent request will force cache revalidation if required. Use this when acting on reception of binary data.

Returns:A response that can be returned from a Flask service method
static response_failed(message='Import failed')[source]

Signal that import for this entity failed. Whether this results in a retry either immediately or later in time is entirely up to the exporting party - this should therefore only be used for error cases, and not used to indicate duplicate data (use the response_complete for this as it tells the exporter that it shouldn’t send the data again)

Parameters:message (string) – An optional message to convey about the failure
Returns:A response that can be returned from a Flask service method
static response_need_file_data(file_id)[source]

Signal the exporter that we need the binary data associated with a given file ID

Parameters:file_id (uuid.UUID) – the UUID of the meteorpi_model.FileRecord for which we don’t currently have data
Returns:A response that can be returned from a Flask service method
response_need_status()[source]

Signal the exporter that we need the status associated with the current entity, in response to which it should then send a status block with the appropriate ID.

Returns:A response that can be returned from a Flask service method
class meteorpi_server.importer_api.MeteorDatabaseImportReceiver(db)[source]

An implementation of meteorpi_server.import_api.BaseImportReceiver that connects to a meteorpi_fdb.MeteorDatabase and pushes any data to it on import, including managing the acquisition of any additional information (camera status, binary file data) required in the process.

__init__(db)[source]
meteorpi_server.importer_api.add_routes(meteor_app, handler=None, url_path='/import')[source]

Add two routes to the specified instance of meteorpi_server.MeteorApp to implement the import API and allow for replication of data to this server.

Parameters:

Database: meteorpi_fdb

Firebird based data access layer, uses a simple directory based file store to handle file data. The database itself provides for search, retrieval and registration operations, with extra modules providing export mark and process functionality. Caching is implemented as an in-memory LRU cache for each main event type, mostly to optimise the kind of burst access we tend to have during imports and exports, and during pagination of searches.

class meteorpi_fdb.MeteorDatabase(db_path, file_store_path, db_user='meteorpi', db_password='meteorpi', installation_id=None)[source]

Class representing a single MeteorPi relational database and file store.

Variables:
  • con – Database connection used to access the db
  • db_path – Path to the database
  • file_store_path – Path to the file store on disk
  • generators – Helper object containing generator functions to retrieve entities lazily
  • installation_id (string) – The installation ID, either supplied explicitly in the constructor or derived automatically from the network interface MAC address (used if None is passed to the constructor for this property)
__init__(db_path, file_store_path, db_user='meteorpi', db_password='meteorpi', installation_id=None)[source]

Create a new db instance. This connects to the specified firebird database and retains a connection which is then used by methods in this class when querying or updating the database.

Parameters:
  • db_path (string) – String passed to the firebird database driver and specifying a file location. Defaults to
  • file_store_path (string) – File data is stored on the file system in a flat structure within the specified directory. If this location doesn’t exist it will be created, along with any necessary parent directories.
  • db_user (string) – User for the database, defaults to ‘meteorpi’
  • db_password (string) – Password for the database, defaults to ‘meteorpi’
  • installation_id (string) – 12 Character string containing the installation ID which will be used as the default when registering new files and events to this database. If set to none this will default to an attempt to calculate the ID from the MAC address of the first network interface found.
clear_database()[source]

Delete ALL THE THINGS!

This doesn’t reset any internal counters used to generate IDs but does otherwise remove all data from the database. Also purges all files from the fileStore

create_or_update_export_configuration(export_config)[source]

Create a new file export configuration or update an existing one

Parameters:export_config (ExportConfiguration) – a meteorpi_model.ExportConfiguration containing the specification for the export. If this doesn’t include a ‘config_id’ field it will be inserted as a new record in the database and the field will be populated, updating the supplied object. If it does exist already this will update the other properties in the database to match the supplied object.
Returns:The supplied meteorpi_model.ExportConfiguration as stored in the DB. This is guaranteed to have its ‘config_id’ uuid.UUID field defined.
create_or_update_user(user_id, password, roles)[source]

Create a new user record, or update an existing one

Parameters:
  • user_id – user ID to update or create
  • password – new password, or None to leave unchanged
  • roles – new roles, or None to leave unchanged
Returns:

the action taken, one of “none”, “update”, “create”
Raises:

ValueError if there is no existing user and either password or roles is None
defer_export_tasks(config_id, seconds)[source]

Increment the export time of all events for a given config such that the earliest is seconds into the future

Parameters:
  • config_id (uuid.UUID) – The UUID of an export configuration
  • seconds (int) – The number of seconds by which tasks associated with this export configuration should be delayed. Any tasks that would occur before (now + seconds) are instead marked to occur at exactly that time. Tasks which were scheduled for further in the future, or for other export configurations, are not changed.
delete_export_configuration(config_id)[source]

Delete a file export configuration by external UUID

Parameters:config_id (uuid.UUID) – the ID of the config to delete
delete_user(user_id)[source]

Completely remove the specified user ID from the system

Parameters:user_id (string) – The user_id to remove
file_path_for_id(file_id)[source]

Get the system file path for a given file ID. Does not guarantee that the file exists!

Parameters:file_id (uuid.UUID) – ID of a file (which may or may not exist, this method doesn’t check)
Returns:System file path for the file
get_camera_status(time=None, camera_id=None)[source]

Return the camera status for a given time, or None if no status is available.

Parameters:
  • time (datetime.datetime) – UTC time at which we want to know the camera status ID, defaults to model.now() if not specified
  • camera_id (string) – The ID of the camera to query, defaults to the current installation ID if not specified
Returns:

A meteorpi_model.CameraStatus or None if no status was available
get_camera_status_by_id(status_id)[source]

Return the camera status block with the given UUID

Parameters:status_id (uuid.UUID) – UUID of the camera status
Returns:An instance of meteorpi_model.CameraStatus or None if none matched.
get_cameras()[source]

Retrieve the IDs of all cameras with active status blocks. As the model has changed somewhat since this code was first written this in effect means the IDs of all cameras we’ve ever seen, as we no longer have a ‘valid_to’ entry in the status table.

Returns:A list of camera IDs for all cameras with status blocks
get_event(event_id)[source]

Retrieve an existing meteorpi_model.Event by its ID

Parameters:event_id (uuid.UUID) – UUID of the event
Returns:A meteorpi_model.Event instance, or None if not found
get_export_configuration(config_id)[source]

Retrieve the ExportConfiguration with the given ID

Parameters:config_id (uuid.UUID) – ID for which to search
Returns:a meteorpi_model.ExportConfiguration or None, or no match was found.
get_export_configurations()[source]

Retrieve all ExportConfigurations held in this db

Returns:a list of all meteorpi_model.ExportConfiguration on this server
get_file(file_id)[source]

Retrieve an existing meteorpi_model.FileRecord by its ID

Parameters:file_id (uuid.UUID) – UUID of the file record
Returns:A meteorpi_model.FileRecord instance, or None if not found
get_high_water_mark(camera_id=None)[source]

Retrieves the high water mark for a given camera, defaulting to the current installation ID

Parameters:camera_id (string) – The camera ID to check for, or the default installation ID if not specified
Returns:A UTC datetime for the high water mark, or None if none was found.
get_next_entity_to_export()[source]

Examines the t_fileExport and t_eventExport tables, finds the earliest incomplete export task and builds either a meteorpi_fdb.FileExportTask or a meteorpi_fdb.EventExportTask as appropriate. These task objects can be used to retrieve the underlying entity and export configuration, and to update the completion state or push the timestamp into the future, deferring evaluation of the task until later. Only considers tasks where the timestamp is before (or equal to) the current time.

Returns:Either None, if no exports are available, or a meteorpi_fdb.FileExportTask or meteorpi_fdb.EventExportTask depending on whether a file or event is next in the queue to export.
get_user(*args, **kwds)[source]

Retrieve a user record

Parameters:
  • user_id – the user ID
  • password – password
Returns:

A meteorpi_model.User if everything is correct
Raises:

ValueError if the user is found but password is incorrect or if the user is not found.
get_users()[source]

Retrieve all users in the system

Returns:A list of meteorpi_model.User
has_event_id(event_id)[source]

Check for the presence of the given event_id

Parameters:event_id (uuid.UUID) – The event ID
Returns:True if we have a meteorpi_model.Event with this ID, False otherwise
has_file_id(file_id)[source]

Check for the presence of the given file_id

Parameters:event_id (uuid.UUID) – The file ID
Returns:True if we have a meteorpi_model.FileRecord with this ID, False otherwise
has_status_id(status_id)[source]

Check for the presence of the given status_id

Parameters:status_id (uuid.UUID) – The camera status ID
Returns:True if we have a meteorpi_model.CameraStatus with this ID, False otherwise
import_camera_status(status)[source]

Import a new camera status block, used by the import server - do not use this for local status changes, it doesn’t update high water marks and will not execute any kind of roll-back. Clears the status cache, as camera status instances are somewhat dynamically generated based on other instances, most particularly for their time ranges.

Parameters:status – The new camera status block to import, this must be pre-populated with all required fields.
import_event(event, user_id)[source]

Functionality used by the import API, pulls in a meteorpi_model.Event including its pre-existing links to camera status and UUID. Do not use this to register a new event, it is only for use by the import system, this system ensures that all pre-requisites (files, camera status etc) are in place. If you use it outside of this context you will break your database. Don’t.

Parameters:
Internal:
import_file_record(file_record, user_id)[source]

Used by the import system to import a previously instantiated and named meteorpi_model.FileRecord. Do not use this method outside of the import system, this system will ensure that any pre-requisite status or file data is in place, using this method directly will certainly break your database.

Parameters:
Returns:

The integer internal ID of the imported record, used when importing events to create the link tables without additional database queries.
Internal:
mark_entities_to_export(export_config)[source]

Apply the specified meteorpi_model.ExportConfiguration to the database, running its contained query and creating rows in t_eventExport or t_fileExport for matching entities.

Parameters:export_config (ExportConfiguration) – An instance of meteorpi_model.ExportConfiguration to apply.
Returns:The integer number of rows added to the export tables
register_event(camera_id, event_time, event_type, file_records=None, event_meta=None)[source]

Register a new event, updating the database and returning the corresponding Event object

Parameters:
  • camera_id (string) – The ID of the camera which produced this event
  • event_time (datetime) – The UTC datetime of the event
  • event_type (NSString) – A meteorpi_model.NSString describing the semantic type of this event
  • file_records – A list of meteorpi_model.FileRecord associated with this event. These must already exist, typically multiple calls would be made to register_file() before registering the associated event
  • event_meta – A list of meteorpi_model.Meta used to provide additional information about this event
Returns:

The meteorpi_model.Event as stored in the database
register_file(file_path, mime_type, semantic_type, file_time, file_metas, camera_id=None, file_name=None)[source]

Register a file in the database, also moving the file into the file store. Returns the corresponding FileRecord object.

Parameters:
  • file_path (string) – The path of the file on disk to register. This file will be moved into the file store and renamed.
  • mime_type (string) – MIME type of the file
  • semantic_type (NSString) – A meteorpi_model.NSString defining the semantic type of the file
  • file_time (datetime) – UTC datetime for the file
  • file_metas (list[Meta]) – A list of meteorpi_model.Meta describing additional properties of the file
  • camera_id (string) – The camera ID which created this file. If not specified defaults to the current installation ID
  • file_name (string) – An optional file name, primarily used to display in the UI and provide help when downloading.
Returns:

The resultant meteorpi_model.FileRecord as stored in the database
search_events(search)[source]

Search for meteorpi_model.Event entities

Parameters:search – an instance of meteorpi_model.EventSearch used to constrain the events returned from the DB
Returns:a structure of {count:int total rows of an unrestricted search, events:list of meteorpi_model.Event}
search_files(search)[source]

Search for meteorpi_model.FileRecord entities

Parameters:search – an instance of meteorpi_model.FileRecordSearch used to constrain the events returned from the DB
Returns:a structure of {count:int total rows of an unrestricted search, events:list of meteorpi_model.FileRecord}
set_high_water_mark(time, camera_id=None, allow_rollback=True, allow_advance=True)[source]

Sets the ‘high water mark’ for this installation.

This is the latest point before which all data has been processed, when this call is made any data products (events, images etc) with time stamps later than the high water mark will be removed from the database. Any camera status blocks with validFrom dates after the high water mark will be removed.

Internal:
update_camera_status(ns, time=None, camera_id=None)[source]

Update the status for a camera, optionally specify a time (defaults to model.now()).

If the time is earlier than the current high water mark for this camera any data products derived after that time will be deleted as if setHighWaterMark was called.

Clears the status cache.

Parameters:
  • ns (CameraStatus) – The new camera status. If an existing camera status is supplied (one with an assigned ID) the ID will be ignored and a new one generated, this makes it easier to update a camera status by retrieving it, modifying fields and then calling this method.
  • time (datetime.datetime) – The time from which the new status should apply, defaults to now if not specified, should be a UTC time.
  • camera_id (string) – The camera to which this status applies, or the default installation ID if not specified
class meteorpi_fdb.exporter.EventExportTask(db, config_id, config_internal_id, event_id, event_internal_id, timestamp, status, target_url, target_user, target_password)[source]

Represents a single active Event export, providing methods to get the underlying meteorpi_model.Event, the meteorpi_model.ExportConfiguration and to update the completion state in the database.

__init__(db, config_id, config_internal_id, event_id, event_internal_id, timestamp, status, target_url, target_user, target_password)[source]
class meteorpi_fdb.exporter.FileExportTask(db, config_id, config_internal_id, file_id, file_internal_id, timestamp, status, target_url, target_user, target_password)[source]

Represents a single active FileRecord export, providing methods to get the underlying meteorpi_model.FileRecord, the meteorpi_model.ExportConfiguration and to update the completion state in the database.

__init__(db, config_id, config_internal_id, file_id, file_internal_id, timestamp, status, target_url, target_user, target_password)[source]
class meteorpi_fdb.exporter.MeteorExporter(db, mark_interval_seconds=300, max_failures_before_disable=4, defer_on_failure_seconds=1800, scheduler=None)[source]

Manages the communication part of MeteorPi’s export mechanism, acquiring meteorpi_fdb.FileExportTask and meteorpi_fdb.EventExportTask instances from the database and sending them on to the appropriate receiver. This class in effect defines the communication protocol used by this process.

The scheduler defined by default will also handle back-off under failure conditions. If an export fails, the count of failures (since the server was started) for that export config will be incremented and all export tasks for that config will be pushed a configurable distance into the future. If it fails more than a certain number of times the config will be marked as disabled. Any successful export will reset the failure count.

Variables:scheduler – An instance of apscheduler.schedulers.background.BackgroundScheduler used to schedule regular mark of entities to export and trigger the actual export of such entities.
class ExportStateCache(export_task, state='not_started')[source]

Used as a continuation when processing a multi-stage export. On sub-task completion, if the export_task is set to None this is an indication that the task is completed (whether this means it’s failed or succeeded, there’s nothing left to do).

__init__(export_task, state='not_started')[source]
MeteorExporter.__init__(db, mark_interval_seconds=300, max_failures_before_disable=4, defer_on_failure_seconds=1800, scheduler=None)[source]

Build a new MeteorExporter. The export process won’t run by default, you must call the appropriate methods on this object to actually start exporting. A scheduler is created to handle automated, regular, exports, but is not started, you must explicitly call its start method if you want regular exports to function.

Parameters:
  • db (MeteorDatabase) – The database to read from, for both entities under replication and the export configurations.
  • mark_interval_seconds (int) – The number of seconds after finishing on mark / export round that the next one will be triggered. Defaults to 300 for a five minute break. Note that at the end of an export round the process is re-run immediately, only finishing when there was nothing to do. This means that if we have a lot of data generated during an export run we won’t wait another five minutes before we process the new data.
  • defer_on_failure_seconds (int) – The number of seconds into the future which will be applied to any tasks pending for a given config when a task created by that config fails (including the failed task). The timestamp for any tasks with timestamps less than now + defer_on_failure_seconds will be set to now + defer_on_failure_seconds.
  • scheduler – The scheduler to use, defaults to a BackgroundScheduler with a non-daemon thread if not specified. Use a blocking one for test purposes.
  • max_failures_before_disable (int) – The number of times an export configuration can have its exports fail before it is disabled.
MeteorExporter.handle_next_export()[source]

Retrieve and fully evaluate the next export task, including resolution of any sub-tasks requested by the import client such as requests for binary data, camera status etc.

Returns:An instance of ExportStateCache, the ‘state’ field contains the state of the export after running as many sub-tasks as required until completion or failure. If there were no jobs to run this returns None.
complete:A job was processed and completed. The job has been marked as complete in the database
continue:A job was processed, more information was requested and sent, but the job is still active
failed:A job was processed but an error occurred during processing
confused:A job was processed, but the importer returned a response which we couldn’t recognise

The core search operations are split into SQL generation in the sql_builder module, and lazy instantiation of the domain entities in the generators module. While most existing APIs in the main database then instantiate lists of results in response to search, if you are extending the server and need to iterate over all files or all events these generators allow you to do so in an efficient manner.

class meteorpi_fdb.generators.MeteorDatabaseGenerators(db)[source]

Generator functions used to retrieve, and cache, items from the database.

__init__(db)[source]
cache_clear(cache)[source]

Clear a named cache

Parameters:cache – The cache to clear, one of (status, event, file, export, user)
cache_info()[source]

Retrieve cache info for the LRU caches used by this database

Returns:A dict of cache name (status, event, file, export, user) to cache info
camera_status_generator(sql, sql_args)[source]

Generator for meteorpi_model.CameraStatus

Parameters:
  • sql – A SQL statement which must return rows with, in order: lens, sensor, instURL, instName, locationLatitude, locationLongitude, locationGPS, locationError, orientationAltitude, orientationAzimuth, orientationError, orientationRotation, widthOfField, validFrom, softwareVersion, internalID, statusID, cameraID
  • sql_args – Any arguments required to populate the query provided in ‘sql’
Returns:

A generator which produces meteorpi_model.CameraStatus instances from the supplied SQL, closing any opened cursors on completion
event_generator(sql, sql_args)[source]

Generator for Event

Parameters:
  • sql – A SQL statement which must return rows with, in order: camera ID, event ID, internal ID, event time, event semantic type, status ID
  • sql_args – Any variables required to populate the query provided in ‘sql’
Returns:

A generator which produces Event instances from the supplied SQL, closing any opened cursors on completion.
export_configuration_generator(sql, sql_args)[source]

Generator for meteorpi_model.ExportConfiguration

Parameters:
  • sql – A SQL statement which must return rows with, in order: internalID, exportConfigID, exportType, searchString, targetURL, targetUser, targetPassword, exportName, description, active
  • sql_args – Any variables required to populate the query provided in ‘sql’
Returns:

A generator which produces meteorpi_model.ExportConfiguration instances from the supplied SQL, closing any opened cursors on completion.
file_generator(sql, sql_args)[source]

Generator for FileRecord

Parameters:
  • sql – A SQL statement which must return rows with, in order, internal ID, camera ID, mime type, semantic type, file time, file size, file ID, file name and file status ID.
  • sql_args – Any variables required to populate the query provided in ‘sql’
Returns:

A generator which produces FileRecord instances from the supplied SQL, closing any opened cursors on completion.
meteorpi_fdb.generators.first_from_generator(generator)[source]

Pull the first value from a generator and return it, closing the generator

Parameters:generator – A generator, this will be mapped onto a list and the first item extracted.
Returns:None if there are no items, or the first item otherwise.
Internal:
meteorpi_fdb.generators.first_non_null(values)[source]

Retrieve the first, non-null item in the specified list

Parameters:values – a list of values from which the first non-null is returned
Returns:the first non-null item
Raises:ValueError if there isn’t any such item in the list.
class meteorpi_fdb.sql_builder.SQLBuilder(tables, where_clauses=None)[source]

Helper class to make it easier to build large, potentially complex, SQL clauses.

This class contains various methods to allow SQL queries to be built without having to manage enormous strings of SQL. It includes facilities to add metadata constraints, and to map from meteorpi_model.NSString and datetime.datetime to the forms we use within the firebird database (strings and big integers respectively). Also helps simplify the discovery and debugging of issues with generated queries as we can pull out the query strings directly from this object.

__init__(tables, where_clauses=None)[source]

Construct a new, empty, SQLBuilder

Parameters:
  • where_clauses – Optionally specify an initial array of WHERE clauses, defaults to an empty sequence. Clauses specified here must not include the string ‘WHERE’, but should be e.g. [‘e.statusID = s.internalID’]
  • tables – A SQL fragment defining the tables used by this SQLBuilder, i.e. ‘t_file f, t_cameraStatus s’
Variables:
  • where_clauses – A list of strings of SQL, which will be prefixed by ‘WHERE’ to construct a constraint. As with the init parameter these will not include the ‘WHERE’ itself.
  • sql_args – A list of values which will be bound into an execution of the SQL query
Returns:

An unpopulated SQLBuilder, including any initial where clauses.
add_metadata_query_properties(meta_constraints, meta_table_name)[source]

Construct WHERE clauses from a list of MetaConstraint objects, adding them to the query state.

Parameters:
  • meta_constraints – A list of MetaConstraint objects, each of which defines a condition over metadata which must be satisfied for results to be included in the overall query.
  • meta_table_name – The name of the link table between the queried entity and metadata, i.e. t_eventMeta or t_fileMeta in the current code.
Raises:

ValueError if an unknown meta constraint type is encountered.
add_set_membership(values, column_name)[source]

Append a set membership test, creating a query of the form ‘WHERE name IN (?,?...?)’.

Parameters:
  • values – A list of values, or a subclass of basestring. If this is non-None and non-empty this will add a set membership test to the state. If the supplied value is a basestring it will be wrapped in a single element list. Values are mapped by SQLBuilder._map_value before being added, so e.g. NSString instances will work here.
  • column_name – The name of the column to use when checking the ‘IN’ condition.
add_sql(value, clause)[source]

Add a WHERE clause to the state. Handles NSString and datetime.datetime sensibly.

Parameters:
  • value – The unknown to bind into the state. Uses SQLBuilder._map_value() to map this into an appropriate database compatible type.
  • clause – A SQL fragment defining the restriction on the unknown value
get_count_sql()[source]

Build a SELECT query which returns the count of items for an unlimited SELECT

Returns:A SQL SELECT query which returns the count of items for an unlimited query based on this SQLBuilder
get_select_sql(columns, order=None, limit=0, skip=0)[source]

Build a SELECT query based on the current state of the builder.

Parameters:
  • columns – SQL fragment describing which columns to select i.e. ‘e.cameraID, s.statusID’
  • order – Optional ordering constraint, i.e. ‘e.eventTime DESC’
  • limit – Optional, used to build the ‘FIRST n’ clause. If not specified no limit is imposed.
  • skip – Optional, used to build the ‘SKIP n’ clause. If not specified results are returned from the first item available. Note that this parameter must be combined with ‘order’, otherwise there’s no ordering imposed on the results and subsequent queries may return overlapping data randomly. It’s unlikely that this will actually happen as almost all databases do in fact create an internal ordering, but there’s no guarantee of this (and some operations such as indexing will definitely break this property unless explicitly set).
Returns:

A SQL SELECT query, which will make use of self.sql_args when executed. To run the query, use e.g.:

b = SQLBuilder()
# Call add_sql etc methods on b here.
sql = b.get_select_sql(columns='e.cameraID, e.eventID, e.internalID, e.eventTime',
                       skip=search.skip,
                       limit=search.limit,
                       order='e.eventTime DESC')
with closing(connection.cursor()) as cursor:
    cursor.execute(sql, b.sql_args)
    # do stuff with results

static map_value(value)[source]

Perform type translation of values to be inserted into SQL queries based on their types.

Parameters:value – The value to map
Returns:The mapped value. This will be the same as the input value other than two special cases: Firstly if the input value is an instance of model.NSString we map it to the stringified form ‘ns:value’. Secondly if the value is an instance of datetime.datetime we map it using model.utc_datetime_to_milliseconds, returning an integer.
meteorpi_fdb.sql_builder.search_events_sql_builder(search)[source]

Create and populate an instance of meteorpi_fdb.SQLBuilder for a given meteorpi_model.EventSearch. This can then be used to retrieve the results of the search, materialise them into meteorpi_model.Event instances etc.

Parameters:search (EventSearch) – The search to realise
Returns:A meteorpi_fdb.SQLBuilder configured from the supplied search
meteorpi_fdb.sql_builder.search_files_sql_builder(search)[source]

Create and populate an instance of meteorpi_fdb.SQLBuilder for a given meteorpi_model.FileRecordSearch. This can then be used to retrieve the results of the search, materialise them into meteorpi_model.FileRecord instances etc.

Parameters:search (FileRecordSearch) – The search to realise
Returns:A meteorpi_fdb.SQLBuilder configured from the supplied search