This extension adds:
- Some new methods to the CKAN API for dealing with storage
- A /storage/upload page to web interface for doing file uploads
It uses OFS to talk to the backing storage so can support anything that OFS supports including local filesytem, S3, Google Storage etc.
Install the extension:
# using pip (could use easy_install)
pip install ckanext-storage
# could install from source
# hg clone https://bitbucket.org/okfn/ckanext-storage
# cd ckanext-storage
# pip install -e .
Note that for use of S3-like backends (S3, Google Storage etc) you will need boto (this is installed by default at the moment). For local filesystem backend you need to install pairtree (pip install pairtree).
In your config you need something like:
ckan.plugins = storage
## OFS configuration
## This is for google storage. Example for another backend is below
## See OFS docs for full details
ofs.impl = google
ofs.gs_access_key_id = GOOGCABCDASDASD
ofs.gs_secret_access_key = 134zsdfjkw4234addad
## bucket to use in storage You *must* set this
ckanext.storage.bucket = ....
## optional
## maximum content size for uploads in bytes, defaults to 1Gb
# ckanext.storage.max_content_length = 1000000000
## prefix for all keys. Useful because we may use a single bucket and want to
## partition file uploads. Defaults to file/
# ckanext.storage.key_prefix = file/
For local file storage you would replace ofs arguments with:
ofs.impl = pairtree
ofs.storage_dir = /my/path/to/storage/root/directory
There will be a new upload page at /storage/upload.
The API is located at:
/api/storage/metadata/{label}
It supports the following methods:
- GET will return the metadata
- POST will add/update metadata
- PUT will replace metadata
Metadata is a json dict of key values which for POST and PUT should be send in body of request.
A standard response looks like:
{
"_bucket": "ckannet-storage",
_content_length: 1074
_format: "text/plain"
_label: "/file/8630a664-0ae4-485f-99c2-126dae95653a"
_last_modified: "Fri, 29 Apr 2011 19:27:31 GMT"
_location: "some-location"
_owner: null
uploaded-by: "bff737ef-b84c-4519-914c-b4285144d8e6"
}
Note that values with ‘_’ are standard OFS metadata and are mostly read-only – _format i.e. content-type can be set).
Get credentials for doing operations on storage directly.
The API is at:
/api/storage/auth/request/{label}
Provide authentication information for a request so a client can interact with backend storage directly:
:param label: label.
:param kwargs: sent either via query string for GET or json-encoded
dict for POST). Interpreted as http headers for request plus an
(optional) method parameter (being the HTTP method).
Examples of headers are:
Content-Type
Content-Encoding (optional)
Content-Length
Content-MD5
Expect (should be '100-Continue')
:return: is a json hash containing various attributes including a
headers dictionary containing an Authorization field which is good for
15m.
The API is located at:
/api/storage/auth/form/{label}
Provide fields for a form upload to storage including authentication:
:param label: label.
:return: json-encoded dictionary with action parameter and fields list.