API Reference ¶

Table of Contents

API Reference
- s3
- s3 Command Line Tool

s3 ¶

This program moves local files to and from Amazon’s Simple Storage System.

Capability:

Test if file exists in S3 bucket
Upload file to S3 bucket
Download file from S3 bucket
Delete file from S3 bucket
Copy S3 file to S3 file
Update metadata of S3 file
Create S3 bucket
Delete S3 bucket
Configure S3 bucket
List S3 bucket keys
List S3 buckets

S3 files also have metadata in addition to their content. metadata is a python dict i.e. a set of key, value pairs.

IMPLEMENTATION NOTES: The implementation is based on tinys3. I’ve kept the connection and request objects but changed their API so it is not compatible with tinys3 at all. I do use tinys3’s util module unchanged, and auth module with a slight change to the signing logic.

Paul Wexler, Jun 3, 2014

class s3.s3.Retry(limit, interval, status_codes)¶

Manage retries. Retry only if the response status code is in the list. Retry until the count is 0.

limit: The number of attempts to try. 0 means unlimited attempts.
interval: The number of seconds (floating point) to wait between attempts.
status_codes: A list of status codes to retry.

next(status_code, method, url, message)¶: manages count. conditionally logs retry message and sleeps for interval.

reset()¶: resets count.

class s3.s3.S3Bucket(bucket={})¶

The bucket object derived from the S3 <Bucket> tag dict.

attributes:

name
creation_date

class s3.s3.S3Facts¶: Facts (i.e. constants) for S3

class s3.s3.S3Key(obj={})¶

The key object listed in the <Contents> tag returned by the GET BUCKET request.

attributes:

key
last_modified
e_tag
size
owner - an S3Owner instance
storage_class

class s3.s3.S3Name(key, bucket=None)¶

An S3 file name consists of a bucket and a key. This pair uniquely identifies the file within S3.

The S3Connection class methods take a remote_name argument which can be either a string which is the key, or an instance of the S3Name class. When no bucket is given the default_bucket established when the connection is instantiated is used.

In other words, the S3Name class provides a means of using a bucket other than the default_bucket.

class s3.s3.S3Owner(obj={})¶

The Owner object in the key object listed in the <Contents> tag returned by the GET BUCKET request.

attributes:

id
display_name

class s3.s3.S3Part(part_number, etag)¶

part_number: ranges from 1 to S3Facts.maximum_part_number
etag: returned by S3 for successful upload_part

class s3.s3.S3Request(conn)¶: Base request object. Altered directly from the tinys3.S3Request implementation.

class s3.s3.S3Response(ok=True, error='', response=None, **kwargs)¶

ok: boolean - the response was/not ok.
error: the error when the response was not ok.
response: the response object returned by the requests module.

s3.s3.get_content_md5(s)¶: Returns the base64-encoded 128-bit MD5 digest of the string s.

class s3.s3.BucketS3Request(conn)¶: send() returns S3Response instance

class s3.s3.CopyS3Request(conn, src_bucket, src_key, dst_bucket, dst_key, headers={})¶: send() returns S3Response instance

class s3.s3.DeleteS3Request(conn, bucket, key)¶: send() returns S3Response instance

class s3.s3.ExistsS3Request(conn, bucket, key)¶: send() returns S3Response instance

class s3.s3.ReadS3Request(conn, bucket, key, local_file)¶: send() returns S3Response instance which includes a metadata attribute containing the metadata of the bucket/key.

class s3.s3.UpdateMetadataS3Request(conn, bucket, key, headers={})¶: send() returns S3Response instance

class s3.s3.WriteS3Request(conn, local_file, bucket, key, headers={})¶: send() returns S3Response instance

class s3.s3.S3Connection(access_key_id, secret_access_key, default_bucket=None, endpoint='s3.amazonaws.com', tls=True, region='us-east-1', retry={})¶

Altered directly from the tinys3.connection implementation.

Creates a new S3 connection

Params:

access_key_id

AWS access key (username)

secret_access_key

AWS secret access key (password)

default_bucket

Sets the default bucket.

Default is None

endpoint

Sets the s3 endpoint.

Default is S3Facts.default_endpoint.

tls

do/not use secure connection.

Default is True.

region

Region associated with the endpoint.

Default is S3Facts.default_region.

retry

dict contains values to retry requests.request()

Default is {‘limit’: 5, ‘interval’: 2.5, ‘status_codes’: [104,]}

limit: Number of times to retry. 0 means unlimited.
interval: Seconds to wait between retries.
status_codes: List of status codes (errors) to retry on.

s3.s3.check_response(f)¶: The decorated function adds basic response handling to a function returning an S3Response object. It also saves the requests response object in self.response

s3.s3.check_response_ok(f)¶: The decorated function adds basic response handling to a function returning an S3Response object. It also saves the requests response object in self.response It also returns self.response.ok

s3.s3.get_json_content(f)¶: The decorated function returns a dict converted from the json string in the response content.

s3.s3.get_xml_content(f)¶: The decorated function returns a dict converted from the xml string in the response content.

class s3.s3.Storage(connection, **kwargs)¶

bucket_list()¶

Returns a generator which returns an S3Bucket instance for each bucket in the account of the authenticated user.

You gotta luv xml. If there are no buckets ‘Buckets’ is an empty tag, otherwise it contains a ‘Bucket’ tag. If there is 1 bucket, ‘Bucket’ contains a dict. If there is more than 1 bucket, ‘Bucket’ is a list of dicts.

bucket_list_keys(bucket, delimiter=None, prefix=None, params={}, **kwargs)¶

Returns a generator which returns the keys in the given bucket.

If delimiter is used, the keys are returned first, followed by the common prefixes. Each key is returned as an instance of S3Key while each common prefix is returned as a string.

exists(remote_name)¶

Tests if remote_name is stored in remote storage.

Returns (exists, metadata)

exists: True if remote_name exists, False otherwise.
metadata: a dict. The file’s metadata (when exists == True)

raises StorageError on failure.

read(remote_name, local_name)¶

Reads (downloads) remote_name from storage and saves it in the local file system as local_name

Returns metadata (a dict) on success, raises StorageError on failure.

s3 Command Line Tool ¶

Commands operate on the default bucket unless the –bucket option is used.

Create a bucket

create-bucket [bucket_name] The default bucket_name is the default bucket.

Delete a file from S3

delete delete_file

Delete a bucket

delete-bucket [bucket_name] The default bucket_name is the default bucket.

Get a file from S3

get remote_src [local_dst]

List all files or list a single file and its metadata.

list [list_file]

List all buckets or list a single bucket.

list-buckets [bucket_name] If bucket_name is given but does not exist, this is printed:

'%s NOT FOUND' % bucket_name

Put a file to S3

put local_src [remote_dst]

arguments:

bucket_name: The name of the bucket to use.
delete_file: The remote file to delete.
list_file: If present, the file to list (with its metadata), otherwise list all files.
local_dst: The name of the local file to create (or overwrite). The default is the basename of the remote_src.
local_src: The name of the local file to put.
remote_dst: The name of the s3 file to create (or overwrite). The default is the basename of the local_src.
remote_src: The name of the file in S3 to get.

API Reference¶

s3¶

s3 Command Line Tool¶

API Reference ¶

s3 ¶

s3 Command Line Tool ¶