A simple filesystem cache for python.
Pyfscache (python filesystem cache) is a filesystem cache that is easy to use. The principal class is pyfscache.FSCache, instances of which may be used as decorators to create cached functions with very little coding overhead:
import pyfscache
cache_it = pyfscache.FSCache('some/cache/directory',
days=13, hours=4, minutes=2.5)
@cache_it
def cached_doit(a, b, c):
return [a, b, c]
It’s that simple!
Now, every time the function cached_doit() is called with a particular set of arguments, the cache cache_it is inspected to see if an identical call has been made before. If it has, then the return value is retrieved from the cache_it cache. If not, the return value is calculated with cached_doit(), stored in the cache, and then returned.
In the code above, the expiration for cache_it is set to 1,137,750 seconds (13 days, 4 hours, and 2.5 minutes), which means that every item created by cache_it has a lifetime of 1,137,750 seconds, beginning when the item is made (i.e not beginning when cache_it is made). Values specifying lifetime may be provided with the keywords years, months, weeks, days, hours, minutes, and seconds. The lifetime is the total for all keywords.
If these optional keyword arguments are not included, then items added by the pyfscache.FSCache object never expire:
no_expiry_cache = pyfscache.FSCache('some/cache/directory')
Note
Several instances of pyfscache.FSCache objects can use the same cache directory. Each will honor the expirations of the items therein. Thus, it is possible to have a cache mixed with objects of many differening lifetimes, made by many instances of pyfscache.FSCache.
Instances of phyles.FSCache work like mapping objects, supporting item getting and setting:
>>> cache_it[('some', ['key'])] = {'some': 'value'}
>>> cache_it[('some', ['key'])]
{'some': 'value}
However, deletion with the del statement only works on memory. To erase an item in the cache directory, use phyles.FSCache.expire():
>>> cache_it.get_loaded()
['LIlWpBZL68MBJaXouRjFBL3fzScyxh5q56hqSZ3DBK']
>>> del cache_it[('some', ['key'])]
>>> cache_it.get_loaded()
[]
>>> ('some', ['key']) in cache_it
True
>>> cache_it[('some', ['key'])]
{'some': 'value}
>>> cache_it.expire(('some', ['key']))
>>> ('some', ['key']) in cache_it
False
What if you didn’t write the function you want to cache? Although their convenience is manifest in the example above, it is not necessary to use decorators:
import pyfscache
cache = pyfscache.FSCache('some/cache/directory',
days=13, hours=4, minutes=2.5)
def uncached_doit(a, b, c):
return [a, b, c]
cached_doit = cache(uncached_doit)
phyles.FSCache objects should work on the vast majority of python “callables”, including instance methods and even built-ins:
# a cached built-in
cached_list = cache_it(list)
# a cached instance method
class AClass(object):
@cahe_it
def some_cached_instance_method(self, a, r, g, s):
return (a + r) / (g * s)
Note
The rule of thumb is that if python’s cPickle module can handle the expected arguments to the cached function, then so can pyfscache.
pyfscache: A file system cache for python. Copyright (c) 2013, James C. Stroud; All rights reserved.
A class that manages a filesystem cache. Works like a dictionary and can decorate functions to make them cached.
A pyfscache.FSCache object is instantiated with a path and optional lifetime keyword arguments:
>>> c = FSCache('cache/dir', days=7)
This command creates a new FSCache instance at the given path (cache/dir). Each item added by this cache has a lifetime of 7 days, starting when the item (not the cache) is created. If the path doesn’t exist, one is made. New items added to the cache are given a lifetime expressed by the keyword arguments with potential keys of years, months, weeks, days, hours, minutes, seconds (see to_seconds()). If no keyword arguments are given, then the items added by the cache do not expire automatically.
Creating an pyfscache.FSCache object does not purge the cache in path if the cache already exists. Instead, the pyfscache.FSCache object will begin to use the cache, loading items and storing items as necessary.
>>> import os
>>> import shutil
>>> from pyfscache import *
>>> if os.path.exists('cache/dir'):
... shutil.rmtree('cache/dir')
...
>>> c = FSCache('cache/dir', days=7)
>>> c['some_key'] = "some_value"
>>> c['some_key']
'some_value'
>>> os.listdir('cache/dir')
['PXBZzwEy3XnbOweuMtoPj9j=PwkfAsTXexmW2v05JD']
>>> c.expire('some_key')
>>> os.listdir('cache/dir')
[]
>>> c['some_key'] = "some_value"
>>> @c
... def doit(avalue):
... print "had to call me!"
... return "some other value"
...
>>> doit('some input')
had to call me!
'some other value'
>>> doit('some input')
'some other value'
>>> shutil.rmtree('cache/dir')
Unloads all loaded cache items from memory. All cache items remain on the disk, however.
Use with care. This permanently removes the object keyed by k from the cache, both in the memory and in the filesystem.
Returns an expiry for the cache in seconds as if the start of the expiration period were the moment at which this the method is called.
>>> import time
>>> c = FSCache('cache/dir', seconds=60)
>>> round(c.expiry() - time.time(), 3)
60.0
Returns the lifetime, in seconds, of new items in the cache. If new items do not expire, then None is returned.
Returns a list of keys for all objects that are loaded.
Returns the names of the files in the cache on the filesystem. These are not keys but one-way hashes (or “digests”) of the keys created by make_digest().
Returns the absolute path to the file system cache represented by the instance.
Returns True if the item keyed by k has been loaded, False if not.
Returns the lifetime, in seconds, of new items in the cache. If new items do not expire, then None is returned.
Causes the object keyed by k to be loaded from the file system and returned. It therefore causes this object to reside in memory.
Returns the absolute path to the file system cache represented by the instance.
Be careful, this empties the cache from both the filesystem and memory!
Removes the object keyed by k from memory but not from the filesystem. To remove it from both memory and permanently from the filesystem, use expire.
Use with care. Updates, both in memory and on the filesystem, the object for key k with the object v. If the key k already exists with a stored object, it will be replaced.
Creates a digest suitable for use within an phyles.FSCache object from the key object k.
>>> adict = {'a' : {'b':1}, 'f': []}
>>> make_digest(adict)
'a2VKynHgDrUIm17r6BQ5QcA5XVmqpNBmiKbZ9kTu0A'
Creates a cached function from function f. The cache can be any mapping object, such as FSCache objects.
The function arguments are expected to be well-behaved for python’s cPickle. Or, in other words, the expected values for the parameters (the arguments) should be instances new-style classes (i.e. inheriting from object) or implement __getstate__() with well-behaved results.
If the arguments to f are not expected to be well-behaved, it is best to use cache_function instead and create a custom keyer.
Takes any function f and a function that creates a key, keyer and caches the result in cache.
The keys created by keyer should be well behaved for python’s cPickle. See the documentation for auto_cache_funtion() for details.
It is best to have a unique keyer for every function.
Converts keyword arguments to seconds.
The the keyword arguments can have the following keys:
- years (31,556,900 seconds per year)
- months (2,629,740 seconds per month)
- weeks (604,800 seconds per week)
- days (86,400 seconds per day)
- hours (3600 seconds per hour)
- minutes (60 seconds per minute)
- seconds
>>> to_seconds(seconds=15, minutes=20)
1215.0
>>> to_seconds(seconds=15.42, hours=10, minutes=18, years=2)
63150895.42