Sizing individual objects

Introduction

This module exposes 9 functions and 2 classes to obtain lengths and sizes of Python objects (for Python 2.6 or later).

Earlier versions of this module supported Python versions down to Python 2.2. If you are running Python < 2.5 please consider downgrading Pympler to version 0.3.x.

Public Functions [1]

Function asizeof calculates the combined (approximate) size of one or several Python objects in bytes. Function asizesof returns a tuple containing the (approximate) size in bytes for each given Python object separately. Function asized returns for each object an instance of class Asized containing all the size information of the object and a tuple with the referents.

Functions basicsize and itemsize return the basic resp. item size of the given object.

Function flatsize returns the flat size of a Python object in bytes defined as the basic size plus the item size times the length of the given object.

Function leng returns the length of an object, like standard len but extended for several types, e.g. the leng of a multi-precision int (or long) is the number of digits [2]. The length of most mutable sequence objects includes an estimate of the over-allocation and therefore, the leng value may differ from the standard len result.

Function refs returns (a generator for) the referents of the given object, i.e. the objects referenced by the given object.

Certain classes are known to be sub-classes of or to behave as dict objects. Function adict can be used to install other class objects to be treated like dict.

Public Classes [1]

An instance of class Asized is returned for each object sized with the asized function or method.

Class Asizer may be used to accumulate the results of several asizeof or asizesof calls. After creating an Asizer instance, use methods asizeof and asizesof as needed to size any number of additional objects.

Call methods exclude_refs and/or exclude_types to exclude references to resp. instances or types of certain objects. Use one of the print_... methods to report the statistics.

Duplicate Objects

Any duplicate, given objects are sized only once and the size is included in the combined total only once. But functions asizesof and asized do return a size value resp. an Asized instance for each given object, including duplicates.

Definitions [3]

The size of an object is defined as the sum of the flat size of the object plus the sizes of any referents. Referents are visited recursively up to a given limit. However, the size of objects referenced multiple times is included only once.

The flat size of an object is defined as the basic size of the object plus the item size times the number of allocated items. The flat size does include the size for the items (references to the referents), but not the referents themselves.

The flat size returned by function flatsize equals the result of the asizeof function with options code=True, ignored=False, limit=0 and option align set to the same value.

The accurate flat size for an object is obtained from function sys.getsizeof() where available. Otherwise, the length and size of sequence objects as dicts, lists, sets, etc. is based on an estimate for the number of allocated items. As a result, the reported length and size may substantially differ from the actual length and size.

The basic and item sizes are obtained from the __basicsize__ resp. __itemsize__ attributes of the (type of the) object. Where necessary (e.g. sequence objects), a zero __itemsize__ is replaced by the size of a corresponding C type. The basic size (of GC managed objects) objects includes the overhead for Python’s garbage collector (GC) as well as the space needed for refcounts (used only in certain Python builds).

Optionally, size values can be aligned to any power of 2 multiple.

Size of (byte)code

The (byte)code size of objects like classes, functions, methods, modules, etc. can be included by setting option code=True.

Iterators are handled similar to sequences: iterated object(s) are sized like referents, only if the recursion limit permits. Also, function gc.get_referents() must return the referent object of iterators.

Generators are sized as (byte)code only, but the generated objects are never sized.

Old- and New-style Classes

All old- and new-style class, instance and type objects, are handled uniformly such that (a) instance objects are distinguished from class objects and (b) instances of different old-style classes can be dealt with separately.

Class and type objects are represented as <class ....* def> resp. <type ... def> where an ‘*’ indicates an old-style class and the def suffix marks the definition object. Instances of old-style classes are shown as new-style ones but with an ‘*’ at the end of the name, like <class module.name*>.

Ignored Objects

To avoid excessive sizes, several object types are ignored [3] by default, e.g. built-in functions, built-in types and classes [4], function globals and module referents. However, any instances thereof are sized and module objects will be sized when passed as given objects. Ignored object types are included if option ignored is set accordingly.

In addition, many __...__ attributes of callable objects are ignored [3], except crucial ones, e.g. class attributes __dict__, __doc__, __name__ and __slots__. For more details, see the type-specific _..._refs() and _len_...() functions below.

Footnotes

[1]	(1, 2) The functions and classes in this module are not thread-safe.

[2]	See Python source file .../Include/longinterp.h for the C typedef of digit used in multi-precision int (or long) objects. The C sizeof(digit) in bytes can be obtained in Python from the int (or long) __itemsize__ attribute. Function leng determines the number of digits of an int (or long) value.

[3]	(1, 2, 3) These definitions and other assumptions are rather arbitrary and may need corrections or adjustments.

[4]	Types and classes are considered built-in if the __module__ of the type or class is listed in _builtin_modules below.

Asizer

class pympler.asizeof.Asized(size, flat, refs=(), name=None)

Store the results of an asized object in these 4 attributes:

size – total size of the object

flat – flat size of the object

name – name or repr of the object

refs – tuple containing an Asized instance for each referent

class pympler.asizeof.Asizer(**opts)

Sizer state and options.

asized(*objs, **opts)

Size each object and return an Asized instance with size information and referents up to the given detail level (and with modified options, see method set).

If only one object is given, the return value is the Asized instance for that object.

asizeof(*objs, **opts)

Return the combined size of the given objects (with modified options, see method set).

asizesof(*objs, **opts)

Return the individual sizes of the given objects (with modified options, see method set).

exclude_refs(*objs)

Exclude any references to the specified objects from sizing.

While any references to the given objects are excluded, the objects will be sized if specified as positional arguments in subsequent calls to methods asizeof and asizesof.

exclude_types(*objs)

Exclude the specified object instances and types from sizing.

All instances and types of the given objects are excluded, even objects specified as positional arguments in subsequent calls to methods asizeof and asizesof.

print_profiles(w=0, cutoff=0, **print3opts)

Print the profiles above cutoff percentage.

w=0 – indentation for each line

cutoff=0 – minimum percentage printed

print3options – print options, ala Python 3.0

print_stats(objs=(), opts={}, sized=(), sizes=(), stats=3.0, **print3opts)

Print the statistics.

w=0 – indentation for each line

objs=() – optional, list of objects

opts={} – optional, dict of options used

sized=() – optional, tuple of Asized instances returned

sizes=() – optional, tuple of sizes returned

stats=0.0 – print stats, see function asizeof

print3options – print options, as in Python 3.0

print_summary(w=0, objs=(), **print3opts)

Print the summary statistics.

w=0 – indentation for each line

objs=() – optional, list of objects

print3options – print options, as in Python 3.0

print_typedefs(w=0, **print3opts)

Print the types and dict tables.

w=0 – indentation for each line

print3options – print options, as in Python 3.0

reset(align=8, clip=80, code=False, derive=False, detail=0, ignored=True, infer=False, limit=100, stats=0, stream=None)

Reset options, state, etc.

The available options and default values are:

align=8 – size alignment

clip=80 – clip repr() strings

code=False – incl. (byte)code size

derive=False – derive from super type

detail=0 – Asized refs level

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0.0 – print statistics, see function asizeof

stream=None – output stream for printing

See function asizeof for a description of the options.

set(align=None, code=None, detail=None, limit=None, stats=None)

Set some options. See also reset.

align – size alignment

code – incl. (byte)code size

detail – Asized refs level

limit – recursion limit

stats – print statistics, see function asizeof

Any options not set remain unchanged from the previous setting.

Public Functions

pympler.asizeof.adict(*classes)

Install one or more classes to be handled as dict.

pympler.asizeof.asized(*objs, **opts)

Return a tuple containing an Asized instance for each object passed as positional argument using the following options.

align=8 – size alignment

clip=80 – clip repr() strings

code=False – incl. (byte)code size

derive=False – derive from super type

detail=0 – Asized refs level

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0.0 – print statistics

If only one object is given, the return value is the Asized instance for that object. Otherwise, the length of the returned tuple matches the number of given objects.

Set detail to the desired referents level (recursion depth).

See function asizeof for descriptions of the other options.

pympler.asizeof.asizeof(*objs, **opts)

Return the combined size in bytes of all objects passed as positional arguments.

The available options and defaults are the following.

align=8 – size alignment

all=False – all current objects

clip=80 – clip repr() strings

code=False – incl. (byte)code size

derive=False – derive from super type

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0.0 – print statistics

Set align to a power of 2 to align sizes. Any value less than 2 avoids size alignment.

All current module, global and stack objects are sized if all is True and if no positional arguments are supplied.

A positive clip value truncates all repr() strings to at most clip characters.

The (byte)code size of callable objects like functions, methods, classes, etc. is included only if code is True.

If derive is True, new types are handled like an existing (super) type provided there is one and only of those.

By default certain base types like object, super, etc. are ignored. Set ignored to False to include those.

If infer is True, new types are inferred from attributes (only implemented for dict types on callable attributes as get, has_key, items, keys and values).

Set limit to a positive value to accumulate the sizes of the referents of each object, recursively up to the limit. Using limit=0 returns the sum of the flat[4] sizes of the given objects. High limit values may cause runtime errors and miss objects for sizing.

A positive value for stats prints up to 8 statistics, (1) a summary of the number of objects sized and seen, (2) a simple profile of the sized objects by type and (3+) up to 6 tables showing the static, dynamic, derived, ignored, inferred and dict types used, found resp. installed. The fractional part of the stats value (x100) is the cutoff percentage for simple profiles.

[4] See the documentation of this module for the definition of flat size.

pympler.asizeof.asizesof(*objs, **opts)

Return a tuple containing the size in bytes of all objects passed as positional arguments using the following options.

align=8 – size alignment

clip=80 – clip repr() strings

code=False – incl. (byte)code size

derive=False – derive from super type

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0.0 – print statistics

See function asizeof for a description of the options.

The length of the returned tuple equals the number of given objects.

pympler.asizeof.basicsize(obj, **opts)

Return the basic size of an object (in bytes).

Valid options and defaults are

derive=False – derive type from super type

infer=False – try to infer types

save=False – save typedef if new

pympler.asizeof.flatsize(obj, align=0, **opts)

Return the flat size of an object (in bytes), optionally aligned to a given power of 2.

See function basicsize for a description of the other options. See the documentation of this module for the definition of flat size.

pympler.asizeof.itemsize(obj, **opts)

Return the item size of an object (in bytes).

See function basicsize for a description of the options.

pympler.asizeof.leng(obj, **opts)

Return the length of an object (in items).

See function basicsize for a description of the options.

pympler.asizeof.refs(obj, **opts)

Return (a generator for) specific referents of an object.

See function basicsize for a description of the options.