Tutorial - Tracking Class Instances in SCons¶
This tutorial demonstrates the class tracking facility to profile and optimize a non-trivial program. SCons is a next-generation build system with a quite elaborate architecture and thus an interesting candidate for profiling attempts.
Before we begin, it should be identified what shall be tracked, i.e. what
classes we want to connect to and whose instances are to be sized and profiled.
In this tutorial, the effect of a patch is analyzed that tries to size-optimize the very
heart of SCons - the Node
class. Naturally, we will connect to the Node
base
class and its sub-classes. It makes sense to put the profiling data
in context and track additional classes that are believed to contribute
significantly to the total memory consumption.
Installing hooks into SCons¶
The first step is to find the proper spots for connecting to the classes that shall be tracked, taking snapshots, and printing the gathered profile data. SCons has a simple memory profiling tool that we will override. The SCons MemStats class provides all we need:
from pympler.classtracker import ClassTracker
class MemStats(Stats):
def __init__(self):
Stats.__init__(self)
classes = [SCons.Node.Node, SCons.Node.FS.Base, SCons.Node.FS.File,
SCons.Node.FS.Dir, SCons.Executor.Executor]
self.tracker = ClassTracker()
for c in classes:
self.tracker.track_class(c)
def do_append(self, label):
self.tracker.create_snapshot(label)
def do_print(self):
stats = self.tracker.stats
stats.print_summary()
stats.dump_stats('pympler.stats')
When SCons starts, MemStats
is instantiated and the ClassTracker is
connected to a number of classes. SCons has predefined spots where it invokes
its statistics facilities with do_append
being called. This is where
snapshosts will be taken of all objects tracked so far.
Because of the large number of instances, only a summary is printed to the
console via stats.print_summary()
and the profile data is dumped to a file
in case per-instance profile information is needed later.
Test run¶
Time for a test. In the following examples, SCons builds a non-trivial program
with a fair number of nodes. Running SCons via scons --debug=memory
will
print the gathered data to the console:
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
---- SUMMARY ------------------------------------------------------------------
before reading SConscript files: active 4.17 MB average pct
SCons.Executor.Executor 7 7.53 KB 1.08 KB 0%
SCons.Node.FS.Base 1 9.30 KB 9.30 KB 0%
SCons.Node.FS.Dir 6 17.77 KB 2.96 KB 0%
SCons.Node.FS.File 1 2.91 KB 2.91 KB 0%
SCons.Node.Node 0 0 B 0 B 0%
after reading SConscript files: active 13.06 MB average pct
[...]
before building targets: active 13.41 MB average pct
[...]
after building targets: active 34.77 MB average pct
SCons.Executor.Executor 1311 3.57 MB 2.79 KB 10%
SCons.Node.FS.Base 1102 4.84 MB 4.50 KB 13%
SCons.Node.FS.Dir 108 5.67 MB 53.72 KB 16%
SCons.Node.FS.File 2302 10.45 MB 4.65 KB 30%
SCons.Node.Node 1 84.93 KB 84.93 KB 0%
-------------------------------------------------------------------------------
Making sense of the data¶
The console output may give a brief overview how much memory is allocated by
instances of the individual tracked classes. A more appealing and well arranged
representation of the data can be generated with the HtmlStats
class. The
dump generated previously can be loaded and a set of HTML pages can be emitted:
from pympler.classtracker_stats import HtmlStats
stats = HtmlStats()
stats.load_stats('pympler.stats')
stats.create_html('pympler.html')
If matplotlib
is installed, charts will be embedded in the HTML output:
At first sight it might seem suspicious that the tracked classes appear to be the sole contributors to the total memory footprint of the application. Because the tracked objects are sized recursively, referenced objects which are not tracked themselves are added to the referrers account. Thus, a root object’s size will include the size of every leaf unless the leaf is also tracked by the ClassTracker.
Optimization attempt¶
After applying the patch by Jean Brouwers, SCons is rerun under the supervision
of the ClassTracker. The differences in the last snapshot show that the
changes indeed reduce the memory footprint of Node
instances:
$ scons --debug=memory
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
---- SUMMARY ------------------------------------------------------------------
[...]
after building targets: active 32.41 MB average pct
SCons.Executor.Executor 1311 3.50 MB 2.73 KB 10%
SCons.Node.FS.Base 1102 4.29 MB 3.98 KB 13%
SCons.Node.FS.Dir 108 5.52 MB 52.30 KB 17%
SCons.Node.FS.File 2302 8.82 MB 3.92 KB 27%
SCons.Node.Node 1 84.32 KB 84.32 KB 0%
-------------------------------------------------------------------------------
The total measured memory footprint dropped from 34.8MB to 32.4MB, File
nodes’ average size from 4.6KB to 3.9KB.
Summary¶
This tutorial illustrated how applications can be profiled with the ClassTracker facility. It has been shown how the memory impact of changes can be quantified.