Tutorial - Tracking Class Instances in SCons

This tutorial demonstrates the class tracking facility to profile and optimize a non-trivial program. SCons is a next-generation build system with a quite elaborate architecture and thus an interesting candidate for profiling attempts.

Before we begin, it should be identified what shall be tracked, i.e. what classes we want to connect to and whose instances are to be sized and profiled. In this tutorial, the effect of a patch is analyzed that tries to size-optimize the very heart of SCons - the Node class. Naturally, we will connect to the Node base class and its sub-classes. It makes sense to put the profiling data in context and track additional classes that are believed to contribute significantly to the total memory consumption.

Installing hooks into SCons

The first step is to find the proper spots for connecting to the classes that shall be tracked, taking snapshots, and printing the gathered profile data. SCons has a simple memory profiling tool that we will override. The SCons MemStats class provides all we need:

from pympler.classtracker import ClassTracker

class MemStats(Stats):
    def __init__(self):
        Stats.__init__(self)
        classes = [SCons.Node.Node, SCons.Node.FS.Base, SCons.Node.FS.File,
                   SCons.Node.FS.Dir, SCons.Executor.Executor]
        self.tracker = ClassTracker()
        for c in classes:
            self.tracker.track_class(c)
    def do_append(self, label):
        self.tracker.create_snapshot(label)
    def do_print(self):
        stats = self.tracker.stats
        stats.print_summary()
        stats.dump_stats('pympler.stats')

When SCons starts, MemStats is instantiated and the ClassTracker is connected to a number of classes. SCons has predefined spots where it invokes its statistics facilities with do_append being called. This is where snapshosts will be taken of all objects tracked so far.

Because of the large number of instances, only a summary is printed to the console via stats.print_summary() and the profile data is dumped to a file in case per-instance profile information is needed later.

Test run

Time for a test. In the following examples, SCons builds a non-trivial program with a fair number of nodes. Running SCons via scons --debug=memory will print the gathered data to the console:

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
---- SUMMARY ------------------------------------------------------------------
before reading SConscript files:         active      4.17 MB      average   pct
  SCons.Executor.Executor                     7      7.53 KB      1.08 KB    0%
  SCons.Node.FS.Base                          1      9.30 KB      9.30 KB    0%
  SCons.Node.FS.Dir                           6     17.77 KB      2.96 KB    0%
  SCons.Node.FS.File                          1      2.91 KB      2.91 KB    0%
  SCons.Node.Node                             0      0     B      0     B    0%
after reading SConscript files:          active     13.06 MB      average   pct
  [...]
before building targets:                 active     13.41 MB      average   pct
  [...]
after building targets:                  active     34.77 MB      average   pct
  SCons.Executor.Executor                  1311      3.57 MB      2.79 KB   10%
  SCons.Node.FS.Base                       1102      4.84 MB      4.50 KB   13%
  SCons.Node.FS.Dir                         108      5.67 MB     53.72 KB   16%
  SCons.Node.FS.File                       2302     10.45 MB      4.65 KB   30%
  SCons.Node.Node                             1     84.93 KB     84.93 KB    0%
-------------------------------------------------------------------------------

Making sense of the data

The console output may give a brief overview how much memory is allocated by instances of the individual tracked classes. A more appealing and well arranged representation of the data can be generated with the HtmlStats class. The dump generated previously can be loaded and a set of HTML pages can be emitted:

from pympler.classtracker_stats import HtmlStats

stats = HtmlStats()
stats.load_stats('pympler.stats')
stats.create_html('pympler.html')

If matplotlib is installed, charts will be embedded in the HTML output:

../_images/classtracker_timespace.png

At first sight it might seem suspicious that the tracked classes appear to be the sole contributors to the total memory footprint of the application. Because the tracked objects are sized recursively, referenced objects which are not tracked themselves are added to the referrers account. Thus, a root object’s size will include the size of every leaf unless the leaf is also tracked by the ClassTracker.

Optimization attempt

After applying the patch by Jean Brouwers, SCons is rerun under the supervision of the ClassTracker. The differences in the last snapshot show that the changes indeed reduce the memory footprint of Node instances:

$ scons --debug=memory
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.
---- SUMMARY ------------------------------------------------------------------
[...]
after building targets:                  active     32.41 MB      average   pct
  SCons.Executor.Executor                  1311      3.50 MB      2.73 KB   10%
  SCons.Node.FS.Base                       1102      4.29 MB      3.98 KB   13%
  SCons.Node.FS.Dir                         108      5.52 MB     52.30 KB   17%
  SCons.Node.FS.File                       2302      8.82 MB      3.92 KB   27%
  SCons.Node.Node                             1     84.32 KB     84.32 KB    0%
-------------------------------------------------------------------------------

The total measured memory footprint dropped from 34.8MB to 32.4MB, File nodes’ average size from 4.6KB to 3.9KB.

Summary

This tutorial illustrated how applications can be profiled with the ClassTracker facility. It has been shown how the memory impact of changes can be quantified.