API Tutorial

Introduction

Using Stalker along with Python is all about interacting with a database by using the Stalker Object Model (SOM). Stalker uses the powerful SQLAlchemy ORM.

This tutorial section let you familiarise with the Stalker Python API and Stalker Object Model (SOM). If you used SQLAlchemy before you will feel at home and if you aren’t you will see that it is fun dealing with databases with SOM.

Part I - Basics

Lets say that we just installed Stalker (as you are right now) and want to use Stalker in our first project.

The first thing we are going to learn about is how to connect to the database so we can enter information about our studio and the projects.

We are going to use a helper script to connect to the default database. Use the following command to connect to the database:

from stalker import db
db.setup({"sqlalchemy.url": "sqlite:///"})

This will create an in-memory SQLite3 database, which is useless other than testing purposes. To be able to get more out of Stalker we should give a proper database information. The most basic setup is to use a file based SQLite3 database:

db.setup({"sqlalchemy.url": "sqlite:///C:/studio.db"}) # assumed Windows

or:

db.setup({"sqlalchemy.url": "sqlite:////home/ozgur/studio.db"}) # under linux or osx

Note

Although with Stalker v0.2.18 the SQLite3 support is dropped, Stalker can still work with an SQLite3 database. But the suggested database backend is PostgreSQL (preferably PostgreSQL 9.5).

Then if this is the first time you are connecting to the database, then you should initialize the database to create some default data:

db.init()

This will create some very important default data required for Stalker to work properly. Although it will not break anything to call db.init() multiple times it is needed only once (so you don’t need to call it again when you close your python shell and open up a new and fresh one).

Lets continue by creating a Studio for our self:

from stalker import Studio
my_studio = Studio(
    name='My Great Studio'
)

For now don’t care what a Studio is about. It will be explained later on this tutorial.

Lets continue by creating a User for ourselves in the database. The first thing we need to do is to import the User class in to the current namespace:

from stalker import User

then create the User object:

me = User(
    name="Erkan Ozgur Yilmaz",
    login="eoyilmaz",
    email="some_email_address@gmail.com",
    password="secret",
    description="This is me"
)

Now we have just created a user which represents us.

Lets create a new Department to define your department:

from stalker import Department
tds_department = Department(
    name="TDs",
    description="This is the TDs department"
)

Now add your user to the department:

tds_department.users.append(me)

or we can do it by using the User instance:

me.departments.append(tds_department)

Even if you didn’t do the latter, when you run:

print(me.departments)
# you should get something like
# [<TDs (Department)>]

We have successfully created a User and a Department and we assigned the user as one of the member of the TDs Department.

Because we didn’t tell Stalker to commit the changes, no data has been saved to the database yet. So lets send it the data to the database:

from stalker.db.session import DBSession
DBSession.add(my_studio)
DBSession.add(me)
DBSession.add(tds_department)
DBSession.commit()

As you see we have used the DBSession object to send (commit) the data to the database. These information are stored in the database right now.

Lets try to get something back from the database by querying all the departments, then getting the second one (the first department is always the “admins” which is created by default) and getting its first members name:

all_departments = Department.query.all()
print(all_departments)
# This should print something like
# [<admins (Department)>, <TDs (Department)>]
# "admins" department is created by default

admins = all_departments[0]
tds = all_departments[1]

all_users = tds.users  # Department.users is a synonym for Department.members
                       # they are essentially the same attribute
print(all_users[0])
# this should print
# <Erkan Ozgur Yilmaz ('eoyilmaz') (User)>

Part II/A - Creating Simple Data

Lets say that we have this new commercial project coming and you want to start using Stalker with it. So we need to create a Project object to hold data about it.

A project instance needs to have a suitable StatusList (see Statuses and Status Lists) and a Repository instance:

# we will reuse the Statuses created by default (in db.init())
from stalker import Status

status_new = Status.query.filter_by(code='NEW').first()
status_wip = Status.query.filter_by(code='WIP').first()
status_cmpl = Status.query.filter_by(code='CMPL').first()

Note

When the Stalker database is first initialized (with db.init()) a set of Statuses for Tasks, Assets, Shots, Sequences and Tickets are created along with a StatusList for each of the data types. Up to this point in the tutorial we have used those Statuses (new, wip and cmpl) that are created by default.

For now we have just created generic statuses. These Status instances can be used with any kind of statusable objects. The idea behind is to define the statuses only once, and use them in mixtures suitable for different type of objects. So you can define all the possible Statuses for your entities, then you can create a list of them for specific type of objects.

Lets create a StatusList suitable for Project instances:

# a status list which is suitable for Project instances
from stalker import StatusList, Project

project_statuses = StatusList(
    name="Project Status List",
    statuses=[
        status_new,
        status_wip,
        status_cmpl
    ],
    target_entity_type='Project'  # you can also use Project which is the
                                  # class itself
)

So we defined a status list which is suitable for Project instances. As you see we didn’t used all the generic Statuses in our project_statuses because for a Project object we thought that these statuses are enough.

And finally, the Repository. The Repository (or Repo if you like) is a path in our file server, where we place files and which is visible to all the workstations/render farmers:

from stalker import Repository

# and the repository itself
commercial_repo = Repository(
    name="Commercial Repository"
)

Repository class will be explained in detail in upcoming sections.

So:

new_project = Project(
    name="Fancy Commercial",
    code='FC',
    status_list=project_statuses,
    repositories=[commercial_repo],
)

So we have created our project now.

Lets enter more information about this new project:

import tzlocal
import datetime
from stalker import ImageFormat

new_project.description = \
"""The commercial is about this fancy product. The
client want us to have a shiny look with their
product bla bla bla..."""

new_project.image_format = ImageFormat(
    name="HD 1080",
    width=1920,
    height=1080
)

new_project.fps = 25
local_tz = tzlocal.get_localzone()
new_project.end = datetime.datetime(2014, 5, 15, tzinfo=local_tz)
new_project.users.append(me)

Lets save all the new data to the database:

DBSession.add(new_project)
DBSession.commit()

As you see, even though we have created multiple objects (new_project, statuses, status lists etc.) we’ve just added the new_project object to the database, but don’t worry all the related objects will be added to the database.

Note

Starting with Stalker v0.2.18 all the datetime information needs to have timezone information (we’ve used the local timezone in the example).

A Project generally is group of Tasks that needs to be completed. A Task in Stalker is a type of entity where we define the total amount of effort need to be done (or the duration or the length of the task, see Task class documentation) to consider that Task as completed. All of the tasks (leaf tasks in fact, coming next) has resources which defines the Users who need to work on that task and complete it. These are all explained in Task class documentation.

For now you just need to now that Assets, Shots and Sequences in Stalker are derived from Task and they are in fact other type of Tasks or a specialized version of Tasks.

So lets create a Sequence:

from stalker import Sequence

seq1 = Sequence(
    name="Sequence 1",
    code="SEQ1",
    project=new_project,
)

And a Sequence generally has Shots:

from stalker import Shot

sh001 = Shot(
    name='SH001',
    code='SH001',
    project=new_project,
    sequences=[seq1]
)
sh002 = Shot(
    code='SH002',
    project=new_project,
    sequences=[seq1]
)
sh003 = Shot(
    code='SH003',
    project=new_project,
    sequences=[seq1]
)

send them to the database:

DBsession.add_all([sh001, sh002, sh003])
DBsession.commit()

Note

Even though, in this tutorial we have created Shots with one Sequence instance, it is not needed. You can create Shots without any Sequence instance needed.

For small projects like commercials, you may skip creating a Sequence at all.

For bigger projects, like feature films, it is a very good idea to use Sequences and then group the Shots under them.

But again, a Shot can be connected to multiple sequences, which is useful if your shot, let say, is a kind of flashback and you will use this shot again without changing it at all, then this feature becomes handy.

Part II/B - Querying, Updating and Deleting Data

So far we just created some simple data. What about updating them. Let say that we created a new shot with wrong info:

sh004 = Shot(
    code='SH004',
    project=new_project,
    sequences=[seq1]
)
DBSession.add(sh004)
DBSession.commit()

and you figured out that you have created and committed a wrong info and you want to correct it:

sh004.code = "SH005"
DBsession.commit()

later on lets say you wanted to get the shot back from database:

# first find the data
wrong_shot = Shot.query.filter_by(code="SH005").first()

# now update it
wrong_shot.code = "SH004"

# commit the changes to the database
DBsession.commit()

and let say that you decided to delete the data:

DBsession.delete(wrong_shot)
DBsession.commit()

If you don’t close your python session, your variable are still going to contain the data but they do not exist in the database anymore:

wrong_shot = Shot.query.filter_by(code="SH005").first()
print(wrong_shot)
# should print None

for more info about update and delete options (like cascades) in SQLAlchemy please see the SQLAlchemy documentation.

Part III - Pipeline

Up until now, we skipped a lot of stuff here to take little steps every time. Even tough we have created users, departments, projects, sequences and shots, Stalker still doesn’t know much about our studio. For example, it doesn’t have any information about the pipeline that we are following and what steps we do to complete those shots, thus to complete the project.

In Stalker, pipeline is managed by Tasks. So you create Tasks for Shots and then you can create dependencies between tasks.

So lets create a couple of tasks for one of the shots we have created before:

from stalker import Task

previs = Task(
    name="Previs",
    parent=sh001
)

matchmove = Task(
    name="Matchmove",
    parent=sh001
)

anim = Task(
    name="Animation",
    parent=sh001
)

lighting = Task(
    name="Lighting",
    parent=sh001
)

comp = Task(
    name="comp",
    parent=sh001
)

Now create the dependencies between them:

comp.depends = [lighting]
lighting.depends = [anim]
anim.depends = [previs, matchmove]

Stalker uses this dependency relation in scheduling these tasks. That is by appending “lighting” task as one of the dependencies of comp, Stalker now know that lighting should be completed to let the resource of the comp task start working. The “Task Scheduling” will be explained in detail later on in this tutorial.

Part IV - Task & Resource Management

Now we have a couple of Shots with couple of tasks inside it but we didn’t assign the tasks to anybody to let them complete this job.

Lets assign all this stuff to our self (for now :) ):

previs.resources = [me]
previs.schedule_timing = 10
previs.schedule_unit = 'd'

matchmove.resources = [me]
matchmove.schedule_timing = 2
matchmove.schedule_unit = 'd'

anim.resources = [me]
anim.schedule_timing = 5
anim.schedule_unit = 'd'

lighting.resources = [me]
lighting.schedule_timing = 3
lighting.schedule_unit = 'd'

comp.resources = [me]
comp.schedule_timing = 6
comp.schedule_unit = 'h'

Now Stalker knows the hierarchy of the tasks and how much effort is needed to complete this tasks. Stalker will use this information to solve the Scheduling problem, and will tell you when to start and complete this tasks.

Lets commit the changes again:

DBsession.commit()

If you noticed, this time we didn’t add anything to the session, cause we have added the sh001 in a previous commit, and because all the objects are attached to this shot object in some way, all the changes has been tracked and added to the database.

Part V - Scheduling

In previous sections of this tutorial we have created a Shot and then created a couple of Tasks to this shot and then assigned our self as the resource of these tasks.

Stalker knows enough about our little project now, but we don’t know where to start the project from. That is which task should we start from.

In Stalker, defining the start and end dates of a Task (also of an Asset, Shot and Sequence) is called “Scheduling”. Stalker, with the help of TaskJuggler, can solve this problem and define when the resource should work on a specific task.

Warning

You should have TaskJuggler installed in your system, and you should have configured your Stalker installation to be able to find the tj3 executable.

On a linux system this should be fairly straight forward, just install TaskJuggler and stalker will be able to use it.

But for other OSes, like OSX and Windows, you should create an environment variable called STALKER_PATH and then place a file called config.py inside the folder that this path is pointing at. And then add the following to this config.py:

tj_command = 'C:\\Path\\to\\tj3.exe'

The default value for tj_command config variable is /usr/local/bin/tj3, so if on a Linux or OSX system when you run:

which tj3

is returning this value (/usr/local/bin/tj3) you don’t need to setup anything.

So, lets schedule our project by using the Studio instance that we have created at the beginning of this tutorial:

from stalker import TaskJugglerScheduler

my_studio.scheduler = TaskJugglerScheduler()
my_studio.duration = datetime.timedelta(days=365)  # we are setting the
my_studio.schedule(scheduled_by=me)                # duration to 1 year just
                                                   # to be sure that TJ3
                                                   # will not complain
                                                   # about the project is not
                                                   # fitting in to the time
                                                   # frame.
DBsession.commit()  # to reflect the change

This should take a little while depending to your projects size (around 1-2 seconds for this tutorial, but around ~15 min for a project with 15000+ tasks).

When it is finished all of your tasks now have their computed_start and computed_end values filled with proper data. Now check the start and end values:

print(previs.computed_start)     # 2014-04-02 16:00:00
print(previs.computed_end)       # 2014-04-15 15:00:00

print(matchmove.computed_start)  # 2014-04-15 15:00:00
print(matchmove.computed_end)    # 2014-04-17 13:00:00

print(anim.computed_start)       # 2014-04-17 13:00:00
print(anim.computed_end)         # 2014-04-23 17:00:00

print(lighting.computed_start)   # 2014-04-23 17:00:00
print(lighting.computed_end)     # 2014-04-24 11:00:00

print(comp.computed_start)       # 2014-04-24 11:00:00
print(comp.computed_end)         # 2014-04-24 17:00:00

The dates are probably going to be different in your computer. But as you see Stalker has computed the start and end date values for each of the tasks. They are simply following one other, this is because we have entered only one resource for each of the task.

You should know that “Scheduling” is a huge concept and it is greatly explained in TaskJuggler documentation.

For a last thing you can check the to_tjp values of each data we have created for now, so try running:

print(my_studio.to_tjp)
print(me.to_tjp)
print(comp.to_tjp)
print(new_project.to_tjp)

If you are familiar with TaskJuggler, you should recognize the output of each to_tjp variable. So essentially Stalker is mapping all of its data to a TaskJuggler compatible string. A very small part of TaskJuggler directives are currently supported. But it is enough to schedule very complex projects with complex dependency relation and Task hierarchies. And with every new version of Stalker the supported TaskJuggler directives are expanded.

Part VI - Asset Management

Now we have created a lot of things but other then storing all the data in the database, we didn’t do much. Stalker still doesn’t have information about a lot of things. For example, it doesn’t know how to handle your asset versions (Version) namely it doesn’t know how to store your data that you are going to create while completing these tasks.

So what we need to define is a place in our file structure. It doesn’t need to be a network shared directory but if you are not working alone than it means that everyone needs to reach your data and the simplest way to do this is to place your files in a network share, there are other alternatives like storing your files locally and sharing your revisions with a Software Configuration Management (SCM) system, Stalker doesn’t support the latter right now.

We are going to see the first alternative, which uses a network share in our fileserver, and this network share is called a Repository in Stalker.

A repository is a file path, preferably a path which is mapped or mounted to the same path on every computer in your studio (also you can use autofs with an OpenLDAP server in which you can synchronize all off the mount points on all of your workstations and render slaves at once).

In Stalker, you can have several repositories, let say one for Commercials and another one for each big Movie projects.

You can define repositories and assign projects to those repositories.

We have already created a repository while creating our first project. But the repository has missing information. A Repository object shows the path that we create our projects into. Lets enter the paths for all the major operating systems:

commercial_repo.linux_path   = "/mnt/M/commercials"
commercial_repo.osx_path     = "/Volumes/M/commercials"
commercial_repo.windows_path = "M:/commercials"  # you can use reverse
                                                 # slashes (\\) if you want

And if you ask for the path to a repository object it will always give the correct answer according to your operating system:

print(commercial_repo.path)
# under Windows outputs:
# M:/commercials
#
# in Linux and variants:
# /mnt/M/commercials
#
# and in OSX:
# /Volumes/M/commercials

Note

Stalker always uses forward slashes no matter what operating system you are using. It is like that even if you define your paths with reverse slashes (\).

Assigning this repository to our project is not enough, Stalker still doesn’t know about the directory structure of this project. To explain the project structure to Stalker we use a Structure instance:

from stalker import Structure

commercial_project_structure = Structure(
    name="Commercial Projects Structure"
)

# now assign this structure to our project
new_project.structure = commercial_project_structure

New in version 0.2.13: Starting with Stalker version 0.2.13 Project instances can have multiple Repository instances attached. So you can create complex templates where you can for example store published versions on a different server/network share or you can setup so the outputs of a version (like the rendered files) are stored on a different server, and etc.

The following examples are updated in a simple way and examples showing the advantage of having multiple repositories will be added on later versions.

Now we have created a very simple structure instance, but we still need to create FilenameTemplate instances for Tasks which then will be used by the Version instances to generate a consistent and meaningful path and filename:

from stalker import FilenameTemplate

task_template = FilenameTemplate(
    name='Task Template for Commercials',
    target_entity_type='Task',
    path='$REPO{{project.repository.id}}/{{project.code}}/{%- for p in parent_tasks -%}{{p.nice_name}}/{%- endfor -%}',
    filename='{{version.nice_name}}_v{{"%03d"|format(version.version_number)}}'
)

# and append it to our project structure
commercial_project_structure.templates.append(task_template)

# commit to database
DBsession.commit()  # no need to add anything, project is already on db

By defining a FilenameTemplate instance we have essentially told Stalker how to store Version instances created for Task entities in our Repository.

The data entered both to the path and filename arguments are Jinja2 directives. The Version class knows how to render these templates while calculating its path and filename attributes.

Also, if you noticed we have used an environment variable “$REPO” along with the id of the first repository in the project “{{project.repository.id}}” (attention! project.repository always shows the first repository in the project), this is a new feature introduced with Stalker version 0.2.13. Stalker creates environment variables on runtime for each of the repository whenever a repository is created and inserted in to the DB or it will create environment variables for already existing repositories upon a successful database connection.

Lets create a Version instance for one of our tasks:

from stalker import Version

vers1 = Version(
    task=comp
)

# we need to update the paths
vers1.update_paths()

# check the path and filename
print(vers1.path)                # '$REPO33/FC/SH001/comp'
print(vers1.filename)            # 'SH001_comp_Main_v001'
print(vers1.full_path)           # '$REPO33/FC/SH001/comp/SH001_comp_Main_v001'

# now the absolute values, values with repository root
# because I'm running this code in a Linux laptop, my results are using the
# linux path of the repository
print(vers1.absolute_path)       # '/mnt/M/commercials/FC/SH001/comp'
print(vers1.absolute_full_path)  # '/mnt/M/commercials/FC/SH001/comp/SH001_comp_Main_v001'

# check the version_number
print(vers1.version_number)      # 1

# commit to database
DBsession.commit()

As you see, the Version instance magically knows where to place itself and what to use as the filename. Thanks to Stalker it is now easy to create version files where you don’t have weird file names (ex: ‘Shot1_comp_Final’, ‘Shot1_comp_Final_revised’, ‘Shot1_comp_Final_revised_Final’, ‘Shot1_comp_Final_revised_Final_real_final’ and the list goes on, we all know those filenames don’t we :) ).

With Stalker the filename and path always follows strict rules.

Also by using the Version.is_published attribute you can define which of the versions are usable and which are versions that you are still working on:

vers1.is_published = False  # I still work on this version, this is not a
                            # usable one

Lets create another version for the same task and see what happens:

# be sure that you've committed the previous version to the database
# to let Stalker now what number to give for the next version
vers2 = Version(task=comp)
vers2.update_paths()  # this call probably will disappear in next version of
                      # Stalker, so Stalker will automatically update the
                      # paths on Version.__init__()

print(vers2.version_number)  # 2
print(vers2.filename)        # 'SH001_comp_Main_v002'

# before creating a new version commit this one to db
DBsession.commit()

# now create a new version
vers3 = Version(task=comp)
vers3.update_paths()

print(vers3.version_number)  # 3
print(vers3.filename)        # 'SH001_comp_Main_v002'

Isn’t that nice, Stalker increments the version number automatically.

Also you can query all the versions of a specific task by:

# using pure Python
vers_from_python = comp.versions  # [<FC_SH001_comp_Main_v001 (Version)>,
                                  #  <FC_SH001_comp_Main_v002 (Version)>,
                                  #  <FC_SH001_comp_Main_v003 (Version)>]

# or using a query
vers_from_query = Version.query.filter_by(task=comp).all()

# again returns
# [<FC_SH001_comp_Main_v001 (Version)>,
#  <FC_SH001_comp_Main_v002 (Version)>,
#  <FC_SH001_comp_Main_v003 (Version)>]

assert vers_from_python == vers_from_query

Note

Stalker stores Version.path and Version.filename attributes in the database, so the values does not contain any OS specific path. It will only show the OS specific path on Version.absolute_path and on Version.absolute_full_path attributes by joining the Repository.path with the path values from database momentarily.

You can also setup your project structure to have default directories:

commercial_project_structure.custom_template = """
Temp
References
References/Movies
References/Images
"""

When the above template is executed each line will refer to a directory.

Part VII - Collaboration (not completed)

We came a lot from the start, but what is the use of an Production Asset Management System if we can not communicate with our colleagues.

In Stalker you can communicate with others in the system, by:

• Leaving a Note to anything created in Stalker (except you can not create a Note to another Note and to a Tag).
• Sending a Message directly to them or to a group of users. (Not implemented yet).
• Anyone can create a Ticket for a Project.
• You can create wiki Pages per Project.

Part VIII - Extending SOM (coming)

This part will be covered soon

Conclusion

In this tutorial, you have nearly learned a quarter of what Stalker supplies as a Python library.

Stalker is a very flexible and powerful Production Asset Management system. As of writing this tutorial it has been developed for the last 5 years (4 years with the only developer being yours truly and for another 1 year where his wife is also attended to the project) and it is currently been used in production of a feature movie.

But it is only a Python library so it doesn’t supply any graphical user interface.

There are other projects, namely Stalker Pyramid and Anima that is using Stalker in their back ends. Stalker Pyramid is an Pyramid based Web application and Anima is a pipeline library.

You can clone their repositories to see how PyQt4 and PySide UIs are created with Stalker (in Anima) and how it is used as the database model for a Web application in Stalker Pyramid.