Development¶
Setting up your Environment¶
Create your VirtualEnvironment¶
$ virtualenv .venv --no-site-packages
New python executable in .venv/bin/python
Installing setuptools, pip...done.
$ source .venv/bin/activate
Check out the code¶
(.venv) $ git clone git@github.com:Nextdoor/kingpin
Cloning into 'kingpin'...
Warning: Permanently added 'github.com,192.30.252.128' (RSA) to the list of known hosts.
remote: Counting objects: 1831, done.
remote: irangedCompressing objects: 100% (17/17), done.
remote: Total 1831 (delta 7), reused 0 (delta 0)
Receiving objects: 100% (1831/1831), 287.68 KiB, done.
Resolving deltas: 100% (1333/1333), done.
Install the test-specific dependencies¶
(.venv) $ pip install -r kingpin/requirements.test.txt
...
(.venv) $ cd kingpin
(.venv) $ python setup.py test
...
Testing¶
Unit Tests¶
The code is 100% unit test coverage complete, and no pull-requests will be accepted that do not maintain this level of coverage. That said, it’s possible (likely) that we have not covered every possible scenario in our unit tests that could cause failures. We will strive to fill out every reasonable failure scenario.
Integration Tests¶
Because it’s hard to predict cloud failures, we provide integration tests for most of our modules. These integration tests actually go off and execute real operations in your accounts, and rely on particular environments being setup in order to run. These tests are great to run though to validate that your credentials are all correct.
Executing the tests
HIPCHAT_TOKEN=<xxx> RIGHTSCALE_TOKEN=<xxx> INTEGRATION_TESTS=<comma separated list> make integration
...
integration_02a_clone (integration_server_array.IntegrationServerArray) ... ok
integration_test_execute_real (integration_hipchat.IntegrationHipchatMessage) ... ok
integration_test_execute_with_invalid_creds (integration_hipchat.IntegrationHipchatMessage) ... ok
integration_test_init_without_environment_creds (integration_hipchat.IntegrationHipchatMessage) ... ok
...
kingpin.utils 67 30 55% 57-69, 78, 93-120, 192-202
----------------------------------------------------------------------
TOTAL 571 143 75%
----------------------------------------------------------------------
Ran 10 tests in 880.274s
OK
running pep8
running pyflakes
Executing Only Certain Test Suites¶
Because not everyone will use or need to test all of our actors, you can
execute only certain subsets of our integration tests if you wish. Simply set
the INTEGRATION_TESTS
environment variable to a comma-separated list of test
suites. See below for the list.
Executing only the HTTP Tests
(.venv)Matts-MacBook-2:kingpin diranged$ INTEGRATION_TESTS=http make integration
INTEGRATION_TESTS=http PYFLAKES_NODOCTEST=True \
python setup.py integration pep8 pyflakes
running integration
integration_base_get (integration_api.IntegrationRestConsumer) ... ok
integration_delete (integration_api.IntegrationRestConsumer) ... ok
integration_get_basic_auth (integration_api.IntegrationRestConsumer) ... ok
integration_get_basic_auth_401 (integration_api.IntegrationRestConsumer) ... ok
integration_get_json (integration_api.IntegrationRestConsumer) ... ok
integration_get_with_args (integration_api.IntegrationRestConsumer) ... ok
integration_post (integration_api.IntegrationRestConsumer) ... ok
integration_put (integration_api.IntegrationRestConsumer) ... ok
integration_status_401 (integration_api.IntegrationRestConsumer) ... ok
integration_status_403 (integration_api.IntegrationRestConsumer) ... ok
integration_status_500 (integration_api.IntegrationRestConsumer) ... ok
integration_status_501 (integration_api.IntegrationRestConsumer) ... ok
...
List of Built-In Integration Test Suites
- aws
- librato
- rightscale
- http
- hipchat
- pingdom
- rollbar
- pingdom
- slack
Class/Object Architecture¶
kingpin.rb
|
+-- deployment.Deployer
| Executes a deployment based on the supplied DSL.
|
+-- actors.rightscale
| | RightScale Cloud Management Actor
| |
| +-- server_array
| +-- Clone
| +-- Destroy
| +-- Execute
| +-- Launch
| +-- Update
|
+-- actors.aws
| | Amazon Web Services Actor
| |
| +-- elb
| | +-- WaitUntilHealthy
| |
| +-- sqs
| +-- Create
| +-- Delete
| +-- WaitUntilEmpty
|
+-- actors.email
| | Email Actor
|
+-- actors.hipchat
| | Hipchat Actor
| |
| +-- Message
|
+-- actors.librato
| Librato Metric Actor
|
+-- Annotation
Actor Design¶
Kingpin Actors are self-contained python classes that execute operations asynchronously. Actors should follow a consistent structure (described below) and be written to be as fault tolerant as possible.
Example - Hello World¶
This is the basic structure for an actor class.
import os
from tornado import gen
from kingpin.actors import base
from kingpin.actors import exceptions
# All actors must have an __author__ tag. This is used actively
# by the Kingpin code, do not forget this!
__author__ = 'Billy Joe Armstrong <american_idiot@broadway.com>'
# Perhaps you need an API token?
TOKEN = os.getenv('HELLO_WORLD_TOKEN', None)
class HelloWorld(base.BaseActor):
# Create an all_options dictionary that contains all of
# the required and optional options that can be passed into
# this actor.
all_options = {
'name': (str, None, 'Your name'),
'world': (str, None, 'World we\'re saying hello to!'),
}
# Optionally, if you need to do any instantiation-level, non-blocking
# validation checks (for example, looking for an API token) you can do
# them in the __init__. Do *not* put blocking code in here.
def __init__(self, *args, **kwargs):
super(HelloWorld, self).__init__(*args, **kwargs)
if not TOKEN:
raise exceptions.InvalidCredentials(
'Missing the "HELLO_WORLD_TOKEN" environment variable.')
# Initialize our hello world sender object. This is non-blocking.
self._hello_world = my.HelloWorldSender(token=TOKEN)
# Its nice to wrap some of your logic into separate methods. This
# method handles sending the message, or pretends to send the
# message if we're in a dry run.
@gen.coroutine
def _send_message(self, name, world):
# Attempt to log into the API to sanity check our credentials
try:
yield self._hello_world.login()
except Shoplifter:
msg = 'Could not log into the world!'
raise exceptions.UnrecoverableActorFailure(msg)
# Make sure to support DRY mode all the time!
if self._dry:
self.log.info('Would have said Hi to %s' % world)
raise gen.Return()
# Finally, send the message!
try:
res = yield self._hello_world.send(
from=name, to=world)
except WalkingAlone as e:
# Lets say that this error is completely un-handleable exception,
# there's no one to say hello to!
self.log.critical('Some extra information about this error...')
# Now, raise an exception that is will stop execution of Kingpin,
# regardless of the warn_on_failure setting.
raise exceptions.UnrecoverableActorException('Oh my: %s' % e)
# Return the value back to the execute method
raise gen.Return(res)
# The meat of the work happens in the _execute() method. This method
# is called by the BaseActor.execute() method. Your method must be
# wrapped in a gen.Coroutine wrapper. Note, the _execute() method takes
# no arguments, all arguments for the acter were passed in to the
# __init__() method.
@gen.coroutine
def _execute(self):
self.log.debug('Warming up the HelloWorld Actor')
# Fire off an async request to a our private method for sending
# hello world messages. Get the response and evaluate
res = yield self._send_message(
self.option('name'), self.option('world'))
# Got a response. Did our message really go through though?
if not res:
# The world refuses to hear our message... A shame, really, but
# not entirely critical.
self.log.error('We failed to get our message out ... just '
'letting you know!')
raise exceptions.RecoverableActorFailure(
'A shame, but I suppose they can listen to what they want')
# We've been heard!
self.log.info('%s people have heard our message!' % res)
# Indicate to Tornado that we're done with our execution.
raise gen.Return()
Actor Parameters¶
The following parameters are baked into our
BaseActor
model and must be supported by any
actor that subclasses it. They are fundamentally critical to the behavior of
Kingpin, and should not be bypassed or ignored.
desc
¶
A string describing the stage or action thats occuring. Meant to be human
readable and useful for logging. You do not need to do anything intentinally to
support this option (it’s handled in BaseActor
).
All logging (when using self.log()) are passed through a custom
LogAdapter
.
dry
¶
All Actors must support a dry
run flag. The codepath thats executed when
_execute()
is yielded should be as wet as possible without actually making
any changes. For example, if you have an actor that checks the state of an
Amazon ELB (hint see kingpin.actors.aws.elb.WaitUntilHealthy
), you
would want the actor to actually search Amazons API for the ELB, actually check
the number of instances that are healthy in the ELB, and then fake a return
value so that the rest of the script can be tested.
options
¶
Your actor can take in custom options (ELB name, Route53 DNS entry name, etc)
through a dictionary named options
thats passed in to every actor and
accessible through the option()
method. The contents of this dictionary are entirely up to you.
These options are defined in your class’s all_options
dict. A simple example:
from kingpin.constants import REQUIRED
class SayHi(object):
all_options = {
'name': (str, REQUIRED, 'What is your name?')
}
@gen.coroutine
def _execute(self):
self.log.info('Hi %s' % self.option('name'))
For more complex user input validation, see kingpin.actors.utils.dry().
warn_on_failure
(optional)¶
If the user sets warn_on_failure=True
, any raised exceptions that subclass
kingpin.actors.exceptions.RecoverableActorFailure
will be swallowed up and
warned about, but will not cause the execution of the kingpin script to end.
Exceptions that subclass kingpin.actors.exceptions.UnrecoverableActorFailure
(or uncaught third party exceptions) will cause the actor to fail and the
script to be aborted no matter what!
Required Methods¶
_execute() method¶
Your actor can execute any code you would like in the _execute()
method. This
method should make sure that it’s a tornado-style generator (thus, can be
yielded), and that it never calls any blocking operations.
Actors must not:
- Call a blocking operation ever
- Call an async operation from inside the init() method
- Bypass normal logging methods
return
a result (shouldraise gen.Return(...)
)
Actors must:
- Subclass kingpin.actors.base.BaseActor
- Include
__author__
attribute thats a single string with the owners listed in it. - Implement a *_execute()* method
- Handle as many possible exceptions of third-party libraries as possible
- Return None when the actor has succeeded.
Actors can:
- Raise kingpin.actors.exceptions.UnrecoverableActorFailure. This is considered an unrecoverable exception and no Kingpin will not execute any further actors when this happens.
- Raise kingpin.actors.exceptions.RecoverableActorFailure.
This is considered an error in execution, but is either expected or at
least cleanly handled in the code. It allows the user to specify
warn_on_failure=True
, where they can then continue on in the script even if an actor fails.
Super simple example Actor _execute() method
@gen.coroutine
def _execute(self):
self.log.info('Making that web call')
res = yield self._post_web_call(URL)
raise gen.Return(res)
Recommended Design Patterns¶
State Management Actors¶
While many of our actors are designed as code that “does something once” – ie, “Create User Foo” – we are increasingly seeing actors that “ensure a resource exists.” This new pattern is a bit more Puppet-like, and more well suited for ensuring the state of cloud resources rather than simply creating or destrying things.
To that end, we have a few recommended guidelines for patterns to follow when creating actors like this. These guidelines will help breed consistency between our various actors so that users are never surprised by their behavior.
Resource attributes should be managed explicitly
(See this http://github.com/Nextdoor/issues/342 for more discussion)
Generally speaking, if an actor manages a resource (call it a User
), any
parameters, sub resources like group memberships or other attributes should
only be managed by the Actor if they are explicitly defined by the user.
For example, the following code should create a user, and do absolutely nothing else to the user. Any additional attirbutes (group memberships, or inline IAM policies) should not be managed:
{ "actor": "aws.iam.User",
"options": {
"name": "myuser",
"state": "present"
}
}
On the other hand, if the user does supply groups or inline_policies, the actor should explicitly manage those and ensure that they exactly match what was supplied:
{ "actor": "aws.iam.User",
"options": {
"name": "myuser",
"state": "present"
"inline_policies": "my-policy.json",
"groups": [
"admin", "engineers"
]
}
}
In this case, the myuser
account should have its groups and inline policies
exactly set to the above settings, and anything that was found to be mismatched
in Amazon should be wiped out.
Helper Methods/Objects¶
self.__class__.desc¶
The “description” of a particular actor is a parameter that the user can supply
through the JSON if they wish. If no description is supplied, a default
description is supplied by the actor’s self.__class__.desc
attribute. If your
actor wants to supply its own default description, it can be done like this:
class Sleep(object):
desc = "Sleeping for {sleep}s"
all_options = {
'sleep': (int), REQUIRED, 'Number of seconds to do nothing.')
}
(.venv)Matts-MacBook-2:kingpin diranged$ python kingpin/bin/deploy.py --color --debug -a misc.Sleep -o sleep=10 --dry
09:55:08 DEBUG 33688 [kingpin.actors.utils ] [get_actor_class ] Tried importing "misc.Sleep" but failed: No module named misc
09:55:08 DEBUG 33688 [kingpin.actors.misc.Sleep ] [_validate_options ] [DRY: Sleeping for 10s] Checking for required options: ['sleep']
09:55:08 DEBUG 33688 [kingpin.actors.misc.Sleep ] [__init__ ] [DRY: Sleeping for 10s] Initialized (warn_on_failure=False, strict_init_context=True)
09:55:08 INFO 33688 [__main__ ] [main ]
09:55:08 WARNING 33688 [__main__ ] [main ] Lights, camera ... action!
09:55:08 INFO 33688 [__main__ ] [main ]
09:55:08 DEBUG 33688 [kingpin.actors.misc.Sleep ] [execute ] [DRY: Sleeping for 10s] Beginning
09:55:08 DEBUG 33688 [kingpin.actors.misc.Sleep ] [_check_condition ] [DRY: Sleeping for 10s] Condition True evaluates to True
09:55:08 DEBUG 33688 [kingpin.actors.misc.Sleep ] [timeout ] [DRY: Sleeping for 10s] kingpin.actors.misc.Sleep._execute() deadline: 3600(s)
09:55:08 DEBUG 33688 [kingpin.actors.misc.Sleep ] [_execute ] [DRY: Sleeping for 10s] Sleeping for 10 seconds
09:55:08 DEBUG 33688 [kingpin.actors.misc.Sleep ] [execute ] [DRY: Sleeping for 10s] Finished successfully, return value: None
09:55:08 DEBUG 33688 [kingpin.actors.misc.Sleep ] [_wrap_in_timer ] [DRY: Sleeping for 10s] kingpin.actors.misc.Sleep.execute() execution time: 0.00s
The format() is called with the following key/values as possible variables that can be parsed at runtime:
actor
: The Actor Package and Class – ie,kingpin.actors.misc.Sleep
in the example above.**self._options
: The entire set of options passed into the actor, broken out by key/value.
self.log()¶
For consistency in logging, a custom Logger object is instantiated for every
Actor. This logging object ensures that prefixes such as the desc
of an Actor
are included in the log messages. Usage examples:
self.log.error('Hey, something failed')
self.log.info('I am doing work')
self.log.warning('I do not think that should have happened')
self.option()¶
Accessing options passed to the actor from the JSON file should be done via
self.option()
method. Accessing self._options
parameter is not recommended,
and the edge cases should be handled via the all_options
class variable.
kingpin.actors.utils.dry()¶
The kingpin.actors.utils.dry()
wrapper quickly allows you to make a
call dry – so it only warns about execution during a dry run rather than
actually executing.
User Option Validation¶
While you can rely on options for simple validation of strings, bools, etc – you may find yourself needing to validate more complex user inputs. Regular expressions, lists of valid strings, or even full JSON schema validations.
The Self-Validating Class¶
If you create a class with a validate()
method, Kingpin will automatically
validate a users input against that method. Here’s a super simple example that
only accepts words that start with the letter X
.
from kingpin.actors.exceptions import InvalidOptions
class OnlyStartsWithX(object):
@classmethod
def validate(self, option):
if not option.startswith('X'):
raise InvalidOptions('Must start with X: %s' % option)
class MyActor(object):
all_options = {
(OnlyStartsWithX, REQUIRED, 'Any string that starts with an X')
}
Pre-Built Option Validators¶
We have created a few useful option validators that you can easily leverage in your own code:
Exception Handling¶
Simple API Access Objects¶
Most of the APIs out there leverage basic REST with JSON or XML as the data
encoding method. Since these APIs behave similarly, we have created a simple
API access object that can be extended for creating actors quickly. The object
is called a RestConsumer
and is in the kingpin.actors.support.api
package.
This RestConsumer
can be subclassed and filled in with a dict
that
describes the API in detail.
HTTPBin Actor with the RestConsumer¶
HTTPBIN = {
'path': '/',
'http_methods': {'get': {}},
'attrs': {
'get': {
'path': '/get',
'http_methods': {'get': {}},
},
'post': {
'path': '/post',
'http_methods': {'post': {}},
},
'put': {
'path': '/put',
'http_methods': {'put': {}},
},
'delete': {
'path': '/delete',
'http_methods': {'delete': {}},
},
}
}
class HTTPBinRestClient(api.RestConsumer):
_CONFIG = HTTPBIN
_ENDPOINT = 'http://httpbin.org'
class HTTPBinGetThenPost(base.BaseActor):
def __init__(self, \*args, \**kwargs):
super(HTTPBinGetThenPost, self).__init__(\*args, \**kwargs)
self._api = HTTPBinRestClient()
@gen.coroutine
def _execute(self):
yield self._api.get().http_get()
if self._dry
raise gen.Return()
yield self._api.post().http_post(foo='bar')
raise gen.Return()
Exception Handling in HTTP Requests¶
The RestClient.fetch()
method has been wrapped in a retry decorator
that
allows you to define different behaviors based on the exceptions returned from
the fetch method. For example, you may want to handle an HTTPError exception
with a 401
error code differently than a 503
error code.
You can customize the exception handling by subclassing the
RestClient
:
class MyRestClient(api.RestClient):
_EXCEPTIONS = {
httpclient.HTTPError: {
'401': my.CustomException(),
'403': exceptions.InvalidCredentials,
'500': my.UnretryableError(),
'502': exceptions.InvalidOptions,
# This acts as a catch-all
'': exceptions.RecoverableActorFailure,
}
}