Amazon Web Services¶
Documentation¶
kingpin.actors.aws.base
¶
The AWS Actors allow you to interact with the resources (such as SQS and ELB)
inside your Amazon AWS account. These actors all support dry runs properly, but
each actor has its own caveats with dry=True
. Please read the instructions
below for using each actor.
Required Environment Variables
_Note, these can be skipped only if you have a .aws/credentials file in place._
AWS_ACCESS_KEY_ID: | |
---|---|
Your AWS access key | |
AWS_SECRET_ACCESS_KEY: | |
Your AWS secret |
-
exception
kingpin.actors.aws.base.
InvalidPolicy
[source] Raised when Amazon indicates that policy JSON is invalid.
-
class
kingpin.actors.aws.base.
EnsurableAWSBaseActor
(*args, **kwargs)[source] Ensurable version of the AWS Base Actor
CloudFormation¶
kingpin.actors.aws.cloudformation
¶
-
exception
kingpin.actors.aws.cloudformation.
StackFailed
[source] Raised any time a Stack fails to be created or updated.
-
class
kingpin.actors.aws.cloudformation.
ParametersConfig
[source] Validates the Parameters option.
A valid
parameters
option is a dictionary with simple Key/Value pairs of strings. No nested dicts, arrays or other objects.
-
class
kingpin.actors.aws.cloudformation.
CapabilitiesConfig
[source] Validates the Capabilities option.
The
capability
options currently available areCAPABILITY_IAM
andCAPABILITY_NAMED_IAM
, either of which can be used to grant a Stack the capability to create IAM resources. You must useCAPABILITY_NAMED_IAM
to create IAM resources with custom names.
-
class
kingpin.actors.aws.cloudformation.
OnFailureConfig
[source] Validates the On Failure option.
- The
on_failure
option can take one of the following settings: DO_NOTHING
,ROLLBACK
,DELETE
This option is applied at stack _creation_ time!
- The
-
class
kingpin.actors.aws.cloudformation.
Create
(*args, **kwargs)[source] Creates a CloudFormation stack.
Creates a CloudFormation stack from scratch and waits until the stack is fully built before exiting the actor.
Options
Name: The name of the queue to create
Capabilities: A list of CF capabilities to add to the stack.
On_failure: One of the following strings:
DO_NOTHING
,ROLLBACK
,DELETE
Default:
DELETE
Parameters: A dictionary of key/value pairs used to fill in the parameters for the CloudFormation template.
Region: AWS region (or zone) string, like ‘us-west-2’
Template: String of path to CloudFormation template. Can either be in the form of a local file path (ie,
/my_template.json
) or a URI (iehttps://my_site.com/cf.json
).Timeout_in_minutes: The amount of time that can pass before the stack status becomes CREATE_FAILED.
Examples
{ "desc": "Create production backend stack", "actor": "aws.cloudformation.Create", "options": { "capabilities": [ "CAPABILITY_IAM" ], "name": "%CF_NAME%", "parameters": { "test_param": "%TEST_PARAM_NAME%", }, "region": "us-west-1", "template": "/examples/cloudformation_test.json", "timeout_in_minutes": 45, } }
Dry Mode
Validates the template, verifies that an existing stack with that name does not exist. Does not create the stack.
-
class
kingpin.actors.aws.cloudformation.
Delete
(*args, **kwargs)[source] Deletes a CloudFormation stack
Options
Name: The name of the queue to create Region: AWS region (or zone) string, like ‘us-west-2’ Examples
{ "desc": "Delete production backend stack", "actor": "aws.cloudformation.Create", "options" { "region": "us-west-1", "name": "%CF_NAME%", } }
Dry Mode
Validates that the CF stack exists, but does not delete it.
-
class
kingpin.actors.aws.cloudformation.
Stack
(*args, **kwargs)[source] Manages the state of a CloudFormation stack.
This actor can manage the following aspects of a CloudFormation stack in Amazon:
- Ensure that the Stack is present or absent.
- Monitor and update the stack Template and Parameters as necessary.
NoEcho Parameters
If your CF stack takes a Password as a paremter or any other value thats secret and you set
NoEcho: True
on that parameter, Kingpin will be unable to diff it and compare whether or not the desired setting matches whats in Amazon. A warning will be thrown, and the rest of the actor will continue to operate as normal.If any other difference triggers a Stack Update, the desired value for the parameter with
NoEcho: True
will be pushed in addition to all of the other stack parameters.Options
Name: The name of the queue to create
State: (str) Present or Absent. Default: “present”
Capabilities: (
CapabilitiesConfig
, None)A list of CF capabilities to add to the stack.
Disable_rollback: Set to True to disable rollback of the stack if creation failed.
On_failure: (
OnFailureConfig
, None)One of the following strings:
DO_NOTHING
,ROLLBACK
,DELETE
Default:
DELETE
Parameters: (
ParametersConfig
, None)A dictionary of key/value pairs used to fill in the parameters for the CloudFormation template.
Region: AWS region (or zone) string, like ‘us-west-2’
Template: String of path to CloudFormation template. Can either be in the form of a local file path (ie,
/my_template.json
) or a URI (iehttps://my_site.com/cf.json
).Timeout_in_minutes: The amount of time that can pass before the stack status becomes CREATE_FAILED.
Examples
{ "actor": "aws.cloudformation.Create", "state": "present", "options": { "capabilities": [ "CAPABILITY_IAM" ], "on_failure": "DELETE", "name": "%CF_NAME%", "parameters": { "test_param": "%TEST_PARAM_NAME%", }, "region": "us-west-1", "template": "/examples/cloudformation_test.json", "timeout_in_minutes": 45, } }
Dry Mode
Validates the template, verifies that an existing stack with that name does not exist. Does not create the stack.
Elastic Container Service (ECS)¶
kingpin.actors.aws.ecs
¶
-
exception
kingpin.actors.aws.ecs.
ECSAPIException
[source] A failure from the ECS API.
-
exception
kingpin.actors.aws.ecs.
ECSTaskFailedException
[source] A failure from an ECS Task.
-
exception
kingpin.actors.aws.ecs.
ServiceNotFound
[source] Failure to find an ECS Service.
-
class
kingpin.actors.aws.ecs.
RunTask
(*args, **kwargs)[source] Register and run a task on ECS.
This actor will loop indefinitely until the task is complete.
Options
Region: AWS region (or zone) name, such as us-west-2 or eu-west-1. Cluster: ECS cluster in the region to launch the task into. Task_definition: String of path to the Task Definition file template. Must be a local file path. Tokens to be interpolated must be of the form %VAR%. Tokens can come in the form of options or environment variables. Tokens: A dictionary of key/value pairs used to fill in the tokens for the Task Definition template. These will override environment variables which can be used as tokens. Default: {}. Count: How many tasks to run. Default: 1. Wait: Whether to wait for the tasks to complete. Default: True. Examples
actor: aws.ecs.Task desc: Run migrations options: task_definition: migrate.yaml region: us-west-2 cluster: us1-internal
Dry Mode
Will only attempt to interpolate env vars into the task definition.
-
class
kingpin.actors.aws.ecs.
Service
(*args, **kwargs)[source] Register and run a service on ECS.
This actor will loop indefinitely until the task is complete.
If the service already exists, it is upgraded.
Options
State: Desired state: present/absent Region: AWS region (or zone) name, such as us-west-2 or eu-west-1. Cluster: ECS cluster in the region to launch the task into. Task_definition: String of path to the Task Definition file template. Must be a local file path. Tokens to be interpolated must be of the form %VAR%. Tokens can come in the form of options or environment variables. Default: None. Service_definition: String of path to the Service Definition file template. Must be a local file path. Tokens to be interpolated must be of the form %VAR%. Implicit fields - do not include these: ‘serviceName’, ‘taskDefinition’, ‘desiredCount’, ‘clientToken’ Allowed fields: ‘loadBalancers’, ‘role’, ‘deploymentConfiguration’ Default: None. Service_name: Service name to use. If not specified, this will use the Task Definition’s family. Necessary if there is no given Task Definition. Default: None. Tokens: A dictionary of key/value pairs used to fill in the tokens for the Task and Service Definition template. These will override environment variables which can be used as tokens. Default: {}. Count: How many instances of the service to deploy. Not used when state is ‘absent’. Default: 1. Wait: Whether to wait for the services to deploy. Not used when state is ‘absent’. Default: True. Use_existing_count: Whether to use the existing service’s count when updating instead of the specified count. Default: True. Deregister_task_definitions: Whether to deregister related Task Definitions. Only used when state is ‘absent’. Examples
actor: aws.ecs.Service desc: Deploy taskworker options: task_definition: taskworker.yaml region: us-west-2 cluster: us1-internal
Dry Mode
Will only validate and interpolate tokens into both the task and service definition files.
Elastic Load Balancing (ELB)¶
kingpin.actors.aws.elb
¶
-
class
kingpin.actors.aws.elb.
WaitUntilHealthy
(*args, **kwargs)[source] Wait indefinitely until a specified ELB is considered “healthy”.
This actor will loop infinitely until a healthy threshold of the ELB is met. The threshold can be reached when the
count
as specified in the options is less than or equal to the number of InService instances in the ELB.Another situation is for
count
to be a string specifying a percentage (see examples). In this case the percent of InService instances has to be greater than thecount
percentage.Options
Name: The name of the ELB to operate on Count: Number, or percentage of InService instance to consider this ELB healthy Region: AWS region (or zone) name, such as us-east-1 or us-west-2 Examples
{ "actor": "aws.elb.WaitUntilHealthy", "desc": "Wait until production-frontend has 16 hosts", "options": { "name": "production-frontend", "count": 16, "region": "us-west-2" } }
{ "actor": "aws.elb.WaitUntilHealthy", "desc": "Wait until production-frontend has 85% of hosts in-service", "options": { "name": "production-frontend", "count": "85%", "region": "us-west-2" } }
Dry Mode
This actor performs the finding of the ELB as well as calculating its health at all times. The only difference in dry mode is that it will not re-count the instances if the ELB is not healthy. A log message will be printed indicating that the run is dry, and the actor will exit with success.
-
class
kingpin.actors.aws.elb.
SetCert
(*args, **kwargs)[source] Find a server cert in IAM and use it for a specified ELB.
Options
Region: (str) AWS region (or zone) name, like us-west-2 Name: (str) Name of the ELB Cert_name: (str) Unique IAM certificate name, or ARN Port: (int) Port associated with the cert. (default: 443) Example
{ "actor": "aws.elb.SetCert", "desc": "Run SetCert", "options": { "cert_name": "new-cert", "name": "some-elb", "region": "us-west-2" } }
Dry run
Will check that ELB and Cert names are existent, and will also check that the credentials provided for AWS have access to the new cert for ssl.
-
class
kingpin.actors.aws.elb.
RegisterInstance
(*args, **kwargs)[source] Add an EC2 instance to a load balancer.
Options
Elb: (str) Name of the ELB Instances: (str, list) Instance id, or list of ids. Default “self” id. Region: (str) AWS region (or zone) name, like us-west-2 Enable_zones: (bool) add all available AZ to the elb. Default: True Example
{ "actor": "aws.elb.RegisterInstance", "desc": "Run RegisterInstance", "options": { "elb": "prod-loadbalancer", "instances": "i-123456", "region": "us-east-1", } }
Dry run
Will find the specified ELB, but not take any actions regarding instances.
-
class
kingpin.actors.aws.elb.
DeregisterInstance
(*args, **kwargs)[source] Remove EC2 instance(s) from an ELB.
Options
Elb: (str) Name of the ELB. Optionally this may also be a *
.Instances: (str, list) Instance id, or list of ids Region: (str) AWS region (or zone) name, like us-west-2 Wait_on_draining: (bool) Whether or not to wait for connection draining Example
{ "actor": "aws.elb.DeregisterInstance", "desc": "Run DeregisterInstance", "options": { "elb": "my-webserver-elb", "instances": "i-abcdeft", "region": "us-west-2" } }
Extremely simple way to remove the local instance running this code from all ELBs its been joined to:
{ "actor": "aws.elb.DeregisterInstance", "desc": "Run DeregisterInstance", "options": { "elb": "*", "region": "us-west-2" } }
Dry run
Will find the ELB but not take any actions regarding the instances.
Identity and Access Management (IAM)¶
kingpin.actors.aws.iam
¶
-
class
kingpin.actors.aws.iam.
User
(*args, **kwargs)[source] Manages an IAM User.
This actor manages the state of an Amazon IAM User.
Currently we can:
- Ensure is present or absent
- Manage the inline policies for the user
- Manage the groups the user is in
Options
Name: (str) Name of the User profile to manage State: (str) Present or Absent. Default: “present” Groups: (str,array) A list of groups for the user to be a member of. Default: None Inline_policies: (str,array) A list of strings that point to JSON files to use as inline policies. Default: None Example
{ "actor": "aws.iam.User", "desc": "Ensure that Bob exists", "options": { "name": "bob", "state": "present", "groups": "my-test-group", "inline_policies": [ "read-all-s3.json", "create-other-stuff.json" ] } }
Dry run
Will let you know if the user exists or not, and what changes it would make to the users policy and settings. Will also parse the inline policies supplied, make sure any tokens in the files are replaced, and that the files are valid JSON.
-
class
kingpin.actors.aws.iam.
Group
(*args, **kwargs)[source] Manages an IAM Group.
This actor manages the state of an Amazon IAM Group.
Currently we can:
- Ensure is present or absent
- Manage the inline policies for the group
- Purge (or not) all group members and delete the group
Options
Name: (str) Name of the Group profile to manage Force: (bool) Forcefully delete the group (explicitly purging all group memberships). Default: false State: (str) Present or Absent. Default: “present” Inline_policies: (str,array) A list of strings that point to JSON files to use as inline policies. You can also pass in a single inline policy as a string. Default: None Example
{ "actor": "aws.iam.Group", "desc": "Ensure that devtools exists", "options": { "name": "devtools", "state": "present", "inline_policies": [ "read-all-s3.json", "create-other-stuff.json" ] } }
Dry run
Will let you know if the group exists or not, and what changes it would make to the groups policy and settings. Will also parse the inline policies supplied, make sure any tokens in the files are replaced, and that the files are valid JSON.
-
class
kingpin.actors.aws.iam.
Role
(*args, **kwargs)[source] Manages an IAM Role.
This actor manages the state of an Amazon IAM Role.
Currently we can:
- Ensure is present or absent
- Manage the inline policies for the role
- Manage the Assume Role Policy Document
Options
Name: (str) Name of the Role to manage State: (str) Present or Absent. Default: “present” Inline_policies: (str,array) A list of strings that point to JSON files to use as inline policies. You can also pass in a single inline policy as a string. Default: None Assume_role_policy_document: (str) A string with an Amazon IAM Assume Role policy. Not providing this causes Kingpin to ignore the value, and Amazon defaults the role to an ‘EC2’ style rule. Supplying the document will cause Kingpin to ensure the assume role policy is correct. Default: None Example
{ "actor": "aws.iam.Role", "desc": "Ensure that myapp exists", "options": { "name": "myapp", "state": "present", "inline_policies": [ "read-all-s3.json", "create-other-stuff.json" ] } }
Dry run
Will let you know if the group exists or not, and what changes it would make to the groups policy and settings. Will also parse the inline policies supplied, make sure any tokens in the files are replaced, and that the files are valid JSON.
-
class
kingpin.actors.aws.iam.
InstanceProfile
(*args, **kwargs)[source] Manages an IAM Instance Profile.
This actor manages the state of an Amazon IAM Instance Profile.
Currently we can:
- Ensure is present or absent
- Assign an IAM Role to the Instance Profile
Options
Name: (str) Name of the Role to manage State: (str) Present or Absent. Default: “present” Role: (str) Name of an IAM Role to assign to the Instance Profile. Default: None Example
{ "actor": "aws.iam.InstanceProfile", "desc": "Ensure that my-ecs-servers exists", "options": { "name": "my-ecs-servers", "state": "present", "role": "some-iam-role", } }
Dry run
Will let you know if the profile exists or not, and what changes it would make to the profile.
-
class
kingpin.actors.aws.iam.
UploadCert
(*args, **kwargs)[source] Uploads a new SSL Cert to AWS IAM.
Options
Private_key_path: (str) Path to the private key. Path: (str) The AWS “path” for the server certificate. Default: “/” Public_key_path: (str) Path to the public key certificate. Name: (str) The name for the server certificate. Cert_chain_path: (str) Path to the certificate chain. Optional. Example
{ "actor": "aws.iam.UploadCert", "desc": "Upload a new cert", "options": { "name": "new-cert", "private_key_path": "/cert.key", "public_key_path": "/cert.pem", "cert_chain_path": "/cert-chain.pem" } }
Dry run
Checks that the passed file paths are valid. In the future will also validate that the files are of correct format and content.
-
class
kingpin.actors.aws.iam.
DeleteCert
(*args, **kwargs)[source] Delete an existing SSL Cert in AWS IAM.
Options
Name: (str) The name for the server certificate. Example
{ "actor": "aws.iam.DeleteCert", "desc": "Run DeleteCert", "options": { "name": "fill-in" } }
Dry run
Will find the cert by name or raise an exception if it’s not found.
Simple Storage Service (S3)¶
kingpin.actors.aws.s3
¶
-
class
kingpin.actors.aws.s3.
LoggingConfig
[source] Provides JSON-Schema based validation of the supplied logging config.
The S3 LoggingConfig format should look like this:
{ "target": "s3_bucket_name_here", "prefix": "an_optional_prefix_here" }
If you supply an empty
target
, then we will explicitly remove the logging configuration from the bucket. Example:{ "target": "" }
-
class
kingpin.actors.aws.s3.
LifecycleConfig
[source] Provides JSON-Schema based validation of the supplied Lifecycle config.
The S3 Lifecycle system allows for many unique configurations. Each configuration object defined in this schema will be turned into a
boto.s3.lifecycle.Rule
object. All of the rules together will be turned into aboto.s3.lifecycle.Lifecycle
object.[ { "id": "unique_rule_identifier", "prefix": "/some_path", "status": "Enabled", "expiration": { "days": 365, }, "noncurrent_version_expiration": { "noncurrent_days": 365, }, "transition": { "days": 90, "date": "2016-05-19T20:04:17+00:00", "storage_class": "GLACIER", }, "noncurrent_version_transition": { "noncurrent_days": 90, "storage_class": "GLACIER", } } ]
-
class
kingpin.actors.aws.s3.
TaggingConfig
[source] Provides JSON-Schema based validation of the supplied tagging config.
The S3 TaggingConfig format should look like this:
[ { "key": "my_key", "value": "some_value" } ]
-
class
kingpin.actors.aws.s3.
Bucket
(*args, **kwargs)[source] Manage the state of a single S3 Bucket.
The actor has the following functionality:
- Ensure that an S3 bucket is present or absent.
- Manage the bucket policy.
- Manage the bucket Lifecycle configurations.
- Enable or Suspend Bucket Versioning. Note: It is impossible to actually _disable_ bucket versioning – once it is enabled, you can only suspend it, or re-enable it.
Note about Buckets with Files
Amazon requires that an S3 bucket be empty in order to delete it. Although we could recursively search for all files in the bucket and then delete them, this is a wildly dangerous thing to do inside the confines of this actor. Instead, we raise an exception and alert the you to the fact that they need to delete the files themselves.
Options
Name: The name of the bucket to operate on
State: (str) Present or Absent. Default: “present”
Lifecycle: (
LifecycleConfig
, None)A list of individual Lifecycle configurations. Each dictionary includes keys for the
id
,prefix
andstatus
as required parameters. Optionally you can supply anexpiration
and/ortransition
dictionary.If an empty list is supplied, or the list in any way does not match what is currently configured in Amazon, the appropriate changes will be made.
Logging: (
LoggingConfig
, None)If a dictionary is supplied (
{'target': 'logging_bucket', 'prefix': '/mylogs'}
), then we will configure bucket logging to the supplied bucket and prefix. Ifprefix
is missing then no prefix will be used.If
target
is supplied as an empty string (''
), then we will disable logging on the bucket. IfNone
is supplied, we will not manage logging either way.Tags: (
TaggingConfig
, None)A list of dictionaries with a
key
andvalue
key. Defaults to an empty list, which means that if you manually add tags, they will be removed.Policy: (str, None) A JSON file with the bucket policy. Passing in a blank string will cause any policy to be deleted. Passing in None (or not passing it in at all) will cause Kingpin to ignore the policy for the bucket entirely. Default: None
Region: AWS region (or zone) name, such as us-east-1 or us-west-2
Versioning: (bool, None): Whether or not to enable Versioning on the bucket. If “None”, then we don’t manage versioning either way. Default: None
Examples
{ "actor": "aws.s3.Bucket", "options": { "name": "kingpin-integration-testing", "region": "us-west-2", "policy": "./examples/aws.s3/amazon_put.json", "lifecycle": [ { "id": "main", "prefix": "/", "status": "Enabled", "expiration": 30, } ], "logging": { "target": "logs.myco.com", "prefix": "/kingpin-integratin-testing" }, "tags": [ {"key": "my_key", "value": "billing-grp-1"}, ], "versioning": true, } }
Dry Mode
Finds the bucket if it exists (or tells you it would create it). Describes each potential change it would make to the bucket depending on the configuration of the live bucket, and the options that were passed into the actor.
Will gracefully fail and alert you if there are files in the bucket and you are trying to delete it.
Simple Queue Service (SQS)¶
kingpin.actors.aws.sqs
¶
-
class
kingpin.actors.aws.sqs.
Create
(*args, **kwargs)[source] Creates a new SQS queue with the specified name
Options
Name: (str) The name of the queue to create Region: (str) AWS region (or zone) string, like ‘us-west-2’ Examples
{ "actor": "aws.sqs.Create", "desc": "Create queue named async-tasks", "options": { "name": "async-tasks", "region": "us-east-1", } }
Dry Mode
Will not create any queue, or even contact SQS. Will create a mock.Mock object and exit with success.
-
class
kingpin.actors.aws.sqs.
Delete
(*args, **kwargs)[source] Deletes the SQS queues
Note: even if it`s not empty
Options
Name: (str) The name of the queue to destroy Region: (str) AWS region (or zone) string, like ‘us-west-2’ Idempotent: (bool) Will not raise errors if no matching queues are found. (default: False) Examples
{ "actor": "aws.sqs.Delete", "desc": "Delete queue async-tasks", "options": { "name": "async-tasks", "region": "us-east-1" } }
{ "actor": "aws.sqs.Delete", "desc": "Delete queues with 1234 in the name", "options": { "name": "1234", "region": "us-east-1" } }
Dry Mode
Will find the specified queue, but will have a noop regarding its deletion. Dry mode will fail if no queues are found, and idempotent flag is set to False.
-
class
kingpin.actors.aws.sqs.
WaitUntilEmpty
(*args, **kwargs)[source] Wait indefinitely until for SQS queues to become empty
This actor will loop infinitely as long as the count of messages in at least one queue is greater than zero. SQS does not guarantee exact count, so this can return a stale value if the number of messages in the queue changes rapidly.
Options
Name: (str) The name or regex pattern of the queues to operate on Region: (str) AWS region (or zone) string, like ‘us-west-2’ Required: (bool) Fail if no matching queues are found. (default: False) Examples
{ "actor": "aws.sqs.WaitUntilEmpty", "desc": "Wait until release-0025a* queues are empty", "options": { "name": "release-0025a", "region": "us-east-1", "required": true } }
Dry Mode
This actor performs the finding of the queue, but will pretend that the count is 0 and return success. Will fail even in dry mode if
required
option is set to True and no queues with the name pattern are found.