jujubigdata.utils¶

`jujubigdata.utils.DistConfig`	This class processes distribution-specific configuration options.
`jujubigdata.utils.TimeoutError`
`jujubigdata.utils.check_connect`
`jujubigdata.utils.cpu_arch`
`jujubigdata.utils.disable_firewall`	Temporarily disable the firewall, via ufw.
`jujubigdata.utils.environment_edit_in_place`	Edit the /etc/environment file in-place.
`jujubigdata.utils.generate_ssh_key`
`jujubigdata.utils.get_kv_hosts`
`jujubigdata.utils.get_ssh_key`
`jujubigdata.utils.ha_node_state`	If given, retries overrides the default number (45) of times each NameNode will try to be connected to.
`jujubigdata.utils.initialize_kv_host`
`jujubigdata.utils.install_ssh_key`
`jujubigdata.utils.jps`	Get PIDs for named Java processes, for any user.
`jujubigdata.utils.manage_etc_hosts`	Manage the /etc/hosts file from the host entries stored in unitdata.kv() by the various relations.
`jujubigdata.utils.normalize_strbool`
`jujubigdata.utils.re_edit_in_place`	Perform a set of in-place edits to a file.
`jujubigdata.utils.read_etc_env`	Read /etc/environment and return it, along with proxy configuration, as a dict.
`jujubigdata.utils.remove_kv_hosts`
`jujubigdata.utils.resolve_private_address`
`jujubigdata.utils.run_as`	Run a command as a particular user, using `/etc/environment` and optionally capturing and returning the output.
`jujubigdata.utils.run_bg_as`	Run a command as the given user in the background.
`jujubigdata.utils.spec_matches`
`jujubigdata.utils.ssh_key_dir`
`jujubigdata.utils.ssh_priv_key`
`jujubigdata.utils.ssh_pub_key`
`jujubigdata.utils.strtobool`
`jujubigdata.utils.update_etc_hosts`	Update /etc/hosts given a mapping of managed IP / hostname pairs.
`jujubigdata.utils.update_kv_host`
`jujubigdata.utils.update_kv_hosts`
`jujubigdata.utils.verify_resources`	Predicate for specific named resources, with useful rendering in the logs.
`jujubigdata.utils.wait_for_connect`
`jujubigdata.utils.wait_for_hdfs`
`jujubigdata.utils.wait_for_jps`
`jujubigdata.utils.xmlpropmap_edit_in_place`	Edit an XML property map (configuration) file in-place.

class jujubigdata.utils.DistConfig(filename='dist.yaml', required_keys=None, data=None)¶

Bases: object

This class processes distribution-specific configuration options.

Some configuration options are specific to the Hadoop distribution, (e.g. Apache, Hortonworks, MapR, etc). These options are immutable and must not change throughout the charm deployment lifecycle.

Helper methods are provided for keys that require action. Presently, this includes adding/removing directories, dependent packages, and groups/users. Other required keys may be listed when instantiating this class, but this will only validate these keys exist in the yaml; it will not provide any helper functionality for unkown keys.

Parameters:	filename (str) – File to process (default dist.yaml) required_keys (list) – A list of keys required to be present in the yaml

Example dist.yaml with supported keys:

vendor: '<name>'
hadoop_version: '<version>'
packages:
    - '<package 1>'
    - '<package 2>'
groups:
    - '<name>'
users:
    <user 1>:
        groups: ['<primary>', '<group>', '<group>']
    <user 2>:
        groups: ['<primary>']
dirs:
    <dir 1>:
        path: '</path/to/dir>'
        perms: 0777
    <dir 2>:
        path: '{config[<option>]}'  # value comes from config option
        owner: '<user>'
        group: '<group>'
        perms: 0755
ports:
    <name1>:
        port: <port>
        exposed_on: <service>  # optional
    <name2>:
        port: <port>
        exposed_on: <service>  # optional

add_dirs()¶

add_packages()¶

add_users()¶

exposed_ports(service)¶

path(key)¶

port(key)¶

remove_dirs()¶

remove_packages()¶

remove_users()¶

exception jujubigdata.utils.TimeoutError¶: Bases: exceptions.Exception

jujubigdata.utils.check_connect(addr, port)¶

jujubigdata.utils.cpu_arch()¶

jujubigdata.utils.disable_firewall(*args, **kwds)¶: Temporarily disable the firewall, via ufw.

jujubigdata.utils.environment_edit_in_place(*args, **kwds)¶

Edit the /etc/environment file in-place.

There is no standard definition for the format of /etc/environment, but the convention, which this helper supports, is simple key-value pairs, separated by =, with optionally quoted values.

Note that this helper will implicitly quote all values.

Also note that the file is not locked during the edits.

jujubigdata.utils.generate_ssh_key(user)¶

jujubigdata.utils.get_kv_hosts()¶

jujubigdata.utils.get_ssh_key(user)¶

jujubigdata.utils.ha_node_state(host, retries=None)¶: If given, retries overrides the default number (45) of times each NameNode will try to be connected to.

jujubigdata.utils.initialize_kv_host()¶

jujubigdata.utils.install_ssh_key(user, ssh_key)¶

jujubigdata.utils.jps(name)¶: Get PIDs for named Java processes, for any user.

jujubigdata.utils.manage_etc_hosts()¶: Manage the /etc/hosts file from the host entries stored in unitdata.kv() by the various relations.

jujubigdata.utils.normalize_strbool(value)¶

jujubigdata.utils.re_edit_in_place(filename, subs, encoding='utf8', append_non_matches=False)¶

Perform a set of in-place edits to a file.

Parameters:	filename (str) – Name of file to edit subs (dict) – Mapping of patterns to replacement strings

jujubigdata.utils.read_etc_env()¶: Read /etc/environment and return it, along with proxy configuration, as a dict.

jujubigdata.utils.remove_kv_hosts(*hosts)¶

jujubigdata.utils.resolve_private_address(addr)¶

jujubigdata.utils.run_as(user, command, *args, **kwargs)¶

Run a command as a particular user, using /etc/environment and optionally capturing and returning the output.

Raises subprocess.CalledProcessError if command fails.

Parameters:	user (str) – Username to run command as command (str) – Command to run args (list) – Additional args to pass to command env (dict) – Additional env variables (will be merged with `/etc/environment`) capture_output (bool) – Capture and return output (default: False) input (str) – Stdin for command

jujubigdata.utils.run_bg_as(user, output_log, command, *args)¶

Run a command as the given user in the background.

Parameters:	user (str) – User to run flume agent command (str) – Command to run args (list) – Additional args to pass to the command

jujubigdata.utils.spec_matches(local_spec, remote_spec)¶

jujubigdata.utils.ssh_key_dir(user)¶

jujubigdata.utils.ssh_priv_key(user)¶

jujubigdata.utils.ssh_pub_key(user)¶

jujubigdata.utils.strtobool(value)¶

jujubigdata.utils.update_etc_hosts(ips_to_names)¶

Update /etc/hosts given a mapping of managed IP / hostname pairs.

Note, you should not use this directly. Instead, use update_kv_hosts() and manage_etc_hosts().

Parameters:	ips_to_names (dict) – mapping of IPs to hostnames (must be one-to-one)

jujubigdata.utils.update_kv_host(ip, host)¶

jujubigdata.utils.update_kv_hosts(ips_to_names)¶

class jujubigdata.utils.verify_resources(*which)¶

Bases: object

Predicate for specific named resources, with useful rendering in the logs.

Parameters:	which (str*) – One or more resource names to fetch & verify. Defaults to all non-optional resources.

jujubigdata.utils.wait_for_connect(addr, port, timeout)¶

jujubigdata.utils.wait_for_hdfs(timeout)¶

jujubigdata.utils.wait_for_jps(process_name, timeout)¶

jujubigdata.utils.xmlpropmap_edit_in_place(*args, **kwds)¶

Edit an XML property map (configuration) file in-place.

This helper acts as a context manager which edits an XML file of the form:

<configuration>
    <property>
        <name>property-name</name>
        <value>property-value</value>
        <description>Optional property description</description>
    </property>
    ...
</configuration>

This context manager yields a dict containing the existing name/value mappings. Properties can then be modified, added, or removed, and the changes will be reflected in the file.

Example usage:

with xmlpropmap_edit_in_place('my.xml') as props:
    props['foo'] = 'bar'
    del props['removed']

Note that the file is not locked during the edits.