Isolating etcd¶

The Deis Control Plane, Data Plane, and Router Mesh components all depend on an etcd cluster for service discovery and configuration.

Whether built for evaluation or to host production applications, when managing a small Deis cluster (three to five nodes), it is reasonable to accept the platform’s default behavior wherein etcd runs on every node within the cluster.

In larger Deis clusters however, running etcd on every node can have a deleterious effect on overall cluster performance since it increases the time required for nodes to reach consensus on writes and leader elections. In such cases, it is beneficial to isolate etcd to a small, fixed number of nodes. All other nodes in the Deis cluster may run an etcd proxy. Proxies will forward read and write requests to active participants in the etcd cluster (leader or followers) without affecting the time required for etcd nodes to reach consensus on writes or leader elections.

Note

The benefit of running an etcd proxy on any node not running a full etcd process is that any container or service depending on etcd can connect to etcd easily via localhost from any node in the Deis cluster.

Also see CoreOS cluster architecture documentation for further details.

Note

The approach documented here works as of Deis 1.9. Older versions of Deis utilize an older version of etcd that did not include the proxy functionality.

cloud-config¶

To realize the topology described above, it is necessary, at the time of provisioning, to provide different cloud-config for those hosts that will run etcd and for those that will only run an etcd proxy.

For the small, fixed number of hosts running full etcd and satisfying the “central services” role (as described in the CoreOS documentation), the cloud-config provided with Deis is sufficient.

For hosts running only an etcd proxy, satisfying the “worker” role (as described in the CoreOS documentation), cloud-config must be tweaked slightly to include the -proxy on flag. For example:

#cloud-config
---
coreos:
  # ...
  - name: etcd.service
    command: start
    content: |
      # ...
      [Service]
      # ...
      ExecStart=/usr/bin/docker run --net=host --rm \
        # ...
        -listen-client-urls http://0.0.0.0:2379,http://0.0.0.0:4001 \
        # ...
        --discovery <discovery url here> \
        -proxy on
      # ...
    # ...
  # ...
# ...

Isolating etcd as described here requires subsets of a cluster’s hosts to be configured differently from one another (including or excluding the -proxy on flag). Deis provisioning scripts do not currently account for this, so managing separate cloud-config for each subset of nodes in the cluster is left as an exercise for the advanced operator.