Docker Swarm turns a pool of Docker hosts into a single, virtual Docker host. Because Docker Swarm serves the standard Docker API, any tool that already communicates with a Docker daemon can use Swarm to transparently scale to multiple hosts.
Within the Flarecast project Docker Swarm can be used to set up a swarm cluster over multiple Slurm partitions making their resources, such as dedicated graphics processing units, accessible for the Flarecast infrastructure over a single Docker host.
Currently, the Flarecast infrastructure is running on a single partition called 'flarecast' (host name: cluster-r730-1) which do not support any generic resources, such as graphics processing units (gpu) or many integrated core processors (mic). Algorithms requiring such resources have to run on separate partitions, e.g. 'K20' (host name: cluster-r730-k20-1), which can be achieved by Docker Swarm. Hereby, a Swarm manager is running within the 'flarecast' partition while another Swarm agent is running within the 'K20' partition - representing a master-slave model. Finally, a third component, the backend discovery service, is required to persistent the state of the cluster within a key-value store.
The following diagram shows the connection between the Swarm manager, a discovery backend and two Swarm nodes, each running a Swarm agent and a Docker daemon.
A test scenario was built up using VirtualBox and two CoreOS machines which is described in a separate article.
Glossary:
- Swarm manager: orchestrates and schedules containers on remote hosts which are registered on the discovery backend
- Discovery backend: backend where Swarm agents can announce to the Swarm manager
- Swarm agent: registers the referenced Docker daemon on the discovery backend and monitors it
References: