Docker: Platform for Microservices

With the advent of micro services architectural style were applications are a collection of loosely coupled services which can be independently deployed, upgraded and scaled, many organizations are switching to micro-services design in order to achieve greater scalability and availability. In order to run individual services on different instances to scale efficiently, a self contained unit such as virtual machines or docker containers could be used.

Virtualization is a technique of importing a guest operating system on a host operating system. It allows multiple operating systems to run on a single machine, which allows easy recovery on failure. A virtual machine is comprised of some level of hardware, kernel virtualization on which runs a guest operating system and a guest kernel that can talk to this virtual hardware. Virtual machine emulates a real machine and runs on top of either hosted hypervisor or a bare-metal hypervisor which in turn runs on host machine. Hosted virtualization hypervisor runs on the operating system of the host machine hence is almost hardware independent while bare metal hypervisor runs directly on the host machine’s hardware providing better performance. The hypervisor drives virtualization by allowing the physical host machine to operate multiple virtual machines as guests to help maximize the effective use of computing resources such as memory, network bandwidth and CPU cycles. It also allows sharing of resources amongst multiple virtual machines. Either way, the hypervisor approach is considered heavy weight as it requires virtualizing multiple parts if not all of the hardware and kernel. The virtual machine packages up the virtual hardware, a kernel (i.e. OS) and user space for each new instance thus requiring lot of hardware resources. Running multiple VMs on the same host machine degrades the system performance, as each virtual OS runs its own kernel and libraries/dependencies taking considerable chunk of host system resources. Virtual machines are also slower to boot up which becomes critical for real time processing production applications. Once any virtual machine is allocated memory, it cannot be taken back later even though it only uses fraction of its allocated memory. Virtualization thus involves in adding extra hardware to achieve desirable performance, and is a tedious and costly affair to maintain.

Containerization is a lightweight alternative to full machine virtualization that involves encapsulating an application in a container with its own operating environment. Containers run on the same host operating system and on host kernel requiring significantly less resources making booting up the container much faster than a virtual machine.

Docker is a Containerization platform which packages the application and all its dependencies together in the form of Containers so as to ensure that the application works seamlessly in any environment, be it Development, Testing or Production. Docker Containers similar to VMs have a common goal to isolate an application and its dependencies into a self-contained unit that can run anywhere. Each container runs independently of the other containers with process level isolation. Docker containers requires very less space, starts up faster and can be easily integrated with many Dev-Ops tools for automation compared to virtual machines. The Docker container gets allocated the exact amount of memory to run each container thus avoiding any unused memory allocated to any container. Unlike virtual machines which require hardware virtualization for machine level isolation, docker containers operate on isolation within the same operation system. The overhead difference between VM and containers becomes really apparent as the number of isolated spaces increase. Further since docker containers runs on the host system kernel it makes them very lightweight and faster to execute.

Docker container is an isolated application platform which contains everything needed to run the application. They are built from one base docker image & dependencies are installed on top of the image as "image layers". A Docker image is equivalent to an executable which run specific services in a particular environment. Hence in other words, a Docker container is a live running instance of a Docker image. Docker registry is a storage component for docker images. The registry can be user's local repository or a public repository like DockerHub in order to collaborate to build an application.
Docker engine is the heart of docker system, and it creates and runs Docker containers. It works as a client server application with server being a Docker Daemon process which is communicated by Docker CLI using rest APIs and socket I/O to create/run docker containers. Docker daemon builds an image based on inputs or pulls an image from docker registry after receiving corresponding docker build command or docker pull command from docker CLI. When docker run command is received from docker CLI, docker daemon creates a running instance of docker image by creating and running docker container. For Windows and Mac OS X, there is an additional Docker Toolbox which acts as an installer to quickly setup docker environment which includes Docker client, Kitematic, Docker machine and Virtual box.
Docker provides various restart policies to allow the containers to start automatically when they exit, or when Docker restarts. It is always preferred to restart the container if it stops mostly in case of failures.

Docker Machine

Docker Machine is a tool which allows to create (and manage) virtual hosts with installed docker engine on either local machine using VirtualBox or on any cloud providers such as DigitalOcean, AWS and Azure. The docker-machine commands allows to start, inspect, stop, and restart a managed host, upgrade the Docker client and daemon, and configure a Docker client to talk to corresponding host. Docker Machine enables to provision multiple remote Docker hosts on various flavors of Linux and allows to run docker on older Windows or Mac operating systems.


Docker Networking

By default docker creates three networks automatically on install: bridge, none, and host.

BRIDGE: All Docker installations represent the docker0 network with bridge; since docker connects to bridge driver by default. Docker also automatically creates a subnet and gateway for the bridge network, and docker run automatically adds containers to it. Containers running on the same network can communicate with one another via IP addresses. Docker does not support automatic service discovery on bridge network. To connect containers with the network use the "--network" option of docker run command.

NONE: The None network offers a container-specific network stack that lacks a network interface.
The container for none network only has a local loopback interface (i.e., no external network interface).

HOST: Host enables a container to attach to your host’s network (meaning the configuration inside the container matches the configuration outside the container).

Containers can communicate within networks but not across networks. A container with attachments to multiple networks can connect with all of the containers on all of those networks. The docker network create command allows to create custom isolated networks. Any other container created on such network can immediately connect to any other container on this network. The network isolates containers from other (including external) networks. However, we can expose and publish container ports on the network, allowing portions of our bridge access to an outside network.

Overlay network provides native multi-host networking and requires a valid key-value store service, such as Consul, Etcd, or ZooKeeper. A key-value store service should be installed and configured before creating the network. Multiple docker hosts within overlay network must communicate with the key-value store service. Hosts can be provisioned by docker machine. Once we connect, every container on the network has access to all the other containers on the network, regardless of the Docker host serving the container.

Docker Compose

When the docker application includes more than one container, building, running, and connecting the containers from separate Dockerfiles is cumbersome and time-consuming. Docker compose solves this by allows to define a multi-container application using a single YAML file and spin up the application using a single command. It allows to build images, scale containers, link containers in a network and define volumes for data storage. Docker compose is a wrapper around the docker CLI in order to gain time. A docker-compose.yml file is organized into four sections:

version: It specifies the docker compose file syntax version
services: A service is the name for the docker container in production. This section defines the containers that will be started as a part of the Docker Compose instance.
networks: This section is used to configure networking for the application. It enables to change the settings of the default network, connect to an external network, or define app-specific networks.
volumes: It enables to mount a linked path on host machine which is used by the container for persistent storage.


Installing Docker on Ubuntu

$ sudo apt update

$ sudo apt install apt-transport-https ca-certificates curl software-properties-common

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

$ sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"

$ sudo apt update

$ sudo apt install docker-ce

$ sudo apt install docker-compose


Docker Commands

Log in to a Docker registry

$ docker login -u <docker-username> -p <docker-password>

Pull an image or a repository from a registry

$ docker pull elkozmon/zoonavigator-api:0.2.3

Docker command to clean up any resources; images, containers, volumes, and networks which are dangling and not associated with a container.

$  docker system prune

To remove any stopped containers and all unused images (not just dangling images), add the -a flag to the command.

$  docker system prune -a

Remove all stopped containers, unused volumes and unused images. The -force option does not ask for confirmation during removal.

$  docker system prune --all --force --volumes

Remove all dangling images were no container is associated to them, skipping confirmation for removal.

$ docker image prune -f

Remove all unused images, not just dangling ones

$ docker image prune -a

Remove all stopped containers.

$ docker container prune

Remove all unused local volumes

$ docker volume prune

To delete all dangling volumes, use the below command

$ docker volume rm `docker volume ls -q -f dangling=true`

Below docker ps command with the -a flag gives the details of the container including its name, container id and ports on which they are running.

$ docker ps -a

The docker ps -a command can also be used to locate the containers and filter them using -f flag by their status: created, restarting, running, paused, or exited.

$ docker ps -a -f status=exited

Build a docker image using the docker build command. The -t option allows to tagging of the image.

$ docker build .

$ docker build -t username/repository-name .

Remove the container by container name or id using rm command.

$ docker rm <container-id> or <container-name>

Removes (and un-tags) one or more images. The -f option removes an image from running container.

$ docker rmi

To stop all the docker containers

$ docker stop $(docker ps -a -q)

Then to remove all the stopped containers, pass the docker container ids from docker ps to docker rm command

$ docker rm $(docker ps -a -q)

Create a volume with specified volume driver using --driver (-d) option. The --options (-o) allows to set driver specific options.

$ docker volume create -d local-persist -o mountpoint=/mnt/ --name=<volume-name>

Display detailed information of the specified volume

$ docker volume inspect <volume-name>

The docker volume ls command is used to locate the volume name or names to be deleted. Remove one or more volumes using the docker volume rm command as below.

$ docker volume ls
$ docker volume rm <volume_name> <volume_name>

Using the --filter (-f) option, list volumes by filtering only those which are dangling.

$ docker volume ls -f dangling=true

Get the assigned address for specified docker container

$ docker inspect <container-name>

To get the process id i.e. PID of the specified docker container we use the below command.

$ docker inspect -f '{{.State.Pid}}' <container-name>

Find IP addresses of the container specified by container name. The argument ultimately passed to the docker inspect command is the container id.

$ docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container-name>

The display the health check of a docker container.

$ docker inspect --format='{{json .State.Health}}' <container-name>

{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2017-07-21T06:10:51.809087707Z",
      "End": "2017-07-21T06:10:51.868940223Z",
      "ExitCode": 0,
      "Output": "Hello world"
    }
  ]
}

Execute the specified command in a running docker container. The command is not restarted if the container gets restarted.

$ docker exec <container-name> ps 
$ docker exec -it <container-name> /bin/bash

The exec command comes in handy for all kinds of debuging purposes, e.g. to ensure UDP ports are being listened we use the below netstat command..

$ docker exec -it <container-name> netstat -au

Run a one-time command in a new container. The -e option allows to set an environment variable while running the command. The -t option allocates a pseudo-TTY, while -i keeps the STDIN open even if not attached. The docker run command first creates a writeable container layer over the specified image, and then starts it using the specified command.

$ docker run -it -e "ENV=dev" <docker-image-name>

By default a container’s file system persists even after the container exits. The --rm flag allows to avoid persisting container file systems for short term processes and to automatically clean up the container and remove the file system when the container exits. The --rm parameter is ignored in detached mode with -d (--detach) parameter. By default all containers are connected to a bridge interface to make any outgoing connections, but a custom network can be provided by --network option.

$ docker run -it --rm --network net postgres-service psql -h postgres-service -U appuser

Get a list of all container IDs, only displaying numeric container ids.

$ docker container ls -aq

Display list of all images along with their repository, tags, and size. On passing the name or tag allows to list images by name and tag.

$ docker images
$ docker images java

Create a network with the specified driver using --driver (-d) option.

$ docker network create -d bridge <network-name>

Use the docker run command with the --net flag to specify the network to which you wish to connect your specified container.

$ docker run <container-name> --net=<network-name>

Get he list of Docker networks

$ docker network ls

Network inspect allows to get further details on networks.

$ docker network inspect <network-name>

To get the details of the default bridge network in JSON format we use below command.

$ docker network inspect bridge

Create a docker machine with a --driver flag indicating the provider on which the machine should be created on e.g. VirtualBox, DigitalOcean, AWS, etc.

$ docker-machine create --driver virtualbox <docker-machine-name>

The docker logs command shows information logged by a running container. The -t option allows to follow the log and -t option displays the timestamp.

$ docker logs -t -f <container-name>

Builds, (re)creates, starts, and attaches to containers for a service. Unless they are already running, docker-compose up also starts any linked services. When the command exits it stops all containers.

$ docker-compose up

Start all docker containers. With the -d (--detach) option specified, docker-compose will start all the services in the foreground and leaves them running. When –no-recreate option, if the container already exits, this will not recreate it.

$ docker-compose up -d

Build and start only the specified docker container.

$ docker-compose up <container-name>

The --scale flag allows to scale the number of instances of the specified service.

$ docker-compose up --scale <service-name>=3

The --no-deps argument for docker-compose up command doesn't start linked services.

$ docker-compose up -d --no-deps --build <container-name>

The --file or -f option allows to specify alternate compose file from the default "docker-compose.yml". Multiple configuration files can be supplied using -f option. The compose combines and builds the configuration in the order compose files were supplied. Subsequent files override and add to their predecessors.

$ docker-compose -f docker-compose.yml -f docker-compose.dev.yml up

The --build option allows to build images before starting containers.

$ docker-compose -f docker-compose-dev.yml up --build

The docker-compose with --force-recreate option allows to stop and recreate containers from fresh images every time even if their configuration and image haven't changed.

$ docker-compose up -d <container-name> --build --force-recreate

The --no-recreate does not create containers if they already exists.

$ docker-compose up -d <container-name> --build --no-recreate

The --remove-orphans removes containers for services not defined in the compose file.

$ docker-compose up -d <container-name> --build --remove-orphans

Starts existing containers for a service.

$ docker-compose start

Stop all the docker containers

$ docker-compose stop

Below down command, stops containers and removes containers, networks, volumes, and images created by up.

$ docker-compose down

The Run command runs a one-time command against a service. It starts a container, runs the command and discards the container.

$ docker-compose run <container-name> bash

The exec command allows to run arbitrary commands in the services. It is similar to run, but allows to attach to a running container and run commands inside it e.g. for debugging. By default it allocates a TTY to get an interactive prompt.

$ docker-compose exec <container-name> sh

Removes the stopped service containers without asking for any confirmation.

$ docker-compose rm -f

Stop the containers, if required, and then remove all the stopped service containers.

$ docker-compose rm -s

Remove the volumes that are attached to the containers

$ docker-compose rm -v

Rebuild the docker containers and tags it. It helps to rebuild whenever there is a change in the Dockerfile or the contents of its build directory.

$ docker-compose build

Get status of docker containers

$ docker-compose ps

The docker-compose log command displays output logs from all running services. The -f option means to follow the log output and the -t option gives the timestamps.

$ docker-compose logs -f -t

Validate and view the docker compose file.

$ docker-compose config

It also allows to test the resultant docker compose file where variables e.g. $SERVICE_PASSWORD need to populated by passing from the command line before the docker command as below. It is important to note that docker detects if environment variables are changed in dependent container compared to the existing running container and recreates the container again.

$ SERVICE_PASSWORD=secret docker-compose config

Docker Swarm

Swarmkit is a separate project which implements Docker’s orchestration layer and cluster management and is embedded in the Docker Engine. Docker swarm is a technique to create and maintain a cluster of Docker Engines. Many docker engines connected to each other form a network , which is called docker swarm cluster.

A docker manager initializes the swarm in a docker swarm cluster, and along with many other nodes executes the services. A node is an instance of the Docker engine participating in the swarm. Though docker manager can execute the services, its primary role is to maintain and manage docker nodes running the services. Docker manager also performs cluster and orchestration management by electing a single leader to conduct orchestration tasks. The manager node uses the submitted service definition to dispatch units of work called tasks to worker nodes. A task is an instance of running container which is part of a swarm cluster managed by docker manager. The docker manager assigns tasks to worker nodes according to the number of replicas set in the service scale. Once a task is assigned to a node, it cannot move to another node. The worker nodes receive and execute the corresponding tasks dispatched from manager node. The docker manager maintains the desired state of each worker node using their current state of assigned tasks reported by the agent running on each worker nodes. The docker Manager has two kinds of tokens, a manager token and a worker token. The worker nodes use the worker token to join the swarm as a worker node, while another node can join as a docker manager by getting the token from docker manager creating multi-manager docker cluster. The multi manager cluster has a single primary docker manager while multiple secondary docker managers. While a request to deploy the application (start a service) can be made to either the primary or secondary manager, any request to secondary manager is automatically routed to primary manager which is responsible for scheduling/starting container on the host. All the docker managers in a multi manager cluster form a Raft consensus group. Raft consensus algorithm enables to design fault-tolerant distributed system were multiple servers agree on the same information. It allows an election of a leader and for each subsequent request to the leader which is appended to its log, logs of every follower is replicated with the same information. It is highly recommended to have odd number of docker managers (typically 1, 3 or 5) to avoid split brain issue were more than one candidate gets equal majority aka tie. The Worker nodes communicate with to each other using gossip network.

There are two modes in which services are executed in docker swarm, namely replicated or global. The replicated mode allows to have multiple instances (task) of the service be executed in same docker host, depending on its load and capacity. Also it allows to have no instance of the service running on already loaded docker node. The global mode however ensures that one instance of the service is running on every node of the docker cluster. It ensures that unless all the nodes fail the service would still up and running across other nodes. It is used for critical services which required to be up all the time e.g. Eureka service.

A service is an higher abstraction which helps to run an image in a swarm cluster while swarm manages individual instances aka tasks. It is a an docker image which docker swarm manages and executes. When a desired service state is declared by creating or updating a service, the orchestrator realizes the desired state by scheduling tasks. Each task is a slot that the scheduler fills by spawning a container. The container is the instantiation of the task. Service creation requires to specify which docker image to use and which commands to execute inside running containers.

The services requested would be divided and executed across multiple docker nodes as tasks to achieve load balancing. Multiple tasks belonging to single or different service can be executed within a docker node. At any point of time when a node goes down in docker swarm cluster, the docker manager starts the tasks for the services running on stopped docker node on another nodes to balance the load and thus providing high availability of services. Auto load balancing ensures that during any node down time the docker manager will execute the corresponding down services on other nodes and also scale the services on multiple nodes during high load time. The docker manager uses an internal DNS server  for load balancing, which connects all the nodes in the docker cluster. Decentralized access allows to the service deployed in any node to be accessed from other nodes in the same cluster. Docker swarm also allows seamless rolling updates for each service with delay between individual nodes. Docker Swarm manages individual containers on the node for us.

A stack is a group of interrelated services that share dependencies, and can be orchestrated and scaled together. A single stack is capable of defining and coordinating the functionality of an entire application. The stack abstraction goes beyond the individual services and deals with the entirety of application services, which are closely interlinked or interdependent. Stacks allow for multiple services, which are containers distributed across a swarm, to be deployed and grouped logically. The services running in a Stack can be configured to run several replicas, which are clones of the underlying container. The stack is configured using a docker-compose file and it takes one command to deploy the stack across an entire swarm of Docker nodes. Stacks are very similar to docker-compose except they define services while docker-compose defines containers. Docker stack simplifies deployment and maintenance of multiple inter-communicating microservices and is ideal for running stateless processing applications.

Docker Swarm Commands

Initialize Docker Swarm using below swarm init command. The specified <ip-addr> would be the docker manager node's ip address which ideally should be same machine.

$ docker swarm init --advertise-addr <ip-addr>

The swarm init command's listen-addr option allows the current node to listen for inbound swarm manager traffic on the specified IP address.

$ docker swarm init --listen-addr <ip-addr>:2377

The swarm join command allows the node with specified IP address to join the swarm cluster as a node and/or manager.

$ docker swarm join --token <token> <ip-addr>:2377
$ docker swarm join --token <worker-token> <manager>:2377

Create a multi master docker cluster by joining the swarm as docker manager and then giving the manager token. Below we have 2 docker managers in the docker cluster.

$ docker swarm join --manager --token <manager_token> --listen-addr <master2-addr>:2377 <master1-addr>:2377

The below join-token allows docker swarm to manage join tokens. It is usually used to add the current node as a manager or a worker to the current swarm cluster (often by generating swarm join --token command).

$ docker swarm join-token (worker|manager)

Leave the current swarm cluster. When command is ran on a worker, the worker node leaves the swarm. The --force option on a docker manager removes it from the swarm cluster.

$ docker swarm leave --force

All below service commands can only run on docker manager.

Below command lists all the services running inside docker swarm.
$ docker service ls

Below command lists tasks running on one or more nodes for specified nodes.
$ docker service ps <name>

Create new services published on specified node port.

$ docker service create <name> -p host-port:container-port <image-name>
$ docker service create <name> --publish host-port:container-port --replicas 2 <image-name>
$ docker service create --name <name> alpine ping <host-name>

The replicas option allows to specify the number of tasks (instances) which the new created service will be executed.

$ docker service create --replicas 3 --name <name> <image-name>

With mode set as global for create service command, it downloads the specified image and starts the corresponding service on each single node of the cluster.

$ docker service create --mode=global --name=<name> <image-name>

Remove service running in docker swarm.

$ docker service rm <name>

Scale one or more replicated services

$ docker service scale <name>=5

Display details regarding the specified service

$ docker service inspect <name> --pretty

Update the service by increasing the number of replicated services.

$ docker service update --replicas 10 <name>

The service logs command shows information logged by all containers participating in a service or task.

$ docker service logs

List all the nodes present in the swarm cluster

$ docker node ls

Lists all the services (tasks) running on current node (by default).

$ docker node ps

Removes one or more nodes specified by id from the swarm cluster

$ docker node rm <id>

Stop allocating services to Manager-1 node

$ docker node update --availability drain <Manager-1>

Start allocating services to Manager-1 node

$ docker node update --availability active <Manager-1>

Deploy a new stack or update an existing stack. The --compose-file (-c) option allows to provide path to the docker compose file.

$ docker stack deploy <stack-name>

$ docker stack deploy -c docker-compose.yml <stack-name>

List all the stacks

$ docker stack ls

List all the services in the specified stack

$ docker stack services <stack-name>

List the tasks in the specified stack

$ docker stack ps <stack-name>