Most projects I am currently working on use Docker Container Engine as target environment. Besides the basic Docker engine most projects use Container Orchestration Platforms to add production ready features to their deployment. Although Kubernetes and its derivates became the go to enterprise solution throughout the last couple of years, I tend to use Docker Swarm Mode on some occasions. It just comes out of the box, does not require much configuration and works for most smaller use cases.
A big trade-off one has to make when using Docker Swarm Mode is the absence of a management user interface. Although I am a big fan of native CLI usage, I can understand the benefits of a nice looking and easy to use UI without the need to connect to docker hosts via SSH. A widely used solution for this shortcoming ist to use third party management UIs, such as Portainer. As stated on the website, „PORTAINER IS AN OPEN-SOURCE LIGHTWEIGHT MANAGEMENT UI WHICH ALLOWS YOU TO EASILY MANAGE YOUR DOCKER HOSTS OR SWARM CLUSTERS“.
The beauty of Portainer is not only its ease of use but also its ease of deployment. Following the documentation you can be up and running just by using a few commands. One can see that the development team’s focus is primarily to provide simple and easy deployment scenarios. All this works perfect, until you get serious and want to run the Portainer Agent deployment on a Docker Swarm Cluster with User Namespace Remap feature enabled for security reasons. Suddenly agents can not connect to the docker daemon via the standard Unix docker.sock port, an issue already described and discussed on GitHub.
Following the workarounds in the issue mentioned above, it is possible to manage each docker engine as separate Docker API over TCP endpoint. However, you never achieve the overall Docker Swarm view which could be achieved by a working agent deployment. After trying out a few things and starting a Thread in Portainer’s Slack Channel, I was able to create a deployment setup which is not so intuitive and easy but seems to provide the full functionality running on a Docker Swarm with User Namespace Remap enabled. Read on if you need a similar setup. In order to go through this configuration the following prerequisites are required:
- Running Docker Swarm
- User Namespace Remap is enabled
- SSH access to Docker Swarm hosts
- Privileges to execute docker commands
Portainer Agent Deployment
To start of with the figure below shows the Docker Swarm setup. We have three docker hosts each running a portainer agent container. In the standard portainer agent deployment all three agents form a cluster. The agent cluster is responsible to consolidate all information required by Portainer UI and make it available through the agent’s API. Portainer UI can then query any agent within the cluster. Portainer UI itself is only running on specific management nodes as service.
First of all we need an overlay network for our Portainer agent cluster:
1 $> docker network create -d overlay --attachable portainer_agent_network
After creating the network check its Subnet setting using docker network inspect. In my case it was set to 10.0.1.0/24. Every container within our deployment will have to use this overlay network. The next step is to start Portainer agents. The standard stack deployment does not work when user namespace remapping is active as agents do not have the permission to access docker.sock file on docker hosts. Remapping is configured on docker engine level and is therefore applied to all containers. Currently there is no option to overwrite the remapping for specific services in swarm mode (open issue regarding this feature on GitHub). However, it is possible to do so when using the plain docker run command:
1 $> docker run -d --name portainer_agent_one --network portainer_agent_network --ip 10.0.1.10 --userns=host -e AGENT_CLUSTER_ADDR=10.0.1.10 --mount type=bind,src=//var/run/docker.soc
This docker run command will start the agent on the first host. The –userns=host option forces docker to use the host user namespaces. As result the agent has proper access rights for docker.sock file. Furthermore, one has to change the volumes mount to the remapped volumes folder at /var/lib/docker/uid.guid/volumes. Finally, due to the way the portainer agents resolve all cluster members in a docker swarm, it is required to set a fix IP address using –ip 10.0.2.10 option (take the overlay network subnet into account). We will need this IP in order to start the agents on other hosts. To start the agent on the second host the following command can be used:
$> docker run -d --name portainer_agent_one --network portainer_agent_network --ip 10.0.1.10 --userns=host -e AGENT_CLUSTER_ADDR=10.0.1.10 --mount type=bind,src=//var/run/docker.sock,dst=/var/run/docker.sock --mount type=bind,src=//var/lib/docker/123456.123456/volumes,dst=/var/lib/docker/volumes portainer/agent
Repeat this run command on each host and provide a unique container name. Note that the -e AGENT_CLUSTER_ADDR option is set to the first agent’s fixed IP address. When you are finished you can check if the agents were able to create a cluster using the following command:
$> docker logs portainer_agent_one
2018/09/04 14:01:19 [INFO] serf: EventMemberJoin: 3d03424816e7 10.0.1.20
2018/09/04 14:01:19 [INFO] serf: EventMemberJoin: 74635795832b 10.0.1.10
2018/09/04 14:01:19 [INFO] - Starting Portainer agent version 1.1.2 on 0.0.0.0:9001 (cluster mode: true)
2018/09/04 14:01:42 [INFO] serf: EventMemberJoin: d0ee6a36e8cf 10.0.1.30
Now that all agents are running, we can create a portainer service within the same network. Make sure to use one of the agent hostnames as Docker API Endpoint:
$> docker service create --name portainer --network portainer_agent_network --publish 9000:9000 --replicas=1 --constraint 'node.role == manager' portainer/portainer -H "tcp://portainer_agent_one:9001" --tlsskipverify
Afterwards the docker swarm should be available in Portainer UI as one endpoint. Although this deployment approach does not comply to portainer’s design principles, it does make it possible to run Portainer UI in a Docker Swarm while using advanced security mechanisms. For large swarms it will not be feasible and error prone to start all portainer agents by hand, nevertheless with a little bit of scripting and DevOps provisioning magic it can be a process worth automating.
Thanks again to the Portainer team for the support in their Slack channel!