Automating Docker Container Monitoring

Zvika Meiseles, CTO CorrelsenseIn my previous blog post (APMing docker-contained applications), I discussed two ways to approaching the monitoring of software running inside docker containers:

  • Internal – all the monitoring software components run inside the monitored container.
  • Shared agent – there are common monitoring components that run on the host.

The benefits of using the first configuration (internal) are that it’s a simpler and more stable deployment as the containers don’t change between the development and production stages.

Following the post, a question came to mind: “What about the existing containers and applications? Is it really necessary for a company with a fully-operational docker infrastructure to modify their docker images in order to include monitoring?”

After browsing the docker API, I discovered that it is possible (and quite easy since docker 1.8) to copy files into a running container, and it is also possible to execute commands inside a running container.

So I decided to brush up on my (non-existing) python skills and automate the process as follows:

  • Monitor the docker environment for container start-ups.
  • When a container starts:
    • Copy files into it
    • Run a command

The docker API [1] enables the easy execution of these tasks, and the python client [2] makes it even easier.

Firstly, make sure you have the docker-py package installed:

Then, dive into python. Step 1 is to monitor docker for container ‘start’ events:

After receiving a notification for every container ‘start’ event, let’s see what the handler can do with it.

The ‘event’ parameter passed to the handler is a dictionary, and we can see a list of its attributes every time a container is started, for example:

Most of the properties here are of no use to us, except for:

  • Action – obviously, we want to monitor ‘start’ events.
  • from – this is the name of the image the container was created from. We will use it for filtering purposes if we want to selectively inject files into containers.
  • id – this is the ID of the new container, which we will use to identify the container we need to operate on.

Now, let’s say we want to inject a script into our containers that registers the new container with our monitoring dashboard.

All we need is to create the script, and then pack it as a [.tar] archive:

Going back to our python code, we will now need to perform the following operations on the newly created container:

  1. Execute the command mkdir -p /tmp/monitoring
  2. Copy the tar into the container and extract it inside the directory created in (1).
  3. Execute the command /tmp/monitoring/register.sh

If this is done using docker, it will appear as follows:

Next, we need to translate this to python, using the docker-py client API:

And voila! Instant “monitoring” for every docker container created on this host, using all images – past present and future. You can of course use this same code to inject non-monitoring tasks and scripts.

All the code is available in github [3], including some improvements:

  • Image name filtering – inject into containers created only from desired images.
  • Command line arguments – for source tar/directory, destination path and image filter.
  • Memory mapped file access – for improved memory consumption.

Important links

[1] http://docs.docker.com/engine/reference/api/docker_remote_api/
[2] http://github.com/docker/docker-py
[3] http://github.com/zvikam/docker-atstart

About the Author: Zvika Meiseles, is the  CTO of Correlsense and  has over 20 years’ of software development experience − ranging from embedded, low-level driver development and hardware integration to scripting and back-end programming. Zvika also manages the Data Collection team which he helped establish.

 

Paste your AdWords Remarketing code here