Advanced container deep-dive

Tomáš Tomeček
Peter Schiffer

Intro

Agenda

  • Debugging
  • Building images
  • Networking
  • Capabilities
  • SELinux
  • Namespaces
  • Seccomp

Environment

  • Fedora 25 Atomic Host
    • $ rpm-ostree status
  • Vagrant
  • Useful aliases: 
    $ alias docker="sudo docker"
$ alias d="sudo docker"
$ alias
$ docker images

Debugging (getting logs)

  • Let's start with nginx... 
    $ docker run -d -p 80:80 --name nginx modularitycontainers/nginx
$ curl -v localhost
$ curl -v localhost/does-not-exist
  • How does my container gets run? 
    $ docker inspect nginx
$ docker inspect modularitycontainers/nginx
$ docker history --no-trunc modularitycontainers/nginx

Debugging (getting logs) II

  • How do I see logs of my application? 
    $ docker logs nginx
$ docker logs -f nginx
  • How do I get logs via API? 
    $ curl --unix-socket /var/run/docker.sock \
  "http:/latest/containers/nginx/logs?stderr=1&tail=all"
$ curl --unix-socket /var/run/docker.sock \
  "http:/latest/containers/nginx/logs?stderr=1&stdout=1&tail=1&follow=true"

Debugging (getting logs) III

Debugging (getting logs) IV

  • I want to see logs from files inside container. 
    $ docker run --name db -it -d -p 5432:5432 \
  -e POSTGRESQL_USER=pguser \
  -e POSTGRESQL_PASSWORD=pgpasswd \
  -e POSTGRESQL_DATABASE=our_db \
  registry.access.redhat.com/rhscl/postgresql-95-rhel7
  • Let's see the logs then... 
    $ docker exec -ti db bash -c \
  "tail -f /var/lib/pgsql/data/userdata/pg_log/*.log"
$ docker stop db
$ docker cp db:/var/lib/pgsql/data/userdata/pg_log/ ./logs

Debugging - tools

  • They said it always works: 
    $ docker run --rm -ti --name=working-container this-always-works
              
    $ sudo strace docker run --rm -ti this-always-works
              
    $ docker run -ti --name=working-container this-always-works bash
              
    $ docker inspect working-container
$ docker history this-always-works
              
    $ docker run -ti --rm --name=working-container \
  --entrypoint="/bin/bash" this-always-works -l

Debugging - tools II

  • Let's finally run strace 
    $ strace /this-errors-out
              
    $ id
$ dnf install strace
              
    $ docker run -ti --rm --user=root \
  --cap-add=SYS_PTRACE --name=working-container \
  --entrypoint="/bin/bash" this-always-works -l
$ dnf install strace
$ strace /this-errors-out

Building images

  • How do I handle secrets during build? 
    $ docker history --no-trunc secret-image
$ docker run --rm -ti secret-image bash
$ ls -lha /home/app/.ssh
$ docker save -o image.tar secret-image
$ sudo mc
  • Is layering that important? 
    $ docker images fat
$ docker history --no-trunc fat
$ docker images slim
$ docker history --no-trunc slim

Building images II

  • Cache 
    $ docker run --rm current-date
$ docker build --tag=current-date ./cache
$ docker run --rm current-date

Capabilities

$ man capabilities
  • What capabilities a container has by default? 
    $ docker run --rm -ti fedora:25 getpcaps 1
  • What capabilities a privileged container has by default? 
    $ docker run --rm -ti --privileged fedora:25 getpcaps 1

Capabilities II

  • CAP_CHOWN — Make arbitrary changes to file UIDs and GIDs 
    $ docker run --rm -ti --cap-drop=chown fedora:25 bash

[root@9c433e5d4427 /]# chown 2:2 root/.bashrc
chown: changing ownership of 'root/.bashrc': Operation not permitted

[root@9c433e5d4427 /]# ll root/.bashrc
-rw-r--r-- 1 root root 176 Feb  5  2016 root/.bashrc

More info: rhelblog.redhat.../secure-your-containers-with-this-one-weird-trick/

Capabilities III

  • CAP_DAC_OVERRIDE
  • CAP_NET_BIND_SERVICE — service can bind to ports <1024
  • CAP_NET_RAW
  • CAP_SYS_ADMIN

Networking

Source: https://www.oreilly.com/learning/what-is-docker-networking

Networking II

  • So how does docker networking work by default? 
    $ docker network ls
$ docker create -ti --name=toy -p 8000:8000 fedora:25 bash
$ ip addr show docker0
$ docker network inspect bridge
$ ip route
$ iptables -t nat -L -nv
$ iptables -L -nv

shaded-enmity/docker-network-capture

More info: docs.docker.com/.../networking/

More info: http://windsock.io/tag/docker-proxy/

SELinux

  • Introduction: 
    $ getenforce
$ docker create -ti -v $PWD/volume:/volume --name=toy fedora:25 bash
$ ll -Z -d ./volume
$ docker start toy
$ docker attach toy
$ ll -Z -d /
$ touch /volume/file

SELinux II

  • -v path:/container_path:z — shared
  • -v path:/container_path:Z — not shared 
    $ ll -Z /etc/shadow
$ docker run --rm -ti -v /etc/shadow:/hostshadow:Z \
  --name=toy fedora:25 bash
$ ll -Z /hostshadow
$ ll -Z /etc/shadow
$ restorecon -v /etc/passwd
$ ll -Z /etc/shadow
$ restorecon -v -F /etc/passwd
$ ll -Z /etc/shadow

Be careful relabeling volumes with Container run times.

SELinux III

  • Is there an alternative? 
    $ docker run --rm -ti -v $PWD/volume2:/volume/:Z \
  --security-opt=label:disable --name=toy fedora:25 bash
$ ll -Z /
$ touch /volume/file
$ ll ./volume2
$ ll -Z -d ./volume2

Namespaces

  • Please show me the details of namespaces. 
    $ ll --color=no /proc/self/ns/
$ docker run --rm -ti fedora:25 ls -lha /proc/self/ns

Cgroups

  • How are Cgroups used? 
    $ docker create --name=toy --memory 256m \
  --cpu-period=50000 --cpu-quota=25000 \
  fedora:25 sleep 9999
$ docker start toy
$ find /sys/fs/cgroup/ | grep ID
$ cat /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-ID.scope/cpu.cfs_period_us

seccomp

  • What is seccomp? 
    $ cat seccomp/profile.json
$ docker run -ti \
  --security-opt seccomp=seccomp/profile.json \
  fedora:25 bash
$ chmod 0000 /root/anaconda-ks.cfg

Questions are welcome!