Kubernetes inside Docker Image

This post runs you through the process of running kubernetes inside containers. This is mostly useful when you want to test some applications which interact with kubernetes configuration.

Example: Kubernetes Operators.

Use Cases

The main use cases we come accross is running the test cases in jenkins where we need to sping up the kubernetes clusters. Let us suppose say we want to run and test helm charts or we want to test the operators we developed on the kubernetes cluster. In this scenarios sometimes we may change the configuration of the kuberntes cluster which may effect other applications deployed inside the kubernetes cluster.

Pre-requisite

Before proceeding, you need to have the knowledge of Docker Inside Docker(DIND) and K3s by Rancher.

• DIND
• K3s

Docker File

FROM docker:19.03.7-dind

RUN apk add curl bash

# Install kubectl
RUN curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
RUN install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Install k3d to run k3s cluster
RUN curl -s https://raw.githubusercontent.com/rancher/k3d/main/install.sh | TAG=v4.0.0 bash

# Set docker host env to use while bringing up docker daemon
ENV DOCKER_HOST=unix:///var/run/docker.sock

# Copy kind entrypoint file
COPY kind-entrypoint.sh /usr/local/bin/kind-entrypoint.sh

RUN ["chmod", "+x", "/usr/local/bin/kind-entrypoint.sh"]

ENTRYPOINT ["/usr/local/bin/kind-entrypoint.sh"]

Kind Entrypoint

#!/bin/bash
# This script is used to run docker daemon before bringing up k3s cluster
# using k3d
# Exit script when there is any error
set -e
echo "Running docker service and creating cluster"

# Bring docker daemon up using DOCKER_HOST variable 
dockerd &>docker.logs &

# Wait until docker daemon is available. 
while (! docker stats --no-stream ); do
  # Docker takes a few seconds to initialize
  echo "Waiting for Docker to launch..."
  sleep 1
done

# Run k3d cluster with name default_cluster
echo "Creating cluster with defaultCluster as name"
k3d cluster create defaultCluster

# Run commands at start of container
if [ "$#" -gt "0" ]; then
  $@
fi

Source Repo

Don’t

As we are running the container with privileged access we will have access to host kernel. So,

• running these containers on production environment is not advisable.
• running these containers in shared resources is not advisable.