1. Knowledge bits

This page aims to provide practical examples of the technologies used the duration of the project.

Each example of a command should work in a terminal when copied. Try to understand the use and syntax of the various tools and techniques by dissecting them.

1.1. Markdown

More thorough Markdown examples & guides.

https://www.mweb.im/markdown.html

---

Bold

**Bold**

Cursive

_Cursive_

~~Striketrough~~

~~Striketrough~~
  • List
* List
  • [ ] Checklist
* [ ] Checklist

Link

[Link](https://duckduckgo.com/)

picture

![picture](https://duckduckgo.com/assets/icons/meta/DDG-icon_256x256.png)

Either use a URL or local link to a picture.

``Just add one more backtick ` up an below
your code
``

inline code

`inline code`

Create a code block by putting three backticks ` above and under your code.

Quote

> Quote
col1 col2 col3
val1 val2 val3 
| col1   | col2   | col3   |
| ------ | ------ | ------ |
| val1   | val2   | val3   |

:) ;) :joy: <3 :fire: :poop: 

:) ;) :joy: <3 :fire: :poop:

Use this GitHub project for a complete list of emoticons.

1.1.1. VS Code Extensions

1.2. Docker

1.2.1. Dockerfile

Container images should be as small as possible to reduce the time to build and pull them. The design of your Dockerfile has a lot of influence on this.

It starts with the base image. Use one from a trusted or even verified vendor/community to begin with. See if there is a slim-variant for this image.

Instead of FROM debian:bullseye, I use FROM debian:bullseye-slim.

Container images are build using layers. These layers should be minimized to the reduce the container image size. Every command in a Dockerfile will create a image layer. Thus you want to combine commands together using && and \.

Example:

export NAME=test123 \
&& echo $NAME

When using the package manager for installing software use --no-install-recommends for limiting what you'll install. And use apt-get clean and rm -rf /var/lib/apt/lists/* to clean stuff afterwards.

The beginning of a Dockerfile could look like this:

FROM debian:bullseye-slim

RUN apt-get update \ 
    && apt-get install -y --no-install-recommends \ 
    openssh-server \
    curl \
    git \
    wget \
    unzip \
    gawk \
    jq \
    python3-pip \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

When you need files in your container a couple options are available.

  • Copy the files in to the container at build time. However, this increases the image size and means that on a file change the whole container must be build again.
  • Docker/Kubernetes has the concept of volumes. A directory on the host is mounted then mounted as directory in the container at runtime.
  • Retrieve files from the internet with either Git, curl or wget. This option is mostly used when building a container image. However, you could implement it as entrypoint script to always retrieve a latest version on container runtime.

1.2.2. Running a container

1.3. Parsing output & files

1.3.1. tr

tr is short for “translate”. It is a member of the GNU coreutils package. Therefore, it’s available in all Linux distros.

The tr command reads a byte stream from standard input (stdin), translates or deletes characters, then writes the result to the standard output (stdout).

Example: Convert lowercase characters to uppercase and a dash to underscore.

export NAME=jcr-demo

echo $NAME

echo $NAME | tr 'a-z' 'A-Z'

echo $NAME | tr 'a-z' 'A-Z' | tr '-' '_'

Converting to uppercase can also be done as follows:

export NAME=jcr-demo

echo {$NAME^^}

1.3.2. gawk

The basic function of awk is to search files for lines (or other units of text) that contain certain patterns. When a line matches one of the patterns, awk performs specified actions on that line. awk continues to process input lines in this way until it reaches the end of the input files.

Parse stdin, use : as seperator and print the first occurrence in the list for all entries.

cat /etc/passwd | gawk -F: '{print $1}'

Have the same result, but use a file for input instead of stdin.

gawk -F: '{print $1}' /etc/passwd

1.3.3. jq

jq is a tool to parse JSON data structures. It is not installed by default, it can be downloaded from here. You could also try the jq interactive playground.

See the example to extract a particular value from a JSON data structure.

cat << EOF > example-data.json
{
    "glossary": {
        "title": "example glossary",
        "GlossDiv": {
            "title": "S",
            "GlossList": {
                "GlossEntry": {
                    "ID": "SGML",
                    "SortAs": "SGML",
                    "GlossTerm": "Standard Generalized Markup Language",
                    "Acronym": "SGML",
                    "Abbrev": "ISO 8879:1986",
                    "GlossDef": {
                        "para": "A meta-markup language, used to create markup languages such as DocBook.",
                        "GlossSeeAlso": ["GML", "XML"]
                    },
                    "GlossSee": "markup"
                }
            }
        }
    }
}
EOF

jq -r ".glossary.title" example-data.json

1.3.4. yq

yq is similar to jq, however yq can parse YAML data structures. You can download it from here.

See the following example to replace a value in a YAML file.

cat << EOF > kubeconfig.yaml
apiVersion: v1
kind: Config
clusters:
- cluster:
    api-version: v1
    certificate-authority-data: CA-cert-xxx
    server: "https://144.101.110.124:6443"
  name: "test-cluster"
contexts:
- context:
    cluster: "test-cluster"
    user: "kube-admin-test-cluster"
  name: "test-cluster"
current-context: "test-cluster"
users:
- name: "kube-admin-test-cluster"
  user:
    client-certificate-data: client-cert-xxx
EOF

yq -i eval '.clusters[].cluster.server = "https://localhost:7443"' kubeconfig.yaml

cat kubeconfig.yaml

1.3.5. Regex (Regular Expressions)

Regular expression testers: Regexr Regex101

1.4. Bash scripting

1.5. Secret management (with Doppler)

1.5.1. Doppler CLI

Create a Doppler account, create a workspace and install the Doppler CLI.

Login to Doppler using the CLI.

doppler login

Create a Doppler project.

doppler projects create test_project

List the environments from the new project.

doppler environments -p test_project

Delete the staging environment with the slug stg.

doppler environments delete -p test_project stg

Create a new config config1 under the development environment, you have to specify the environment’s slug as prefix when creating a branch config.

doppler configs create dev_config1 -p test_project -e dev

Create a service token for dev_config1.

doppler configs tokens create --project test_project --config dev_config1 service_token --plain

Delete the test_project.

doppler projects delete test_project

1.5.1.1. Use case for SSH key and Kubeconfig file

Uploading a SSH key.

cat ~/.ssh/surf | doppler secrets set SSH_PRIVATE_KEY -p test_project -c dev

Downloading a SSH key.

doppler secrets get SSH_PRIVATE_KEY --plain > ~/.ssh/ -p test_project -c dev

Uploading a kubeconfig file.

cat ~/.kube/config | doppler secrets set KUBECONFIG -p test_project -c dev

Downloading a kubeconfig file

doppler secrets get KUBECONFIG --plain > ~/.kube/config -p test_project -c dev

1.5.2. Python

1.5.3. Docker

1.5.4. Kubernetes

1.6. Kubernetes

1.6.1. Configuration management

1.6.1.1. Kustomize

See more thorough examples and explanations of Kustomize at its repository and documentation.

Using Kustomize build and pipe kubectl apply.

kustomize build overlays/development | kubectl apply -f -

Using the built-in kubectl -k option. Take note that kubectl doesn't use the latest Kustomize version and thus not all functionality is included.

kubectl apply -k overlays/development

kustomization.yaml

Every kustomization.yaml should, like every Kubernetes manifest declare the apiVersion and kind to be used.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

Resources

Resources define which Kubernetes manifests are applied as part of the kustomization.yaml.

resources:
  - deployment.yaml
  - service.yaml
  - ingress.yaml

Bases

You can use other kustomization.yaml files as base and apply extra configuration to it. The parent kustomization.yaml is called an overlay.

bases:
- ../bases/application

namespace

All resources that are touched by a kustomization.yaml can be configured to use the same Kubernetes namespace.

namespace: your-app-namespace

images

Images used in your Kubernetes deployment manifests can be patched as followed.

images:
  - name: python
    newName: python-bullseye
    newTag: 3.10-slim-bullseye
  - name: mariadb
    newName: mariadb-focal
    newTag: 10.7.1-focal

patchesStrategicMerge

patchesStrategicMerge can be used to patch an existing resource. You can define it as inline YAML or a file.

patchesStrategicMerge:
  - |-
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: app
    spec:
      template:
        spec:
          containers:
            - name: ctfd
              envFrom:
                - secretRef:
                    name: secret-app
                - configMapRef:
                    name: configmap-app
  - ingress.yaml

configMapGenerator

ConfigMap is the Kubernetes concept of defining environment variables. configMapGenerator is the Kustomize implementation of a ConfigMap and generates it on a Kustomize build. It can either use the behaviors to merge or replace existing values.

configMapGenerator:
  - name: configmap-literal
    behavior: merge
    literals:
      - ENV1=Value1
      - ENV2=Value2
  - name: configmap-file
    behavior: replace
    env: values.env

secretGenerator

The concept of a secretGenerator is similar to that of the configMapGenerator, but for Kubernetes secrets.

secretGenerator:
  - name: k8s-secret
    literals:
      - secret=Value1
  - name: container-registry-secret
    files:
      - secrets/.dockerconfigjson
    type: kubernetes.io/dockerconfigjson

The accompanying file for this secret looks as followed. See the Kubernetes docs for more information on Pull an Image from a Private Registry. Remember to base64 encode the value USERNAME:PASSWORD for auth.

{
  "auths": {
    "https://registry.gitlab.com": {
      "username": "USERNAME",
      "password": "PASSWORD",
      "email": "",
      "auth": "USERNAME:PASSWORD"
    }
  }
}

generatorOptions

Normally generators generate NameSuffixHash, but then it's hard to reference this in a deployment because the name will be different everytime. You can disable this with the following snippet.

generatorOptions:
  disableNameSuffixHash: true

namePrefix & nameSuffix

namePrefix: development-
nameSuffix: -2021

commonLabels

Labels provide the foundation for grouping objects, they can be used to select objects and to find collections of objects that satisfy certain conditions.

commonLabels:
  app.kubernetes.io/environment:
  app.kubernetes.io/name:
  app.kubernetes.io/instance:
  app.kubernetes.io/version:
  app.kubernetes.io/component:
  app.kubernetes.io/tier:
  app.kubernetes.io/purpose:

commonAnnotations

Kubernetes annotations are key/value pairs designed to hold non-identifying information that can be leveraged by tools and libraries.

commonAnnotations:
  a8r.io/description:
  a8r.io/owner:
  a8r.io/chat:
  a8r.io/bugs:
  a8r.io/logs:
  a8r.io/documentation:
  a8r.io/repository:
  a8r.io/support:
  a8r.io/runbook:
  a8r.io/incidents:
  a8r.io/uptime:
  a8r.io/performance:
  a8r.io/dependencies:

1.6.1.2. Helm

Find a Helm chart you want to install at artifacthub.io.

Add the repo data to your workstation.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

You can lookup the chart version for what you are going to install.

helm search repo prometheus-community/kube-prometheus-stack

Install the Helm chart.

helm install kube-prometheus prometheus-community/kube-prometheus-stack --create-namespace --namespace monitoring

Uninstall the chart.

helm uninstall kube-prometheus

When the default configuration of a Helm chart is not sufficient additional values can be configured. You can get a list of all possible values documented by the vendor.

helm show values prometheus-community/kube-prometheus-stack

Create a values file for the kube-promethues-stack Helm chart.

cat << EOF > helm-values.yaml
alertmanager:
  enabled: false
EOF

Install the Helm chart as follows.

helm install kube-prometheus prometheus-community/kube-prometheus-stack --create-namespace --namespace monitoring -f values.yaml

However, when Helm is not an option for you alternative methods exist for still using the original chart.

helm template kube-prometheus prometheus-community/kube-prometheus-stack > kube-prometheus.yaml

1.6.1.3. ytt templating

1.6.2. Quality of life improvements

1.6.2.1. Helm package manager

Helm is often described as the package manager for Kubernetes, like apt is for Debian/Ubuntu. It packages or bundles various Kubernetes manifests to deploy multiple application components in Kubernetes. Popular applications are available as Helm charts in public repositories, like container images on Docker Hub. Artifacthub.io hosts various Cloud-Native artifacts availible to the community, including Helm charts. Install Helm for your system with the following link.

1.6.2.2. Cloud-Native Artifacts & HUBs

When something already exists you don't have to waist resources by building it yourself. Cloud-Native artifacts are often shared on a HUB of sorts. From whom Docker hub and ArtifactHUB are the most prominent ones.

Besides Helm Charts, ArtifactHUB also offers: - OLM operators - Krew kubectl plugins - Tekton tasks - Helm plugins - Falco rules - Keptn integrations - Tinkerbell actions - OPA policies - KEDA scalers - Tekton pipelines

Other HUBs are: - OperatorHub for finding Kubernetes Operators. - R2Devops hub for finding ready to use GitLab CI Jobs. - ExporterHub for finding Prometheus exporter, Alert rules and Dashboards. - PromCat for finding Prometheus exporter, Alert rules and Dashboards.

1.6.2.3. k9s CLI dashboard

When deploying to Kubernetes, kubectl get pods will be run often. K9s replaces this command with a nice dashboard in your terminal. It also provides a log viewer and quick port forward shortcut. Install it for your system with the following link. The command k9s will spawn the terminal dashboard. Availible shortcuts are shown at the top. Cluster resources can be navigated with the arrow keys and selected with Enter, press esc to exit a selected resource and Ctrl+C for exiting the application. Press 0 to show all namespaces. Your're also able to quickly port-forward a container and edit and delete resources.

1.6.2.4. Lens GUI dashboard

When in need of something more feature-rich and complete, Lens the Kubernetes IDE can be used. This looks more like the Rancher dashboard used by the JCR in production. With the big difference being that Lens runs on your workstation. It utilizes your Kubeconfig for finding and authenticating to your cluster and only provides access to resources provided in your RBAC (Roll Based Access Control) configuration. Lens makes it more easy to mange multiple clusters and provides a general overview of resources. Lens automatically captures basic Prometheus metrics and also provides Helm integration and mangement.

My use cases for Lens are; more quickly identify issues with its logging capabilities and manually editing/deleting resources, for when i'm tinkering. Because of all its capabilities, Lens is described as the Kubernetes IDE. Install it for your system with the following link.

1.6.2.5. krew: kubectl plugin manager

Kubectl plugins extend kubectl's native capabilities, these plugins can be searched and installed using krew.

1.6.2.6. Bash Aliases

For common commands u can use aliases to shorten them. You can add aliases to ~/.bashrc with your favorite text editor and activate them with the following command . ~/.bashrc. I've added the following to the end of my .bashrc file:

alias k='kubectl'
alias kgn='kubectl get nodes'
alias kgp='kubectl get pods'
alias kgpa='kubectl get pods -A'
alias kgs='kubectl get service'
alias kdes='kubectl describe'
alias kl='kubectl logs'
alias ka='kubectl apply -f'
alias kdel='kubectl delete'
alias kdf='kubectl delete -f'

1.6.2.7. Bash auto-completion

Enable auto-completion on Bash with the following commands:

source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

1.6.2.8. Multiple Kubernetes clusters & contexts

See current cluster contexts:

kubectl config get-contexts

Create cluster contexts for the development & production cluster with:

export KUBECONFIG=$KUBECONFIG:kube_config_dev.yaml:kube_config_prod.yaml

Check if the cluster contexts are updated:

kubectl config get-contexts

Use a cluster context:

kubectl config use-context dev-cluster

1.6.3. Troubleshooting tips

1.6.3.1. Namespace stuck in termination

When a namespace is stuck in termination you can resolve it with the following oneliner. It will ask you to enter the name of the corresponding namespace you want to delete.

echo 'namespace to delete: ' && read && kubectl get namespace ${REPLY} -o json | jq '.spec = {"finalizers":[]}' | kubectl replace --raw "/api/v1/namespaces/${REPLY}/finalize" -f -

Finalizers are a protection for related resources within a namepace. The oneliner will empty the finalizers array. This oneliner however requires jq to be installed.

You can install jq with:

sudo apt-get install jq

1.6.3.2. Spawning a DNS troubleshooting pod

The following command can be used to spawn a pod with DNS utilities for troubleshooting purposes. When you exit the pod will be deleted.

kubectl run -it --rm --image anubhavmishra/tiny-tools dnstools`

1.6.3.3. Delete Evicted Kubernetes pods

Evicted pods can relate to various porblems. A common one is resource limitations and various services competing for consumption of those resources. Resource constraints should be set in place for Kuberentes pods. More explanation on pod eviction

https://support.huaweicloud.com/intl/en-us/cce_faq/en-us_image_0281530643.png

You can delete evicted pods as follows, you have to specify the namespace twice. This command can be more effiecient, e.g. iterating over namespaces or setting values. Do you know how? Feel free to publish an improvement!

kubectl get pods -n {namespace} | grep Evicted | awk '{print $1}' | xargs kubectl delete pod -n {namespace}

1.6.3.4. Ingress-nginx deployment error

Redeployment of the ingress-nginx controller might fail on a cluster to which it has been deployed before, with a error like this:

helm upgrade --install ingress-nginx --create-namespace --namespace=ingress-nginx -f ./dont/ingress-nginx-helm-values.yaml ingress-nginx/ingress-nginx
Release "ingress-nginx" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "ingress-nginx" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annot
ation validation error: missing key "meta.helm.sh/release-name": must be set to "ingress-nginx"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "ingress-nginx"

If this happens, you can create a YAML-file containing the deployment as it would be ran by Helm and then delete it’s contents with kubectl (and remove the temporary YAML-file afterwards):

helm template ingress-nginx --create-namespace --namespace=ingress-nginx -f ./dont/ingress-nginx-helm-values.yaml ingress-nginx/ingress-nginx > nginx.yaml
kubectl delete -f nginx.yaml
rm nginx.yaml

1.7. Miscellaneous

Generate a random password with Bash commands.

< /dev/urandom tr -dc _A-Z-a-z-0-9 | head -c${1:-15};echo;

1.8. Other resources