I’m setting up a Kubernetes cluster at home on my Raspberry Pi machines, mostly because I can. At the time of writing, some of the things to be done are running containerized Pi-Hole and Jellyfin services. I have 2 Raspberry Pi 3Bs and 1 Raspberry Pi 4B. I’m using the RPi 4B to run an OMV NAS mostly because the USB3.0 ports will allow for much faster file transfer rates on my HDD.

Stuff being used Link to heading

  • 2x RPi 3B
  • 1x RPi 4B (running OMV with NFS enabled)
  • 1x WDBlue 2TB HDD, connected to RPi 4B

Any compute machine capable of running Linux and having some storage attached will work. Results may vary based on actual hardware.

Prerequisites Link to heading

All the usual setup for a new Raspberry Pi should be done, such as creating an admin user and removing the pi user, setting up SSH access keys and static IPs.

Prepare Nodes Link to heading

Update the firewall to legacy configs.

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Enable cgroup_memory for each Raspberry Pi. Add the following to the end of the line in /boot/cmdline.txt

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Reboot after saving.

sudo reboot now

Setup K3s Master Node Link to heading

Access master node via SSH and install k3s using the install script provided.

# Needed for running in non-root
export K3S_KUBECONFIG_MODE=644

# Don't install traefik and load balancer
export INSTALL_K3S_EXEC=" --no-deploy servicelb --no-deploy traefik"

curl -sfL https://get.k3s.io | sh -

Ensure k3s is running.

sudo systemctl status k3s

The Active: status should show active (running). If there are errors, they would show up in this check as well.

Setup K3s agent on the worker nodes Link to heading

Set the same ENV variables as for the master node as well as the master node URL and token. Copy the token from sudo cat /var/lib/rancher/k3s/server/node-token in the master node.

export K3S_KUBECONFIG_MODE=644

export K3S_URL="https://<master_node_ip>:6443"

export K3S_TOKEN=<token>

curl -sfL https://get.k3s.io | sh -

Verify that k3s-agent is running with

sudo systemctl status k3s-agent

Verify Nodes are up Link to heading 

# In the master node
kubectl get nodes -o wide

Both nodes should be visible.

Run kubectl locally Link to heading

  • Copy the contents of /etc/rancher/k3s/k3s.yaml from the master node to your local machine
  • Replace all instances of 127.0.0.1 with the IP address of the master node
  • Use that file for your kubeconfig export KUBECONFIG=<file_path>

Local Helm chart setup Link to heading

Install helm, for MacOS brew install helm or any other way.

Create a Helm Chart Link to heading

Create a helm chart that will have all the required components like the Metallb load balancer and Nginx Ingress Controller.

helm create home-k3s

cd home-k3s

Open the newly created chart in an editor. Clean up the charts by removing the templated stuff, everything in the ./templates directory can be deleted and empty out the values.yaml file.

Add the MetalB Load Balancer to the chart Link to heading

Review the MetalB Installation with Helm instructions.

In Chart.yaml add the metallb dependency.

dependencies:
- name: metallb
  repository: https://metallb.github.io/metallb
  version: 0.11.0
  tags:
  - load_balancer
  import-values

metallb:
  configInline:
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      # Assigns addresses in range 192.168.1.X for use by the load balancer. The address must be within the range set on the router DHCP settings.
      - 192.168.1.50/27

Test Link to heading

This command should show the metallb pods running in kube-system namespace.

kubectl get pods -n kube-system -o wide

Add Nginx Ingress Controller Link to heading

Add the ingress-nginx dependency.

- name: ingress-nginx
  repository: https://kubernetes.github.io/ingress-nginx
  version: 4.0.13
  tags:
  - ingress-controller

Disable the default backend. No backend service called default-backend will be created.

ingress-nginx:
  defaultBackend:
    enabled: false

Test Link to heading

Nginx Ingress controller should be visible in the running pods

kubectl get pods -n kube-system -o wide

Under services there should be the LoadBalancer service for the ingress controller.

kubectl get services  -n kube-system

Accessing the External IP of the ingress at port 80 should give you a 404 error.

curl -v http://<ip_address>

Add CertManager Link to heading

CertManager requires CRDs which need to be install before installing it with Helm. Install the CRDs, then add cert-manager as a dependency in Chart.yaml

- name: cert-manager
  repository: https://charts.jetstack.io
  version: 1.7.0
  tags:
  - cert-manager

Install the CRDs automatically when installing the chart.

cert-manager:
  installCRDs: true

Once installation is complete check that the objects are created. You should see the cert-manager, cert-manager-cainjector and cert-manager-webhook pods.

kubectl get pods -n kube-system

What’s Next Link to heading

Installing and setting up PiHole and Jellyfin services in the cluster.

References Link to heading