Ian Harris

Raspberry Pi Cluster, Part 8 - Automation

2021-11-07T00:00:00+00:00

Raspberry Pi Cluster, Part 1 - Hardware
Raspberry Pi Cluster, Part 2 - Assembly
Raspberry Pi Cluster, Part 3 - Operating Systems
Raspberry Pi Cluster, Part 4 - Jekyll and Docker
Raspberry Pi Cluster, Part 5 - Kubernetes
Raspberry Pi Cluster, Part 6 - Dynamic DNS
Raspberry Pi Cluster, Part 7 - SSL
Raspberry Pi Cluster, Part 8 - Automation

Wow, this has turned into a much bigger series that I initially planned! Well done if you have made it this far! I promise this is the last once (for now).

At this point, the cluster is up and running and it is serving secure website pages over the Internet. However, there is a small issue with our setup whereby any changes that we wish to publish on the site requires us to got through the process of recreating the docker image and then logging into Kubernetes to redeploy it. It’s a bit of a faff.

To do this after pushing up a change to DockerHub, we can restart the Kubernetes deployment to pull down the latest image with:

kubectl rollout restart deploy blog-website

But that will get tiresome quickly. This is where Keel comes in. It’s a very handy little app that will monitor our DockerHub repo and then update our Kubernetes deployment as and when we push up a new version of our website image.

Installing Helm

To install Keel, we need to first install Helm. It’s the package manager for Kubernetes and uses charts to deployment predefined manifests on to Kubernetes clusters. It’s easy to install, run each of these three commands in order:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

As per the K3S documentation, we need to set and Environment Variable to allow Helm to work properly, set this with:

export KUBECONFIG=~/.kube/config
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
chmod 600 ~/.kube/config

Setup Webhook Relay forwarding

helm upgrade –install webhookrelay-operator –namespace=auto-deploy webhookrelay/webhookrelay-operator \ –set credentials.key=$RELAY_KEY –set credentials.secret=$RELAY_SECRET

What’s next?

Host a private Docker image repository
Use solar/wind energy
Optimising the Kubernetes deployment. Maybe remove Traefik and install Nginx Ingress.
Use a CI/CD service like CircleCI
Replace the last 2 3b+ Pi with 2Gb Pi 4s

Raspberry Pi Cluster, Part 7 - SSL

2021-10-04T00:00:00+00:00

Here we are, the last push to complete this project. Let’s add a SSL certificate to encrypt all traffic to the cluster ensure that the site is as secure as possible. Once that is done, we can take a look at automating the deployment process.

If you have been following along and you have setup Cloudflare as per the previous post, you should see that your website is already protected with a SSL certificate. So why do we need to add another certificate on our server?

Good question. What is happening now is that all connections to the Cloudflare proxy are encrypted, but any connections from Cloudflare to our website are not and any data will be sent over the wire in plain text. Ideally, we want full encryption to our cluster, so we are going to install Let’s Encrypt on the cluster to solve this problem.

Installing Cert Manager

Let’s dive straight into it - the first task is to create a new namespace in Kubernetes so that we can keep all the SSL stuff in its own little bubble:

kubectl create namespace cert-manager

Installing Cert Manager is pretty easy, it’s a single command. Just make sure you replace the version number with the current available version. For me, this was v1.6.1:

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml

Once the install has finished, we next need to create a new ClusterIssuer resource on the Create a new file with sudo nano letsencrypt.yml and copy in the following:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: <email-address>
    privateKeySecretRef:
      name: blog-website-tls
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: traefik

Replace <email-address> with your email address and keep an note of the name that you assigned for privateKeySecretRef property.

Finally, tell Kubernetes to use our cert-manager manifest file by running:

kubectl apply -f letsencrypt.yml

We can check to make sure everything is working expected by running:

kubectl describe clusterissuer letsencrypt-prod

All being well, we should see that everything is running and the Issuer is returning a Ready status. You may need to give it a couple of minutes to sort itself out.

Updating the ingress

After we have install the means of issuing a Let’s Encrypt certificate on the cluster, we now need to update the Ingress that we configured previously to use TLS and to apply for a certificate. Open up the Ingress manifest again with sudo nano ingress.yml and update it so it looks like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blog-website-ingress
  annotations:
    kubernetes.io/ingress.class: traefik
    cert-manager.io/cluster-issuer: letsencrypt-prod
    acme.cert-manager.io/http01-edit-in-place: "true"
spec:
  tls:
    - secretName: blog-website-tls
      hosts:
        - <external-domain>
  rules:
  - host: <external-domain>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: blog-website-service
            port:
              number: 80

Make sure you double-check the names and domains like you did before. Also make sure your annotations match the names that you used when you created the ClusterIssuer, for me this is letsencrypt-prod. It’s very easy to get a name mis-match here and you can spend a long, long time trying to track this down. Again, I bet you can’t guess how I know this.

Last job, we need to reapply the manifest:

kubectl apply -f ingress.yml

It will take a few minutes, Let’s Encrypt with dial home with the cluster details and after a few minutes, you should be able to access the cluster directly on https:. If you go into the Cloudflare control panel, you should now be able to set up Full encryption to ensure that all of the data transmissions are secure.

Next, Part 8 - Automation

Raspberry Pi Cluster, Part 6 - Dynamic DNS

2021-09-29T00:00:00+00:00

Raspberry Pi Cluster, Part 1 - Hardware
Raspberry Pi Cluster, Part 2 - Assembly
Raspberry Pi Cluster, Part 3 - Operating Systems
Raspberry Pi Cluster, Part 4 - Jekyll and Docker
Raspberry Pi Cluster, Part 5 - Kubernetes
Raspberry Pi Cluster, Part 6 - Dynamic DNS
Raspberry Pi Cluster, Part 7 - SSL
Raspberry Pi Cluster, Part 8 - Automation

We are nearing the end of this epic journey, now we just need to point a domain to the cluster so that it accessible to the outside world and apply a SSL certificate to make sure that any connections are secure.

Simple right? Well, no it isn’t (it could have been, but where is the fun in that).

This isn’t the place to delve too deeply into the inner workings of DNS, quite frankly, neither of us has the time, but there are plenty of resources available. As the cluster is sitting inside my home network, it means that I need to point my domain name to the external IP address of my router.

Here is my first problem - my ISP is BT Infinity and as any customer of theirs will know, they do not provide static IP’s for their home customers. This is a fairly major problem when it comes to DNS, it means that my external IP will change often, maybe even a few times a day. That will result in me having to go and manually update my DNS records every time the IP changes - not ideal.

Fortunately there a pretty well established solution to the problem, it’s called Dynamic DNS. The concept is pretty simple, there is normally an agent sitting somewhere inside the network, it repeatedly checks the external IP address and automatically updates a Dynamic DNS provider.

I’ve been using No-IP for years. My router runs the agent and it works perfectly. However, one minor downside is that to use their free tier, I need to manually confirm the hostname every month, which isn’t that much of a pain, but we can do better…

Cloudflare

In my professional life, I am a web developer and I’ve used Cloudflare a few times for various clients and I’ve always been impressed by their service.

(I must point out here that I am in no way associated with them, I just like their service - this ain’t no ad!)

There are many, many parts to Cloudflare that sets it apart from other services, mainly their security and DDOS protection, but there is one feature I am particularly interested in and that is their DNS proxy.

As I’m running this on my home network, it means that I am exposing my IP to the world, while in reality isn’t too much of an issue, but I would sleep easier at night knowing that I’m not creating easy access to my home network. Quite simply, Cloudflare has an option called DNS Proxy for your DNS records where they replace my IP address with one of theirs, then forward any requests in the background. Nice.

That is enough talking, let’s set it up…

Updating the DNS records

First of all, you’ll need a Cloudflare account, head over there and create one if you haven’t already.

Second, watch this awesome tutorial by NetworkChuck over on YouTube. The general idea is that we create our own DDNS agent (like we discussed earlier) via a very handy script by someone called K0p1-Git (thanks for that!) that makes it easy to communicate with the Cloudflare API.

All we need do is update the script with our API credentials and then execute the script on a schedule via a cron job. Job done!

Except, I didn’t go through the pain of building an awesome Kubernetes cluster and not to use it, did I! Surely we can run this script via a Kubernetes deployment?

Migrating the script to Kubernetes

As I mentioned in part 5, the current stable release of Kubernetes at the time of writing this series is v1.21.4, which is handy because in v1.21 they introduced the Cronjob resource.

This should have made this migration to Kubernetes easy right? Well, not quite. It did make setting the cron task super easy, but trying to run this task on Kubernetes became one of those things where it became a matter of principle to find a way of getting it to work.

You’ll be please to hear, dear reader, that I persevered and in the end found an excellent solution to the problem. I also learnt a lot about Kubernetes in the process. I’ll guide you through the steps here to prevent you from spending hours trying to figure it out!

Step 1

Make sure you have watched the video above, completed the steps there and you have confirmed the script is working by testing on your server. Make sure you are in the same folder as your cloudflare.sh script.

Step 2

To run the cron task, we need a container to do that in with an OS, I found that the best base was Alpine Linux. There are a couple of reasons for this, the first is the size, it’s barely 4Mb in size, which is essential as we want to keep this efficient as possible. Using Ubuntu or another flavour of Linux would be way too bloated for what we want to do here. This is what Alpine was created for.

The Second reason is that we need to be able to install packages into the distro (more in this in a sec), which is something that Busybox can’t do.

The problem we have now is that Alpine doesn’t include Bash, so we need to modify our script to tell it which shell binary to use. Fortunately, that is very easy. Open up the script again with:

sudo nano cloudflare.sh

Change the very first line of the file from #!/bin/bash to #!/bin/sh, save and exit.

Step 3

The Cronjob deployment we are creating needs a way to be able to read our script. As it’s just a simple file, we can utilise a config map here. This is a really nice way to give Kubernetes access to the script:

kubectl create configmap cloudflare --from-file=cloudflare.sh

Step 4

Create a new file with sudo nano cloudflare-cronjob.yml and copy in the following:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: cloudflare
spec:
  schedule: "*/5 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cloudflare
            image: carronmedia/alpine-curl-arm64
            command: ["/scripts/cloudflare.sh"]
            volumeMounts:
            - name: cloudflare
              mountPath: /scripts
          volumes:
          - name: cloudflare
            configMap:
              name: cloudflare
              defaultMode: 0744
          restartPolicy: OnFailure

You can edit this to your needs, but here are a couple of notes. This will work on older versions of Kubernetes, but you’ll need to change the first line to apiVersion: batch/v1beta1.

The schedule is currently set to run the script every minute, this is standard cron notation, so you can update this as you wish.

Finally, the last point is the image. This is where a lot of my headaches came from.

At first, I set this to use the base alpine image, but the script was failing with the dreaded exec user process caused “exec format error” error. I mentioned this in part 4, which I forgot about. So after a lot of reading and messing about, I remembered that this is because arm64 CPU architecture of the Raspberry Pi isn’t compatible with the standard alpine image. I needed to use the arm64/alpine image instead.

Next it threw an error about cURL not being installed. Alpine by default doesn’t include cURL, so I created a pair of simple Docker images to run the cURL install for us. These are available on DockerHub: carronmedia/alpine-curl for the standard version and carronmedia/alpine-curl-arm64 for the arm64 version for Raspberry Pis.

Step 5

Finally, let’s deploy our Cronjob with:

kubectl create -f cloudflare-cronjob.yml

This creates all the necessary pods on Kubernetes and this will run our script to update our DNS settings on Cloudflare. We can confirm that everything is working by running:

kubectl get cronjob cloudflare

All being well, you should see your Cronjob deployment in an active state.

Well done! That was a long post, but hopefully you found it useful.

Next, Part 7 - SSL

Raspberry Pi Cluster, Part 5 - Kubernetes

2021-09-25T00:00:00+00:00

Raspberry Pi Cluster, Part 1 - Hardware
Raspberry Pi Cluster, Part 2 - Assembly
Raspberry Pi Cluster, Part 3 - Operating Systems
Raspberry Pi Cluster, Part 4 - Jekyll and Docker
Raspberry Pi Cluster, Part 5 - Kubernetes
Raspberry Pi Cluster, Part 6 - Dynamic DNS
Raspberry Pi Cluster, Part 7 - SSL
Raspberry Pi Cluster, Part 8 - Automation

It’s time to join all of these dots together and really get into the depths of it. We have a functional Raspberry Pi Cluster, we have a functional Docker image hosting a website, so let’s install the container orchestration system known as Kubernetes.

I mentioned Kubernetes briefly in the previous post and what it is, but instead of going into the details here, I’ll leave that to The Linux Foundation to do a much better job.

As Kubernetes is an enterprise grade system, it has enterprise grade system requirements and under normal circumstances, we wouldn’t even consider running such a thing on a Raspberry Pi. Thankfully, the nice people at Rancher built a super lightweight stripped back version of Kubernetes called K3S that plays very nicely on Raspberry Pi’s.

Installing K3S

You’d think this part would be complicated, it couldn’t really be much easier. On your master node, run the following:

curl -sfL https://get.k3s.io | sh -s - --write-kubeconfig-mode 644

Note the --write-kubeconfig-mode 644 argument here. This is important as it installs Kubernetes with the correct permissions. Without it, when the cluster reboots, it won’t be able to access the file system. You won’t need many guesses to know how I found this out.

Give a few minutes and that is it. Kubernetes v1.21 (at the time of writing) is installed and in theory, the cluster is alive! We can check this by running:

kubectl get pods -A

All good! The next task is to install Kubernetes of each of the worker nodes. However, before we can do this, we need to get the cluster’s unique access token that we use to connect the worker nodes to the master. Run this and make a note of the result:

sudo cat /var/lib/rancher/k3s/server/token

Almost there! On each of the worker nodes, run the following:

curl -sfL https://get.k3s.io | K3S_URL=https://<server>:6443 K3S_TOKEN=<token> sh -

Where <server> is the IP address of your master node and <token> is the token we got from the master node in the previous step.

And once complete, the Raspberry Pi based Kubernetes cluster is complete!

Major problem No. 2

Full disclosure, I didn’t realise I had the problem that I’m just about to describe until a bit further down the line, but it makes sense to talk about it here quickly.

Whilst Kubernetes installs without any issues on a Raspberry Pi 3b+, after around 24 hours of use, the master node runs out of steam and locks itself up. After quite a bit research and a fair amount of swearing, I discovered that any versions of K3S later that v1.17 don’t play nicely on the 3b+ variants of the Pi.

The solution was to upgrade the master node to a Raspberry Pi 4 4Gb. I probably should have just bitten the bullet on done this in the first place. Never mind. As the Pi 4 runs host, especially running Kubernetes, I added a fan shim to keep it cool.

Here was additional shopping list:

Raspberry Pi 4 4Gb (Pimoroni)
Fan SHIM for Raspberry Pi (Pimoroni)

This added a further £63.90 to the budget.

Configuring our cluster

We now need to tell Kubernetes what to do and we do this via a series of manifest files of different types. Again, this post is long enough already, so I’m not going into the details here, but this will give you an idea of how I got it all working.

Fortunately, there isn’t much in the way of configuration needed, the K3S install pretty much does it all. However, as the Docker image we created is in a private repo on DockerHub, Kubernetes needs the access credentials to pull the image down. Run the following:

kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-password> --docker-email=<your-email>

Replacing <your-registry-server>, <your-name>, <your-password> and <your-email> with your credentials. Kubernetes can now pull down our private image from DockerHub.

WARNING: Be aware that there is a potential security issue here where you credentials are visible in your bash history. If this is going to be a problem, there are options available to help. Give it a Google. For me this is not a factor - if someone can view my bash history, I have bigger problems to deal with!

Setting up a Deployment

A deployment manifest in basic terms tells the Kubernetes where to find the image that will be used as the base of the application that we want to run on the cluster. Let’s create the manifest file with sudo nano deployment.yml and copy in the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: blog-website
spec:
  replicas: 3
  selector:
    matchLabels:
      app: blog-website
  template:
    metadata:
      labels:
        app: blog-website
    spec:
      containers:
      - env:
        image: <username>/<containername>:latest
        imagePullPolicy: Always
        name: blog-website
        ports:
        - containerPort: 80
      imagePullSecrets:
        - name: regcred

Replace blog-website with what ever name you want. Just remember what you change it to as we’ll need to link to this shortly. There are some important lines to consider, the first is replicas. I chose 3 as I have 3 worker nodes in my cluster. You can set this how you like, you can have more that 1 replica per node too. I could have set this to 20, but that is probably a little excessive.

Secondly, replace <username> and <containername> with the image details that you created in the last post.

After you have saved and closed the manifest file, we need to pass it to Kubernetes. Run:

kubectl apply -f deployment.yml

After a few minutes, if you run kubectl get pods, you should see that 3 pods come online all running instances of our Docker container. Nice.

WARNING: This is where you are likely to see issues if you have any problems with your configuration or servers. One to look out for is an **EXEC FORMAT** error. This is most likely caused by you trying to run a container that was built to be run on a different processor architecture. Remember Raspberry Pi's have ARM processors. Images built for x86 based systems will not work on a Pi.

Setting up a Service

The next manifest we want to create is a service, this will give network access to our deployment. There are a few different types of service, however, we don’t need to specify one and this will use the default ClusterIP type. As before, let’s create a new file with sudo nano service.yml and copy in the following:

apiVersion: v1
kind: Service
metadata:
  name: blog-website-service
spec:
  selector:
    app: blog-website
  ports:
  - protocol: TCP
    port: 80

You can change the names to what you wish, just make sure they match between your different manifest files. Apply this service manifest with:

kubectl apply -f service.yml

Again, give it a few minutes and check your service is running with kubectl get service.

Setting up an Ingress

Like everything Kubernetes, there appears to be at least 100 ways of doing anything, so it’s no surprise that setting up Ingress can be confusing. An Ingress essentially routes external traffic to our service (which in turn exposes our deployment). Most people appear to favour the Nginx Ingress in all of the tutorials that I read researching this, but K3S ships with an Ingress controller called Traefik, so we’ll go with that.

As before, create a new file with sudo nano ingress.yml and copy in the following:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blog-website-ingress
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: <external-domain>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: blog-website-service
            port:
              number: 80

As previously, make sure your names match what you put in the other manifest and replace <external-domain> with your own domain name that you want to use for your website.

Apply this service manifest with:

kubectl apply -f ingress.yml

Congratulations! Once the ingress comes online, you should have a cluster based website that is accessible on the network. There is one last task to make our master piece available to the outside world…

TOP TIP: You can combine all of these manifest into a single file, all you need to do is separate each declaration with ---. I separated it out here to make it easier to explain.

Setup port forwarding on network router

In terms of instructions, you are on your own I’m afraid. Every router is different, but the process is essentially the same. In your network router configuration, find the port forwarding section and forward port TCP 80 (might as well do 443 too if you enable SSL like I do in a future post) to the internal IP of your master node.

This was a pretty epic post, well done if you have made it this far. Now we have the website up and running, let’s help people find it by pointing a domain at it.

Next, Part 6 - Dynamic DNS

Raspberry Pi Cluster, Part 4 - Jekyll and Docker

2021-09-22T00:00:00+00:00

Raspberry Pi Cluster, Part 1 - Hardware
Raspberry Pi Cluster, Part 2 - Assembly
Raspberry Pi Cluster, Part 3 - Operating Systems
Raspberry Pi Cluster, Part 4- Jekyll and Docker
Raspberry Pi Cluster, Part 5 - Kubernetes
Raspberry Pi Cluster, Part 6 - Dynamic DNS
Raspberry Pi Cluster, Part 7 - SSL
Raspberry Pi Cluster, Part 8 - Automation

Now I have my Pi’s all set up and ready to go, I’m ready to lose myself down the rabbit hole that is Kubernetes. However, before I can go there, I need to think about what I’m actually going to use the cluster for (apart from a Minecraft server, obvs).

Well, as I mentioned in the first part of this series, my main plan is to host this very website on the cluster, so let’s set that up now.

Even though Raspberry Pi’s are amazing, they do have limitations and I want to squeeze every bit of performance and longevity out of my cluster. This essentially means that the site needs to be as lightweight as possible. While I could just install WordPress or Drupal on the cluster, this means that I would have to install PHP and MySQL, which will add quite a big strain to the Pi’s and their relatively fragile SD cards.

Jekyll

The solution is to keep everything simple, all we want to serve from the site is plain on HTML, without an expensive (in terms or performance) CMS using up all the memory. This is where Jekyll comes in, it’s a static site generator that spits out plain HTML. Ideal.

Jekyll is a ruby gem that has been around for ages and works by compiling markdown templates into a static HTML site. It’s pretty simple to setup, if you are following along, checkout the quick start guide. If everything goes well, you should end up with a HTML site in a folder called _site. We’ll need that next.

Packaging up the website

Kubernetes is an open source container orchestration tool, which is basically means it manages the deployment of virtual machines on clusters of computers. Containers can contain pretty much anything and there many different flavours of containers, but I’m going to use Docker and I’m going to run an Nginx instance within the container to host my shiny new Jekyll website.

It’s actually quite a straight-forward process, in the root of the Jekyll site (i.e the parent folder of _site), create a new file called Dockerfile and add the following:

FROM arm64v8/nginx

COPY _site /usr/share/nginx/html

It’s probably about as simple as a Dockerfile can get. To explain, the first line tells Docker to use nginx, but not just the standard image - as we are deploying this on 64 bit Raspberry Pi’s, we have to use a specific version that is compatible with the Pi’s ARM based processor architecture. This is what the arm64v8 part signifies.

The second line simply copies our Jekyll website to the default root folder of nginx.

Now we need to build our new Docker container. The best way to do this is to install the Docker Desktop app, this will give you access to the CLI. At this point, create an account too, we’ll need this in a sec. Once installed, build the container:

docker build -t <username>/<containername> .

Replace <username> and <containername> with the relevant info. This will take a few seconds, but once finished, we have successfully built our container image that hosts our Jekyll website.

Pushing up to the cloud

I had been toying with the idea of creating my own private docker container repository on the cluster itself. It would fit in with the project ethos of making everything as complicated as possible. But I want to be able to update the site from anywhere in the world and not just on my local network. I could create a VPN, but even by my standards, that is a bit overkill.

A much better solution is to simply push my container image up to DockerHub, which is a cloud based repository specifically for storing Docker images (as the name would suggest).

If you created an account earlier when installing Docker Desktop, it’s quite easy. Make sure you are logged into your account in Docker Desktop, then run:

docker push <username>/<containername>

Et volia, the image is pushed up to the cloud. There is one extra step I took here: I logged into DockerHub and found my repo, I then marked it as private. Each DockerHub account gets one free private repo in the free tier. As this is a cloud based system, it means that any public repos can be found and pulled down by anyone. So technically, if someone found our image, the could spin up their own version of our site pretty quick. This probably isn’t ideal.

If you are following along, excellent work. Now we’re going to setup Kubernetes. Which is proper geeky. You’ll love it.

Next, Part 5 - Installing and configuring Kubernetes

Raspberry Pi Cluster, Part 3 - Operating Systems

2021-09-20T00:00:00+00:00

Raspberry Pi Cluster, Part 1 - Hardware
Raspberry Pi Cluster, Part 2 - Assembly
Raspberry Pi Cluster, Part 3 - Operating Systems
Raspberry Pi Cluster, Part 4 - Jekyll and Docker
Raspberry Pi Cluster, Part 5 - Kubernetes
Raspberry Pi Cluster, Part 6 - Dynamic DNS
Raspberry Pi Cluster, Part 7 - SSL
Raspberry Pi Cluster, Part 8 - Automation

Now the hardware is sorted, it’s time to turn my attention to the software side of the project. More accurately, the decision I need to make is which operating system I’m going to run on the Pi’s.

There is actually a wider choice here that I originally anticipated, but I’ll keep this bit brief and explain my choices. I worked my way through a couple of options before settling on my operating system of choice.

The initial experiments

To keep things small and simple, one requirement that I scoped out straight-away was that there is no need for any sort of GUI. I want to utilise as much of the onboard power of the Pi’s to run the cluster software, there isn’t any need to waste it on a desktop, etc.

I originally installed the 32 bit version of the official Raspberry Pi OS, but I really wanted to utilise the full 64 bit goodness of the Arm64 processor that the Raspberry Pi’s have, so I rebuilt all of the Pi’s with Ubuntu 21.04 which supports 64 bit. However, and I’m aware that this sounds a little weird, but it just didn’t feel right not running the dedicated Pi OS.

So third time lucky, I rebuilt all of the SD cards again, this time using the 64 bit edition of Raspberry Pi OS.

Even though Kubernetes (or more accurately, the Rancher K3S flavour of Kubernetes) does work on 32 bit OS’s, it seems to be much happier on the 64 bit OS’s and most of the docker images that I originally worked with all were built for 64 bit architectures, so this seems like the best way to go.

Installing Raspbien OS 64 bit

At the time of writing, the 64bit version of the Raspberry Pi OS is still not officially published by the Raspberry Pi Foundation, but it is available to download here: https://downloads.raspberrypi.org/raspios_arm64/images/. Grab the latest .zip and extract the files.

While you’re there, if you haven’t already, download the Raspberry Pi Imager application too as this makes pushing the OS to the SD card a breeze.

The very handy Raspberry Pi Imager

Click on the Choose OS button, scroll to the bottom of the list, select the Use custom option and load up the Raspberry Pi OS 64 image we downloaded earlier. Similarly, click the Choose Storage button and select the SD card that you want to write to.

DON’T CLICK THE WRITE BUTTON YET!

As with any OS install, we need to do some configuration. Fortunately, there is a hidden menu in the app that gives us an easy way to configure the most important settings. On a Mac, if you press Ctrl + Shift + x, you’ll get a new menu popup (I’m not sure what this would be on Windows, I’m sure a quick Google will sort you out).

Now, check Set hostname and give the Pi a unique hostname. I called my first Pi einstein-master, then the next three einstein-worker-01, einstein-worker-02 and einstein-worker-03.

Next, check Enable SSH, I choose the Use password authentication and then set a good password. I’m actually going to setup the public key authentication once the Pi is online, but this means that I can set a good sudo password now. The rest you can leave, unless of course you wish to add some more config now.

Click Save, now you can click that Write button to image the SD card.

Just for your info, this setup can all be done on the Pi itself and when I built the 32 bit versions, I did all this configuration manually, but it’s much quicker to do it this way.

Setting up the Pi’s

We’re getting there now! Stick the SD card into the Pi and power it up. To be honest, I’m not sure if it is necessary to have a static IP for the Pi, but at the very least, it makes it a lot easier to manage. So set a static IP by which ever means floats your boat. I just log into my router and find the IP address and then set a DHCP reservation so that the Pi gets the same IP all the time.

Now we know the IP, we can transfer over our public key to secure up the Pi a little bit better. If you don’t have a key setup, follow these instructions.

Copy over the key with (insert the IP address of your Pi):

ssh-copy-id pi@<IP-ADDRESS>

We can now login to the Pi:

ssh pi@<IP-ADDRESS>

Kubernetes requires the control groups kernel feature to be enabled (more info here), run:

sudo nano /boot/cmdline.txt

And add the following to the end of the line, making sure that everything is on the same line:

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Save and close. Next, we can reduce the amount of memory that is allocated to the GPU as we don’t require any graphical features. Run:

sudo nano /boot/config.txt

And add the following to the end of the file:

gpu_mem=16

Almost there! Next we need to disable the SWAP file, again this is for Kubernetes, run each of the following commands:

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall
sudo apt purge dphys-swapfile

I mentioned earlier that we are going to disabled password authentication to increase the security, run:

sudo nano /etc/ssh/sshd_config

And update the following option (you may need to removed the # at the beginning of the line):

PasswordAuthentication no

Finally, let’s update everything (the -y automatically says yes to every question):

sudo apt update && sudo apt dist-upgrade -y

This is probably take a little while, but we are done configuring the Pi. We just need to run reboot the Pi with sudo reboot to make sure all of the new configuration options take effect.

Rinse and repeat for all of your Pi’s, they all need the same setup.

Next, Part 4 - Setting up Jekyll and Docker

Raspberry Pi Cluster, Part 2 - Assembly

2021-09-18T00:00:00+00:00

Raspberry Pi Cluster, Part 1 - Hardware
Raspberry Pi Cluster, Part 2 - Assembly
Raspberry Pi Cluster, Part 3 - Operating Systems
Raspberry Pi Cluster, Part 4 - Jekyll and Docker
Raspberry Pi Cluster, Part 5 - Kubernetes
Raspberry Pi Cluster, Part 6 - Dynamic DNS
Raspberry Pi Cluster, Part 7 - SSL
Raspberry Pi Cluster, Part 8 - Automation

Here was the bit I was looking forward too, actually sticking all the bits together to build the Raspberry Pi cluster. It’s a pretty straight-forward process, it’s just a case of following the instructions - a bit like Lego.

Putting together the layers of the cluster

Start by bolting the Pi’s onto the acrylic layers of the case. At this point, stick the heatsinks onto the Pi’s chips as it will be obviously harder to do this later down the line.

Using the case kit stand offs, attached the layers together to build the Pi stack (there must be a gag in there somewhere).

Finally attach the mini CAT5 cables to the switch and the USB cables to the powered hub. That’s, err, pretty much it.

The assembled Raspberry Pi cluster. Pretty isn't it.

Major problem No. 1

Every was going great, until I turned it all on. The timeline of the posts goes a little out of order here, as I didn’t find this problem out until I had created the operating system images, but it makes sense to talk about this here in this post.

The problem was that only two of the Pi’s turned on, the other two kind of flickered on and off, but mainly stayed off. Then one of the good Pi’s turned itself off and one of the dead ones sprang into life. And so on.

To be completely honest with you, I had a feeling that the powered USB Hub that I dug out of junk cupboard might not cut the mustard. Turns out that was the case, it simply couldn’t supply enough power for all of the Pi’s to be on at the same time.

The power solution

Anker PowerPort 5. Much more capable.

What I needed is a power supply that is designed to delivery at least 2A on each port simultaneously and a quick search turned up the Anker PowerPort. This is much more suitable to powering the Pi’s. Now all the Pi’s can run happily, even when under a high load. Problem solved.

As the new power supply has 5 ports, that left a spare port and that got me thinking. I wondered if I could power the Netgear switch via USB as well? Another quick search later and it turns out you can!

This tiny cable made my OCD much happier.

Now I only need one power cable for the whole cluster and all the ports on both the PSU and the switch are filled. The whole thing looks better and that makes me very happy.

Here was additional shopping list:

Anker USB Charger PowerPort 5 (Amazon)
USB Type A Male to DC Plug Barrel Connector (Amazon)

This added another £30 to the hardware budget.

SPOILER: This STILL isn't the final setup. There are more updates needed, but I'll cover that in a future post.

A rather bad photo of the cluster on my desk, taken before I realised I could power the switch from the Anker PowerPort as well.

Now it’s time to work out what I’m actually going to do with this thing.

Next, Part 3 - Installing the operating systems

Raspberry Pi Cluster, Part 1 - Hardware

2021-09-15T00:00:00+00:00

Raspberry Pi Cluster, Part 1 - Hardware
Raspberry Pi Cluster, Part 2 - Assembly
Raspberry Pi Cluster, Part 3 - Operating Systems
Raspberry Pi Cluster, Part 4 - Jekyll and Docker
Raspberry Pi Cluster, Part 5 - Kubernetes
Raspberry Pi Cluster, Part 6 - Dynamic DNS
Raspberry Pi Cluster, Part 7 - SSL
Raspberry Pi Cluster, Part 8 - Automation

Introduction

Ever since I first saw a Raspberry Pi back in 2012, I’ve been a little bit obsessed with them. I don’t know if it was simply because of it’s size, it’s super low price or the endless possibilities that it gives an enquiring mind, but I remember how my mind was blown by this credit card size computer that cost around £20 (if my memory serves me correct).

My original Raspberry Pi 1B

Fast forward a few years. I started to read more and more about clustering Raspberry Pi’s and how people have built some amazing projects on their clusters, so I decided that I definitely, 100% needed one in my life. In this series of blog posts, I’m going to detail the journey and the approach that I took to get the cluster online.

If this project goes anything like most of my previous projects, it will most likely be totally over-the-top and unnecessarily complicated. But that’s fine with me, it’s all about the learning and fixing issues as I find them.

This post is a bit meta - it has been written before I have completed the project. The plan is to host this site on the cluster itself, so if you are reading this, it means I must have worked it all out and got the site live.

Well done future me!

What this series is and isn’t

The series is intended to document what I did to get this cluster up and running, it’s not designed to be a step-by-step tutorial. There are plenty of other tutorials like this out there on the Internet, so have a look around and you’ll find a few such tutorials.

However, it’s worth saying that it is a still a pretty niche subject and the tutorials do vary in quality. Also, bear in mind that the hardware and software is updated regularly, so what worked a year ago may not (actually, that is more like probably won’t) work today. It should give you a good start through.

The hardware list

SPOILER: This isn't the final setup. If you are following along, read the entire series first as there are many issues that I had to resolve before I got it all working.

The main choice I had to make at the beginning was which flavour of Raspberry Pi I was going to use. At the time of writing, the Pi 4 has been around for a few years and is pretty much the recommended base for a Pi Cluster (Alex Ellis recommends giving away the Pi 3b+’s as they are not viable anymore), but a few of the tutorials out there do use the older Pi 3b+ boards.

One of the main objectives of the project was to keep the costs down as much as possible. As I already owned a couple of Pi 3b+ boards, I decided to simply purchase a couple more of these to build a 4 node Pi 3b+ cluster.

So here was my initial shopping list:

4 x Raspberry Pi 3b+ (2 I already owned, 2 I bought second-hand on eBay)
Netgear GS305 Switch (Amazon)
4 layer Raspberry Pi cluster case (Amazon)
5 x 0.25m CAT5 cables (Amazon)
4 x 0.25m microUSB cables (Amazon)
4 x 8Gb mini SD cards (already owned)
Belkin powered USB hub (already owned)
4 x Pi 3b+ heatsink kits (Pimoroni)

All in, I spent around £110 on the extra bits, on top of the parts I already owned.

The individual hardware components

Now I have all the components, it’s time to stick them all together to build myself an awesome little super computer.

Next, Part 2 - Assembling the cluster

Introducing My Blog

2021-08-21T00:00:00+00:00

Welcome to my blog!

This is a place where I am sharing my notes and progress on a number of different projects that I have on the go. In all honesty, it exists primarily so I remember what I did and I can refer back to when needed. But if you are here and this is useful to you, then that is great!

Why did you bother?

Good question, thanks for asking. I like a challenge and I like learning even more. There are a (literal) million ways to get a simple blog online, but I wanted to develop a place where I can learn new skills and have fun learning at the same time.

I’m a web developer who likes to get involved in the tech side of things too. You can expect to see many web based projects, but also a lot of hardware based ones too. I’ve started dabbling in electronics, so expect more posts about that. I’m also currently building various garden buildings and features, so that will probably feature on here too. Basically, anything that I’m working on that may prove useful to someone else, I’ll probably post on here.

I’ve used the Internet to pretty much self-teach myself everything I know. It will be nice to give as much of that knowledge back to future generations of builders and makers.

What is coming next?

The first series of posts will be a detailed rundown of how this site was built, where it is hosted (SPOILER: it’s self hosted on a rather awesome Raspberry Pi cluster) and the challenges I faced in getting it up and running.

The chances are that when you are reading this, you are being served this very post from a shelf in my office, in which I am probably working away building a website for a client (or working on one of the many ongoing projects that I seem to be amassing).

I think that is pretty awesome.

Take a look at the about page for more information about the site and the technology used to bring this article to the world.

Thanks for reading, please feel free to get in touch!