kodbasen
Where the compass is spinning around software
I’ve been running my Kubernetes-On-ARM cluster a couple of weeks now and I’ll try to summarize my experiences. If you followed my earlier posts, then you know that I’m running Gogs and Drone on a Kubernetes cluster with GlusterFS for storing. I’m using Kubernetes-On-ARM on HypriotOS.
In this post I will try to summarize my experiences so far.
One of the reasons that I wanted to run Drone was it’s use of Docker, and to
build and push docker images to a registry. Drone has a plugin that does all
that and uses dind (Docker in Docker).
The examples and documentation is rather straightforward, nothing complicated. You need to install and setup the client, and setup your secrets so that you don’t store your username and password in your build file.
There was only one problem, it didn’t work …
I could not get the plugin to work. After turning on DEBUG:
# .drone.yml
...
publish:
docker:
environment:
- DOCKER_LAUNCH_DEBUG=true
...
I could see this error message:
It gave me a real headache and finally, after some googling, I found these issues:
The kubelet is causing the /sys/fs/docker file-system to becoming read-only,
this will hopefully be resolved in Kubernetes 1.2. Meantime, the solution is,
to remove one of the nodes from the Kubernetes cluster and use it as a remote
worker-node in Drone.
There are issues, when running the kubelet in a container, and is work in
progress, to resolve those issues.
I’ve been wanting to try out Go CD for some time now, and try to set up a pipeline on Kubernetes. If you take a look at the system requirements for Go CD, you will see that this is not suitable for a Raspberry Pi. The Go Server is a giant monolith, but what the heck, sometimes you need to check out the limit. I created a Docker image for ARM and tried to start up a pod.
I got the GO CD Server started, but it took an hour. Most of the cpu-time during
start was spent on glusterfs. The GO CD Server uses a lot of I/O during
start and I measured the read/write speed on the glusterfs to around 1,5-2 MB/s.
One should not expect any great performance when running glusterfs on
Raspberry Pi. There are some real bottlenecks on network and USB.
The Hypriot team has written a post about Test, build and package Docker for ARM the official way.
That was one of reasons why I wanted to upgrade Docker to v1.10.2.
But has shown to have some issues. The kubectl exec command doesn’t work
anymore and is reported here:
https://github.com/kubernetes/kubernetes/issues/20884
and is a duplicate of:
Running Kubernetes-On-ARM together with new releases of Docker is nothing for the faint hearted. There are some issues that hopefully will be resolved in a near future. I’m still very pleased with the cluster and it’s a great environment for building microservices and validating your system architecture. For instance, it’s not possible to build a big monolith. If your system runs and scales on Kubernetes-On-ARM (built on Raspberry Pi) then you can be sure of that it runs and scales on any platform.
3 Mar 2016 #Development #Docker #Kubernetes #Raspberry PI #GlusterFS #Gogs #Drone #ARM