sh0rez

HomeLab

Starting with just a Raspberry Pi connected to a 4-Port 100M-Switch, it quickly evolved to a full-fledged HomeLab with a 24-Core Xeon host capable of running lots of virtual machines, a VLAN-based, business-grade, firewalled, software-defined network, distributed storage, declarative configuration and infrastructure as code.

Networking

Mainly for security reasons, I wanted to have network separation, for example to prevent chatty IOT-devices from sending internal data to their manufacturer. After doing some research, I chose VLAN as the technology to achieve this. But VLAN does not happen by it's own, so I needed compatible devices. After trying some older Netgear equipment I got on Ebay, it got clear that this was not the way to go, as these devices had to be configured manually, by hand and using a probably insecure web interface. So I looked for alternatives and found Unifi from UBNT. The concept looked promising: A central controller software manages the whole network. The individual devices are adopted by it and the configuration pushed. Everything is encrypted using TLS.
But switching is not all for successfully implementing VLAN, because devices need to be able to talk to each other across VLAN boundaries, in a controlled way. So I needed a firewall. Generally being in love with open-source software, I chose pfSense for this task, and it works perfectly to this day. Documentation is close to perfect and the system runs reliable and just works. Sadly, there is neither an API nor any kind of automation.

Virtualization

To run software, machines are needed. As I wanted to get as close to the cloud as possible, I needed virtualization as a basis. Originally considering vSphere ESXI, I quickly abandoned this idea due to the insane price point for a single machine. After all it was still a hobby. So I looked at the open-source ecosystem (again) and gave Proxmox a shot. And I was rewarded with a Linux KVM based, simple yet solid virtualization platform. While it is not perfect in all aspects (weird cloud-init support, no official Terraform provider, inconsistent documentation). But it does its job and is free, so I am happy!
Speaking of cloud-alike, I always wanted to use infrastructure as code. While there is no official Terraform provider for Proxmox, an unofficial one exists. This one gets the job done, although it is a disaster of untested Go code and no documentation. If I find some time, I will probably write a new one.
Terraform creates my machines, but effectively stops here. So I needed somebody to take over from here: Enter Ansible. It allows me to describe all my machines as clean .yaml-files in git repositories. Even if my host burns down, I would be able to get up and running in a couple of hours.

Storage

Proxmox et al. solved machines itself, but storage needs to be taken care of as well. To keep the door to a future HA setup open, I wanted distributed storage. While there are very fancy solutions out there (Ceph, etc.), I needed to get started so I went with good old NFS. FreeNAS is an awesome FreeBSD NAS appliance that provides a lot of features, including amazing ZFS and even S3 support. It does not support automation but works well as a simple storage backbone.

Kubernetes

As I had a network, virtual machines and storage, I was finally able to focus on running something on top of it. It's 2019 and workloads of any kind belong into Docker containers. And while I could certainly manage my >10 actual applications by hand or using smaller tools like docker-compose or even ansible, I decided to overshoot the mark a little bit and ran Kubernetes with 3 masters and 5 workers, all on my single machine. While this is certainly not needed, it was an awesome experience and I definitely learned a lot. I have written about it here.

Backups

While IAC reduces the amount of resources to backup, applications still have state and one does not want to lose it. To run backups, I chose restic. But restic is rather clumsy in terms of usability, so I wrote some software around it (not on GitHub yet), to simplify operations (scheduling, monitoring using Prometheus, secrets using vault).

Secrets

To keep track of secrets (these little strings you have to take care of), I chose HashiCorp Vault. This tool is worth a whole page itself but in short, I was able to keep all secrets in Vault, let computers authenticate against it using AppRole, humans use GitHub OAuth2.

Awesome things discovered along the way

routing, vlan, firewall, dns, vpn, LoadBalancing

pfSense

learn more
Software Defined Network

UBNT Unifi

learn more
Virtualization

Proxmox

learn more
provisioning

Ansible

learn more
Storage

FreeNAS

learn more
declarative infrastructure

Terraform

learn more
container runtime

Docker

learn more