A Simple Talos Kubernetes Cluster on Proxmox
In this post, I’ll walk through creating a small Talos Linux Kubernetes cluster on Proxmox VE, with one control plane and two workers. No Cilium, no Cluster API, just three VMs, talosctl and the defaults Talos ships with.
Talos Linux is an operating system built for one job, running Kubernetes. There is no SSH, no shell and no package
manager. Everything is managed through an API with talosctl, and the whole node is described by a single machine
configuration file. That sounds restrictive at first, but it means a node is either configured exactly like its
config file says, or it is not running.
It also makes Talos a very good fit, not only for the enterprise but for homelabs as well. Creating a cluster is a handful of commands, and tearing it down and starting over costs you a few minutes.
The cluster in this post looks like this.
| Node | Role | IP | MAC | Size |
|---|---|---|---|---|
| talos-cp-1 | Control plane | 10.0.0.21 | BC:24:11:00:00:21 |
2 vCPU / 2 GB / 10 GB |
| talos-w-1 | Worker | 10.0.0.22 | BC:24:11:00:00:22 |
2 vCPU / 2 GB / 10 GB |
| talos-w-2 | Worker | 10.0.0.23 | BC:24:11:00:00:23 |
2 vCPU / 2 GB / 10 GB |
Each VM gets a static MAC address, so the DHCP server can hand out a static lease for it. The nodes
need fixed IPs because the Kubernetes API endpoint and the talosctl configuration both point at them. Talos can
be configured with static IPs in its machine config, but then the addresses are spread across config files. With
static leases, the DHCP server stays the single source of truth for addressing, and you can see every node and its IP in
one place, next to the rest of your network.
Replace the IPs with something from your own network. The versions used are Talos v1.14.5 and whatever Kubernetes version that Talos release ships as default.
Prerequisites¶
You need talosctl and kubectl on your workstation. The talosctl version should match the Talos version you
are installing.
On the network side, create a static lease on your DHCP server for each MAC address in the table above.
Building the Talos Image¶
Talos images are built by the Image Factory. The factory lets you add system extensions to the image, and on Proxmox the one you want is the QEMU guest agent. Without it, Proxmox can't see the IP of the VM and a shutdown from the Proxmox UI will not be passed on to Talos gracefully.
An image is identified by a schematic ID, which is a hash of the customization. Since it is a hash, the same customization always gives the same ID. You can get it through the web UI or the API, and both end up with the exact same ID.
Using the Web UI¶
Go to factory.talos.dev and click through the wizard.
- Hardware Type: Cloud Server.
- Talos Linux Version: 1.14.5.
- Cloud: Nocloud.
- Machine Architecture: amd64, with SecureBoot left off.
- System Extensions: search for and select
siderolabs/qemu-guest-agent. - Customization: leave the extra kernel arguments empty.
The last page shows the schematic ID along with download links for the ISO and the installer image to use when generating the configuration. Copy the schematic ID, you’ll need it twice.
Using the API¶
If you would rather script it, describe the customization in a file called schematic.yaml.
Post it to the factory to get the ID back.
Downloading the ISO¶
Download the ISO and upload it to the ISO storage in Proxmox, or download it directly on the Proxmox host.
SCHEMATIC=ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515
TALOS=v1.14.5
curl -Lo talos-$TALOS-nocloud-amd64.iso \
https://factory.talos.dev/image/$SCHEMATIC/$TALOS/nocloud-amd64.iso
Creating the VMs¶
Create the three VMs in the Proxmox UI with the sizes from the table, and boot them from the Talos ISO.
These are proof-of-concept sizes, just enough to run Talos and a test app. 2 GB of memory, 2 cores and a 10 GB disk is the minimum Talos asks for on a control plane node, and the recommended sizes are a lot larger, so give the nodes more if you plan to run real workloads on them.
There are three settings you need to remember.
- QEMU Guest Agent: enable it under System in the create wizard, or later under Options. This adds the device that the guest agent extension in our image talks to.
- MAC address: set the static MAC address from the table under Network, so the VM gets its static lease from the DHCP server.
- Ballooning: untick Ballooning Device under Memory in the create wizard. It is hidden until you tick Advanced at the bottom of the wizard. Kubernetes schedules workloads based on the memory the node reports, and memory the hypervisor takes back later is not something it can plan around.
Leave the CPU type at the default x86-64-v2-AES. Talos requires the x86-64-v2 instruction set, so don’t pick the
old kvm64 type, which makes the VM hang during boot at 100% CPU. host works too, but it ties the VM to the host’s
CPU model and can block live migration between different hosts.
Since all three VMs are identical, you only need to create the first one. Before starting it, right-click it and choose Clone to create the other two. A clone gets a new random MAC address, so remember to change it to the static MAC address from the table under Hardware, Network Device, before starting the clone.
Start the VMs. Each VM boots into maintenance mode. Talos is running from the ISO, has an IP address from DHCP and waits for a machine configuration. Open the console of one of the VMs and you’ll see the Talos dashboard with the IP in the top section. Check that it matches the DHCP lease you created.
Generating the Configuration¶
Back on your workstation, start with the cluster secrets. This file holds the certificate authorities, keys and tokens for the cluster, and every config file is generated from it.
Before generating the configs, check which disk Talos sees on the nodes. While a node is in maintenance mode, there
is no client certificate yet, which is why --insecure is needed.
With VirtIO SCSI the disk shows up as /dev/sda, which is also what Talos installs to by default. If yours is
different, add --install-disk to the command below.
Now generate the configuration. The first argument is the cluster name, the second is the Kubernetes API endpoint. With a single control plane, that is simply the IP of the control plane node.
SCHEMATIC=ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515
TALOS=v1.14.5
talosctl gen config homelab https://10.0.0.21:6443 \
--with-secrets secrets.yaml \
--install-image factory.talos.dev/nocloud-installer/$SCHEMATIC:$TALOS
Pointing the endpoint at a virtual IP
Using the node IP as the endpoint ties the Kubernetes API to a single machine. Talos can also share a virtual IP between the control plane nodes, which is needed if you want more than one control plane. That is covered in Moving the Talos Kubernetes API to a Virtual IP, the follow-up to this post.
The --install-image flag is easy to miss but important. The ISO only decides what runs in maintenance mode. The
installer image is what actually ends up on the disk, and if you leave it out, you get the plain Talos image without
the guest agent extension.
You now have three new files.
| File | Purpose |
|---|---|
controlplane.yaml |
Machine configuration for control plane nodes |
worker.yaml |
Machine configuration for worker nodes |
talosconfig |
Client configuration for talosctl, with admin cert |
Together with secrets.yaml, these files are the keys to the cluster. Keep them somewhere safe and out of public
Git repositories. If you need to regenerate the configs later, always use --with-secrets secrets.yaml, otherwise you
get a brand new set of certificates that don't match the running cluster.
Notice that we don't need any config patches. Out of the box Talos runs Flannel as the CNI and kube-proxy for services, which is exactly what we want for a plain cluster.
Applying the Configuration¶
Push the configs to the nodes. The control plane gets controlplane.yaml and both workers get the same worker.yaml.
talosctl apply-config --insecure -n 10.0.0.21 -f controlplane.yaml
talosctl apply-config --insecure -n 10.0.0.22 -f worker.yaml
talosctl apply-config --insecure -n 10.0.0.23 -f worker.yaml
Each node now installs Talos to its disk and reboots, coming back up from disk with its configuration applied. If a VM boots into the ISO again, move the disk before the CD-ROM in the boot order under Options, or remove the ISO from the VM.
Bootstrapping the Cluster¶
Next, set up talosctl to talk to the cluster. Merge the generated talosconfig into your default talosconfig file
at ~/.talos/config. The cluster is added as a new context and becomes the current one, so talosctl works from
any directory.
Optionally, keep the talosconfig local
Instead of merging, you can point talosctl at the file in the current directory with the TALOSCONFIG
environment variable. This keeps the cluster’s credentials together with the rest of its files, and the endpoint
and node are then written to ./talosconfig. The variable only lives in the current shell, and since the path is
relative, run talosctl from the cluster directory and export it again in a new terminal.
Then set the endpoint and the default node. The endpoint is where talosctl connects, and the node is the default
target for commands when you don't give -n.
Setting the default node to the control plane only is on purpose. Commands like reboot and upgrade would
otherwise hit every node at once, and the next command must only run against a single node.
Bootstrap etcd on the control plane. This is done exactly once, for the lifetime of the cluster.
Check that all three nodes have found each other. Talos nodes discover the other members of the cluster on their own, so this works before Kubernetes is fully up.
NODE NAMESPACE TYPE ID VERSION HOSTNAME MACHINE TYPE OS ADDRESSES
10.0.0.21 cluster Member talos-def-ghi 1 talos-def-ghi worker Talos (v1.14.5) ["10.0.0.22"]
10.0.0.21 cluster Member talos-jkl-mno 1 talos-jkl-mno worker Talos (v1.14.5) ["10.0.0.23"]
10.0.0.21 cluster Member talos-xyz-abc 1 talos-xyz-abc controlplane Talos (v1.14.5) ["10.0.0.21"]
There should be one member per node. If a node is missing, check that the config was applied to it and that it can reach the control plane.
It takes a few minutes for etcd and the Kubernetes control plane components to come up. You can follow along with the dashboard, or wait for the health check to pass.
Accessing the Cluster¶
When the health check is green, fetch the kubeconfig. Without any arguments it is merged into ~/.kube/config and
set as the current context.
Optionally, keep the kubeconfig local
Instead of merging, give talosctl kubeconfig a directory to write a kubeconfig file there, and point
kubectl at it with the KUBECONFIG environment variable. Just like with TALOSCONFIG, the variable only lives
in the current shell and the path is relative to the cluster directory.
Get the nodes.
NAME STATUS ROLES AGE VERSION INTERNAL-IP ...
talos-xyz-abc Ready control-plane 4m v1.37.x 10.0.0.21 ...
talos-def-ghi Ready <none> 3m v1.37.x 10.0.0.22 ...
talos-jkl-mno Ready <none> 3m v1.37.x 10.0.0.23 ...
All three nodes should be Ready. Since Flannel is installed by Talos as part of the bootstrap, there is no extra
CNI step. The node names are random unless your DHCP server hands out hostnames, which is another good reason for the
static leases.
A quick test that pods can actually be scheduled and reached.
kubectl create deployment nginx --image=nginx --replicas=2
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pods -o wide
kubectl get svc nginx
Point a browser at any node IP on the NodePort, and you should be greeted by the nginx welcome page. Clean up again afterwards.
Day Two¶
A few commands that are useful once the cluster is running.
talosctl -n 10.0.0.22 dashboard # live view of a node
talosctl -n 10.0.0.22 logs kubelet
talosctl -n 10.0.0.22 reboot
talosctl -n 10.0.0.22 shutdown # cordon, drain and power off
To change the configuration of a running node, edit the config file and apply it again. This time without
--insecure, since talosctl now has a client certificate.
Adding another worker is the same procedure as before. Create a VM, wait for maintenance mode and apply worker.yaml
to it. It joins the cluster on its own.
And if you want to start over, delete the VMs and recreate them. That is the nice thing about Talos, the cluster is nothing more than a few files on your workstation and some empty VMs.