Skip to content

A Simple Talos Kubernetes Cluster on Proxmox

In this post, I’ll walk through creating a small Talos Linux Kubernetes cluster on Proxmox VE, with one control plane and two workers. No Cilium, no Cluster API, just three VMs, talosctl and the defaults Talos ships with.

Talos Linux is an operating system built for one job, running Kubernetes. There is no SSH, no shell and no package manager. Everything is managed through an API with talosctl, and the whole node is described by a single machine configuration file. That sounds restrictive at first, but it means a node is either configured exactly like its config file says, or it is not running.

It also makes Talos a very good fit, not only for the enterprise but for homelabs as well. Creating a cluster is a handful of commands, and tearing it down and starting over costs you a few minutes.

The cluster in this post looks like this.

Node Role IP MAC Size
talos-cp-1 Control plane 10.0.0.21 BC:24:11:00:00:21 2 vCPU / 2 GB / 10 GB
talos-w-1 Worker 10.0.0.22 BC:24:11:00:00:22 2 vCPU / 2 GB / 10 GB
talos-w-2 Worker 10.0.0.23 BC:24:11:00:00:23 2 vCPU / 2 GB / 10 GB

Each VM gets a static MAC address, so the DHCP server can hand out a static lease for it. The nodes need fixed IPs because the Kubernetes API endpoint and the talosctl configuration both point at them. Talos can be configured with static IPs in its machine config, but then the addresses are spread across config files. With static leases, the DHCP server stays the single source of truth for addressing, and you can see every node and its IP in one place, next to the rest of your network.

Replace the IPs with something from your own network. The versions used are Talos v1.14.5 and whatever Kubernetes version that Talos release ships as default.

Prerequisites

You need talosctl and kubectl on your workstation. The talosctl version should match the Talos version you are installing.

curl -sL https://talos.dev/install | sh
talosctl version --client

On the network side, create a static lease on your DHCP server for each MAC address in the table above.

Building the Talos Image

Talos images are built by the Image Factory. The factory lets you add system extensions to the image, and on Proxmox the one you want is the QEMU guest agent. Without it, Proxmox can't see the IP of the VM and a shutdown from the Proxmox UI will not be passed on to Talos gracefully.

An image is identified by a schematic ID, which is a hash of the customization. Since it is a hash, the same customization always gives the same ID. You can get it through the web UI or the API, and both end up with the exact same ID.

Using the Web UI

Go to factory.talos.dev and click through the wizard.

  1. Hardware Type: Cloud Server.
  2. Talos Linux Version: 1.14.5.
  3. Cloud: Nocloud.
  4. Machine Architecture: amd64, with SecureBoot left off.
  5. System Extensions: search for and select siderolabs/qemu-guest-agent.
  6. Customization: leave the extra kernel arguments empty.

The last page shows the schematic ID along with download links for the ISO and the installer image to use when generating the configuration. Copy the schematic ID, you’ll need it twice.

Using the API

If you would rather script it, describe the customization in a file called schematic.yaml.

customization:
  systemExtensions:
    officialExtensions:
      - siderolabs/qemu-guest-agent

Post it to the factory to get the ID back.

curl -X POST --data-binary @schematic.yaml https://factory.talos.dev/schematics
{"id":"ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515"}

Downloading the ISO

Download the ISO and upload it to the ISO storage in Proxmox, or download it directly on the Proxmox host.

SCHEMATIC=ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515
TALOS=v1.14.5

curl -Lo talos-$TALOS-nocloud-amd64.iso \
  https://factory.talos.dev/image/$SCHEMATIC/$TALOS/nocloud-amd64.iso

Creating the VMs

Create the three VMs in the Proxmox UI with the sizes from the table, and boot them from the Talos ISO.

These are proof-of-concept sizes, just enough to run Talos and a test app. 2 GB of memory, 2 cores and a 10 GB disk is the minimum Talos asks for on a control plane node, and the recommended sizes are a lot larger, so give the nodes more if you plan to run real workloads on them.

There are three settings you need to remember.

  • QEMU Guest Agent: enable it under System in the create wizard, or later under Options. This adds the device that the guest agent extension in our image talks to.
  • MAC address: set the static MAC address from the table under Network, so the VM gets its static lease from the DHCP server.
  • Ballooning: untick Ballooning Device under Memory in the create wizard. It is hidden until you tick Advanced at the bottom of the wizard. Kubernetes schedules workloads based on the memory the node reports, and memory the hypervisor takes back later is not something it can plan around.

Leave the CPU type at the default x86-64-v2-AES. Talos requires the x86-64-v2 instruction set, so don’t pick the old kvm64 type, which makes the VM hang during boot at 100% CPU. host works too, but it ties the VM to the host’s CPU model and can block live migration between different hosts.

Since all three VMs are identical, you only need to create the first one. Before starting it, right-click it and choose Clone to create the other two. A clone gets a new random MAC address, so remember to change it to the static MAC address from the table under Hardware, Network Device, before starting the clone.

Start the VMs. Each VM boots into maintenance mode. Talos is running from the ISO, has an IP address from DHCP and waits for a machine configuration. Open the console of one of the VMs and you’ll see the Talos dashboard with the IP in the top section. Check that it matches the DHCP lease you created.

Generating the Configuration

Back on your workstation, start with the cluster secrets. This file holds the certificate authorities, keys and tokens for the cluster, and every config file is generated from it.

talosctl gen secrets -o secrets.yaml

Before generating the configs, check which disk Talos sees on the nodes. While a node is in maintenance mode, there is no client certificate yet, which is why --insecure is needed.

talosctl get disks --insecure -n 10.0.0.21

With VirtIO SCSI the disk shows up as /dev/sda, which is also what Talos installs to by default. If yours is different, add --install-disk to the command below.

Now generate the configuration. The first argument is the cluster name, the second is the Kubernetes API endpoint. With a single control plane, that is simply the IP of the control plane node.

SCHEMATIC=ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515
TALOS=v1.14.5

talosctl gen config homelab https://10.0.0.21:6443 \
  --with-secrets secrets.yaml \
  --install-image factory.talos.dev/nocloud-installer/$SCHEMATIC:$TALOS

Pointing the endpoint at a virtual IP

Using the node IP as the endpoint ties the Kubernetes API to a single machine. Talos can also share a virtual IP between the control plane nodes, which is needed if you want more than one control plane. That is covered in Moving the Talos Kubernetes API to a Virtual IP, the follow-up to this post.

The --install-image flag is easy to miss but important. The ISO only decides what runs in maintenance mode. The installer image is what actually ends up on the disk, and if you leave it out, you get the plain Talos image without the guest agent extension.

You now have three new files.

File Purpose
controlplane.yaml Machine configuration for control plane nodes
worker.yaml Machine configuration for worker nodes
talosconfig Client configuration for talosctl, with admin cert

Together with secrets.yaml, these files are the keys to the cluster. Keep them somewhere safe and out of public Git repositories. If you need to regenerate the configs later, always use --with-secrets secrets.yaml, otherwise you get a brand new set of certificates that don't match the running cluster.

Notice that we don't need any config patches. Out of the box Talos runs Flannel as the CNI and kube-proxy for services, which is exactly what we want for a plain cluster.

Applying the Configuration

Push the configs to the nodes. The control plane gets controlplane.yaml and both workers get the same worker.yaml.

talosctl apply-config --insecure -n 10.0.0.21 -f controlplane.yaml
talosctl apply-config --insecure -n 10.0.0.22 -f worker.yaml
talosctl apply-config --insecure -n 10.0.0.23 -f worker.yaml

Each node now installs Talos to its disk and reboots, coming back up from disk with its configuration applied. If a VM boots into the ISO again, move the disk before the CD-ROM in the boot order under Options, or remove the ISO from the VM.

Bootstrapping the Cluster

Next, set up talosctl to talk to the cluster. Merge the generated talosconfig into your default talosconfig file at ~/.talos/config. The cluster is added as a new context and becomes the current one, so talosctl works from any directory.

talosctl config merge ./talosconfig

Optionally, keep the talosconfig local

Instead of merging, you can point talosctl at the file in the current directory with the TALOSCONFIG environment variable. This keeps the cluster’s credentials together with the rest of its files, and the endpoint and node are then written to ./talosconfig. The variable only lives in the current shell, and since the path is relative, run talosctl from the cluster directory and export it again in a new terminal.

export TALOSCONFIG=./talosconfig

Then set the endpoint and the default node. The endpoint is where talosctl connects, and the node is the default target for commands when you don't give -n.

talosctl config endpoint 10.0.0.21
talosctl config node 10.0.0.21

Setting the default node to the control plane only is on purpose. Commands like reboot and upgrade would otherwise hit every node at once, and the next command must only run against a single node.

Bootstrap etcd on the control plane. This is done exactly once, for the lifetime of the cluster.

talosctl bootstrap

Check that all three nodes have found each other. Talos nodes discover the other members of the cluster on their own, so this works before Kubernetes is fully up.

talosctl get members
NODE        NAMESPACE   TYPE     ID              VERSION   HOSTNAME        MACHINE TYPE   OS                ADDRESSES
10.0.0.21   cluster     Member   talos-def-ghi   1         talos-def-ghi   worker         Talos (v1.14.5)   ["10.0.0.22"]
10.0.0.21   cluster     Member   talos-jkl-mno   1         talos-jkl-mno   worker         Talos (v1.14.5)   ["10.0.0.23"]
10.0.0.21   cluster     Member   talos-xyz-abc   1         talos-xyz-abc   controlplane   Talos (v1.14.5)   ["10.0.0.21"]

There should be one member per node. If a node is missing, check that the config was applied to it and that it can reach the control plane.

It takes a few minutes for etcd and the Kubernetes control plane components to come up. You can follow along with the dashboard, or wait for the health check to pass.

talosctl dashboard
talosctl health

Accessing the Cluster

When the health check is green, fetch the kubeconfig. Without any arguments it is merged into ~/.kube/config and set as the current context.

talosctl kubeconfig

Optionally, keep the kubeconfig local

Instead of merging, give talosctl kubeconfig a directory to write a kubeconfig file there, and point kubectl at it with the KUBECONFIG environment variable. Just like with TALOSCONFIG, the variable only lives in the current shell and the path is relative to the cluster directory.

talosctl kubeconfig .
export KUBECONFIG=./kubeconfig

Get the nodes.

kubectl get nodes -o wide
NAME            STATUS   ROLES           AGE   VERSION   INTERNAL-IP    ...
talos-xyz-abc   Ready    control-plane   4m    v1.37.x   10.0.0.21   ...
talos-def-ghi   Ready    <none>          3m    v1.37.x   10.0.0.22   ...
talos-jkl-mno   Ready    <none>          3m    v1.37.x   10.0.0.23   ...

All three nodes should be Ready. Since Flannel is installed by Talos as part of the bootstrap, there is no extra CNI step. The node names are random unless your DHCP server hands out hostnames, which is another good reason for the static leases.

A quick test that pods can actually be scheduled and reached.

kubectl create deployment nginx --image=nginx --replicas=2
kubectl expose deployment nginx --port=80 --type=NodePort
kubectl get pods -o wide
kubectl get svc nginx

Point a browser at any node IP on the NodePort, and you should be greeted by the nginx welcome page. Clean up again afterwards.

kubectl delete svc,deployment nginx

Day Two

A few commands that are useful once the cluster is running.

talosctl -n 10.0.0.22 dashboard   # live view of a node
talosctl -n 10.0.0.22 logs kubelet
talosctl -n 10.0.0.22 reboot
talosctl -n 10.0.0.22 shutdown    # cordon, drain and power off

To change the configuration of a running node, edit the config file and apply it again. This time without --insecure, since talosctl now has a client certificate.

talosctl apply-config -n 10.0.0.22,10.0.0.23 -f worker.yaml

Adding another worker is the same procedure as before. Create a VM, wait for maintenance mode and apply worker.yaml to it. It joins the cluster on its own.

And if you want to start over, delete the VMs and recreate them. That is the nice thing about Talos, the cluster is nothing more than a few files on your workstation and some empty VMs.