Almost 3 years ago, I paid for a few VPSs on which I host a variety of services. (Vaultwarden, gitea, drone, meshcentral, metabase, gptresearcher, etc)
Interspersed among the VPSs are a series of data processing containers to handle crypto data.
With the contract coming up for renewal, I’m exploring how to separate the hardware from the software so I’d only need to deploy the container to a pool of servers, and the infrastructure decides on which server to run the container, correctly route incoming requests, and update cloudflare dns for containers which are meant to be oublicly facing.
I went through the kubernetes the hard way tutorial and have a cursory understanding of kubernetes but with some substantial gaps which I couldn’t Google away.
For the replacement platform, I’m thinking to:
- Combine multiple VPSs as a baseline cluster to run internet-facing loads
- Use some home servers for backend/non-internet facing processes and make the data available on the Internet facing hosts.
- Add the ability to dynamically add more VPSs or preemptible instances from GCP/AWS
I’m still stuck on the first part. Standing up a kubernetes cluster using multiple VPS with different public IPV4 addresses.
Googling around heavily suggests this is not a common use case. Or at least I’m not using the correct terms.
Is there a better solution for me to pursue?
Look at K3s. Since a while it has built-in support for Tailscale (can also use Headscale).
Alternatively, it doesn’t really matter how or where your nodes are located, if you add a VPN to allow them to talk to each other.
Your main issue would be storage. But that’s easily fixed with a topology aware CSI and then keeping your stateful workloads either wherever they got their volumes provisioned, or forcing them to be provisioned on your home servers.
That is what rook-ceph or longhorn is for. Longhorn is good for beginners
Good point about Tailscale. It hadn’t occurred to me.
How does DNS work for a Tailscale-networked cluster of servers?
My understanding is at least one of the nodes would need to be designated as the ingress. I could potentially also have all the master nodes hold the ingress, but then I believe that means I’d need to use round robin DNS in cloudflare to ensure the domains are always pointing at the cluster.
Storage might be a problem, but being more cloud aware potentially means I can run DuckDB against minio to scan S3 objects when doing data-intensive tasks.