Almost 3 years ago, I paid for a few VPSs on which I host a variety of services. (Vaultwarden, gitea, drone, meshcentral, metabase, gptresearcher, etc)

Interspersed among the VPSs are a series of data processing containers to handle crypto data.

With the contract coming up for renewal, I’m exploring how to separate the hardware from the software so I’d only need to deploy the container to a pool of servers, and the infrastructure decides on which server to run the container, correctly route incoming requests, and update cloudflare dns for containers which are meant to be oublicly facing.

I went through the kubernetes the hard way tutorial and have a cursory understanding of kubernetes but with some substantial gaps which I couldn’t Google away.

For the replacement platform, I’m thinking to:

- Combine multiple VPSs as a baseline cluster to run internet-facing loads

- Use some home servers for backend/non-internet facing processes and make the data available on the Internet facing hosts.

- Add the ability to dynamically add more VPSs or preemptible instances from GCP/AWS

I’m still stuck on the first part. Standing up a kubernetes cluster using multiple VPS with different public IPV4 addresses.

Googling around heavily suggests this is not a common use case. Or at least I’m not using the correct terms.

Is there a better solution for me to pursue?

  • natermer@alien.topBEnglish
    1·
    11 months ago

    The problem with hosting kubernetes on VPSes is that exposing the Kubernetes API to the public is pretty sketchy. I know a lot of people do it, but I don’t like the idea.

    I also like having multiple smaller Kubernetes clusters then a single big one. Easier to manage and breakage is more isolated. You can incorporate external services into kubernetes pretty easily using kubes services and endpoints.

    I suggest using K3s as it is very lightweight, easy to deploy, and is k8s compliant. There are default set of services k3s deploys by default and are designed for more ‘IOT’ applications. Things like service-lb. These things can be disabled if you want during install time.

    For managing it I like to ArgoCD on a ‘administrative’ kubes cluster local to you. It has no problem connecting to multiple clusters and has a nice declarative yaml files for configuring things that work well with a git-based workflow. The web UI is nice and is used widely.

    • hardyrekshin@alien.topOPBEnglish
      1·
      11 months ago

      What’s your take on inter-cluster communication?

      E.g. I could hypothetically have 3 clusters:

      1. Administrative
      2. Web-Facing
      3. Backend

      Potential use cases:

      • Backend might produce updated parquet files which needs to be transferred and made accessible on metabase in the Web-Facing cluster.
      • Web-Facing might need to send batched inputs (from webhooks for example) to Backend for processing.
      • ionfury@alien.topBEnglish
        1·
        11 months ago

        Service mesh is what you’re looking for here. Istio is a front runner in this space.

        Without knowing more of your use case though multi cluster is really adding a lot of complexity here and I’m not sure what you’re getting over, say, namespaces and network policy.