kubernetes manifest / kustomize #442

@oschwartz10612 commented on GitHub (Jun 16, 2025):

Yeah absolutely all PRs are welcome! I know the community has been eager for k8s support but our experience is a bit lacking with it so we have not had the chance.

For the config - all the containers dont actually have to reference the same config. As long as they can address each other over the network for API calls the configs can be separated out of that config directory. Gerbil does not need it at all though it is in our deployment which might need to change. Traefik just needs its config files and Pangolin needs just the config.yml.

Only pangolin needs to access the DB.

@oschwartz10612 commented on GitHub (Jun 16, 2025): Yeah absolutely all PRs are welcome! I know the community has been eager for k8s support but our experience is a bit lacking with it so we have not had the chance. For the config - all the containers dont actually have to reference the same config. As long as they can address each other over the network for API calls the configs can be separated out of that config directory. Gerbil does not need it at all though it is in our deployment which might need to change. Traefik just needs its config files and Pangolin needs just the config.yml. Only pangolin needs to access the DB.

GiteaMirror commented

@rdxmb commented on GitHub (Jun 16, 2025):

Thanks for the detailed answer! That's good to know.

In Kubernetes, it's important to differentiate between read-only configurations and data that will be written by the application. We would also prefer to only mount the configurations needed per container. Could you clean up the docker-compose.yml file in that way? (Read-only configurations could be mounted with ro.)

It would be great to have this preparation to clean up our k8s-manifest as well and create a scalable microservice stack.

@rdxmb commented on GitHub (Jun 16, 2025): Thanks for the detailed answer! That's good to know. In Kubernetes, it's important to differentiate between read-only configurations and data that will be written by the application. We would also prefer to only mount the configurations needed per container. Could you clean up the docker-compose.yml file in that way? (Read-only configurations could be mounted with ro.) It would be great to have this preparation to clean up our k8s-manifest as well and create a scalable microservice stack.

GiteaMirror commented

@github-actions[bot] commented on GitHub (Jul 1, 2025):

This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.

@github-actions[bot] commented on GitHub (Jul 1, 2025): This issue has been automatically marked as stale due to 14 days of inactivity. It will be closed in 14 days if no further activity occurs.

GiteaMirror commented

@vtmocanu commented on GitHub (Jul 9, 2025):

also interested in this one, not stale

@vtmocanu commented on GitHub (Jul 9, 2025): also interested in this one, not stale

GiteaMirror commented

@marcschaeferger commented on GitHub (Aug 17, 2025):

I'm working on it. They will be released soon. Currently Alpha Phase

@marcschaeferger commented on GitHub (Aug 17, 2025): I'm working on it. They will be released soon. Currently Alpha Phase

GiteaMirror commented

@chiqors commented on GitHub (Sep 5, 2025):

I am interested with this. Helm chart would be helpful

@chiqors commented on GitHub (Sep 5, 2025): I am interested with this. Helm chart would be helpful

GiteaMirror commented

@marcschaeferger commented on GitHub (Sep 8, 2025):

@chiqors I can send you the Helm chart for Newt or Pangolin if you need it immediately.

The official release will take another 1–2 weeks for the final setup, e.g., publishing on ArtifactHub, setting up CI, and creating beginner-friendly Helm documentation.

@marcschaeferger commented on GitHub (Sep 8, 2025): @chiqors I can send you the Helm chart for Newt or Pangolin if you need it immediately. The official release will take another 1–2 weeks for the final setup, e.g., publishing on ArtifactHub, setting up CI, and creating beginner-friendly Helm documentation.

GiteaMirror commented

@marcschaeferger commented on GitHub (Sep 8, 2025):

The same applies for the manifest files and Kustomize, with options for Service/PodMonitor, PrometheusRules, and multiple deployment setups — for example, a single all-in-one pod or a multi-pod setup where each component runs separately.

Both approaches can be configured to use an external managed PostgreSQL database (e.g., via a PostgreSQL operator like CloudNative-PG or a separate Helm installation of PostgreSQL) or a PostgreSQL instance deployed directly by the Application Manifest/Kustomize.

@marcschaeferger commented on GitHub (Sep 8, 2025): The same applies for the manifest files and Kustomize, with options for Service/PodMonitor, PrometheusRules, and multiple deployment setups — for example, a single all-in-one pod or a multi-pod setup where each component runs separately. Both approaches can be configured to use an external managed PostgreSQL database (e.g., via a PostgreSQL operator like CloudNative-PG or a separate Helm installation of PostgreSQL) or a PostgreSQL instance deployed directly by the Application Manifest/Kustomize.

GiteaMirror commented

@chiqors commented on GitHub (Sep 8, 2025):

@chiqors I can send you the Helm chart for Newt or Pangolin if you need it immediately.

The official release will take another 1–2 weeks for the final setup, e.g., publishing on ArtifactHub, setting up CI, and creating beginner-friendly Helm documentation.

i see.. it's fine, i can wait. Does that use ingress, or gateway api or other method for reverse proxy/expose the service?

@chiqors commented on GitHub (Sep 8, 2025): > [@chiqors](https://github.com/chiqors) I can send you the Helm chart for Newt or Pangolin if you need it immediately. > > The official release will take another 1–2 weeks for the final setup, e.g., publishing on ArtifactHub, setting up CI, and creating beginner-friendly Helm documentation. i see.. it's fine, i can wait. Does that use ingress, or gateway api or other method for reverse proxy/expose the service?

GiteaMirror commented

@marcschaeferger commented on GitHub (Sep 8, 2025):

@chiqors
There are two options:

Option 1:

Use the Traefik pod with an external IP for example, with MetalLB or a load balancer via Cloud Controller Manager annotations, depending on your cloud provider. This approach is similar to the Docker-Compose installation, just transferred to Kubernetes. There’s no need for Ingress or anything else. Traefik Pod reads the Config via HttpFile Provider from Pangolin as it does on the normal Pangolin Installation.

Option 2 (preferred):

Use the Pangolin Traefik Controller, which is a small microservice I wrote in Go. It also supports OTEL (metrics, tracing and many more) and interacts with the K8s API to create IngressRoutes, middlewares, etc. for all resources you create within Pangolin. With this, you don’t need a dedicated Traefik pod for Pangolin just use the Traefik IngressController and later, I plan to add NGINX support as well.

@marcschaeferger commented on GitHub (Sep 8, 2025): @chiqors There are two options: ## Option 1: Use the Traefik pod with an external IP for example, with MetalLB or a load balancer via Cloud Controller Manager annotations, depending on your cloud provider. This approach is similar to the Docker-Compose installation, just transferred to Kubernetes. There’s no need for Ingress or anything else. Traefik Pod reads the Config via HttpFile Provider from Pangolin as it does on the normal Pangolin Installation. ## Option 2 (preferred): Use the Pangolin Traefik Controller, which is a small microservice I wrote in Go. It also supports OTEL (metrics, tracing and many more) and interacts with the K8s API to create IngressRoutes, middlewares, etc. for all resources you create within Pangolin. With this, you don’t need a dedicated Traefik pod for Pangolin just use the Traefik IngressController and later, I plan to add NGINX support as well.

GiteaMirror commented

@chiqors commented on GitHub (Sep 11, 2025):

@chiqors There are two options:

Option 1:

Use the Traefik pod with an external IP for example, with MetalLB or a load balancer via Cloud Controller Manager annotations, depending on your cloud provider. This approach is similar to the Docker-Compose installation, just transferred to Kubernetes. There’s no need for Ingress or anything else. Traefik Pod reads the Config via HttpFile Provider from Pangolin as it does on the normal Pangolin Installation.

Option 2 (preferred):

Use the Pangolin Traefik Controller, which is a small microservice I wrote in Go. It also supports OTEL (metrics, tracing and many more) and interacts with the K8s API to create IngressRoutes, middlewares, etc. for all resources you create within Pangolin. With this, you don’t need a dedicated Traefik pod for Pangolin just use the Traefik IngressController and later, I plan to add NGINX support as well.

Thank you for the detailed explanation.

I'm currently using Pangolin on Docker as a reverse proxy manager, which works flawlessly for my setup: exposing multiple services via subdomains from a single public IP.

Now, I'm planning to migrate to a bare-metal, single-node Kubernetes cluster with Cilium CNI. I want to run the entire Pangolin stack inside the cluster.

My initial thought was to expose Pangolin using a NodePort and then let it handle the reverse proxying from there. However, I was under the impression that this approach might require the Pangolin instance to be outside the cluster, which is what I want to avoid.

Given that my goal is to have everything running within Kubernetes, which of your two options would you recommend for my bare-metal setup?

@chiqors commented on GitHub (Sep 11, 2025): > [@chiqors](https://github.com/chiqors) There are two options: > ## Option 1: > > Use the Traefik pod with an external IP for example, with MetalLB or a load balancer via Cloud Controller Manager annotations, depending on your cloud provider. This approach is similar to the Docker-Compose installation, just transferred to Kubernetes. There’s no need for Ingress or anything else. Traefik Pod reads the Config via HttpFile Provider from Pangolin as it does on the normal Pangolin Installation. > ## Option 2 (preferred): > > Use the Pangolin Traefik Controller, which is a small microservice I wrote in Go. It also supports OTEL (metrics, tracing and many more) and interacts with the K8s API to create IngressRoutes, middlewares, etc. for all resources you create within Pangolin. With this, you don’t need a dedicated Traefik pod for Pangolin just use the Traefik IngressController and later, I plan to add NGINX support as well. Thank you for the detailed explanation. I'm currently using Pangolin on Docker as a reverse proxy manager, which works flawlessly for my setup: exposing multiple services via subdomains from a single public IP. Now, I'm planning to migrate to a bare-metal, single-node Kubernetes cluster with Cilium CNI. I want to run the entire Pangolin stack inside the cluster. My initial thought was to expose Pangolin using a NodePort and then let it handle the reverse proxying from there. However, I was under the impression that this approach might require the Pangolin instance to be outside the cluster, which is what I want to avoid. Given that my goal is to have everything running within Kubernetes, which of your two options would you recommend for my bare-metal setup?

GiteaMirror commented

@marcschaeferger commented on GitHub (Sep 11, 2025):

@chiqors

Given your bare-metal, single-node Kubernetes setup with Cilium CNI and the requirement to run everything inside the cluster, the recommended approach is Option 2 — use the Pangolin Traefik Controller.

Why this is the better choice

Kubernetes-native management and orchestration
This approach aligns perfectly with Kubernetes best practices by leveraging CRDs (Custom Resource Definitions) and the Kubernetes API directly.
Scalability, observability, and maintainability
It’s easier to scale, monitor, and integrate with other Kubernetes-native tools such as Prometheus, Grafana, and Jaeger. This makes the stack more modular, maintainable, and future-proof, especially when adding more nodes in the future.
Reusable for other applications
Since the Traefik IngressController can be used for exposing any application in your cluster. Not only Pangolin which would be the case in Option 1.
Easier high availability compared to a standalone Traefik Pod
Running Traefik as an IngressController allows HA by simply scaling the deployment (e.g., one pod per node), whereas a plain Traefik pod needs manual tweaks for redundancy.

And many other reasons...

Explanation

An Ingress Controller in Kubernetes is essentially a routing component that integrates deeply with the Kubernetes API. Compared to running a plain Traefik pod, it offers richer and more native functionality:

All routing configurations are stored as Kubernetes resources (Ingress, IngressRoute (Traefik-specific), Middleware (Traefik-specific), etc.) instead of large, centralized configuration files.
Switching between controllers (e.g., NGINX ↔ Traefik) is much simpler if you stick to standard Ingress resources, since these are compatible with any controller.
These resources can be inspected, modified, and applied at any time via kubectl, making changes far more transparent and less error-prone.
Being standard Kubernetes YAML definitions, they can easily be incorporated into GitOps workflows (Argo CD, Flux, etc.), allowing automated deployments, version control, and reproducibility.

There is more...

SingleNode consideration:
If possible, avoid running production workloads on a single-node cluster. Aim for at least a 3-node cluster for high availability (HA) and operational resilience. You can even use low-cost hardware such as Raspberry Pi / Compute Modules for additional nodes — these can serve as control-plane nodes or workers, depending on your needs.
At minimum, I would recommend 3 Raspberry Pi control-plane nodes and 1 dedicated worker node (the one you initially planned to use as your single node). This setup will save you troubleshooting time and provide much better fault tolerance.

Bare-metal consideration:
If your node(s) are self-hosted and you opt for Option 2, you will also need something like MetalLB (or another bare-metal load balancer) to provide external IPs for your Traefik IngressController service. For Could you can sometimes use MetalLB as well or the Provider Options e.g Hetzner LB.

If you have more questions just let me know.

@marcschaeferger commented on GitHub (Sep 11, 2025): @chiqors Given your bare-metal, single-node Kubernetes setup with Cilium CNI and the requirement to run everything inside the cluster, the recommended approach is **Option 2 — use the Pangolin Traefik Controller**. --- ### **Why this is the better choice** - **Kubernetes-native management and orchestration** This approach aligns perfectly with Kubernetes best practices by leveraging CRDs (Custom Resource Definitions) and the Kubernetes API directly. - **Scalability, observability, and maintainability** It’s easier to scale, monitor, and integrate with other Kubernetes-native tools such as Prometheus, Grafana, and Jaeger. This makes the stack more **modular, maintainable, and future-proof**, especially when adding more nodes in the future. - **Reusable for other applications** Since the Traefik IngressController can be used for **exposing any application** in your cluster. Not only Pangolin which would be the case in Option 1. - **Easier high availability compared to a standalone Traefik Pod** Running Traefik as an IngressController allows HA by simply scaling the deployment (e.g., one pod per node), whereas a plain Traefik pod needs manual tweaks for redundancy. And many other reasons... --- ### **Explanation** An **Ingress Controller** in Kubernetes is essentially a routing component that integrates deeply with the Kubernetes API. Compared to running a plain Traefik pod, it offers richer and more native functionality: - All routing configurations are stored as **Kubernetes resources** (`Ingress`, `IngressRoute` (Traefik-specific), `Middleware` (Traefik-specific), etc.) instead of large, centralized configuration files. - Switching between controllers (e.g., NGINX ↔ Traefik) is much simpler if you stick to standard `Ingress` resources, since these are compatible with any controller. - These resources can be inspected, modified, and applied at any time via `kubectl`, making changes far more transparent and less error-prone. - Being standard Kubernetes YAML definitions, they can easily be incorporated into **GitOps workflows** (Argo CD, Flux, etc.), allowing automated deployments, version control, and reproducibility. There is more... --- **SingleNode consideration:** If possible, **avoid running production workloads on a single-node cluster**. Aim for at least a **3-node cluster** for high availability (HA) and operational resilience. You can even use low-cost hardware such as **Raspberry Pi / Compute Modules** for additional nodes — these can serve as control-plane nodes or workers, depending on your needs. At minimum, I would recommend **3 Raspberry Pi control-plane nodes** and **1 dedicated worker node** (the one you initially planned to use as your single node). This setup will save you troubleshooting time and provide much better fault tolerance. **Bare-metal consideration:** If your node(s) are self-hosted and you opt for Option 2, you will also need something like **MetalLB** (or another bare-metal load balancer) to provide external IPs for your Traefik IngressController service. For Could you can sometimes use MetalLB as well or the Provider Options e.g Hetzner LB. If you have more questions just let me know.

GiteaMirror commented