Run multiple Gitea pods in Kubernetes #2418

@BetaCat0 commented on GitHub (Oct 19, 2018):

That's a problem and we are planning to create an independent index server to solve this. @abulfat-masimov

@BetaCat0 commented on GitHub (Oct 19, 2018): That's a problem and we are planning to create an independent index server to solve this. @abulfat-masimov

GiteaMirror commented

@techknowlogick commented on GitHub (Oct 19, 2018):

Presslabs have had success running Gitea in k8s: https://www.presslabs.com/news/sftp-gitea/ you may be able to ask them for some implementation details.

@techknowlogick commented on GitHub (Oct 19, 2018): Presslabs have had success running Gitea in k8s: https://www.presslabs.com/news/sftp-gitea/ you may be able to ask them for some implementation details.

GiteaMirror commented

@Sluggerman commented on GitHub (Oct 19, 2018):

great, thanks @techknowlogick! I'll reach out to them to ask if they have Gitea running in scaled mode.

@Sluggerman commented on GitHub (Oct 19, 2018): great, thanks @techknowlogick! I'll reach out to them to ask if they have Gitea running in scaled mode.

GiteaMirror commented

@feluxe commented on GitHub (Oct 22, 2018):

I was thinking about a solution for this as well. I currently have this in mind:

Create a shared volume (one that supports ReadWriteMany, e.g. CephFS cluster deployed via Rook).
Run multiple instances of gitea:

connect each instance to the shared volume from 1).
use redis for cache and session for each instance.

Sharding: To ensure that each repo is written by a single instance at a time, the loadbalancer directs URL requests in alphabetical ranges, e.g.:

www.mygitea.io/[a-g]   ---> Instance-1
www.mygitea.io/[h-n]   ---> Instance-2
www.mygitea.io/[o-u]   ---> Instance-3
www.mygitea.io/[v-z]   ---> Instance-4

Note: I'm not sure if 3) is even necessary with CephFS? From my understanding, it comes with a locking mechanism to handle reads/writes of multiple clients.

To solve the problem with the indexer(s):

All instances of gitea have indexing disabled. These instances can read but not write indexes.
Create another pod that has Indexing enabled. This pod is responsible for writing indexes only.

Note: I haven't tested this yet... I think for 4) to work gitea would need to have an option to disable indexing for an instance entirely.

Note: Again. I'm not sure if 4) is necessary with CephFS.

I'd love to hear some feedback before I jump into this. :)

@feluxe commented on GitHub (Oct 22, 2018): I was thinking about a solution for this as well. I currently have this in mind: 1) Create a shared volume (one that supports `ReadWriteMany`, e.g. [CephFS](https://ceph.com/) cluster deployed via [Rook](https://rook.io/)). 2) Run multiple instances of gitea: * connect each instance to the shared volume from 1). * use *redis* for `cache` and `session` for each instance. 3) Sharding: To ensure that each repo is written by a single instance at a time, the loadbalancer directs URL requests in alphabetical ranges, e.g.: ``` www.mygitea.io/[a-g] ---> Instance-1 www.mygitea.io/[h-n] ---> Instance-2 www.mygitea.io/[o-u] ---> Instance-3 www.mygitea.io/[v-z] ---> Instance-4 ``` Note: I'm not sure if 3) is even necessary with CephFS? From my understanding, it comes with a locking mechanism to handle reads/writes of multiple clients. 4) To solve the problem with the *indexer(s)*: * All instances of gitea have indexing **disabled**. These instances can *read* but not *write* indexes. * Create another pod that has Indexing **enabled**. This pod is responsible for *writing* indexes only. Note: I haven't tested this yet... I think for 4) to work gitea would need to have an option to disable indexing for an instance entirely. Note: Again. I'm not sure if 4) is necessary with CephFS. I'd love to hear some feedback before I jump into this. :)

GiteaMirror commented

@bogdanpetrea commented on GitHub (Oct 23, 2018):

@techknowlogick: That article is about dropping the SFTP support and unfortunately it's about running a single instance of Gitea in a non-Kubernetes environment. We are currently working on running Gitea on Kubernetes, but we decided to have multiple Gitea instances, logically separated from each other. That's not really the scaling you are probably looking for.

@bogdanpetrea commented on GitHub (Oct 23, 2018): @techknowlogick: That article is about dropping the SFTP support and unfortunately it's about running a single instance of Gitea in a non-Kubernetes environment. We are currently working on running Gitea on Kubernetes, but we decided to have multiple Gitea instances, logically separated from each other. That's not really the scaling you are probably looking for.

GiteaMirror commented

@abulfatmasimov commented on GitHub (Oct 23, 2018):

I guess even if we manage to forward write traffic (POST,PUT) to a single Gitea instance to mitigate risks with corruption, yet it will not be scalable solution as such. I think If something like pessimistic locking could be implemented in Gitea to support simultaneous writes, that would be a real game changer.

@abulfatmasimov commented on GitHub (Oct 23, 2018): I guess even if we manage to forward write traffic (POST,PUT) to a single Gitea instance to mitigate risks with corruption, yet it will not be scalable solution as such. I think If something like pessimistic locking could be implemented in Gitea to support simultaneous writes, that would be a real game changer.

GiteaMirror commented

@lunny commented on GitHub (Oct 23, 2018):

@feluxe I think 3) is unnecessary. 4) needs some codes change.

@lunny commented on GitHub (Oct 23, 2018): @feluxe I think 3) is unnecessary. 4) needs some codes change.

GiteaMirror commented

@BetaCat0 commented on GitHub (Oct 23, 2018):

@feluxe This looks like somekind of a master-replica solution. As @abulfat-masimov said, it still needs a R/W router to avoid risks of corruption and may not alleviate the writing pressure of main instance (potential SPOF problem).

@BetaCat0 commented on GitHub (Oct 23, 2018): @feluxe This looks like somekind of a master-replica solution. As @abulfat-masimov said, it still needs a R/W router to avoid risks of corruption and may not alleviate the writing pressure of main instance (potential SPOF problem).

GiteaMirror commented

@jfelten commented on GitHub (Nov 7, 2018):

I've been thinking about this, but lack understanding of where the resource conflicts might be. I think the best way would be to use a stateful set that uses a common queue service for write requests. A lot would depend on the type of file storage and persistence, but I believe postgres at least can be fed from a queue. Other than that you could have a stateful set with db replication set up somehow and fancy file storage that handles concurrency for you. I created a helm chart for gitea but it only uses a single replica, which is enough for our usage.

@jfelten commented on GitHub (Nov 7, 2018): I've been thinking about this, but lack understanding of where the resource conflicts might be. I think the best way would be to use a stateful set that uses a common queue service for write requests. A lot would depend on the type of file storage and persistence, but I believe postgres at least can be fed from a queue. Other than that you could have a stateful set with db replication set up somehow and fancy file storage that handles concurrency for you. I created a helm chart for gitea but it only uses a single replica, which is enough for our usage.

GiteaMirror commented

@schmitch commented on GitHub (Dec 6, 2018):

actually the biggest problem is that redis-ha is not supported.
I mean you can run a redis service of your cloud provider. but you can't use redis-ha helm chart since redis clients need to be sentinal aware.

@schmitch commented on GitHub (Dec 6, 2018): actually the biggest problem is that redis-ha is not supported. I mean you can run a redis service of your cloud provider. but you can't use redis-ha helm chart since redis clients need to be sentinal aware.

GiteaMirror commented

@benyanke commented on GitHub (Dec 25, 2018):

What's the current state of this? I'm currently considering setting up Gitea for a production environment, and curious if fully HA mode is supported yet.

@benyanke commented on GitHub (Dec 25, 2018): What's the current state of this? I'm currently considering setting up Gitea for a production environment, and curious if fully HA mode is supported yet.

GiteaMirror commented

@stale[bot] commented on GitHub (Feb 23, 2019):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

@stale[bot] commented on GitHub (Feb 23, 2019): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

GiteaMirror commented

@stale[bot] commented on GitHub (Mar 9, 2019):

This issue has been automatically closed because of inactivity. You can re-open it if needed.

@stale[bot] commented on GitHub (Mar 9, 2019): This issue has been automatically closed because of inactivity. You can re-open it if needed.

GiteaMirror commented