Session: Redis PANIC #350

Closed
opened 2025-11-02 03:19:46 -06:00 by GiteaMirror · 6 comments
Owner

Originally created by @jag3773 on GitHub (Feb 14, 2017).

  • Gitea version (or commit ref):
  • Git version: 2.7.4
  • Operating system: aws linux / centos
  • Database (use [x]):
    • PostgreSQL
    • MySQL
    • SQLite
  • Can you reproduce the bug at https://try.gitea.io:
    • Yes (provide example URL)
    • No
    • Not relevant
  • Log gist:

Description

I've got 3 identical servers setup and am hooking them up to use the same redis server for session data. Config:

[session]
PROVIDER = redis
COOKIE_NAME = dcs_session
PROVIDER_CONFIG: network=tcp,addr=myredis.zw3b9b.clustercfg.usw2.cache.amazonaws.com:6379,password=,db=0,pool_size=100,idle_timeout=180

The first node has connected and appears to be functioning normally.

Gitea starts on the other two nodes but every time a page is loaded they are giving me PANIC messages like this:

[Macaron] PANIC: session(start): MOVED 4023 172.31.37.108:6379
/usr/local/go/src/runtime/panic.go:426 (0x42df39)
/home/git/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/session/session.go:156 (0x8be368)
/usr/local/go/src/runtime/asm_amd64.s:472 (0x45dcce)
/usr/local/go/src/reflect/value.go:435 (0x92de0d)
/usr/local/go/src/reflect/value.go:303 (0x92cad1)
/home/git/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:177 (0xc92429)
/home/git/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:137 (0xc91bec)
/home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:113 (0x8c81a3)
/home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:104 (0x8c808c)
/home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/recovery.go:161 (0x8de3d2)
/usr/local/go/src/runtime/asm_amd64.s:472 (0x45dcce)
/usr/local/go/src/reflect/value.go:435 (0x92de0d)
/usr/local/go/src/reflect/value.go:303 (0x92cad1)
/home/git/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:177 (0xc92429)
/home/git/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:137 (0xc91bec)
/home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:113 (0x8c81a3)
/home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/router.go:184 (0x8dfd64)
/home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/router.go:286 (0x8d633d)
/home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/macaron.go:177 (0x8cd7ea)
/usr/local/go/src/net/http/server.go:2081 (0x75edce)
/usr/local/go/src/net/http/server.go:1472 (0x75b75e)
/usr/local/go/src/runtime/asm_amd64.s:1998 (0x4603c1)

As I monitor the redis instance I can't see these two nodes running any commands. I can manually connect to redis from all of the nodes so it doesn't appear to be a connection issue.

Any thoughts?

...

Originally created by @jag3773 on GitHub (Feb 14, 2017). - Gitea version (or commit ref): - Git version: 2.7.4 - Operating system: aws linux / centos - Database (use `[x]`): - [ ] PostgreSQL - [x] MySQL - [ ] SQLite - Can you reproduce the bug at https://try.gitea.io: - [ ] Yes (provide example URL) - [ ] No - [x] Not relevant - Log gist: ## Description I've got 3 identical servers setup and am hooking them up to use the same redis server for session data. Config: ``` [session] PROVIDER = redis COOKIE_NAME = dcs_session PROVIDER_CONFIG: network=tcp,addr=myredis.zw3b9b.clustercfg.usw2.cache.amazonaws.com:6379,password=,db=0,pool_size=100,idle_timeout=180 ``` The first node has connected and appears to be functioning normally. Gitea starts on the other two nodes but every time a page is loaded they are giving me PANIC messages like this: ``` [Macaron] PANIC: session(start): MOVED 4023 172.31.37.108:6379 /usr/local/go/src/runtime/panic.go:426 (0x42df39) /home/git/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/session/session.go:156 (0x8be368) /usr/local/go/src/runtime/asm_amd64.s:472 (0x45dcce) /usr/local/go/src/reflect/value.go:435 (0x92de0d) /usr/local/go/src/reflect/value.go:303 (0x92cad1) /home/git/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:177 (0xc92429) /home/git/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:137 (0xc91bec) /home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:113 (0x8c81a3) /home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:104 (0x8c808c) /home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/recovery.go:161 (0x8de3d2) /usr/local/go/src/runtime/asm_amd64.s:472 (0x45dcce) /usr/local/go/src/reflect/value.go:435 (0x92de0d) /usr/local/go/src/reflect/value.go:303 (0x92cad1) /home/git/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:177 (0xc92429) /home/git/go/src/code.gitea.io/gitea/vendor/github.com/go-macaron/inject/inject.go:137 (0xc91bec) /home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/context.go:113 (0x8c81a3) /home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/router.go:184 (0x8dfd64) /home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/router.go:286 (0x8d633d) /home/git/go/src/code.gitea.io/gitea/vendor/gopkg.in/macaron.v1/macaron.go:177 (0x8cd7ea) /usr/local/go/src/net/http/server.go:2081 (0x75edce) /usr/local/go/src/net/http/server.go:1472 (0x75b75e) /usr/local/go/src/runtime/asm_amd64.s:1998 (0x4603c1) ``` As I monitor the redis instance I can't see these two nodes running any commands. I can manually connect to redis from all of the nodes so it doesn't appear to be a connection issue. Any thoughts? ...
GiteaMirror added the topic/deployment label 2025-11-02 03:19:46 -06:00
Author
Owner

@lunny commented on GitHub (Feb 14, 2017):

It seems it's a macaron-sessions' bug. @Unknwon

@lunny commented on GitHub (Feb 14, 2017): It seems it's a macaron-sessions' bug. @Unknwon
Author
Owner

@unknwon commented on GitHub (Feb 14, 2017):

Did a quick research and I think this exception should be handled (retry) by Go's redis lib: http://serverfault.com/questions/812156/redis-cluster-error-moved

@unknwon commented on GitHub (Feb 14, 2017): Did a quick research and I think this exception should be handled (retry) by Go's redis lib: http://serverfault.com/questions/812156/redis-cluster-error-moved
Author
Owner

@unknwon commented on GitHub (Feb 14, 2017):

I think the cause is gopkg.in/redis.v2 doesn't have cluster support, so you should probably just use single instance of redis?

@unknwon commented on GitHub (Feb 14, 2017): I think the cause is `gopkg.in/redis.v2` doesn't have cluster support, so you should probably just use single instance of redis?
Author
Owner

@jag3773 commented on GitHub (Feb 14, 2017):

I think AWS should be transparently providing the cluster support. In other words, I don't think the clients need to know anything about it. However, that is a good thing to try. Tomorrow I can try to spin up a single redis instance and see what happens.

@jag3773 commented on GitHub (Feb 14, 2017): I *think* AWS should be transparently providing the cluster support. In other words, I don't think the clients need to know anything about it. However, that is a good thing to try. Tomorrow I can try to spin up a single redis instance and see what happens.
Author
Owner

@unknwon commented on GitHub (Feb 14, 2017):

MOVED seems a special command for Redis cluster which tells client to retry on given new address.

@unknwon commented on GitHub (Feb 14, 2017): `MOVED` seems a special command for Redis cluster which tells client to retry on given new address.
Author
Owner

@jag3773 commented on GitHub (Feb 14, 2017):

@Unknwon Thanks! That was the issue. I've setup a single ElastiCache Redis instance and now all the gitea nodes are connected and functioning properly.

@jag3773 commented on GitHub (Feb 14, 2017): @Unknwon Thanks! That was the issue. I've setup a single ElastiCache Redis instance and now all the gitea nodes are connected and functioning properly.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#350