You are not logged in.
Pages: 1
Upgraded gitlab and the service constantly cycles. Testing in /usr/share/webapps/gitlab
sudo -u gitlab bundle exec puma -C config/puma.rb -e production
{"timestamp":"2023-11-05T14:36:53.841Z","pid":829788,"message":"Puma starting in cluster mode..."}
{"timestamp":"2023-11-05T14:36:53.855Z","pid":829788,"message":"* Puma version: 6.4.0 (ruby 3.0.6-p216) (\"The Eagle of Durango\")"}
{"timestamp":"2023-11-05T14:36:53.855Z","pid":829788,"message":"* Min threads: 0"}
{"timestamp":"2023-11-05T14:36:53.855Z","pid":829788,"message":"* Max threads: 16"}
{"timestamp":"2023-11-05T14:36:53.855Z","pid":829788,"message":"* Environment: production"}
{"timestamp":"2023-11-05T14:36:53.855Z","pid":829788,"message":"* Master PID: 829788"}
{"timestamp":"2023-11-05T14:36:53.855Z","pid":829788,"message":"* Workers: 3"}
{"timestamp":"2023-11-05T14:36:53.855Z","pid":829788,"message":"* Restarts: (✔) hot (✖) phased"}
{"timestamp":"2023-11-05T14:36:53.855Z","pid":829788,"message":"* Preloading application"}
WARNING: The Rust extension for prometheus-client-mmap is unavailable, falling back to the legacy C extension.
The Rust extension will be required in the next version. If you are compiling this gem from source,
ensure your build system has a Rust compiler and clang: https://gitlab.com/gitlab-org/ruby/gems … lient-mmap
/usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: already initialized constant Gitlab::Instrumentation::Redis::ActionCable
/usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: previous definition of ActionCable was here
{"timestamp":"2023-11-05T14:38:07.727Z","pid":829788,"message":"* Listening on unix:///run/gitlab/gitlab.socket"}
{"timestamp":"2023-11-05T14:38:07.727Z","pid":829788,"message":"! WARNING: Detected 2 Thread(s) started in app boot:"}
{"timestamp":"2023-11-05T14:38:07.727Z","pid":829788,"message":"! #\u003cThread:0x000055a261d84150 /usr/share/webapps/gitlab/vendor/bundle/ruby/3.0.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/scheduler.rb:73 sleep\u003e - /usr/share/webapps/gitlab/vendor/bundle/ruby/3.0.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/scheduler.rb:91:in `sleep'"}
{"timestamp":"2023-11-05T14:38:07.727Z","pid":829788,"message":"! #\u003cThread:0x000055a24ac9d960 /usr/share/webapps/gitlab/vendor/bundle/ruby/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/session_flusher.rb:81 sleep\u003e - /usr/share/webapps/gitlab/vendor/bundle/ruby/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/session_flusher.rb:83:in `sleep'"}
{"timestamp":"2023-11-05T14:38:07.727Z","pid":829788,"message":"Use Ctrl-C to stop"}
For me it almost seems like it starts correctly but returns a faulty state?
Any ideas are welcome, thanks
Offline
Offline
Updated, not the problem. Found out that gitlab-workhorse keeps restarting with the following log output:
Nov 08 07:51:57 triback systemd[1]: Started Gitlab Workhorse.
Nov 08 07:51:57 triback gitlab-workhorse[974708]: time="2023-11-08T07:51:57+01:00" level=info msg=Starting build_time=20231107.065356 version=v16.5.1
Nov 08 07:51:57 triback gitlab-workhorse[974708]: time="2023-11-08T07:51:57+01:00" level=info msg="Using gomodule/redigo"
Nov 08 07:51:57 triback gitlab-workhorse[974708]: time="2023-11-08T07:51:57+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /run/gitlab/gitlab.socket: connect: connection refused" method=GET uri=
Nov 08 07:51:57 triback gitlab-workhorse[974708]: time="2023-11-08T07:51:57+01:00" level=info msg="Running upstream server" address=/run/gitlab/gitlab-workhorse.socket network=unix
Nov 08 07:52:07 triback gitlab-workhorse[974708]: time="2023-11-08T07:52:07+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /run/gitlab/gitlab.socket: connect: connection refused" method=GET uri=
Nov 08 07:52:17 triback gitlab-workhorse[974708]: time="2023-11-08T07:52:17+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /run/gitlab/gitlab.socket: connect: connection refused" method=GET uri=
Nov 08 07:52:27 triback gitlab-workhorse[974708]: time="2023-11-08T07:52:27+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /run/gitlab/gitlab.socket: connect: connection refused" method=GET uri=
Nov 08 07:52:37 triback gitlab-workhorse[974708]: time="2023-11-08T07:52:37+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /run/gitlab/gitlab.socket: connect: connection refused" method=GET uri=
Nov 08 07:52:42 triback systemd[1]: Stopping Gitlab Workhorse...
Nov 08 07:52:42 triback gitlab-workhorse[974708]: SIGQUIT: quit
Nov 08 07:52:42 triback gitlab-workhorse[974708]: PC=0x55e4a2245341 m=0 sigcode=0
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 0 [idle]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.futex()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/sys_linux_amd64.s:557 +0x21 fp=0x7fff05fad008 sp=0x7fff05fad000 pc=0x55e4a2245341
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.futexsleep(0x0?, 0x0?, 0x1ff05fad0b0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/os_linux.go:69 +0x30 fp=0x7fff05fad058 sp=0x7fff05fad008 pc=0x55e4a2209650
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.notesleep(0x55e4a495baa8)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/lock_futex.go:160 +0x87 fp=0x7fff05fad090 sp=0x7fff05fad058 pc=0x55e4a21e0a07
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.mPark(...)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:1632
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.stopm()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:2536 +0x8c fp=0x7fff05fad0c0 sp=0x7fff05fad090 pc=0x55e4a2213e6c
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.findRunnable()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:3229 +0xb9c fp=0x7fff05fad1d0 sp=0x7fff05fad0c0 pc=0x55e4a221579c
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.schedule()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:3582 +0xb1 fp=0x7fff05fad208 sp=0x7fff05fad1d0 pc=0x55e4a2216591
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.park_m(0xc0000071e0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:3745 +0x11f fp=0x7fff05fad250 sp=0x7fff05fad208 pc=0x55e4a2216a9f
Nov 08 07:52:42 triback gitlab-workhorse[974708]: traceback: unexpected SPWRITE function runtime.mcall
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.mcall()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:458 +0x50 fp=0x7fff05fad268 sp=0x7fff05fad250 pc=0x55e4a22414f0
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 1 [runnable]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0xc0008d5768?, 0x2?, 0x45?, 0x6e?, 0xc0008d54bc?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc0008d5198 sp=0xc0008d5178 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.selectgo(0xc0008d5768, 0xc0008d54b8, 0x0?, 0x0, 0x0?, 0x1)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/select.go:327 +0x725 fp=0xc0008d52b8 sp=0xc0008d5198 pc=0x55e4a2220645
Nov 08 07:52:42 triback gitlab-workhorse[974708]: main.run({{0x55e4a3028c83, 0x1a}, {0x7fff05faedae, 0x23}, {0x7fff05faed9d, 0x4}, 0x0, {0x0, 0x0}, {0x0, ...}, ...}, ...)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: gitlab.com/gitlab-org/gitlab/workhorse/main.go:294 +0x144d fp=0xc0008d5ab0 sp=0xc0008d52b8 pc=0x55e4a2ffc2ed
Nov 08 07:52:42 triback gitlab-workhorse[974708]: main.main()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: gitlab.com/gitlab-org/gitlab/workhorse/main.go:68 +0x3b8 fp=0xc0008d5f40 sp=0xc0008d5ab0 pc=0x55e4a2ff9cb8
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.main()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:267 +0x2d2 fp=0xc0008d5fe0 sp=0xc0008d5f40 pc=0x55e4a220fc12
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0008d5fe8 sp=0xc0008d5fe0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 2 [force gc (idle)]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc000064fa8 sp=0xc000064f88 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goparkunlock(...)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:404
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.forcegchelper()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:322 +0xb8 fp=0xc000064fe0 sp=0xc000064fa8 pc=0x55e4a220fef8
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000064fe8 sp=0xc000064fe0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by runtime.init.6 in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:310 +0x1a
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 3 [GC sweep wait]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc000065778 sp=0xc000065758 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goparkunlock(...)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:404
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.bgsweep(0x0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgcsweep.go:321 +0xdf fp=0xc0000657c8 sp=0xc000065778 pc=0x55e4a21fa1ff
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gcenable.func1()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:200 +0x25 fp=0xc0000657e0 sp=0xc0000657c8 pc=0x55e4a21ef305
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0000657e8 sp=0xc0000657e0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by runtime.gcenable in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:200 +0x66
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 4 [GC scavenge wait]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0xc00007c000?, 0x55e4a363e138?, 0x0?, 0x0?, 0x0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc000065f70 sp=0xc000065f50 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goparkunlock(...)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:404
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.(*scavengerState).park(0x55e4a495ab60)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgcscavenge.go:425 +0x49 fp=0xc000065fa0 sp=0xc000065f70 pc=0x55e4a21f7a49
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.bgscavenge(0x0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgcscavenge.go:658 +0x59 fp=0xc000065fc8 sp=0xc000065fa0 pc=0x55e4a21f7ff9
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gcenable.func2()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:201 +0x25 fp=0xc000065fe0 sp=0xc000065fc8 pc=0x55e4a21ef2a5
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000065fe8 sp=0xc000065fe0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by runtime.gcenable in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:201 +0xa5
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 5 [finalizer wait]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0x55e4a3dfdc80?, 0x1a2211201?, 0x0?, 0x0?, 0x55e4a22182a5?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc000064628 sp=0xc000064608 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.runfinq()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mfinal.go:193 +0x107 fp=0xc0000647e0 sp=0xc000064628 pc=0x55e4a21ee387
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0000647e8 sp=0xc0000647e0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by runtime.createfing in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mfinal.go:163 +0x3d
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 6 [GC worker (idle)]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc000066750 sp=0xc000066730 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gcBgMarkWorker()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:1293 +0xe5 fp=0xc0000667e0 sp=0xc000066750 pc=0x55e4a21f0ec5
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0000667e8 sp=0xc0000667e0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:1217 +0x1c
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 7 [GC worker (idle)]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0x25217569e10a1?, 0x0?, 0x0?, 0x0?, 0x0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc000066f50 sp=0xc000066f30 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gcBgMarkWorker()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:1293 +0xe5 fp=0xc000066fe0 sp=0xc000066f50 pc=0x55e4a21f0ec5
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000066fe8 sp=0xc000066fe0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:1217 +0x1c
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 18 [GC worker (idle)]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0x25217572d206e?, 0x0?, 0x0?, 0x0?, 0x0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc000060750 sp=0xc000060730 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gcBgMarkWorker()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:1293 +0xe5 fp=0xc0000607e0 sp=0xc000060750 pc=0x55e4a21f0ec5
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0000607e8 sp=0xc0000607e0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:1217 +0x1c
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 8 [GC worker (idle)]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0x25217572d1d50?, 0x0?, 0x0?, 0x0?, 0x0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc000067750 sp=0xc000067730 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gcBgMarkWorker()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:1293 +0xe5 fp=0xc0000677e0 sp=0xc000067750 pc=0x55e4a21f0ec5
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0000677e8 sp=0xc0000677e0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by runtime.gcBgMarkStartWorkers in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/mgc.go:1217 +0x1c
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 34 [select]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0xc000060f88?, 0x3?, 0xa0?, 0xb9?, 0xc000060f72?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc000060e18 sp=0xc000060df8 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.selectgo(0xc000060f88, 0xc000060f6c, 0xc0005a0880?, 0x0, 0x0?, 0x1)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/select.go:327 +0x725 fp=0xc000060f38 sp=0xc000060e18 pc=0x55e4a2220645
Nov 08 07:52:42 triback gitlab-workhorse[974708]: go.opencensus.io/stats/view.(*worker).start(0xc0005a0880)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc000060fc8 sp=0xc000060f38 pc=0x55e4a2601e5f
Nov 08 07:52:42 triback gitlab-workhorse[974708]: go.opencensus.io/stats/view.init.0.func1()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc000060fe0 sp=0xc000060fc8 pc=0x55e4a2601185
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000060fe8 sp=0xc000060fe0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by go.opencensus.io/stats/view.init.0 in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 449 [sleep]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0x25220a80a18c8?, 0x55e4a21db2d1?, 0xd7?, 0xa6?, 0xc000757260?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc0007aff50 sp=0xc0007aff30 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: time.Sleep(0x2540be400)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/time.go:195 +0x125 fp=0xc0007aff90 sp=0xc0007aff50 pc=0x55e4a2240245
Nov 08 07:52:42 triback gitlab-workhorse[974708]: gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream.(*upstream).pollGeoProxyAPI(0xc000004f00)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream/upstream.go:218 +0x6f fp=0xc0007affc8 sp=0xc0007aff90 pc=0x55e4a2ff67ef
Nov 08 07:52:42 triback gitlab-workhorse[974708]: gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream.newUpstream.func5()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream/upstream.go:101 +0x25 fp=0xc0007affe0 sp=0xc0007affc8 pc=0x55e4a2ff5f45
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0007affe8 sp=0xc0007affe0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream.newUpstream in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream/upstream.go:101 +0x533
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 177 [IO wait]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0xc000929c68?, 0x55e4a21e239d?, 0x70?, 0x9c?, 0x55e4a22b927d?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc000929c00 sp=0xc000929be0 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.netpollblock(0x7f63bcd4c208?, 0xa21d8b66?, 0xe4?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/netpoll.go:564 +0xf7 fp=0xc000929c38 sp=0xc000929c00 pc=0x55e4a2208a17
Nov 08 07:52:42 triback gitlab-workhorse[974708]: internal/poll.runtime_pollWait(0x7f63bcd40750, 0x72)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/netpoll.go:343 +0x85 fp=0xc000929c58 sp=0xc000929c38 pc=0x55e4a223d8c5
Nov 08 07:52:42 triback gitlab-workhorse[974708]: internal/poll.(*pollDesc).wait(0xc0005a0200?, 0xc000929d40?, 0x0)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000929c80 sp=0xc000929c58 pc=0x55e4a22b1ee7
Nov 08 07:52:42 triback gitlab-workhorse[974708]: internal/poll.(*pollDesc).waitRead(...)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: internal/poll/fd_poll_runtime.go:89
Nov 08 07:52:42 triback gitlab-workhorse[974708]: internal/poll.(*FD).Accept(0xc0005a0200)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: internal/poll/fd_unix.go:611 +0x2ac fp=0xc000929d28 sp=0xc000929c80 pc=0x55e4a22b73cc
Nov 08 07:52:42 triback gitlab-workhorse[974708]: net.(*netFD).accept(0xc0005a0200)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: net/fd_unix.go:172 +0x29 fp=0xc000929de0 sp=0xc000929d28 pc=0x55e4a23a4389
Nov 08 07:52:42 triback gitlab-workhorse[974708]: net.(*UnixListener).accept(0x55e4a2382f66?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: net/unixsock_posix.go:172 +0x16 fp=0xc000929e08 sp=0xc000929de0 pc=0x55e4a23c2d76
Nov 08 07:52:42 triback gitlab-workhorse[974708]: net.(*UnixListener).Accept(0xc000904150)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: net/unixsock.go:260 +0x30 fp=0xc000929e38 sp=0xc000929e08 pc=0x55e4a23c1810
Nov 08 07:52:42 triback gitlab-workhorse[974708]: net/http.(*onceCloseListener).Accept(0x55e4a3e2f230?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: <autogenerated>:1 +0x24 fp=0xc000929e50 sp=0xc000929e38 pc=0x55e4a24e3f44
Nov 08 07:52:42 triback gitlab-workhorse[974708]: net/http.(*Server).Serve(0xc0004c42d0, {0x55e4a3e2b5e0, 0xc000904150})
Nov 08 07:52:42 triback gitlab-workhorse[974708]: net/http/server.go:3056 +0x364 fp=0xc000929f80 sp=0xc000929e50 pc=0x55e4a24c07c4
Nov 08 07:52:42 triback gitlab-workhorse[974708]: main.run.func3({0x55e4a3e2b5e0?, 0xc000904150?})
Nov 08 07:52:42 triback gitlab-workhorse[974708]: gitlab.com/gitlab-org/gitlab/workhorse/main.go:291 +0x33 fp=0xc000929fc0 sp=0xc000929f80 pc=0x55e4a2ffc693
Nov 08 07:52:42 triback gitlab-workhorse[974708]: main.run.func11()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: gitlab.com/gitlab-org/gitlab/workhorse/main.go:291 +0x2b fp=0xc000929fe0 sp=0xc000929fc0 pc=0x55e4a2ffc62b
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000929fe8 sp=0xc000929fe0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by main.run in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: gitlab.com/gitlab-org/gitlab/workhorse/main.go:291 +0x1297
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 450 [select, locked to thread]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0xc0002f1fa8?, 0x2?, 0x29?, 0x3?, 0xc0002f1fa4?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc0002f1e38 sp=0xc0002f1e18 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.selectgo(0xc0002f1fa8, 0xc0002f1fa0, 0x0?, 0x0, 0x1?, 0x1)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/select.go:327 +0x725 fp=0xc0002f1f58 sp=0xc0002f1e38 pc=0x55e4a2220645
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.ensureSigM.func1()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/signal_unix.go:1014 +0x1a5 fp=0xc0002f1fe0 sp=0xc0002f1f58 pc=0x55e4a2239ea5
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0002f1fe8 sp=0xc0002f1fe0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by runtime.ensureSigM in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/signal_unix.go:997 +0xc8
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 451 [syscall]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.notetsleepg(0xffffffffffffffff?, 0xc0002f1728?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/lock_futex.go:236 +0x29 fp=0xc0002f17a0 sp=0xc0002f1768 pc=0x55e4a21e0ce9
Nov 08 07:52:42 triback gitlab-workhorse[974708]: os/signal.signal_recv()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/sigqueue.go:152 +0x29 fp=0xc0002f17c0 sp=0xc0002f17a0 pc=0x55e4a223f8c9
Nov 08 07:52:42 triback gitlab-workhorse[974708]: os/signal.loop()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: os/signal/signal_unix.go:23 +0x13 fp=0xc0002f17e0 sp=0xc0002f17c0 pc=0x55e4a2529693
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0002f17e8 sp=0xc0002f17e0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by os/signal.Notify.func1.1 in goroutine 1
Nov 08 07:52:42 triback gitlab-workhorse[974708]: os/signal/signal.go:151 +0x1f
Nov 08 07:52:42 triback gitlab-workhorse[974708]: goroutine 453 [chan receive]:
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.gopark(0xc000757560?, 0x55e4a21dadd5?, 0xd0?, 0x7e?, 0x0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/proc.go:398 +0xce fp=0xc000077ea0 sp=0xc000077e80 pc=0x55e4a221008e
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.chanrecv(0xc000312240, 0xc000077fa0, 0x1)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/chan.go:583 +0x3cd fp=0xc000077f18 sp=0xc000077ea0 pc=0x55e4a21db7ed
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.chanrecv2(0x0?, 0x0?)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/chan.go:447 +0x12 fp=0xc000077f40 sp=0xc000077f18 pc=0x55e4a21db412
Nov 08 07:52:42 triback gitlab-workhorse[974708]: github.com/getsentry/raven-go.(*Client).worker(0xc0004c41e0)
Nov 08 07:52:42 triback gitlab-workhorse[974708]: github.com/getsentry/raven-go@v0.2.0/client.go:546 +0x97 fp=0xc000077fc8 sp=0xc000077f40 pc=0x55e4a2adb757
Nov 08 07:52:42 triback gitlab-workhorse[974708]: github.com/getsentry/raven-go.(*Client).Capture.func1.1()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: github.com/getsentry/raven-go@v0.2.0/client.go:625 +0x25 fp=0xc000077fe0 sp=0xc000077fc8 pc=0x55e4a2adc245
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime.goexit()
Nov 08 07:52:42 triback gitlab-workhorse[974708]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x55e4a2243501
Nov 08 07:52:42 triback gitlab-workhorse[974708]: created by github.com/getsentry/raven-go.(*Client).Capture.func1 in goroutine 449
Nov 08 07:52:42 triback gitlab-workhorse[974708]: github.com/getsentry/raven-go@v0.2.0/client.go:625 +0x56
Nov 08 07:52:42 triback gitlab-workhorse[974708]: rax 0xca
Nov 08 07:52:42 triback gitlab-workhorse[974708]: rbx 0x0
Nov 08 07:52:42 triback gitlab-workhorse[974708]: rcx 0x55e4a2245343
Nov 08 07:52:42 triback gitlab-workhorse[974708]: rdx 0x0
Nov 08 07:52:42 triback gitlab-workhorse[974708]: rdi 0x55e4a495baa8
Nov 08 07:52:42 triback gitlab-workhorse[974708]: rsi 0x80
Nov 08 07:52:42 triback gitlab-workhorse[974708]: rbp 0x7fff05fad048
Nov 08 07:52:42 triback gitlab-workhorse[974708]: rsp 0x7fff05fad000
Nov 08 07:52:42 triback gitlab-workhorse[974708]: r8 0x0
Nov 08 07:52:42 triback gitlab-workhorse[974708]: r9 0x0
Nov 08 07:52:42 triback gitlab-workhorse[974708]: r10 0x0
Nov 08 07:52:42 triback gitlab-workhorse[974708]: r11 0x286
Nov 08 07:52:42 triback gitlab-workhorse[974708]: r12 0x0
Nov 08 07:52:42 triback gitlab-workhorse[974708]: r13 0xc000578000
Nov 08 07:52:42 triback gitlab-workhorse[974708]: r14 0x55e4a495b160
Nov 08 07:52:42 triback gitlab-workhorse[974708]: r15 0x2
Nov 08 07:52:42 triback gitlab-workhorse[974708]: rip 0x55e4a2245341
Nov 08 07:52:42 triback gitlab-workhorse[974708]: rflags 0x286
Nov 08 07:52:42 triback gitlab-workhorse[974708]: cs 0x33
Nov 08 07:52:42 triback gitlab-workhorse[974708]: fs 0x0
Nov 08 07:52:42 triback gitlab-workhorse[974708]: gs 0x0
Nov 08 07:52:42 triback systemd[1]: gitlab-workhorse.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 08 07:52:42 triback systemd[1]: gitlab-workhorse.service: Failed with result 'exit-code'.
Nov 08 07:52:42 triback systemd[1]: Stopped Gitlab Workhorse./run/gitlab/gitlab.socket gets created by gitlab-workhorse, shoudn't it just be waiting for incoming requests?
Last edited by wabi (2023-11-12 12:13:19)
Offline
Anybody any ideas how to get gitlab puma to provide any more information why it turns off?
- rails console works
- rake tasks work
Logs in /var/log/gitlab/
- puma.stdout.log : empty
- puma.stderr.log : empty
- production_json.log :
{"method":"GET","path":"/","format":"html","controller":"RootController","action":"index","status":302,"location":"http://example.org/users/sign_in","time":"2
023-11-19T13:07:14.958Z","params":[],"correlation_id":"fdd12a31-3089-45ab-9efb-8d0f8b135871","meta.caller_id":"RootController#index","meta.feature_category":"
groups_and_projects","meta.client_id":"ip/","request_urgency":"low","target_duration_s":5,"redis_calls":26,"redis_duration_s":0.00745,"redis_read_bytes":26,"r
edis_write_bytes":4279,"redis_feature_flag_calls":26,"redis_feature_flag_duration_s":0.00745,"redis_feature_flag_read_bytes":26,"redis_feature_flag_write_byte
s":4279,"db_count":13,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":13,"db_main_count":13,"db_ci_count":0,"db_main_replica_co
unt":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached
_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_coun
t":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_m
ain_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.006,"db_main_duration_s":0.006,"db_ci
_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":4.478509,"pid":1020376,"worker_id":"puma_master","rate_limiting_gates
":[],"db_duration_s":0.00305,"view_duration_s":0.0,"duration_s":0.03222}
- production.log:
Raven 3.1.2 configured not to capture errors: DSN not set
Offline
Pages: 1