You are not logged in.
After upgrading Gitlab I get issues with the service no longer running as expected, re-installing the gitlab package does not seem to help.
The major issues that I can find in my journal is:
Error with sidekiqctl:
nov 04 03:00:12 hostname gitlab-sidekiq[8430]: bundler: command not found: sidekiqctl
nov 04 03:00:12 hostname gitlab-sidekiq[8430]: Install missing gem executables with `bundle install`
It stops the services needed by Gitlab:
nov 04 03:00:10 hostname systemd[1]: gitlab-puma.service: Main process exited, code=exited, status=1/FAILURE
nov 04 03:00:10 hostname systemd[1]: gitlab-puma.service: Failed with result 'exit-code'.
nov 04 03:00:10 hostname systemd[1]: gitlab-puma.service: Consumed 31.084s CPU time.
nov 04 03:00:11 hostname systemd[1]: gitlab-puma.service: Scheduled restart job, restart counter is at 67.
nov 04 03:00:11 hostname systemd[1]: Stopped target GitLab - Self Hosted Git Management.
nov 04 03:00:11 hostname systemd[1]: Stopping GitLab - Self Hosted Git Management...
nov 04 03:00:11 hostname systemd[1]: Stopping Gitlab Mailroom Worker...
nov 04 03:00:11 hostname systemd[1]: Stopping GitLab Sidekiq Worker...
nov 04 03:00:11 hostname systemd[1]: Stopping Gitlab Workhorse...
nov 04 03:00:12 hostname systemd[1]: Started Process Core Dump (PID 8433/UID 0).
What is wrong, and how can I fix this to have Gitlab running again?
Below is the full log
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.420Z","message":"Cron Jobs - added job with name: authorized_project_update_periodic_recalculate_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.422Z","message":"Cron Jobs - added job with name: update_container_registry_info_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.423Z","message":"Cron Jobs - added job with name: postgres_dynamic_partitions_manager"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.425Z","message":"Cron Jobs - added job with name: postgres_dynamic_partitions_dropper"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.427Z","message":"Cron Jobs - added job with name: analytics_usage_trends_count_job_trigger_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.428Z","message":"Cron Jobs - added job with name: member_invitation_reminder_emails_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.430Z","message":"Cron Jobs - added job with name: schedule_merge_request_cleanup_refs_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.431Z","message":"Cron Jobs - added job with name: manage_evidence_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.432Z","message":"Cron Jobs - added job with name: user_status_cleanup_batch_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.434Z","message":"Cron Jobs - added job with name: ssh_keys_expired_notification_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.436Z","message":"Cron Jobs - added job with name: ssh_keys_expiring_soon_notification_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.437Z","message":"Cron Jobs - added job with name: users_deactivate_dormant_users_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.439Z","message":"Cron Jobs - added job with name: ci_delete_unit_tests_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.440Z","message":"Cron Jobs - added job with name: batched_background_migrations_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.442Z","message":"Cron Jobs - added job with name: batched_background_migration_worker_ci_database"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.443Z","message":"Cron Jobs - added job with name: issues_reschedule_stuck_issue_rebalances"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.445Z","message":"Cron Jobs - added job with name: projects_schedule_refresh_build_artifacts_size_statistics_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.446Z","message":"Cron Jobs - added job with name: inactive_projects_deletion_cron_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.448Z","message":"Cron Jobs - added job with name: loose_foreign_keys_cleanup_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.449Z","message":"Cron Jobs - added job with name: batched_git_ref_updates_cleanup_scheduler_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.451Z","message":"Cron Jobs - added job with name: users_migrate_records_to_ghost_user_in_batches_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.452Z","message":"Cron Jobs - added job with name: packages_cleanup_delete_orphaned_dependencies_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.454Z","message":"Cron Jobs - added job with name: cleanup_dangling_debian_package_files_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.455Z","message":"Cron Jobs - added job with name: object_storage_delete_stale_direct_uploads_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.457Z","message":"Cron Jobs - added job with name: service_desk_custom_email_verification_cleanup"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.458Z","message":"Cron Jobs - added job with name: ensure_merge_requests_prepared_worker"}
nov 04 02:59:54 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:54.460Z","message":"Cron Jobs - added job with name: deactivated_pages_deployments_delete_cron_worker"}
nov 04 02:59:55 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T01:59:55.774Z","message":"stopped","memwd_reason":"monitors are not configured","memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::SidekiqHandler","memwd_sleep_time_s":3,"pid":8269,"worker_id":"sidekiq","memwd_rss_bytes":455094272,"retry":0}
nov 04 02:59:58 hostname gitlab-workhorse[8270]: time="2023-11-04T02:59:58+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /run/gitlab/gitlab.socket: connect: connection refused" method=GET uri=
nov 04 03:00:00 hostname systemd[1]: Started Timeline of Snapper Snapshots.
nov 04 03:00:00 hostname dbus-daemon[627]: [system] Activating via systemd: service name='org.opensuse.Snapper' unit='snapperd.service' requested by ':1.244' (uid=0 pid=8394 comm="/usr/lib/snapper/systemd-helper --timeline")
nov 04 03:00:00 hostname systemd[1]: Starting DBus interface for snapper...
nov 04 03:00:00 hostname dbus-daemon[627]: [system] Successfully activated service 'org.opensuse.Snapper'
nov 04 03:00:00 hostname systemd[1]: Started DBus interface for snapper.
nov 04 03:00:00 hostname systemd-helper[8394]: running timeline for 'storage'.
nov 04 03:00:01 hostname systemd[1]: snapper-timeline.service: Deactivated successfully.
nov 04 03:00:07 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T02:00:07.456Z","message":"Booted Rails 7.0.8 application in production environment"}
nov 04 03:00:07 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T02:00:07.456Z","message":"Running in ruby 3.0.6p216 (2023-03-30 revision 23a532679b) [x86_64-linux]"}
nov 04 03:00:07 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T02:00:07.456Z","message":"See LICENSE and the LGPL-3.0 for licensing details."}
nov 04 03:00:07 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T02:00:07.456Z","message":"Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org"}
nov 04 03:00:08 hostname gitlab-workhorse[8270]: time="2023-11-04T03:00:08+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /run/gitlab/gitlab.socket: connect: connection refused" method=GET uri=
nov 04 03:00:10 hostname gitlab-puma[8267]: {"timestamp":"2023-11-04T02:00:10.786Z","pid":8267,"message":"* Listening on unix:///run/gitlab/gitlab.socket"}
nov 04 03:00:10 hostname gitlab-puma[8267]: {"timestamp":"2023-11-04T02:00:10.786Z","pid":8267,"message":"! WARNING: Detected 2 Thread(s) started in app boot:"}
nov 04 03:00:10 hostname gitlab-puma[8267]: {"timestamp":"2023-11-04T02:00:10.786Z","pid":8267,"message":"! #\u003cThread:0x0000559be3efac70 /usr/share/webapps/gitlab/vendor/bundle/ruby/3.0.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/scheduler.rb:73 sleep\u003e - /usr/share/webapps/gitlab/vendor/bundle/ruby/3.0.0/gems/rack-timeout-0.6.3/lib/rack/timeout/support/scheduler.rb:91:in `sleep'"}
nov 04 03:00:10 hostname gitlab-puma[8267]: {"timestamp":"2023-11-04T02:00:10.786Z","pid":8267,"message":"! #\u003cThread:0x0000559be56c7a60 /usr/share/webapps/gitlab/vendor/bundle/ruby/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/session_flusher.rb:81 sleep\u003e - /usr/share/webapps/gitlab/vendor/bundle/ruby/3.0.0/gems/sentry-ruby-5.8.0/lib/sentry/session_flusher.rb:83:in `sleep'"}
nov 04 03:00:10 hostname gitlab-puma[8267]: {"timestamp":"2023-11-04T02:00:10.786Z","pid":8267,"message":"Use Ctrl-C to stop"}
nov 04 03:00:10 hostname systemd[1]: gitlab-puma.service: Main process exited, code=exited, status=1/FAILURE
nov 04 03:00:10 hostname systemd[1]: gitlab-puma.service: Failed with result 'exit-code'.
nov 04 03:00:10 hostname systemd[1]: gitlab-puma.service: Consumed 31.084s CPU time.
nov 04 03:00:11 hostname systemd[1]: gitlab-puma.service: Scheduled restart job, restart counter is at 67.
nov 04 03:00:11 hostname systemd[1]: Stopped target GitLab - Self Hosted Git Management.
nov 04 03:00:11 hostname systemd[1]: Stopping GitLab - Self Hosted Git Management...
nov 04 03:00:11 hostname systemd[1]: Stopping Gitlab Mailroom Worker...
nov 04 03:00:11 hostname systemd[1]: Stopping GitLab Sidekiq Worker...
nov 04 03:00:11 hostname systemd[1]: Stopping Gitlab Workhorse...
nov 04 03:00:12 hostname systemd[1]: Started Process Core Dump (PID 8433/UID 0).
nov 04 03:00:12 hostname gitlab-workhorse[8270]: SIGQUIT: quit
nov 04 03:00:12 hostname gitlab-workhorse[8270]: PC=0x563b85b48341 m=0 sigcode=0
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 0 [idle]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.futex()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/sys_linux_amd64.s:557 +0x21 fp=0x7ffed1bca1c0 sp=0x7ffed1bca1b8 pc=0x563b85b48341
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.futexsleep(0x7ffed1bca238?, 0x85b174b3?, 0x7ffed1bca238?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/os_linux.go:69 +0x30 fp=0x7ffed1bca210 sp=0x7ffed1bca1c0 pc=0x563b85b0c650
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.notesleep(0x563b8825eaa8)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/lock_futex.go:160 +0x87 fp=0x7ffed1bca248 sp=0x7ffed1bca210 pc=0x563b85ae3a07
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.mPark(...)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:1632
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.stoplockedm()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:2780 +0x73 fp=0x7ffed1bca2a0 sp=0x7ffed1bca248 pc=0x563b85b176d3
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.schedule()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:3561 +0x3a fp=0x7ffed1bca2d8 sp=0x7ffed1bca2a0 pc=0x563b85b1951a
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.park_m(0xc000583040?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:3745 +0x11f fp=0x7ffed1bca320 sp=0x7ffed1bca2d8 pc=0x563b85b19a9f
nov 04 03:00:12 hostname gitlab-workhorse[8270]: traceback: unexpected SPWRITE function runtime.mcall
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.mcall()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:458 +0x50 fp=0x7ffed1bca338 sp=0x7ffed1bca320 pc=0x563b85b444f0
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 1 [runnable]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0xc0008c1768?, 0x2?, 0x45?, 0x9e?, 0xc0008c14bc?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc0008c1198 sp=0xc0008c1178 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.selectgo(0xc0008c1768, 0xc0008c14b8, 0x0?, 0x0, 0x0?, 0x1)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/select.go:327 +0x725 fp=0xc0008c12b8 sp=0xc0008c1198 pc=0x563b85b23645
nov 04 03:00:12 hostname gitlab-workhorse[8270]: main.run({{0x563b8692bc83, 0x1a}, {0x7ffed1bcbdb3, 0x23}, {0x7ffed1bcbda2, 0x4}, 0x0, {0x0, 0x0}, {0x0, ...}, ...}, ...)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: gitlab.com/gitlab-org/gitlab/workhorse/main.go:294 +0x144d fp=0xc0008c1ab0 sp=0xc0008c12b8 pc=0x563b868ff2ed
nov 04 03:00:12 hostname gitlab-workhorse[8270]: main.main()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: gitlab.com/gitlab-org/gitlab/workhorse/main.go:68 +0x3b8 fp=0xc0008c1f40 sp=0xc0008c1ab0 pc=0x563b868fccb8
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.main()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:267 +0x2d2 fp=0xc0008c1fe0 sp=0xc0008c1f40 pc=0x563b85b12c12
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0008c1fe8 sp=0xc0008c1fe0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 2 [force gc (idle)]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000070fa8 sp=0xc000070f88 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goparkunlock(...)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:404
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.forcegchelper()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:322 +0xb8 fp=0xc000070fe0 sp=0xc000070fa8 pc=0x563b85b12ef8
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000070fe8 sp=0xc000070fe0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.init.6 in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:310 +0x1a
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 3 [GC sweep wait]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000071778 sp=0xc000071758 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goparkunlock(...)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:404
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.bgsweep(0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgcsweep.go:321 +0xdf fp=0xc0000717c8 sp=0xc000071778 pc=0x563b85afd1ff
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gcenable.func1()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:200 +0x25 fp=0xc0000717e0 sp=0xc0000717c8 pc=0x563b85af2305
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0000717e8 sp=0xc0000717e0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.gcenable in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:200 +0x66
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 4 [GC scavenge wait]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0xc00009a000?, 0x563b86f41138?, 0x0?, 0x0?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000071f70 sp=0xc000071f50 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goparkunlock(...)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:404
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.(*scavengerState).park(0x563b8825db60)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgcscavenge.go:425 +0x49 fp=0xc000071fa0 sp=0xc000071f70 pc=0x563b85afaa49
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.bgscavenge(0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgcscavenge.go:658 +0x59 fp=0xc000071fc8 sp=0xc000071fa0 pc=0x563b85afaff9
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gcenable.func2()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:201 +0x25 fp=0xc000071fe0 sp=0xc000071fc8 pc=0x563b85af22a5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000071fe8 sp=0xc000071fe0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.gcenable in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:201 +0xa5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 18 [finalizer wait]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x563b87700c80?, 0x185b14201?, 0x0?, 0x0?, 0x563b85b1b2a5?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000070628 sp=0xc000070608 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.runfinq()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mfinal.go:193 +0x107 fp=0xc0000707e0 sp=0xc000070628 pc=0x563b85af1387
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0000707e8 sp=0xc0000707e0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.createfing in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mfinal.go:163 +0x3d
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 19 [GC worker (idle)]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc00006c750 sp=0xc00006c730 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gcBgMarkWorker()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1293 +0xe5 fp=0xc00006c7e0 sp=0xc00006c750 pc=0x563b85af3ec5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc00006c7e8 sp=0xc00006c7e0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.gcBgMarkStartWorkers in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1217 +0x1c
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 20 [GC worker (idle)]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc00006cf50 sp=0xc00006cf30 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gcBgMarkWorker()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1293 +0xe5 fp=0xc00006cfe0 sp=0xc00006cf50 pc=0x563b85af3ec5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc00006cfe8 sp=0xc00006cfe0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.gcBgMarkStartWorkers in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1217 +0x1c
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 21 [GC worker (idle)]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x328f9668141?, 0x3?, 0x38?, 0x68?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc00006d750 sp=0xc00006d730 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gcBgMarkWorker()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1293 +0xe5 fp=0xc00006d7e0 sp=0xc00006d750 pc=0x563b85af3ec5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc00006d7e8 sp=0xc00006d7e0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.gcBgMarkStartWorkers in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1217 +0x1c
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 5 [GC worker (idle)]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x563b8828fd20?, 0x1?, 0x1c?, 0x67?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000072750 sp=0xc000072730 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gcBgMarkWorker()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1293 +0xe5 fp=0xc0000727e0 sp=0xc000072750 pc=0x563b85af3ec5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0000727e8 sp=0xc0000727e0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.gcBgMarkStartWorkers in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1217 +0x1c
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 34 [GC worker (idle)]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x563b8828fd20?, 0x3?, 0xc8?, 0xae?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000508750 sp=0xc000508730 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gcBgMarkWorker()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1293 +0xe5 fp=0xc0005087e0 sp=0xc000508750 pc=0x563b85af3ec5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0005087e8 sp=0xc0005087e0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.gcBgMarkStartWorkers in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1217 +0x1c
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 6 [GC worker (idle)]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x328f966f94c?, 0x3?, 0xbb?, 0x5?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000072f50 sp=0xc000072f30 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gcBgMarkWorker()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1293 +0xe5 fp=0xc000072fe0 sp=0xc000072f50 pc=0x563b85af3ec5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000072fe8 sp=0xc000072fe0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.gcBgMarkStartWorkers in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1217 +0x1c
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 35 [GC worker (idle)]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x328f966f94c?, 0x0?, 0x0?, 0x0?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000508f50 sp=0xc000508f30 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gcBgMarkWorker()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1293 +0xe5 fp=0xc000508fe0 sp=0xc000508f50 pc=0x563b85af3ec5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000508fe8 sp=0xc000508fe0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.gcBgMarkStartWorkers in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1217 +0x1c
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 7 [GC worker (idle)]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x328f966f94c?, 0x0?, 0x0?, 0x0?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000073750 sp=0xc000073730 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gcBgMarkWorker()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1293 +0xe5 fp=0xc0000737e0 sp=0xc000073750 pc=0x563b85af3ec5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc0000737e8 sp=0xc0000737e0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.gcBgMarkStartWorkers in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/mgc.go:1217 +0x1c
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 8 [select]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0xc00050b788?, 0x3?, 0x40?, 0x7f?, 0xc00050b772?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc00050b618 sp=0xc00050b5f8 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.selectgo(0xc00050b788, 0xc00050b76c, 0xc0000c0680?, 0x0, 0x0?, 0x1)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/select.go:327 +0x725 fp=0xc00050b738 sp=0xc00050b618 pc=0x563b85b23645
nov 04 03:00:12 hostname gitlab-workhorse[8270]: go.opencensus.io/stats/view.(*worker).start(0xc0000c0680)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f fp=0xc00050b7c8 sp=0xc00050b738 pc=0x563b85f04e5f
nov 04 03:00:12 hostname gitlab-workhorse[8270]: go.opencensus.io/stats/view.init.0.func1()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x25 fp=0xc00050b7e0 sp=0xc00050b7c8 pc=0x563b85f04185
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc00050b7e8 sp=0xc00050b7e0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by go.opencensus.io/stats/view.init.0 in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 68 [syscall]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.notetsleepg(0xffffffffffffffff?, 0xc000825f28?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/lock_futex.go:236 +0x29 fp=0xc000825fa0 sp=0xc000825f68 pc=0x563b85ae3ce9
nov 04 03:00:12 hostname gitlab-workhorse[8270]: os/signal.signal_recv()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/sigqueue.go:152 +0x29 fp=0xc000825fc0 sp=0xc000825fa0 pc=0x563b85b428c9
nov 04 03:00:12 hostname gitlab-workhorse[8270]: os/signal.loop()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: os/signal/signal_unix.go:23 +0x13 fp=0xc000825fe0 sp=0xc000825fc0 pc=0x563b85e2c693
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000825fe8 sp=0xc000825fe0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by os/signal.Notify.func1.1 in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: os/signal/signal.go:151 +0x1f
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 470 [sleep]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0x32ff602b53a?, 0x563b85ade2d1?, 0xd7?, 0xd6?, 0xc0006e20c0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000713f50 sp=0xc000713f30 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: time.Sleep(0x2540be400)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/time.go:195 +0x125 fp=0xc000713f90 sp=0xc000713f50 pc=0x563b85b43245
nov 04 03:00:12 hostname gitlab-workhorse[8270]: gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream.(*upstream).pollGeoProxyAPI(0xc000004900)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream/upstream.go:218 +0x6f fp=0xc000713fc8 sp=0xc000713f90 pc=0x563b868f97ef
nov 04 03:00:12 hostname gitlab-workhorse[8270]: gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream.newUpstream.func5()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream/upstream.go:101 +0x25 fp=0xc000713fe0 sp=0xc000713fc8 pc=0x563b868f8f45
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000713fe8 sp=0xc000713fe0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream.newUpstream in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: gitlab.com/gitlab-org/gitlab/workhorse/internal/upstream/upstream.go:101 +0x533
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 471 [select, locked to thread]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0xc000509fa8?, 0x2?, 0x29?, 0x33?, 0xc000509fa4?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000509e38 sp=0xc000509e18 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.selectgo(0xc000509fa8, 0xc000509fa0, 0x0?, 0x0, 0x1?, 0x1)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/select.go:327 +0x725 fp=0xc000509f58 sp=0xc000509e38 pc=0x563b85b23645
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.ensureSigM.func1()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/signal_unix.go:1014 +0x1a5 fp=0xc000509fe0 sp=0xc000509f58 pc=0x563b85b3cea5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000509fe8 sp=0xc000509fe0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by runtime.ensureSigM in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/signal_unix.go:997 +0xc8
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 37 [chan receive]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0xc000702180?, 0x563b85adddd5?, 0xd0?, 0xe?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000080ea0 sp=0xc000080e80 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.chanrecv(0xc000525080, 0xc000080fa0, 0x1)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/chan.go:583 +0x3cd fp=0xc000080f18 sp=0xc000080ea0 pc=0x563b85ade7ed
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.chanrecv2(0x0?, 0x0?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/chan.go:447 +0x12 fp=0xc000080f40 sp=0xc000080f18 pc=0x563b85ade412
nov 04 03:00:12 hostname gitlab-workhorse[8270]: github.com/getsentry/raven-go.(*Client).worker(0xc0004c41e0)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: github.com/getsentry/raven-go@v0.2.0/client.go:546 +0x97 fp=0xc000080fc8 sp=0xc000080f40 pc=0x563b863de757
nov 04 03:00:12 hostname gitlab-workhorse[8270]: github.com/getsentry/raven-go.(*Client).Capture.func1.1()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: github.com/getsentry/raven-go@v0.2.0/client.go:625 +0x25 fp=0xc000080fe0 sp=0xc000080fc8 pc=0x563b863df245
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000080fe8 sp=0xc000080fe0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by github.com/getsentry/raven-go.(*Client).Capture.func1 in goroutine 470
nov 04 03:00:12 hostname gitlab-workhorse[8270]: github.com/getsentry/raven-go@v0.2.0/client.go:625 +0x56
nov 04 03:00:12 hostname gitlab-workhorse[8270]: goroutine 69 [IO wait]:
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.gopark(0xc000085c68?, 0x563b85ae539d?, 0x70?, 0x5c?, 0x563b85bbc27d?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/proc.go:398 +0xce fp=0xc000085c00 sp=0xc000085be0 pc=0x563b85b1308e
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.netpollblock(0x7f2f784f7a50?, 0x85adbb66?, 0x3b?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/netpoll.go:564 +0xf7 fp=0xc000085c38 sp=0xc000085c00 pc=0x563b85b0ba17
nov 04 03:00:12 hostname gitlab-workhorse[8270]: internal/poll.runtime_pollWait(0x7f2f7a0696d0, 0x72)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/netpoll.go:343 +0x85 fp=0xc000085c58 sp=0xc000085c38 pc=0x563b85b408c5
nov 04 03:00:12 hostname gitlab-workhorse[8270]: internal/poll.(*pollDesc).wait(0xc0001b0180?, 0xc000085d40?, 0x0)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc000085c80 sp=0xc000085c58 pc=0x563b85bb4ee7
nov 04 03:00:12 hostname gitlab-workhorse[8270]: internal/poll.(*pollDesc).waitRead(...)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: internal/poll/fd_poll_runtime.go:89
nov 04 03:00:12 hostname gitlab-workhorse[8270]: internal/poll.(*FD).Accept(0xc0001b0180)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: internal/poll/fd_unix.go:611 +0x2ac fp=0xc000085d28 sp=0xc000085c80 pc=0x563b85bba3cc
nov 04 03:00:12 hostname gitlab-workhorse[8270]: net.(*netFD).accept(0xc0001b0180)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: net/fd_unix.go:172 +0x29 fp=0xc000085de0 sp=0xc000085d28 pc=0x563b85ca7389
nov 04 03:00:12 hostname gitlab-workhorse[8270]: net.(*UnixListener).accept(0x563b85c85f66?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: net/unixsock_posix.go:172 +0x16 fp=0xc000085e08 sp=0xc000085de0 pc=0x563b85cc5d76
nov 04 03:00:12 hostname gitlab-workhorse[8270]: net.(*UnixListener).Accept(0xc000780150)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: net/unixsock.go:260 +0x30 fp=0xc000085e38 sp=0xc000085e08 pc=0x563b85cc4810
nov 04 03:00:12 hostname gitlab-workhorse[8270]: net/http.(*onceCloseListener).Accept(0x563b87732230?)
nov 04 03:00:12 hostname gitlab-workhorse[8270]: <autogenerated>:1 +0x24 fp=0xc000085e50 sp=0xc000085e38 pc=0x563b85de6f44
nov 04 03:00:12 hostname gitlab-workhorse[8270]: net/http.(*Server).Serve(0xc0004c42d0, {0x563b8772e5e0, 0xc000780150})
nov 04 03:00:12 hostname gitlab-workhorse[8270]: net/http/server.go:3056 +0x364 fp=0xc000085f80 sp=0xc000085e50 pc=0x563b85dc37c4
nov 04 03:00:12 hostname gitlab-workhorse[8270]: main.run.func3({0x563b8772e5e0?, 0xc000780150?})
nov 04 03:00:12 hostname gitlab-workhorse[8270]: gitlab.com/gitlab-org/gitlab/workhorse/main.go:291 +0x33 fp=0xc000085fc0 sp=0xc000085f80 pc=0x563b868ff693
nov 04 03:00:12 hostname gitlab-workhorse[8270]: main.run.func11()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: gitlab.com/gitlab-org/gitlab/workhorse/main.go:291 +0x2b fp=0xc000085fe0 sp=0xc000085fc0 pc=0x563b868ff62b
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime.goexit()
nov 04 03:00:12 hostname gitlab-workhorse[8270]: runtime/asm_amd64.s:1650 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x563b85b46501
nov 04 03:00:12 hostname gitlab-workhorse[8270]: created by main.run in goroutine 1
nov 04 03:00:12 hostname gitlab-workhorse[8270]: gitlab.com/gitlab-org/gitlab/workhorse/main.go:291 +0x1297
nov 04 03:00:12 hostname gitlab-workhorse[8270]: rax 0xca
nov 04 03:00:12 hostname gitlab-workhorse[8270]: rbx 0x0
nov 04 03:00:12 hostname gitlab-workhorse[8270]: rcx 0x563b85b48343
nov 04 03:00:12 hostname gitlab-workhorse[8270]: rdx 0x0
nov 04 03:00:12 hostname gitlab-workhorse[8270]: rdi 0x563b8825eaa8
nov 04 03:00:12 hostname gitlab-workhorse[8270]: rsi 0x80
nov 04 03:00:12 hostname gitlab-workhorse[8270]: rbp 0x7ffed1bca200
nov 04 03:00:12 hostname gitlab-workhorse[8270]: rsp 0x7ffed1bca1b8
nov 04 03:00:12 hostname gitlab-workhorse[8270]: r8 0x0
nov 04 03:00:12 hostname gitlab-workhorse[8270]: r9 0x0
nov 04 03:00:12 hostname gitlab-workhorse[8270]: r10 0x0
nov 04 03:00:12 hostname gitlab-workhorse[8270]: r11 0x286
nov 04 03:00:12 hostname gitlab-workhorse[8270]: r12 0x7ffed1bca1d8
nov 04 03:00:12 hostname gitlab-workhorse[8270]: r13 0xc000509fa4
nov 04 03:00:12 hostname gitlab-workhorse[8270]: r14 0x563b8825e160
nov 04 03:00:12 hostname gitlab-workhorse[8270]: r15 0x0
nov 04 03:00:12 hostname gitlab-workhorse[8270]: rip 0x563b85b48341
nov 04 03:00:12 hostname gitlab-workhorse[8270]: rflags 0x286
nov 04 03:00:12 hostname gitlab-workhorse[8270]: cs 0x33
nov 04 03:00:12 hostname gitlab-workhorse[8270]: fs 0x0
nov 04 03:00:12 hostname gitlab-workhorse[8270]: gs 0x0
nov 04 03:00:12 hostname systemd[1]: gitlab-workhorse.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
nov 04 03:00:12 hostname systemd[1]: gitlab-workhorse.service: Failed with result 'exit-code'.
nov 04 03:00:12 hostname systemd[1]: Stopped Gitlab Workhorse.
nov 04 03:00:12 hostname gitlab-sidekiq[8430]: bundler: command not found: sidekiqctl
nov 04 03:00:12 hostname gitlab-sidekiq[8430]: Install missing gem executables with `bundle install`
nov 04 03:00:12 hostname systemd[1]: gitlab-sidekiq.service: Control process exited, code=exited, status=127/n/a
nov 04 03:00:12 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T02:00:12.247Z","message":"Shutting down"}
nov 04 03:00:12 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T02:00:12.248Z","message":"Scheduler exiting..."}
nov 04 03:00:12 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T02:00:12.248Z","message":"Terminating quiet threads"}
nov 04 03:00:12 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T02:00:12.248Z","message":"Scheduler exiting..."}
nov 04 03:00:12 hostname systemd-coredump[8434]: [?] Process 8268 (bundle) of user 105 dumped core.
Stack trace of thread 8268:
#0 0x00007f260c9bf83c n/a (libc.so.6 + 0x8e83c)
#1 0x00007f260c96f668 raise (libc.so.6 + 0x3e668)
#2 0x00007f260ccb2d18 n/a (libruby.so.3.0 + 0xb2d18)
#3 0x00007f260ccb2f15 ruby_run_node (libruby.so.3.0 + 0xb2f15)
#4 0x0000556be1cbe085 n/a (ruby + 0x1085)
#5 0x00007f260c958cd0 n/a (libc.so.6 + 0x27cd0)
#6 0x00007f260c958d8a __libc_start_main (libc.so.6 + 0x27d8a)
#7 0x0000556be1cbe0c5 _start (ruby + 0x10c5)
ELF object binary architecture: AMD x86-64
nov 04 03:00:12 hostname systemd[1]: systemd-coredump@53-8433-0.service: Deactivated successfully.
nov 04 03:00:12 hostname systemd[1]: gitlab-mailroom.service: Main process exited, code=dumped, status=3/QUIT
nov 04 03:00:12 hostname systemd[1]: gitlab-mailroom.service: Failed with result 'core-dump'.
nov 04 03:00:12 hostname systemd[1]: Stopped Gitlab Mailroom Worker.
nov 04 03:00:12 hostname systemd[1]: gitlab-mailroom.service: Consumed 1.325s CPU time.
nov 04 03:00:12 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T02:00:12.749Z","message":"Pausing to allow jobs to finish..."}
nov 04 03:00:13 hostname gitlab-sidekiq[8269]: {"severity":"INFO","time":"2023-11-04T02:00:13.752Z","message":"Bye!"}
nov 04 03:00:13 hostname systemd[1]: gitlab-sidekiq.service: Failed with result 'exit-code'.
nov 04 03:00:13 hostname systemd[1]: Stopped GitLab Sidekiq Worker.
nov 04 03:00:13 hostname systemd[1]: gitlab-sidekiq.service: Consumed 28.084s CPU time.
nov 04 03:00:13 hostname systemd[1]: Started GitLab Puma Server.
nov 04 03:00:13 hostname systemd[1]: Started Gitlab Mailroom Worker.
nov 04 03:00:13 hostname systemd[1]: Started GitLab Sidekiq Worker.
nov 04 03:00:13 hostname systemd[1]: Started Gitlab Workhorse.
nov 04 03:00:13 hostname systemd[1]: Reached target GitLab - Self Hosted Git Management.
nov 04 03:00:13 hostname gitlab-workhorse[8455]: time="2023-11-04T03:00:13+01:00" level=info msg=Starting build_time=20231025.225211 version=v16.5.0
nov 04 03:00:13 hostname gitlab-workhorse[8455]: time="2023-11-04T03:00:13+01:00" level=info msg="Using gomodule/redigo"
nov 04 03:00:13 hostname gitlab-workhorse[8455]: time="2023-11-04T03:00:13+01:00" level=info msg="Running upstream server" address=/run/gitlab/gitlab-workhorse.socket network=unix
nov 04 03:00:13 hostname gitlab-workhorse[8455]: time="2023-11-04T03:00:13+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /run/gitlab/gitlab.socket: connect: connection refused" method=GET uri=
nov 04 03:00:15 hostname gitlab-puma[8452]: {"timestamp":"2023-11-04T02:00:15.600Z","pid":8452,"message":"Puma starting in cluster mode..."}
nov 04 03:00:15 hostname gitlab-puma[8452]: {"timestamp":"2023-11-04T02:00:15.608Z","pid":8452,"message":"* Puma version: 6.4.0 (ruby 3.0.6-p216) (\"The Eagle of Durango\")"}
nov 04 03:00:15 hostname gitlab-puma[8452]: {"timestamp":"2023-11-04T02:00:15.608Z","pid":8452,"message":"* Min threads: 1"}
nov 04 03:00:15 hostname gitlab-puma[8452]: {"timestamp":"2023-11-04T02:00:15.608Z","pid":8452,"message":"* Max threads: 16"}
nov 04 03:00:15 hostname gitlab-puma[8452]: {"timestamp":"2023-11-04T02:00:15.608Z","pid":8452,"message":"* Environment: production"}
nov 04 03:00:15 hostname gitlab-puma[8452]: {"timestamp":"2023-11-04T02:00:15.609Z","pid":8452,"message":"* Master PID: 8452"}
nov 04 03:00:15 hostname gitlab-puma[8452]: {"timestamp":"2023-11-04T02:00:15.609Z","pid":8452,"message":"* Workers: 3"}
nov 04 03:00:15 hostname gitlab-puma[8452]: {"timestamp":"2023-11-04T02:00:15.609Z","pid":8452,"message":"* Restarts: (✔) hot (✖) phased"}
nov 04 03:00:15 hostname gitlab-puma[8452]: {"timestamp":"2023-11-04T02:00:15.609Z","pid":8452,"message":"* Preloading application"}
nov 04 03:00:16 hostname gitlab-runner[697]: WARNING: Checking for jobs... failed runner=stcos33b status=502 Bad Gateway
nov 04 03:00:16 hostname gitlab-workhorse[8455]: time="2023-11-04T03:00:16+01:00" level=error correlation_id=01HEC0YEJ09ZN71H8J29RJNT44 duration_ms=0 error="badgateway: failed to receive response: dial unix /run/gitlab/gitlab.socket: connect: connection refused" method=POST uri=/api/v4/jobs/request
nov 04 03:00:16 hostname gitlab-workhorse[8455]: ci.goget.dev 127.0.0.1 - - [2023/11/04:03:00:16 +0100] "POST /api/v4/jobs/request HTTP/1.0" 502 24 "" "gitlab-runner 16.5.0 (16-5-stable; go1.21.3; linux/amd64)" 0
nov 04 03:00:23 hostname gitlab-workhorse[8455]: time="2023-11-04T03:00:23+01:00" level=error correlation_id= duration_ms=0 error="badgateway: failed to receive response: dial unix /run/gitlab/gitlab.socket: connect: connection refused" method=GET uri=
nov 04 03:00:24 hostname gitlab-puma[8452]: WARNING: The Rust extension for prometheus-client-mmap is unavailable, falling back to the legacy C extension.
nov 04 03:00:24 hostname gitlab-puma[8452]: The Rust extension will be required in the next version. If you are compiling this gem from source,
nov 04 03:00:24 hostname gitlab-puma[8452]: ensure your build system has a Rust compiler and clang: https://gitlab.com/gitlab-org/ruby/gems/prometheus-client-mmap
nov 04 03:00:24 hostname gitlab-sidekiq[8454]: WARNING: The Rust extension for prometheus-client-mmap is unavailable, falling back to the legacy C extension.
nov 04 03:00:24 hostname gitlab-sidekiq[8454]: The Rust extension will be required in the next version. If you are compiling this gem from source,
nov 04 03:00:24 hostname gitlab-sidekiq[8454]: ensure your build system has a Rust compiler and clang: https://gitlab.com/gitlab-org/ruby/gems/prometheus-client-mmap
nov 04 03:00:25 hostname sshd[8475]: Failed password for root from 31.41.244.62 port 32246 ssh2
nov 04 03:00:26 hostname gitlab-sidekiq[8454]: /usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: already initialized constant Gitlab::Instrumentation::Redis::ActionCable
nov 04 03:00:26 hostname gitlab-sidekiq[8454]: /usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: previous definition of ActionCable was here
nov 04 03:00:26 hostname sshd[8475]: Received disconnect from 31.41.244.62 port 32246:11: Bye Bye [preauth]
nov 04 03:00:26 hostname sshd[8475]: Disconnected from authenticating user root 31.41.244.62 port 32246 [preauth]
nov 04 03:00:26 hostname gitlab-puma[8452]: /usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: already initialized constant Gitlab::Instrumentation::Redis::ActionCable
nov 04 03:00:26 hostname gitlab-puma[8452]: /usr/share/webapps/gitlab/lib/gitlab/instrumentation/redis.rb:9: warning: previous definition of ActionCable was here
nov 04 03:00:28 hostname gitlab-sidekiq[8454]: 2023-11-04T02:00:28.114Z pid=8454 tid=2ea6 INFO: {:message=>"started", :memwd_handler_class=>"Gitlab::Memory::Watchdog::Handlers::SidekiqHandler", :memwd_sleep_time_s=>3, :pid=>8454, :worker_id=>"sidekiq", :memwd_rss_bytes=>373497856}
nov 04 03:00:28 hostname gitlab-sidekiq[8454]: {"severity":"INFO","time":"2023-11-04T02:00:28.575Z","message":"Listening on queues [\"abuse_new_abuse_report\", \"adjourned_project_deletion\", \"admin_emails\", \"analytics_code_review_metrics\", \"analytics_devops_adoption_create_snapshot\", \"analytics_usage_trends_counter_job\", \"app_sec_dast_scanner_profiles_builds_consistency\", \"app_sec_dast_scans_consistency\", \"app_sec_dast_site_profiles_builds_consistency\", \"approval_rules_external_approval_rule_payload\", \"approve_blocked_pending_approval_users\", \"audit_events_audit_event_streaming\", \"audit_events_user_impersonation_event_create\", \"auth_saml_group_sync\", \"authorized_keys\", \"authorized_project_update:authorized_project_update_project_recalculate\", \"authorized_project_update:authorized_project_update_project_recalculate_per_user\", \"authorized_project_update:authorized_project_update_user_refresh_from_replica\", \"au
thorized_project_update:authorized_project_update_user_refresh_over_user_range\", \"authorized_project_update:authorized_project_update_user_refresh_with_low_urgency\", \"authorized_projects\", \"auto_devops\", \"auto_devops:auto_devops_disable\", \"auto_merge\", \"auto_merge:auto_merge_process\", \"automation_execute_rule\", \"background_migration\", \"background_migration_ci_database\", \"batched_background_migrations\", \"batched_background_migrations:database_batched_background_migration_ci_execution\", \"batched_background_migrations:database_batched_background_migration_main_execution\", \"batched_git_ref_updates_project_cleanup\", \"bitbucket_import_advance_stage\", \"bitbucket_import_import_issue\", \"bitbucket_import_import_issue_notes\", \"bitbucket_import_import_lfs_object\", \"bitbucket_import_import_pull_request\", \"bitbucket_import_import_pull_request_notes\", \"bitbucket_import_stage_finish_import\", \"bitbucket_import_stage_
import_issues\", \"bitbucket_import_stage_import_issues_notes\", \"bitbucket_import_stage_import_lfs_objects\", \"bitbucket_import_stage_import_pull_requests\", \"bitbucket_import_stage_import_pull_requests_notes\", \"bitbucket_import_stage_import_repository\", \"bitbucket_server_import_advance_stage\", \"bitbucket_server_import_import_lfs_object\", \"bitbucket_server_import_import_pull_request\", \"bitbucket_server_import_import_pull_request_notes\", \"bitbucket_server_import_stage_finish_import\", \"bitbucket_server_import_stage_import_lfs_objects\", \"bitbucket_server_import_stage_import_notes\", \"bitbucket_server_import_stage_import_pull_requests\", \"bitbucket_server_import_stage_import_repository\", \"bulk_import\", \"bulk_imports_entity\", \"bulk_imports_export_request\", \"bulk_imports_finish_batched_pipeline\", \"bulk_imports_finish_batched_relation_export\", \"bulk_imports_finish_project_import\", \"bulk_imports_pipeline\", \"bulk
_imports_pipeline_batch\", \"bulk_imports_relation_batch_export\", \"bulk_imports_relation_export\", \"chaos\", \"chaos:chaos_cpu_spin\", \"chaos:chaos_db_spin\", \"chaos:chaos_kill\", \"chaos:chaos_leak_mem\", \"chaos:chaos_sleep\", \"chat_notification\", \"ci_cancel_pipeline\", \"ci_cancel_redundant_pipelines\", \"ci_delete_objects\", \"ci_initialize_pipelines_iid_sequence\", \"ci_job_artifacts_expire_project_build_artifacts\", \"ci_llm_generate_config\", \"ci_parse_secure_file_metadata\", \"ci_runners_process_runner_version_update\", \"ci_unlock_pipelines_in_queue\", \"ci_upstream_projects_subscriptions_cleanup\", \"click_house_ci_finished_builds_sync\", \"cluster_agent\", \"cluster_agent:clusters_agents_delete_expired_events\", \"cluster_agent:clusters_agents_notify_git_push\", \"compliance_management_chain_of_custody_report\", \"compliance_management_framework_export_mailer\", \"compliance_management_merge_requests_compliance_violations
\", \"compliance_management_pending_status_check\", \"compliance_management_standards_gitlab_at_least_two_approvals\", \"compliance_management_standards_gitlab_base\", \"compliance_management_standards_gitlab_group_base\", \"compliance_management_standards_gitlab_prevent_approval_by_author\", \"compliance_management_standards_gitlab_prevent_approval_by_author_group\", \"compliance_management_standards_gitlab_prevent_approval_by_committer\", \"compliance_management_standards_gitlab_prevent_approval_by_committer_group\", \"compliance_management_update_default_framework\", \"compliance_management_violation_export_mailer\", \"container_repository\", \"container_repository:cleanup_container_repository\", \"container_repository:container_expiration_policies_cleanup_container_repository\", \"container_repository:container_registry_record_data_repair_detail\", \"container_repository_delete\", \"container_repository_delete:container_registry_delete_c
ontainer_repository\", \"counters_cleanup_refresh\", \"create_commit_signature\", \"create_github_webhook\", \"create_note_diff_file\", \"cronjob\", \"cronjob:admin_email\", \"cronjob:analytics_usage_trends_count_job_trigger\", \"cronjob:authorized_project_update_periodic_recalculate\", \"cronjob:batched_git_ref_updates_cleanup_scheduler\", \"cronjob:bulk_imports_stuck_import\", \"cronjob:ci_archive_traces_cron\", \"cronjob:ci_delete_unit_tests\", \"cronjob:ci_pipeline_artifacts_expire_artifacts\", \"cronjob:ci_platform_metrics_update_cron\", \"cronjob:ci_runners_reconcile_existing_runner_versions_cron\", \"cronjob:ci_runners_stale_machines_cleanup_cron\", \"cronjob:ci_schedule_delete_objects_cron\", \"cronjob:ci_schedule_unlock_pipelines_in_queue_cron\", \"cronjob:ci_stuck_builds_drop_running\", \"cronjob:ci_stuck_builds_drop_scheduled\", \"cronjob:ci_update_locked_unknown_artifacts\", \"cronjob:click_house_events_sync\", \"cronjob:containe
r_expiration_policy\", \"cronjob:container_registry_cleanup\", \"cronjob:container_registry_migration_enqueuer\", \"cronjob:container_registry_migration_guard\", \"cronjob:container_registry_migration_observer\", \"cronjob:database_batched_background_migration\", \"cronjob:database_batched_background_migration_ci_database\", \"cronjob:database_ci_namespace_mirrors_consistency_check\", \"cronjob:database_ci_project_mirrors_consistency_check\", \"cronjob:database_drop_detached_partitions\", \"cronjob:database_monitor_locked_tables\", \"cronjob:database_partition_management\", \"cronjob:dependency_proxy_cleanup_dependency_proxy\", \"cronjob:dependency_proxy_image_ttl_group_policy\", \"cronjob:environments_auto_delete_cron\", \"cronjob:environments_auto_stop_cron\", \"cronjob:expire_build_artifacts\", \"cronjob:export_prune_project_export_jobs\", \"cronjob:gitlab_service_ping\", \"cronjob:import_export_project_cleanup\", \"cronjob:import_stuck_p
roject_import_jobs\", \"cronjob:issue_due_scheduler\", \"cronjob:issues_reschedule_stuck_issue_rebalances\", \"cronjob:jira_import_stuck_jira_import_jobs\", \"cronjob:loose_foreign_keys_cleanup\", \"cronjob:member_invitation_reminder_emails\", \"cronjob:members_expiring\", \"cronjob:merge_requests_ensure_prepared\", \"cronjob:metrics_global_metrics_update\", \"cronjob:namespaces_prune_aggregation_schedules\", \"cronjob:object_storage_delete_stale_direct_uploads\", \"cronjob:packages_cleanup_delete_orphaned_dependencies\", \"cronjob:packages_cleanup_package_registry\", \"cronjob:packages_composer_cache_cleanup\", \"cronjob:packages_debian_cleanup_dangling_package_files\", \"cronjob:pages_deactivated_deployments_delete_cron\", \"cronjob:pages_domain_removal_cron\", \"cronjob:pages_domain_ssl_renewal_cron\", \"cronjob:pages_domain_verification_cron\", \"cronjob:partition_creation\", \"cronjob:pause_control_resume\", \"cronjob:personal_access_to
kens_expired_notification\", \"cronjob:personal_access_tokens_expiring\", \"cronjob:pipeline_schedule\", \"cronjob:projects_inactive_projects_deletion_cron\", \"cronjob:projects_schedule_refresh_build_artifacts_size_statistics\", \"cronjob:prune_old_events\", \"cronjob:releases_manage_evidence\", \"cronjob:remove_expired_group_links\", \"cronjob:remove_expired_members\", \"cronjob:remove_unaccepted_member_invites\", \"cronjob:remove_unreferenced_lfs_objects\", \"cronjob:repository_archive_cache\", \"cronjob:repository_check_dispatch\", \"cronjob:schedule_merge_request_cleanup_refs\", \"cronjob:schedule_migrate_external_diffs\", \"cronjob:service_desk_custom_email_verification_cleanup\", \"cronjob:ssh_keys_expired_notification\", \"cronjob:ssh_keys_expiring_soon_notification\", \"cronjob:stuck_ci_jobs\", \"cronjob:stuck_export_jobs\", \"cronjob:stuck_merge_jobs\", \"cronjob:trending_projects\", \"cronjob:update_container_registry_info\", \"cr
onjob:user_status_cleanup_batch\", \"cronjob:users_create_statistics\", \"cronjob:users_deactivate_dormant_users\", \"cronjob:users_migrate_records_to_ghost_user_in_batches\", \"cronjob:x509_issuer_crl_check\", \"database_lock_tables\", \"default\", \"delete_diff_files\", \"delete_merged_branches\", \"delete_stored_files\", \"delete_user\", \"dependencies_destroy_export\", \"dependencies_export\", \"dependency_proxy\", \"dependency_proxy:purge_dependency_proxy_cache\", \"dependency_proxy_blob\", \"dependency_proxy_blob:dependency_proxy_cleanup_blob\", \"dependency_proxy_manifest\", \"dependency_proxy_manifest:dependency_proxy_cleanup_manifest\", \"deployment\", \"deployment:deployments_archive_in_project\", \"deployment:deployments_hooks\", \"deployment:deployments_link_merge_request\", \"deployment:deployments_update_environment\", \"design_management_copy_design_collection\", \"design_management_new_version\", \"destroy_pages_deployments\"
, \"detect_repository_languages\", \"disallow_two_factor_for_group\", \"disallow_two_factor_for_subgroups\", \"dora_metrics\", \"elastic_association_indexer\", \"elastic_commit_indexer\", \"elastic_delete_project\", \"elastic_full_index\", \"elastic_indexing_control\", \"elastic_namespace_indexer\", \"elastic_namespace_rollout\", \"elastic_namespace_update\", \"elastic_project_transfer\", \"elastic_wiki_indexer\", \"email_receiver\", \"emails_on_push\", \"environments_auto_stop\", \"environments_canary_ingress_update\", \"environments_stop_job_failed\", \"environments_stop_job_success\", \"epics\", \"epics_new_epic_issue\", \"error_tracking_issue_link\", \"export_csv\", \"external_service_reactive_caching\", \"file_hook\", \"flush_counter_increments\", \"gcp_cluster\", \"gcp_cluster:cluster_configure_istio\", \"gcp_cluster:cluster_install_app\", \"gcp_cluster:cluster_patch_app\", \"gcp_cluster:cluster_provision\", \"gcp_cluster:cluster_updat
e_app\", \"gcp_cluster:cluster_upgrade_app\", \"gcp_cluster:cluster_wait_for_app_installation\", \"gcp_cluster:cluster_wait_for_app_update\", \"gcp_cluster:cluster_wait_for_ingress_ip_address\", \"gcp_cluster:clusters_applications_activate_integration\", \"gcp_cluster:clusters_applications_deactivate_integration\", \"gcp_cluster:clusters_applications_uninstall\", \"gcp_cluster:clusters_applications_wait_for_uninstall_app\", \"gcp_cluster:clusters_cleanup_project_namespace\", \"gcp_cluster:clusters_cleanup_service_account\", \"gcp_cluster:wait_for_cluster_creation\", \"geo\", \"github_gists_importer\", \"github_gists_importer:github_gists_import_finish_import\", \"github_gists_importer:github_gists_import_import_gist\", \"github_gists_importer:github_gists_import_start_import\", \"github_import_advance_stage\", \"github_importer\", \"github_importer:github_import_attachments_import_issue\", \"github_importer:github_import_attachments_import_m
erge_request\", \"github_importer:github_import_attachments_import_note\", \"github_importer:github_import_attachments_import_release\", \"github_importer:github_import_import_collaborator\", \"github_importer:github_import_import_diff_note\", \"github_importer:github_import_import_issue\", \"github_importer:github_import_import_issue_event\", \"github_importer:github_import_import_lfs_object\", \"github_importer:github_import_import_note\", \"github_importer:github_import_import_protected_branch\", \"github_importer:github_import_import_pull_request\", \"github_importer:github_import_pull_requests_import_merged_by\", \"github_importer:github_import_pull_requests_import_review\", \"github_importer:github_import_pull_requests_import_review_request\", \"github_importer:github_import_refresh_import_jid\", \"github_importer:github_import_stage_finish_import\", \"github_importer:github_import_stage_import_attachments\", \"github_importer:github_i
mport_stage_import_base_data\", \"github_importer:github_import_stage_import_collaborators\", \"github_importer:github_import_stage_import_issue_events\", \"github_importer:github_import_stage_import_issues_and_diff_notes\", \"github_importer:github_import_stage_import_lfs_objects\", \"github_importer:github_import_stage_import_notes\", \"github_importer:github_import_stage_import_protected_branches\", \"github_importer:github_import_stage_import_pull_requests\", \"github_importer:github_import_stage_import_pull_requests_merged_by\", \"github_importer:github_import_stage_import_pull_requests_review_requests\", \"github_importer:github_import_stage_import_pull_requests_reviews\", \"github_importer:github_import_stage_import_repository\", \"gitlab_performance_bar_stats\", \"gitlab_subscriptions_add_on_purchases_bulk_refresh_user_assignments\", \"gitlab_subscriptions_add_on_purchases_cleanup_user_add_on_assignment\", \"gitlab_subscriptions_add_
on_purchases_refresh_user_assignments\", \"gitlab_subscriptions_refresh_seats\", \"gitlab_subscriptions_trials_apply_trial\", \"google_cloud_create_cloudsql_instance\", \"google_cloud_fetch_google_ip_list\", \"group_destroy\", \"group_export\", \"group_import\", \"group_saml_group_sync\", \"group_wikis_git_garbage_collect\", \"groups_create_event\", \"groups_enterprise_users_associate\", \"groups_enterprise_users_bulk_associate_by_domain\", \"groups_enterprise_users_disassociate\", \"groups_export_memberships\", \"groups_schedule_bulk_repository_shard_moves\", \"groups_update_repository_storage\", \"groups_update_statistics\", \"groups_update_two_factor_requirement_for_members\", \"hashed_storage\", \"hashed_storage:hashed_storage_migrator\", \"hashed_storage:hashed_storage_project_migrate\", \"hashed_storage:hashed_storage_project_rollback\", \"hashed_storage:hashed_storage_rollbacker\", \"import_issues_csv\", \"incident_management\", \"inc
ident_management:incident_management_add_severity_system_note\", \"incident_management:incident_management_pager_duty_process_incident\", \"incident_management:incident_management_process_alert_worker_v2\", \"incident_management_apply_incident_sla_exceeded_label\", \"incident_management_close_incident\", \"incident_management_oncall_rotations_persist_shifts_job\", \"incident_management_pending_escalations_alert_check\", \"incident_management_pending_escalations_alert_create\", \"incident_management_pending_escalations_issue_check\", \"incident_management_pending_escalations_issue_create\", \"integrations_create_external_cross_reference\", \"integrations_execute\", \"integrations_group_mention\", \"integrations_irker\", \"integrations_slack_event\", \"invalid_gpg_signature_update\", \"issuable_export_csv\", \"issuable_label_links_destroy\", \"issuable_related_links_create\", \"issuables_clear_groups_issue_counter\", \"issues_close\", \"issues
_placement\", \"issues_rebalancing\", \"iterations\", \"jira_connect\", \"jira_connect:jira_connect_forward_event\", \"jira_connect:jira_connect_retry_request\", \"jira_connect:jira_connect_send_uninstalled_hook\", \"jira_connect:jira_connect_sync_branch\", \"jira_connect:jira_connect_sync_builds\", \"jira_connect:jira_connect_sync_deployments\", \"jira_connect:jira_connect_sync_feature_flags\", \"jira_connect:jira_connect_sync_merge_request\", \"jira_connect:jira_connect_sync_project\", \"jira_importer\", \"jira_importer:jira_import_advance_stage\", \"jira_importer:jira_import_import_issue\", \"jira_importer:jira_import_stage_finish_import\", \"jira_importer:jira_import_stage_import_attachments\", \"jira_importer:jira_import_stage_import_issues\", \"jira_importer:jira_import_stage_import_labels\", \"jira_importer:jira_import_stage_import_notes\", \"jira_importer:jira_import_stage_start_import\", \"ldap_group_sync\", \"llm_completion\", \"ll
m_embedding_gitlab_documentation_create_db_embeddings_per_doc_file\", \"llm_embedding_gitlab_documentation_set_embeddings_on_the_record\", \"llm_namespace_access_cache_reset\", \"llm_tanuki_bot_update\", \"llm_vertex_ai_access_token_refresh\", \"mail_scheduler\", \"mail_scheduler:mail_scheduler_issue_due\", \"mail_scheduler:mail_scheduler_notification_service\", \"mailers\", \"members_destroyer_clean_up_group_protected_branch_rules\", \"members_expiring_email_notification\", \"merge\", \"merge_request_cleanup_refs\", \"merge_request_mergeability_check\", \"merge_request_reset_approvals\", \"merge_requests_capture_suggested_reviewers_accepted\", \"merge_requests_cleanup_ref\", \"merge_requests_close_issue\", \"merge_requests_create_approval_event\", \"merge_requests_create_approval_note\", \"merge_requests_delete_source_branch\", \"merge_requests_execute_approval_hooks\", \"merge_requests_fetch_suggested_reviewers\", \"merge_requests_handle_a
ssignees_change\", \"merge_requests_llm_summarize_merge_request\", \"merge_requests_mergeability_check_batch\", \"merge_requests_process_approval_auto_merge\", \"merge_requests_resolve_todos\", \"merge_requests_resolve_todos_after_approval\", \"merge_requests_set_reviewer_reviewed\", \"merge_requests_stream_approval_audit_event\", \"merge_requests_sync_code_owner_approval_rules\", \"merge_requests_update_head_pipeline\", \"migrate_external_diffs\", \"ml_experiment_tracking_associate_ml_candidate_to_package\", \"namespaces_process_sync_events\", \"namespaces_storage_usage_export\", \"namespaces_sync_namespace_name\", \"namespaces_update_root_statistics\", \"new_epic\", \"new_issue\", \"new_merge_request\", \"new_note\", \"object_pool\", \"object_pool:object_pool_create\", \"object_pool:object_pool_destroy\", \"object_pool:object_pool_join\", \"object_pool:object_pool_schedule_join\", \"object_storage\", \"object_storage:object_storage_migrate
_uploads\", \"onboarding_create_iterable_trigger\", \"onboarding_issue_created\", \"onboarding_pipeline_created\", \"onboarding_progress\", \"onboarding_user_added\", \"package_cleanup\", \"package_cleanup:packages_cleanup_execute_policy\", \"package_cleanup:packages_cleanup_package_file\", \"package_cleanup:packages_mark_package_files_for_destruction\", \"package_metadata_advisory_scan\", \"package_repositories\", \"package_repositories:packages_debian_generate_distribution\", \"package_repositories:packages_debian_process_package_file\", \"package_repositories:packages_go_sync_packages\", \"package_repositories:packages_helm_extraction\", \"package_repositories:packages_maven_metadata_sync\", \"package_repositories:packages_npm_create_metadata_cache\", \"package_repositories:packages_npm_deprecate_package\", \"package_repositories:packages_nuget_extraction\", \"package_repositories:packages_rubygems_extraction\", \"packages_composer_cache_
update\", \"pages\", \"pages_domain_ssl_renewal\", \"pages_domain_verification\", \"personal_access_tokens\", \"pipeline_background\", \"pipeline_background:ci_archive_trace\", \"pipeline_background:ci_build_trace_chunk_flush\", \"pipeline_background:ci_daily_build_group_report_results\", \"pipeline_background:ci_job_artifacts_track_artifact_report\", \"pipeline_background:ci_pending_builds_update_group\", \"pipeline_background:ci_pending_builds_update_project\", \"pipeline_background:ci_pipeline_artifacts_coverage_report\", \"pipeline_background:ci_pipeline_artifacts_create_quality_report\", \"pipeline_background:ci_pipeline_success_unlock_artifacts\", \"pipeline_background:ci_ref_delete_unlock_artifacts\", \"pipeline_background:ci_refs_unlock_previous_pipelines\", \"pipeline_background:ci_test_failure_history\", \"pipeline_background:ci_track_failed_build\", \"pipeline_creation\", \"pipeline_creation:ci_external_pull_requests_create_pipeli
ne\", \"pipeline_creation:create_pipeline\", \"pipeline_creation:merge_requests_create_pipeline\", \"pipeline_creation:run_pipeline_schedule\", \"pipeline_default\", \"pipeline_default:ci_create_downstream_pipeline\", \"pipeline_default:ci_drop_pipeline\", \"pipeline_default:ci_merge_requests_add_todo_when_build_fails\", \"pipeline_default:ci_pipeline_bridge_status\", \"pipeline_default:ci_pipeline_cleanup_ref\", \"pipeline_default:ci_retry_pipeline\", \"pipeline_default:pipeline_metrics\", \"pipeline_default:pipeline_notification\", \"pipeline_hooks\", \"pipeline_hooks:build_hooks\", \"pipeline_hooks:pipeline_hooks\", \"pipeline_processing\", \"pipeline_processing:build_queue\", \"pipeline_processing:build_success\", \"pipeline_processing:ci_build_finished\", \"pipeline_processing:ci_build_prepare\", \"pipeline_processing:ci_build_schedule\", \"pipeline_processing:ci_initial_pipeline_process\", \"pipeline_processing:ci_resource_groups_assig
n_resource_from_resource_group\", \"pipeline_processing:pipeline_process\", \"pipeline_processing:stage_update\", \"pipeline_processing:update_head_pipeline_for_merge_request\", \"post_receive\", \"process_commit\", \"product_analytics_initialize_snowplow_product_analytics\", \"product_analytics_post_push\", \"project_cache\", \"project_destroy\", \"project_export\", \"project_import_schedule\", \"project_template_export\", \"projects_after_import\", \"projects_delete_branch\", \"projects_deregister_suggested_reviewers_project\", \"projects_finalize_project_statistics_refresh\", \"projects_forks_sync\", \"projects_git_garbage_collect\", \"projects_import_export_create_relation_exports\", \"projects_import_export_parallel_project_export\", \"projects_import_export_relation_export\", \"projects_import_export_wait_relation_exports\", \"projects_inactive_projects_deletion_notification\", \"projects_post_creation\", \"projects_process_sync_events
\", \"projects_record_target_platforms\", \"projects_refresh_build_artifacts_size_statistics\", \"projects_register_suggested_reviewers_project\", \"projects_repository_destroy\", \"projects_schedule_bulk_repository_shard_moves\", \"projects_update_repository_storage\", \"propagate_integration\", \"propagate_integration_group\", \"propagate_integration_inherit\", \"propagate_integration_inherit_descendant\", \"propagate_integration_project\", \"pull_mirrors_reenable_configuration\", \"reactive_caching\", \"rebase\", \"redis_migration\", \"refresh_license_compliance_checks\", \"releases_create_evidence\", \"remote_mirror_notification\", \"repository_check\", \"repository_check:repository_check_batch\", \"repository_check:repository_check_clear\", \"repository_check:repository_check_single_repository\", \"repository_cleanup\", \"repository_fork\", \"repository_import\", \"repository_update_mirror\", \"repository_update_remote_mirror\", \"requi
rements_management_import_requirements_csv\", \"requirements_management_process_requirements_reports\", \"sbom_reports\", \"search_elastic_default_branch_changed\", \"search_elastic_group_association_deletion\", \"search_namespace_index_integrity\", \"search_project_index_integrity\", \"search_wiki_elastic_delete_group_wiki\", \"search_zoekt_default_branch_changed\", \"search_zoekt_delete_project\", \"search_zoekt_namespace_indexer\", \"search_zoekt_project_transfer\", \"security_auto_fix\", \"security_generate_policy_violation_comment\", \"security_orchestration_configuration_create_bot\", \"security_orchestration_policy_rule_schedule_namespace\", \"security_process_scan_result_policy\", \"security_refresh_project_policies\", \"security_scan_execution_policies_rule_schedule\", \"security_scan_result_policies_sync_any_merge_request_approval_rules\", \"security_scan_result_policies_sync_opened_merge_requests\", \"security_scan_result_policies
_sync_project\", \"security_scans\", \"security_scans_purge_by_job_id\", \"security_sync_scan_policies\", \"service_desk_email_receiver\", \"set_user_status_based_on_user_cap_setting\", \"snippets_schedule_bulk_repository_shard_moves\", \"snippets_update_repository_storage\", \"ssh_keys_update_last_used_at\", \"status_page_publish\", \"sync_seat_link_request\", \"system_access_group_saml_microsoft_group_sync\", \"system_access_saml_microsoft_group_sync\", \"system_hook_push\", \"tasks_to_be_done_create\", \"terraform\", \"terraform:terraform_states_destroy\", \"todos_destroyer\", \"todos_destroyer:todos_destroyer_confidential_issue\", \"todos_destroyer:todos_destroyer_destroyed_designs\", \"todos_destroyer:todos_destroyer_destroyed_issuable\", \"todos_destroyer:todos_destroyer_entity_leave\", \"todos_destroyer:todos_destroyer_group_private\", \"todos_destroyer:todos_destroyer_private_features\", \"todos_destroyer:todos_destroyer_project_priv
ate\", \"unassign_issuables\", \"unassign_issuables:members_destroyer_unassign_issuables\", \"update_external_pull_requests\", \"update_highest_role\", \"update_merge_requests\", \"update_namespace_statistics\", \"update_namespace_statistics:namespaces_root_statistics\", \"update_namespace_statistics:namespaces_schedule_aggregation\", \"update_project_statistics\", \"upload_checksum\", \"users_track_namespace_visits\", \"vulnerabilities_mark_dropped_as_resolved\", \"vulnerabilities_statistics_adjustment\", \"vulnerabilities_update_namespace_ids_of_vulnerability_reads\", \"vulnerability_exports_export\", \"vulnerability_exports_export_deletion\", \"web_hook\", \"web_hooks_log_destroy\", \"web_hooks_log_execution\", \"wikis_git_garbage_collect\", \"work_items_import_work_items_csv\", \"work_items_update_parent_objectives_progress\", \"x509_certificate_revoke\", \"zoekt_indexer\"]"}
nov 04 03:00:28 hostname gitlab-sidekiq[8454]: {"severity":"INFO","time":"2023-11-04T02:00:28.575Z","message":"Booting Sidekiq 6.5.7 with Sidekiq::RedisConnection::RedisAdapter options {:instrumentation_class=\u003eGitlab::Instrumentation::Redis::Queues, :path=\u003e\"/var/run/redis/redis.sock\", :namespace=\u003e\"resque:gitlab\"}"}
nov 04 03:00:29 hostname gitlab-sidekiq[8454]: {"severity":"INFO","time":"2023-11-04T02:00:29.765Z","message":"GitLab reliable fetch activated!"}Offline