Arch Linux

iavr

Member

Registered: 2024-02-07

Posts: 20

[SOLVED] Log spam from GitLab Sidekiq

I've installed gitlab 16.11.1-1

It works more-or-less as it should.  One issue, however, is that it uses a default log level of "INFO", and this causes it (specifically the gitlab-sidekiq service) to generate a huge volume of not-especially-important logs.  The question is, how can I bump the log level up to "WARN" to reduce the logging overhead?

I have a solution that works, but I'm not happy with this solution because it's hackish and it gets overwritten every time gitlab updates:
- Edit the file /usr/share/webapps/gitlab/config/initializers/sidekiq.rb
- Add the following line to the Sidekiq.configure_server section:

  config.logger.level = Logger::WARN

The GitLab docs says that setting the GITLAB_LOG_LEVEL environment variable would have the effect of changing the minimum log level.  I tried to do this by creating the following /etc/systemd/system/gitlab-sidekiq.service.d/sidekiq_env.conf file:

[Service]
# ensure that logging is WARN or above only, to prevent log spam
Environment="GITLAB_LOG_LEVEL=warn"

Playing with the code, I've confirmed that: the GITLAB_LOG_LEVEL environment variable does have the correct value in sidekiq.rb, but the logger is still set to the INFO level, and the program still generates log spam.  Any ideas what I might be doing wrong?

Last edited by iavr (2024-07-03 17:02:13)

iavr

Member

Registered: 2024-02-07

Posts: 20

Re: [SOLVED] Log spam from GitLab Sidekiq

This is the solution I'm currently using, by the way, inspired by the script from the wiki:

/ect/pacman.d/scripts/gitlab-migrate-database.sh

#!/bin/sh

# Perform the database migration.
cd "/usr/share/webapps/gitlab"
sudo -u gitlab $(cat environment | xargs) bundle exec rake db:migrate

# Ensure that the log level for Sidekiq is set to WARN
SIDEKIQ_FILE=/usr/share/webapps/gitlab/config/initializers/sidekiq.rb

if [[ -f $SIDEKIQ_FILE && -r $SIDEKIQ_FILE && -w $SIDEKIQ_FILE ]]; then
    grep -q Logger::WARN $SIDEKIQ_FILE || sed -i '/Sidekiq[.]configure_server do [|]config[|]/a \ \ config.logger.level = Logger::WARN' $SIDEKIQ_FILE
fi

# The hook runs after 30-systemd-daemon-reload.hook so another systemctl daemon-reload is not necessary.
systemctl start gitlab.target

This "solves" the log spam issue by inserting a line into the sidekiq.rb script whenever it gets updated.  The line sets the logger level to the WARN level, reducing the amount of logging.
I don't like this solution because it's a hack that makes assumptions about how the script is structured; I'm still not sure why setting the environment doesn't work.  Regardless, marking this as solved.