Arch Linux

marcus

Member

Registered: 2012-09-26

Posts: 23

[Solved] GitLab: unable to push/pull after upgrade

Hi Forum,

after the latest upgrade, I am no longer able to push or pull code with my GitLab installation:

$ pacman -Qs gitlab
local/gitlab 14.3.1-1
    Project management and code hosting application
local/gitlab-gitaly 14.3.0-1
    Speed up Git access using caching
local/gitlab-runner 14.3.1-1
    The official GitLab CI runner written in Go
local/gitlab-shell 2:13.21.1-1
    Git management software
local/gitlab-workhorse 8.65.0-1
    A smart reverse proxy for GitLab that handles large HTTP requests such as file downloads, file uploads,
    git push/pull and git archive downloads

Both commands throw:

$ git push
remote:
remote: ========================================================================
remote:
remote: The project you were looking for could not be found or you don't have permission to view it.
remote:
remote: ========================================================================
remote:
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

/var/log/gitlab/gitlab-shell.log (formatted for readability):

{
    "correlation_id": "01FHDJZ09SSR4XBPG6Z2AENZXN",
    "duration_ms": 42,
    "error": "The project you were looking for could not be found or you don't have permission to view it.",
    "level": "error",
    "method": "POST",
    "msg": "Internal API error",
    "status": 404,
    "time": "2021-10-07T14:36:52Z",
    "url": "http://unix/api/v4/internal/allowed"
}

/var/log/gitlab/api_json.log (formatted for readability):

{
    "time": "2021-10-07T14:36:52.450Z",
    "severity": "INFO",
    "duration_s": 0.02291,
    "db_duration_s": 0.00276,
    "view_duration_s": 0.02015,
    "status": 404,
    "method": "POST",
    "path": "/api/v4/internal/allowed",
    "params": [
        {
            "key": "action",
            "value": "git-receive-pack"
        },
        {
            "key": "project",
            "value": "some/project.git"
        },
        {
            "key": "changes",
            "value": "_any"
        },
        {
            "key": "protocol",
            "value": "ssh"
        },
        {
            "key": "check_ip",
            "value": "10.0.0.9"
        }
    ],
    "host": "unix",
    "remote_ip": "127.0.0.1",
    "ua": "GitLab-Shell",
    "route": "/api/:version/internal/allowed",
    "queue_duration_s": 0.014652,
    "redis_calls": 2,
    "redis_duration_s": 0.000851,
    "redis_read_bytes": 405,
    "redis_write_bytes": 119,
    "redis_cache_calls": 2,
    "redis_cache_duration_s": 0.000851,
    "redis_cache_read_bytes": 405,
    "redis_cache_write_bytes": 119,
    "db_count": 1,
    "db_write_count": 0,
    "db_cached_count": 0,
    "cpu_s": 0.032507,
    "pid": 1418,
    "correlation_id": "01FHDJZ09VPRVXM9VTE07NZ4QP",
    "meta.project": "some/project",
    "meta.root_namespace": "some",
    "meta.caller_id": "POST /api/:version/internal/allowed",
    "meta.remote_ip": "127.0.0.1",
    "meta.feature_category": "source_code_management",
    "meta.client_id": "ip/127.0.0.1",
    "content_length": "115"
}

Testing the ssh connection:

$ ssh -T gitlab@git.noflyz.one
remote:
remote: ========================================================================
remote:
remote: Failed to get username: who='' is invalid
remote:
remote: ========================================================================
remote:

With gitlab-shell downgraded to 2:13.20.0-1 everything works fine.

Gitlab user has its shell correctly set:

$ getent passwd | grep gitlab
gitlab:x:105:105::/var/lib/gitlab:/usr/share/webapps/gitlab-shell/bin/gitlab-shell

Any ideas appreciated!

Cheers, Marcus

Last edited by marcus (2021-10-09 07:46:37)

caleb

Member

Registered: 2016-04-16

Posts: 14

Re: [Solved] GitLab: unable to push/pull after upgrade

Yup yup, same problem here. My issue was complicated because Apache 2.4.49 broke the proxy redirect I was using to serve GitLab. I assumed the broken SSH access was related to my hack to get that up. With Apache 2.4.50 I reverted to my original configs only to find `gitlab-shell` is still acting up as you describe.

caleb

Member

Registered: 2016-04-16

Posts: 14

Re: [Solved] GitLab: unable to push/pull after upgrade

This seems to be opened on the bug tracker here: https://bugs.archlinux.org/task/72298?p … tlab-shell

caleb

Member

Registered: 2016-04-16

Posts: 14

Re: [Solved] GitLab: unable to push/pull after upgrade

Also see upstream issue report here: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/530

That has alternative workarounds to downgrading, but still no proper fix.

marcus

Member

Registered: 2012-09-26

Posts: 23

Re: [Solved] GitLab: unable to push/pull after upgrade

Hi caleb

Also see upstream issue report here: https://gitlab.com/gitlab-org/gitlab-shell/-/issues/530

Thanks for the hint, this comment had the solution for me:

usermod -s /bin/sh gitlab

I have also enabled fast ssh key lookup:

Match User gitlab
    AuthorizedKeysCommand     /var/lib/gitlab/gitlab-shell/bin/gitlab-shell-authorized-keys-check gitlab %u %k
    AuthorizedKeysCommandUser gitlab

Haven't checked though, if this is really needed.

I wonder why arch has set up gitlab user's shell to gitlab-shell, if upstream doesn't:

As a note, GitLab Omnibus sets up the git user (the one that runs the GitLab rails application) with a login shell of /bin/sh.

Marking issue closed.