You are not logged in.

#1 2020-10-10 16:04:09

amo
Member
Registered: 2016-01-05
Posts: 13

[SOLVED] Need advise packaging "papermerge" django web app (python3)

Dear arch community

I am trying to package papermerge. The packaging itself works fine, the package installs, but I run into issues when trying to run it:
Papermerge is a python3 app based on the django framework. All dependencies are installed with pacman (or from AUR and a few additional PKGBUILDs I prepared). Before the initial start of the server, papermerge needs to do a few migrations: first using

django-admin makemigrations

(to my understanding, these are migrations in the libraries which papermerge depends on) and

django-admin migrate

(these are migrations in papermerge itself).
The problem is now that the first migration call fails to run successfully because it is run with the papermerge user (created by the package) who has no write access inside the directory containing the migrations (owned by root since it was installed by pacman).
My question is, how do I go about this? I would like to prevent running stuff as root as I think it is not intended to be used that way.

For reference and help: here is a short discussion about the exact problem and here are the files I use to build and install the package (credits to Lucki who initially made these files):
PKGBUILD

# Contributor: Lucki <https://aur.archlinux.org/account/Lucki>
# shellcheck disable=SC2034,2154,2148
# Maintainer: amo <https://aur.archlinux.org/account/amo>

pkgname=papermerge
pkgver=1.4.5
pkgrel=1
pkgdesc="An open source document management system (DMS) primarily designed for archiving and retrieving your digital documents."
arch=('any')
url="https://github.com/ciur/papermerge"
license=('Apache')
backup=("etc/$pkgname.conf.py" "usr/lib/python3.8/site-packages/config/settings/production.py")
install=$pkgname.install
depends=('imagemagick' 'pdftk' 'poppler' 'python-celery' 'python-django' 'python-django-allauth' 'python-django-celery-results' 'python-django-dynamic-preferences'
      'python-django-modelcluster' 'python-django-mptt' 'python-django-polymorphic-tree' 'python-django-rest-framework' 'python-django-rest-knox' 'python-django-taggit'
      'python-imapclient' 'python-lxml' 'python-mgclipboard' 'python-mglib' 'python-psycopg2' 'python-pyaml' 'tesseract')
optdepends=('gunicorn: run papermerge in a production environment with nginx'
          'nginx: run papermerge through gunicorn in a production environment')
source=("$pkgname-$pkgver.src.tar.gz::https://github.com/ciur/$pkgname/archive/v$pkgver.tar.gz"
      "$pkgname.install"
      "$pkgname.sysusers"
      "$pkgname.tmpfiles"
      "$pkgname-gunicorn.service"
      "$pkgname-worker.service"
      "$pkgname.sh")
sha512sums=('e2f2227d9e57ef38efbc70d0a64c99134907a92a84b31d54adbe643bd69bd0f16a65251eddef61e402b20213ad2baf047f570d1609c948cb320d4505dd2ede8c'
          '22a83a9f26c6991ef6710c08157ff24e11f246a45ccac4b88406fb4f4b57d07608f5fa75bc42f28e7db920ce7ff05617cd03f9febc476f8d654e9fa7d6eb9019'
          'eb408d6d6c0072b4883cae9b6aa591935b910585e93aa6fdacbb543dbc4233588678d396c06d0f10b7c74b4d393af19b8be526d094821de23792c29df593d4a5'
          '07726d47deb2438dbeb3f5dc6ebc2218a4d5f25dacb7e0304102deddfb914d4d6d649643e26cb844ac4d02c7f21b58835edd4026ff91a6481fee59f786b90925'
          '58cd8936ccd5700361f01af2c01af5645ef08c2b1045efd28eb4697af60e49230615b0b0a57dcca1f9813b5658d5244bee8f7315fc62aa5c903cf9586e9a8cc8'
          'fcdd49e0b6b93ade84ddafd77111f28bdea75a4f12829ec249d414d46a14399198be7bf29d0ce5166de6dba1d6386eda4ab9b39a1f5376b7c7758c8966a53ce3'
          'b838d54dca56e45f99690ddaae1b058e3f72b314adf9d689869f14bb141a5bbf209df974eea20cb2305b127d5119d0b8d11293494d47350f6974edc6fffc374f')

prepare() {
  {
      echo 'DBDIR = "/var/lib/papermerge/database"';
      echo 'MEDIA_DIR = "/var/lib/papermerge/media"';
      echo 'STATIC_DIR = "/var/lib/papermerge/static"';
      echo 'TASK_QUEUE_DIR = "/var/tmp/papermerge/queue"'
  } >> "$pkgname-$pkgver/$pkgname.conf.py.example"

  # https://papermerge.readthedocs.io/en/latest/setup/server_configurations.html#step-1-install-gunicorn
  {
      echo 'from .base import *  # noqa';
      echo 'DEBUG = False';
      echo "ALLOWED_HOSTS = ['*']"
  } > "$pkgname-$pkgver/config/settings/production.py"

  # The admin package is missing somehow so let's move it plain stupid directly in place
  mkdir -p "$pkgname-$pkgver/build/lib/$pkgname/"
  cp -dpr --no-preserve=ownership "$pkgname-$pkgver/$pkgname/contrib" "$pkgname-$pkgver/build/lib/$pkgname/"
}

build() {
  cd "$pkgname-$pkgver" || exit
  python setup.py build
}

check() {
  cd "$pkgname-$pkgver" || exit
  ./run_tests.sh
}

package() {
  # tmpfiles.d
  install -Dm 644 "$pkgname.tmpfiles" \
      "$pkgdir/usr/lib/tmpfiles.d/$pkgname.conf"

  # sysusers.d
  install -Dm 644 "$pkgname.sysusers" \
      "$pkgdir/usr/lib/sysusers.d/$pkgname.conf"

  # service files
  install -Dm 644 "$pkgname-gunicorn.service" \
      "$pkgdir/usr/lib/systemd/system/$pkgname-gunicorn.service"
  install -Dm 644 "$pkgname-worker.service" \
      "$pkgdir/usr/lib/systemd/system/$pkgname-worker.service"

  # wrapper
  install -Dm 755 "$pkgname.sh" \
      "$pkgdir/usr/bin/$pkgname-manage"

  cd "$pkgname-$pkgver" || exit

  # config
  install -Dm 644 "$pkgname.conf.py.example" \
      "$pkgdir/etc/$pkgname.conf.py"

  export PYTHONHASHSEED=0
  python setup.py install --root="$pkgdir" --optimize=1 --skip-build
}

papermerge.tmpfiles

d /var/lib/papermerge/media - papermerge papermerge -
d /var/lib/papermerge/static - papermerge papermerge -
d /var/lib/papermerge/database - papermerge papermerge -
d /run/papermerge - papermerge papermerge -
d /var/log/papermerge - papermerge papermerge -

papermerge.sh

#!/bin/sh
export DJANGO_SETTINGS_MODULE=config.settings.dev
cd /tmp || exit
exec /usr/bin/django-admin "$@"

papermerge.install

post_install() {
	printf '\n --> Run papermerge with "systemctl start papermerge-gunicorn papermerge-worker". Point your reverse proxy (e.g. nginx) to 127.0.0.1:9001.\n'
	printf '\n --> If this is a new Papermerge installation, please run "sudo -u papermerge papermerge-manage createsuperuser" to create the admin user.\n\n'
}

papermerge.sysusers

u papermerge - "open source document management system" -

papermerge-gunicorn.service

[Unit]
Description=Gunicorn providing papermerge
Documentation=https://papermerge.readthedocs.io/en/latest/setup/server_configurations.html,https://docs.gunicorn.org/en/latest/deploy.html#systemd
After=network.target
Wants=papermerge-worker.service

[Service]
Type=notify
User=papermerge
Group=papermerge
RuntimeDirectory=papermerge
WorkingDirectory=/var/lib/papermerge
Environment="DJANGO_SETTINGS_MODULE=config.settings.production"
ExecStartPre=/usr/bin/papermerge-manage makemigrations
ExecStartPre=/usr/bin/papermerge-manage migrate
ExecStartPre=/usr/bin/papermerge-manage collectstatic --noinput
ExecStart=/usr/bin/gunicorn config.wsgi:application --bind 127.0.0.1:9001 --workers 2 --access-logfile "/var/log/papermerge/gunicorn.access" --error-logfile "/var/log/papermerge/gunicorn.error"
ExecReload=/bin/kill -s HUP $MAINPID
KillMode=mixed
TimeoutStopSec=5
PrivateTmp=true
Restart=on-failure

[Install]
WantedBy=multi-user.target

papermerge-worker.service

[Unit]
Description=Papermerge Worker
Documentation=https://papermerge.readthedocs.io/en/latest/setup/server_configurations.html
After=network.target
PartOf=papermerge-gunicorn.service

[Service]
Type=simple
User=papermerge
Group=papermerge
WorkingDirectory=/tmp
Environment="DJANGO_SETTINGS_MODULE=config.settings.production"
ExecStart=/usr/bin/django-admin worker --pidfile /var/run/papermerge/papermerge-worker.pid
Restart=on-failure

[Install]
WantedBy=multi-user.target

Maybe somebody with experience packaging python applications can help here. Any advise or hint is greatly appreciated!

Last edited by amo (2020-11-10 15:10:29)

Offline

#2 2020-10-29 09:02:21

amo
Member
Registered: 2016-01-05
Posts: 13

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

Unfortunately, no better idea came to my mind in the meantime than just to set the

/usr/lib/python3.8/site-packages/papermerge/core/migrations

directory to be owned by the papermerge user.
Is there anything to be said against it?

If I am not mistaken, nothing other than papermerge would need the content of that dir anyway.
Also: would it be a better "packaging practice" to have this migrations dir owned by the papermerhe user reside somewhere else in the filesystem and be symlinked there?

Offline

#3 2020-10-30 00:34:06

eschwartz
Fellow
Registered: 2014-08-08
Posts: 4,097

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

The symlink method is better, let the actual files be edited in e.g. /var/lib/papermerge/migrations

This does seem like a stupid limitation though.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#4 2020-10-30 07:59:48

amo
Member
Registered: 2016-01-05
Posts: 13

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

Thank you a lot for your reply.

How would I go about creating the symlink with the PKGBUILD?
My idea would be to add

d /var/lib/papermerge/migrations - papermerge papermerge -

to papermerge.tmpfiles in order to have the directory created for the package. But where would I put the ln -s statement? To my understanding it would need to be after the creation of the link's parent directory but before the extraction of files from the package into the linked destination. Is that possible at all?

Last edited by amo (2020-10-30 08:00:39)

Offline

#5 2020-10-30 12:05:58

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

man ln wrote:

When creating hard links, each TARGET must exist.
       Symbolic links can hold arbitrary text

Just create the symbolic link in the package function.
for an example check package_mesa() in https://github.com/archlinux/svntogit-p … k/PKGBUILD


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#6 2020-10-30 14:21:48

amo
Member
Registered: 2016-01-05
Posts: 13

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

Unfortunately, it is not that easy:
If I put the ln -s statement into the package() function, the packaging procedure fails at that point with a "file exists" error. I guess the problem is that it tries to extract files to the not-yet-existing target folder of the link.
Now, first thing that comes to mind is: create the target folder beforehand. But this approach would not work because the target folder would then be owned by root, but it explicitely need to be owned by the papermerge user that, at this point, does not yet exist either. This is what I hoped to work around when adding "d /var/lib/papermerge/migrations - papermerge papermerge -" to papermerge.tmpfiles, but these folder are only created later.

What I also tried was to chown the folders in a ExecPre statement of the systemd service, but this approach also fails because the service is to be run as the papermerge user which does not have the privileges to do that.

Any further ideas would be greatly appreciated!

Offline

#7 2020-10-31 18:37:59

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

amo wrote:

f I put the ln -s statement into the package() function, the packaging procedure fails at that point with a "file exists" error.

Does that also happen when you build in a clean chroot ?
See https://wiki.archlinux.org/index.php/De … ean_chroot


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#8 2020-11-01 04:02:08

eschwartz
Fellow
Registered: 2014-08-08
Posts: 4,097

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

If the directory "$pkgdir"/usr/lib/python3.8/site-packages/papermerge/core/migrations already exists because python setup.py install has created it, then you'll need to rmdir it before using ln -s.

I presume it does not contain files which would need to be moved to /var/lib/papermerge/migrations?

Lone_Wolf,

Note that package() always nukes $pkgdir beforehand, so building in a clean chroot is irrelevant here... clean chroots affect $srcdir only.

Last edited by eschwartz (2020-11-01 04:03:36)


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#9 2020-11-01 09:05:47

amo
Member
Registered: 2016-01-05
Posts: 13

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

Thank you for the replies!

eschwartz wrote:

I presume it does not contain files which would need to be moved to /var/lib/papermerge/migrations?

Actually, there are files in there that would need to be moved. Unfortunately, this leads to the problem, that the user who needs to own them does not yet exist at this point.

Offline

#10 2020-11-08 15:21:54

amo
Member
Registered: 2016-01-05
Posts: 13

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

As it looks to me like it is not really possible to package this the clean way, but I still feel determined to provide an easy way of installing and updating papermerge on arch, I came up with another idea:

I would just create /var/lib/papermerge/migrations as root and symlink /usr/lib/python3.8/site-packages/papermerge/core/migrations to that place and let it all be owned by root for now. I'd then remove ExecStartPre=/usr/bin/papermerge-manage makemigrations, ExecStartPre=/usr/bin/papermerge-manage migrate, and ExecStartPre=/usr/bin/papermerge-manage collectstatic --noinput from the systemd service file and create a pacman hook containing these with a preceding chown -R papermerge:papermerge /var/lib/papermerge/migrations. This pacman hook would go into the package and be installed automatically. Any updates to the package would trigger the hook in turn triggering the migrations.
The only problem seems to be that the hook would not be triggered (yet) upon initial installation of the package, because the hook itself would not yet be installed. Therefore, initial migrations would not run automatically but would require user action.

I'd appreciate any opinion on this idea! For example: Is that way of handling the packaging difficulties acceptable? Is providing and automatically installing a pacman hook too intrusive (and should be replaced with instructions on a dedicated archwiki page)?

Offline

#11 2020-11-08 16:26:49

eschwartz
Fellow
Registered: 2014-08-08
Posts: 4,097

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

pacman hooks are the equivalent of post_install scripts, but for things that need to be run after multiple packages. Which other packages need to trigger this migration???

I recommend using a post_install/post_upgrade script rather than a pacman hook. If it needs to run after sysusers.d, simply run "systemd-sysusers" and/or "systemd-tmpfiles --create" in the post_install first (and do chown etc.)

As I assume the papermerge-manage program is expected to run as the "papermerge" user, please execute it with "runuser -u papermerge papermerge-manage ..."

...

If it's appropriate to run these commands on the package upgrade but not on every start of the service, then they should never have been put in the service file to begin with.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#12 2020-11-10 08:43:50

amo
Member
Registered: 2016-01-05
Posts: 13

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

eschwartz wrote:

If it needs to run after sysusers.d, simply run "systemd-sysusers" and/or "systemd-tmpfiles --create" in the post_install first (and do chown etc.)"

Awesome, this is exactly the advise I needed, thank you so much! I did not know about these two commands. (Unfortunately, the archwiki does not mention them either in all articles about packaging.)

I did not manage to have the migrations install to a symlinked folder, but I guess it is not a big deal.
I will now test it for a while on my server, see if the next 2 or 3 updates install smoothly and eventually submit it to the AUR.

Offline

#13 2020-11-10 18:43:30

eschwartz
Fellow
Registered: 2014-08-08
Posts: 4,097

Re: [SOLVED] Need advise packaging "papermerge" django web app (python3)

They're not really mentioned because typically you don't need to run them in install scriptlets, and this package is an edge case.

If the migrations don't install to the symlinked folder, you could mv them after setup.py install.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

Board footer

Powered by FluxBB