You are not logged in.

#1 2017-07-14 16:54:41

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

[SOLVED] A little confusion over PKGBUILD files and python source

I'm a comp sci student and an arch user for the last 6 months. I've developed a piece of software that I'd like to submit to the AUR, so I've read the 'creating packages' guide on the wiki, but I'm a little confused on how exactly to use the PKGBUILD file because my software is entirely python. It consists of 3 .py files and a systemd .service file, and so installation only requires the system to check for dependencies and then move the .py files to /usr/bin and the .service file to ~/.config/systemd/user/. This is the first piece of software I've ever written, and I'm still relatively new at arch and linux in general, so please forgive me if this is something easy or obvious.

How should my PKGBUILD be formatted, considering I do not need to ./configure or make anything? How do I ensure that python module dependencies are installed (cprint)?

Thanks in advance.

Last edited by tgsachse (2017-07-14 21:43:10)

Offline

#2 2017-07-14 16:58:56

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

ALSO: Since this is entirely python, should I consider uploading to PyPI only and not developing a setup for AUR?

Offline

#3 2017-07-14 17:10:17

slithery
Member
Registered: 2013-12-01
Posts: 1,640

Re: [SOLVED] A little confusion over PKGBUILD files and python source

Just follow the instructions on the wiki for creating a PKGBUILD. You wont need  prepare() or build()  functions, just use the install command in your package() function to copy the files to their required location.

Edit - Fill in as much as you can of the PKGBUILD and then post it here so we can take a look.

Last edited by slithery (2017-07-14 17:11:12)

Offline

#4 2017-07-14 17:16:29

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 14,982

Re: [SOLVED] A little confusion over PKGBUILD files and python source

Welcome to Arch Linux.  If the software is going to be installed outside of the user's directory, then definitely make a package for it and install it with pacmanpacman should be cognizant of everything in /usr/binBut, a package should never, ever change something in the uses' home directory. Ever.  Put the .service file somewhere else -- maybe under /usr/share and tell the user about it by having your PKGBUILD have pacman post a message telling the user where the file is and where to copy it.

Your build() function will do nothing.  Your package() function will put the files in the proper place in the file system structure under pkg -- BUT DON'T MESS WITH /home/*.

Edit:

How do I ensure that python module dependencies are installed (cprint)

Put the package that contains it in the depends list

Last edited by ewaller (2017-07-14 17:19:34)


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
----
How to Ask Questions the Smart Way

Offline

#5 2017-07-14 17:36:12

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

It looks like I'll have to retool my software some then. I had planned on storing logs and a config file in a .dir in the user's home directory. Where should these be stored instead, as the config is user specific so I figured I should emulate what other programs do by creating dotfiles. For the logs I know I could use journal or something, so I will look into that more. My biggest problem is with the .service file. It is meant to enable my script to run as a daemon using this type of command: 'systemctl --user enable myscript.py' and so guides said that the .service file for systemctl needs to be in the ~/.config/systemd/user/ directory. How should I work around this? Could my program install it elsewhere on the system and then in my CLI setup script I could have the file copied into the user's home directory?

Offline

#6 2017-07-14 17:43:42

slithery
Member
Registered: 2013-12-01
Posts: 1,640

Re: [SOLVED] A little confusion over PKGBUILD files and python source

You should install the service file to /usr/lib/systemd/user/

https://wiki.archlinux.org/index.php/Sy … w_it_works

Offline

#7 2017-07-14 17:57:15

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 14,982

Re: [SOLVED] A little confusion over PKGBUILD files and python source

As far as the logs and stuff, that happens at run time.  That is fine.  It is just that pacman should not update the user's directory because (a) it is not aware of the contents of the user's directory, (b) no two user's directories are the same, and (c) the user does not expect that pacman will touch their directory.  Many things write configs and logs to the user's directory and is expected -- just not pacman.

The service file can be copied by the user to their local directory if and when they want it.  Once they have made a copy to their directory, they can even change it.   An example of this is /etc/skel.  There are seeds in there for .bashrc, .bash_logout, .screenrc, .... (Note they are all dot files, so you need the -a flag for ls to see them).  These files allow the user a starting point without clobbering any hand crafted files that exist in the user directory.  If you notify the user that it is up to them to copy the service file, that is enough.  We expect the user to read those messages and we do chastise those who do not in these forums wink

Last edited by ewaller (2017-07-14 17:59:01)


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
----
How to Ask Questions the Smart Way

Offline

#8 2017-07-14 17:58:37

Trilby
Forum Fellow
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 17,053
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

tgsachse wrote:

It looks like I'll have to retool my software some then. I had planned on storing logs and a config file in a .dir in the user's home directory.

No no, your python program/script can definitely put files in the user's home directory, this is expected.  I'd recommend checking $XDG_CONFIG_HOME and falling back on ~/.config if this is not set.  The above point was that the PKGBUILD cannot touch the user's home directory (nor should any post_install script or hook run by pacman for your package).  Once installed, your software can feel free to write files to the user's home directory.

tgsachse wrote:

... `systemctl --user enable myscript.py' and so guides said that the .service file for systemctl needs to be in the ~/.config/systemd/user/ directory. How should I work around this?

Don't work around it.  The user can enable your service, don't try to enable it for them.  The service file does not need to be in their home directory:
https://wiki.archlinux.org/index.php/Sy … w_it_works

Offline

#9 2017-07-14 18:18:03

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

Here's as far as I got with the PKGBUILD. I removed all of the example variables that I'm pretty sure I don't need, also I removed all the functions before package(), however it is possible that they need to be there even if they are empty, I'm unsure.

# Maintainer: tgsachse (Tiger Sachse) <tgsachse@gmail.com>
pkgname=Wind
pkgver=0.2.0
pkgrel=1

pkgdesc="Daemon and CLI interface for cloud storage management."
arch=('any')
url="https://github.com/tgsachse/Wind"
license=('GPL')
depends=('python>=3.3.0' 'rclone>=1.36')
source=("$pkgname-$pkgver.tar.gz"
        "$pkgname-$pkgver.patch")

package() {
	cd "$pkgname-$pkgver"
	make DESTDIR="$pkgdir/" install
}

Offline

#10 2017-07-14 18:20:05

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

Trilby wrote:

No no, your python program/script can definitely put files in the user's home directory, this is expected.  I'd recommend checking $XDG_CONFIG_HOME and falling back on ~/.config if this is not set.  The above point was that the PKGBUILD cannot touch the user's home directory (nor should any post_install script or hook run by pacman for your package).  Once installed, your software can feel free to write files to the user's home directory.

Oh alright, that makes sense. Thanks for the clarification.

Offline

#11 2017-07-14 18:47:34

Trilby
Forum Fellow
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 17,053
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

The other functions don't need to be there.  Do you include a Makefile with your python source?  That's great if you do, but not common.

The source array needs to include a url where the source can be retrieved from.  You cannot upload the source files themselves to the AUR.  Also, you don't have a patch, do you - just include the source actually used in the source array.

Please also test the PKGBUILD yourself - this will definitely not work yet.  (also your url is a 404 so I can't even check the source files to try to help).

EDIT: is this it?  Are you renaming it (Wind vs AutoClone)?

EDIT2: I'm guessing you were in the process of renaming this as I posted, and github hadn't caught up with the repo transfer yet.

Offline

#12 2017-07-14 19:04:15

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

I've been renaming the software this morning, just now renamed the github repo. Link should work now.
I'm not too sure what a Makefile is ;( I'm brand new to all of it.
What is the reasoning for using a patch? The link to the source should always be latest, so what does a patch link do?

EDIT: The files on github right now are still all named improperly. I've renamed my local files but have not uploaded them yet. I'm working on the code now so an upload should happen later today.

Last edited by tgsachse (2017-07-14 19:05:47)

Offline

#13 2017-07-14 19:16:45

Trilby
Forum Fellow
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 17,053
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

You don't need a patch.  I gather you just copied that from an example - get rid of it.  AUR packages can patch the upstream source if something from upstream needs to be changed to work with arch.  While not uncommon patching is not the norm.

A Makefile is needed if you want to use `make install` as you had in your package function.  That's why I suggested you should be trying out your own PKGBUILD - your package function will fail as you try to use `make` but there is not makefile.  As slithery said in post 3, just copy the files to where they need to be:

   mkdir -p ${pkgdir}/usr/bin
   cp myfile1.py ${pkgdir}/usr/bin/
   cp myfile2.py ${pkgdir}/usr/bin/

Copying with cp would work fine most of the time, and I include it as it's probably familiar and highlights how simple the package function is: it just needs to put files in the right place under ${pkgdir}, that's it.  However, while cp will be fine for this most of the time, I'd recommend `install` instead:

   install -Dm 0755 -t ${pkgdir}/usr/bin myfile1.py myfile2.py
   install -Dm 0644 myservice.service ${pkgdir}/usr/lib/systemd/user/

Offline

#14 2017-07-14 19:54:13

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

# Maintainer: tgsachse (Tiger Sachse) <tgsachse@gmail.com>
pkgname=Wind
pkgver=0.2.0
pkgrel=1

pkgdesc="Daemon and CLI interface for cloud storage management."
arch=('any')
url="https://github.com/tgsachse/Wind"
license=('GPL')
depends=('python>=3.3.0' 'rclone>=1.36')
source=('https://github.com/tgsachse/Wind.git')

package() {
	cd "$pkgdir"
	install -Dm 0777 -t ${pkgdir}/usr/bin wind wind_setup.py wind_daemon.py 
}

I ran

$makepkg --skipinteg

with the PKGBUILD listed above. It created the /usr/bin tree structure but it was unable to find the .py files to install. This is likely because I'm formatting the source=() wrong. In the source=() am I doing it correctly by putting the github clone link in the array? Or is something else supposed to go in there?

EDIT: I don't need to install a .service file anymore. Instead, when prompted in the setup utility within my program, the user can create the file him/herself based on a template.

Last edited by tgsachse (2017-07-14 19:55:29)

Offline

#15 2017-07-14 19:59:34

Trilby
Forum Fellow
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 17,053
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

There are two reasons it can't find your files.  You are right about the first.  See the wiki to see how to add your source link.  You're very close, but missing an important part at the start.

The second problem is in your package function you have `cd $pkgdir`.  $pkgdir is where the new package contents will be assembled - when makepkg starts, that is an empty directory.  So you are trying to install your source files by copying them from an empty directory to an empty directory.  You should cd to where your source files are - again see the examples in the wiki.

On a different note, the pkgname should be "Wind-git" as this builds from the current git source, and you should have a pkgver function - but we can worry about both of those points once you get the basics together.

On yet another note for probably further in the future, most python packages do not install all their files in /usr/bin - it gets a bit cluttered.  If users don't generally need to execute those .py files directly, they should be somewhere else like /usr/lib/python3.6/site-packages/Wind/ but this will also require some revision to your python code as it currently assumes the other .py files are in the same directory.

Offline

#16 2017-07-14 20:18:01

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

# Maintainer: tgsachse (Tiger Sachse) <tgsachse@gmail.com>
pkgname=Wind-git
pkgver=0.2.0
pkgrel=1

pkgdesc="Daemon and CLI interface for cloud storage management."
arch=('any')
url="https://github.com/tgsachse/Wind"
license=('GPL')
depends=('python>=3.3.0' 'rclone>=1.36')
source=('git+https://github.com/tgsachse/Wind.git')

package() {
	cd Wind
	install -Dm 0777 -t ${pkgdir}/usr/bin wind wind_setup.py wind_daemon.py 
}

When I run this the makepkg command completes successfully and creates a Wind-git-0.2.0-1-any.pkg.tar.xz file!
I ran

 sudo pacman -U Wind-git-0.2.0-1-any.pkg.tar.xz

and it installed! This is awesome.

So I think I understand now, the AUR is a repo full of PKGBUILDs, which you download and run with makepkg, then makepkg downloads the source and builds as told by the PKGBUILD, and finally pacman, the package manager, takes the newly built package and installs it onto the system.

Offline

#17 2017-07-14 20:20:26

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

Trilby wrote:

On yet another note for probably further in the future, most python packages do not install all their files in /usr/bin - it gets a bit cluttered.  If users don't generally need to execute those .py files directly, they should be somewhere else like /usr/lib/python3.6/site-packages/Wind/ but this will also require some revision to your python code as it currently assumes the other .py files are in the same directory.

I figured, since some projects could have a ton of .py files. I will move them into a better location soon.

Offline

#18 2017-07-14 20:26:22

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

So here's a nub question. When I upgrade the software and I upload changes to git, I assume I'll have to update the PKGBUILD with AUR and that's how pacman knows when to upgrade? Will my package upgrade with

pacman -Syyu

like other packages or does it not upgrade from that function because it's from the AUR.

Offline

#19 2017-07-14 21:02:07

Trilby
Forum Fellow
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 17,053
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

Congrats on a working PKGBUILD.  There are a few more steps before it's ready for upload to the AUR, but having it working is a rewarding milestone.

But to answer your question: pacman does not updated AUR packages.  You need to rebuild them yourself with makepkg.  Once you get the hang of how the AUR works, you can look into using an AUR helper which will help streamline the process - but I'd suggest waiting a bit for that.  It is important that you have a reasonable understanding of how the AUR works with just makepkg first.

tgsachse wrote:

So I think I understand now, the AUR is a repo full of PKGBUILDs, which you download and run with makepkg, then makepkg downloads the source and builds as told by the PKGBUILD, and finally pacman, the package manager, takes the newly built package and installs it onto the system.

Exactly.  I've written elsewhere that the AUR is basically just a recipe book.  In your case, you are both writing the recipe and (writing a PKGBUILD for packaging) and you are the farmer growing the ingredients (editing the python code on github).  In other words, you are both the packager and upstream developer.  Frequently AUR packagers are just the packager and the source code comes from somewhere else all together.  (This is just a tangential note to put into context some parts of the process that might seem confusing as you are in both roles at the moment).

Also, the nice thing about VCS PKGBUILD's (like your that pulls from git) is that you generally don't need to change anything in the PKGBUILD to update your installed Wind version.  Just cd to the directory where you have the PKGBUILD, and run makepkg again.  You can also see `man makepkg` to learn about the options it can take: one you'll get quick use out of is the -i flag `makepkg -i` will build your package and hand it off to pacman to install.  So when you make changes on git, just `makepkg -i` and the newest code will be installed (instead of both `makepkg` and `pacman -U ....`).

One note on the above, though, for now you'll need to use `makepkg -fi`.  This is because you don't yet have a pkgver function, so makepkg can't tell that there is new stuff on git, so it just stops as you've already built version 0.2.0 of the package.  The -f flag tells it to rebuild it anyways.

Next steps:
1) check the wiki on how to add a pkgver function.
2) remove the versioned dependencies if they aren't needed (remove the ">=" version numbers).
3) if/when you revise the code on github for the .py files to be under /usr/lib you'll need to also revise the package function so it installs them to the right place.
  3A) you could future proof this by making a simple makefile - so the package function would just call `make` and what the Makefile actually does can be determined by your Makefile stored on github ... but this might be for later down the road as learning Makefile syntax (as simple as it really is) might not need to be piled on top of this just yet.

Tangential moderator note: please use the "edit" button to add new information to posts if no one else has posted.  This is preferred to bumping your thread repeatedly.

Offline

#20 2017-07-14 21:42:33

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

Thanks so much for the help, I'll mark the post as solved. Also once I saw I posted 3 times I realized I probably shoulda made it all one post. Will do so going forward. Thanks again.

Offline

#21 2017-07-17 11:06:55

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 5,545

Re: [SOLVED] A little confusion over PKGBUILD files and python source

PKGBUILD is looking good, i see 1 minor issue :

license=('GPL')

On archlinux GPL means GPL v2 or any later version .
The README.md file on github mentions GPL v3 as license, but you haven't included the license-file itself.


Booting with apg Openrc, NOT systemd.
Automounting : not needed, i prefer pmount
Aur helpers : makepkg + my own local repo === rarely need them

Offline

#22 2017-07-18 19:43:24

tgsachse
Member
From: Florida
Registered: 2017-07-14
Posts: 12
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

Hello again, I have some updates on the project and a new question regarding integrity checks. I've also renamed the package again due to a potential conflict with uploading to PyPI. First, the latest PKGBUILD and repo link:

# Maintainer: tgsachse (Tiger Sachse) <tgsachse@gmail.com>
pkgname=viento-git
pkgver=0.3.0
pkgrel=1

pkgdesc="Daemon and CLI interface for cloud storage management."
arch=('any')
url="https://github.com/tgsachse/viento"
license=('GPLv3')
depends=('python' 'rclone' 'python-termcolor')
source=('git+https://github.com/tgsachse/viento.git')

pkgver(){
    cd viento
    git describe --long | sed 's/^v-//;s/\([^-]*-g\)/r\1/;s/-/./g'
}


package() {
    cd viento
	install -Dm 0777 -t ${pkgdir}/usr/bin viento
    install -Dm 0777 -t "${pkgdir}/usr/lib/python3.6/site-packages" viento_s.py viento_d.py 
}

To address Lone_Wolf, I've corrected the license variable to be 'GPLv3' and I've added a LICENSE.md in the github repo. I've added a pkgver() function, revised the dependencies and changed where most of the program is installed to (now with other python packages). I have 2 questions however where I'm still a little confused.

1) How do I implement validity checks? I understand the concepts behind checksums and hashes and such for the most part, but I have very little experience generating those sums. Once generated, do I put a sums file in the github repo and then add some sort of variable to the PKGBUILD? When I read about it on the wiki it still confuses me as to exactly what I must put in the PKGBUILD.
2) Is there anything else that stands out that I must change before this is ready for the AUR? I'll give the guidelines a once-over again in the meantime, but I think I'm pretty close.

Thanks everyone.

EDIT: Just made a .SRCINFO file.

Last edited by tgsachse (2017-07-18 19:53:51)

Offline

#23 2017-07-18 20:04:31

Trilby
Forum Fellow
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 17,053
Website

Re: [SOLVED] A little confusion over PKGBUILD files and python source

For non-vcs PKGBUILDs, you can use the makepkg -g flag (see the man page): `makpekg -g >> PKGBUILD` will append the checksum to the PKGBUILD.  But for -git packages:

wiki wrote:

Because the sources are not static, skip the checksum in md5sums=() by adding 'SKIP'

I suppose that is a bit odd without an example, but there are ample examples out there you could check which all have something like:

source=("git+https://...")
md5sums=('SKIP')

As for the site-packages path, you should use a flexible approach to get the proper site-packages path rather than hard-coding them.  This differs by distro and python version.  I'm crap with python, but it seems these lines in your code will get you the path you need to pass to your os.system() calls:

import site
site.getsitepackages()[0]

The second will give you the leading directory up to where the .py files should be.

You'd also need to get this path in the PKGBUILD.  I'm pretty sure there are much better ways that python users could suggest here, but the following hack will work:

   install -Dm 0777 -t "${pkgdir}/$(python -c "import site; print(site.getsitepackages()[0])")" viento_s.py viento_d.py

I suspect this is what things like python's setup tool(s) help with.

Offline

#24 2017-07-19 00:49:33

Eschwartz
Bug Wrangler
Registered: 2014-08-08
Posts: 1,092

Re: [SOLVED] A little confusion over PKGBUILD files and python source

1) You failed to quote $pkgdir in one location.

2) For the love of G-d, please use a setup.py file to install this, it will make things so much easier and no need to do inline python scripts to find the right installation directory.

3) The license should really be GPL3 without the "v", to match the name in /usr/share/licenses/

4) As soon as pacman 5.1 is released, makepkg will be able to verify GPG-signed git repositories. In preparation for this, consider enabling `git config --global commit.gpgsign true`, and when pacman 5.1 is released you will add "?signed" to the end of the git clone URL in the source array, and add your GPG key fingerprint to the validpgpkeys array.
This helps for proving authorship of the code, which checksums don't really do (or rather, they do so in an inefficient, error-prone way). Checksums really only protect against file download corruption, and git has built-in error checking.

Offline

Board footer

Powered by FluxBB