You are not logged in.

#1 2020-05-30 18:00:40

kinru
Member
From: East Coast USA
Registered: 2019-03-23
Posts: 69

Problem with dependencies on a python package

Not too long ago, I submitted an AUR Package and to my knowledge it was functional. However, I got a comment the brought my attention to the package's dependencies. The package was functional on my PC, but I made a fresh Arch VM which I then installed the package onto. It did install fine, but when I ran the program it has the error:

pkg_resources.DistributionNotFound: The 'lxml>=4.5.0' distribution was not found and is required by wifipumpkin3

and if I install the package python-lxml it has the same error with another package.

I added all the packages required in the Install Guide to the PKGBUILD. There were several of those. That PKGBUILD was good on my pc.

I've previously communicated with upstream about the dependencies and optional dependencies. He told me (he is difficult to understand)

so, in requirements.txt anda the guide of installation tools
iptables
net-tools
wireless-tools
Hostapd
net-tools only ifconfig command , from wireless-tools only iw command

python3 setup.py install, will be install everything in requirements.txt and make somes config
Frist install this

iptables
net-tools
wireless-tools
Hostapd
@mh4x0f
After run python3 setup.py install

There are many dependencies listed in the requirements.txt. I now know the package doesn't work without (most of?) them. Its a mystery to me as to how all the dependencies were installed on my pc (maybe a previously installed package enabled this?). So, when I found out the PKGBUILD didnt work on the VM, I went about collecting package names for all the dependencies in dependencies.txt. The problem is, several of the dependencies there aren't available in Arch Packages or AUR packages, only via pip. So what would be the correct way to add the dependencies? As far as I know, it is not correct for the setup.py to install a bunch of depends thru pip.

Offline

#2 2020-05-30 19:13:21

loqs
Member
Registered: 2014-03-06
Posts: 10,969

Re: Problem with dependencies on a python package

Is the following a rough summary of the issues?
isc_dhcp_leases,  Responder3,  beautifultable,  scapy_http, dhcplib are unpackaged
python2-bs4 has no python build

package               required version    shipped version
python-dnslib         0.9.7               0.9.12-1
python-pyopenssl      19.0.0              19.1.0
python-twisted        19.7.0              20.3.0
python-urwid          1.1.2               2.1.0
python-tabulate       0.8.5               0.8.7
python-configparser   4.0.2               5.0.0
python-ipaddress      1.0.22              1.0.23

The rest of the versioned dependencies look met to me on a brief inspection.
Edit:
Are the unexplained files tracked by pacman and or pip?

Last edited by loqs (2020-05-30 20:08:32)

Offline

#3 2020-05-31 03:29:56

kinru
Member
From: East Coast USA
Registered: 2019-03-23
Posts: 69

Re: Problem with dependencies on a python package

Sorry for the slow reply,
Here is a table:

pip name          pip installed          arch name                    arch installed          required version         arch shipped version

(listed in the requirements.txt)
netifaces          yes                   python-netifaces             no                      0.10.9                   0.10.9-3
netaddr            yes                   python-netaddr               no                      0.7.19                   0.7.19-5
dhcplib            yes                   (none)                       no                      0.1.1                    N/A
ipaddress          yes                   python2-ipaddress            no                      1.0.22                   1.0.23-2
scapy_http         yes                   (none)                       no                      1.8.2                    N/A
configparser       yes                   python2-configparser         yes                     4.0.2                    4.0.2-2
tabulate           yes                   python-tabulate              no                      0.8.5                    0.8.7-1
beautifultable     yes                   (none)                       no                      0.8.0                    N/A
urwid              yes                   python-urwid                 no                      1.1.2                    2.1.0-2
termcolor          yes                   python-termcolor             no                      1.1.0                    1.1.0-8
twisted            yes                   python-twisted               no                      19.7.0                   20.3.0-1
pyqt5              yes                   python-pyqt5                 yes                     >5.14                    5.14.2-1
pyqt5-sip          yes                   python-pyqt5-sip             yes                     (unspecfied)             12.7.2-1
pyopenssl          yes                   python-pyopenssl             no                      19.0.0                   19.1.0-2
asn1crypto         yes                   python-asn1crypto            no                      >1.0.0                   1.3.0-1
responder3         yes                   (none)                       no                      0.0.1                    N/A
bs4                yes                   (none, dummy pkg)            no                      0.0.1                    N/A
dnslib             yes                   python-dnslib                no                      0.9.7                    0.9.12-1
loguru             yes                   (none)                       no                      >0.4.0                   N/A
scapy              yes                   scapy                        no                      2.4.3                    2.4.3-2
isc_dhcp_leases    yes                   (none)                       no                      0.9.1                    N/A
dnspython          yes                   python-dnspython             no                      1.16.0                   1.16.0-3
flask              yes                   python-flask                 no                      >1.1.1                   1.1.2-2
lxml               yes                   python-lxml                  no                      >4.5.0                   4.5.1-1
requests           yes                   python-requests              yes                     >2.18.4                  2.23.0-1

And by "installed" I mean that either sudo pacman -Qs package or pip show package return that the package is installed thru that method.

By the table, the packages dhcplib, scapy-http, beatifultable, responder3, loguru, and isc_dhcp_leases have no Arch packages.

Also it can be seen from here that bs4 is a dummy package for beautifulsoup4 which does actually have an Arch package, python-beautifulsoup4.

There were a few more which were from the installation guide and from communications with upstream, you can see them in the PKGBUILD

I didn't manually install any (I don't think) of the packages installed via pip. I think they are installed in the setup.py where it says

with open("requirements.txt") as fp:
    required = [line.strip() for line in fp if line.strip() != ""] 

I don't have any knowledge of python, so I can only assume this gets all lines in dependencies.txt and sets them as deps.

Furthermore, I was told that it was a VCS git package. What should I do? I would like to avoid doing very frequent maintenance.

As a bit of a meta-question, is there a way to make tables on the arch forums easier? It is inefficient manual work for me to line up all the columns due to the website displaying a monospace font in the final post and a proportional font in the drafting box.

Offline

#4 2020-05-31 11:15:16

loqs
Member
Registered: 2014-03-06
Posts: 10,969

Re: Problem with dependencies on a python package

If you did not intend the package to be a git package you could pin the git source to the latest release or replace the git source entry with the release tarball.

The preferable solution for the missing packages would be for you to create new AUR packages for them.

For the packages with incompatible versions if you alter requirements.txt so the versions are accepted can you detect any issues using the application?

For bs4 ask upstream to change to beautifulsoup4,  you could alter requirements.txt in the PKGBUILD.

I had the same issue creating a table.

Offline

#5 2020-05-31 20:33:42

kinru
Member
From: East Coast USA
Registered: 2019-03-23
Posts: 69

Re: Problem with dependencies on a python package

I'm not sure I understand what you mean. Are you saying that I should build the package and add the tarball to the package's AUR git repo? I thought that the AUR was only a means to share PKGBUILDs which just download the source code and build it.

I am not sure I am interested in creating many new AUR packages. Seems like a pain to maintain, but I might. However, (I think) the setup.py installs all the deps via pip even if they're already downloaded. Would I need to make a patch or something to stop this?

I asked upstream to change it and I'm awaiting a response. Hopefully there is no need to patch on my end? (If I understand correctly it is possible for a 'patch' to be included in the AUR which would change the content of the files after downloading.)

sad - I think its possible to get a browser plugin to toggle fonts. This could be used to temporarily display a monospace font for the submission box so that it is easy to make the table.

Offline

#6 2020-05-31 20:50:05

loqs
Member
Registered: 2014-03-06
Posts: 10,969

Re: Problem with dependencies on a python package

source=("git+https://github.com/P0cL4bs/wifipumpkin3.git/")

The source entry is the git repository,  the checkout will be the default branch,  tracks upstream changes.

_commit=cd985184d471a85d0a7b1c826b93f798ef478772
source=("git+https://github.com/P0cL4bs/wifipumpkin3.git#commit=$_commit")

The source entry is the same git repository but the checkout will be the specified commit,  does not track upstream changes.

pkgver=1.0.5R1
source=("$pkgname-$pkgver.tar.gz::https://github.com/P0cL4bs/$pkgname/archive/v$pkgver.tar.gz")

The source entry uses the tarball provided by upstream,  like the pinned commit does not track upstream changes.
The tarball is renamed to include the package name so it will not collided on systems using a shared SRCDEST directory.

Offline

#7 2020-06-01 02:57:11

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,426

Re: Problem with dependencies on a python package

As I recall, I mentioned in #archlinux-aur when you were initially uploading this package, that you should investigate the dependencies in requirements.txt due to the fact that they were pinned, and either add them as PKGBUILD dependencies or remove them from the source code to prevent DistributionNotFound runtime errors. And discuss with upstream whether they're actually needed.

Last edited by eschwartz (2020-06-01 02:58:08)


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#8 2020-06-01 16:15:58

kinru
Member
From: East Coast USA
Registered: 2019-03-23
Posts: 69

Re: Problem with dependencies on a python package

Ok, I understand that for packages without -git suffixes, the provided per-version tarball is used and that pkgver should equal the latest 'stable' release. That's fine, but is it possible to (rather than manually get the stable pkgver for each release) get the pkgver of the master branch via a pkgver () function? I presume there is some reason this is not done. (or is it?)

I did investigate, however the developer didn't provide me any useful info. He only gave me instructions on installing, not anything on optional dependencies. What I can tell is that the setup.py probably installs a bunch of pip packages.  So, it seems that in a patch, either requirements.txt or the line

with open("requirements.txt") as fp:
    required = [line.strip() for line in fp if line.strip() != ""]

need to be removed. I'm currently following this wiki article about it.

About creating packages for the dependencies not covered by any Arch packages, I will look into it.

Offline

#9 2020-06-01 19:22:38

loqs
Member
Registered: 2014-03-06
Posts: 10,969

Re: Problem with dependencies on a python package

kinru wrote:

Ok, I understand that for packages without -git suffixes, the provided per-version tarball is used and that pkgver should equal the latest 'stable' release. That's fine, but is it possible to (rather than manually get the stable pkgver for each release) get the pkgver of the master branch via a pkgver () function? I presume there is some reason this is not done. (or is it?)

That would make it a -git package.
You can pin to a commit and have the pkgver function update the pkgver variable after you have updated the commit such as is done by https://git.archlinux.org/svntogit/pack … ages/glib2

kinru wrote:

What I can tell is that the setup.py probably installs a bunch of pip packages.

If setup.py was installing other packages they would be present in the archive.  Also the original error indicates the packages are not installed as a result of the package being installed.

pkg_resources.DistributionNotFound: The 'lxml>=4.5.0' distribution was not found and is required by wifipumpkin3

Last edited by loqs (2020-06-01 19:28:58)

Offline

#10 2020-06-02 14:35:55

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 8,197

Re: Problem with dependencies on a python package


Multi-init booting with apg Openrc and systemd coexisting
Automounting : not needed, i prefer pmount
Aur helpers : makepkg + my own local repo === rarely need them

Offline

#11 2020-06-03 15:13:24

kinru
Member
From: East Coast USA
Registered: 2019-03-23
Posts: 69

Re: Problem with dependencies on a python package

The reason I am saying

What I can tell is that the setup.py probably installs a bunch of pip packages.

is because the dev wrote:

mh4x0f wrote:

python3 setup.py install, will be install everything in requirements.txt and make somes config

So, I amended the PKGBUILD to be this now:

pkgname=wifipumpkin3
pkgver=v1.0.5R1
pkgrel=4
pkgdesc="Framework for Rogue Wi-Fi Access Point Attacks"
arch=('x86_64')
url="https://github.com/P0cL4bs/wifipumpkin3"
license=('Apache')
depends=('openssl' 'python>=3.0.0' 'libffi' 'python-pyqt5' 'hostapd' 'iptables' 'iw' 'net-tools' 'wireless_tools' 'python-setuptools' 'python-pip' )
makedepends=('git')
source=("$pkgname-$pkgver.tar.gz::https://github.com/P0cL4bs/$pkgname/archive/v$pkgver.tar.gz" '.install')
sha1sums=('SKIP')

build() {
	cd "$srcdir/$pkgname"
	python setup.py build	
}

package() {
	cd "$srcdir/$pkgname"
	python setup.py install --root="$pkgdir" --optimize=1 --skip-build
}

I want to confirm that I have repaired the issue with it being a -git type package. As you can see, the dependencies still haven't been finalized. I'm still communicating with upstream about deps, especially about bs4. Also, is the git package needed for makedepends?

Yes, I am aware of the Python Packing Guidelines wiki and I'm trying to follow it, however the issue of the setup.py (supposedly?) installing packages is making this complicated.

Offline

#12 2020-06-03 15:39:41

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,426

Re: Problem with dependencies on a python package

'python setup.py install' doesn't install dependencies.

'pip install' does, unless you use --no-deps, and that's why the Python Package guidelines utterly forbids the use of pip install unless you do, in fact, use --no-deps.

Because packages are not ALLOWED to package both a module and its dependencies.

...

makedepends on git is usually needed if your source is a "git+https://github.com/user/repo.git" clone url. If you're using tarball archives, then unless the Makefile or setup.py uses git also (doubtful) you do not need to makedepends on git.

(I have a package that uses yarn to download nodejs modules, and yarn needs git in order to download modules that are specified with git. This package does just that. So I makedepend on git. It's gross, but then, anything contaminated by nodejs is gross.)


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#13 2020-06-03 16:12:46

kinru
Member
From: East Coast USA
Registered: 2019-03-23
Posts: 69

Re: Problem with dependencies on a python package

people from uptstream have replied:

mx4x0f wrote:

yes, but the wp3 import as bs4, if you find the all word "bs4" on project and change for beautifulsoup4 will be work fine without problems

as well as

cjb900 wrote:

well if you look on the site https://wifipumpkin3.github.io/ you will not see Arch linux listed meaning that it has not been tested to properly work on arch. its only been tested on debian based OS's and docker. The only thing i can suggest like @mh4x0f mentioned is to replace everything in the code that says bs4 with beautifulsoup4

Replacing all instances of "bs4" with "beautifulsoup4" sounds inefficient- but maybe it is made easy with a command like:

find . \( -type d -name .git -prune \) -o -type f -print0 | xargs -0 sed -i 's/bs4/beautifulsoup4/g'

  which was found here: stackoverflow The command ignores files with .git in their names, so, I think it is okay to run. However, I don't know if this is necessary, since the wifipumpkin/package.new directory's .git files are separate from those in the wifipumpkin directory.

So, I ran this command within the package.new directory (as outlined here) and I checked in the requirements.txt, where bs4 has actually been replaced with beautifulsoup4, but I don't know if all instances of bs4 in all the code were replaced.

In order to troubleshoot deps, does following Building in a clean chroot prevent my personally installed packages from potentially fouling testing? I'd like to avoid using a vm again.

From Python_package_guidelines#Installation_methods:

However, for managing Python packages from within PKGBUILDs, the standard-provided distutils proves to be the most convenient solution since it uses the downloaded source package's setup.py and easily installs files under $pkgdir/usr/lib/python<python version>/site-packages/$pkgname directory.
Note: Dependencies from setup.py must be defined in the depends array otherwise they will not be installed.

Is the above stating that distutils will handle the deps for me, as long as they are defined in depends? If this is true, must all the deps arch packages or can they be pip ones? It's been shown not all the deps have an arch package.

eschwartz wrote:

makedepends on git is usually needed if your source is a "git+https://github.com/user/repo.git" clone url. If you're using tarball archives, then unless the Makefile or setup.py uses git also (doubtful) you do not need to makedepends on git.

Ok, I have read through the setup.py and ran

cat setup.py | grep git 

and found the only mention to git to be within the project's url. Therefore, I will remove git from makedepends, unless there is some objection.

Offline

#14 2020-06-03 17:38:33

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,426

Re: Problem with dependencies on a python package

kinru wrote:

people from uptstream have replied:

mx4x0f wrote:

yes, but the wp3 import as bs4, if you find the all word "bs4" on project and change for beautifulsoup4 will be work fine without problems

as well as

cjb900 wrote:

well if you look on the site https://wifipumpkin3.github.io/ you will not see Arch linux listed meaning that it has not been tested to properly work on arch. its only been tested on debian based OS's and docker. The only thing i can suggest like @mh4x0f mentioned is to replace everything in the code that says bs4 with beautifulsoup4

Replacing all instances of "bs4" with "beautifulsoup4" sounds inefficient- but maybe it is made easy with a command like:

find . \( -type d -name .git -prune \) -o -type f -print0 | xargs -0 sed -i 's/bs4/beautifulsoup4/g'

  which was found here: stackoverflow The command ignores files with .git in their names, so, I think it is okay to run. However, I don't know if this is necessary, since the wifipumpkin/package.new directory's .git files are separate from those in the wifipumpkin directory.

So, I ran this command within the package.new directory (as outlined here) and I checked in the requirements.txt, where bs4 has actually been replaced with beautifulsoup4, but I don't know if all instances of bs4 in all the code were replaced.

Where did they say this? Do they even know how to code in the first place? That's one of the stupidest things I've ever heard.

the PyPI project for bs4 wrote:

This is a dummy package managed by the developer of Beautiful Soup to prevent name squatting. The official name of PyPI’s Beautiful Soup Python package is beautifulsoup4. This package ensures that if you type pip install bs4 by mistake you will end up with Beautiful Soup.

And here is some metadata from the bs4 python module.

$ cat /usr/lib/python3.8/site-packages/beautifulsoup4-4.9.1-py3.8.egg-info/top_level.txt
bs4
$ pacman -Qql python-beautifulsoup4
/usr/
/usr/lib/
/usr/lib/python3.8/
/usr/lib/python3.8/site-packages/
/usr/lib/python3.8/site-packages/beautifulsoup4-4.9.1-py3.8.egg-info/
/usr/lib/python3.8/site-packages/beautifulsoup4-4.9.1-py3.8.egg-info/PKG-INFO
/usr/lib/python3.8/site-packages/beautifulsoup4-4.9.1-py3.8.egg-info/SOURCES.txt
/usr/lib/python3.8/site-packages/beautifulsoup4-4.9.1-py3.8.egg-info/dependency_links.txt
/usr/lib/python3.8/site-packages/beautifulsoup4-4.9.1-py3.8.egg-info/requires.txt
/usr/lib/python3.8/site-packages/beautifulsoup4-4.9.1-py3.8.egg-info/top_level.txt
/usr/lib/python3.8/site-packages/bs4/
/usr/lib/python3.8/site-packages/bs4/__init__.py
/usr/lib/python3.8/site-packages/bs4/__pycache__/
/usr/lib/python3.8/site-packages/bs4/__pycache__/__init__.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/__pycache__/__init__.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/__pycache__/dammit.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/__pycache__/dammit.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/__pycache__/diagnose.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/__pycache__/diagnose.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/__pycache__/element.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/__pycache__/element.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/__pycache__/formatter.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/__pycache__/formatter.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/__pycache__/testing.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/__pycache__/testing.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/builder/
/usr/lib/python3.8/site-packages/bs4/builder/__init__.py
/usr/lib/python3.8/site-packages/bs4/builder/__pycache__/
/usr/lib/python3.8/site-packages/bs4/builder/__pycache__/__init__.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/builder/__pycache__/__init__.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/builder/__pycache__/_html5lib.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/builder/__pycache__/_html5lib.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/builder/__pycache__/_htmlparser.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/builder/__pycache__/_htmlparser.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/builder/__pycache__/_lxml.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/builder/__pycache__/_lxml.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/builder/_html5lib.py
/usr/lib/python3.8/site-packages/bs4/builder/_htmlparser.py
/usr/lib/python3.8/site-packages/bs4/builder/_lxml.py
/usr/lib/python3.8/site-packages/bs4/dammit.py
/usr/lib/python3.8/site-packages/bs4/diagnose.py
/usr/lib/python3.8/site-packages/bs4/element.py
/usr/lib/python3.8/site-packages/bs4/formatter.py
/usr/lib/python3.8/site-packages/bs4/testing.py
/usr/lib/python3.8/site-packages/bs4/tests/
/usr/lib/python3.8/site-packages/bs4/tests/__init__.py
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/__init__.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/__init__.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_builder_registry.cpython-38-pytest-5.4.2.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_builder_registry.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_builder_registry.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_docs.cpython-38-pytest-5.4.2.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_docs.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_docs.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_html5lib.cpython-38-pytest-5.4.2.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_html5lib.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_html5lib.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_htmlparser.cpython-38-pytest-5.4.2.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_htmlparser.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_htmlparser.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_lxml.cpython-38-pytest-5.4.2.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_lxml.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_lxml.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_soup.cpython-38-pytest-5.4.2.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_soup.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_soup.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_tree.cpython-38-pytest-5.4.2.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_tree.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/bs4/tests/__pycache__/test_tree.cpython-38.pyc
/usr/lib/python3.8/site-packages/bs4/tests/test_builder_registry.py
/usr/lib/python3.8/site-packages/bs4/tests/test_docs.py
/usr/lib/python3.8/site-packages/bs4/tests/test_html5lib.py
/usr/lib/python3.8/site-packages/bs4/tests/test_htmlparser.py
/usr/lib/python3.8/site-packages/bs4/tests/test_lxml.py
/usr/lib/python3.8/site-packages/bs4/tests/test_soup.py
/usr/lib/python3.8/site-packages/bs4/tests/test_tree.py

The "beautifulsoup4" project name provides:
- the "beautifulsoup4-4.9.1-py3.8" egg-info metadata (name: beautifulsoup4, version: 4.9.1, python interpreter: py3.8)
- the "bs4" importable python module (import bs4, bs4.builder, bs4.dammit, bs4.diagnose, bs4.element, bs4.formatter, bs4.testing ).

Do they really claim they've tested their code on Debian? If so, they're lying. Debian provides this package: https://packages.debian.org/buster/all/ … 4/filelist

Which once again is the "beautifulsoup4" project name and pip requirement, which provides the "bs4" importable toplevel module.

"its only been tested on debian based OS's and docker"

Well, if they've only tested it in docker, then maybe they're not lying, they're simply fools who never tested it with debian's packages. But they should stop speaking FUD about how "it's tested to work on debian" when they are NOT using debian. Docker is not debian. Virtualenvs are not debian either. It is completely irrelevant which LINUX KERNEL they were BOOTED into when they tested their virtualenv.

Please tell me where this discussion happened. It's not in github issues. I would like to discuss this with them.

kinru wrote:

In order to troubleshoot deps, does following Building in a clean chroot prevent my personally installed packages from potentially fouling testing? I'd like to avoid using a vm again.

A VM, or a clean chroot, will both let you use a container environment to test the package. Note in order to test the package using the clean chroot, you should use the "arch-nspawn" program to enter the chroot environment.

kinru wrote:

From Python_package_guidelines#Installation_methods:

However, for managing Python packages from within PKGBUILDs, the standard-provided distutils proves to be the most convenient solution since it uses the downloaded source package's setup.py and easily installs files under $pkgdir/usr/lib/python<python version>/site-packages/$pkgname directory.
Note: Dependencies from setup.py must be defined in the depends array otherwise they will not be installed.

Is the above stating that distutils will handle the deps for me, as long as they are defined in depends? If this is true, must all the deps arch packages or can they be pip ones? It's been shown not all the deps have an arch package.

Distutils does not handle deps. setuptools does not handle deps. The PyPA recommends you use pip install, because pip install handles deps.

pip install --no-deps does not handle deps.

The only thing which is allowed to handle deps in a pacman package is *pacman*. Only one tool can be in charge. This is EITHER pacman OR pip.

pip is a package manager. It is a *competitor* of pacman.

Distutils, setuptools, and pip install --no-deps will install metadata that declares which python dependencies must exist. pacman's job is to install those python dependencies via the "depends" keyword.

Any package which does not install all dependencies *as arch packages*, is a broken package. You cannot use pip for this. If you want to use pip for this, you could just "pip install wifipumpkin3", but then it's not handled by pacman and will be in $HOME/.local/lib/python3.8/site-packages/, which may or may not be what you want.

Last edited by eschwartz (2020-06-03 17:41:13)


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#15 2020-06-03 18:28:53

kinru
Member
From: East Coast USA
Registered: 2019-03-23
Posts: 69

Re: Problem with dependencies on a python package

As you ask: the discussion took place on their discord server. (Invitation Link) This link can be found at https://github.com/P0cL4bs/wifipumpkin3 under the community section of README.md. In order to communicate via discord an account is needed. The discussion is within the #development channel. I guess it is good I didn't commit & push the changes yet, bc I have created a patch that changes all bs4 to beautifulsoup4. In fact, here they recommend Ubuntu. Then, the procedure is called "Debian/Ubuntu"

Ok, since all deps in a package must be arch packages, I guess I will start work on those missing packages. As a sidenote:

pip search wifipumpkin3

returns no output, therefore I believe there package is not available thru pip.

About bs4: (It should be known that I have 0 python knowledge) If I am correctly interpreting the information, bs4 is a package that is meant to install beautifulsoup4 (similar to an arch package group but with only one package?) but in reality, bs4 provides some independent files. What adds to the wackness is that I find the packages bs4 on ubuntu and debian (are they the same packages?) always provide more files than the beautifulsoup packages. bs4 debian, beautfulsoup debian, bs4 ubuntu, beautifulsoap ubuntu.

Offline

#16 2020-06-03 18:55:27

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,426

Re: Problem with dependencies on a python package

kinru wrote:

As you ask: the discussion took place on their discord server. (Invitation Link) This link can be found at https://github.com/P0cL4bs/wifipumpkin3 under the community section of README.md. In order to communicate via discord an account is needed. The discussion is within the #development channel. I guess it is good I didn't commit & push the changes yet, bc I have created a patch that changes all bs4 to beautifulsoup4. In fact, here they recommend Ubuntu. Then, the procedure is called "Debian/Ubuntu"

Discord again, lol.

kinru wrote:

Ok, since all deps in a package must be arch packages, I guess I will start work on those missing packages. As a sidenote:

pip search wifipumpkin3

returns no output, therefore I believe there package is not available thru pip.

That would be best, yes. Or figure out if some of them aren't strictly needed, I guess.

As for pip search, it's possible that wifipumpkin3 wasn't uploaded to the PyPI. However, pip can install from many different sources, see "man pip-install" for details. So you could still do:

pip install https://github.com/P0cL4bs/wifipumpkin3/archive/dev.zip
kinru wrote:

About bs4: (It should be known that I have 0 python knowledge) If I am correctly interpreting the information, bs4 is a package that is meant to install beautifulsoup4 (similar to an arch package group but with only one package?) but in reality, bs4 provides some independent files. What adds to the wackness is that I find the packages bs4 on ubuntu and debian (are they the same packages?) always provide more files than the beautifulsoup packages. bs4 debian, beautfulsoup debian, bs4 ubuntu, beautifulsoap ubuntu.

It's understandable if you don't understand these nuances. smile I'm more alarmed, however, that the wifipumpkin3 dev doesn't either.

So here's the thing.

python-beautifulsoup 3.2.1 is one package, which provides:
- BeautifulSoup-3.2.1.egg-info
- BeautifulSoup.py

python3-bs4 4.7.1 is another package which provides:
- beautifulsoup4-4.7.1.egg-info
- bs4/

Upstream Beautiful Soup went from major version 3 to major version 4, and changed the import name from "Beautifulsoup" to "bs4", then lowercased the PyPI name, and added a 4 to the end.

So, "Beautifulsoup" is version 3, and has the same *import* name and *project* name.

But "beautifulsoup4" is version 4, and has a different import name (bs4) and project name (beautifulsoup4).

By coincidence, Debian packages beautifulsoup4 using an "apt install" package name which matches the import name rather than the project name, which seems a bit odd IMO, but I'm not a debian developer so what do I know?


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#17 2020-06-20 15:13:57

kinru
Member
From: East Coast USA
Registered: 2019-03-23
Posts: 69

Re: Problem with dependencies on a python package

Sorry for the very long delay,

The dev branch is ahead of the master branch. It contains some of the useful advancements that were made, such as changing bs4 for beautifulsoup4. I'm supposed to be packaging master (correct?) and I can't because of bs4 still being in the master branch requirements.txt. Most of the deps are also a very specific version which I'm not sure is necessary.

Also see this issue, scapy-http is deprecated, and scapy has included those features once provided by scapy-http. Can be seen in this pull. I don't want to package scapy-http since its deprecated. According to the scapy-http README.md even the syntax is the same. Shouldn't it be as simple as removing scapy-http from requirements.txt?

Last edited by kinru (2020-06-20 15:15:17)

Offline

#18 2020-06-21 17:46:58

eschwartz
Trusted User/Bug Wrangler
Registered: 2014-08-08
Posts: 3,426

Re: Problem with dependencies on a python package

"I'm wait for a stable version, like 2.4.4 for remove scapy_http."

I don't comprehend this, why is the wifipumpkin3 developer unhappy with the current stable scapy 2.4.3 version? What exactly is unstable about this stable release? Please request the issue be reopened.


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

#19 2020-06-26 17:04:30

kinru
Member
From: East Coast USA
Registered: 2019-03-23
Posts: 69

Re: Problem with dependencies on a python package

The issue is open but its on priority: low, status: review needed
It was reopened 5 days ago when the original poster of the issue asked.

Offline

Board footer

Powered by FluxBB