[Solved] Julia package

user28371 · 2022-12-20 20:40:57

Hello,
I have a question that's been bothering me. What is the deal with the Julia package? What is it for? It is unsupported by the upstream, and even basic things, such as creating a dictionary from the standard library sometimes crash. External things almost always crash.

Julia devs claim it's simply broken and they don't care about it. Yet someone is for some reason dedicating the time to keep this broken-by-design version up-to-date in the community repository. Why? There already is a working version in the "unsupported" AUR.

This seems like it just exists out of some expectation of consistency "this is how packages should work, so this is how it will be" despite it being broken. Is there another reason? Can we get the working version from aur in the community repo? Or at least a mention in the package description that it's broken? It says it on the wiki, but the wiki is extremely overused already. Why have a "trusted" repository when it can't be trusted to work? Even not having the broken package at all may be an improvement...

This just seems like a trap for unsuspecting noobs.

Thanks

Last edited by user28371 (2022-12-20 21:12:27)

arojas · 2022-12-20 20:51:33

> It is unsupported by the upstream, and even basic things, such as creating a dictionary from the standard library sometimes crash. External things almost always crash.

Please post links to the bug reports you have created for these issues.

user28371 · 2022-12-20 21:00:12

arojas wrote:

> It is unsupported by the upstream, and even basic things, such as creating a dictionary from the standard library sometimes crash. External things almost always crash.
Please post links to the bug reports you have created for these issues.

I only have one recent one. I've never opened anything relating to crashes with external libraries as instructed by the wiki. (but try e.g. to use a solver such as SCS.jl. That ought to cause a crash in no time. At least it always used to)

Are you a maintainer of the package to the point that you fix bugs in the package caused by lib mismatch? :-O

Is there a github issue tracker for the repo where I could post the issue instead? Can I post it to you privately somehow?

Last edited by user28371 (2022-12-20 21:01:34)

arojas · 2022-12-20 21:07:31

Bug reports for issues which are not reproducible in the upstream binary package should be reported in the Arch bug tracker. Many of them have already been fixed.

user28371 · 2022-12-20 21:11:33

arojas wrote:

Bug reports for issues which are not reproducible in the upstream binary package should be reported in the Arch bug tracker. Many of them have already been fixed.

I never knew. Seems like a lot of work. I will post the issue there. Thanks!

Trilby · 2022-12-20 23:47:55

As noted above, bugs should be reported. But being "unsupported by upstream" is not surprising given that they distribute their own precompiled binary. They want to simplify their support by saying "use what we packaged, or we can't help you". That's not so different from the AUR package being unsupported by arch devs as they similarly say "use what we package, or we can't help you".

No one wants (or is well equipped) to support someone else's packaging. That's a universal. The only peculiarity here is that an exceedingly vast majority of open source software upstream developers play well with distro packagers[1]: upstream writes the code and works with distro maintainers allowing the latter to do the compiling packaging. From what you report, it sounds like Julia's upstream dev team does not want to interact with distro maintainers. That's kinda' a crappy approach [2].

[1] "playing well" being a relative term - here meaning they at least allow distro maintainers to build the packages for different distros. Distributing pre-compiled binaries is typical of closed-source software.

[2] Assuming what you say is true that upstream does not "support" distro packages. There is also a potential miscommunication: no upstream developer should be expected to support distro packaging bugs. But with good working relationship between distro developer / packager and upstream developer there is a devision of labor between packaging bugs and true software bugs and each flavor is "supported" (or fixed) by different people. So it's possible that Julia's upstream is just reporting that they can't support packaging bugs from distro packages - nor should they. But if they would not be responsive to actual software bug reports (at least if / when they come from distro maintainers) unless they came from their precompiled binaries that'd be the "crappy approach".

Last edited by Trilby (2022-12-20 23:57:24)

cfr · 2022-12-21 01:22:58

I had a poke around the bug tracker for julia and I cannot see anything to suggest they don't support distro packages - unless lack of support means they're not willing to figure out all the packaging issues or distribute such packages themselves.

The first 'non-support' comment I found is a 2019 response to a complaint that the project did not provide Ubuntu or Debian packages itself (whereas packages were available for some other distros, though I'm not clear who provided those). While things might have changed since then, the attitude at that time was certainly very far from 'crappy'.

To give a little bit more context; we used to provide an Ubuntu PPA and such, but it was a lot of extra work, and getting Julia included in the default distribution packaging was much more work than we wanted to do. Distributions have pressures on them to not duplicate libraries (such as LLVM, libuv, etc...) however since Julia requires heavily patched versions of these libraries there is an inherent tension between the JuliaLang project (which requires functionality from LLVM that precludes using a "system-wide" libLLVM) and distribution packagers (which reject packages that want to bundle 'duplicates' of libraries as they don't want to have 30 different versions of the same library on the system).
Both viewpoints are understandable, and while we have found some distributions that are willing to work with us, in the end, there are too many advantages to generic linux tarballs for us to spend much time and effort doing anything else. Tarballs can be unpacked and run without sudo privileges, they are self-contained and do not interfere with other system packages, and they are so simple as to work on all glibc Linux systems (modulo some kernel and glibc requirements).
For more complex arrangements (Such as deb packages for Debian and Ubuntu) we're happy to help someone else work with the distribution packagers to make it happen, but we don't have the bandwidth to devote time and energy to this.

Out of interest, have you ever reported a an issue with the kernel upstream? Because I suspect you may be horrified to discover the Linux kernel itself is equally 'unsupported' by its developers.

user28371 · 2022-12-21 02:21:19

cfr wrote:

I cannot see anything to suggest they don't support distro packages

I am probably not explaining myself well as I have never had to deal with the (presumably) horrors of packaging someone else's software.

To put it very concretely
1) Experimenting with Julia using the community repo, I ran into so many crashes and I've given up on using it. Julia's devs and the community simply ask you to use the official binary.
2) My bug report of a crash discovered during "basic usage" was met with "Closed. If you can replicate on our binary feel free to re-open." (I've only ever reported one, despite clear instructions not to open any)
3) I am astonished that anyone would be willing to go to so much effort to package software who's authors don't seem to care (fair enough. Maybe that is normal. I really would not know.) -- regardless of what their actual response towards actual package maintainers is.
5) The julia-bin package usually works.

Therefore my thinking is: maintaining the package ought to be difficult for Arch devs, julia devs seem pretty clear they are not interested, the resulting package is often unsatisfactory even for "basic use", there already exists the functional official binary.

It would be nice to have a proper julia package maintained cooperatively by both sides, but the one we have currently feels like some sort of sisyphean exercise to hurt noobs. Same story on other distros. I have had this confirmed several times when trying to convince people to try julia. Beginners are genuinely put off due to the number of bugs and crashes.

tl.dr. julia devs say "just use the official binary;" the community says "just use the official binary;" every julia user will come the same conclusion eventually. Then why have the arch package?

Last edited by user28371 (2022-12-21 02:31:53)

cfr · 2022-12-21 04:33:00

user28371 wrote:

tl.dr. julia devs say "just use the official binary;" the community says "just use the official binary;" every julia user will come the same conclusion eventually. Then why have the arch package?

Because binaries don't get packaged for the official repos. Generally, they're not found in AUR either, but there are quite a lot of exceptions.

It is not unreasonable for julia's developers to ignore bugs which cannot be reproduced with their binaries or with binaries built according to their instructions. Not all projects take this approach, but some do: if users of distro packages have problems, they expect them to be reported to their distros first. The package maintainers act as a filter, filtering out problems introduced at the distro level and passing the remainder upstream.

The most reasonable option, if things are as bad as you say, might be to remove julia from the official repos altogether. Regardless of policy, upstream's julia is not suitable for distro packaging. It comes with a bunch of customised versions of libraries and is designed to be run from a directory containing these libraries and other resources. That is not a job for a distro package manager. The libraries can't be installed system-wide because they will conflict with the standard versions of those libraries.

Upstream julia seems to be closer to a flatpak, appimage, snap or whatever than anything else. Although Arch includes those tools in its official repos, it doesn't package the software they are used to install.

Moreover, there is a security issue in packaging binaries from upstream which distros are likely to be extremely cautious about, to say the very least. The people who package for the official repos are responsible for the security of the software they provide. They cannot legitimately outsource that responsibility to julia's developers. I would certainly switch distors if they did and I do not say that lightly. I'm not really comfortable with tools like flatpak etc. in the official repos, but the packagers can still vouch for the integrity of those tools as functioning as intended.

I use pre-compiled binaries from AUR. Some are commercial (e.g. zoom, spideroak) and some are just convenience (e.g. pdftk-bin). That's my choice and I know the risks. Using AUR inevitably introduces greater risks. Using binaries from elsewhere arguably doesn't introduce more than using AUR in the first place. But having binaries in the official repos would alter the risk profile for everything I use on my machine, because I could not make the same kinds of assumptions about packages just in virtue of their being in the official repos. Those assumptions mean I do not scrutinise the PKGBUILDs (unless I want to know something they may tell me), the installation scripts, the file lists. I just assume if it is from the repos, it is safe (enough).

There's not a snowflake's chance in the bubbling cauldron's chance of julia's upstream binaries making it into the repos. I just wanted to explain why what may seem simple convenience to you is likely to seem beyond the pale to many others.

Report the bugs. If the package really can't be packaged satisfactorily, it probably should be dropped from the repos. Understand that I have no idea whether that's true. I'm just saying that if what you're saying is accurate and if it cannot be remedied in any reasonable way, the optimal solution may be not officially packaging it.

In general, projects which insist on relying on forked/custom-built/patched dependencies are a poor fit for distribution by Linux distros. Even distros which maintain multiple versions of libraries and utilities have a hard time with them. Arch typically tries to have only one of such things, when that is practical. Older versions of libraries, compilers etc. get dropped to AUR relatively quickly. I imagine it's julia's insistence on having a whole suite of its own libraries and dependencies which means it ends up not working well - because distro packagers will try to avoid those and end up relying on standard versions.

What this means is that, so long as julia relies on such things, its developers are probably right to recommend it be installed in a special directory with its own little fiefdom of software. Note that there is no guarantee that julia's developers will patch security vulnerabilities in that idiosyncratic fiefdom and it will likely be more difficult for them to do so, however well motivated, than distro maintainers. So however conscientious julia's developers are, using upstream's little cocoon of software is likely to mean you're running code with known vulnerabilities sooner or later.

Unfortunately, flatpak et al. are encouraging similar moves by other projects, which is not a positive trend. If you use flatpack, appimage, snap etc., you should already be aware of this kind of risk.

julia's developers are not being unreasonable: they clearly don't want to be packaged by distros. Arguably distros would be better not packaging them. But it would be unreasonable for them to request their software fiefdom be packaged by distros.

Edit: good grief. I honestly had no plans to write an essay.

Last edited by cfr (2022-12-21 04:35:39)

user28371 · 2022-12-21 04:51:38

That makes a lot of sense. Thank you for the explanation/ perspective.

Trilby · 2022-12-21 17:54:34

user28371 wrote:

... the (presumably) horrors of packaging someone else's software.

It's definitely work for the packagers, but generally not horrific. And it is a huge benefit for upstream developers - which is why most upstream devs are happy to work with distro packagers.

cfr wrote:

The package maintainers act as a filter

That's a key point. If you skip the line and go upstream with an issue that really should be handled at the distro level, upstream should just close it. In a perfect world, they might take the time to point you to were you should report it - but that's not really their job.

I am, in a very small and limited manner, the "upstream dev" of one or two projects. I share them on the AUR and "support" their use on arch linux. I was pleasantly surprised to here they are also used by some users of other distros - but I learned the hard way that I am completely ill-equipped to "support" those distros. Their libs work differently, their build system is different ... I provide a Makefile and a list of dependencies, but I don't know how to pick the right packages from their distro's repos to satisfy those dependencies. I also don't (didn't) know that their distro doesn't even have a compiler as part of base install ... so yeah, they'll need to install a c compiler of some sort - and other than suggesting 'gcc' I can't really help them set up their non-arch-linux system for compiling software. That's outside of the scope of supporting the software I wrote.

Me as an "upstream developer" trying to support users of my software in a distro I'm completely unfamiliar held all sorts of horrors for everyone involved. But then when my software got the interest of a distro-dev / packager from that distro, life got sooo much better. They knew the ins and outs of building software on their distro, and I knew my software. So between the two of us it was easy to communicate the needs and reach the end goal efficiently. And if there is any dependecy issue with the package, or library incompatibility, that's just not my job to address: that goes to the distro-devs/packagers. If there are changes I can make to ensure my software isn't to heavily reliant on bleeding edge libs so that it will work on stable release distros, that is my job; but that's a point that would likely be brought forward by a packager or distro-dev (for the present purposes I'm using these two terms as interchangeable).

It's not all sunshine and roses; there are hurdles. The most notable one deriving from my being spoiled with a rolling release paired with my software being very experimental and quickly developing with new commits fixing existing issues while the other distro is a point release distro that doesn't want to rebuild the package with every commit (which is one good reason for tagged releases at important checkpoints, but that's a different topic).

So any upstream developer with their head on strait should love distro packagers. Any end user with their head on strait should also love packagers. As for the distro devs / packagers, they may have occasional justification to loathe everyone else, but they rarely do; that highlights the burden they stoically take on so that the rest of us can benefit from their work. Joking aside, a sane upstream team should value packagers, and in turn work with them to help them rememdy any issues with running the software in their specific distro - because working with knowledgable packagers is easy, working with masses of end-users can be challenging.

Last edited by Trilby (2022-12-21 17:59:43)

Arch Linux

#1 2022-12-20 20:40:57

[Solved] Julia package

#2 2022-12-20 20:51:33

Re: [Solved] Julia package

#3 2022-12-20 21:00:12

Re: [Solved] Julia package

#4 2022-12-20 21:07:31

Re: [Solved] Julia package

#5 2022-12-20 21:11:33

Re: [Solved] Julia package

#6 2022-12-20 23:47:55

Re: [Solved] Julia package

#7 2022-12-21 01:22:58

Re: [Solved] Julia package

#8 2022-12-21 02:21:19

Re: [Solved] Julia package

#9 2022-12-21 04:33:00

Re: [Solved] Julia package

#10 2022-12-21 04:51:38

Re: [Solved] Julia package

#11 2022-12-21 17:54:34

Re: [Solved] Julia package

Board footer