Competing philosophies for Anaconda install, which to follow?

charper_flow · 2022-07-19 16:06:39

I use a product called MFiX for research which must be installed through conda. I know that package managers can cause a huge mess in the organization of your system, so I was doing some research on these forums to discern a "best practice" for installing conda. There seems to be two competing approaches to handling conda:

Approach 1 Don't do a system-wide install of conda, it defeats the purpose and that is exactly how things get messy! To follow this approach I would just install anaconda following the directions from their website. This installs anaconda in my /home/charper/anaconda directory, so this is NOT system wide. Furthermore there is a step where the ask "Do you wish the installer to initialize Anaconda3 by running conda init?” As long as I say no here, anaconda will not touch my dot files and I can just activate anaconda whenever I need by running source <path to conda>/bin/activate. This seems like a successful "isolated" install and will keep conda nicely quarantined from wreaking havoc on my system.

Approach 2 You should always build from AUR. If you consistently build everything from AUR then pacman will keep track of everything for you. If you end up creating a mess and creating conflicts you can just uninstall anaconda and rebuild it from source whenever you need to run it. Building from AUR helps you stay consistent and keep everything in the same place. HOWEVER, installing through AUR by definition is a sytemwide install. This would put conda in my /usr/opt/bin and /usr/opt/lib...but one of the main pieces of advice I read when reading about installing anaconda is to NEVER do a system wide install.

Is that not as much of a threat with arch linux because of how easy it is to just uninstall and rebuild different packages as you need them? Thus we are just easily creating whatever environment we need as we need it? Im a bit confused. This is my first arch install and I have already learned so much about how everything interacts and works, but it is a lot to take in. If I mess anything up , I feel confident I could do a clean reinstall and get back to here pretty quickly, however, I have sunk a lot of time into setting up this current build and would like to keep it clean and properly built. I appreciate any insights

dogknowsnx · 2022-07-25 09:29:31

https://aur.archlinux.org/packages/miniconda3
https://aur.archlinux.org/cgit/aur.git/ … miniconda3
https://docs.conda.io/en/latest/miniconda.html

Last edited by dogknowsnx (2022-07-25 09:34:04)

Lone_Wolf · 2022-07-25 09:53:58

Anaconda is not the same as conda.

By installing miniconda3 as advised by dogknowsnx you will avoid the issues with anaconda and still be able to use conda.

ngoonee · 2022-07-25 23:26:26

You seem to misunderstand the specific issue system-wide vs user-only has. conda is a package manager as well, just like pacman! Which means as long as you use conda, you are duplicating/conflicting with pacman.

For Approach 2, pacman will ONLY keep track of the initial install of anaconda/miniconda, which means it keeps track of a list of files. When you run conda and it installs something else, there's a whole bunch of other files created which pacman knows nothing about. All this is expected behaviour, because that's just how conda works. Unsure if you can use the system conda run as your user to only install things in your home folder, as I've never tried that.

So if you HAVE to use conda, then Approach 1 makes more sense, simply because conda only runs as your user, hence it can only install things in directories your user has permissions for. In this case, your /home/charper/anaconda.

Practically.... conda isn't too bad even in Approach 2. If you init it properly it will install its packages in predictable locations etc (always use separate environments of course). However long term use of conda can eventually result in use of pip for packages you can't get in conda, this is where things get problematic in Approach 2 (especially if you're not using the environments own pip, but your system one).

The 'Arch' approach is not to use conda at all, but to install all the packages (including whatever MFiX is) from the repos, from AUR, and perhaps from source with your own PKGBUILD. Meaning no usage of conda or pip. For machine learning research at least, this is not often very feasible, but if you can manage it then that's the only approach where pacman knows about everything. Your Approach 2, pacman doesn't.

FWIW I've used conda for years and Approach 1 is the way to go. Don't understand why Approach 2 would ever be considered except on a shared research machine with multiple users using it and being careful to set up their own user-specific environments using the system anaconda (even so I would just have each user install their own disparate copies).

Last edited by ngoonee (2022-07-25 23:40:21)

Trilby · 2022-07-25 23:42:09

ngoonee wrote:

For machine learning research at least, this is not often very feasible...

Could you elaborate on this? I'm skeptical.

I've worked in machine learning (via bioinformatics). This work depends on finely crafted software tools written by remarkably brilliant people. But the same ecosystem is also flooded with other software tools written by people who think they are remarkably brilliant ... but they make software that is incredibly fragile requiring it's own specific set of incantations for it to be "installed" and cannot be reasonably managed by any sane package manager. That's not the sort of software I'd trust with my data. If the author can't make a reasonably compliant makefile (or cmake list file,or setup.py, or...) then I wouldn't trust the results of the tool any further than I could throw it.

Last edited by Trilby (2022-07-25 23:42:47)

ngoonee · 2022-07-26 00:08:09

Trilby wrote:

ngoonee wrote:
For machine learning research at least, this is not often very feasible...
Could you elaborate on this? I'm skeptical.
I've worked in machine learning (via bioinformatics). This work depends on finely crafted software tools written by remarkably brilliant people. But the same ecosystem is also flooded with other software tools written by people who think they are remarkably brilliant ... but they make software that is incredibly fragile requiring it's own specific set of incantations for it to be "installed" and cannot be reasonably managed by any sane package manager. That's not the sort of software I'd trust with my data. If the author can't make a reasonably compliant makefile (or cmake list file,or setup.py, or...) then I wouldn't trust the results of the tool any further than I could throw it.

I mean, that's a fair point. But from an education/academic/research perspective, quality of code is subservient. Any of my undergrad or postgrad students who came to me and said "I don't want to run a comparison against this tool because it doesn't have a good makefile" would get chewed out because this isn't primarily about the engineering, its about the algorithms and results. Anaconda's team is specifically targeted at a narrow range of software in a way that no distro's packaging team is, and will be much more likely to have the needed packages than our repos+AUR will (also, unmaintained AUR packages are in my experience just the worst to deal with).

Side benefit is that when using conda-based installs, its easier for me to guide Windows-based students.

Arch Linux

#1 2022-07-19 16:06:39

Competing philosophies for Anaconda install, which to follow?

#2 2022-07-25 09:29:31

Re: Competing philosophies for Anaconda install, which to follow?

#3 2022-07-25 09:53:58

Re: Competing philosophies for Anaconda install, which to follow?

#4 2022-07-25 23:26:26

Re: Competing philosophies for Anaconda install, which to follow?

#5 2022-07-25 23:42:09

Re: Competing philosophies for Anaconda install, which to follow?

#6 2022-07-26 00:08:09

Re: Competing philosophies for Anaconda install, which to follow?

Board footer