You are not logged in.

#1 2021-09-14 20:05:39

teckk
Member
Registered: 2013-02-21
Posts: 518

Pacman redownloading cached packages

I can't find the answer. I don't see any post that covers this. I'm not using wget.

Something with pacman has changed. Pacman used to look for a package in /var/cache/pacman/pkg that matched the database first, then download if not there.

Example:

sudo pacman -S gcc
resolving dependencies...
looking for conflicting packages...

Packages (4) binutils-2.36.1-3  elfutils-0.185-1  libmpc-1.2.1-1  gcc-11.1.0-1

Total Installed Size:  199.45 MiB

:: Proceed with installation? [Y/n]

If all of those packages were there, in the cache, it should install them from the cache if they match the database.

This prevents me from

pacman -Syuw --cachedir

Then sharing that temp cache with all the machines on the network, and updating them with those packages.

I can copy all of those packaged into the /var/cache/pacman/pkg of a machine then run pacman -Syu, and pacman still tries to download everything.

What have I missed?

pacman.conf

[options]
# The following paths are commented out with their default values listed.
# If you wish to use different paths, uncomment and update the paths.
#RootDir     = /
#DBPath      = /var/lib/pacman/
#CacheDir    = /var/cache/pacman/pkg/
#LogFile     = /var/log/pacman.log
#GPGDir      = /etc/pacman.d/gnupg/
#HookDir     = /etc/pacman.d/hooks/
HoldPkg     = pacman glibc
#XferCommand = /usr/bin/curl -L -C - -f -o %o %u
#XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u
#CleanMethod = KeepInstalled
Architecture = auto

#IgnorePkg   =
#IgnoreGroup =

#NoUpgrade   =
#NoExtract   =

#UseSyslog
#Color
#NoProgressBar
CheckSpace
#VerbosePkgLists
#ParallelDownloads = 5

SigLevel    = Required DatabaseOptional
LocalFileSigLevel = Optional
#RemoteFileSigLevel = Required

[core]
Include = /etc/pacman.d/mirrorlist

[extra]
Include = /etc/pacman.d/mirrorlist

[community]
Include = /etc/pacman.d/mirrorlist

#[testing]
#Include = /etc/pacman.d/mirrorlist

#[community-testing]
#Include = /etc/pacman.d/mirrorlist

#[multilib-testing]
#Include = /etc/pacman.d/mirrorlist

#[multilib]
#Include = /etc/pacman.d/mirrorlist
pacman --version

 .--.                  Pacman v6.0.0 - libalpm v13.0.0
...

All the machines are the same, and act the same. And I know that they are from July.

Last edited by teckk (2021-09-14 20:34:07)

Offline

#2 2021-09-14 20:30:33

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,441
Website

Re: Pacman redownloading cached packages

I can't help with this issue - but I didn't know that until I read the thread.  You'll likely get better results if you put something with *any* meaningful content in the title.  You've posted in the pacman subforum ... so "pacman problem" is a given.  Perhaps "pacman redownloading cached packages" would be better.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#3 2021-09-14 21:47:26

teckk
Member
Registered: 2013-02-21
Posts: 518

Re: Pacman redownloading cached packages

If I add to the top of /etc/pacman.d/mirrorlist

Server = file://var/cache/pacman/pkg

Then in /etc/pacman.conf

SigLevel = Optional TrustAll

Then pacman will install the packages that are already in it's cache. I can do that and then remark it out until I find out why this is happening.

Offline

#4 2021-09-14 21:52:15

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Pacman redownloading cached packages

Works for me:

┌─[Shiv ~]
└─╼ pacman -Q zsh                                                                                                                                                           
zsh 5.8-1
┌─[Shiv ~]
└─╼ sudo pacman -S zsh
warning: zsh-5.8-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Package (1)  Old Version  New Version  Net Change

extra/zsh    5.8-1        5.8-1          0.00 MiB

Total Installed Size:  6.22 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] 
:: Retrieving packages...
 zsh-5.8-1-x86_64 is up to date
(1/1) checking keys in keyring                                                                          [--------------------------------------------------------------] 100%
(1/1) checking package integrity                                                                        [--------------------------------------------------------------] 100%
(1/1) loading package files                                                                             [--------------------------------------------------------------] 100%
(1/1) checking for file conflicts                                                                       [--------------------------------------------------------------] 100%
(1/1) checking available disk space                                                                     [--------------------------------------------------------------] 100%
:: Processing package changes...
(1/1) reinstalling zsh                                                                                  [--------------------------------------------------------------] 100%
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...

Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#5 2021-09-14 22:23:31

teckk
Member
Registered: 2013-02-21
Posts: 518

Re: Pacman redownloading cached packages

You already had that installed. My machines will do that also.

If I have gcc and friends installed on one machine, I copy /var/cache/pacman/pkg/ and /var/lib/pacman/sync/community.db /var/lib/pacman/sync/core.db /var/lib/pacman/sync/extra.db to a second machine, I should be able to pacman -S gcc on the second machine and have it install from cache, not having to download anything. Or anything else that is in the package cache, as long as it is the version that pacmans database wants.

Or at least that it how pacman has worked for a decade. That is what has changed here. I just noticed it a few days ago. And it affects downloading packages once, install on many machines.

Last edited by teckk (2021-09-14 22:24:29)

Offline

#6 2021-09-14 22:28:02

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Pacman redownloading cached packages

What's the output of pacman -Sp pacman? (or any other package that is already in your cache)

You can also compare the remote and local modification times to see if there's a problem with timestamps in your cache directory.

curl -s  --head "$(pacman -Sp pacman --cachedir /dev/null)" | grep last-modified
ls -lh /var/cache/pacman/pkg/pacman-[0-9]*

If your mirrorlist trick works then the file checksums should be correct but you can double-check them if need be with (which may point you to a misconfigured cache directory in pacman.conf, for example)

bsdtar -xOf /var/lib/pacman/sync/core.db 'pacman-[0-9]*/desc' | grep -A1 '%SHA256SUM%'
sha256sum /var/cache/pacman/pkg/pacman-[0-9]*

edit

teckk wrote:

You already had that installed. My machines will do that also.

If I have gcc and friends installed on one machine, I copy /var/cache/pacman/pkg/ and /var/lib/pacman/sync/community.db /var/lib/pacman/sync/core.db /var/lib/pacman/sync/extra.db to a second machine, I should be able to pacman -S gcc on the second machine and have it install from cache, not having to download anything. Or anything else that is in the package cache, as long as it is the version that pacmans database wants.

Or at least that it how pacman has worked for a decade. That is what has changed here. I just noticed it a few days ago. And it affects downloading packages once, install on many machines.

What seems to have changed is that pacman now does a remote check of all packages before installing them so I suspect that there's something wrong with the timestamps or checksums of the files that indicates to pacman that the file in the cache is old or corrupted. I don't know why the extra remote check was added.

edit 2
If it's relying on timestamps and not just signatures and checksums that it's worthy filing a bug report imo.

Last edited by Xyne (2021-09-14 22:31:40)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#7 2021-09-14 22:38:52

teckk
Member
Registered: 2013-02-21
Posts: 518

Re: Pacman redownloading cached packages

pacman -Sp pacman
file:///var/cache/pacman/pkg/pacman-6.0.0-5-x86_64.pkg.tar.zst

curl -s  --head "$(pacman -Sp pacman --cachedir /dev/null)" | grep last-modified
ls -lh /var/cache/pacman/pkg/pacman-[0-9]*
-rw-r--r-- 1 root root 839K Feb  8  2021 /var/cache/pacman/pkg/pacman-5.2.2-2-x86_64.pkg.tar.zst
-rw-r--r-- 1 root root 839K Mar 28 06:22 /var/cache/pacman/pkg/pacman-5.2.2-3-x86_64.pkg.tar.zst
-rw-r--r-- 1 root root 893K Jul 15 14:27 /var/cache/pacman/pkg/pacman-6.0.0-5-x86_64.pkg.tar.zst

bsdtar -xOf /var/lib/pacman/sync/core.db 'pacman-[0-9]*/desc' | grep -A1 '%SHA256SUM%'
%SHA256SUM%
0af273a1719f8e40dfa0a9e674419b375145aa31568bc21861e4bbad35b3d396

sha256sum /var/cache/pacman/pkg/pacman-[0-9]*
56fc524a8685c5faa25bd0085965ae46e8ec301c91881562f8e8fd1726ee5714  /var/cache/pacman/pkg/pacman-5.2.2-2-x86_64.pkg.tar.zst
a8fe90908efa314f59ffd0af3997c1caba994e0922085847d8f615db408f8ba4  /var/cache/pacman/pkg/pacman-5.2.2-3-x86_64.pkg.tar.zst
0af273a1719f8e40dfa0a9e674419b375145aa31568bc21861e4bbad35b3d396  /var/cache/pacman/pkg/pacman-6.0.0-5-x86_64.pkg.tar.zs

It's very possible that the next update will solve this. I can share packages in the meantime.

I thought that it would be a change to pacman. And i don't update as often as others.

Last edited by teckk (2021-09-14 22:40:09)

Offline

#8 2021-09-14 23:03:34

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: Pacman redownloading cached packages

The output of "curl -s  --head "$(pacman -Sp pacman --cachedir /dev/null)" | grep last-modified" was lost in your reply. That will give you the remote file modification time on the mirror. If it's newer than the one in the shared cache then you could try updating the modification time with "touch".

However, your comment about not updating that often may indicate the error. You're copying the databases but running "pacman -Syu" so pacman will download newer sync databases and newer packages from the mirrors if they have them. Check pacman's output and the contents of /var/lib/pacman/sync to see if pacman is in fact updating the databases, and then check if the downloaded packages actually match the version that you copied from the other system.

Post the full output of the pacman command that shows pacman downloading the packages from the server.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#9 2021-09-14 23:54:55

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: Pacman redownloading cached packages

You have packages but no signature files.  Pacman-6.0 downloads the signature files.  This has the side effect of restarting the download if a cached package does not have a signature file AND is not in your primary cache directory.   If the package is in your primary cache directory, it looks like it is redownloading, but just gets the signature.

It is a bug that will go away if you clean the cache completely or after the first redownload, so all package download also have signatures.  I put it on the zero priority to fix in pacman, as it will fix itself...

Offline

#10 2021-09-15 03:46:28

teckk
Member
Registered: 2013-02-21
Posts: 518

Re: Pacman redownloading cached packages

Thanks for the explanation.

Offline

Board footer

Powered by FluxBB