paccheck - pacman package authenticity check

milomouse · 2011-02-25 07:06:35

Me again. Thanks for the update, grep|sed works just fine.

1) One thing I noticed is that a repo that's specified in pacman.conf with the [repo-name] Server=. (not Include=.) is still attempted on fetches from the server mirrors and therefore always reports a download failure. Not a big deal since the user should know that's a specific server they added, (like [xyne-any] in my config, also causing error reports since xyne-any's self maintained are usually newer than community). Not sure if this has any bearing on anything, just thought I'd bring it up although you've probably already seen this (as in previous output posts). :}

2) Also, not sure if I'm misreading things but it didn't seem to scan packages for my first mirror. I'll post the output of paccheck and here my first mirror is hpc.arc.georgetown.edu and it does the analyzing but after checking databases for that first mirror it jumps to the other mirrors but in those mirrors it does "Scanning packages..." whereas it didn't for hpc.arc.georgetown.edu. Does that mean it didn't find any mispackages and everything is okay or did something go wrong? I just thought it should show "Scanning packages..." on that mirror too.

3) As a side query, is it normal for a mirror to not show "Upstream: $URL" when analyzing or does that just mean it couldn't detect it?

Reading mirrors from /etc/paccheck/mirrorlist...

NOTE: pacman must be synced and needed packages downloaded
      for paccheck to work properly.
pacman sync...
>>> sudo pacman -Sy
:: Synchronizing package databases...
 testing                                                             18.6K   78.8K/s 00:00:00 [######################################################] 100%
 staging is up to date
 community-testing is up to date
 community-staging is up to date
 xyne-any is up to date
 core is up to date
 community                                                          425.1K   89.0K/s 00:00:05 [######################################################] 100%
 extra                                                              464.2K   89.1K/s 00:00:05 [######################################################] 100%

Download needed packages without installing...
>>> sudo pacman -w --noconfirm -Su
:: Starting full system upgrade...
warning: arch-wiki-docs: local (20110219-1) is newer than community (20110207-1)
warning: curl: local (7.21.4-3) is newer than extra (7.21.4-2)
warning: glib2: local (2.27.93-1) is newer than core (2.26.1-1)
warning: libcups: local (1.4.6-2) is newer than extra (1.4.6-1)
warning: sbcl: ignoring package upgrade (1.0.44-1 => 1.0.46-1)
 there is nothing to do

Timestamps:
/var/cache/packages/database/testing.db.tar.gz: 2011-02-24 16:12:17 -0500
/var/cache/packages/database/staging.db.tar.gz: 2011-02-23 20:50:35 -0500
/var/cache/packages/database/xyne-any.db.tar.gz: 2011-02-23 03:07:57 -0500
/var/cache/packages/database/core.db.tar.gz: 2011-02-22 07:39:59 -0500
/var/cache/packages/database/community-testing.db.tar.gz: 2011-02-23 20:43:20 -0500
/var/cache/packages/database/community.db.tar.gz: 2011-02-24 14:12:56 -0500
/var/cache/packages/database/extra.db.tar.gz: 2011-02-24 14:56:15 -0500
/var/cache/packages/database/community-staging.db.tar.gz: 2011-02-23 20:43:21 -0500

========== DOWNLOADING ============

...
-- snip -- snip -- snip -- snip -- (everything fine here, just downloading databases)
...

=========== ANALYZING =============

hpc.arc.georgetown.edu:  Tier 2 (United States) Upstream: gtlib.gatech.edu
    testing.db: OK
    staging.db: OK
    xyne-any.db: DOWNLOAD FAILED
    core.db: OK
    community-testing.db: OK
    community.db: OK
    extra.db: OK
    community-staging.db: OK

distro.ibiblio.org:  Tier 2 (United States) Upstream: gtlib.gatech.edu
    testing.db: timestamp mismatch   CONTENT MISMATCH
    staging.db: OK
    xyne-any.db: DOWNLOAD FAILED
    core.db: OK
    community-testing.db: OK
    community.db: timestamp mismatch   CONTENT MISMATCH
    extra.db: timestamp mismatch   CONTENT MISMATCH
    community-staging.db: OK
    Scanning packages...
        bauerbill-2011.01.28.1-1: MISSING
        file-5.05-2: MISSING
        libmpc-0.9-1: MISSING
        librsvg-2.32.1-2: MISSING
        reflector-2011.01.24.2-1: MISSING
        tsocks-1.8beta5-4: MISSING

mirror.yellowfiber.net:  Tier 2 (United States)
    testing.db: OK
    staging.db: OK
    xyne-any.db: DOWNLOAD FAILED
    core.db: OK
    community-testing.db: timestamp mismatch   CONTENT MISMATCH
    community.db: timestamp mismatch   CONTENT MISMATCH
    extra.db: timestamp mismatch   CONTENT MISMATCH
    community-staging.db: OK
    Scanning packages...
        bauerbill-2011.01.28.1-1: MISSING
        reflector-2011.01.24.2-1: MISSING

lug.mtu.edu:  Tier 2 (United States) Upstream: gtlib.gatech.edu
    testing.db: OK
    staging.db: OK
    xyne-any.db: DOWNLOAD FAILED
    core.db: OK
    community-testing.db: OK
    community.db: timestamp mismatch   CONTENT MISMATCH
    extra.db: timestamp mismatch   CONTENT MISMATCH
    community-staging.db: OK
    Scanning packages...
        bauerbill-2011.01.28.1-1: MISSING
        reflector-2011.01.24.2-1: MISSING

mirror.ece.vt.edu:  Tier 2 (United States) Upstream: gtlib.gatech.edu
    testing.db: OK
    staging.db: OK
    xyne-any.db: DOWNLOAD FAILED
    core.db: OK
    community-testing.db: timestamp mismatch   CONTENT MISMATCH
    community.db: timestamp mismatch   CONTENT MISMATCH
    extra.db: timestamp mismatch   CONTENT MISMATCH
    community-staging.db: OK
    Scanning packages...
        bauerbill-2011.01.28.1-1: MISSING
        reflector-2011.01.24.2-1: MISSING

mirror.rit.edu:  Tier 1 (United States)
    testing.db: OK
    staging.db: OK
    xyne-any.db: DOWNLOAD FAILED
    core.db: OK
    community-testing.db: timestamp mismatch   CONTENT MISMATCH
    community.db: timestamp mismatch   CONTENT MISMATCH
    extra.db: timestamp mismatch   CONTENT MISMATCH
    community-staging.db: OK
    Scanning packages...
        bauerbill-2011.01.28.1-1: MISSING
        reflector-2011.01.24.2-1: MISSING

mirrors.xmission.com:
    testing.db: timestamp mismatch   CONTENT MISMATCH
    staging.db: OK
    xyne-any.db: DOWNLOAD FAILED
    core.db: OK
    community-testing.db: OK
    community.db: OK
    extra.db: OK
    community-staging.db: OK
    Scanning packages...
        bauerbill-2011.01.28.1-1: MISSING
        reflector-2011.01.24.2-1: MISSING

Checking package sizes...
    abs-2.4.2-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    attr-2.4.44-2: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    cdrkit-1.1.11-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    cvs-1.11.23-6: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    enca-1.13-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    enchant-1.6.0-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    gc-7.1-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    gnutls-2.10.4-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    gtk2-2.22.1-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    iptables-1.4.10-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    libnotify-0.5.2-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    libsndfile-1.0.23-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    neon-0.29.3-2: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    netcfg-2.5.4-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    net-tools-1.60-14: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    reflector-2011.01.24.2-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    wireless-regdb-2010.11.24-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    wpa_actiond-1.1-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@
    wpa_supplicant-0.7.3-1: @@@@@@@@ SIZE MISMATCH @@@@@@@@@

============ SUMMARY ==============

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
THE FOLLOWING PACKAGES IN PACMAN'S PKG CACHE ARE THE WRONG SIZE - this
indicates they are corrupt, or they or the database has been tampered
with:

    abs-2.4.2-1  (7968 != 7960)
    attr-2.4.44-2  (56948 != 65124)
    cdrkit-1.1.11-1  (610172 != 608804)
    cvs-1.11.23-6  (534728 != 536504)
    enca-1.13-1  (120304 != 203786)
    enchant-1.6.0-1  (47468 != 52400)
    gc-7.1-1  (272044 != 381879)
    gnutls-2.10.4-1  (1623092 != 1625292)
    gtk2-2.22.1-1  (5160176 != 5109444)
    iptables-1.4.10-1  (235460 != 259988)
    libnotify-0.5.2-1  (27640 != 27552)
    libsndfile-1.0.23-1  (283832 != 285452)
    neon-0.29.3-2  (176372 != 180316)
    netcfg-2.5.4-1  (16920 != 19366)
    net-tools-1.60-14  (111068 != 176961)
    reflector-2011.01.24.2-1  (1992 != 1988)
    wireless-regdb-2010.11.24-1  (3984 != 4000)
    wpa_actiond-1.1-1  (6324 != 7194)
    wpa_supplicant-0.7.3-1  (252528 != 254024)
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

THERE WERE DOWNLOAD FAILURES - This indicates unresponsive mirror(s).

THE FOLLOWING PACKAGES ARE MISSING FROM THE INDICATED MIRROR - if they
are listed as missing from all mirrors above, this indicates they could
not be tested for authenticity:

    bauerbill-2011.01.28.1-1  (distro.ibiblio.org)
    file-5.05-2  (distro.ibiblio.org)
    libmpc-0.9-1  (distro.ibiblio.org)
    librsvg-2.32.1-2  (distro.ibiblio.org)
    reflector-2011.01.24.2-1  (distro.ibiblio.org)
    tsocks-1.8beta5-4  (distro.ibiblio.org)
    bauerbill-2011.01.28.1-1  (mirror.yellowfiber.net)
    reflector-2011.01.24.2-1  (mirror.yellowfiber.net)
    bauerbill-2011.01.28.1-1  (lug.mtu.edu)
    reflector-2011.01.24.2-1  (lug.mtu.edu)
    bauerbill-2011.01.28.1-1  (mirror.ece.vt.edu)
    reflector-2011.01.24.2-1  (mirror.ece.vt.edu)
    bauerbill-2011.01.28.1-1  (mirror.rit.edu)
    reflector-2011.01.24.2-1  (mirror.rit.edu)
    bauerbill-2011.01.28.1-1  (mirrors.xmission.com)
    reflector-2011.01.24.2-1  (mirrors.xmission.com)

THERE WERE DATABASE CONTENT MISMATCHES - This usually indicates some
mirrors were out of date with your pacman sync, but alone does not
indicate compromised mirrors.

System update is NOT recommended until the above problems are fixed.

IgnorantGuru · 2011-02-25 07:38:41

milomouse wrote:

1) One thing I noticed is that a repo that's specified in pacman.conf with the [repo-name] Server=. (not Include=.) is still attempted on fetches from the server mirrors and therefore always reports a download failure. Not a big deal since the user should know that's a specific server they added, (like [xyne-any] in my config, also causing error reports since xyne-any's self maintained are usually newer than community). Not sure if this has any bearing on anything, just thought I'd bring it up although you've probably already seen this (as in previous output posts). :}

I could have it ignore those - I'll take a look.

2) Also, not sure if I'm misreading things but it didn't seem to scan packages for my first mirror. I'll post the output of paccheck and here my first mirror is hpc.arc.georgetown.edu and it does the analyzing but after checking databases for that first mirror it jumps to the other mirrors but in those mirrors it does "Scanning packages..." whereas it didn't for hpc.arc.georgetown.edu. Does that mean it didn't find any mispackages and everything is okay or did something go wrong? I just thought it should show "Scanning packages..." on that mirror too.

If there isn't a CONTENT MISMATCH on the database file, it doesn't need to extract it and scan individual package entries, since the whole database is identical. So that is normal.

3) As a side query, is it normal for a mirror to not show "Upstream: $URL" when analyzing or does that just mean it couldn't detect it?

It just gets that info from http://www.archlinux.org/mirrors/ and in that case the mirror is listed as "Untiered", which threw it. I'll accomodate that, but in general that info is just a convenience - not authoritative or required.

The real question is why you have all those size mismatches - that should not be. It means the size of the packages in your local cache don't match the listed size in the local sync database (/var/lib/pacman/sync/REPO/PACKAGE/desc %CSIZE% value). I can't explain that.

Allan · 2011-02-25 07:55:41

IgnorantGuru wrote:

The real question is why you have all those size mismatches - that should not be. It means the size of the packages in your local cache don't match the listed size in the local sync database (/var/lib/pacman/sync/REPO/PACKAGE/desc %CSIZE% value). I can't explain that.

File system type, block size, filesystem compression... the list goes on...

IgnorantGuru · 2011-02-25 08:02:08

Allan wrote:

IgnorantGuru wrote:
The real question is why you have all those size mismatches - that should not be. It means the size of the packages in your local cache don't match the listed size in the local sync database (/var/lib/pacman/sync/REPO/PACKAGE/desc %CSIZE% value). I can't explain that.
File system type, block size, filesystem compression... the list goes on...

I thought CSIZE represented the size of the package archive in bytes, which it seems to on my system. Apparently not? If not, then I'll have to remove the package size check, which is too bad because it makes MD5 collisions a bit more difficult.

milomouse · 2011-02-25 08:22:42

IgnorantGuru wrote:

I could have it ignore those - I'll take a look.

Not a big deal about the Server=. but I thought it might save on "polling" or something if I didn't check for those.

IgnorantGuru wrote:

If there isn't a CONTENT MISMATCH on the database file, it doesn't need to extract it and scan individual package entries, since the whole database is identical. So that is normal.

Allright, that's good to know. I thought it might be the case but wasn't sure.

IgnorantGuru wrote:

The real question is why you have all those size mismatches - that should not be. It means the size of the packages in your local cache don't match the listed size in the local sync database (/var/lib/pacman/sync/REPO/PACKAGE/desc %CSIZE% value). I can't explain that.

Ah, yeah. I have a solid state drive with btrfs using the 'compress' mount option (file size in btrfs is strange anyway) so maybe that's why there's size differences, although I would think it would show a size difference for almost all my packages in that case, not just a handful. I don't think btrfs does "smart" compression like reiser4 and probably compresses already compressed data too. Makes me want to read up on it some more as it could make this sort of integrity checking difficult for me (or I might have to mount my CacheDir independently without compress option). Hmm!

Last edited by milomouse (2011-02-25 08:28:33)

IgnorantGuru · 2011-02-25 08:33:53

milomouse wrote:

Ah, yeah. I have a solid state drive with btrfs using the 'compress' mount option (file size in btrfs is strange anyway) so maybe that's why there's size differences, although I would think it would show a size difference for almost all my packages in that case, not just a handful. I don't think btrfs does "smart" compression like reiser4 and probably compresses already compressed data too. Makes me want to read up on it some more as it could make this sort of integrity checking difficult for me (or I might have to mount my CacheDir independently without compress option). Hmm!

File size should not vary with disk compression or filesystem, even if 'size on disk' or space consumed does. But I can't find any spec for Arch's desc file, so I don't know what it's supposed to represent or why.

The problem could also be the stat command with btrfs - maybe it isn't returning the file size correctly.

Last edited by IgnorantGuru (2011-02-25 08:41:15)

Allan · 2011-02-25 08:52:57

You can look at the repo-add script to see how it is generated.

From memory, the tools for getting sizes just do not work particularly well on btrfs with compression...

milomouse · 2011-02-25 09:14:28

Well, I get the same mismatch sizes with or without compression so I guess it's just btrfs because I'm rather positive everything in my CacheDir should be correct as those are my untampered pacman downloads (not the majority of my packages which are made with makepkg and sent to a different directory, and I've made sure I don't have both at the same time. e.g. same package in both CacheDir and my makepkg PKGDEST). Without going too far off topic some reading for thought: https://btrfs.wiki.kernel.org/index.php … e_space.3F I guess I'll just ignore the size results for now as I get the same thing even after force re-downloading.

Last edited by milomouse (2011-02-25 09:21:03)

IgnorantGuru · 2011-02-25 14:46:45

milomouse wrote:

Well, I get the same mismatch sizes with or without compression so I guess it's just btrfs because I'm rather positive everything in my CacheDir should be correct as those are my untampered pacman downloads (not the majority of my packages which are made with makepkg and sent to a different directory, and I've made sure I don't have both at the same time. e.g. same package in both CacheDir and my makepkg PKGDEST). Without going too far off topic some reading for thought: https://btrfs.wiki.kernel.org/index.php … e_space.3F I guess I'll just ignore the size results for now as I get the same thing even after force re-downloading.

It is appalling that stat isn't returning the actual file size with btrfs. stat is considered the most reliable way of determining file size. You can try these methods to see what gives you an accurate answer ("testsize" script):

#!/bin/bash

FILENAME="$1"

stat -c %s "$FILENAME"
/bin/ls -l "$FILENAME"  | awk -F " " '{ print $5 }'
du -b "$FILENAME" | sed 's/\([0-9]*\)\(.*\)/\1/'
cat "$FILENAME" | wc -c

They should all be the same, but ls is known to be inaccurate, and cat requires reading the entire file. If you find what works I may be able to change paccheck's method or give you an option. For now it appears to be a bug in stat.

$ testsize /var/cache/pacman/pkg/cvs-1.11.23-6-x86_64.pkg.tar.xz
536504
536504
536504
536504

$sed -n '1h;1!H;${;g;s/%CSIZE%\x0A\([0-9]\)/pkgsize=\1/;p;}' \
    "/var/lib/pacman/sync/extra/cvs-1.11.23-6/desc" | grep -m 1 "^pkgsize=" \
    | sed 's/^pkgsize=\([0-9]\)/\1/'
536504

repo-add uses:

csize=$(stat -L -c %s "$deltafile")

which is the same as paccheck, and which should represent the archive file size. From paccheck:

pkgsize=`sed -n '1h;1!H;${;g;s/%CSIZE%\x0A\([0-9]\)/pkgsize=\1/;p;}' \
        "$dsync/desc" | grep -m 1 "^pkgsize=" \
        | sed 's/^pkgsize=\([0-9]\)/\1/'`
if [ "$pkgsize" != "" ]; then
    if [ "$(stat -c %s "$f")" != "$pkgsize" ]; then
        echo "    $pkg: @@@@@@@@ SIZE MISMATCH @@@@@@@@@"

Also, you can try copying the file to a non-btrfs partition and see if the file size report changes.

Last edited by IgnorantGuru (2011-02-25 14:47:35)

Allan · 2011-02-25 15:01:40

du is even worse on compressed filesystems...

berbae · 2011-02-25 16:09:28

IgnorantGuru wrote:

Personally I think giving it a few tier 1 mirrors is sufficient for reasonable security - certainly an improvement over no polling at all. But of course you must make your own security assessments for your purposes.

Ok I decided to change my habit and to use your script instead of just the traditional 'pacman -Syu'.
I chose to use several different mirrors and to avoid the --compare option presently.
Certainly it is better to do something now to reduce the risks when a vulnerability is known and potentially dangerous.

I hope that the package signing will be implemented soon, because it seems to me more appropriate to this problem and largely used by other distros.

Thank you for your work to elaborate this useful script waiting for an official solution.

Last edited by berbae (2011-02-25 16:19:15)

Inxsible · 2011-02-25 16:18:02

As a friendly reminder, lets not make this thread as another package signing holy war. We all agree to what needs to be done, and there are developers working towards it. Keep the posts related to paccheck and its features.

milomouse · 2011-02-25 17:14:13

@IgnorantGuru: I wouldn't go as far as to removing the size function completely as it seems to be just me(?). If others with btrfs (and perhaps nilfs2 or zfs) don't have the problem then it could very well be an unseen mistake on my part though I'm not sure as to what it would be. I do a lot of work with packages and tend to do things unconventionally so human error is likely. Regardless, the packages remain the same size regardless of what filesystem type I move them to (I tried your tests and some of my own but results are consistent) which the exception of one that did change size after I flushed, synced, and committed the filesystem at different intervals of redownloading, stating, etc. So I'm getting there, but until someone else has an issue I wouldn't worry too much over this. Perhaps it will work itself out over time. I had no intention of cluttering your thread as much as I have but certainly appreicate your efforts, not everyone would do so.

Last edited by milomouse (2011-02-25 17:15:01)

IgnorantGuru · 2011-02-25 18:20:48

paccheck 0.8.7 is available and includes the following changes and additions:

* repo selection is limited to the repos on official mirrors to prevent download errors
* added --alt-size and --skip-size (see below)
* automatic --alt-size with btrfs
* added --no-sync for scripts (see below)
* mirrorlist accepts Server= format & old style entries that hard code $arch
* info for untiered mirrors
* AUR PKGBUILD creates /etc/paccheck/mirrorlist

paccheck version 0.8.7
Compares Arch Linux pacman sync and package cache to multiple mirrors to help
detect compromised mirrors

Usage: paccheck [OPTIONS]
OPTIONS:
--keep              Don't remove temporary files in /tmp/paccheck.tmp
--compare 'MIRROR'  Fully download and compare all non-expired packages in
                    pacman's pkg cache to MIRROR.  Can alternatively be
                    listed in /etc/paccheck/mirrorlist as "compare=MIRROR".
                    For best speed, remove unneeded (already installed)
                    packages from pacman's cache before using this option.
--verbose           show debugging output
--alt-size          use alternate slower test of package sizes (useful due
                    to stat bug with btrfs which gives inaccurate results)
--skip-size         skip test of package sizes
--no-sync           no pacman update (mainly for use in scripts - paccheck
                    requires an updated pacman sync and packages)

pacman Update Procedure:
   1) Run paccheck and examine report
   2) If no package MISMATCH, then run sudo pacman -Su to update your system

Desired mirrors may be configured inside this script or 
saved to /etc/paccheck/mirrorlist

Exit Status:
    3  Package MISMATCH, download failures, or other errors
    2  Packages missing from some mirrors
    1  Out of sync mirrors (DATABASE CONTENT MISMATCH) or other warnings
    0  All OK

If paccheck sees the cache on a btrfs filesystem, it will enable the --alt-size option automatically, which uses "cat | wc -c" to avoid stat's inaccuracy. This may be a little slower but doesn't seem to make a big difference. Or you can specify --skip-size to skip the package size check entirely.

--no-sync is provided for scripts. Just be sure to update pacman and download needed packages before running paccheck with this option, or you won't check what's needed. Prior to running paccheck --nosync you should run:

sudo pacman -Sy
sudo pacman -w --noconfirm -Su # download, no install

milomouse wrote:

I had no intention of cluttering your thread as much as I have but certainly appreicate your efforts, not everyone would do so.

No point in having false positives to ignore, because that can get confusing. Please see how the --alt-size option works (should enable itself automatically on your system). Or you can use --skip-size. The package size check is on top of pacman's routine md5 check, so it's only there to help detect MD5 collisions, which are unlikely.

Last edited by IgnorantGuru (2011-02-25 18:25:15)

milomouse · 2011-02-25 19:04:16

IgnorantGuru wrote:

milomouse wrote:
I had no intention of cluttering your thread as much as I have but certainly appreicate your efforts, not everyone would do so.
No point in having false positives to ignore, because that can get confusing. Please see how the --alt-size option works (should enable itself automatically on your system). Or you can use --skip-size. The package size check is on top of pacman's routine md5 check, so it's only there to help detect MD5 collisions, which are unlikely.

After doing some integrity checks to test compression corruption and the like, and installing new version of paccheck it seems all is good now. I saw in the code it should stat for %T so I let it do it automatically, and it did, and there were no package size mismatches. I did of course get the usual summary about bauerbill and reflector (and kernel26-headers-2.6.37-2-1, but this is just because some haven't synced yet). So I'm good to go. Also nice to see xyne-any was skipped and I can just symlink my /etc/pacman.d/mirrorlist to /etc/paccheck/mirrorlist without removing the Server stuff. Thanks!

ANOKNUSA · 2011-02-25 19:09:58

@ IgnorantGuru: It's nice to see some community effort in this direction; I've built paccheck and will set it to run as part of my update process, though at the moment I don't have the time to play with it too much. After your posts on the previous package signing thread and the article on your blog, I thought it'd be a good idea to leave an unsavory flame in the comments section; I was in a bad mood at the time, but that's really no excuse for unnecessary personal attacks. Allow me to take this opportunity to apologize and choke on my words. Way to move things forward, if only this little bit.

@milomouse: Space management is odd in btrfs; you've already found the wiki entry on that, I see. I still haven't had much time to play around with my btrfs drive, but the unique logic it uses to handle subvolumes and such makes the issue confusing, since graphical tools and old-school built-ins like du don't read it properly. It's my understanding that this, the fscheck (due out any time now) and the COW defrag bug are the only things holding back the final release. Back on topic, what process does the --alt-size flag use for its comparison?

milomouse · 2011-02-25 19:21:16

ANOKNUSA wrote:

@milomouse: Space management is odd in btrfs; you've already found the wiki entry on that, I see. I still haven't had much time to play around with my btrfs drive, but the unique logic it uses to handle subvolumes and such makes the issue confusing, since graphical tools and old-school built-ins like du don't read it properly. It's my understanding that this, the fscheck (due out any time now) and the COW defrag bug are the only things holding back the final release. Back on topic, what process does the --alt-size flag use for its comparison?

I'm pretty sure it uses `cat "FILE" | wc -c'. Though I'm not sure if this is what made my results work or if it was all the manipulation and compression corruption tests I was doing (etc), or a combination effort. Regardless, it works now and I'm not about to digress.

kgas · 2011-02-26 08:27:01

* AUR PKGBUILD creates /etc/paccheck/mirrorlist

thanks. If it is there the installation/upgrade should not fail with the following error
error: failed to commit transaction (conflicting files)
paccheck: /etc/paccheck/mirrorlist exists in filesystem
local database is up to date

hope this minor issue will be corrected in the next release.

lucak3 · 2011-02-26 09:01:34

Why not use `wc -c $FILE | awk '{print $1}'` instead of `cat $FILE | wc -c`? I think it's faster.
Nonetheless, it's not like 'cat' is abysmally slow, it needs like 10s for a 300MB file...

IgnorantGuru · 2011-02-26 15:54:21

paccheck 0.8.8 is available with a few minor changes.

* The mirror info download should now work more reliably (eg problem corrected with archlinux.mirrors.uk2.net)

* packages not in the official repos will now be ignored, except when checking package sizes, which checks everything it can

* now reports repo/packagename instead of just packagename

kgas wrote:

* AUR PKGBUILD creates /etc/paccheck/mirrorlist
thanks. If it is there the installation/upgrade should not fail with the following error
error: failed to commit transaction (conflicting files)
paccheck: /etc/paccheck/mirrorlist exists in filesystem
local database is up to date
hope this minor issue will be corrected in the next release.

Thanks - This is explained here. You got this error because you created mirrorlist manually - thus it was not owned by paccheck, thus you received an error instead of it being saved as mirrorlist.pacnew. If you delete your mirrorlist, then install the AUR paccheck, then replace your mirrorlist, future updates should work correctly.

lucak3 wrote:

Why not use `wc -c $FILE | awk '{print $1}'` instead of `cat $FILE | wc -c`? I think it's faster.
Nonetheless, it's not like 'cat' is abysmally slow, it needs like 10s for a 300MB file...

The former won't actually read the bytes in the file, just the file size. But that is what I'm trying to avoid, as btrfs does not give reliable file sizes. cat actually sends each byte to wc, which is slower, but should give the correct file size (unless the file is actually corrupt, which is detected by pacman using the md5 sum).

IgnorantGuru · 2011-02-27 19:25:10

paccheck 0.8.10 fixes a url problem with the compare function. Also, I just realized that some official packages still use pkg.tar.gz instead of .xz, so 0.8.10 will include .gz packages in its tests.

And a compare mirror can now be a local dir, in which case nothing will be downloaded. Packages must already be in MIRROR/pkg/ This is mainly useful for scripts, or if you have a dir of packages already downloaded that you want to test against pacman's pkg cache.

IgnorantGuru · 2011-02-28 02:17:44

paccheck 0.8.11 is available, which adds two features.

paccheck --help

Compares Arch Linux pacman sync and package cache to multiple mirrors to help
detect compromised mirrors

Usage: paccheck [OPTIONS]
OPTIONS:
--install PKG [...] Download packages (without sync) and check ONLY those
                    packages, then offer to install
--compare 'MIRROR'  Fully download and compare all non-expired packages in
                    pacman's pkg cache to MIRROR.  Can alternatively be
                    listed in /etc/paccheck/mirrorlist as "compare=MIRROR".
                    MIRROR can also be local dir with packages in MIRROR/pkg/
--targets           Limit check and download to current update targets only
--verbose           Show debugging output
--keep              Don't remove temporary files in /tmp/paccheck.tmp
--alt-size          Use alternate slower test of package sizes (useful due
                    to stat bug with btrfs which gives inaccurate results)
--skip-size         Skip test of package sizes
--no-sync           No pacman update - mainly for use in scripts. paccheck
                    requires an updated pacman sync and package cache.
                    Before running "paccheck --no-sync" be sure to run:
                        sudo pacman -Sy
                        sudo pacman -w --noconfirm -Su

Full System Update Procedure:
   1) Run paccheck and examine report
   2) If no package MISMATCH then run "sudo pacman -Su" to update your system

Desired mirrors may be configured in /etc/paccheck/mirrorlist

NOTE: paccheck only tests these official repositories (if configured):
      core extra community community-staging community-testing
      gnome-unstable kde-unstable multilib multilib-testing staging testing

Exit Status:
    3  Package MISMATCH, download failures, or other errors
    2  Packages missing from some mirrors
    1  Out of sync mirrors (DATABASE CONTENT MISMATCH) or other warnings
    0  All OK

To simplify the process of installing new packages safely, the --install PKG option has been added. This will download the listed packages in pacman, then check only those targets (packages and dependencies), and then offer to install them. For example:

$ sudo pacman -Sy   # sync first if desired

$ paccheck --install gv

Reading mirrors from /etc/paccheck/mirrorlist...

NOTE: pacman must be synced and needed packages downloaded
      for paccheck to work properly.
Download needed packages without installing...
>>> sudo pacman -w --noconfirm -S  gv
Password: 
resolving dependencies...

Targets (2): xaw3d-1.5E-2  gv-3.7.1-2

Total Download Size:    0.37 MB

Proceed with download? [Y/n] 
:: Retrieving packages from extra...
 xaw3d-1.5E-2-x86_64     221.8K  209.7K/s 00:00:01 [######################] 100%
 gv-3.7.1-2-x86_64       157.2K  303.3K/s 00:00:01 [######################] 100%
checking package integrity...

TARGET LIST:  (only these packages will be checked)
xaw3d-1.5E-2
gv-3.7.1-2

Timestamps:
core.db.tar.gz: 2011-02-27 04:03:43 -0700
extra.db.tar.gz: 2011-02-27 16:13:43 -0700
community.db.tar.gz: 2011-02-27 12:48:31 -0700

========== DOWNLOADING ============

Downloading info on archlinux.mirrors.uk2.net
2011-02-27 19:10:33 URL:http://archlinux.mirrors.uk2.net/core/os/x86_64/core.db.tar.gz [37617/37617] -> "core.db.tar.gz" [1]
2011-02-27 19:10:38 URL:http://archlinux.mirrors.uk2.net/extra/os/x86_64/extra.db.tar.gz [475354/475354] -> "extra.db.tar.gz" [1]
2011-02-27 19:10:48 URL:http://archlinux.mirrors.uk2.net/community/os/x86_64/community.db.tar.gz [436352/436352] -> "community.db.tar.gz" [1]
FINISHED --2011-02-27 19:10:48--
Downloaded: 3 files, 927K in 0s (1768252 GB/s)

=========== ANALYZING =============

archlinux.mirrors.uk2.net:  Tier 1 (Great Britain) 
    core.db: OK
    extra.db: timestamp mismatch   CONTENT MISMATCH
    community.db: OK
    Scanning package database (due to database mismatch)...

Checking package sizes...

============ SUMMARY ==============

THERE WERE DATABASE CONTENT MISMATCHES - This usually indicates some
mirrors were out of date with your pacman sync, but alone does not
indicate compromised mirrors.

WARNING: USING MORE THAN ONE MIRROR IS RECOMMENDED


Install Package List: gv
Targets: 
    xaw3d-1.5E-2
    gv-3.7.1-2

Proceed with installation? [Y/n] 


>>> sudo pacman --noconfirm -S  gv
resolving dependencies...
looking for inter-conflicts...

Targets (2): xaw3d-1.5E-2  gv-3.7.1-2

Total Download Size:    0.00 MB
Total Installed Size:   1.48 MB

Proceed with installation? [Y/n] 
checking package integrity...
(2/2) checking for file conflicts                  [######################] 100%
(1/2) installing xaw3d                             [######################] 100%
(2/2) installing gv                                [######################] 100%

Note that no "pacman -Sy" is performed with install - you must do this manually first if desired.

Also, the new --targets option limits checking to only those packages due for update, instead of all the non-expired packages in the cache. This is mainly useful when used with compare, as it will reduce the amount downloaded, especially if you haven't cleaned out your cache lately. When used without compare, the benefit is limited - the same amount of data will be downloaded (the databases), and processing is fairly quick.

paccheck --targets

Last edited by IgnorantGuru (2011-02-28 02:26:45)

berbae · 2011-02-28 11:13:24

The '--install PKG' option is a useful logical adding to the script. Thanks for it.

As for the --targets option, I had already worked around that by moving all the currently installed packages in /var/cache/pacman/pkg to another directory.
So there are only the just downloaded and to be installed/updated packages in the pacman cache.
Because there is no need to verify all the installed packages every times a difference is found in the sync database files, and no need to always verify their size each times the script is used.
And the script runs very fast like that, without much cpu usage and disk accesses.

Just a little remark/wish :

I think that the OK results of

Scanning package database (due to database mismatch)...

Checking package sizes...

could be showed without the --verbose option activated.
So it will explicitly tell that these controls were good.
Thanks.

IgnorantGuru · 2011-02-28 13:58:14

berbae wrote:

I think that the OK results of
Scanning package database (due to database mismatch)...

Checking package sizes...
could be showed without the --verbose option activated.
So it will explicitly tell that these controls were good.

Thanks - I changed these messages to give an "all ok" and a file count. Also, I corrected a non-critical problem with --install which caused a 'sudo could not find working dir' message to appear. These changes are in 0.8.12.

IgnorantGuru · 2011-03-14 00:06:37

paccheck, as well as all the other downloads on my site, are now signed. At the top of each download page, you'll see a verify link in the Download Links section, which gives instructions for verifying that download. This is as simple as pasting a few lines into your terminal (you can even paste all the lines at once).

I have created a PGP key and signed all the current versions of the files available for download. The reason I took the time to do this is to improve your security. I recommend verifying downloads, especially in the case of paccheck.

Note that the AUR currently provides no way to verify signatures. For now I recommend following the 'verify' instructions prior to using the AUR to install paccheck, and also prior to installing any updates to it.

If you ever encounter a bad signature, please don’t ignore it, and let me know about it so I can check the server - thanks.

Arch Linux

#26 2011-02-25 07:06:35

Re: paccheck - pacman package authenticity check

#27 2011-02-25 07:38:41

Re: paccheck - pacman package authenticity check

#28 2011-02-25 07:55:41

Re: paccheck - pacman package authenticity check

#29 2011-02-25 08:02:08

Re: paccheck - pacman package authenticity check

#30 2011-02-25 08:22:42

Re: paccheck - pacman package authenticity check

#31 2011-02-25 08:33:53

Re: paccheck - pacman package authenticity check

#32 2011-02-25 08:52:57

Re: paccheck - pacman package authenticity check

#33 2011-02-25 09:14:28

Re: paccheck - pacman package authenticity check

#34 2011-02-25 14:46:45

Re: paccheck - pacman package authenticity check

#35 2011-02-25 15:01:40

Re: paccheck - pacman package authenticity check

#36 2011-02-25 16:09:28

Re: paccheck - pacman package authenticity check

#37 2011-02-25 16:18:02

Re: paccheck - pacman package authenticity check

#38 2011-02-25 17:14:13

Re: paccheck - pacman package authenticity check

#39 2011-02-25 18:20:48

Re: paccheck - pacman package authenticity check

#40 2011-02-25 19:04:16

Re: paccheck - pacman package authenticity check

#41 2011-02-25 19:09:58

Re: paccheck - pacman package authenticity check

#42 2011-02-25 19:21:16

Re: paccheck - pacman package authenticity check

#43 2011-02-26 08:27:01

Re: paccheck - pacman package authenticity check

#44 2011-02-26 09:01:34

Re: paccheck - pacman package authenticity check

#45 2011-02-26 15:54:21

Re: paccheck - pacman package authenticity check

#46 2011-02-27 19:25:10

Re: paccheck - pacman package authenticity check

#47 2011-02-28 02:17:44

Re: paccheck - pacman package authenticity check

#48 2011-02-28 11:13:24

Re: paccheck - pacman package authenticity check

#49 2011-02-28 13:58:14

Re: paccheck - pacman package authenticity check

#50 2011-03-14 00:06:37

Re: paccheck - pacman package authenticity check

Board footer