You are not logged in.

#1 2011-08-31 23:22:43

ssri
Member
Registered: 2010-02-16
Posts: 216

kernel.org - Security Breach

---------- Forwarded message ----------
    From: J.H. <warthog9@kernel.org>
    Date: 2011/8/29
    Subject: [kernel.org users] [KORG] Master back-end break-in
    To: users@kernel.org
     
     
    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1
     
    Afternoon Everyone,
     
    As you can guess from the subject line, I've not had what many would
    consider a "good" day.  Earlier today discovered a trojan existing on
    HPA's personal colo machine, as well as hera.  Upon some investigation
    there are a couple of kernel.org boxes, specifically hera and odin1,
    with potential pre-cursors on demeter2, zeus1 and zeus2, that have been
    hit by this.
     
    As it stands right now, HPA is working on cleaning his box, and
    I'm working on hera (odin1 and zeus1 are out of rotation still for other
    reasons), mainly so that if one of us finds something of interest, we
    can deal with it and compare notes on the other box.
     
    Points of interest:
     
    - - Break-in seems to have initially occurred no later than August 12th
     
    - - Files belonging to ssh (openssh, openssh-server and openssh-clients)
    were modified and running live.  These have been uninstalled and
    removed, all processes were killed and known good copies were
    reinstalled.  That said all users may wish to consider taking this
    opportunity to change their passwords and update ssh keys (particularly
    if you had an ssh private key on hera).  This seems to have occurred on
    or around August 19th.
     
    - - A trojan startup file was added to rc3.d
     
    - - User interactions were logged, as well as some exploit code.  We have
    retained this for now.
     
    - - Trojan initially discovered due to the Xnest /dev/mem error message
    w/o Xnest installed; have been seen on other systems.  It is unclear if
    systems that exhibit this message are susceptible, compromised or not.
    If you see this, and you don't have Xnest installed, please investigate.
     
    - - It *appears* that 3.1-rc2 might have blocked the exploit injector, we
    don't know if this is intentional or a side affect of another bugfix or
    change.
     
    - - System is being verified from backups, signatures, etc.  As of right
    now things look correct, however we may take the system down soon to do
    a full reinstall and for more invasive checking.
     
    - - As a precaution a number of packages have been removed from the
    system, if something was removed that you were using please let us know
    so we can put it back.
     
    - - At this time we do not know the vector that was used to get into the
    systems, but the attackers had gained root access level privileges.
     
    That's what we know right now, some of the recent instabilities may have
    been caused by these intrusions, and we are looking into everything.
     
    If you are on the box, keep an eye out, and if you see something please
    let us know immediately.
     
    Beyond that, verify your git trees and make sure things are correct.
     
    - - John 'Warthog9' Hawley
    Chief Kernel.org Administrator
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.4.11 (GNU/Linux)
    Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
     
    iEYEARECAAYFAk5a5U0ACgkQ/E3kyWU9dif+1ACfYPlgq/keFrFO77AmQVduKGwx
    TAcAnRAu6nHt74+5aC+fPeb8aT0hcy2K
    =Semd
    -----END PGP SIGNATURE-----

It looks like the attack targeted mainly the kernel development.  I wonder if the machines that housed software repos for various distros were compromised too?

Offline

#2 2011-08-31 23:26:54

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: kernel.org - Security Breach

Any links? It's already 1st Sept here and that's the first I hear about it.


Update: Some googling shows http://marc.info/?a=113904295100002&r=1&w=2 J. "warthog9" H.'s e-mails and the one you posted isn't there.

Last edited by karol (2011-08-31 23:38:33)

Offline

#3 2011-09-01 00:04:52

ssri
Member
Registered: 2010-02-16
Posts: 216

Re: kernel.org - Security Breach

karol wrote:

Any links? It's already 1st Sept here and that's the first I hear about it.


Update: Some googling shows http://marc.info/?a=113904295100002&r=1&w=2 J. "warthog9" H.'s e-mails and the one you posted isn't there.

The news is on the main page: https://www.kernel.org/

Offline

#4 2011-09-01 00:09:51

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: kernel.org - Security Breach

Ah, thanks.
I never knew there was a 'Site News' section on kernel.org ;P

Offline

#5 2011-09-01 00:13:03

ssri
Member
Registered: 2010-02-16
Posts: 216

Re: kernel.org - Security Breach

Wow, the news travels pretty fast...

Offline

#6 2011-09-01 00:39:23

Brandon_R
Member
Registered: 2011-08-22
Posts: 36

Re: kernel.org - Security Breach

Oh oh. I pacmaned -syu and got a new version of the linux kernel and i think i saw a new update for the linux kernel.. Am i infected?

Offline

#7 2011-09-01 01:16:51

Misfit138
Misfit Emeritus
From: USA
Registered: 2006-11-27
Posts: 4,189

Re: kernel.org - Security Breach

Good thing kernel.org runs on Linux...
wait...

Offline

#8 2011-09-01 02:08:56

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: kernel.org - Security Breach

I just saw this on Slashdot now (and came to post about it here if no one else had).

Cue revitalized package signing discussion in 5, 4...


Brandon_R wrote:

Oh oh. I pacmaned -syu and got a new version of the linux kernel and i think i saw a new update for the linux kernel.. Am i infected?

If the Arch mirror was compromised then it's possible, even if my first impulse is to say that it's unlikely. Arch is a relatively big distro though, so it may be worth the effort to target it, and hacking the kernel package is probably not too hard to do given the simplicity of Arch packages.

Considering the possibility, this should be mentioned on the front page news, at least until we know more.



If you want to reassure yourself, you could boot up your system with a live CD, generate checksums of updated files and compare them to packages from other servers. You could also reinstall all recently updated packages from a different server (via the live cd). Rebuild the kernel image and re-install the boot loader too... or just back up your data and re-install the whole system if you're really paranoid.

Here's a script that can generate sha512 checksums for all files in a package:

#!/bin/bash

old_ifs="$IFS"
IFS="
"
for _file in $(pacman -Qql "$@"); do
  [[ -f $_file ]] && sha512sum -- "$_file"
done

IFS="$old_ifs"

Just invoke it with package names (e.g. "linux", "grub") and redirect the output to a file.




Until we get package signing, you can get a little more security if you change your main mirror to ftp.archlinux.org. It's throttled, but if you Install pacman2aria2 then you can download packages with powerpill-light. It will download packages in parallel from multiple servers, but all downloads will be checked against the database checksums from archlinux.org. The only way to get infected then is if archlinux.org is compromised (which would compromise all mirrors), or if someone manages to hack a package on some server without changing the checksum, which is prohibitively difficult.


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#9 2011-09-01 03:01:31

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: kernel.org - Security Breach

Or just check the package signature that is on the mirrors beside the package file.  No idea why people think it would only affect the linux kernel package though... Surely every package on a kernel.org mirror is at risk (if indeed the server that was accessed has any access to the mirrored content...)

Also, ftp.archlinux.org is just a mirror and no different to any other mirror.  So using that give no security boost.

Offline

#10 2011-09-01 03:06:39

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: kernel.org - Security Breach

Allan wrote:

No idea why people think it would only affect the linux kernel package though... Surely every package on a kernel.org mirror is at risk (if indeed the server that was accessed has any access to the mirrored content...)

Not everyone knows it's just a mirror, some people might imagine this is where the kernel packages comes from.

Offline

#11 2011-09-01 03:07:08

Xyne
Administrator/PM
Registered: 2008-08-03
Posts: 6,963
Website

Re: kernel.org - Security Breach

Allan wrote:

Surely every package on a kernel.org mirror is at risk

Who said otherwise?

*edit*
Nevermind. I see how you can interpret Brandon_R's post that way.


Allan wrote:

Also, ftp.archlinux.org is just a mirror and no different to any other mirror.  So using that give no security boost.

I thought other mirrors synced from the server that hosts ftp.archlinux.org. Is there really no way to access database files or checksums directly from the release server?

Last edited by Xyne (2011-09-01 03:09:18)


My Arch Linux StuffForum EtiquetteCommunity Ethos - Arch is not for everyone

Offline

#12 2011-09-01 03:21:24

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: kernel.org - Security Breach

Xyne wrote:
Allan wrote:

Also, ftp.archlinux.org is just a mirror and no different to any other mirror.  So using that give no security boost.

I thought other mirrors synced from the server that hosts ftp.archlinux.org. Is there really no way to access database files or checksums directly from the release server?

Not really.  Although you could compare databases across tier 1 mirrors and check for differences.   Someone did implement that once...

Offline

#13 2011-09-01 03:22:42

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: kernel.org - Security Breach

Allan wrote:
Xyne wrote:
Allan wrote:

Also, ftp.archlinux.org is just a mirror and no different to any other mirror.  So using that give no security boost.

I thought other mirrors synced from the server that hosts ftp.archlinux.org. Is there really no way to access database files or checksums directly from the release server?

Not really.  Although you could compare databases across tier 1 mirrors and check for differences.   Someone did implement that once...

https://bbs.archlinux.org/viewtopic.php?id=113917 ?

Offline

#14 2011-09-01 03:31:17

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: kernel.org - Security Breach

For what it is worth, I just did a brief repo db comparison (only for i686, [core] and [extra]) and found no differences between that on ftp.kernel.org and the master server.  That means any package exploit would require a md5sum match.  Of course, that is now and any prior attempt could have been rsynced away...

Offline

#15 2011-09-01 15:47:11

Gullible Jones
Member
Registered: 2004-12-29
Posts: 4,863

Re: kernel.org - Security Breach

Quick question - the kernel.org people say that, because of the way git works, they'll be able to see if anything was changed in the kernel tree. But if the intruder had root, wouldn't it have been possible for them to manually mess around with the git metadata, covering their tracks?

Offline

#16 2011-09-01 17:18:10

ethail
Member
From: Spain
Registered: 2011-02-10
Posts: 225

Re: kernel.org - Security Breach

With my little understanding of git, that would mess with git on the server,  but not with the git of the other 99.9% of the world. Other people will still be able to see changes from the code 1 second before attack and 1 second later.

Say archlinux svntogit repos get attacked and messed up. If I pull from them, I still see changes

Last edited by ethail (2011-09-01 17:19:38)


My GitHub Page

Best Testing Repo Warning: [testing] means it can eat you hamster, catch fire and you should keep it away from children. And I'm serious here, it's not an April 1st joke.

Offline

#17 2011-09-01 19:13:23

Leonid.I
Member
From: Aethyr
Registered: 2009-03-22
Posts: 999

Re: kernel.org - Security Breach

Allan wrote:

For what it is worth, I just did a brief repo db comparison (only for i686, [core] and [extra]) and found no differences between that on ftp.kernel.org and the master server.  That means any package exploit would require a md5sum match.  Of course, that is now and any prior attempt could have been rsynced away...

Confirmed for [core] 64bit.

I have updated on Aug. 31 and today compared md5sums for each intalled pkg against {core,extra}.db.tar.gz from ftp.archlinux.org -- 100% agreement. Unless someone generated an md5sum collision (if it's ever practical) or hacked ftp.archlinux.org simultaneously with kernel.org, I think this a good indication that arch mirrors @kernel.org are unlikely  to be affected.


Arch Linux is more than just GNU/Linux -- it's an adventure
pkill -9 systemd

Offline

#18 2011-09-01 19:35:21

1archgamenon2
Banned
Registered: 2011-07-12
Posts: 201

Re: kernel.org - Security Breach

Brandon_R wrote:

Am i infected?

Idem!...
seen on some bsd forum =>http://ompldr.org/vYTVtcQ

Last edited by 1archgamenon2 (2011-09-01 19:47:10)

Offline

#19 2011-09-01 20:48:09

barzam
Member
From: Sweden
Registered: 2009-01-27
Posts: 277

Re: kernel.org - Security Breach

ethail wrote:

With my little understanding of git, that would mess with git on the server,  but not with the git of the other 99.9% of the world. Other people will still be able to see changes from the code 1 second before attack and 1 second later.

Say archlinux svntogit repos get attacked and messed up. If I pull from them, I still see changes

I think this is a very valid point. From the kernel.org post:

However, it's also useful to note that the potential damage of cracking kernel.org is far less than typical software repositories. That's because kernel development takes place using the git distributed revision control system, designed by Linus Torvalds. For each of the nearly 40,000 files in the Linux kernel, a cryptographically secure SHA-1 hash is calculated to uniquely define the exact contents of that file. Git is designed so that the name of each version of the kernel depends upon the complete development history leading up to that version. Once it is published, it is not possible to change the old versions without it being noticed.


Those files and the corresponding hashes exist not just on the kernel.org machine and its mirrors, but on the hard drives of each several thousand kernel developers, distribution maintainers, and other users of kernel.org. Any tampering with any file in the kernel.org repository would immediately be noticed by each developer as they updated their personal repository, which most do daily.

Offline

#20 2011-09-04 05:53:11

Kilzool
Member
From: Ireland
Registered: 2010-08-04
Posts: 232

Re: kernel.org - Security Breach

http://kernel.org/  -- read the news.

Last edited by Kilzool (2011-09-04 05:55:01)

Offline

#21 2011-09-04 06:02:40

ewaller
Administrator
From: Pasadena, CA
Registered: 2009-07-13
Posts: 19,739

Re: kernel.org - Security Breach

Moderator: Merged threads


Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Offline

#22 2011-09-11 16:49:39

fsckd
Forum Fellow
Registered: 2009-06-15
Posts: 4,173

Re: kernel.org - Security Breach

The plot thickens: http://linux.com/


aur S & M :: forum rules :: Community Ethos
Resources for Women, POC, LGBT*, and allies

Offline

#23 2011-09-11 16:59:35

WorMzy
Forum Moderator
From: Scotland
Registered: 2010-06-16
Posts: 11,783
Website

Re: kernel.org - Security Breach

Blimey.

Looks like somebody's got it in for the Linux community. neutral


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#24 2011-09-11 17:26:09

Grinch
Member
Registered: 2010-11-07
Posts: 265

Re: kernel.org - Security Breach

Likely compromised credentials from the first breach was used to compromise the other. That said, Linux.com can't have been much of a prize given that it's just a Linux promotional site and has nothing to do with Linux development.

Offline

#25 2011-09-11 17:26:23

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: kernel.org - Security Breach

There goes security through obscurity ;-)

Offline

Board footer

Powered by FluxBB