LinuxQuestions.org - Security Breach at kernel.org

- Linux - Security (http://www.linuxquestions.org/questions/linux-security-4/)

- - Security Breach at kernel.org (http://www.linuxquestions.org/questions/linux-security-4/security-breach-at-kernel-org-900485/)

Security Breach at kernel.org

Kernel.org is currently carrying this announcement:

Quote:

Security breach on kernel.org

Earlier this month, a number of servers in the kernel.org infrastructure were compromised. We discovered this August 28th. While we currently believe that the source code repositories were unaffected, we are in the process of verifying this and taking steps to enhance security across the kernel.org infrastructure.

Standby for Tech-media frenzy to ensue.

If the kernel source code was actually accessed/modified, the implications of this could be huge. Here's hoping it never got that far.

Thanks, I'll sticky this for a few days.

More details and more in this email.

Quote:

Originally Posted by MS3FGX (Post 4458401)

If the kernel source code was actually accessed/modified, the implications of this could be huge. Here's hoping it never got that far.

While a valid question it is IMO completely unnecessary to ask as www.kernel.org explains in detail:

Quote:

However, it's also useful to note that the potential damage of cracking kernel.org is far less than typical software repositories. That's because kernel development takes place using the git distributed revision control system, designed by Linus Torvalds. For each of the nearly 40,000 files in the Linux kernel, a cryptographically secure SHA-1 hash is calculated to uniquely define the exact contents of that file. Git is designed so that the name of each version of the kernel depends upon the complete development history leading up to that version. Once it is published, it is not possible to change the old versions without it being noticed.

Those files and the corresponding hashes exist not just on the kernel.org machine and its mirrors, but on the hard drives of each several thousand kernel developers, distribution maintainers, and other users of kernel.org. Any tampering with any file in the kernel.org repository would immediately be noticed by each developer as they updated their personal repository, which most do daily.

Quote:

Originally Posted by unSpawn (Post 4458533)

While a valid question it is IMO completely unnecessary to ask as www.kernel.org explains in detail:

While that is some comfort, it only really covers the stuff inside of git and there are the tarballs and patch files on the ftp site to be considered. I check the gpg signatures for these as a matter of routine, so as long as the ftpadmin_AT_kernel.org signing key hasn't been compromised I'm not overly concerned. Clarification on the status of that key would make me feel a little more comfortable.

update:

Just found this comment on the lwn story about the break in:

Quote:

Since asking these questions, I've been informed that downloadable tarballs could have been trojaned and then signed by the intruder on hera, implying that the private signing key + passphrase are normally present there.

In that light, the presence of *.sign files published alongside the tarballs isn't useful for ensuring security integrity of source tarballs on kernel.org. It's useful only for making sure that kernel.org mirrors correctly track the upstream site. Kernel tarballs on kernel.org can be vetted by generating them from an sha1-vetted git repo checkout, but that is currently the only way to check their integrity.

I'm a little surprised at that. Those *.sign files and the published Linux Kernel Archives OpenPGP key thus end up, IMO, being a little misleading.

I've no idea who the guy is, or where his information comes from, but if true, then I think he's right when he says that the signatures are of limited value and give a false sense of security. I'm suddenly feeling less comfortable again.

update 2:
Seems he was right...

http://www.kernel.org/signature.html

Quote:

Files placed in the Linux Kernel Archives are automatically OpenPGP-signed by the archive. This signature can be used to prove that a file, which may have been obtained from a mirror site or other location, really originated at the Linux Kernel Archives.

This signature does not guarantee that the Linux Kernel Archives master site itself has not been compromised. However, if we suffer an intrusion we will revoke the key and post information here as quickly as possible.

So, if you don't get your files from a mirror, verifying the signatures is pointless. This is disappointing.

Quote:

Originally Posted by unSpawn (Post 4458533)

While a valid question it is IMO completely unnecessary to ask as www.kernel.org explains in detail

Well, I didn't actually ask a question to begin with, but the information you posted hardly rules out the possibility of a compromised tarball getting on the FTP; it just means that the working kernel source in git is safe.

A non-answer to a non-question, it would seem.

I remember several years ago the Linux kernel sources were attacked and a malware payload code was uploaded but was caught in time through commit changes and the malicious code was removed before any damage could be done.

Quote:

Originally Posted by GazL (Post 4458662)

While that is some comfort, it only really covers the stuff inside of git and there are the tarballs and patch files on the ftp site to be considered.

Indeed, and this doesn't diminish anything wrt contamination or Hans Peter Anvin having his colo box rooted or Hera having been rooted for too long anyway (as they have an admin on the payroll), but at least it means there's a source that can be verified to still be untainted and tarballs can be regenerated from that.

Quote:

Originally Posted by GazL (Post 4458662)

Seems he was right...
So, if you don't get your files from a mirror, verifying the signatures is pointless. This is disappointing.

That it is.

The perspective of one semi-clueless Debian user

Rick Moen said (at lwn.net):

Quote:

What I've basically been hearing amounts to: 'Silly sysadmin, don't download kernel tarballs from kernel.org and expect to authenticate them by checking the accompanying .sign file using the published gpg key. If you must authenticate a tarball, pull down the sources from the git repo thereby ensuring the sources' integrity, and then tar it up.'

I might now do that, but the point is that many of us have been accustomed to ftp'ing (etc.) Linux kernel tarballs since pre-1.0 days, and many people will naturally interpret the presence alongside those tarballs of .sign files plus the published gpg key as suggesting a means for the public to verify code signature that had been made using a carefully guarded private key. Given that that is not the case, I would humbly suggest that the kernel.org operators, at some point, see if that perceptual pitfall can be eliminated.

Broadening such concerns from the linux kernel source code to the iso images used by most users to install a linux distribution, and the debs/rpms used by most users to install packages in some distribution:

When I recently installed Squeeze on a PC, I was offered a choice of mirrors from which to get all the debs not on the installation CD#1. One of the first choices was kernel.org, and I actually thought about asking for that repo specifically, hoping that they would care more about security than some random academic mirror.

A year ago I tried hard to get clarification from debian.org about just what the gpg signed files with the hashes for the various Debian iso images accomplish, and what they do not. I was told to take it to the forums--- where I got no answers at all. I too very much desire authoritative clarification on exactly how the ordinary user (who is perforce acting as the sysadmin of his/her own PC) is to understand the hashes and gpg signatures at places like this (for Debian linux users installing Squeeze):
http://cdimage.debian.org/cdimage/re...t/i386/iso-cd/
My best guess is that the Debian installer does automatically load and import into the new system's apt-keyring the signing keys contained in the iso image it finds at this website (and your bad luck if the site has been compromised, unless you are connected by the GPG Web of Trust to the signer(s) and check the key post-install), verifies the signature of the file with the SHA1 hashes for the various iso images, and verifies the hashes of individual packages before installation.

Furthermore, as far as I know, when the user uses apt-get (or synaptic or...) to install or upgrade debs, the SHA1 hash (and/or GPG signature?) of that deb is checked before installation. At least, when I tried to install a contributed package signed with a key not included in the Debian developer's keyring, which I had forgotten to import, I did get a warning.

I hope I do not seriously overestimate the security of Debian package management, especially because I have long believed that the GPG signatures could be replaced by an intruder without most new Debian users noticing. I'd probably notice because I previously installed what I believe to be the genuine keys, but only because I found them at debian.org---- it's worrying that at least one Debian signing key was only self-signed, last I checked. It would be good to have a hundred signatures for security-critical keys whose compromise would affect thousands of users.

Does anyone know more?

While everyone talks about possibly compromised tarballs with (technically) valid signatures, I wonder: Which developer account got compromised, and did the attacker inject some (obfuscated) code into the valid check-ins of the developer?

I am rather sure this would not survive the code audits by other devs, but anyway...

I guess it all depends on how they generate the tarballs, how they are copied into the archive and at what point they get signed.

It's the 'automatically signed' that make me raise an eyebrow, but I really don't know anything about their signing methodology.
I just hope that it's not something as dumb as the following in a cron job on the server itself (which was root compromised)

Code:

#!/bin/bash



for file in /pub/archive/*

do



   test -f "${file}.sig" \ 

   || gpg --detach-sign --local-user ftpadmin@kernel.org --passphrase "Idiotic use of gpg signing" "$file"



done

Oh, I forgot that every git commit is signed in the first place. That means the git sources cannot be compromised if just one commit access to kernel.org was hacked, because the signing happens on a clean development machine.

Quote:

Originally Posted by GazL (Post 4461949)

II just hope that it's not something as dumb as ...

:redface:

D'oh.

kernel org security breach

pinched this from another forum

Quote:

Kernel.org has been compromised by an intruder gaining root access to parts of their infrastructure which hosts the kernel source code.

A number of servers have been accessed, apparently via compromised user credentials. The intruder installed several rootkits and monitored user activity.

The intrusion went unnoticed for almost a month until the kernel.org staff discovered it on August 28th.

What happened?

* Intruders gained root access on the server Hera. We believe they may have gained this access via a compromised user credential; how they managed to exploit that to root access is currently unknown and is being investigated.
* Files belonging to ssh (openssh, openssh-server and openssh-clients) were modified and running live.
* A trojan startup file was added to the system start up scripts
User interactions were logged, as well as some exploit code. We have retained this for now.
* Trojan initially discovered due to the Xnest /dev/mem error message w/o Xnest installed; have been seen on other systems. It is unclear if systems that exhibit this message are susceptible, compromised or not. If developers see this, and you don't have Xnest installed, please investigate.
* It *appears* that 3.1-rc2 might have blocked the exploit injector, we don't know if this is intentional or a side affect of another bugfix or change.

They list what they’ve done so far as a response to the security breach:

* We have currently taken boxes off line to do a backup and are in the process of doing complete reinstalls.
* We have notified authorities in the United States and in Europe to assist with the investigation
* We will be doing a full reinstall on all boxes on kernel.org
* We are in the process of doing an analysis on the code within git, and the tarballs to confirm that nothing has been modified']

http://kernel.org/

the site is down for maintenance as I write
looks like serious biz

floppy

That was posted about already on 31/aug/11 (http://www.linuxquestions.org/questi...el-org-900485/). AFAIK the LKML should have a post from Linus telling he will only use Git for the time being.