LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 08-31-2011, 07:44 PM   #1
GazL
LQ Veteran
 
Registered: May 2008
Posts: 6,873

Rep: Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982
Exclamation Security Breach at kernel.org


Kernel.org is currently carrying this announcement:
Quote:
Security breach on kernel.org

Earlier this month, a number of servers in the kernel.org infrastructure were compromised. We discovered this August 28th. While we currently believe that the source code repositories were unaffected, we are in the process of verifying this and taking steps to enhance security across the kernel.org infrastructure.

Standby for Tech-media frenzy to ensue.
 
Old 08-31-2011, 08:00 PM   #2
MS3FGX
LQ Guru
 
Registered: Jan 2004
Location: NJ, USA
Distribution: Slackware, Debian
Posts: 5,852

Rep: Reputation: 361Reputation: 361Reputation: 361Reputation: 361
If the kernel source code was actually accessed/modified, the implications of this could be huge. Here's hoping it never got that far.
 
1 members found this post helpful.
Old 08-31-2011, 10:43 PM   #3
win32sux
LQ Guru
 
Registered: Jul 2003
Location: Los Angeles
Distribution: Ubuntu
Posts: 9,870

Rep: Reputation: 380Reputation: 380Reputation: 380Reputation: 380
Thanks, I'll sticky this for a few days.
 
Old 09-01-2011, 12:25 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
More details and more in this email.


Quote:
Originally Posted by MS3FGX View Post
If the kernel source code was actually accessed/modified, the implications of this could be huge. Here's hoping it never got that far.
While a valid question it is IMO completely unnecessary to ask as www.kernel.org explains in detail:
Quote:
However, it's also useful to note that the potential damage of cracking kernel.org is far less than typical software repositories. That's because kernel development takes place using the git distributed revision control system, designed by Linus Torvalds. For each of the nearly 40,000 files in the Linux kernel, a cryptographically secure SHA-1 hash is calculated to uniquely define the exact contents of that file. Git is designed so that the name of each version of the kernel depends upon the complete development history leading up to that version. Once it is published, it is not possible to change the old versions without it being noticed.


Those files and the corresponding hashes exist not just on the kernel.org machine and its mirrors, but on the hard drives of each several thousand kernel developers, distribution maintainers, and other users of kernel.org. Any tampering with any file in the kernel.org repository would immediately be noticed by each developer as they updated their personal repository, which most do daily.
 
1 members found this post helpful.
Old 09-01-2011, 04:15 AM   #5
GazL
LQ Veteran
 
Registered: May 2008
Posts: 6,873

Original Poster
Rep: Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982
Quote:
Originally Posted by unSpawn View Post
While a valid question it is IMO completely unnecessary to ask as www.kernel.org explains in detail:
While that is some comfort, it only really covers the stuff inside of git and there are the tarballs and patch files on the ftp site to be considered. I check the gpg signatures for these as a matter of routine, so as long as the ftpadmin_AT_kernel.org signing key hasn't been compromised I'm not overly concerned. Clarification on the status of that key would make me feel a little more comfortable.


update:

Just found this comment on the lwn story about the break in:
Quote:
Since asking these questions, I've been informed that downloadable tarballs could have been trojaned and then signed by the intruder on hera, implying that the private signing key + passphrase are normally present there.

In that light, the presence of *.sign files published alongside the tarballs isn't useful for ensuring security integrity of source tarballs on kernel.org. It's useful only for making sure that kernel.org mirrors correctly track the upstream site. Kernel tarballs on kernel.org can be vetted by generating them from an sha1-vetted git repo checkout, but that is currently the only way to check their integrity.

I'm a little surprised at that. Those *.sign files and the published Linux Kernel Archives OpenPGP key thus end up, IMO, being a little misleading.
I've no idea who the guy is, or where his information comes from, but if true, then I think he's right when he says that the signatures are of limited value and give a false sense of security. I'm suddenly feeling less comfortable again.


update 2:
Seems he was right...

http://www.kernel.org/signature.html
Quote:
Files placed in the Linux Kernel Archives are automatically OpenPGP-signed by the archive. This signature can be used to prove that a file, which may have been obtained from a mirror site or other location, really originated at the Linux Kernel Archives.

This signature does not guarantee that the Linux Kernel Archives master site itself has not been compromised. However, if we suffer an intrusion we will revoke the key and post information here as quickly as possible.
So, if you don't get your files from a mirror, verifying the signatures is pointless. This is disappointing.

Last edited by GazL; 09-01-2011 at 04:59 AM.
 
Old 09-01-2011, 06:59 PM   #6
MS3FGX
LQ Guru
 
Registered: Jan 2004
Location: NJ, USA
Distribution: Slackware, Debian
Posts: 5,852

Rep: Reputation: 361Reputation: 361Reputation: 361Reputation: 361
Quote:
Originally Posted by unSpawn View Post
While a valid question it is IMO completely unnecessary to ask as www.kernel.org explains in detail
Well, I didn't actually ask a question to begin with, but the information you posted hardly rules out the possibility of a compromised tarball getting on the FTP; it just means that the working kernel source in git is safe.

A non-answer to a non-question, it would seem.
 
Old 09-01-2011, 10:13 PM   #7
ReaperX7
LQ Guru
 
Registered: Jul 2011
Location: California
Distribution: Slackware64-15.0 Multilib
Posts: 6,554
Blog Entries: 15

Rep: Reputation: 2097Reputation: 2097Reputation: 2097Reputation: 2097Reputation: 2097Reputation: 2097Reputation: 2097Reputation: 2097Reputation: 2097Reputation: 2097Reputation: 2097
I remember several years ago the Linux kernel sources were attacked and a malware payload code was uploaded but was caught in time through commit changes and the malicious code was removed before any damage could be done.
 
Old 09-02-2011, 12:54 AM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by GazL View Post
While that is some comfort, it only really covers the stuff inside of git and there are the tarballs and patch files on the ftp site to be considered.
Indeed, and this doesn't diminish anything wrt contamination or Hans Peter Anvin having his colo box rooted or Hera having been rooted for too long anyway (as they have an admin on the payroll), but at least it means there's a source that can be verified to still be untainted and tarballs can be regenerated from that.


Quote:
Originally Posted by GazL View Post
Seems he was right...
So, if you don't get your files from a mirror, verifying the signatures is pointless. This is disappointing.
That it is.
 
Old 09-02-2011, 04:09 PM   #9
Peufelon
Member
 
Registered: Jul 2005
Posts: 164
Blog Entries: 1

Rep: Reputation: Disabled
The perspective of one semi-clueless Debian user

Rick Moen said (at lwn.net):
Quote:
What I've basically been hearing amounts to: 'Silly sysadmin, don't download kernel tarballs from kernel.org and expect to authenticate them by checking the accompanying .sign file using the published gpg key. If you must authenticate a tarball, pull down the sources from the git repo thereby ensuring the sources' integrity, and then tar it up.'

I might now do that, but the point is that many of us have been accustomed to ftp'ing (etc.) Linux kernel tarballs since pre-1.0 days, and many people will naturally interpret the presence alongside those tarballs of .sign files plus the published gpg key as suggesting a means for the public to verify code signature that had been made using a carefully guarded private key. Given that that is not the case, I would humbly suggest that the kernel.org operators, at some point, see if that perceptual pitfall can be eliminated.
Broadening such concerns from the linux kernel source code to the iso images used by most users to install a linux distribution, and the debs/rpms used by most users to install packages in some distribution:

When I recently installed Squeeze on a PC, I was offered a choice of mirrors from which to get all the debs not on the installation CD#1. One of the first choices was kernel.org, and I actually thought about asking for that repo specifically, hoping that they would care more about security than some random academic mirror.

A year ago I tried hard to get clarification from debian.org about just what the gpg signed files with the hashes for the various Debian iso images accomplish, and what they do not. I was told to take it to the forums--- where I got no answers at all. I too very much desire authoritative clarification on exactly how the ordinary user (who is perforce acting as the sysadmin of his/her own PC) is to understand the hashes and gpg signatures at places like this (for Debian linux users installing Squeeze):
http://cdimage.debian.org/cdimage/re...t/i386/iso-cd/
My best guess is that the Debian installer does automatically load and import into the new system's apt-keyring the signing keys contained in the iso image it finds at this website (and your bad luck if the site has been compromised, unless you are connected by the GPG Web of Trust to the signer(s) and check the key post-install), verifies the signature of the file with the SHA1 hashes for the various iso images, and verifies the hashes of individual packages before installation.

Furthermore, as far as I know, when the user uses apt-get (or synaptic or...) to install or upgrade debs, the SHA1 hash (and/or GPG signature?) of that deb is checked before installation. At least, when I tried to install a contributed package signed with a key not included in the Debian developer's keyring, which I had forgotten to import, I did get a warning.

I hope I do not seriously overestimate the security of Debian package management, especially because I have long believed that the GPG signatures could be replaced by an intruder without most new Debian users noticing. I'd probably notice because I previously installed what I believe to be the genuine keys, but only because I found them at debian.org---- it's worrying that at least one Debian signing key was only self-signed, last I checked. It would be good to have a hundred signatures for security-critical keys whose compromise would affect thousands of users.

Does anyone know more?

Last edited by Peufelon; 09-02-2011 at 04:48 PM.
 
Old 09-05-2011, 05:56 AM   #10
cepheus11
Member
 
Registered: Nov 2010
Location: Germany
Distribution: Gentoo
Posts: 286

Rep: Reputation: 91
While everyone talks about possibly compromised tarballs with (technically) valid signatures, I wonder: Which developer account got compromised, and did the attacker inject some (obfuscated) code into the valid check-ins of the developer?

I am rather sure this would not survive the code audits by other devs, but anyway...
 
Old 09-05-2011, 06:52 AM   #11
GazL
LQ Veteran
 
Registered: May 2008
Posts: 6,873

Original Poster
Rep: Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982Reputation: 4982
I guess it all depends on how they generate the tarballs, how they are copied into the archive and at what point they get signed.

It's the 'automatically signed' that make me raise an eyebrow, but I really don't know anything about their signing methodology.
I just hope that it's not something as dumb as the following in a cron job on the server itself (which was root compromised)
Code:
#!/bin/bash

for file in /pub/archive/*
do

   test -f "${file}.sig" \ 
   || gpg --detach-sign --local-user ftpadmin@kernel.org --passphrase "Idiotic use of gpg signing" "$file"

done

Last edited by GazL; 09-05-2011 at 07:49 AM.
 
Old 09-05-2011, 07:12 AM   #12
cepheus11
Member
 
Registered: Nov 2010
Location: Germany
Distribution: Gentoo
Posts: 286

Rep: Reputation: 91
Oh, I forgot that every git commit is signed in the first place. That means the git sources cannot be compromised if just one commit access to kernel.org was hacked, because the signing happens on a clean development machine.
 
Old 09-05-2011, 07:14 AM   #13
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 21,099

Rep: Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117Reputation: 4117
Quote:
Originally Posted by GazL View Post
II just hope that it's not something as dumb as ...


D'oh.
 
Old 09-11-2011, 02:24 AM   #14
floppywhopper
Member
 
Registered: Aug 2004
Location: Western Australia
Distribution: Mageia , Centos
Posts: 643
Blog Entries: 2

Rep: Reputation: 136Reputation: 136
kernel org security breach

pinched this from another forum
Quote:
Kernel.org has been compromised by an intruder gaining root access to parts of their infrastructure which hosts the kernel source code.

A number of servers have been accessed, apparently via compromised user credentials. The intruder installed several rootkits and monitored user activity.

The intrusion went unnoticed for almost a month until the kernel.org staff discovered it on August 28th.

What happened?

* Intruders gained root access on the server Hera. We believe they may have gained this access via a compromised user credential; how they managed to exploit that to root access is currently unknown and is being investigated.
* Files belonging to ssh (openssh, openssh-server and openssh-clients) were modified and running live.
* A trojan startup file was added to the system start up scripts
User interactions were logged, as well as some exploit code. We have retained this for now.
* Trojan initially discovered due to the Xnest /dev/mem error message w/o Xnest installed; have been seen on other systems. It is unclear if systems that exhibit this message are susceptible, compromised or not. If developers see this, and you don't have Xnest installed, please investigate.
* It *appears* that 3.1-rc2 might have blocked the exploit injector, we don't know if this is intentional or a side affect of another bugfix or change.

They list what they’ve done so far as a response to the security breach:

* We have currently taken boxes off line to do a backup and are in the process of doing complete reinstalls.
* We have notified authorities in the United States and in Europe to assist with the investigation
* We will be doing a full reinstall on all boxes on kernel.org
* We are in the process of doing an analysis on the code within git, and the tarballs to confirm that nothing has been modified']

http://kernel.org/
the site is down for maintenance as I write
looks like serious biz

floppy
 
Old 09-11-2011, 04:20 AM   #15
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
That was posted about already on 31/aug/11 (http://www.linuxquestions.org/questi...el-org-900485/). AFAIK the LKML should have a post from Linus telling he will only use Git for the time being.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Yet another thread about a security breach Fredde87 Linux - Security 19 10-16-2009 08:12 AM
[SOLVED] possible security breach johnh10000 Linux - Security 18 10-13-2009 11:23 AM
Breach in Sendmail Security? bper Linux - Security 2 08-02-2005 05:40 PM
[Security Questions] Last Login, how good is this feature for security breach info? t3gah Linux - Security 2 06-14-2005 01:02 AM
Security breach? lhoff Linux - Security 5 02-15-2002 01:33 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 11:29 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration