LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices

Reply
 
Search this Thread
Old 04-02-2013, 01:40 AM   #781
JZL240I-U
Senior Member
 
Registered: Apr 2003
Location: Germany
Distribution: openSuSE 13.1 / 12.3_64-KDE, Ubuntu 14.04, Fedora 20, Mint 17, Chakra
Posts: 3,650

Rep: Reputation: Disabled

Okay, that's what I was afraid of but thank you .
 
Old 05-09-2013, 03:59 AM   #782
xl
Member
 
Registered: Feb 2013
Location: Germany
Distribution: Wheezy
Posts: 46

Rep: Reputation: 7
I just realized that it's much faster to zero fill a drive after it's been formatted. Don't know what the reason is maybe file system enhancements or alignment ...
So I used gparted (0.7) to format the whole drive ext4 and create a big file like this:
Code:
sudo dd if=/dev/zero of=/media/DRIVE/big.file bs=4k
It's 135 instead of 100 MB/s and should be as fast as the vendor's zero fill.

Last edited by xl; 05-09-2013 at 04:05 AM.
 
Old 05-09-2013, 09:49 AM   #783
xl
Member
 
Registered: Feb 2013
Location: Germany
Distribution: Wheezy
Posts: 46

Rep: Reputation: 7
Alright, seems it drops again to about 100 MB/s at the end at least on a 2TB drive.
I'm still looking for a way to create several random files in a loop. Something like:
Code:
while true; do sudo dd if=/dev/urandom of=/media/DRIVE/file.$RANDOM bs=4k count=$RANDOM; done
This will create files from 2-120 MB but can only be aborted with Ctrl+Z.
Still: Why is /dev/urandom only 5 MB/s and /dev/random only about 10B/s!?

Last edited by xl; 05-09-2013 at 09:56 AM.
 
Old 05-10-2013, 01:11 AM   #784
Beryllos
Member
 
Registered: Apr 2013
Location: Massachusetts
Distribution: Debian
Posts: 148

Rep: Reputation: 59
Quote:
Originally Posted by xl View Post
Still: Why is /dev/urandom only 5 MB/s and /dev/random only about 10B/s!?
The random number generator harvests "entropy" (randomness) from environmental sources like the keyboard and mouse, and maintains an "entropy pool." (I suspect the randomness is in the timing of keyboard, mouse, and perhaps other activity, not the content.) The entropy pool can be used as a source of random numbers, but because of the limited data rate of the keyboard/mouse/environment sources, the entropy pool can be emptied rather quickly.

/dev/random will provide random numbers only as long as there is sufficient entropy in the pool. When the pool is depleted, /dev/random will pause until more entropy is gathered from the environment.

On the other hand, /dev/urandom uses a pseudo-random number generator algorithm to provide any amount of output at a high rate limited mainly by your CPU speed.

Try "man random" (or "man urandom") to see the official help files.

P.S. I get 90-100MB/s for dd transfer from /dev/zero, 4.5MB/s from /dev/urandom using the default blocksize of 512 bytes, and 8.1MB/s from /dev/urandom with bs=64k or bs=1M.

Last edited by Beryllos; 05-10-2013 at 01:21 AM. Reason: added my typical speeds for comparison
 
Old 05-10-2013, 04:38 AM   #785
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,591

Original Poster
Rep: Reputation: 162Reputation: 162
Post Random and Urandom

Quote:
Originally Posted by xl View Post
Alright, seems it drops again to about 100 MB/s at the end – at least on a 2TB drive.
I'm still looking for a way to create several random files in a loop. Something like:
Code:
while true; do sudo dd if=/dev/urandom of=/media/DRIVE/file.$RANDOM bs=4k count=$RANDOM; done
This will create files from 2-120 MB but can only be aborted with Ctrl+Z.
Still: Why is /dev/urandom only 5 MB/s and /dev/random only about 10B/s!?
/dev/random and /dev/urandom need randomness (entropy). The system only stores enough randomness to create a certain amount of random data.

/dev/random is high-quality entropy. It will block after the entropy pool is exhausted. /dev/urandom will deliver random data that exactly repeats itself if you ask for more than the entropy pool can supply in one shot.

To get an idea, try:

dd if=/dev/random | hexdump -C

and
dd if=/dev/urandom | hexdump -C

and watch how the data output behaves.

But, for your application, /dev/urandom is permissible without increased security risk. If you ever encrypt a partition, before you do anything else, fill it with random data.

Otherwise the encrypted data is vulnerable to crypto attacks using the encryption algorithm itself.

If you do it the proper way, it's impossible to tell what algorithm was used to make the volume.

Oh, and lest I forget, beware of disk wipe methods that save time. Try this if you will:

Take a drive and wipe it with the manufacturer's zero-out, or low-level format program. Time the process with your own clock, not the program-display time

Then, boot with a live CD and search:

dd if=/dev/sda | strings -n 8 | less

nd check for data. Then wipe the drive with the first utility, again. Compare the total time to the first wipe. Why is that?

Last edited by AwesomeMachine; 05-10-2013 at 04:42 AM.
 
Old 05-10-2013, 09:15 AM   #786
LinuxSuporte
LQ Newbie
 
Registered: May 2013
Distribution: Ubuntu
Posts: 2

Rep: Reputation: Disabled
Great.
 
Old 05-10-2013, 03:57 PM   #787
xl
Member
 
Registered: Feb 2013
Location: Germany
Distribution: Wheezy
Posts: 46

Rep: Reputation: 7
Thanks, Beryllos & AM.

Quote:
Originally Posted by AwesomeMachine View Post
Take a drive and wipe it with the manufacturer's zero-out, or low-level format program. Time the process with your own clock, not the program-display time
Haven't come around to that. Basically I only want to zero the drive for a RMA.
Random files I needed to trigger bad sectors. Or will zero files do that too?!

So is it faster the second time? Does it only zero out the non-zero content? I thought zerofree would be able to do that ...

And why is a Win7 disk tool as a random file generator as fast as 17 MB/s (over USB!)?
Could timer_entropyd help in Linux?

- - - - - - - - - - - - - - - - - - - - - - - -
EDIT:
Code:
sudo sh -c 'cryptsetup -d /dev/urandom -c aes-xts-plain create delete /dev/SDX  && shred -vzn 0 /dev/mapper/delete && sync && sleep 4& cryptsetup remove delete'
Seems fast at first glance ...
You need cryptsetup and modprobe dm-crypt for that. (Source)

Last edited by xl; 05-10-2013 at 04:18 PM.
 
Old 05-10-2013, 07:07 PM   #788
odiseo77
Member
 
Registered: Dec 2004
Location: London, UK
Distribution: Debian Sid, OpenSUSE 13.1
Posts: 968

Rep: Reputation: 282Reputation: 282Reputation: 282
Hi,

I have used this guide hundreds of times when I've had to resort to dd, so I thought I'd drop by to say thank you! Great resource.

Regards.
 
Old 05-11-2013, 12:12 AM   #789
Beryllos
Member
 
Registered: Apr 2013
Location: Massachusetts
Distribution: Debian
Posts: 148

Rep: Reputation: 59
Quote:
Originally Posted by AwesomeMachine View Post
/dev/urandom will deliver random data that exactly repeats itself if you ask for more than the entropy pool can supply in one shot.

To get an idea, try:

dd if=/dev/random | hexdump -C
and
dd if=/dev/urandom | hexdump -C

and watch how the data output behaves.
I tried that, and do not see anything suspicious. What do you see?
 
Old 05-17-2013, 12:12 PM   #790
xl
Member
 
Registered: Feb 2013
Location: Germany
Distribution: Wheezy
Posts: 46

Rep: Reputation: 7
Beryllos, I think this was more for me. But I can see the difference:
Code:
~$ while true; do if dd if=/dev/random | hexdump -C | grep -q ""; then date "+%F %T"; fi; sleep 2; done
2013-05-17 18:50:33
2013-05-17 18:56:12
2013-05-17 19:03:25
Source

I guess it also depends also on my activity with the computer.
 
Old 05-17-2013, 02:55 PM   #791
Beryllos
Member
 
Registered: Apr 2013
Location: Massachusetts
Distribution: Debian
Posts: 148

Rep: Reputation: 59
Quote:
Originally Posted by xl View Post
Beryllos, I think this was more for me. But I can see the difference...
Okay, I should have mentioned that I put in the "bs" and "count" parameters, and I was examining whether "/dev/urandom will deliver random data that exactly repeats itself if you ask for more than the entropy pool can supply in one shot." I did not see any repetition.

Here's an example of what I did:
Code:
dd if=/dev/urandom bs=256M count=1 | hexdump -v -e '1/8 "%16.16x"' -e '"\n"' | sort | uniq -dc | sort -n | wc -l
1+0 records in
1+0 records out
268435456 bytes (268 MB) copied, 200.523 s, 1.3 MB/s
0
This gets a 256MB block of data from /dev/urandom, divides it into 8-byte words, and looks for duplicates. The 0 (zero) at the end indicates that there were no duplicates within the 32M consecutive 8-byte words which were sampled. This is not the most thorough test one can envision, but it tells us at least that /dev/urandom is not very repetitive.

I would like to explore the cryptographic strength of /dev/urandom someday. Probably the best way to do that is to look at the source code and see whether the programmers selected an established, proven, cryptographic pseudo-random number generator. Cryptographers know how to do it right.
 
1 members found this post helpful.
Old 06-03-2013, 06:01 PM   #792
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,591

Original Poster
Rep: Reputation: 162Reputation: 162
Post Randomness

Quote:
Originally Posted by Beryllos View Post
I tried that, and do not see anything suspicious. What do you see?
It's more the speed and rhythm that should be observed. /dev/urandom does not repeat very often. You can't see it repeat. It repeats maybe every 1024 bytes. But if the entire drive is overwritten, it doesn't matter.

Overwriting with random data is really for encrypted volumes or files. If you want to wipe a drive you can use zeroes.

If you wipe a drive twice, the second pass takes much longer. That tells you the first pass doesn't wipe everything.

/dev/random is for use in long-term keys that can't easily be changed, and that someone might have years of time to crack.

If I were to recommend the best way to secure against cryptographic exploits, it would be to hash the salt value for 30 seconds.

Every passphrase attempted would then require not 0.0001 seconds but 30.0 seconds, raising the time required by 6 orders of magnitude, and triple that.

If it took 20 days normally to crack a passphrase, hashing the salt value for thirty seconds would effectively increase that to 6.0 million days.

But the logical extrapolation of that method is to make the data unusable, which is 100% secure.

The federal government uses idle cpu cycles on 110,000 clustered PCs to crack all but the most meticulously crafted encryption.
 
Old 06-12-2013, 02:18 AM   #793
exvor
Senior Member
 
Registered: Jul 2004
Location: Phoenix, Arizona
Distribution: LFS-Version SVN-20091202, Arch 2009.08
Posts: 1,485

Rep: Reputation: 66
Quote:
Originally Posted by xl View Post
Thanks, Beryllos & AM.


Haven't come around to that. Basically I only want to zero the drive for a RMA.
Random files I needed to trigger bad sectors. Or will zero files do that too?!

So is it faster the second time? Does it only zero out the non-zero content? I thought zerofree would be able to do that ...

And why is a Win7 disk tool as a random file generator as fast as 17 MB/s (over USB!)?
Could timer_entropyd help in Linux?

- - - - - - - - - - - - - - - - - - - - - - - -
EDIT:
Code:
sudo sh -c 'cryptsetup -d /dev/urandom -c aes-xts-plain create delete /dev/SDX  && shred -vzn 0 /dev/mapper/delete && sync && sleep 4& cryptsetup remove delete'
Seems fast at first glance ...
You need cryptsetup and modprobe dm-crypt for that. (Source)


Best way to write random data to a drive quickly would be to create a luks container then write zeros to the container. When you do this it will write random data to the drive as it translates it down to the real drive from the container. It seams to work quicker then writing directly from urandom and its a pretty good way to be sure the data is scrambled.

Also on the note of security if you have a SSD writing zeros to the drive sometimes will not clear the drive of data. The firmware for the drive will see that there is no data being sent and will cache it in a way that will preserve the data. It does this for drive leveling. With flash media and non spinning disks the best practice is to physically destroy the device.

Last edited by exvor; 06-12-2013 at 02:23 AM.
 
Old 08-07-2013, 01:31 AM   #794
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,591

Original Poster
Rep: Reputation: 162Reputation: 162
Testing at the Center for Magnetic Storage Systems found that Solid-State drives cannot be securely wiped by any known means. They are disposable items. Exvor is correct that it is due to the wear-leveling-management controller in the SSD drive electronics. The same holds true for all solid-state flash-type storage devices. Thankfully, practically no one sells used USB drives.

We're not talking tiny bits of data left on the drives, like one sector here and there. We're talking hundreds of megabytes that will survive even repeated wipes by the most sophisticated drive sterilization software. This means that SSD storage cannot be used for any type of forensic duplication, so the entire computer-forensics industry is still very dependent on mechanical drives, if forensic analysis is to have any legitimacy at all.

It also means that no means exist to securely wipe SSDs for any purpose, not the least of which would be to avoid possible political persecution at the hands of tyrannical regimes.
 
Old 08-12-2013, 05:06 AM   #795
AwesomeMachine
Senior Member
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian jessie/sid; OpenSuSE; Fedora
Posts: 1,591

Original Poster
Rep: Reputation: 162Reputation: 162
Most people probably hot-link to this discussion thread off a Google.com hit, so you don't see that the OP has reached over a million views. Sorry, but I am not able to update the OP, because the edit button disappears after a certain time duration. But it's pretty good the way it is. I have no plans to fork it. It wouldn't be the same if it was anywhere but linuxquestions.org. I'm pleased if the OP has been of use to others. That's what was intended. I hope many more people will benefit. Knowledge is power! Truth will set you free. I give the power to the people.

But I take a lot of heat for it! There are at least a few people who are not too happy with me for putting together the OP, simply because it puts the power in the hands of the common folk. My attitude is: if someone is talented, that person will try to be of some benefit to others, and he won't worry about it. It's those who lack motivation and talent who must keep knowledge hidden, because if everyone knows, they'll do for themselves what others would charge money to do for them.

I learned a great lesson from writing the OP: if you ignore obstacles they become stepping stones toward new heights.
 
  


Reply

Tags
backup, best, clonezilla, cloning, command, data, dd, disk, drive, duplicate, erase, explanation, formatting, ghost, hard, image, iso, memory, ping, popular, recover, recovery, rescue, search, security, stick, upgrade, usb, wipe


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Learn The DD command AwesomeMachine Linux - Newbie 17 08-17-2006 04:22 AM
The best way to learn? iz3r Programming 7 02-06-2005 11:00 PM
Best way to learn Linux from the command line patpawlowski Linux - General 2 03-01-2004 03:37 PM
I want to learn C. KptnKrill Programming 14 12-18-2003 01:03 PM
Best way to learn.... InEeDhElPlInUx Linux - Newbie 5 10-11-2003 01:02 AM


All times are GMT -5. The time now is 07:43 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration