LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 08-15-2020, 10:03 AM   #1
zero.one
LQ Newbie
 
Registered: Aug 2020
Posts: 5

Rep: Reputation: Disabled
Question How can I pipe od to dd


I am running:

od /dev/sda

on a uninitialized drive. This outputs all non-zero characters on the drive.

I would like to pipe it to dd or any other command that can do this and overwrite each non-zero character that is found as zero.

I'm willing to try any other command or script to do this, but I don't want to format the entire drive. What is available to achieve this?
 
Old 08-15-2020, 10:34 AM   #2
berndbausch
LQ Guru
 
Registered: Nov 2013
Location: Tokyo
Distribution: A few
Posts: 5,322

Rep: Reputation: 1602Reputation: 1602Reputation: 1602Reputation: 1602Reputation: 1602Reputation: 1602Reputation: 1602Reputation: 1602Reputation: 1602Reputation: 1602Reputation: 1602
Quote:
Originally Posted by zero.one View Post
I don't want to format the entire drive.
Why? You gain nothing by just writing non-zero characters.

If you absolutely must, I'd write a simple C program. Read a block, check if it contains non-zero bytes, and if yes write the block.
 
Old 08-15-2020, 10:39 AM   #3
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 19,096

Rep: Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325Reputation: 3325
xyproblem

Don't tell us what you think you want as a solution, tell us what the problem is. Or at least what you think the problem is.
 
Old 08-15-2020, 10:39 AM   #4
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,085

Rep: Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957
I don't know the size of the drive, but that will take a very long time, probably a full day.
If you want a faster way write it in c, perl, python, whatever handle a full block in one.
And again, why? What do you want to achieve?
 
Old 08-15-2020, 10:41 AM   #5
zero.one
LQ Newbie
 
Registered: Aug 2020
Posts: 5

Original Poster
Rep: Reputation: Disabled
I would write a c program but I don't have experience with system programming. It would have to access the drive from 0 to the end. Every byte. Is there an example on how to do this?
 
Old 08-15-2020, 10:49 AM   #6
michaelk
Moderator
 
Registered: Aug 2002
Posts: 20,406

Rep: Reputation: 3551Reputation: 3551Reputation: 3551Reputation: 3551Reputation: 3551Reputation: 3551Reputation: 3551Reputation: 3551Reputation: 3551Reputation: 3551Reputation: 3551
Is this a mechanical, Solid state or USB flash drive?

Again why do you want to zero the drive? Are you trying to protect personal data?

I don't see any real time saver then just running a stock utility to erase the drive which is much simpler.
 
Old 08-15-2020, 10:54 AM   #7
rnturn
Senior Member
 
Registered: Jan 2003
Location: Illinois (SW Chicago 'burbs)
Distribution: Currently: openSUSE, Raspbian, Slackware. Formerly: CentOS, MacOS, Red Hat. Other: Solaris, Tru64
Posts: 2,059

Rep: Reputation: 338Reputation: 338Reputation: 338Reputation: 338
Quote:
Originally Posted by zero.one View Post
I am running:

od /dev/sda

on a uninitialized drive. This outputs all non-zero characters on the drive.

I would like to pipe it to dd or any other command that can do this and overwrite each non-zero character that is found as zero.

I'm willing to try any other command or script to do this, but I don't want to format the entire drive. What is available to achieve this?
I hope /dev/sda is not your boot device. [heh heh]

What is the goal here? A data wipe of the drive? There are tools that do that. I'd guess that "cat /dev/zero target-device" would do nicely---using /dev/random would probably be a better choice. I wrote a Perl script many years ago to write random numbers (multiple times) to disks on systems we were decommissioning and sending off for scrap---to satisfy the company's internal auditors' requirement to completely obliterate data disks and be HIPAA-compliant before letting them out the door.

If you're not into scripting a solution, look in your distribution's repository for "wipe". It's supposed to do a thorough destruction of a disk Note: I haven't used it; I'm just assuming this from the package description. There may be other, similar tools out there.

If wiping the data is the end goal, simply formatting the drive would likely not meet your needs, anyway. (I'm guessing that you already know this, though.)

HTH...
 
Old 08-15-2020, 10:55 AM   #8
zero.one
LQ Newbie
 
Registered: Aug 2020
Posts: 5

Original Poster
Rep: Reputation: Disabled
I dump large amounts of data on 10+tb drive from custom software. Once in a while I need to search for a string other times a range of characters. So for example if I have a 2gb range of the letter a, then I would search that range for a's and set them to zeros. The drive is written to uninitialized because it mimics short term memory for an AI (machine learning). I run simulations that are scripted so I dont do much programming, though I have some c experience.
 
Old 08-15-2020, 11:00 AM   #9
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,085

Rep: Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957
https://stackoverflow.com/questions/...ogram-on-linux
 
Old 08-15-2020, 11:01 AM   #10
zero.one
LQ Newbie
 
Registered: Aug 2020
Posts: 5

Original Poster
Rep: Reputation: Disabled
If you guys want a little more detail (It's really irrelevant to solving this problem, but if you must know). An artificial neuron has different parts that make up its connections to a neural net. When these activities are stored on a hard drive, they are patterns. By changing certain patterns new layers are accessed. It's kind of like data compression, but it corresponds to resetting the matrix to a different state. I'm trying to automate utility functions for this.
 
Old 08-15-2020, 11:39 AM   #11
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,085

Rep: Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957
scanning a 10+tb storage will take a long time. Parsing it and writing back data will make it even slower.
You may try a
Code:
time dd if=/dev/sda of=/dev/null
to check how fast can you read the whole disk. Using a shell script to manipulate data will make it run forever.
Here is a sample code to read a block: https://www.unix.com/linux/152828-c-...hard-disk.html
 
Old 08-15-2020, 11:49 AM   #12
zero.one
LQ Newbie
 
Registered: Aug 2020
Posts: 5

Original Poster
Rep: Reputation: Disabled
No, no ... I think I was misunderstood. I appreciate anyone trying to help me ... Let me try to better explain.

So let's say in one session I would want to search for the character 'a'. This would be spread over an area that has many other characters, but let's say a 1 mb area. So let's say there are 900 'a' characters. Everyone that is found would be set to '0'. That's the end of it. Now the search might not be for a character. It can be a string and etc. I would make the script usable in other scenarios. All these things I can do with dd and other utilities, I have written and read bytes before. Even strings, but automating it, so I can simply have some utility functions to do this quickly would be better. The most important drawback is, that the drive does not have a partition. It's just a flat storage area. No filesystem etc. just raw data. So od would help, but it does all nonzero characters. However, I know it's possible to pipe it to grep then dd and etc. So I'm learning about this now. Another option might be sleuthkit.

Last edited by zero.one; 08-15-2020 at 11:52 AM.
 
Old 08-15-2020, 11:52 AM   #13
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 15,085

Rep: Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957Reputation: 4957
so you have some code already. Would be nice to show us to help you to continue....
Additionally if you have an algorithm to implement either you explain us or ????
 
Old 08-15-2020, 11:52 AM   #14
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,054
Blog Entries: 3

Rep: Reputation: 2521Reputation: 2521Reputation: 2521Reputation: 2521Reputation: 2521Reputation: 2521Reputation: 2521Reputation: 2521Reputation: 2521Reputation: 2521Reputation: 2521
If you are only dealing with characters, and they are 8-bit, then tr might be enough:

Code:
tr 'a' '0' < input.file.txt > output.file.txt
If they are Unicode or something then you could use a perl one-liner instead.
 
Old 08-15-2020, 01:28 PM   #15
Shadow_7
Senior Member
 
Registered: Feb 2003
Distribution: debian
Posts: 4,088
Blog Entries: 1

Rep: Reputation: 867Reputation: 867Reputation: 867Reputation: 867Reputation: 867Reputation: 867Reputation: 867
dd has input file and output file. The output file could be stdout. And piped through other things. Each layer makes things slower though. Probably of little concern relative to the speed of storage devices. Not sure how useful grep would be on raw data. Are you just wanting a count of instances of a character? Or need disc position data and other information? You could pipe through sed 's/a/\000/g' or something to change the data. But it would be dog slow reading/writing the whole disc like that. And kind of hard on the drive's lifespan.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] How to handle a broken pipe exception (SIGPIPE) in FIFO pipe? zyroot998 Programming 5 03-03-2011 08:10 PM
PING .... pipe 2 vs pipe 3 hanen03 Linux - Networking 1 04-10-2009 10:50 AM
pipe using no pipe soathana Programming 2 02-22-2003 04:33 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 10:39 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration