LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-25-2017, 10:03 AM   #1
ddenial
Member
 
Registered: Dec 2016
Distribution: CentOS, Fedora, Ubuntu
Posts: 221

Rep: Reputation: 41
Is there a way to see disk space that is deleted but still exist in disk?


Hello All

I was wondering is there a way to see the disk space that has been deleted and whats considered as unallocated free space with non-zero value. I'm not taking about data recovery, but the free space from where data recovery tools recover data.

I don't know i'm making any sense, but let me illustrate it better.
Suppose
  1. I have 1 TB new disk.
  2. I fill it with files of 200 GB.
  3. I delete them all 200 GB.
  4. Then , I fill it again with new 50 GB files.

Now tools like df, du will show the space as 50 GB used and 950 GB unused.

But i know 200 GB of files are still out there, but are not shown to me. I can only recover them using data recovery tools.

so, I want to know is there any tool or trick that will show me in numeric size or in block/chart the free space NOT filled by zero.

I hope I'm clear.

Thanks
 
Old 02-25-2017, 12:43 PM   #2
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware & Android
Posts: 11,026

Rep: Reputation: 1251Reputation: 1251Reputation: 1251Reputation: 1251Reputation: 1251Reputation: 1251Reputation: 1251Reputation: 1251Reputation: 1251
What's your file system? It depends; easy on vfat, near impossible on some others.
 
Old 02-25-2017, 12:54 PM   #3
r3sistance
Senior Member
 
Registered: Mar 2004
Location: UK
Distribution: CentOS 6/7
Posts: 1,375

Rep: Reputation: 217Reputation: 217Reputation: 217
Why do you care about what space is and isn't allocated? It'll eventually get written over eventually in the disks life if you use it actively.

Zerofree maybe what you are after to zero fill any unused non-zero space, however even such a tool might not make the data unrecoverable, more so on HDDs.
 
Old 02-25-2017, 01:25 PM   #4
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 13,611

Rep: Reputation: 4364Reputation: 4364Reputation: 4364Reputation: 4364Reputation: 4364Reputation: 4364Reputation: 4364Reputation: 4364Reputation: 4364Reputation: 4364Reputation: 4364
Quote:
Originally Posted by ddenial View Post


so, I want to know is there any tool or trick that will show me in numeric size or in block/chart the free space NOT filled by zero.
If you don't use that disk actively: just run a tool like
Code:
dd if=/dev/<your disk> | <count number of non-zeros>
and you will get the space not filled by zero. You can count your files too and you can calculate what you need.
If this disk was in use, the counting of bytes will be useless, because the content of the disk will be changed during that process.

There is no process/tool/device which will take care on "touched" bits and bytes - at least I don't know about that. It will definitely slow down that device and will not give any benefits.
 
1 members found this post helpful.
Old 02-25-2017, 01:57 PM   #5
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,422
Blog Entries: 9

Rep: Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668
yes, most of the data is still there, but it's now a jumbled heap instead of an ordered list.

youw ill need data recovery tools to recover most of it, but it is possible - until it gets overwritten.
 
Old 02-25-2017, 03:25 PM   #6
ddenial
Member
 
Registered: Dec 2016
Distribution: CentOS, Fedora, Ubuntu
Posts: 221

Original Poster
Rep: Reputation: 41
Quote:
Originally Posted by business_kid View Post
What's your file system? It depends; easy on vfat, near impossible on some others.
I'm using ext4

Quote:
Originally Posted by r3sistance View Post
Why do you care about what space is and isn't allocated? It'll eventually get written over eventually in the disks life if you use it actively.

Zerofree maybe what you are after to zero fill any unused non-zero space, however even such a tool might not make the data unrecoverable, more so on HDDs.
Of course It'll eventually get the space that will be over-written, but its not the point, I'm not even worried about the data I lost, I'am just curious (or suspicious), the tools which claim they have replaced free space with zero and that your private data is gone forever for good (with all the military grade overwriting with fancy names and with n times overwriting), where's the proof?

I'am sure they have some good algorithm to clear data, but as a ordinary user, I want to know before and after unallocated data that was there and now there isn't. Just be so sure that blocks are actually filled by zero.

Its just a curiosity, if tools that claim they will replace some written but unused space with zero, why cant I?

Quote:
Originally Posted by pan64 View Post
If you don't use that disk actively: just run a tool like
Code:
dd if=/dev/<your disk> | <count number of non-zeros>
and you will get the space not filled by zero. You can count your files too and you can calculate what you need.
If this disk was in use, the counting of bytes will be useless, because the content of the disk will be changed during that process.

There is no process/tool/device which will take care on "touched" bits and bytes - at least I don't know about that. It will definitely slow down that device and will not give any benefits.
This is very interesting....I'll try it as soon as I get to my lab.

Quote:
Originally Posted by ondoho View Post
yes, most of the data is still there, but it's now a jumbled heap instead of an ordered list.

youw ill need data recovery tools to recover most of it, but it is possible - until it gets overwritten.
Yes I know. But I want to know where and how much space that jumbled heap is used.

Thanks
 
Old 02-25-2017, 03:36 PM   #7
r3sistance
Senior Member
 
Registered: Mar 2004
Location: UK
Distribution: CentOS 6/7
Posts: 1,375

Rep: Reputation: 217Reputation: 217Reputation: 217
Quote:
Originally Posted by ddenial View Post
Of course It'll eventually get the space that will be over-written, but its not the point, I'm not even worried about the data I lost, I'am just curious (or suspicious), the tools which claim they have replaced free space with zero and that your private data is gone forever for good (with all the military grade overwriting with fancy names and with n times overwriting), where's the proof?

I'am sure they have some good algorithm to clear data, but as a ordinary user, I want to know before and after unallocated data that was there and now there isn't. Just be so sure that blocks are actually filled by zero.

Its just a curiosity, if tools that claim they will replace some written but unused space with zero, why cant I?
Think about the way you are talking to the disk, the filing system only cares about if space is taken or not and you are speaking to the disk using utilities that are using fast, quick and easy methods of writing to the disk, leaving anything more advanced to the filing system itself (I.E. Journaling). You'd need a tool or method at a higher level to actually do the analysis itself, most likely via directly accessing and modifying the filing system.

Last edited by r3sistance; 02-25-2017 at 03:43 PM.
 
Old 02-25-2017, 03:53 PM   #8
ddenial
Member
 
Registered: Dec 2016
Distribution: CentOS, Fedora, Ubuntu
Posts: 221

Original Poster
Rep: Reputation: 41
Quote:
Originally Posted by r3sistance View Post
Think about the way you are talking to the disk, the filing system only cares about if space is taken or not and you are speaking to the disk using utilities that are using fast, quick and easy methods of writing to the disk, leaving anything more advanced to the filing system itself. You'd need a tool or method at a higher level to actually do the analysis itself, most likely via directly accessing and modifying the filing system.
OK trust me on this...I'am loosing my disk (its 5 yrs old) and I'am donating my disk to others. Its 1TB but only 900 GB are usable. I ran this command


Code:
   dd if=/dev/zero of=junk
   sync
   rm junk
KB: https://frippery.org/uml/

But then I ran extundelete from systemrescuecd (give me a big cheers), I recovered about 400GB of my private data full intact. Now explain this.
 
Old 02-25-2017, 05:22 PM   #9
ondoho
LQ Addict
 
Registered: Dec 2013
Posts: 13,422
Blog Entries: 9

Rep: Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668Reputation: 3668
^ how long did that comamnd run? many hours, i presume?

ALSO, maybe you have more than 1 partition?

HOW EXACTLY did you run extundelete?
 
Old 02-25-2017, 07:03 PM   #10
ddenial
Member
 
Registered: Dec 2016
Distribution: CentOS, Fedora, Ubuntu
Posts: 221

Original Poster
Rep: Reputation: 41
Quote:
Originally Posted by ondoho View Post
^ how long did that comamnd run? many hours, i presume?

ALSO, maybe you have more than 1 partition?

HOW EXACTLY did you run extundelete?
It took full night probably between 8-15 hours. I don't know, I was asleep. I don't remember at what time i slept, but when i woke up at morning, it was still running.

Last edited by ddenial; 02-25-2017 at 07:07 PM.
 
Old 02-25-2017, 07:06 PM   #11
ddenial
Member
 
Registered: Dec 2016
Distribution: CentOS, Fedora, Ubuntu
Posts: 221

Original Poster
Rep: Reputation: 41
Quote:
Originally Posted by ondoho View Post
HOW EXACTLY did you run extundelete?
Code:
# extundelete /dev/sda --restore-all
as i recall
 
Old 02-25-2017, 07:15 PM   #12
r3sistance
Senior Member
 
Registered: Mar 2004
Location: UK
Distribution: CentOS 6/7
Posts: 1,375

Rep: Reputation: 217Reputation: 217Reputation: 217
Quote:
Originally Posted by ddenial View Post
OK trust me on this...I'am loosing my disk (its 5 yrs old) and I'am donating my disk to others. Its 1TB but only 900 GB are usable. I ran this command
I was responding specifically to this part:

Quote:
Originally Posted by ddenial View Post
Its just a curiosity, if tools that claim they will replace some written but unused space with zero, why cant I?
So when I am saying, why do you care? For most users, zero filling data out is meaningless since most things deleted aren't cared about anymore. So just leave it there and write over it later. Thus when you deal with the filing system it is using this ultra efficient method. Also on systems/hardware that may have write limits, zero filling may also reduce the lifespan of such devices, so there is more than a few reasons for the behavior. I myself did in my first response give you the name of a utility that would do what you asked for, and you've been quoted a few other methods first such as using DD to create a zeroed file to cover the disk...
 
Old 02-25-2017, 07:34 PM   #13
JeremyBoden
Senior Member
 
Registered: Nov 2011
Location: London, UK
Distribution: Debian
Posts: 1,653

Rep: Reputation: 362Reputation: 362Reputation: 362Reputation: 362
Quote:
Originally Posted by ddenial View Post
OK trust me on this...I'am loosing my disk (its 5 yrs old) and I'am donating my disk to others. Its 1TB but only 900 GB are usable.
Strange. You think your disk is dying, so you make a gift of it to someone?

Actually, a disk sold as 1TByte will only be 931.5 actual GiBytes anyway. Of this, your filesystem will require some space and may reserve a certain amount of space for its own purposes.

I would get a SMART report of any bad blocks before you give it away.
 
Old 02-25-2017, 07:36 PM   #14
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: CentOS
Posts: 4,362

Rep: Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001Reputation: 2001
Quote:
Originally Posted by ddenial View Post
I'am just curious (or suspicious), the tools which claim they have replaced free space with zero and that your private data is gone forever for good (with all the military grade overwriting with fancy names and with n times overwriting), where's the proof?

I'am sure they have some good algorithm to clear data, but as a ordinary user, I want to know before and after unallocated data that was there and now there isn't. Just be so sure that blocks are actually filled by zero.
Before the file has been deleted, you can use "hdparm --fibmap" to see what sectors that file occupies on the disk. After deleting the file and using one of the "clearing" tools, you can use a tool like hexedit to look at those sectors and see what's there. Of course that isn't proof that parts of the file don't still exist elsewhere (in a journal area, for example), and it doesn't prove anything with an SSD or hybrid drive, where overwriting a block just means that a different physical block is mapped to the LBA, and the original block is just scheduled to be erased whenever the drive gets the proverbial "Round Tuit".
 
Old 02-26-2017, 02:00 AM   #15
AwesomeMachine
LQ Guru
 
Registered: Jan 2005
Location: USA and Italy
Distribution: Debian testing/sid; OpenSuSE; Fedora; Mint
Posts: 5,511

Rep: Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007Reputation: 1007
Sleuthkit, and its GUI, autopsy, will allow you to gather non-zero unallocated space and view it. OR, you can try:
Code:
dd if=/dev/sda | hexdump -C | grep [^00]
but that only works if the entire drive is zeroed.

I believe autopsy also allows to gather slack space. But if you want to clear a drive the best thing is BXDR. It's a bit tricky to use, but it doesn't rely on software to clear the drive. It triggers the drives internal SECURITY_ERASE_UNIT command, and the drive takes over from there, overwriting every sector with zeroes.

In the case of enhanced security erase, it overwrites the platters with random data. But be under no illusions! Someone can still remove the platters from the drive and use a quantum interference generator to determine the drive contents before it was erased!

If you want to overwrite the unallocated space, this always works:
Code:
dd if=/dev/zero of=/bigfile.file conv=sync
Bigfile.file will simply grow to consume every last bit of space that isn't allocated. Then, dd will stop. But you have to then delete bigfile.file
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Zero out deleted files to recover disk space? JoeyB Linux - Newbie 3 03-23-2016 08:54 PM
Disk full. Deleted files but no space available. atrejo Linux - General 3 05-03-2010 03:00 PM
[SOLVED] Deleted 25 million records from database, had no effect on disk space the182guy Linux - Newbie 5 10-03-2009 11:35 AM
3Gb of disk space lost! Disk space problem or mother board conflicts with HDD Mistreated Linux - Hardware 4 12-06-2004 04:58 PM
what files can be safely deleted to save disk space? rohan208 Linux - General 9 05-27-2004 09:06 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 03:38 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration