LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 08-12-2009, 04:35 AM   #1
hpsmouse
LQ Newbie
 
Registered: Aug 2009
Location: China
Distribution: ubuntu
Posts: 11

Rep: Reputation: 2
How to remove some blocks from a sparse file


The ext2/ext3 filesystem automatically allocate blocks when you write a sparse file, but when I no longer want some blocks of them, I found no ways to do it. It feels like using malloc() without free().
Is it possible to "free" some blocks of a sparse file? If it is, how?
Don't tell me to copy it to a new file. It's too boring and needs a lot of disk space.
 
Old 08-12-2009, 07:57 AM   #2
neonsignal
Senior Member
 
Registered: Jan 2005
Location: Melbourne, Australia
Distribution: Debian Wheezy (Fluxbox WM)
Posts: 1,363
Blog Entries: 52

Rep: Reputation: 353Reputation: 353Reputation: 353Reputation: 353
Good question, I can imagine situations where it would be useful.

I don't think it is possible in ext2/ext3. Because the suspended allocation is an optimization rather than part of the posix filesystem 'model' that the api describes, it doesn't necessarily have to be accessible, and any posix file system that provides access must do so through flags to the system calls or non-standard extensions.

In the XFS they provide access to freeing sections through xfsctl, using the XFS_IOC_RESVSP and XFS_IOC_UNRESVSP flags.

And in some Unix systems eg SVr4 you could use 'fcntl' with the FREESP flag.

This doesn't leave you with good options. Copying the file is an expensive operation if there is a lot of data (and as you say, boring).

You could implement your own 'block' layer on top of an ext file (with a block index such that when a block becomes empty, it can be copy-swapped with the last block in the file so that 'truncate' can be used to deallocate it. But this would be like writing your own "malloc/free" on top of the file system, a lot of extra work to gain the feature you want.

Last edited by neonsignal; 08-12-2009 at 08:10 AM.
 
Old 08-12-2009, 11:02 PM   #3
hpsmouse
LQ Newbie
 
Registered: Aug 2009
Location: China
Distribution: ubuntu
Posts: 11

Original Poster
Rep: Reputation: 2
Thank you, neonsignal, although I didn't find a 'fcntl' with FREESP support on my system.

Your advice of creating my own 'block layer' seems interesting. However, a problem still exists:
After swapping the blocks, the content of the file is changed. So all other data related to the swapped tail block must be modified. That's terrible...

Anyway I won't really try to implement it. Just for discussing.
 
Old 08-12-2009, 11:30 PM   #4
neonsignal
Senior Member
 
Registered: Jan 2005
Location: Melbourne, Australia
Distribution: Debian Wheezy (Fluxbox WM)
Posts: 1,363
Blog Entries: 52

Rep: Reputation: 353Reputation: 353Reputation: 353Reputation: 353
When I was talking about a block index over the top, everything has to go through that index: read, write, seek, etc. So swapping blocks underneath just means swapping the index pointers to match. This is transparent to the program, as long as it uses your interface and does not directly access the file.

But you end up reimplementing the whole interface over the top, just so you can gain one additional feature.
 
Old 08-13-2009, 07:47 AM   #5
hpsmouse
LQ Newbie
 
Registered: Aug 2009
Location: China
Distribution: ubuntu
Posts: 11

Original Poster
Rep: Reputation: 2
Oh, sorry, I misunderstood it.

Then that seems to be a huge project.
You are right, a lot of extra work. That's not worthy.

It can't be hard for the filesystem to implement such a useful feature. Since it's already working in that way, all needed is to provide an interface. But nobody does...
 
Old 08-13-2009, 07:45 PM   #6
wje_lq
Member
 
Registered: Sep 2007
Location: Mariposa
Distribution: Debian lenny, Slackware 12
Posts: 809

Rep: Reputation: 178Reputation: 178
Quote:
Originally Posted by hpsmouse View Post
It can't be hard for the filesystem to implement such a useful feature. Since it's already working in that way, all needed is to provide an interface. But nobody does...
Oh, good! You're volunteering, then?
 
Old 08-14-2009, 09:46 AM   #7
hpsmouse
LQ Newbie
 
Registered: Aug 2009
Location: China
Distribution: ubuntu
Posts: 11

Original Poster
Rep: Reputation: 2
Talking

Quote:
Originally Posted by wje_lq View Post
Oh, good! You're volunteering, then?
Don't think so highly of me. I am far far away from that level. I just expressed some of my feeling...
 
Old 08-14-2009, 07:51 PM   #8
neonsignal
Senior Member
 
Registered: Jan 2005
Location: Melbourne, Australia
Distribution: Debian Wheezy (Fluxbox WM)
Posts: 1,363
Blog Entries: 52

Rep: Reputation: 353Reputation: 353Reputation: 353Reputation: 353
I suspect WJE's point is that it is not as easy as it might seem. It might be reasonably trivial in a file system that only supports a single thread and has no performance constraints. But in a complex multi-user system you have to consider a lot of issues like race conditions, journalling, fragmentation, and so on.

Your best chance of getting the feature is to make a request to those developing new filesystems so that they can gauge how much demand there is for this feature. There was work on a FA_DEALLOCATE flag in ext4, but I don't know the status of this (I haven't used ext4 yet). It is unlikely to be added to older file systems such as ext2, because the possible consequence of breaking existing applications far exceeds any advantage.
 
Old 08-14-2009, 08:11 PM   #9
sundialsvcs
Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 5,401

Rep: Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119Reputation: 1119
I can buy a 1.5TB hard-drive for about $200 at a local department store, so...
 
Old 08-16-2009, 02:25 AM   #10
hpsmouse
LQ Newbie
 
Registered: Aug 2009
Location: China
Distribution: ubuntu
Posts: 11

Original Poster
Rep: Reputation: 2
I had to admit I sometimes do make mistakes, thinking things too simple. At that time I was thinking that NTFS had implemented a similar feature for a long time, so it couldn't be so challenging. Now it seems too optimistic.

I upgraded my system last month and I have been using ext4 for a few weeks. But I have no idea of the FA_DEALLOCATE flag. At many sites they say it's used with the 'fallocate()' function. However, the document said there's only one flag to use with it: 'FALLOC_FL_KEEP_SIZE'. I even searched the headers of my kernel(2.6.28). Still, nothing was found. I doubt whether it's still in their plan.

Thanks for all the ideas above!
 
Old 08-16-2009, 08:32 AM   #11
Sergei Steshenko
Senior Member
 
Registered: May 2005
Posts: 4,481

Rep: Reputation: 453Reputation: 453Reputation: 453Reputation: 453Reputation: 453
Quote:
Originally Posted by hpsmouse View Post
Oh, sorry, I misunderstood it.

Then that seems to be a huge project.
You are right, a lot of extra work. That's not worthy.

It can't be hard for the filesystem to implement such a useful feature. Since it's already working in that way, all needed is to provide an interface. But nobody does...

Have a look at FUSE: http://fuse.sourceforge.net/ .
 
  


Reply

Tags
filesystem


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
sort lines inside blocks in a file llattan Linux - Newbie 9 04-05-2009 01:19 PM
how to remove the bad blocks in unix lathasekhar Linux - Newbie 1 11-18-2008 11:14 AM
regarding bad blocks in extended file system rajesh_b Programming 1 06-13-2006 12:38 PM
Remove automatic boldface for quoted blocks? Simon Bridge LQ Suggestions & Feedback 6 06-06-2005 02:07 AM
Breaking up a file into blocks otnaicus Programming 9 05-19-2004 11:13 PM


All times are GMT -5. The time now is 11:50 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration