Download your favorite Linux distribution at LQ ISO.
Go Back > Forums > Linux Forums > Linux - Software
User Name
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.


  Search this Thread
Old 08-26-2009, 01:32 AM   #1
LQ Newbie
Registered: Aug 2009
Posts: 5

Rep: Reputation: 0
Concatinate files by just manipulating cluster chains

Is there any way for concatenating (particularly large video files) files without having to copying one after the other. But, manipulating the file entries in the directory and joining the second file's cluster chain to the first file's chain.

This facility allows us to fetch a large (movie) file from any offset and write into a chunk file and when all chunks are complete we can quickly join them together into one file.Or even when two consecutive chunks are fully fetched we can join them into one chunk.

Old 08-27-2009, 10:00 AM   #2
LQ 5k Club
Registered: May 2003
Location: London, UK
Distribution: Fedora38
Posts: 6,147

Rep: Reputation: 435Reputation: 435Reputation: 435Reputation: 435Reputation: 435
Isn't this pretty much how torrents work? They pre-allocate filespace on your filesystem, and then as each chunk is downloaded, it is inserted into the empty file at the correct offset until the file is complete.

So, maybe take a look at the sourcecode of any torrent up/downloading program, and you'll see how it's done.

You probably won't want to do directory-entry manipulations directly to the disk, because different disks may be using different filesystems, which will have very different methods for handling directory structures and it'll all get very complicated very quickly.
Old 08-28-2009, 12:34 AM   #3
LQ Newbie
Registered: Aug 2009
Posts: 5

Original Poster
Rep: Reputation: 0
The idea was not to pre-allocate space as movie files are around 6-8 GB size. Moreover user may or may not completely download and watch it.He may just see portions of it (streaming VOD contents & and ability to play from any chapter like in DVD, but without fetching complete file). Therefore, we would like to create chunk files during random access downloads and join them later.

Old 08-28-2009, 12:57 AM   #4
LQ Guru
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 682Reputation: 682Reputation: 682Reputation: 682Reputation: 682Reputation: 682
Linux supports many types of filesystems. Even if you could do this at the filesystem level for one, you won't be able for another.
Unix systems are also inode based. That means you might have deleted files which still occupy disk space as long as a process has it open. This is something to keep in mind in writing your app. Look at `lsof +L1' for examples.

One thing to look into is using sparse files.

Last edited by jschiwal; 08-28-2009 at 12:59 AM.
Old 09-02-2009, 01:01 AM   #5
LQ Newbie
Registered: Aug 2009
Posts: 5

Original Poster
Rep: Reputation: 0
Indeed Sparse Files meets my requirement. It does not allocate data blocks and waste disk space.


concatenate, files, join

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Manipulating IMG Files jimmy512 Linux - Newbie 2 05-24-2007 05:06 AM
manipulating files tableyou Linux - Newbie 8 11-08-2005 02:38 AM
Editing/manipulating text files winchester169 Linux - Software 3 08-18-2005 06:01 PM
bash script on manipulating files fiomba Linux - Software 8 10-30-2004 08:31 PM
Manipulating Files in vi Baldorg Linux - General 1 09-19-2003 07:25 PM > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 07:23 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration