LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 08-26-2009, 01:32 AM   #1
mmreddy54
LQ Newbie
 
Registered: Aug 2009
Posts: 5

Rep: Reputation: 0
Concatinate files by just manipulating cluster chains


Hi,
Is there any way for concatenating (particularly large video files) files without having to copying one after the other. But, manipulating the file entries in the directory and joining the second file's cluster chain to the first file's chain.

This facility allows us to fetch a large (movie) file from any offset and write into a chunk file and when all chunks are complete we can quickly join them together into one file.Or even when two consecutive chunks are fully fetched we can join them into one chunk.

Thanks
 
Old 08-27-2009, 10:00 AM   #2
tredegar
LQ 5k Club
 
Registered: May 2003
Location: London, UK
Distribution: Debian "Jessie"
Posts: 6,017

Rep: Reputation: 369Reputation: 369Reputation: 369Reputation: 369
Isn't this pretty much how torrents work? They pre-allocate filespace on your filesystem, and then as each chunk is downloaded, it is inserted into the empty file at the correct offset until the file is complete.

So, maybe take a look at the sourcecode of any torrent up/downloading program, and you'll see how it's done.

You probably won't want to do directory-entry manipulations directly to the disk, because different disks may be using different filesystems, which will have very different methods for handling directory structures and it'll all get very complicated very quickly.
 
Old 08-28-2009, 12:34 AM   #3
mmreddy54
LQ Newbie
 
Registered: Aug 2009
Posts: 5

Original Poster
Rep: Reputation: 0
The idea was not to pre-allocate space as movie files are around 6-8 GB size. Moreover user may or may not completely download and watch it.He may just see portions of it (streaming VOD contents & and ability to play from any chapter like in DVD, but without fetching complete file). Therefore, we would like to create chunk files during random access downloads and join them later.

Thanks
 
Old 08-28-2009, 12:57 AM   #4
jschiwal
Guru
 
Registered: Aug 2001
Location: Fargo, ND
Distribution: SuSE AMD64
Posts: 15,733

Rep: Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654Reputation: 654
Linux supports many types of filesystems. Even if you could do this at the filesystem level for one, you won't be able for another.
Unix systems are also inode based. That means you might have deleted files which still occupy disk space as long as a process has it open. This is something to keep in mind in writing your app. Look at `lsof +L1' for examples.

One thing to look into is using sparse files.

Last edited by jschiwal; 08-28-2009 at 12:59 AM.
 
Old 09-02-2009, 01:01 AM   #5
mmreddy54
LQ Newbie
 
Registered: Aug 2009
Posts: 5

Original Poster
Rep: Reputation: 0
Indeed Sparse Files meets my requirement. It does not allocate data blocks and waste disk space.
 
  


Reply

Tags
concatenate, files, join


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Manipulating IMG Files jimmy512 Linux - Newbie 2 05-24-2007 05:06 AM
manipulating files tableyou Linux - Newbie 8 11-08-2005 02:38 AM
Editing/manipulating text files winchester169 Linux - Software 3 08-18-2005 06:01 PM
bash script on manipulating files fiomba Linux - Software 8 10-30-2004 08:31 PM
Manipulating Files in vi Baldorg Linux - General 1 09-19-2003 07:25 PM


All times are GMT -5. The time now is 04:28 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration