LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices



Reply
 
Search this Thread
Old 07-24-2007, 11:48 AM   #1
dtimms
LQ Newbie
 
Registered: Sep 2004
Location: au.vic.melb
Distribution: Fedora 7 x3, FC6, FC3.
Posts: 27

Rep: Reputation: 15
how to cut an existing file into pieces without space


To: linux legend

I have a big mpg {dvb tuner recording} 100GB. I have 1GB of free space. I tried to split the file, but I then need another 100+GB of space, and a lot of time/disk activity to achieve it.

I'm hoping there is a tool to split an on disk {ext3} file into two parts at a specific offset. This could:
- adjust the first file entry to offset length
- create a new file entry
- set the new file start position {sector?} to be offset +1
- fix up the disk links

My search so far tried:
linux split file {split, cut {doh}, but it needs double the file size space}
linux split mpg file {tools like mpgtx ffmpeg, able to take a frame position, but also need the space}.

If there is such a thing, or script, perhaps it would also be possible to re-assemble two files into a single file {again without copying the bytes - just by adjusting the file pointers} ?

DaveT.
 
Old 07-24-2007, 08:53 PM   #2
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
how about dd ?
There's a long discussion on here about that tool.
 
Old 07-25-2007, 05:27 AM   #3
dtimms
LQ Newbie
 
Registered: Sep 2004
Location: au.vic.melb
Distribution: Fedora 7 x3, FC6, FC3.
Posts: 27

Original Poster
Rep: Reputation: 15
dd can't do it - can it ?

Quote:
Originally Posted by chrism01
how about dd ?
There's a long discussion on here about that tool.
I have used dd a fair bit. Can you give me a hint on parameters I could use to achieve my problem ?

I {if I had 50GB of free disk space} could use dd to copy either the first half or the second half of the file to a new file on disk. At this point the disk would be full, so I wouldn't be able to use dd to copy out the other half of the file. In fact I only have 1GB free.

If it was a fat file system, and the file was continuous, it would be possible to create a new fat table entry, point it to the position in the on disk file that I need to be the second file start position. Then adjust the first files length to truncate the fat's view of the file. Is this possible on ext3 ?
Are there tools available to do it ?

DaveT.
 
Old 07-25-2007, 07:41 AM   #4
gloriant
Member
 
Registered: Sep 2004
Distribution: Debian{Woody,Sarge,Etch}, UbuntuLTS6.06, SuSE{6.2,8.0}
Posts: 42

Rep: Reputation: 15
You'd use dd to get the last part of your file (which will use space), en then skim that end of the original file.
Code:
infile=original.mov
MB=$(( 1024*1024 ));
filetwo="${infile}.part2";
filesize=$(stat -c %s ${infile};);
szone=$(( 100*${MB} ));
sztwo=$(( ${filesize}-${szone} ));
dd if=${infile} of=${filetwo} bs=${sztwo} count=1 skip=${szone};
will get portion beyond 100MB of original.mov into original.mov.part2

Then the second portion (shrink) the original file, I don't know about an existing command for truncating a file, but suppose you have it, you could then execute
Code:
truncate ${infile} ${szone} && mv ${infile} ${infile}.part1
Now we *do* have the unistd-provided function truncate.... so if we write the truncate program ourself,....
it requires make and gcc (probably also libc6-dev)

Code:
cat >truncate.c <<EOFTRUNCATECODE
#include <unistd.h>
#include <sys/types.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>

void printUsage(void) {
  printf("usage:\n");
  printf("truncate <file> <size_in_byte>\n\n");
  printf("Will *reduce* size of <file> to <size_in_byte>.\n\n");
  printf("exitcodes:\n");
  printf("\t0\tSUCCESS (<file> has now size exactly <size_in_byte>.\n");
  printf("\t1\tinvalid arguments.\n");
  printf("\t2\t<file> unexisting or too small.\n");
  printf("\t3\tACCESS failure (cannot read or write <file>).\n");
}

int main(const int argc,
	const char* const* const argv) {
  int rv;

  if (3 != argc) {
    fprintf(stderr,"ERROR: Invalid number of arguments (%d).\n",argc);
    printUsage();
    rv = 1;
  } else {
    const char* const fn = argv[1];
    int base = 10;
    char* endpt;
    long int truncsize;
    
    truncsize=strtol(argv[2],&endpt,base);
    if ((NULL == endpt) || (0 != *endpt) || (0 < strlen(endpt))) {
      fprintf(stderr,"ERROR: (%s) invalid syntax.\n",argv[2]);
      printUsage();
      rv = 1;
    } else {
      FILE* f;

      f = fopen(fn,"r");
      if (!f) {
	fprintf(stderr,"Could not open %s.\n",fn);
	printUsage();
	rv = 3;
      } else {
	long int size;

	fseek(f,0L,SEEK_END);
	size = ftell(f);
	fclose(f);
	if (size < truncsize) {
	  fprintf(stderr,"size of %s (%ld byte) is too small to truncate.\n",fn,size);
	  rv = 2;
	} else if (0 != truncate(fn,(off_t)truncsize)) {
	  int errval;

	  errval=errno;
	  fprintf(stderr,"error truncating %s %d [%s].\n",fn,errval,strerror(errval));
	  rv = 2;
	} else {
	  printf("file %s truncated successfully to %ld byte.\n",fn,truncsize);
	  rv = 0;
	}
      }
    }
  }
  return rv;
}
EOFTRUNCATECODE
make truncate;
 
Old 07-25-2007, 09:42 AM   #5
dtimms
LQ Newbie
 
Registered: Sep 2004
Location: au.vic.melb
Distribution: Fedora 7 x3, FC6, FC3.
Posts: 27

Original Poster
Rep: Reputation: 15
Thanks for the code for a c program implementing truncate.

Quote:
Originally Posted by gloriant
...
long int truncsize;

truncsize=strtol(argv[2],&endpt,base);
...
Do you know off hand the range of long int and l ?
Would it matter on whether I am on i386 or x86_64 distribution ?

Just wondering whether it will actually work on a file that is just shy of 100GB ?

In any case, thanks heaps, obviously you have put in a heap of work to develop the code, DaveT.
 
Old 07-25-2007, 10:07 AM   #6
dtimms
LQ Newbie
 
Registered: Sep 2004
Location: au.vic.melb
Distribution: Fedora 7 x3, FC6, FC3.
Posts: 27

Original Poster
Rep: Reputation: 15
Searching for the word truncate has also been helpful:
http://www.pixelbeat.org/scripts/truncate
is a short script using dd to trim a file, so it seems it can be done with a pair of dd commands after all as long as I have disk space exceeding the part that I want to split of the end of the file {I do not have that sort of space available}.

...<code>
>&2
exit 1
fi

size=$1
file="$2"

error=`dd bs=1 seek=$size if=/dev/null of="$file" 2>&1`
ret=$?
echo "$error" | grep -v "^0" >&2
exit $ret
</code>

Not being that familiar with shell scripts, what is the 2>&1 and >&2 achieve, and also the "^0" ?

If it was possible to manipulate the ext3 file system directly, would doing so safely require dismounting the file system ?

DaveT.
 
Old 11-27-2009, 08:45 AM   #7
gloriant
Member
 
Registered: Sep 2004
Distribution: Debian{Woody,Sarge,Etch}, UbuntuLTS6.06, SuSE{6.2,8.0}
Posts: 42

Rep: Reputation: 15
Quote:
Originally Posted by dtimms View Post
Searching for the word truncate has also been helpful:
http://www.pixelbeat.org/scripts/truncate

Code:
error=`dd bs=1 seek=$size if=/dev/null of="$file" 2>&1`
ret=$?
echo "$error" | grep -v "^0" >&2
exit $ret
Not being that familiar with shell scripts, what is the 2>&1 and >&2 achieve, and also the "^0" ?
Code:
>&2
will output the default output of the execution into the standard error output pipe of the process.

Code:
grep -v "^0"
will look in it's input for all lines that do NOT start with "0"

why ??? or how ???

Code:
process >destination
and
Code:
process >>destination
will put default output of prcess execution into file destination. the former (>) will create a file (or overwrite an existing one), the latter (>>) will create a non existing one, or append to an existing one.

Code:
&2
and
Code:
&1
as destination in aformentioned code, will direct the output concerned into standard output (for &1) and into standard ERROR output pipe (for &2).
so the indication
Code:
process 2>&1
will output the standard error output (2) into the default output (&1)


if you do
Code:
grep -v searchstring
will return all results NOT matching the searchstring (-v stands for revert the condition).

Code:
grep "^0"
this will output all lines BEGINNING (special significance of ^ character in regular expressions) with "0"
 
  


Reply

Tags
dd, mpg, split


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Adding space to an existing partition. Codeman36 Linux - Software 2 06-11-2007 06:45 PM
How to cut a big file into pieces of a certain size? kornerr Linux - General 9 12-31-2005 04:52 PM
Create new partition from existing free space distortedstar Linux - General 4 06-07-2005 10:47 PM
Mounting new HD space into existing mount points? GT_Onizuka Linux - Hardware 4 05-26-2004 06:15 AM
Reallocating space for existing install invictus Linux - General 6 09-08-2003 05:13 PM


All times are GMT -5. The time now is 05:34 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration