how to cut an existing file into pieces without space

dtimms · 07-24-2007, 10:48 AM

To: linux legend

I have a big mpg {dvb tuner recording} 100GB. I have 1GB of free space. I tried to split the file, but I then need another 100+GB of space, and a lot of time/disk activity to achieve it.

I'm hoping there is a tool to split an on disk {ext3} file into two parts at a specific offset. This could:
- adjust the first file entry to offset length
- create a new file entry
- set the new file start position {sector?} to be offset +1
- fix up the disk links

My search so far tried:
linux split file {split, cut {doh}, but it needs double the file size space}
linux split mpg file {tools like mpgtx ffmpeg, able to take a frame position, but also need the space}.

If there is such a thing, or script, perhaps it would also be possible to re-assemble two files into a single file {again without copying the bytes - just by adjusting the file pointers} ?

DaveT.

chrism01 · 07-24-2007, 07:53 PM

how about dd ?
There's a long discussion on here about that tool.

dtimms · 07-25-2007, 04:27 AM

Quote:

Originally Posted by chrism01

how about dd ?
There's a long discussion on here about that tool.

I have used dd a fair bit. Can you give me a hint on parameters I could use to achieve my problem ?

I {if I had 50GB of free disk space} could use dd to copy either the first half or the second half of the file to a new file on disk. At this point the disk would be full, so I wouldn't be able to use dd to copy out the other half of the file. In fact I only have 1GB free.

If it was a fat file system, and the file was continuous, it would be possible to create a new fat table entry, point it to the position in the on disk file that I need to be the second file start position. Then adjust the first files length to truncate the fat's view of the file. Is this possible on ext3 ?
Are there tools available to do it ?

DaveT.

gloriant · 07-25-2007, 06:41 AM

You'd use dd to get the last part of your file (which will use space), en then skim that end of the original file.

Code:

infile=original.mov
MB=$(( 1024*1024 ));
filetwo="${infile}.part2";
filesize=$(stat -c %s ${infile};);
szone=$(( 100*${MB} ));
sztwo=$(( ${filesize}-${szone} ));
dd if=${infile} of=${filetwo} bs=${sztwo} count=1 skip=${szone};

will get portion beyond 100MB of original.mov into original.mov.part2

Then the second portion (shrink) the original file, I don't know about an existing command for truncating a file, but suppose you have it, you could then execute

Code:

truncate ${infile} ${szone} && mv ${infile} ${infile}.part1

Now we *do* have the unistd-provided function truncate.... so if we write the truncate program ourself,....
it requires make and gcc (probably also libc6-dev)

Code:

cat >truncate.c <<EOFTRUNCATECODE
#include <unistd.h>
#include <sys/types.h>
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <string.h>

void printUsage(void) {
  printf("usage:\n");
  printf("truncate <file> <size_in_byte>\n\n");
  printf("Will *reduce* size of <file> to <size_in_byte>.\n\n");
  printf("exitcodes:\n");
  printf("\t0\tSUCCESS (<file> has now size exactly <size_in_byte>.\n");
  printf("\t1\tinvalid arguments.\n");
  printf("\t2\t<file> unexisting or too small.\n");
  printf("\t3\tACCESS failure (cannot read or write <file>).\n");
}

int main(const int argc,
	const char* const* const argv) {
  int rv;

  if (3 != argc) {
    fprintf(stderr,"ERROR: Invalid number of arguments (%d).\n",argc);
    printUsage();
    rv = 1;
  } else {
    const char* const fn = argv[1];
    int base = 10;
    char* endpt;
    long int truncsize;
    
    truncsize=strtol(argv[2],&endpt,base);
    if ((NULL == endpt) || (0 != *endpt) || (0 < strlen(endpt))) {
      fprintf(stderr,"ERROR: (%s) invalid syntax.\n",argv[2]);
      printUsage();
      rv = 1;
    } else {
      FILE* f;

      f = fopen(fn,"r");
      if (!f) {
	fprintf(stderr,"Could not open %s.\n",fn);
	printUsage();
	rv = 3;
      } else {
	long int size;

	fseek(f,0L,SEEK_END);
	size = ftell(f);
	fclose(f);
	if (size < truncsize) {
	  fprintf(stderr,"size of %s (%ld byte) is too small to truncate.\n",fn,size);
	  rv = 2;
	} else if (0 != truncate(fn,(off_t)truncsize)) {
	  int errval;

	  errval=errno;
	  fprintf(stderr,"error truncating %s %d [%s].\n",fn,errval,strerror(errval));
	  rv = 2;
	} else {
	  printf("file %s truncated successfully to %ld byte.\n",fn,truncsize);
	  rv = 0;
	}
      }
    }
  }
  return rv;
}
EOFTRUNCATECODE
make truncate;

dtimms · 07-25-2007, 08:42 AM

Thanks for the code for a c program implementing truncate.

Quote:

Originally Posted by gloriant

...
long int truncsize;

truncsize=strtol(argv[2],&endpt,base);
...

Do you know off hand the range of long int and l ?
Would it matter on whether I am on i386 or x86_64 distribution ?

Just wondering whether it will actually work on a file that is just shy of 100GB ?

In any case, thanks heaps, obviously you have put in a heap of work to develop the code, DaveT.

dtimms · 07-25-2007, 09:07 AM

Searching for the word truncate has also been helpful:
http://www.pixelbeat.org/scripts/truncate
is a short script using dd to trim a file, so it seems it can be done with a pair of dd commands after all

as long as I have disk space exceeding the part that I want to split of the end of the file {I do not have that sort of space available}.

...<code>
>&2
exit 1
fi

size=$1
file="$2"

error=`dd bs=1 seek=$size if=/dev/null of="$file" 2>&1`
ret=$?
echo "$error" | grep -v "^0" >&2
exit $ret
</code>

Not being that familiar with shell scripts, what is the 2>&1 and >&2 achieve, and also the "^0" ?

If it was possible to manipulate the ext3 file system directly, would doing so safely require dismounting the file system ?

DaveT.

gloriant · 11-27-2009, 07:45 AM

Quote:

Originally Posted by dtimms

Searching for the word truncate has also been helpful:
http://www.pixelbeat.org/scripts/truncate

Code:

error=`dd bs=1 seek=$size if=/dev/null of="$file" 2>&1`
ret=$?
echo "$error" | grep -v "^0" >&2
exit $ret

Not being that familiar with shell scripts, what is the 2>&1 and >&2 achieve, and also the "^0" ?

Code:

>&2

will output the default output of the execution into the standard error output pipe of the process.

Code:

grep -v "^0"

will look in it's input for all lines that do NOT start with "0"

why ??? or how ???

Code:

process >destination

and

Code:

process >>destination

will put default output of prcess execution into file destination. the former (>) will create a file (or overwrite an existing one), the latter (>>) will create a non existing one, or append to an existing one.

Code:

&2

and

Code:

&1

as destination in aformentioned code, will direct the output concerned into standard output (for &1) and into standard ERROR output pipe (for &2).
so the indication

Code:

process 2>&1

will output the standard error output (2) into the default output (&1)

if you do

Code:

grep -v searchstring

will return all results NOT matching the searchstring (-v stands for revert the condition).

Code:

grep "^0"

this will output all lines BEGINNING (special significance of ^ character in regular expressions) with "0"