how to copy binary files using Unix API's

rohanak · 05-06-2008, 11:16 AM

Hello everyone,
As a part of a data backup project, I need to use Unix API's for performing file copy from source to destination.I have used the API's open(),read(),write() and close() for performing this.It works fine for regular files but not for binary files(like audio or video files).The newly created binary file gets partially corrupted(I got to know that the file is corrupted when VLC media player said : "the AVI file is broken".How should I use these API's for copying binary files?Also I dont want to copy the files byte by byte for efficiency reasons.I am using a buffer of size 1MB.

Thank You,
Rohan

osor · 05-06-2008, 01:54 PM

Unfortunately, there is no one magical system call that copies a file for you. You should be able to do it correctly using read() and write(). If you post the code, maybe someone will find the mistake. If you want a slight performance increase, you can try using mmap() to do the copy as well.

rohanak · 05-07-2008, 04:16 AM

This is the code I have written for copying files from source to destination:

Code:

fps=open(source,O_RDONLY);
		//printf("dest = %s\n\n",dest);
		fpd=open(dest,O_WRONLY|O_CREAT,0777);
		if(fps==-1)
		{
			printf("source = %s\n",source);
			printf("error : fps\n");
			break;
		}
		if(fpd==-1)
			printf("error : fpd\n");
		fstat(fps,&buf);
		bzero(buffer,BSIZE);
		filesize=buf.st_size;
		while(filesize)
		{
			if(filesize>BSIZE-1)
			{
				filesize=filesize-BSIZE-1;
				read(fps,buffer,BSIZE-1);
				write(fpd,buffer,BSIZE-1);
				bzero(buffer,BSIZE);
			}
			else
			{
				read(fps,buffer,filesize);
				write(fpd,buffer,filesize);
				filesize=0;
				bzero(buffer,BSIZE);
			}
		}

This works fine for text files but problems for binary files.
Can anyone suggest changes??

osor · 05-07-2008, 04:54 PM

Quote:

Originally Posted by rohanak

This works fine for text files but problems for binary files.
Can anyone suggest changes??

You’re making it too complicated. You don’t need to check the file size after each read. The reason I think this does not work on “binary files” is your false assumption that the number of bytes read is exactly the amount requested unless the file has fewer bytes left. In reality, read() may return with any number of bytes between 1 and the number requested for any request (and it is your job to check this number). Here is a slightly simpler version which should work on all types of files:

Code:

	ssize_t bytes;
	int fps, fpd;
	char buffer[BSIZE];

	if((fps = open(source, O_RDONLY)) == -1) {
		perror("open(source)");
		exit(EXIT_FAILURE);
	}
	if((fpd = open(dest, O_WRONLY | O_CREAT | O_TRUNC, 0644)) == -1) {
		perror("open(dest)");
		exit(EXIT_FAILURE);
	}

	while((bytes = read(fps, buffer, BSIZE)) > 0)
		write(fpd, buffer, bytes);

	close(fpd);
	close(fps);

Of course you probably want the mode of the destination file to match that of the source file.

As I said before, you can use mmap() as well (you need to truncate the destination file and then force it to be the same size as the source by writing a single byte to the correct offset).

Code:

	int fps, fpd;
	struct stat statbuf;
	void *bufs, *bufd;

	if((fps = open(source, O_RDONLY)) == -1) {
		perror("open(source)");
		exit(EXIT_FAILURE);
	}
	if((fpd = open(dest, O_RDWR | O_CREAT | O_TRUNC, 0644)) == -1) {
		perror("open(dest)");
		exit(EXIT_FAILURE);
	}
	if(fstat(fps, &statbuf) == -1) {
		perror("fstat");
		exit(EXIT_FAILURE);
	}
	if(pwrite(fpd, "", 1, statbuf.st_size - 1) != 1) {
		perror("pwrite");
		exit(EXIT_FAILURE);
	}
	if((bufs = mmap(0, statbuf.st_size, PROT_READ, MAP_SHARED, fps, 0)) == MAP_FAILED) {
		perror("mmap(source)");
		exit(EXIT_FAILURE);
	}
	if((bufd = mmap(0, statbuf.st_size, PROT_WRITE, MAP_SHARED, fpd, 0)) == MAP_FAILED) {
		perror("mmap(dest)");
		exit(EXIT_FAILURE);
	}

	memcpy (bufd, bufs, statbuf.st_size);

	munmap(bufd, statbuf.st_size);
	munmap(bufs, statbuf.st_size);

	close(fpd);
	close(fps);

rohanak · 05-08-2008, 05:59 AM

You have made use of mmap and memcpy for file copy by passing statbuf.st_size as a parameter.Will this technique work for very large files(order of GB's) due to insufficient memory?

osor · 05-09-2008, 04:26 PM

Quote:

Originally Posted by rohanak

Will this technique work for very large files(order of GB's) due to insufficient memory?

It will work as long as your machine’s address space is large enough. I have an x86-64 machine, on which it is noticeably faster:

Code:

$ du -h rand
4.1G    rand
$ time ./bufcopy rand rand2

real    1m43.420s
user    0m0.480s
sys     0m32.020s
$ time ./mapcopy rand rand3

real    1m23.766s
user    0m4.710s
sys     0m9.730s

Of course, YMMV. On the machine on which the above was run, I have 2GB of ram and 1GB of swap, so clearly a 4GB file is not completely mapped into memory at once. If you have a 32-bit machine, you cannot usually even open files greater than 2GB in size without open64() or O_LARGEFILE. If you use various other large file support macros (e.g., _LARGEFILE64_SOURCE or _FILE_OFFSET_BITS) you should have access to mmap2() or mmap64() on 32-bit machines. Remember to use the correct typedefs instead of the convenient ones (e.g., the last argument to mmap() is of type off_t which is not necessarily the same as int).