LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 05-12-2007, 03:28 PM   #1
Agresvig
LQ Newbie
 
Registered: May 2007
Posts: 6

Rep: Reputation: 0
Need help with 'dd'and 'mmap()'


Hi

I am working on a project that involves creating a file recovery program in C++.
I need some help with the Unix commands to get started.

First off, here is what I want to do, in short:
1. Create an image file of a FAT32 thumbdrive.
2. In the C++ program, map chunks of this image file into memory, and process them chunk by chunk.


Now, for step 1, I am trying to use 'dd' to create an image of the thumbdrive. I have tried:
Code:
dd if=/media/RALLY of=tmp/tmpimg.img
where 'RALLY' is the name of the thumbdrive. This does not work, with the error "dd: reading '/media/RALLY': is a directory"

Whats the correct way to create the image?

When I have that sorted out, I would love to get some pointers on how to split the image into chunks to feed mmap().


Thanks for all help.
 
Old 05-12-2007, 03:33 PM   #2
Daws
Member
 
Registered: May 2006
Location: UK
Distribution: Debian
Posts: 448

Rep: Reputation: 37
Quote:
dd if=/media/RALLY of=tmp/tmpimg.img
You want the device node. For a thumbdrive it's probably /dev/sda or /dev/sdb

Quote:
When I have that sorted out, I would love to get some pointers on how to split the image into chunks to feed mmap().
Hmm. Afraid I don't know what that is ... oh well
 
Old 05-12-2007, 04:26 PM   #3
osor
HCL Maintainer
 
Registered: Jan 2006
Distribution: (H)LFS, Gentoo
Posts: 2,450

Rep: Reputation: 75
Quote:
Originally Posted by Daws
You want the device node. For a thumbdrive it's probably /dev/sda or /dev/sdb
Yes. To figure out for sure, type “mount | grep /media/RALLY” or whatever. That gives you the block device mounted on /media/RALLY.

As for the second part, I’m not quite sure what you want. You can always use the “split” utility to split a file into multiple chunks. You can also mmap only part of a file at a time (vary the len and offset parameters).

For more info, see
man 1 split
man 2 mmap
 
Old 05-12-2007, 04:34 PM   #4
Agresvig
LQ Newbie
 
Registered: May 2007
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by Daws
You want the device node. For a thumbdrive it's probably /dev/sda or /dev/sdb
Perfect, /dev/sdb did the trick. Thanks a lot.

Now, I would love to get some suggestions on how to chop that image file into pieces.

About mmap():
Code:
void *mmap(void *start, size_t length, int prot, int flags,int fd, off_t offset);
If anyone has ideas on how to scan the image file using mmap(), I would love to hear them.

Thanks!
 
Old 05-12-2007, 04:45 PM   #5
Agresvig
LQ Newbie
 
Registered: May 2007
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by osor
Yes. To figure out for sure, type “mount | grep /media/RALLY” or whatever. That gives you the block device mounted on /media/RALLY.

As for the second part, I’m not quite sure what you want. You can always use the “split” utility to split a file into multiple chunks. You can also mmap only part of a file at a time (vary the len and offset parameters).

For more info, see
man 1 split
man 2 mmap

sorry didnt see your psot until I posted mine.
But thanks, that useful, will check it out.
 
Old 05-12-2007, 05:53 PM   #6
osor
HCL Maintainer
 
Registered: Jan 2006
Distribution: (H)LFS, Gentoo
Posts: 2,450

Rep: Reputation: 75
In case you didn’t understand, sample implementation (in C) would go something like this:
Code:
#define filename "tmp/tmpimg.img" /* filename */
#define chunksize 1048576 /* size of a chunk in bytes = 1 megabyte */

int fd, numchunks, i;
struct stat buf;
void *map = 0;

if ((fd = open(filename, O_RDONLY )) == -1) {
	perror("Couldn not open " filename);
	exit(EXIT_FAILURE);
}
if (fstat(fd, &buf) == -1) {
	perror("Could not stat " filename);
	exit(EXIT_FAILURE);
}

numchunks = buf.st_size / chunksize;
if (numchunks % chunksize) // if the size is not a multiple of chunksize,
	numchunks++;       // we need an extra chunk

for (i = 0; i < numchunks; i++) {
	if((map = mmap(NULL, chunksize, PROT_READ, MAP_PRIVATE, fd, i * chunksize)) == MAP_FAILED) {
		perror("map failed");
		exit(EXIT_FAILURE);
	}

	process_chunk(map, chunksize);

	if(munmap(map, chunksize) == -1) {
		perror("unmap failed");
		exit(EXIT_FAILURE);
	}
}
The process_chunk function shall process a chunk at a time.

P.S., there might be an off by one error somewhere
 
Old 05-14-2007, 02:45 PM   #7
Agresvig
LQ Newbie
 
Registered: May 2007
Posts: 6

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by osor
In case you didn’t understand, sample implementation (in C) would go something like this:

The process_chunk function shall process a chunk at a time.

P.S., there might be an off by one error somewhere

Thanks, that is of great help. Just one thing, could you explain your logic here a little more?:
Code:
numchunks = buf.st_size / chunksize;
if (numchunks % chunksize) // if the size is not a multiple of chunksize,
	numchunks++;       // we need an extra chunk
Thanks!
 
Old 05-14-2007, 08:19 PM   #8
osor
HCL Maintainer
 
Registered: Jan 2006
Distribution: (H)LFS, Gentoo
Posts: 2,450

Rep: Reputation: 75
Quote:
Originally Posted by Agresvig
Just one thing, could you explain your logic here a little more?
Code:
numchunks = buf.st_size / chunksize;
if (numchunks % chunksize) // if the size is not a multiple of chunksize,
	numchunks++;       // we need an extra chunk
First, I use fstat() to get file’s size. Then, I assign to the variable numchunks the integer quotient of the file’s size to the chunksize. Recall that integer division (at least in C — at least with ‘/’) always rounds “down” (toward zero). But that means if the file’s size is not a multiple of the chunksize, we will have a “fractional” chunk left over. Therefore, I test to see if there is a non-zero remainder when numchunks is divided by chunksize. If so, we know we have a fractional chunk left over, so we increment numchunks to account for this. The mmap() function will conveniently pad the excess memory with zeros (if this didn’t happen, we might instead request the exact remainder of the division of numchunks by chunksize and use it in one final call to mmap() after the loop).
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
mmap culin Programming 5 12-13-2006 10:51 PM
mmap function professional1983 Programming 1 06-14-2005 04:23 AM
mmap() error FarAway Programming 3 03-30-2005 07:38 AM
using mmap AngryLlama Programming 1 02-09-2005 08:53 AM
kiobuf vs. mmap j4r0d Programming 8 01-10-2005 03:18 AM


All times are GMT -5. The time now is 12:01 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration