LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 12-02-2007, 02:15 PM   #16
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,928

Original Poster
Rep: Reputation: 612Reputation: 612Reputation: 612Reputation: 612Reputation: 612Reputation: 612

Hello Osor -I've spent the day playing with the patch you posted (I live in germany).
I wrote a few lines into test-installwatch to try out the openat function and was hoping you'd write something better than what I came up with because I wanted to check if the backup feature was working and couldn't get the directory and file creation working with the proper perms so that the backup would take place. Since all the normal testing takes place under /tmp that was a problem also as /dev and /tmp are ignored. Even though you can choose exclude dirs, these two plus the installwatch rootdir are skipped over by hard-coded lines.

I'll have to take some time tonite and tomorrow to try out your testing code. If I knew of some source-code that was definitely making use of the openat fetaures I could give it more of a whirl.

I even tried hacking your code around to divert all calls to openat to be done by the regular open function but didn't quite get that right. I did notice one thing that I think is wrong in the openat function.

Code:
instw_getstatus(&instw,&status);

	if(status&INSTW_TRANSLATED) 
		result=true_openat(instw.translpath,flags,mode);
	else
		result=true_openat(instw.path,flags,mode);
	
	if(flags & (O_WRONLY | O_RDWR)) 
		logg("%d\topenat\t%s\t#%s\n", result,instw.path, error(result));

	instw_delete(&instw);
If I understand correctly what is happening in the other functions, these two calls should be to true_openat as they are meant to actually make the real changes after the backup is done(if called for).

I also change the format of the logg output to match the normal output since parsing the log output is the principle way of using the program.
I did get the openat function working okay and even creating backups, but it didn't seem to be doing the real write somehow.
There's no doubt that the translation code is the weakness -anytime users had problems with installwatch they were recommended to disable the feature.

I also changed this in your code:
Code:
snprintf(proc_path, PROC_PATH_LEN, "/proc/self/fd/%d", dirfd);
	if(true_stat(proc_path, &s) == -1)
		goto out;
	if(!(newpath = malloc(s.st_size+strlen(relpath)+2)))
		goto out;
	if((l = true_readlink(proc_path, newpath, s.st_size)) == -1)
		goto free_out;
	newpath[l] = '/';
Since stat and readlink are used internally there should be no reason
to print the extra noise in the debugfile, at least once it is working correctly.

In order to check the backup and translation features you need to pass the options to installwatch --backup=yes --transl=yes and dbglvl=2 (or 4).


I even did some rude changes like what's below in order to force the backup feature to work. And about the logging -some of the other functions something like this depending on whether translation is being used. Notice I was sending everything to the regular open function here also. I even tried gutting nearly the whole function and just using instw_setpathrel to fix the path before sending the call to open.

Code:
int openat (int dirfd, const char *path, int flags, ...) {
	mode_t mode = 0;
	va_list arg;
	if(flags & O_CREAT) {
		va_start(arg, flags);
		mode = va_arg(arg, mode_t);
		va_end (arg);
	}
	int result, status;
	instw_t instw;

	/* If all we are doing is normal open, forgo refcounting, etc. */
        if(dirfd == AT_FDCWD || *path == '/') {
                printf("hello1\n");
		return open(path, flags, mode);
		logg("%d\topenat\t%s\t#%s\n", result,path, error(result));
	}

	REFCOUNT;

	if (!libc_handle)
		initialize();

#if DEBUG
	debug(2, "openat(%d, %s, 0x%x, 0%o)\n", dirfd, path, flags, mode);
#endif
	

	instw_new(&instw);
	instw_setpathrel(&instw,dirfd,path);
	
	  /* We were asked to work in "real" mode */
	if(!(__instw.gstatus & INSTW_INITIALIZED) ||
	   !(__instw.gstatus & INSTW_OKWRAP))
		return open(path,flags,mode);
	
#if DEBUG
	instw_print(&instw);
#endif

	/* if(flags & (O_WRONLY | O_RDWR)) {
		backup(instw.truepath);
		instw_apply(&instw);
	} */
	
	backup(instw.truepath);
	instw_apply(&instw);

	instw_getstatus(&instw,&status);

	if(status&INSTW_TRANSLATED) {
		printf("hello2\n");
		result=open(instw.translpath,flags,mode);
		logg("%d\topenat\t%s\t#%s\n", result,instw.translpath, error(result));
	} else {
		printf("hello3\n");
		result=open(instw.path,flags,mode);
		logg("%d\topenat\t%s\t#%s\n", result,instw.path, error(result));	
		}
	
	/* if(flags & (O_WRONLY | O_RDWR))
		logg("%d\topenat\t%s\t#%s\n", result,
		    instw.reslvpath, error(result)); */

	instw_delete(&instw);

	return result;
}

#endif /* GLIBC_MINOR >= 4 */
I wish I knew enough about this to be more helpful. Thanks for helping out so much. I'll try out the new testing code and see what I can foul up there...
 
Old 12-02-2007, 04:12 PM   #17
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,928

Original Poster
Rep: Reputation: 612Reputation: 612Reputation: 612Reputation: 612Reputation: 612Reputation: 612
It's getting late here you I can't play anymore today. But I wanted to let you know that it seems to be working pretty okay here. The only thing that's not working correctly is read/write using a relative path and translation. This is when using the first test program.

If you'll run it plain the way you have been doing, then run it using the installwatch --backup=yes option and then with --transl=yes option, and then compare the logs you'll see what's happening. The existing translation code chokes on the /some/path/../like/this construction. Be sure to use the --dbglvl=4 option every time so you can see the debug file int /tmp/xxxx

I've tried it like what's here below. The only change is to comment out the logging -since the normal open function logs what it does, the logging here is redundant.

Code:
int openat (int dirfd, const char *path, int flags, ...) {
	mode_t mode = 0;
	va_list arg;
	if(flags & O_CREAT) {
		va_start(arg, flags);
		mode = va_arg(arg, mode_t);
		va_end (arg);
	}
	int result, status;
	instw_t instw;

	/* If all we are doing is normal open, forgo refcounting, etc. */
        if(dirfd == AT_FDCWD || *path == '/')
                return open(path, flags, mode);

	REFCOUNT;

	if (!libc_handle)
		initialize();

	debug(2, "openat(%d, %s, 0x%x, 0%o)\n", dirfd, path, flags, mode);

	  /* We were asked to work in "real" mode */
	if(!(__instw.gstatus & INSTW_INITIALIZED) ||
	   !(__instw.gstatus & INSTW_OKWRAP))
		return true_openat(dirfd,path,flags,mode);

	instw_new(&instw);
	instw_setpathrel(&instw,dirfd,path);
	
#if DEBUG
	instw_print(&instw);
#endif

	if(flags & (O_WRONLY | O_RDWR)) {
		backup(instw.truepath);
		instw_apply(&instw);
	}

	instw_getstatus(&instw,&status);

	if(status&INSTW_TRANSLATED) 
		result=open(instw.translpath,flags,mode);
	else
		result=open(instw.path,flags,mode);
	
	/* if(flags & (O_WRONLY | O_RDWR)) 
		logg("%d\topenat\t%s\t#%s\n", result,instw.path, error(result)); */

	instw_delete(&instw);

	return result;
}

#endif /* GLIBC_MINOR >= 4 */
It looks like it all might work if setrelpath could simplify this:
/some/path/../like/this
to this:
/some/like/this
 
Old 12-03-2007, 01:24 PM   #18
osor
HCL Maintainer
 
Registered: Jan 2006
Distribution: (H)LFS, Gentoo
Posts: 2,450

Rep: Reputation: 78
Quote:
Originally Posted by gnashley View Post
I even tried hacking your code around to divert all calls to openat to be done by the regular open function but didn't quite get that right. I did notice one thing that I think is wrong in the openat function.

Code:
instw_getstatus(&instw,&status);

	if(status&INSTW_TRANSLATED) 
		result=true_openat(instw.translpath,flags,mode);
	else
		result=true_openat(instw.path,flags,mode);
	
	if(flags & (O_WRONLY | O_RDWR)) 
		logg("%d\topenat\t%s\t#%s\n", result,instw.path, error(result));

	instw_delete(&instw);
If I understand correctly what is happening in the other functions, these two calls should be to true_openat as they are meant to actually make the real changes after the backup is done(if called for).
The reason you can’t call true_openat() is that you’ve already translated the pathname to be absolute (instead of relative to the original fd).
Quote:
Originally Posted by gnashley View Post
I also changed this in your code:
Code:
snprintf(proc_path, PROC_PATH_LEN, "/proc/self/fd/%d", dirfd);
	if(true_stat(proc_path, &s) == -1)
		goto out;
	if(!(newpath = malloc(s.st_size+strlen(relpath)+2)))
		goto out;
	if((l = true_readlink(proc_path, newpath, s.st_size)) == -1)
		goto free_out;
	newpath[l] = '/';
Since stat and readlink are used internally there should be no reason
to print the extra noise in the debugfile, at least once it is working correctly.
Good point. The only difference made should be less noise in the logfile (since /proc is by default already one of the excluded dirs).
Quote:
Originally Posted by gnashley View Post
I even tried gutting nearly the whole function and just using instw_setpathrel to fix the path before sending the call to open.
I think this would actually be a good idea. It would move any and all logging/backup logic to the (intercepting) open() function (which would cut down on code duplication).

Quote:
Originally Posted by gnashley View Post
It's getting late here you I can't play anymore today. But I wanted to let you know that it seems to be working pretty okay here. The only thing that's not working correctly is read/write using a relative path and translation. This is when using the first test program.

If you'll run it plain the way you have been doing, then run it using the installwatch --backup=yes option and then with --transl=yes option, and then compare the logs you'll see what's happening. The existing translation code chokes on the /some/path/../like/this construction. Be sure to use the --dbglvl=4 option every time so you can see the debug file int /tmp/xxxx
Yes, I noticed that. So this problem actually also occurs in this snippet of code (which is slightly different from the code in post 15:
Code:
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdio.h>
#include <unistd.h>

#define NEW_DIR "/usr/foo"
#define REL_READ_FILE "../lib/libssl.so"

int new_dir_rel_read_file(void)
{
	printf("Checking if you can read a relative file in an new dir...\n");

	int fd, retval = 0;
	static char buf[5];
	if((fd = open(NEW_DIR "/" REL_READ_FILE, O_RDONLY)) == -1)
		goto out;
	if(read(fd, buf, sizeof(buf) - 1) != sizeof(buf) - 1)
		goto close_out;
	retval = 1;
	printf("\tthe next three letters should be ELF: %s.\n", buf+1);

close_out:
	close(fd);
out:
	if(!retval)
		perror(__func__);
	return retval;
}

int main(int argc, char *argv[])
{
	printf("Attempting to create " NEW_DIR "...\n");
	if(mkdir(NEW_DIR, 0755) == -1) {
		perror(NEW_DIR " could not be created");
		return -1;
	}
	else
		printf("\t" NEW_DIR " successfully created.\n");

	new_dir_rel_read_file();

	return 0;
}
Quote:
Originally Posted by gnashley View Post
It looks like it all might work if setrelpath could simplify this:
/some/path/../like/this
to this:
/some/like/this
Yes, in fact this is the job of the function called canonicalize(). The function canonicalize() is in fact implemented using the POSIX interface called realpath(). There is only one problem in placing a call to canonicalize() within setpath()—links. You see POSIX specified that realpath() will expand links as well as simplify excess occurrences of “..” and “/” in a path. For our purposes, we want all the functionality of realpath() EXCEPT the link expansion (and AFAIK, no such standard function exists). This is because link expansion cause a possible race condition. In fact, it if we put link-expansion in our translating function, calls using O_NOFOLLOW will always succeed (even if they would fail without translation). Additionally, if we put the code in the general area for translation, it will mess up calls to lstat and the like.

The only solution I see here is to create our own “half-realpath” function (that is, exactly like realpath except for link translation). The good news is that it shouldn’t be too complicated (since the destination string is always shorter than the initial string, we can even translate a path in-place). I’ll get an appropriate patch going soon.
 
Old 12-03-2007, 03:28 PM   #19
osor
HCL Maintainer
 
Registered: Jan 2006
Distribution: (H)LFS, Gentoo
Posts: 2,450

Rep: Reputation: 78
Quote:
Originally Posted by osor View Post
I’ll get an appropriate patch going soon.
Well, here is the solution I have up until now. The patches are incremental and build upon 04.patch.

05.patch
Code:
    Since stat and readlink are used internally there should be no reason to print
    the extra noise in the debugfile, at least once it is working correctly.

--- a/installwatch.c
+++ b/installwatch.c
@@ -1548,11 +1548,11 @@ static int instw_setpathrel(instw_t *instw, int dirfd, const char *relpath) {
 	struct stat s;
 
 	snprintf(proc_path, PROC_PATH_LEN, "/proc/self/fd/%d", dirfd);
-	if(stat(proc_path, &s) == -1)
+	if(true_stat(proc_path, &s) == -1)
 		goto out;
 	if(!(newpath = malloc(s.st_size+strlen(relpath)+2)))
 		goto out;
-	if((l = readlink(proc_path, newpath, s.st_size)) == -1)
+	if((l = true_readlink(proc_path, newpath, s.st_size)) == -1)
 		goto free_out;
 	newpath[l] = '/';
 	strcpy(newpath + l + 1, relpath);
06.patch
Code:
    Let the wrapper for open handle all logging and backing up (even though it’s)
    less efficient.

--- a/installwatch.c
+++ b/installwatch.c
@@ -3724,7 +3724,7 @@ int openat (int dirfd, const char *path, int flags, ...) {
 		mode = va_arg(arg, mode_t);
 		va_end (arg);
 	}
-	int result, status;
+	int result;
 	instw_t instw;
 
 	/* If all we are doing is normal open, forgo refcounting, etc. */
@@ -3750,21 +3750,7 @@ int openat (int dirfd, const char *path, int flags, ...) {
 	instw_print(&instw);
 #endif
 
-	if(flags & (O_WRONLY | O_RDWR)) {
-		backup(instw.truepath);
-		instw_apply(&instw);
-	}
-
-	instw_getstatus(&instw,&status);
-
-	if(status&INSTW_TRANSLATED) 
-		result=true_open(instw.translpath,flags,mode);
-	else
-		result=true_open(instw.path,flags,mode);
-	
-	if(flags & (O_WRONLY | O_RDWR)) 
-		logg("%d\topenat\t%d\t%s\t0x%x\t0%o\t#%s\n", result, dirfd,
-		    path, flags, mode, error(result));
+	result = open(instw.path, flags, mode);
 
 	instw_delete(&instw);
07.patch
Code:
    Introduce a recursive function called “reduce()” which does in-place path
    reduction (i.e., gets rid of "//", "..", and ".") of absolute paths. This
    is like realpath() except it doesn’t process links and doesn’t need an
    extra buffer. This new function will be called from setpath.

--- a/installwatch.c
+++ b/installwatch.c
@@ -213,6 +213,7 @@ typedef struct instw_t {
 static instw_t __instw;
 
 static int canonicalize(const char *,char *);
+static int reduce(char *);
 static int make_path(const char *);
 static int copy_path(const char *,const char *);
 static inline int path_excluded(const char *);
@@ -525,6 +526,61 @@ static int canonicalize(const char *path, char *resolved_path) {
 	return 0;
 } 
 
+/*
+ * procedure = / rc:=reduce(path) /
+ *
+ * task      = /   reduces all occurences of "..", ".", and extra "/" in path.
+ *
+ * inputs    = / path               The modifiable string containing the path
+ * outputs   = / path               The reduced path.
+ *
+ * returns   = /  0 ok. path reduced
+ *               -1 failed. cf errno /
+ * note      = /
+ *      --Very similar to canonicalize()/realpath() except we don’t do link-
+ *      expansion
+ *      --This is purely a string manipulation function (i.e., no verification
+ *      of a path’s validity occurs).
+ *      --Additionally, we try to do reduction “in-place” since the ending
+ *      path is shorter than the beginning path.
+ *      --Also, we want only absolute paths (other paths will throw an error)
+ * /
+ */
+static int reduce(char *path) {
+	int len;
+	char *off;
+
+	if(path == NULL || *path != '/') {
+		errno = EINVAL;
+		return -1;
+	}
+
+	len = strlen(path);
+
+	/* First, get rid of double / */
+	if(off = strstr(path, "//")) {
+		memmove(off, off+1, len - (off-path));
+		return reduce(path);
+	}
+
+	/* Then, worry about /../ */
+	if(off = strstr(path, "/../")) {
+		char *off2 = off;
+		if(off2++ != path)
+			while((--off2)[-1] != '/');
+		memmove(off2, off+4, len - 3 - (off-path));
+		return reduce(path);
+	}
+
+	/* Finally, do /./ */
+	if(off = strstr(path, "/./")) {
+		memmove(off, off+2, len - 1 - (off-path));
+		return reduce(path);
+	}
+
+	return 0;
+}
+
 static int make_path (const char *path) {
 	char checkdir[BUFSIZ];
 	struct stat inode;
@@ -1448,6 +1504,7 @@ static int instw_setpath(instw_t *instw,const char *path) {
 		}
 		strcat(instw->truepath,instw->path);
 	} else {
+		reduce(instw->path);
 		strcpy(instw->truepath,instw->path);
 	}
 	relen=strlen(instw->truepath);
With the last of these patches, both the test programs from post 14 and post 18 run correctly with translation. As of yet, the test program from post 15 still doesn’t pass.

Last edited by osor; 12-03-2007 at 05:47 PM. Reason: typo
 
Old 12-03-2007, 10:04 PM   #20
osor
HCL Maintainer
 
Registered: Jan 2006
Distribution: (H)LFS, Gentoo
Posts: 2,450

Rep: Reputation: 78
Here are a few more patches for when you wake up :
08.patch
Code:
    Switch the order of processing /./ and /../ in reduce().

--- a/installwatch.c
+++ b/installwatch.c
@@ -563,7 +563,13 @@ static int reduce(char *path) {
 		return reduce(path);
 	}
 
-	/* Then, worry about /../ */
+	/* Then, worry about /./ */
+	if(off = strstr(path, "/./")) {
+		memmove(off, off+2, len - 1 - (off-path));
+		return reduce(path);
+	}
+
+	/* Finally, do /../ */
 	if(off = strstr(path, "/../")) {
 		char *off2 = off;
 		if(off2++ != path)
@@ -572,12 +578,6 @@ static int reduce(char *path) {
 		return reduce(path);
 	}
 
-	/* Finally, do /./ */
-	if(off = strstr(path, "/./")) {
-		memmove(off, off+2, len - 1 - (off-path));
-		return reduce(path);
-	}
-
 	return 0;
 }
09.patch
Code:
    This improves reduce() for those pesky cases where the culprit is at the end of
    the path (e.g., “/foo/bar/”, “/foo/bar/.”, or “/foo/bar/..”). We can’t just
    scan for a substring because for all we know that is the desired filename (e.g.,
    both “/foo/bar/..baz” and “/foo/bar../baz” would be clobbered). So we treat the
    case of the ending separately.
    
    Additionally, it seems we might have a candidate for IOCCC :)

--- a/installwatch.c
+++ b/installwatch.c
@@ -577,8 +577,31 @@ static int reduce(char *path) {
 		memmove(off2, off+4, len - 3 - (off-path));
 		return reduce(path);
 	}
-
-	return 0;
+	/* Beautify ending */
+	switch(path[len - 1]) {
+		case '.':
+			switch(path[len - 2]) {
+				default:
+					return 0;
+				case '.':
+					if(len != 3) {
+						off = path+len-3;
+						if(*off-- != '/')
+							return 0;
+						while(*--off != '/');
+						off[1] = 0;
+						return reduce(path);
+					}
+				case '/': ;
+			}
+		case '/':
+			if(len != 1) {
+				path[len-1] = 0;
+				return reduce(path);
+			}
+		default:
+			return 0;
+	}
 }
 
 static int make_path (const char *path) {
The first (08.patch) fixes a small bug in my first reduce(). The second patch isn’t strictly necessary for our purposes (at least not yet), but it finishes off the odd case where we need to reduce the very end of a path.

Personally, I think this whole project (implementing *at() functions in installwatch so that it works with newer coreutils) is a very temporary fix. It might be better to get a new backend for src2pkg (one that doesn’t rely on glibc internals or LD_PRELOAD hacks). It would be much more elegant to use something like ptrace() to intercept at the system call-level. From this I see a few advantages (just off the top of my head):
  1. The number of functions you need to manipulate would go down (since you would be dealing only with kernel-level system calls).
  2. You could run as a parent controlling his child instead of through LD_PRELOAD.
  3. Since the parent program would be unaffected by the interception, you wouldn’t have to step around yourself with things like unset_okwrap()/reset_okwrap().
  4. You wouldn’t need to care about glibc internals such as __lxstat. In fact, you would only worry about additions of new kernel-level system calls. This means that you can use installwatch with an alternate libc (e.g., uClibc).
Anyhow, if you do want to continue by patching up the current installwatch for the *at() functions (at least for now), I think I’ve given you a pretty good boilerplate.
 
Old 12-04-2007, 11:04 AM   #21
gnashley
Amigo developer
 
Registered: Dec 2003
Location: Germany
Distribution: Slackware
Posts: 4,928

Original Poster
Rep: Reputation: 612Reputation: 612Reputation: 612Reputation: 612Reputation: 612Reputation: 612
Thanks for the new code... I sincerely hope you are resting today from your -I mean my labors! Of course I have not, but I hope you are. I'm going to write a little more tomorrow and sum up what the outcome is. In the meantime, I'd like to have your name and e-mail address in order to properly credit your excellent work. You can drop me a line at amigo@ibiblio.org if you don't want to post it here.
This all seems to be working pretty solidly except for backup of links -especially relative links. I notice in the sources for coreutils that they quickly back off and revert to more normal usgae when there are problem areas. Most of the openat stuff seems to be used internally -I haven't been able to uncover any programs which call them directly yet. Well, I'll tell you more about it tomorrow. Thanks again for sharing my headache.
 
Old 12-05-2007, 12:59 PM   #22
osor
HCL Maintainer
 
Registered: Jan 2006
Distribution: (H)LFS, Gentoo
Posts: 2,450

Rep: Reputation: 78
Quote:
Originally Posted by gnashley View Post
In the meantime, I'd like to have your name and e-mail address in order to properly credit your excellent work.
I would prefer not to be credited (in part because I don’t think of my work here as being even close to excellent).
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: OpenOffice.org Calc functions, part 1: Understanding functions LXer Syndicated Linux News 0 03-31-2007 12:01 PM
How to know if I am using mbox-style or maildir-style? Niceman2005 Linux - General 1 09-23-2005 12:08 PM
Converting php5 socket functions to php3 socket functions mrobertson Programming 0 06-23-2005 09:11 AM
Proper style for functions exvor Programming 4 02-13-2005 12:07 PM
VIM-style wrapping to OpenOffice style schmmd Linux - Software 1 12-21-2004 06:50 PM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 04:29 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration