Why doesn't inotify detect the creation of files in home directories?

RandomTroll · 12-03-2015, 09:40 PM

I wrote a program to detect (with inotify) the creation of rogue .htaccess files that hackers are creating in some of the directories of our website. It doesn't detect the creation of files in home directories (e.g. /root or /home/randomtroll) but does in subdirectories thereof. It reports that everything is working up to the point that the infinite loop starts; the process shows up in 'ps' (if it didn't go into the infinite loop it would exit immediately); when I send it a quit signal it reports quitting properly.

Because it runs as root it can see those directories. Fortunately I don't need this for the current application. I'm just wondering why it works this way.

chrism01 · 12-03-2015, 10:58 PM

It would be a start if you showed your code - otherwise we're just guessing.

Habitual · 12-04-2015, 05:19 AM

Quote:

Originally Posted by RandomTroll

I wrote a program to detect (with inotify) the creation of rogue .htaccess files that hackers are creating in some of the directories of our website.

You have a larger problem than inotify, my friend.

RandomTroll · 12-04-2015, 09:13 AM

Quote:

Originally Posted by chrism01

It would be a start if you showed your code - otherwise we're just guessing.

Code:

  InotifyDescriptor=inotify_init();
.
.
.
  Watch=inotify_add_watch(InotifyDescriptor, TargetDir, IN_CREATE);
.
.
.

  while(ExitFlag==0){
    read(InotifyDescriptor, buffer, buffer_length);
    WatchEvent = (const struct inotify_event *) buffer;
    if (strcmp(WatchEvent->name,TargetFileName) == 0 ){
      time(&timenow);
      LocalTime=localtime(&timenow);
      fprintf(LogFileHandle,"Deleting %s at %d %02d %02d, %02d:%02d\n", TargetFile, (LocalTime->tm_year)+1900, (LocalTime->tm_mon)+1 , LocalTime->tm_mday, LocalTime->tm_hour, LocalTime->tm_min);
      fflush(LogFileHandle);
      remove(TargetFile);
    }
  }

I tested a variety of permissions, to no avail.

RandomTroll · 12-04-2015, 09:13 AM

Quote:

Originally Posted by Habitual

You have a larger problem than inotify, my friend.

Is this response cryptic or obvious?

Habitual · 12-04-2015, 10:03 AM

Quote:

Originally Posted by RandomTroll

Is this response cryptic or obvious?

It should bite you on the @ss it's so obvious.

Quote:

Originally Posted by RandomTroll

rogue .htaccess files that hackers are creating

The issue is how they are creating them.
Files should be 644
Directories should be 755
Exceptions may be cgi scripts in expected locations.

RandomTroll · 12-04-2015, 01:15 PM

Quote:

Originally Posted by Habitual

It should bite you on the @ss it's so obvious.

Nothing has bitten me anywhere. I'm not the webmaster; I don't get to decide how the site is managed. I just help out with technical problems. (It's a charitable environmental org for which I worked 10 years ago; I work for free now.) I didn't ask for a solution to the site's hacking but for an explanation of why inotify behaves in this way. I explained the background not to ask how to address the larger problem but to make what I did ask about more understandable. I also observed that this behavior doesn't happen for a target directory for which I use the program, so it isn't a problem, but a puzzle.

Quote:

Originally Posted by Habitual

The issue is how they are creating them.

I didn't ask about this problem.

Quote:

Originally Posted by Habitual

Files should be 644
Directories should be 755
Exceptions may be cgi scripts in expected locations.

Every file has correct permissions. We did this a long time ago. For a few years now the only hacks have been through Wordpress vulnerabilities, which we keep abreast of and solve when they're reported.

norobro · 12-04-2015, 01:35 PM

Perhaps I misunderstand your problem, but your code modified as follows works for me:

Code:

    fd = inotify_init();
    wd=inotify_add_watch(fd, "/home/user", IN_CREATE);
    wd1=inotify_add_watch(fd, "/home/user/Downloads/", IN_CREATE);
    while(1){
        read(fd, buffer, EVENT_BUF_LEN);
        WatchEvent = (const struct inotify_event *) buffer;
        if (strcmp(WatchEvent->name,"test.txt") == 0 ){
          time(&timenow);
          LocalTime=localtime(&timenow);
          printf("Deleting %s at %d %02d %02d, %02d:%02d\n", "test.txt", (LocalTime->tm_year)+1900, (LocalTime->tm_mon)+1 , LocalTime->tm_mday, LocalTime->tm_hour, LocalTime->tm_min);
          fflush(stdout);
        }
    }

Creating a file in /home/user yields: Deleting test.txt at 2015 12 04, 13:28

You mention file permissions, but what about directory permissions. For example, if you run the program as user it will not detect files created in /root even if the file is 755.

Habitual · 12-04-2015, 01:55 PM

Quote:

Originally Posted by RandomTroll

I didn't ask about this problem.

No harm, no foul then?

Have a Great Day!

astrogeek · 12-04-2015, 02:38 PM

Quote:

Originally Posted by Habitual

You have a larger problem than inotify, my friend.

No good deed goes unpunished...

I have had mixed results trying to use inotify, in particular I always found file creation detection to be unreliable.

At one time I began to write a How-To on my efforts for slackdocs.org, but after spending far too much time writing and testing scripts without a definite resolution, I abandoned the effort and the use of inotify for my own project.

I do not know if this is the cause, as I found missed events that did not closely correlate to directory creation, but from man inotifywait:

Code:

BUGS
       There are race *conditions in the recursive directory watching code which can cause events to  be  missed
       if they occur in a directory immediately after that directory is created.  This is probably not fixable.

       It is assumed the inotify event queue will never overflow.

*Comment - that is conditions, plural, which is probably why it is so gnarly to quantify...

I looked very closely at contrived examples to figure out how to avoid and/or work around the race condition(s), but found it to be very elusive and ultimately impossible to reliably fix - probably why the author says it is ultimately not fixable...

The queue overflow can also be a problem when watching a large and/or dynamic tree.

Also this...

Code:

       -r, --recursive
              Watch  all  subdirectories of any directories passed as arguments.  Watches will be set up recur
              sively to an unlimited depth.  Symbolic links are not traversed.   Newly  created  subdirectories
              will also be watched.

              Warning:  If  you  use this option while watching the root directory of a large tree, it may take
              quite a while until all inotify watches are established, and events will not be received in  this
              time.   Also,  since  one inotify watch will be established per subdirectory, it is possible that
              the maximum amount of inotify watches per user will be reached.  The default maximum is 8192;  it
              can be increased by writing to /proc/sys/fs/inotify/max_user_watches.

I would think this is an easy limit to hit if you are watching a WordPress directory tree.

Finally, as Habitual pointed out (quite helpfully I thought), if an intruder has the ability to write files on that machine, then really all other bets are off - including inotify event watches. Whether or not you are responsible for fixing that problem is irrelevant - that problem must be fixed before you can seriously deal with anything else. Just a helpful observation.

For example, this is one possible answer to the question, "Why doesn't inotify detect the creation of files...?". Knowing that you might be using inotify watch, an intruder might do this...

Code:

mkdir dummydir; echo "... malicious directives" >.htaccess; rmdir dummydir;

... using the known race conditions to mask creation of the .htaccess file.

You cannot win that race so long as the intruder can write to the filesystem.

RandomTroll · 12-04-2015, 05:37 PM

I added a bit of code to test the read of the inotify descriptor in the loop. When it doesn't work, it returns EINVAL. The man page for read reads:

Quote:

EINVAL fd is attached to an object which is unsuitable for reading; or
the file was opened with the O_DIRECT flag, and either the address
specified in buf, the value specified in count, or the current file
offset is not suitably aligned.

The last thing that happens before the loop starts is a status report, for example:

Quote:

Watching /home/randomtroll at 2015 12 04, 16:18 ; InotifyDescriptor 3 ; Watch 1 ; pid 0 ; sid 25101

a valid directory, inotify descriptor, and watch; the same values returned when it works. The program works for /home/randomtroll/download, which has the same permissions and owner as /home/randomtroll. I run the program as root.

RandomTroll · 12-04-2015, 05:57 PM

Quote:

Originally Posted by astrogeek

I do not know if this is the cause, as I found missed events that did not closely correlate to directory creation, but from man inotifywait:

I don't use inotifywait. Directory creation isn't a problem. I wrote this tiny app to monitor 1 directory then a script to start it for the 191 directories we want to watch. It doesn't search for new directories. As near as I can tell on the website it works as intended. When testing it (on my own computer, with very little else happening, no other inotify tasks) I found it doesn't work for all directories and couldn't find any difference between the ones in which it works and the others. I ask out of curiosity and because I may need to use it again some day.

Quote:

Originally Posted by astrogeek

Finally, as Habitual pointed out (quite helpfully I thought)

It was noise at best, an attempt to disparage at worst. If I can write a program that uses inotify I know that hackers hacking the website are a bigger problem. We're working on that. On my end I've secured ssh and ftp and monitor for other kinds of breakins. No one has gotten in, at least in a way I can detect, in years, through them. The webmaster handles the publication of content, which he does with Wordpress. I don't know anything about Wordpress and he hasn't asked me to. I'm a volunteer, so I need to be asked.

Quote:

Originally Posted by astrogeek

if an intruder has the ability to write files on that machine, then really all other bets are off

Webmaster tells me that Wordpress allows creation of .htaccess files. We've both searched for other rogue files. I told him to stop using .htaccess and offered to write a program to kill them whenever they pop up. I haven't cured cancer either.

Quote:

Originally Posted by astrogeek

Whether or not you are responsible for fixing that problem is irrelevant - that problem must be fixed before you can seriously deal with anything else. Just a helpful observation.

No, it's not: it's an insulting observation.

astrogeek · 12-04-2015, 06:47 PM

Quote:

Originally Posted by RandomTroll

It was noise at best, an attempt to disparage at worst.
...
No, it's not: it's an insulting observation.

None of the above and sorry you feel insulted, only trying to help.

Good luck with that.

berndbausch · 12-04-2015, 08:32 PM

Quote:

Originally Posted by RandomTroll

No, it's not: it's an insulting observation.

This is a bit off-topic, but perhaps it helps. I stepped on somebody's toes like this recently. The thing is, even if you know how to write a C program that uses inotify, whoever responds here doesn't know you; pointing out that there is a larger problem is not an insult.

Also, you are not the only one reading this thread. The remark might well be useful for somebody else who perhaps has problems with .htaccess files popping out of nowhere.