LinuxQuestions.org
Latest LQ Deal: Complete CCNA, CCNP & Red Hat Certification Training Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 08-27-2014, 06:22 AM   #1
rex1254
LQ Newbie
 
Registered: Aug 2014
Posts: 15

Rep: Reputation: Disabled
How to recursively monitor an directory besides using inotify


Hi,
I want to make an C application that will monitor file changes in its directory or its sub-directories. I know inotify can monitor single level directory changes and need to manually add a watch for each sub-directory to detect changes in the sub-directory.
I need to know if there is any other way to recursively monitor changes in a directory in Linux other than inotify.
 
Old 08-27-2014, 11:24 PM   #2
ventsyv
Member
 
Registered: Apr 2006
Posts: 57

Rep: Reputation: 3
Loop over each entry in the directory, if the entry is a directory, go in it and repeat. If the entry is a file, check it's timestamp to see if it was modified since the last time you ran the check.
It's a bit tricky if you want to follow links because that could lead to a loop, but if you don't it's rather simple.
Have the check run on a loop every so often - every few seconds should be OK I think.
 
Old 08-30-2014, 08:10 AM   #3
rex1254
LQ Newbie
 
Registered: Aug 2014
Posts: 15

Original Poster
Rep: Reputation: Disabled
Thanks for your reply ventsyv but won't this approach cost a lot of memory. Is there any other solution that will cost less memory.
 
Old 09-01-2014, 03:03 AM   #4
rex1254
LQ Newbie
 
Registered: Aug 2014
Posts: 15

Original Poster
Rep: Reputation: Disabled
Thanks for the reply. Is there any other method that will be less costly in terms of memory or any other subsystem like inotify that i can use to monitor directory changes recursively
 
Old 09-01-2014, 05:35 PM   #5
joe_2000
Member
 
Registered: Jul 2012
Location: Aachen, Germany
Distribution: Void, Debian
Posts: 808

Rep: Reputation: 216Reputation: 216Reputation: 216
If I understand you right your reason not to use inotify is the limitation that it does not support recursion?
So why do you not write a script / application that automates the recursion process of adding the inotify watches? (I should add that I have no experience with inotify, so I might be missing something obvious).

The big advantage of inotify obviously is its integration with the kernel. I don't think you'll be able to code something anywhere near as efficient in a pure user-land application.

That said, if I really *had* to code something completely manually and if there was a requirement to handle *big* directory structures I would probably not try to store them as a whole but look into options like generating md5hashes or so... Obviously that would make it more CPU intensive. Notification would not be immediate, but you would save memory. You would basically trade CPU usage for RAM. Does that make sense?

EDIT: One other thought that came up after I hit reply: Depending on what your infrastructure is you could also think about storing dirstructure status in a database. That would save memory, but increase disk usage. Either way, it would help if you gave some more background on your requirements / usecase

Last edited by joe_2000; 09-01-2014 at 05:38 PM.
 
Old 09-01-2014, 07:02 PM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,331
Blog Entries: 55

Rep: Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530Reputation: 3530
Quote:
Originally Posted by rex1254 View Post
I know inotify can monitor single level directory changes and need to manually add a watch for each sub-directory to detect changes in the sub-directory. I need to know if there is any other way to recursively monitor changes in a directory in Linux other than inotify.
I think you should first read this explanation by Robert Love. Food for thought. Then see man inotifywait (wrt "--recursive"), Recursive Inotify Daemon, inotify-tools, fanotify or else any LD_PRELOAD or Linux Kernel Module (see for example Linux Antivirus products) that will intercept open, read, write and close syscalls?
 
Old 09-02-2014, 08:15 AM   #7
rex1254
LQ Newbie
 
Registered: Aug 2014
Posts: 15

Original Poster
Rep: Reputation: Disabled
Thanks for the suggestions joe_2000
 
Old 09-02-2014, 08:44 AM   #8
rex1254
LQ Newbie
 
Registered: Aug 2014
Posts: 15

Original Poster
Rep: Reputation: Disabled
Thanks for the reply unSpawn.
 
Old 09-05-2014, 03:36 AM   #9
rex1254
LQ Newbie
 
Registered: Aug 2014
Posts: 15

Original Poster
Rep: Reputation: Disabled
Do i need to create a new fd for each sub directory that i encounter in the directory tree or do i just need to create a fd for the base directory that i want to monitor and just keep on adding watch for new sub directories with this same fd.
 
Old 09-11-2014, 08:29 AM   #10
rex1254
LQ Newbie
 
Registered: Aug 2014
Posts: 15

Original Poster
Rep: Reputation: Disabled
Need only 1 fd i.e. for watch dir. No need to create a new fd to sub dir in a dir tree
 
Old 09-11-2014, 03:15 PM   #11
jstephens84
Senior Member
 
Registered: Sep 2004
Location: Nashville
Distribution: Manjaro, RHEL, CentOS
Posts: 2,098

Rep: Reputation: 102Reputation: 102
Well I may be a little late to the party but here is a simple program I just built using python.

Code:
'''
demonstrates how to write and read files.
'''
__author__ = 'Josh Stephens'

import os

def walk(path):
    directories = []
    files = []
    contents = {'files': [], 'directories': []}
    for root, subdirectories, files in os.walk(path):
        for file in files:
            contents['files'].append(os.path.join(root, file))
        for directory in subdirectories:
            contents['directories'].append(os.path.join(root, directory))

    return contents

'''
When reading files use the r string to stand for read. Use w to write to files. (NOTE: this will clear out
the contents of a file as soon as it opens.) Use the a string to append to a file.
'''

fout = open('pywriter.log', 'w')
print fout

base_directory = r'/home/stephensj/'
msg = '''
Base Directory {}
''' .format(base_directory)
base_directory_items = walk(base_directory)

fout.writelines(msg + '\n')
fout.writelines('\n')
fout.writelines('[ Files ]\n')
for file_items in base_directory_items['files']:
    fout.writelines(file_items + '\n')

fout.writelines('\n')
fout.writelines('[ Directories ]\n')
for directory_items in base_directory_items['directories']:
    fout.writelines(directory_items + '\n')


print('Finished writing to log {}'.format(os.path.join(os.getcwd(), 'pywriter.log')))
fout.writelines('\n')
fout.writelines('Number of Files: {}\n'.format(str(len(base_directory_items['files']))))
fout.writelines('Number of directories: {}\n'.format(str(len(base_directory_items['directories']))))
fout.close()
Currently all it does is take files and directories and adds them into a dictionary and then writes it out to a file. However you could extend it to do something upon new directory or file by doing the following

Code:
if new_item not in base_directory_items:
    ...do something
Probably will need to add an main loop that will sleep for x seconds just to cut down on the IO consumption. You could also take the skeleton and port it over to C.
 
  


Reply

Tags
c++, linux


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
inotify to watch a directory then print and move algreig Linux - Newbie 4 05-25-2012 11:14 AM
inotify and filename / directory name change events euroquisling Programming 3 05-20-2009 04:26 AM
inotify for recursive directory monitoring venkat_k Programming 4 12-24-2008 04:45 AM


All times are GMT -5. The time now is 08:26 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration