read and write permissions, directories versus files

Manganus · 06-23-2017, 08:19 AM

As long as I can remember, I've been quite confused with regard to those pesky permission flags - particularly when it comes to directories.

Now, I have a specific need, and would need dedicated help.
Thanks in advance!

I'm writing a ("small") script that rebuilds the directory structure.

My attempt is "rule based":
[Rules of the kind:

direct old windows backups in one direction,
direct music directories in another direcion, and
direct picture-directories without music to yet another sub-tree,
let certain criteria make files become totally forgotten (like, for instance, obsolete executables)
identify frequently changed files, like databases, and adopt the script's treatment of them
replace certain copies with hard links, others with soft links, others not at all.]

This rebuilding is not done quickly. And my idea is that new, or alternative, rules should be applied step by step, tentatively, until the user is happy with the result. I think of the process as one of several weeks...

Until then, I want the new directory structure and its files to be visible for all, but protected against any changes.

The script is intended to be run without root privileges, i.e. as a user like myself. That means that I need some kind of "protection" also against the same user who runs the script.

(Well, that's not entirely true. Let's rather say: The script may be run by root, to handle those directories and/or files the actual user aren't allowed to handle. But I try to avoid sudo whenever possible.)

The need for this script may return at a time when I've forgotten most about it and am to be considered no wiser than any other user.

Edit: I write in python 3.4. Much of the coding is done by now. But I didn't foresee this need to protect the new structure until it's finished, as I didn't exactly realize how many days/weeks the reconstruction may need. I'm building the new directory structure on the same file system, and hard link the files. Copying the files would require both time and space. That's to be avoided.

Question:
How do I best set the file access rights for these new directories, until the process is ready, and the old directory structure can be unlinked?

The actual files may be owned by anyone, and linked into the new directories. I hope I don't have to os.chmod the individual files, but could os.mkdir(path, mode=?) the new structure's directories and then link (os.link(src, dst)) from the old directory structure to the new directories.

I'm confused also with regard to hard-links and file permissions. Maybe I'll have to go with soft-links until the process is finished? Or maybe it ends up with running the script as root anyway.

Probably, this is easy to understand once you've understood it. But...

rtmistler · 06-23-2017, 09:19 AM

There are a lot of listed requirements here.

Permissions and ownership are things which root can change and I find it useful to have data files be owned by the user who has them in their possession, however as I've said, ownership can be changed at any time by root.

Links can be maintained, however if the places where the links resolve too no longer exist, then the link is no longer useful. Meanwhile links can be followed and the actual file can be accessed, it depends how you set up your copy flags.

I would recommend you start your script to accomplish some few fundamental things, test it well, and grow it over time. Especially considering that you cite that you expect you may move away from the script, forget what it does, or how it does things, and then revisit it much later.

The suggestions there are to maintain revisions of it. Keep the older versions, name then differently, or use the .1, .2 added extensions like logrotate may use to coin your former versions of the script which keeping your latest copy active with the .sh or other extension relevant to the language you're using. The other suggestion is to include verbose comments in your script to explain what you did and why you did it. I've done this after having learned similar, which is that I make something, use it for years, and then one day need to modify it, to learn that I've long forgotten how I did it in the first place, or even where I put the script. Minor recovery and re-learning actions have occurred on my part too.

For when you have something specific where you have questions, feel free to post it or some relevant parts of your script for comments if you wish some suggestions.

BW-userx · 06-23-2017, 09:39 AM

If you understand who ever is creating the dir / files they get that users ownership and everything attracted to it. Unless changed by the owner of same said dir and files.

so if you run a script that is creating dir and files being root user then root will own everything it created, unless you add the needed code to change it.

Hard links vs soft links, soft links is almost always recommended as a use over hard links. Hard links make another copy of the same thing in a different place with a link pointing to it. Whereas the soft link does not.

taking your needs you need to sit down and take each need and analyze it. Coming up with the solution in how to preform said operation(s) to get desired results.

For example:
I want the new directory structure and its files to be visible for all, but protected against any changes.

This would require what permissions attached to each file and the directory they are stored in?
read but not write.

how does one accomplish this on the command line?

Keeping in mind the first rule, whoever creates the dir and or file has ownership: with that ownership they can change permissions on each dir and file.

if root owns it then a reg user can do nothing to it to change its permissions. if Joe runs the script then bob comes along and runs that same script to help finish it using his login. Then if it by change does not get finished. So Joe has to come back and run it again in his login.

Then what has taken place to the permissions of all of the dir and files that have been processed?

they have mixed ownership.

....

You'll need to keep that in mind and perhaps find a work around so Joe and Bob can run the script and the ownership and permissions you want attached to them to be kept properly so you can achieve you goal of "files to be visible for all, but protected against any changes."

if you remove all write permissions to a file for everyone including root, then what have you done to that file? Write permission govern the ability to do what to a file?

This is doable in a script, but it will take time putting it piece by piece together until you get a complete script.

Quote:

direct old windows backups in one direction,
direct music directories in another direction, and

a slice of my one script where I am doing this. putting copies in other places. Defined as redirecting files to other places for safe keeping.

In the header of the BASH scrip I use this method for easier changing of the main directories if I decide to change them. This eliminates having to go through the entire script and change each instance of when that path is used, and it cuts down on errors if one was to miss one of them that needed to be changed.

Code:

count=0 max=0

#uses one source

working_dir=/media/data/Music-1

#sets the destinations 

copy_to1=/run/media/userx/3TB-External/Music

copy_to2=/run/media/userx/750hybrid/Music

move_to=/media/data/Music

move_old_flac=/run/media/userx/3TB-External/OldFlac

#for the re sampling of mp3/flac files 
#to get the new file and process it. 
#because lame and flac create the new file in the 
#same dir the script is being ran.

script_dir=/home/userx/scripts/production

# for sub-string checking

GirlSubString="Reference Of Female"

max="$(find "$working_dir" -type f \( -iname "*.mp3" -o -name "*.MP3" -o -name "*.flac" \) | wc -l )"

Quote:

direct picture-directories without music to yet another sub-tree,

If statements checking to see if true then to direct dir with pictures somewhere else.
My line of thought would be this.
you are going to have to keep a dir and look inside of it for pictures if yes, then look to see if music, if yes - skip? - if no then process it. move to next directory.

Code:

while read FILENAME
do

f=$FILENAME
path=${f%/*}
xfile=${f##*/}
title=${xfile%.*}
ext=${xfile##*.}

if [[ "$ext" != 'mp3' ]] ; then
do whatever you need to do here
fi

taking into consideration of everything you're looking for and not wanting. that one needs work still. You may have to use two loops with find /path -maxdepth (x) {and/or} -mindepth (x) ... to keep the dir that meets the requirements so you can then act on that dir alone before moving on to the next dir to check for matching criteria.

Just some quick sniplets without the background how I got some of the values.

Code:

      elif [[ "$ALBUM" =~ "$Gal1SubString" ]] ; then
      {
	copyIt1="$copy_to1/$ALBUM"
	copyIt="$copy_to2/$ALBUM"
	moveIt="$move_to/$ALBUM"
	mkdir -pv "$copyIt1"
	mkdir -pv "$copyIt2"
	mkdir -pv "$moveIt"
	cp -rv "$FILENAME" "$copyIt1/$NewFile"
	cp -rv "$FILENAME" "$copyIt2/$NewFile"
	mv -v "$FILENAME" "$moveIt/$NewFile"
      } elif [[ "$ALBUM" =~ "$Gal2SubString" ]] ; then

using the elif [ true ] .. elif [ true ] for each condition

Quote:

let certain criteria make files become totally forgotten (like, for instance, obsolete executable)

Just skip the ones you do not want by not looking for them so you can process the ones you want then they will be moved and not the ones yo udo not want. they are kept in place for later deletion. perhaps even within the script when it has finished processing a directory before moving on to the next one.

Quote:

identify frequently changed files, like databases, and adopt the script's treatment of them
replace certain copies with hard links, others with soft links, others not at all.

that one is going to take a bit of coding. but can be done.

I do not want to put too much in here that might just confuse you more. you need to take it one step at a time. create test dirs and the data types within it for each condition you want to deal with, then use your script to test them until you get one to work then move on to the next one.

rknichols · 06-23-2017, 09:48 AM

Quote:

Originally Posted by BW-userx

Hard links make another copy of the same thing in a different place with a link pointing to it.

Absolutely not! Every hard link is just a directory entry that links to the same file. There is just one copy of the file.

BW-userx · 06-23-2017, 10:08 AM

Quote:

Originally Posted by rknichols

Absolutely not! Every hard link is just a directory entry that links to the same file. There is just one copy of the file.

thank you for correcting me?
googled to check myself on that one.

Code:

Hard links can't cross file systems. Soft links do. you know immediately where a symbolic link points to
while with hard links, you need to explore the whole file system to find files sharing the same inode.
hard-links cannot point to directories.

http://www.maxi-pedia.com/difference...+and+hard+link

Thanks for correcting me.
I just use soft links from what I've read on it, so that is what I wrote on it.
but now I am finding different information on this. go figure.

(let me remove that info from my other post]

Beryllos · 06-23-2017, 10:16 AM

Quote:

Originally Posted by Manganus

Until then, I want the new directory structure and its files to be visible for all, but protected against any changes.

A convenient way to do this is by bind mount, which is mounting a first directory (with all its contents) onto a second directory, and configuring that mount readonly. You probably have to be root to do this.

Read man mount, particularly the section on bind mounts.

One of the beautiful things about this method is that when the time comes to open up the permissions, all you need to do is remove the readonly property of the second directory, or just give everyone access to the first directory.

Here is a little example:

Prior to mount, user beryllos sees this:

Code:

$ ls -l *
temp1:
total 8
drwxr-xr-x 2 beryllos beryllos 4096 Jun 23 09:29 dir1
-rw-r--r-- 1 beryllos beryllos   29 Jun 23 10:01 file1

temp2:
total 0

Then root does this:

Code:

# mount --bind temp1 temp2
# mount -o remount,ro,bind temp2

After that, user beryllos sees this:

Code:

$ ls -l *
temp1:
total 8
drwxr-xr-x 2 beryllos beryllos 4096 Jun 23 09:29 dir1
-rw-r--r-- 1 beryllos beryllos   29 Jun 23 10:01 file1

temp2:
total 8
drwxr-xr-x 2 beryllos beryllos 4096 Jun 23 09:29 dir1
-rw-r--r-- 1 beryllos beryllos   29 Jun 23 10:01 file1

The permissions shown in that listing would suggest that beryllos can delete or modify the contents of temp2, but he can't, because it is mounted as a readonly filesystem:

Code:

$ rm temp2/file1
rm: cannot remove `temp2/file1': Read-only file system
$ rm -r temp2/dir1
rm: cannot remove `temp2/dir1': Read-only file system
$ echo hello > temp2/file2
bash: temp2/file2: Read-only file system
$ chmod o+w temp2/file1
chmod: changing permissions of `temp2/file1': Read-only file system

Beryllos · 06-23-2017, 10:26 AM

Quote:

Originally Posted by Manganus

Edit: I write in python 3.4. Much of the coding is done by now. But I didn't foresee this need to protect the new structure until it's finished, as I didn't exactly realize how many days/weeks the reconstruction may need. I'm building the new directory structure on the same file system, and hard link the files. Copying the files would require both time and space. That's to be avoided.

Question:
How do I best set the file access rights for these new directories, until the process is ready, and the old directory structure can be unlinked?

I think this addition to your original post is also covered by my previous post.

Manganus · 06-23-2017, 10:54 AM

Thank you, all four of you!

Quote:

Originally Posted by BW-userx

if Joe runs the script then bob comes along and runs that same script to help finish it using his login. Then if it by change does not get finished. So Joe has to come back and run it again in his login.

Then what has taken place to the permissions of all of the dir and files that have been processed?

they have mixed ownership.

It's embarrassing.
I hadn't thought of this.

(I guess the directories have to be "executable" and readable by all, regardless of who owns them. After all, it's only temporarily. And the final step must be run by root anyway. Then root has the power to chown and chmod the directories as much as needed.

But still, it was a hole in my thinking. Really!)

BW-userx · 06-23-2017, 11:01 AM

Quote:

Originally Posted by Manganus

Thank you, all four of you!

It's embarrassing.
I hadn't thought of this.

(I guess the directories have to be "executable" and readable by all, regardless of who owns them. After all, it's only temporarily. And the final step must be run by root anyway. Then root has the power to chown and chmod the directories as much as needed.

But still, it was a hole in my thinking. Really!)

perhaps but you've already figured out a solution. so that's a good thing.

just, I'd keep in mind if this is going across the system outside of one user /home/dir structure then you'll have to make allowances to check where it is at and then add owner / permissions accordingly.

off top of head if python uses sub-string matching, then you could just use key words against the path inside of variable then run the code to change ownership

Code:

search_path=/
sub-string-sys1=/usr/bin
sub-string-sys2=/etc

#the FILENAME value will be the entire path and file within that variabe

while read FILENAME
do
{
if [[ "$FILENAME" =~ "$sub-string-sys1" ]] ; then
{
 sudo chown user:group "$FILENAME"
} elif [[ "$FILENAME" =~ "$sub-string-sys2" ]] ; then
{
  sudo chown user:group "$FILENAME"

}
fi
done< <(find "$search_path" -type d (or f) )

something like that perhaps.

rknichols · 06-23-2017, 11:04 AM

Quote:

Originally Posted by BW-userx

Code:

Hard links can't cross file systems. Soft links do. you know immediately where a symbolic link points to
while with hard links, you need to explore the whole file system to find files sharing the same inode

And with soft links you have to explore the entire universe to find all the soft links that point to the same file. (Some of those soft links could be on a flash drive in your desk drawer, safe deposit box, etc.)

Manganus · 06-23-2017, 11:18 AM

Quote:

Originally Posted by BW-userx

Hard links can't cross file systems. Soft links do. you know immediately where a symbolic link points to while with hard links, you need to explore the whole file system to find files sharing the same inode.

That's no disadvantage.

There is no need to trace any file paths from the new directory structure to the old one. When the process is finalized, then the old directory structure is to be unlinked (recursively, of course).

If soft links were used, then each and every of them would have to be replaced by a hard link to the file during that finalization. That's doable, but are there any advantages?

BW-userx · 06-23-2017, 11:28 AM

Quote:

Originally Posted by Manganus

That's no disadvantage.

There is no need to trace any file paths from the new directory structure to the old one. When the process is finalized, then the old directory structure is to be unlinked (recursively, of course).

If soft links were used, then each and every of them would have to be replaced by a hard link to the file during that finalization. That's doable, but are there any advantages?

if you are asking me personally. I have never dealt with links of either kind on such a magnitude. within and outside of a script to remove same said type link then take the original file and move it then reestablish a link to it where ever it needs to be.

Though it sounds like an easy matter. but you are using python 3.4 so I do not know what provisions that it has to check for a link hard or soft then remove it, then move that original file, then reestablish the same type of link back on to it.

that to me seems to be all that is necessary to be done to get it to work.

if link, what type, remove link, move file, reestablish same type link to file from the new place its at now linked to where it was link to before.

rtmistler · 06-23-2017, 11:48 AM

To me, links are links and I never seem to care if they are hard vs. soft.

My concerns end at the point where I have determined if I wish to preserve the structure because I'm copying the tree versus whether or not I want the actual file.

As far as Python goes, ... wouldn't the copying or moving of files, links, trees, etc be instead a command line call over a Python call? Or rather, doesn't the Python os.function resolve directly to the OS call? What's unclear to me are how the command options are applied. I do not see how the generic Python os functions accept and pass options such as --preserve or --force to the operating system.

scasey · 06-23-2017, 01:50 PM

I don't have much to add to the thread as it's developed so far, but while reading the original post, I immediately thought of rsync, which should be able to do everything required with a minimum of effort...mostly scripting the running of rsync to fit the specific requirements in the bullet list. rsync will preserve the permissions and timestamps on the source files, make a true copy of everything (including links), and put the copies where directed, preserving the directory structure otherwise. Just a thought.

PS
My comment doesn't address the question of understanding "the permission flags" on files and directories.

AwesomeMachine · 06-23-2017, 03:53 PM

You must use the -a switch with rsync to preserve the attributes, timestamps, etc. A good script to start with might be ftpsync from the Debian.org website. It is used to mirror the Debian archive (tens of thousands of files, directories and links). But it can be easily modified to suit ones own requirements. It has some nice features.