shorten long paths in deeply nested directory structure
 

I would like to shorten some long paths in a deeply nested directory structure. The directory structure itself starts several levels down from root.

So let's say i have a directory structure with path depths varying from 1 to 15 (counting the directory structure as level 1).

I want to find all paths that go to depth of at least 5.

& then snip each path at level 5 and move it (along with its subdirectories) to level 2.

The net result of all this will have been to reduce maxdepth by 3 (i.e., from level 15 down to 12).

I don't want to flatten the directory structure -- that's more flattening than I need. I just need to flatten it a little so that it doesn't create errors in certain operating systems doing certain procedures.

I'd also like to include echoing each step to the standard output to make sure I'm not doing this on the entire file system.

Is there a way to do this?
Thanks.
EVL

Is a Perl program acceptable to you?

Or -- lets talk about the "errors in certain operating systems doing certain procedures" -- perhaps flattening the directory structure is not the best solution to the problem.

Perl Program to shorten directory paths
 

Yes, absolutely .. As long as I know how to execute it .. not sure I've ever run a Perl program.

flatten directory structure vs errors in OS
 

Great idea .. how do I translate that into a solution?

BTW, my printout of the directory tree suggests nesting up to 20 levels deep.

I'd probly wanna flatten it to maybe 10 levels -- even if it's an OS problem I can circumvent.

Which procedures of which OS might be problematic with deeply nested directories?

Thanks much for your help.

Eric

So if you have a directory /path/to/some/of/my/really/cool/computer/files you want to move of/my/really/cool/computer/files to /path/to/cool/computer/files? What if there's a directory with that name already there? Merge them, or rename it by adding 1, 2, 3 etc.?

I'm basically suggesting that the solution to your problems might not be to flatten the directory structure, but lie elsewhere. I could be wrong, of course, or you may not care to investigate another solution.

If you do, please explain what problems you are having that you're attributing to the depth of your directory structure. That is: Which procedures are causing errors because of the long path names? What are you trying to do that not working because of long path names?

If not, please excuse me...

If there's a directory with that name already there, rename with incrementing number.
 

Good point .. Probly rename with incrementing number.

Thanks.

Eric

what problems you are having that you're attributing to the depth of your directory structure
 

[QUOTE=scasey;5729562]I'm basically suggesting that the solution to your problems might not be to flatten the directory structure, but lie elsewhere. I could be wrong, of course, or you may not care to investigate another solution.

If you do, please explain what problems you are having that you're attributing to the depth of your directory structure. That is: Which procedures are causing errors because of the long path names? What are you trying to do that not working because of long path names?


I SEE YOUR POINT .. One thing is copying for the sake of backup. Searching for files by content.

But even more, just I never know what I'm going to try to do with the files in future -- or on which system (I use mac/win/linux/android). So I just kinda wanna know that whatever I try to do in future will go without a hitch -- at least not cuz excessive nesting depth.

BTW -- the whole thing right now is on exfat file system hooked up to mac (easy to reconnect to linux, though)

Thank you.

Eric

Ahh yes. I have experienced those kinds of issues, but the problem has been the number of files the command tried to return, not the depth of the file structure. Flattening the structure won't reduce the number of files. Can you show us what happens?

problem is number of files -- not depth of nesting
 

I'm going to try to remember or duplicate what happened & post it then .... For now, I seem to recall copying all files from a failing drive, but all files didn't copy, & I believe nesting may have been the issue. I'd like to toss the old drive, but not till all files have copied.

For now, I just have a general memory of something going wrong when data was deeply nested -- but not when i reduced the nesting.
I don't recall how many files there were in either case.

I'd be curious if there's a max number of files one can use in one operation on mac/linux/win etc .... If there is such limit, it would be a good thing to know.

I've heard of a limit on path length, i.e., number of characters. But the previous 256 (+ or -) has been transcended on windows & doesn't exist on mac or linux, I believe.

Thanks for your continued feedback.

Why not use a symbolic link?

e.g. create a directory at level 5 called 'some-links' which points to a some deeply nested "parent directory"
It will ease your navigation through loads of files.

For example:-
then,

Let's talk about backups. ;)

Certainly. I suspect a got "too many arguments" sometime in the past, but (as you know), that's probably not the best way to make a backup.

I use rsnapshot, which uses rsync, to make off-site backups of my production server and same-site backups of my backup server.

One of the problems for backups is that it is easier to take a backup than to do a proper restore!

rsync has quite a complex set of possible parameters...

I use backup2l which produces daily tar.gz files, so in a dire emergency, I could do a "manual" restore; even if I'd lost all details of the backup parameters.

Appending incrementing number to directory name.
 

Laserbeak, thanks for the reply, and please excuse my delay in responding.

Yes, this is exactly the idea. I figured out (I think) how to "transplant" all directories at level 10 to level 3 (or whatever) -- with a simple mv.

But the problemo, as you observe, is name duplicates. I'd prefer to rename by appending incrementing number, though merging would probably work, too.

I've been using this, which unfortunately does not address duplicates:

CODE
find -maxdepth 10 -mindepth 10 -type d in ./* -exec echo mv -i

BACK TO TEXT
If you can suggest how to modify directory names to avoid collisions, that would help a lot.

Thanks much for your help with this.

Eric

THANK YOU ALL FOR YOUR HELPFUL REPLIES.

I actually wrote a more detailed reply for each of your various approaches, but unfortunately it got zapped by accident, and it's too late for me to redo.

I'm going with appending incrementing number to avoid directory name collisions.

I might also try (if that doesn't work) combining path and filename into one filename.

Thank you again for your help, and please excuse my absence for the last month.

Eric

Hi Jeremy . . just getting around to seeing this old post.
I agree symbolic links present an effective solution. However, I've had links broken in the past -- not sure how it happened. I'm afraid it will happen again. I'd like a solution that doesn't require me to remember there are symbolic links that can be broken.

Not sure if it was your suggestion, but appending incrementing number to directory name before moving it to a higher level seems like a good idea.

Nothing in particular -- it just kinda scares me to have way deeply nested directories . . sometimes as deep as 40 levels.
I'm afraid at some point in the future it might become an issue -- perhaps for some other OS, or some other backup system.

One of my favorite topics.
Right now I just do a manual backup every so often -- at risk of losing stuff since the previous backup.
I've tried rsync, but it doesn't copy all the files for some reason.

I've just recently learned how to file-share, and I'd like to set this up with some rsync-like backup solution.
Placing the backup on a separate system may be more secure.

Also, it slows the system file operations down -- sometimes it seems like it gets stuck.

blank post
 

...

Definitely searching operations are slowed down to the point of stopping, it seems.
Also, copying deeply nested directory structures results in incomplete copying -- I think.
It can be hard to tell when you don't know if the system has just slowed down or stopped altogether.

to Astrogeek
 

I wasn't sure if I could continue a thread from 5 years ago.
I figured it would be most likely seen if posted anew.
Thanks for letting me know.

I believe the "solution" that I used at the time of original post, 2017, was to move any directory at
-mindepth 15 -maxdepth 15
or something similar. But this posed 2 problems . .
1) It didn't check for duplicate names before moving to target directory.
Recently (5 years after OP) I learned how to append incrementing numbers in python.

2) It didn't determine a file's nesting depth. I can use the awkward -maxdepth & -mindepth from bash, guessing at the depth and then seeing what happens. This flattened only part of a huge directory.
More recently, I've researched how to find path length in python. This guy on reddit/learnpython describes my experience . .

Suppose I have a directory structure that includes /a/b/c/d/ee/FN1.ext -- and it also includes several other files and subdirectories with a path like /a/b/c/FN2.ext -- but no longer than that. In that case, the first one, /a/b/c/d/e/FN1.ext would be the longest -- of path length 5. (Or 6 if you count the file).
I'd like this output . .
Knowing that, I can then "prune the tree" by moving all files with a path length greater than Maxdepth to a target directory, appending incrementing number.
This may make the directory structure easier to search, back up, remove duplicates, and transfer to other file systems.
There's also a variation where the number returned is not the integer number of components in the directory path -- but rather the number of characters (alphanumeric, punctuation, delimiters) in the entire path. So the character count for
is 13.

Then I can meet NTFS limit of 4096 characters for a path.
I can use my script to move empty folders to a target folder. But instead of testing if folder is empty, I'd test for whether its length exceeds some limit.

Darn those "certain" (MS) operating systems! ;)

Anyway, this isn't difficult to do in Python, and based on your posting history, you're more than capable.

Use this:

https://docs.python.org/3/library/pathlib.html

THIS IS THE CODE THAT FINALLY WORKED. THANKS TO ALL