LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 04-29-2021, 07:53 AM   #1
HTop
Member
 
Registered: Mar 2019
Posts: 40

Rep: Reputation: Disabled
Rename files and folders to remove forbidden chars


Hello,
I need a bash (or python) script to rename files and folders (and subfolders, subfiles...) in order to remove chars that are not allowed on Windows NTFS Filesystem (see https://gist.github.com/doctaphred/d...-filenames-txt) that should be these:

<>:"/\|?*

Linux allows to use virtually every char (except from slash and NULL, I think) but on Windows chars like semicolon, colon and some kind of quote are infeasible.
I would like to replace Windows forbidden chars with hypen (or underscore, empty space...no matter).

Can you suggest some script?

Regards,
HTop
 
Old 04-29-2021, 08:10 AM   #2
TenTenths
Senior Member
 
Registered: Aug 2011
Location: Dublin
Distribution: Centos 5 / 6 / 7
Posts: 3,185

Rep: Reputation: 1377Reputation: 1377Reputation: 1377Reputation: 1377Reputation: 1377Reputation: 1377Reputation: 1377Reputation: 1377Reputation: 1377Reputation: 1377
I did some work on this a good few years ago, you might find this of interest:

https://centos.tips/fixing-troublesome-filenames/
 
Old 04-29-2021, 08:17 AM   #3
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,617
Blog Entries: 3

Rep: Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871
If you have the rename utility, preferably the perl version, then you can do find and replace on file names using patterns.

Code:
rename -n 's/[\<\>\:\"\/\\\|\?\*]/-/g' ./*
The perl version will not only take any perl regular expression, but even any general perl expression so variables and calculations are possible.
 
Old 04-29-2021, 09:37 AM   #4
teckk
Senior Member
 
Registered: Oct 2004
Distribution: FreeBSD Arch
Posts: 3,520

Rep: Reputation: 1095Reputation: 1095Reputation: 1095Reputation: 1095Reputation: 1095Reputation: 1095Reputation: 1095Reputation: 1095
Example:
Code:
a='file$! @#%^?<abc.txt'

echo "$a" | sed -e 's/[^A-Za-z0-9._-]//g'
fileabc.txt

echo ${a//[^A-Za-z0-9._-]/}
fileabc.txt
Edit:
Code:
b='file<>:"/\|?*.txt'
echo ${b//[^A-Za-z0-9._-]/}
file.txt

Last edited by teckk; 04-29-2021 at 09:40 AM.
 
Old 04-29-2021, 01:39 PM   #5
HTop
Member
 
Registered: Mar 2019
Posts: 40

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
If you have the rename utility, preferably the perl version, then you can do find and replace on file names using patterns.

Code:
rename -n 's/[\<\>\:\"\/\\\|\?\*]/-/g' ./*
The perl version will not only take any perl regular expression, but even any general perl expression so variables and calculations are possible.
It does not work.
I created a file called bi-"touch.txt and frenzy:.txt on a subfolder then I launched the command and the files remained bi-"touch.txt and frenzy:.txt

rename --version states that rename comes from util-linux 2.36.1


I also tried the centos.tips solution but it's not recusive.

Last edited by HTop; 04-29-2021 at 01:48 PM.
 
Old 04-29-2021, 01:52 PM   #6
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,617
Blog Entries: 3

Rep: Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871Reputation: 2871
Quote:
Originally Posted by HTop View Post
... and the files remained bi-"touch.txt and frenzy:.txt.
Then it worked as intended: You have received a defanged example and need to skim the manual page to see how to activated it. Pay attention to the options' descriptions.
 
Old 04-29-2021, 01:56 PM   #7
HTop
Member
 
Registered: Mar 2019
Posts: 40

Original Poster
Rep: Reputation: Disabled
I found a solution replacing ls with find.

Code:
find | while read -r FILE ; do
  mv -u "${FILE}" $(echo "${FILE}" | tr -d '\"[{}(),\!]' | tr ':' '_' | sed 's/__/_/g') 2>/dev/null
done
 
Old 04-30-2021, 12:31 AM   #8
MadeInGermany
Senior Member
 
Registered: Dec 2011
Location: Simplicity
Posts: 1,745

Rep: Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791Reputation: 791
According to post #1 you want to delete
Code:
<>:"/\|?*[]
The following replaces : and optional leading or trailing _ with a _
Code:
mv -u "${FILE}" "$(printf "%s\n" "${FILE}" | tr -d '<>"/\|?*[]' | sed 's/_\{0,1\}:_\{0,1\}/_/g')" 2>/dev/null
In the shell a backslash within single quotes is literal.
echo has problems if a filename looks like an option ("-ne"), use printf instead.
The $( ) must be in double quotes to avoid further expansion (a space causes a word split)
 
Old 04-30-2021, 12:42 AM   #9
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 16,510

Rep: Reputation: 5552Reputation: 5552Reputation: 5552Reputation: 5552Reputation: 5552Reputation: 5552Reputation: 5552Reputation: 5552Reputation: 5552Reputation: 5552Reputation: 5552
from one side it is really good (OP found a solution) from another side it is not really good the advices were not taken into account:
First of all the perl version of rename can do that job, there is no need to use find, mv, tr, sed ....
But if you wish to use find:
Code:
find | while read -r FILE ; do
   new_name="${FILE//[^A-Za-z0-9._-]/_}"
   [[ $FILE == $new_name ]] || mv "${FILE}" "${new_name}"
done
would be still much better (although that is not exactly the same).
 
Old 04-30-2021, 06:53 AM   #10
boughtonp
Senior Member
 
Registered: Feb 2007
Location: UK
Distribution: Debian
Posts: 1,361

Rep: Reputation: 1136Reputation: 1136Reputation: 1136Reputation: 1136Reputation: 1136Reputation: 1136Reputation: 1136Reputation: 1136Reputation: 1136
Quote:
Originally Posted by MadeInGermany View Post
According to post #1 you want to delete
Code:
<>:"/\|?*[]
According to post #1 they want to delete "<>:"/\|?*" - there are no brackets in that list, and there does not need to be!

In addition to those nine characters, there are several specific filenames which are not permitted, along with other restrictions (like not ending with a dot).

There is code which appears to handle all this already in the link in post #1 - it would simply need to be translated from JavaScript to a suitable CLI-friendly language.

The main reason this post doesn't include such a translation is because at some point in the past three decades someone is certain to have written such a function already, and they probably also made it robust with features such as collision detection - not just between incompatibly named files, but between acceptable files which happen to reside in directories that need renaming, not to mention between files/directories that only have acceptable characters but differ in case.

It seems a waste of time playing with scripts rather than identifying that established and tested solution.

 
Old 04-30-2021, 09:19 AM   #11
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 2,326

Rep: Reputation: Disabled
Quote:
Originally Posted by boughtonp View Post
someone is certain to have written such a function already
Well, mtools can do this, but I was hesitating to mention it because it will be done only when copying/moving to a FAT/VFAT filesystem.
Code:
$ touch prn pr:n;mcopy -Da pr* /tmp;ls /tmp/pr*
/tmp/pr:n  /tmp/prn
$ mkdosfs -C /tmp/fat.img 160
mkfs.fat 4.1 (2017-01-24)
$ mcopy -Da -i /tmp/fat.img pr* ::
$ mdir -i /tmp/fat.img
 Volume in drive : has no label
 Volume Serial Number is 11FC-260A
Directory for ::/

pr_n-1               0 2021-04-30  16:16
prn-1                0 2021-04-30  16:16
        2 files                   0 bytes
                            145 408 bytes free

Last edited by shruggy; 04-30-2021 at 09:29 AM.
 
Old 05-01-2021, 12:38 AM   #12
Michael Uplawski
Senior Member
 
Registered: Dec 2015
Location: Apples
Distribution: Apple-selling shops, markets and direct marketing
Posts: 1,143
Blog Entries: 29

Rep: Reputation: 652Reputation: 652Reputation: 652Reputation: 652Reputation: 652Reputation: 652
Quote:
Originally Posted by HTop View Post
It does not work.
It does work.
The '-n' option will not rename but only shows the “beforeafter” situations. Omit '-n' and say '-f' (force):

Code:
rasalind@dna:/tmp/rename_test$ ls
'bi-"touch.txt'
rosalind@dna:/tmp/rename_test$ `rename -f 's/[\<\>\:\"\/\\\|\?\*]/-/g' bi-\"touch.txt`
rosalind@dna:/tmp/rename_test$ ls
bi--touch.txt
  1. Perl
  2. man rename

Last edited by Michael Uplawski; 05-01-2021 at 12:39 AM.
 
Old 05-01-2021, 02:39 AM   #13
shruggy
Senior Member
 
Registered: Mar 2020
Posts: 2,326

Rep: Reputation: Disabled
Quote:
Originally Posted by Michael Uplawski View Post
It does work.
But not with this rename:
Quote:
Originally Posted by HTop View Post
rename --version states that rename comes from util-linux 2.36.1
@OP. If you're on CentOS(* then you're out of luck: Perl there doesn't provide the rename script. Rebuild the package prename from Fedora sources or install File::Rename from CPAN.
______
* I doubt this is the case as CentOS 8 Stream currently only has util-linux 2.32.1; util-linux 2.36.1 looks more like Ubuntu 21.04 Hirsute, Debian 11 Bullseye, Mageia 8 or Fedora 33.

Last edited by shruggy; 05-02-2021 at 06:09 AM.
 
Old 05-01-2021, 05:56 AM   #14
Michael Uplawski
Senior Member
 
Registered: Dec 2015
Location: Apples
Distribution: Apple-selling shops, markets and direct marketing
Posts: 1,143
Blog Entries: 29

Rep: Reputation: 652Reputation: 652Reputation: 652Reputation: 652Reputation: 652Reputation: 652
Quote:
Originally Posted by shruggy View Post
But not with this rename
That is why I wrote “1. Perl” but I may have skipped a few words, as Turbocapitalist had already done the work.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: GUI To Batch Rename Files On Linux With Exif And Music Tags Support: Inviska Rename LXer Syndicated Linux News 0 05-25-2019 12:07 PM
[SOLVED] How to remove all hidden directories and folders, and only hidden directories and folders rm_-rf_windows Linux - General 5 04-12-2016 07:28 AM
Shell Script to compare folders,Sub-Folders and Sub-Sub-Folders unix_72427 Programming 8 08-08-2012 02:51 PM
Need bash script to remove spaces and non alpha chars from folders/ files ne0shell Programming 6 06-22-2012 11:10 AM
[SOLVED] sed: replace regexp w/ variable #s of chars with the same # of (diff.) chars? kmkocot Linux - Newbie 6 11-18-2011 05:36 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 12:58 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration