LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 09-09-2019, 03:52 PM   #1
SaintDanBert
Senior Member
 
Registered: Jan 2009
Location: "North Shore" Louisiana USA
Distribution: Mint-20.1 with Cinnamon
Posts: 1,767
Blog Entries: 3

Rep: Reputation: 108Reputation: 108
using rsync with files/folders using embedded blanks in names


How do I use rsync to read and write when the file and folder names use embedded blanks?

DISCLAIMER -- This issues is typographically awkward. I'll try my best but please bear with me.

I have a flock of external drives, SD cards, and thumb drives that were written using embedded blanks in file and folder names. I'm tasked to consolidate these files and folders onto a linux file server. A simple copy using:
Code:
prompt$  cp -av  "some\ Source"  "some\ Target"
One issue,
(or similar) does not seem to be good enough.

One tangle comes from the "volume name" or "volume label" that typically results in a mount point like
/media/username/some sort of string with blanks

Now when you add folder and file names you quickly run out of allowed characters on a command line in scripts or internal to utilities like rsync.

While running, rsync reports "I/O error writing..." for some blank-embedded files. It succeeds with others. The failing files can be moved manually using:
Code:
prompt$  cp -av "source/a file/that uses blanks" "target/a file/keeping blanks"
Thanks in advance,
~~~ 0;-Dan
 
Old 09-09-2019, 04:07 PM   #2
michaelk
Moderator
 
Registered: Aug 2002
Posts: 25,592

Rep: Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880
Myabe using the --protect-args option will fix space problem.
 
Old 09-10-2019, 02:28 AM   #3
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,258
Blog Entries: 3

Rep: Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713
I have a lot of files and directories with spaces in the names, but haven't seen that error before.
Can you show the exact parameters you are using with rsync?
 
1 members found this post helpful.
Old 09-10-2019, 05:36 AM   #4
michaelk
Moderator
 
Registered: Aug 2002
Posts: 25,592

Rep: Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880Reputation: 5880
Looks like I may have misinterpreted the problem.

I backup sub-directories with spaces in the path/files names like the following with no problems.

rsync -avxP /source/ /path/to/destination

However, I don't think the I/O error writing messages are attributed to just having spaces in the path file name but maybe filesystem errors.

The --protect-args option is used for remote rsync operations to keep the remote shell from interpreting spaces and special characters.
 
Old 09-12-2019, 02:51 PM   #5
SaintDanBert
Senior Member
 
Registered: Jan 2009
Location: "North Shore" Louisiana USA
Distribution: Mint-20.1 with Cinnamon
Posts: 1,767

Original Poster
Blog Entries: 3

Rep: Reputation: 108Reputation: 108
Quote:
Originally Posted by Turbocapitalist View Post
I have a lot of files and directories with spaces in the names, but haven't seen that error before.
Can you show the exact parameters you are using with rsync?
First, this is a local copy -- disk to disk on the same workstation. Okay, one is an internal (typ. source) and one is an external (typ. target)

I pull files from EXT4, without embedded blanks, without issue using:
Code:
prompt$  rsync -av /someSource/  /someTarget
Since its a local copy, I don't need or want compression.

I only get the errors when I have embedded blanks. I'll try --protect-args like *michaelk* suggested.

I have scanned and checked and tested both the source and target file systems and find no errors. When I'm doing this, I limit the workload so that the disk I/O gets most of the workstation (an 8GB, i5 Thinkpad™ laptop).

I really suspect that the embedded blanks and long path+name character counts are part of the troubles but I've no evidence ... I'll have to do a test and submit a bug report if I get proof.
 
Old 09-12-2019, 11:55 PM   #6
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 7,258
Blog Entries: 3

Rep: Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713Reputation: 3713
Thanks. Make sure you are quoting the paths or escaping the spaces.

This won't work:

Code:
rsync -av /some Source/with/a space or two/  /some Target/
but these will

Code:
rsync -av '/some Source/with/a space or two/'  '/some Target/'
rsync -av "/some Source/with/a space or two/"  "/some Target/"
rsync -av /some\ Source/with/a\ space\ or\ two/  /some\ Target/
There are a lot of shell guides and tutorials around which go into more detail on quoting and escaping.
 
Old 09-13-2019, 03:50 AM   #7
TenTenths
Senior Member
 
Registered: Aug 2011
Location: Dublin
Distribution: Centos 5 / 6 / 7
Posts: 3,461

Rep: Reputation: 1552Reputation: 1552Reputation: 1552Reputation: 1552Reputation: 1552Reputation: 1552Reputation: 1552Reputation: 1552Reputation: 1552Reputation: 1552Reputation: 1552
I came across a similar problem a number of years ago and decided to take the stance that spaces in filenames were just plain wrong!

Not sure if you'll find this useful or not: https://centos.tips/fixing-troublesome-filenames/
 
1 members found this post helpful.
Old 09-13-2019, 05:08 AM   #8
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,691

Rep: Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274Reputation: 7274
the i/o error means probably the filename was split into parts and rsync wanted to write to (or read from) an invalid device/path - which was only a part of the real filename.
You need to use quotations to protect filenames against this.
you can use rsync -v to have verbose output or --itemize-changes to get more info.
You would also need to give a better example, we can only guess without seeing the actual filenames/error messages.
 
1 members found this post helpful.
Old 09-30-2019, 04:49 PM   #9
SaintDanBert
Senior Member
 
Registered: Jan 2009
Location: "North Shore" Louisiana USA
Distribution: Mint-20.1 with Cinnamon
Posts: 1,767

Original Poster
Blog Entries: 3

Rep: Reputation: 108Reputation: 108
Quote:
Originally Posted by michaelk View Post
Looks like I may have misinterpreted the problem.
...
Let me restate things to clarify:
  • I start a local run using rsync. Either SOURCE or TARGET might be an internal HDD or an external usb-connected store or a media device.
  • Somewhere along the way, the SOURCE folder names or file names or both, start to use embedded blanks. The resulting path+filename is usually long, but nowhere near the PATH_MAX=4095 oro 4096 value.
  • rsync now must construct a path+filename to use against the TARGET
  • rsync reports "I/O error" while writing the target; there errors are consistently reported for blank-embedded path+filename on the TARGET
  • there are no reported I/O errors with the same operation where there are zero, none, nada, path+filename using embedded blanks

Thanks in advanced because I'm thorougly stumped,
~~~ 0;-/ Dan
 
Old 10-03-2019, 01:02 AM   #10
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,348

Rep: Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749
I agree with pan64, find a minimal 'broken' rsync example, use -v option and show us the exact cmd, and o/p you get.
This is one of those qns where we need to be as if we are sitting in front of the terminal ourselves.

I'd also agree with TenTenths: spaces considered harmful.
Although the Unix spec says spaces are allowed in filenames, practically every *nix cmd assumes spaces means param separation, unless you take special precautions.

If these are your files, rename them without spaces - you'll thank me later as they will continue to cause problems forever otherwise.
Even if they are not, you may be able to get away with temp renaming them, then processing them, then renaming back - I've had to to that when processing MSWin sourced files - it makes *nix cmds/tools etc 'just work'
 
Old 10-03-2019, 01:04 AM   #11
OstermanA
Member
 
Registered: Dec 2006
Location: Seattle, WA
Distribution: CentOS 8
Posts: 99

Rep: Reputation: 20
Do you need to preserve the spaces? If you are able/willing to replace them with underscores then you ought to be able to use something like this:
Code:
while read -d '' file; do mv "${file}" "${file// /_}"; done < <(find . -type f -print0)
If you can't just ditch the spaces, what happens if you type one of the files in directly as the source for rsync?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
How to recursively merge directories or folders into one, retaining folder names in file names stozi Linux - General 6 05-22-2019 01:16 AM
Shell Script to compare folders,Sub-Folders and Sub-Sub-Folders unix_72427 Programming 8 08-08-2012 02:51 PM
Is it possible to remove all files & folders in a directory without using their names LAPIII Linux - Newbie 8 02-02-2012 04:18 PM
[SOLVED] How to replace expressions with embedded blanks btacuso Linux - Newbie 6 03-16-2010 12:39 AM
SAMBA: display of share names is OK but files names are wrong superandrzej Linux - Software 5 02-02-2004 09:14 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 03:00 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration