Just had an idea (if the prob comes back); could be a power glitch during the backup.
That could maybe cause a sudden corruption of an open file and could affect any disk at all.

Again, just speculation on my part

Do please keep this thread updated; I'm intrigued now ...

Why is this thread marked as solved? Sounds like an unsolved issue to me.

The rsync might hang in a call to select() (which strace -p <pid> should show you). You should then figure out which file descriptor it is trying to write to or read from. That might give a clue on why it is hanging.

For rsyncs running headless (e. g. via cron) you should redirect all of its output (stdout + stderr) to /dev/null, otherwise the caller of rsync (e. g. cron) might try to gather its output in some too-small buffer in order to, for instance, send it via mail to some user. On buffer full, this might hang.

I marked it as solved because the problem went away as soon as I excluded /var/log/journal from the transfer. While the underlying cause isn't solved, the symptoms are, and I moved on. It hasn't hung again in the year and a half since making that change.

The output of rsync was already being redirected to a log file so I could keep track of where the transfer was failing, so cron's internal buffer wasn't an issue.

I would be interested in a thorough solution because we experience a similar situation here.

When investigating, we found out that there are three rsync processes which are waiting for each other in a cycle, each hanging in a select(), waiting for a local socket which is connected to the next. This sounds rather uncanny and weird. (A self-made deadlock?)

We have not yet investigated which current file might be involved because it didn't seem like the input was the source of the problem, and stdout and stderr were currently just thrown away. But the ACL might be part of the problem with us, too, as our nfs is somehow providing these (but this is not my field of expertise, so I lack the basics to have a clear idea).

This is a very old thread, but this may help someone else. I've encountered the same thing, with rsync getting stuck on the select system call at around the same file everytime. The only thing to note in my case is that there were a lot of failed chgrp operations due to the -a flag and rsync trying to change the group on the destination side and failing due to lack of permissions.

In this case removing the -a flag and using the equivalent (from the man page) and dropping the -g stops the error messages from appearing and avoids the problem with stopping part way through the sync.

Piling onto the "very old thread but may help someone else"; I was seeing the same problem with rsync hanging while trying to backup a large NTFS volume over NFS to an ext4 filesystem. I was erroneously using the -X option on rsync (not supported over NFS) and getting tons of errors of the form:

Removing the -X option eliminated all the errors and resolved the issue.

I suspect there may be some kind of bug in rsync that can cause it to hang indefinitely when it hits failures in lsetxattr/chgrp/ACL.