Did you try with -u option (update)? From the man page,
This forces rsync to skip any files which exist on the destina-
tion and have a modified time that is newer than the source
file. (If an existing destination file has a modify time equal
to the source file's, it will be updated if the sizes are dif-
Another one which you might be interested in is the --ignore-existing, if you're not into updating at all.