S3cmd failed with Error Code 74 when uploading to Amazon S3

aikempshall · 06-04-2021, 11:38 AM

My backup to Amazon S3 started failing Tuesday morning. This is the command

Code:

 s3cmd sync --delete-removed --limit-rate=288K --multipart-chunk-size-mb=5 --no-progress LOCAL_DIR s3://BUCKET

Searched everywhere for an explanation/solution to this problem all I could come up with was a very terse

Quote:

An error occurred while doing I/O on some file.

After much playing with --verbose and --debug flags discovered that if I fed the data piecemeal directory by directory it would work. For some reason I then decided to double the very lowly transfer rate from 288K to 576K and it started to work.

Hope this helps someone.

aikempshall · 06-15-2021, 06:08 AM

Well it failed again today for same reason.

In the eight or more years of running s3cmd never had this error.

Seemed to fix it after increasing the transfer rate. In the past had problems with the chunk rate, in the past had to decrease it.

Investigation continues.

aikempshall · 08-02-2021, 05:53 AM

After more than a month with error free syncing with aws it failed again yesterday morning and again this morning

Quote:

Sun 1 Aug 09:59:55 BST 2021: I9033 - s3cmd sync --delete-removed --limit-rate=576K --multipart-chunk-size-mb=5 --no-progress /home/alex/FastStorage1/amazon_drive_encrypted/alex/Documents/ s3://mcmurchy1917-MyDocuments

Sun 1 Aug 10:06:14 BST 2021: E9030 - Failed sync with s3://mcmurchy1917-MyDocuments - errorcode 74

It always seems to fail at a point before it starts syncing any data. This morning it just sat there for 6 minutes silently doing something before failing. Presumably it's trying to work out what it needs to sync and then something timesout. On the last two failures it was trying to remove over 8,000 files from aws.

I've posted a query to the S3cmd mailing list. My post never got onto the list.

Anyway as I've said previously if I fed the data piecemeal directory by directory that's worked in the past when I get these 74 errors

I've also now found that if is use the awscli package, from SlackBuilds.org, instead of s3cmd package, from SlackBuilds.org, and change my call from

Code:

s3cmd sync $DRYRUN ${DEBUG_STRING} --delete-removed ${LIMITRATE} ${CHUNKSIZE} ${PROGRESS} myfolder s3://mybucket/myfolder

to

Code:

aws s3 sync --delete myfolder s3://mybucket/myfolder

It works!

I shall try again to post to the s3cmd forum mailing list, there's only been 3 postings this year which probably explains why my posting of a month or so ago got missed.

aikempshall · 08-21-2021, 03:22 AM

Posted a bug report with the people at s3cmd.

Also found another workaround for this "feature" which only raises an error when it's deleting loads of objects from a bucket. The ensures that the whole bucket isn't inadvertently emptied. There might be an undocumented limit of 1,000 objects and if it hits this it bombs out with the 74 error code.

My workaround was to -

run the s3cmd with the --dry-run flag
count the number of objects it wished to delete
verify the count is correct
add the flag --max-delete=${COUNT} where $COUNT is the number of objects to delete
run the s3cmd without the --dry-run flag