AWS S3 Selectively copy files based on timestamp

hz36t5 · 03-22-2021, 11:04 AM

I have struggled wit this for time time now.
My goal is to do the following:

Copy files that are newer than the indicated date in the CLI script from my aws bucket to my local linux server.

aws s3 ls --recursive s3://my bucket/ | awk '$1 > "2021-03-18 00:00:00" {print $0}' | aws s3 cp --recursive my bucket/{} localfolder

I keep receiving: [Errno 32] Broken pipe

Any ideas?

pan64 · 03-22-2021, 11:29 AM

missing xargs:

Code:

aws ... | awk ... | xargs aws ....

probably helps

Turbocapitalist · 03-22-2021, 11:30 AM

I'm not familiar with AWS but just as a guess you would need xargs for a pipe like that:

Code:

aws s3 ls --recursive s3://my bucket/ \
| awk '$1 > "2021-03-18 00:00:00" {print $0}' \
| xargs -I{} aws s3 cp --recursive "my bucket/{}" localfolder

What is preventing use of find and either the -newer or -newermt option?

computersavvy · 03-22-2021, 11:34 AM

Quote:

Originally Posted by hz36t5

I have struggled wit this for time time now.
My goal is to do the following:

Copy files that are newer than the indicated date in the CLI script from my aws bucket to my local linux server.

aws s3 ls --recursive s3://my bucket/ | awk '$1 > "2021-03-18 00:00:00" {print $0}' | aws s3 cp --recursive my bucket/{} localfolder

I keep receiving: [Errno 32] Broken pipe

Any ideas?

IIRC find will easily locate files based on timestamp with one of the tests options and the list can then be piped to cp. Have not used that for a while so the man page can help better than I can with syntax.

pan64 · 03-22-2021, 12:06 PM

Quote:

Originally Posted by Turbocapitalist

What is preventing use of find and either the -newer or -newermt option?

Quote:

Originally Posted by computersavvy

IIRC find will easily locate files based on timestamp with one of the tests options and the list can then be piped to cp. Have not used that for a while so the man page can help better than I can with syntax.

Sorry guys, I think aws is a remote service, so there is no direct access to that filesystem. That's why OP wanted first execute an [aws] ls, filter the result and finally [aws] cp the required files.
From the other hand I wanted to mention, the date comparison is probably not that simple, but you know if that works for you.

shruggy · 03-22-2021, 02:14 PM

The awk expression is wrong. What you probably want is

Code:

awk '$1>="2021-03-18"{print$NF}'

Instead of awk, here I'd rather filter the aws output with JMESPath using the --query option:

Code:

aws s3api list-objects-v2 --bucket 'my bucket' --query 'Contents[?LastModified>=`2021-03-18`].Key' --output text --no-paginate

Not sure this filter will work though. JMESPath spec states

Quote:

Ordering operators >, >=, <, <= are only valid for numbers.

OTOH, AWS CLI User Guide includes an example similar to above. Also see jmespath.py issues #124, #126 and jmespath.rb issues #47, #49.

What certainly will work in a JMESPath query is sorting by date:

Code:

aws s3api list-objects-v2 --bucket 'my bucket' --query 'sort_by(Contents, &LastModified)[].Key' --output text --no-paginate

At least, jq should always work as expected

Code:

aws s3api list-objects-v2 --bucket 'my bucket' --output json|
jq '.Contents[]|select(.LastModified>="2021-03-18").Key'

hz36t5 · 03-22-2021, 02:28 PM

Quote:

Originally Posted by Turbocapitalist

I'm not familiar with AWS but just as a guess you would need xargs for a pipe like that:

Code:

aws s3 ls --recursive s3://my bucket/ \
| awk '$1 > "2021-03-18 00:00:00" {print $0}' \
| xargs -I{} aws s3 cp --recursive "my bucket/{}" localfolder

What is preventing use of find and either the -newer or -newermt option?

Nothing. I didnt pursue that as an option. Would it be a more simple solution?

I am now running the following and not I get this message over and over until I abort:
fatal error: An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid

aws s3 ls --recursive s3://"my bucket" | awk '$1>="2021-03-01"{print$NF}'| xargs -I{} aws s3 cp --recursive "s3:"my bucket/{}" localfolder

hz36t5 · 03-23-2021, 09:11 AM

So I now run the script:

aws s3 ls --recursive s3://bucket name \
| awk '$1 >"2021-03-22 00:00:00" {print $0}' \
| xargs -I {} aws s3 cp --recursive "s3://bucket name/{}" folder

But no files get copied to the location specified (folder)

If I run the first two parts I do get the expected results:
2021-03-23 00:35:33 104742 source/7700ABJIBA442720210811934.20210323003442.txt
2021-03-23 00:35:37 33 source/7700ABJIBA442720210811934.md5.20210323003443.txt

What am I missing?

pan64 · 03-23-2021, 09:18 AM

I guess the filename itself is $2 (or $3 ?), not $0, so you need to print $2 in your awk.
Probably better to print $NF, which is the last field.

shruggy · 03-23-2021, 09:21 AM

No, I guess it's $3 because $1 is date and $2 is time. That means $1>"2021-03-22 00:00:00" won't work, too. See my post #6 above.

allend · 03-23-2021, 11:23 PM

I am new to AWS, but it seems that the AWS tools are deficient in handling date/time stamps.
It is possible to use awk to convert the text date and time fields to a numeric time stamp. Perhaps there are some ideas here

rnturn · 03-23-2021, 11:48 PM

Quote:

What is preventing use of find and either the -newer or -newermt option?

Quote:

Nothing. I didnt pursue that as an option. Would it be a more simple solution?

I'm unsure of the AWS command syntax you'll need, but on generic Linux, you could do something like:

Code:

$ SRCDIR=your-source-location
$ TGTDIR=your-target-location
$ MARKER=some.file
$ cd ${TGTDIR}
$ touch -d "2021-03-22 00:00:00" ${MARKER}
$ find ${SRCDIR} -type f -newer ${MARKER} -exec cp -p {} . \;
$ rm ${MARKER}

Tweak as needed to work with your AWS buckets

HTH...