LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 04-29-2019, 08:34 PM   #1
RandomTroll
Senior Member
 
Registered: Mar 2010
Distribution: Slackware
Posts: 1,424

Rep: Reputation: 238Reputation: 238Reputation: 238
Downloading New Yorker audm files


I want to download the audio versions of The New Yorker's articles, which they make available through their Audm app. I can get them with Video Download Helper but I'd prefer to download them straight.
http://audm.herokuapp.com/player-emb...d6222e51c0be46 for example
 
Old 04-30-2019, 01:28 PM   #2
individual
Member
 
Registered: Jul 2018
Posts: 276
Blog Entries: 1

Rep: Reputation: 214Reputation: 214Reputation: 214
If you know the article id, you can use this bash script I wrote. You'll need jq. Please don't abuse it.
Code:
#!/bin/bash

set -o pipefail

usage="$0 publisher articleID"
url="https://audm.herokuapp.com/parse/classes/ArticleVersion"

publisher="$1"  # e.g. newyorker
articleID="$2"  # e.g. 5cc07abdf1d6222e51c0be46

if [[ ! "$publisher" || ! "$articleID" ]]
then
    echo $usage && exit
fi

data="{\"where\":{\"publisherUniqueID\":\"$articleID\",\"publicationList\":{\"\$inQuery\":{\"where\":{\"slug\":\"$publisher\"},\"className\":\"PublicationList\"}}},\"include\":\"publication\",\"limit\":1,\"_method\":\"GET\",\"_ApplicationId\":\"M2PeWDdRSFsl28CzdYlAjG2mZGCdhSvosTQGBJhI\",\"_JavaScriptKey\":\"eUUwgiLTAQnj0tG1FrwtM7CPBhw3rDq6KMuKRhYO\",\"_ClientVersion\":\"js1.10.0\",\"_InstallationId\":\"1ed00cbf-8fde-162a-f891-6728e1bf7012\"}"

dataLength="$(echo -n $data | wc -c)"

curl -s \
-H "Host: audm.herokuapp.com" \
-H "User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0" \
-H "Accept: */*" \
-H "Accept-Language: en-US,en;q=0.5" \
-H "Accept-Encoding: gzip, deflate, br" \
-H "Referer: https://audm.herokuapp.com/player-embed/?pub=$publisher&articleID=$articleID" \
-H "Content-Type: text/plain" \
-H "Content-Length: $dataLength" \
-H "Connection: keep-alive" \
-X POST "$url" \
-d "$data" | jq -r '.results[0]["audioUrlM4a"]' | xargs curl -sO
 
1 members found this post helpful.
Old 04-30-2019, 10:48 PM   #3
RandomTroll
Senior Member
 
Registered: Mar 2010
Distribution: Slackware
Posts: 1,424

Original Poster
Rep: Reputation: 238Reputation: 238Reputation: 238
That works: thanks!

How can I abuse it? I prefer to listen using mplayer, sometimes on my portable mp3 player. While The New Yorker publishes segments aimed at a radio audience, they make these files available on their player, seeming to make one run it while on-line. I'm only getting around that more conveniently.

Are ApplicationID , JavascriptKey, and InstallationID magic numbers? Am I making a privileged query to a JSON database?
 
Old 05-01-2019, 06:28 AM   #4
individual
Member
 
Registered: Jul 2018
Posts: 276
Blog Entries: 1

Rep: Reputation: 214Reputation: 214Reputation: 214
Quote:
Originally Posted by RandomTroll View Post
That works: thanks!

How can I abuse it? I prefer to listen using mplayer, sometimes on my portable mp3 player. While The New Yorker publishes segments aimed at a radio audience, they make these files available on their player, seeming to make one run it while on-line. I'm only getting around that more conveniently.

Are ApplicationID , JavascriptKey, and InstallationID magic numbers? Am I making a privileged query to a JSON database?
By downloading all the things! I didn't think you would, just throwing it out there.
 
Old 05-01-2019, 08:29 AM   #5
teckk
Senior Member
 
Registered: Oct 2004
Distribution: FreeBSD Arch
Posts: 2,900

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
Code:
mpv http://s3.amazonaws.com/audm-prod/singlefile-5cc07abdf1d6222e51c0be46-d3ab5024-45bd-40b0-821d-f332c1951d8a.m4a

ffprobe http://s3.amazonaws.com/audm-prod/singlefile-5cc07abdf1d6222e51c0be46-d3ab5024-45bd-40b0-821d-f332c1951d8a.m4a
...
 Duration: 00:24:37.06, start: 0.000000, bitrate: 65 kb/s
    Stream #0:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 64 kb/s (default)
...
 
Old 05-01-2019, 05:28 PM   #6
RandomTroll
Senior Member
 
Registered: Mar 2010
Distribution: Slackware
Posts: 1,424

Original Poster
Rep: Reputation: 238Reputation: 238Reputation: 238
I don't understand your reply teckk. Getting the URI is the hard part.
 
Old 05-02-2019, 08:23 AM   #7
teckk
Senior Member
 
Registered: Oct 2004
Distribution: FreeBSD Arch
Posts: 2,900

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
That is the audio url for the link you posted. I can't post a hacker script on this forum @RandomTroll

You'll have to follow the scripts on that page/link to see where the media is located.

You can open a web inspector in your favorite browser and look for the virtual url, or playlist, or link to a media file. Little bit slower but if a browser can open it then your web inspector should be able to list it.

Sometimes youtube-dl will get you a url. youtube-dl --list-extractors Sometimes it will work for urls that are not listed.

You can make a python script using a web browsers engine, beautiful soup, selenium, nodejs etc., to dump web page requests to terminal.
That's my preferred method.

Sometimes I look at the pages source, and if I can figure it out, I use bash and friends to get the info I want.

Looks like you have to be subscribed to load those pages. So if you want more then post links to them.

You say that you can get them with Video Download Helper. That might be the easiest way for you.
 
Old 05-25-2020, 09:45 AM   #8
RandomTroll
Senior Member
 
Registered: Mar 2010
Distribution: Slackware
Posts: 1,424

Original Poster
Rep: Reputation: 238Reputation: 238Reputation: 238
This stopped working since last week. Perhaps one of the magic numbers has expired.
 
Old 05-25-2020, 11:27 AM   #9
teckk
Senior Member
 
Registered: Oct 2004
Distribution: FreeBSD Arch
Posts: 2,900

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
Code:
mpv http://s3.amazonaws.com/audm-prod/singlefile-5cc07abdf1d6222e51c0be46-d3ab5024-45bd-40b0-821d-f332c1951d8a.m4a
 (+) Audio --aid=1 (*) (aac 2ch 44100Hz)
AO: [alsa] 48000Hz stereo 2ch float
A: 00:00:04 / 00:24:37 (0%) Cache: 1472s/42MB
Divine guidance
adam kirch
He wrote a book...etc.

That's the url for the .aac stream for the link that you posted.

If you want something else then post a link. Also, you are going to have to learn how to scrape.
 
Old 05-25-2020, 11:38 AM   #10
shruggy
Member
 
Registered: Mar 2020
Posts: 967

Rep: Reputation: Disabled
@teckk. This is the same link you posted in #5 .

@OP. The script posted by individual in #4 still works fine with the data you provided in the top post and yields the link given by teckk above.
 
Old 05-25-2020, 11:46 AM   #11
teckk
Senior Member
 
Registered: Oct 2004
Distribution: FreeBSD Arch
Posts: 2,900

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
Quote:
@teckk. This is the same link you posted in #5 .
I know. That's all we have been given to work with. The link in post 1.

I scraped that audio link with python, webengine, pyqt5 by the way.
 
Old 05-25-2020, 11:55 AM   #12
RandomTroll
Senior Member
 
Registered: Mar 2010
Distribution: Slackware
Posts: 1,424

Original Poster
Rep: Reputation: 238Reputation: 238Reputation: 238
Quote:
Originally Posted by teckk View Post
Divine guidance
adam kirch
He wrote a book...etc.
All of us could figure out everything ourselves instead of asking in a public forum.

Quote:
Originally Posted by teckk View Post
If you want something else then post a link.
https://audm.herokuapp.com/player-em...89dfa6eda9d86a


Quote:
Originally Posted by teckk View Post
you are going to have to learn how to scrape.
I looked for an alternative.

Quote:
Originally Posted by shruggy View Post
@OP. The script posted by individual in #4 still works fine with the data you provided in the top post and yields the link given by teckk above.
It doesn't surprise me that old keys work on old URIs. I tried multiple times with 3 links from the current issue.

Quote:
Originally Posted by teckk View Post
I scraped that audio link with python, webengine, pyqt5 by the way.
You ran
Code:
python https://audm.herokuapp.com/player-embed?pub=newyorker&articleID=5ec5745c4889dfa6eda9d86a
?

I opened in web console (Firefox) and found the target URI. @individual responded usefully.
 
Old 05-25-2020, 12:18 PM   #13
teckk
Senior Member
 
Registered: Oct 2004
Distribution: FreeBSD Arch
Posts: 2,900

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
https://audm.herokuapp.com/player-em...89dfa6eda9d86a

Letter from SanFrancisco, too far from home.
Code:
http://s3.amazonaws.com/audm-prod/too-far-from-home-heller-singlefile.m4a
 
Old 05-25-2020, 10:01 PM   #14
RandomTroll
Senior Member
 
Registered: Mar 2010
Distribution: Slackware
Posts: 1,424

Original Poster
Rep: Reputation: 238Reputation: 238Reputation: 238
Quote:
Originally Posted by teckk View Post
https://audm.herokuapp.com/player-em...89dfa6eda9d86a

Letter from SanFrancisco, too far from home.
Code:
http://s3.amazonaws.com/audm-prod/too-far-from-home-heller-singlefile.m4a
As I said in my previous message, I already figured that out. I was asking about automating it.
 
Old 06-14-2020, 01:34 PM   #15
jawanurp
LQ Newbie
 
Registered: Jun 2020
Posts: 1

Rep: Reputation: Disabled
Hi everyone,

I came across this thread and I'm to accomplish this (downloading Audm New Yorker readings) but I have Mac OS. I registered just to post here and see if anyone could help me accomplish this in my situation. I realize this is not what this website is intended for, but I don't know where else to ask.

Thanks so much, hope to hear back from anyone
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
The cover of October 23's 'New Yorker' RandomTroll General 11 11-02-2017 09:55 AM
New apache install, php files are downloading rather than displaying abefroman Linux - Software 2 10-17-2009 06:25 PM
while downloading html files it becomes mp3 files tam Linux - Newbie 3 08-28-2007 03:33 AM
Downloading files into root or write files to root RaleighRalley Linux - Newbie 2 07-01-2006 10:56 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 06:33 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration