I commonly use curl or wget in a RHEL5.6 Bash environment; however, a recent request has stumped me.

The web server I am sending a http request to is sending an initial response object, then multiple subsequent .js files after the end of the response object. The ongoing server pushes are all .js files and continually come. Obviously curl and wget only send a request and listen for a response - nothing past that.

I am looking for a way, whether it be Shell Script, Perl, PHP, or really any linux-based solution to receive these ongoing responses and place them into flat files for parsing.

The files all appear to be JSON objects of data.

I interrogated the header of these server pushes and the header look like the following:

Any feedback, suggestions, or information would be greatly appreciated. Without simply watching the data update in a browser I am having difficulties finding a shell-based solution to retrieve this data.

What does the response body look like?

The initial response is standard HTML/Javascript. The multiple subsequent responses I can only "catch" and view right now if I use a debugging tool; however, it is 100% javascript:

if (__ally && typeof __ally == 'function') __ally( {"allyId":"d11b142d-f173-4da4-b035-0cab25cbaf62","timestamp":"1337114643753","updated":{"799279949":{"session_event_duration":14},"7990 28201":{"session_event_duration":8},"797664083":{"session_event_duration":7},"799152871":{"session_s tate":"HANDLE","session_object_time":1337114634000,"session_queue_time":"","lastname":"Doe Jr.","firstname":"John","a_id":130342,"groups":"0E83E5B4DB42CEAC672F49440CE99066 2F394C67360D9A14BAB432A0CA5AD68E 7A2D090BCD18167BBA368D6F97197270 891496B07C3EE9A4A2BDEB7DC31AC5FD 8CA41D15F5380783DECA8D6EC219EDB9 AE01A5756B781CDB13C1A83386A68544 DAB45A2C3FB618856F6B12814A6B929F F8D2625AB3DB5CBEA9FE1F34EF10321D"}},"tablePath":"\/source\/SessionDetail56","roleName":"rtm_a","rowCount":56,"since":"1337114635185","type":"delta","deleted":["799276716","799527820"],"bootstrapping":false});

In typing this, I guess there is a chance that the initial response is making subsequent request calls via javascript too. Maybe I should look into seeing if subsequent ajax requests are being done behind the scenes.

I know when you go to the page in a web browser the page loads, then data is continually refreshed/updated. Obviously when in a shell environment, using curl or wget, all I see is the html of the initial response and not all the subsequent data. I am trying to figure out how to get this data in a programatic fashion.

An HTTP server can't "send" anything after an initial response. I'm afraid I'm not understanding what's happening.

Would scheduling a wget every minute/second/hour/whatever or so work?

That makes sense now - I think I was trying to make this more magical than it is. What must be happening is the initial response begins a polling session sending subsequent requests to get data. Thus, after the initial response is received ongoing request/responses are occurring. I just need to figure out the subsequent responses and duplicate them.

If you visit the site using Chrome or Firefox you should be able to view what is happening between the browser and the server using the
Developer Tools or Javascript Console.
wrench icon -> Tools -> Javascript Console, pick the Network tab and you will see what requests the browser is sending.