LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices



Reply
 
Search this Thread
Old 04-25-2008, 02:36 PM   #1
indiancosmonaut
Member
 
Registered: Feb 2007
Posts: 65

Rep: Reputation: 15
How to ensure that the zip file is completely downloaded.


Hi,

I am using unzip utility to unzip .zip file.
If unzip finds the file while it is downloading, it gives an error something like

End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.


Is there a way by which we can ensure that the .zip file has been properly downloaded before we give it to unzip utility.

Kind regards,

indiancosmonaut
 
Old 04-25-2008, 02:41 PM   #2
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 15,078

Rep: Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713
What are you using to download the file?

Some downloaders will put a <filename>.zip.part file out there, along with just <filename>.zip. The .zip file will be zero bytes, while the .part file will contain the data. As soon as the download is complete, the .part file is moved to .zip, and removed, leaving you with one complete file.

Wget may act differently, as so some download utilities used through your web browser. You don't say what you're using, so it's hard to help.

If this is a program you're writing yourself, you should be able to easily monitor the download by looking for the thread in the process table, or checking the return status of the code....
 
Old 04-25-2008, 02:58 PM   #3
indiancosmonaut
Member
 
Registered: Feb 2007
Posts: 65

Original Poster
Rep: Reputation: 15
Hi TBOne,

Thanks for replying,

I am using unzip utility to unzip my zip file.
I get the .zip file using secure ftp. And as soon as the file arrives, the utility tries to unzip it and fails giving the error mentioned below.

I am looking for a standard procedure (maybe) to ensure that the file is downloaded 100% and then go to the next line of code where the unzip utility is waiting to unzip it.

I tried,

cp /working_dir/file.zip to /tmp/file.zip
mv /tmp/file.zip to /working_dir/file.zip

but this didn't work.

Kind regards,

indiancosmonaut

Last edited by indiancosmonaut; 04-25-2008 at 03:00 PM.
 
Old 04-25-2008, 03:35 PM   #4
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 15,078

Rep: Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713
Quote:
Originally Posted by indiancosmonaut View Post
I am using unzip utility to unzip my zip file.
Yes, I know...I had asked what you were using to DOWNLOAD the file. Please re-read my post.

What you're using to download the file, will have some sort of rules you can go by.
 
Old 04-25-2008, 03:49 PM   #5
indiancosmonaut
Member
 
Registered: Feb 2007
Posts: 65

Original Poster
Rep: Reputation: 15
I got it.

Actually I run a script on my IBM AIX to unzip the file, which continuously checks for the zip file. Meanwhile at any point in time I ftp the file to the AIX box using secureshell (sftp). I am just transferring the file from my local machine to the UNIX box.

I hope I am able to explain

Kind regards,

indiancosmonaut
 
Old 04-25-2008, 04:01 PM   #6
TB0ne
Guru
 
Registered: Jul 2003
Location: Birmingham, Alabama
Distribution: SuSE, RedHat, Slack,CentOS
Posts: 15,078

Rep: Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713Reputation: 2713
Quote:
Originally Posted by indiancosmonaut View Post
Actually I run a script on my IBM AIX to unzip the file, which continuously checks for the zip file. Meanwhile at any point in time I ftp the file to the AIX box using secureshell (sftp). I am just transferring the file from my local machine to the UNIX box.
What you're describing can't be done. It's like asking "How high is up?"; the file(s) you're transferring show up at "any point in time", and can be any size. Since you don't know when they'll show up, or how big they can be, there's no way to know what's going on.

There are some klunky ways of doing things, though. You might be able to transmit a second "confirmation" file, such as a <filename>.txt file, containing the file size of the .zip file. Your script on the 'receiving' box could look for that file, and if it finds it, begins processing. Further error checking could be done by opening the confirmation file, and comparing the size in it, to the actual size of the file, to make sure there were no transmission errors.
 
Old 04-28-2008, 09:28 AM   #7
indiancosmonaut
Member
 
Registered: Feb 2007
Posts: 65

Original Poster
Rep: Reputation: 15
Ok.

I think one more thing that could be done is that we check for the file size at regular intervals of time. After two consecutive checks, if the file size is unchanged then we can safely assume that the file is copied 100% and so we can go ahead with the unzipping.

Kind regards,

indiancosmonaut
 
Old 04-28-2008, 02:39 PM   #8
ntubski
Senior Member
 
Registered: Nov 2005
Distribution: Debian
Posts: 2,541

Rep: Reputation: 878Reputation: 878Reputation: 878Reputation: 878Reputation: 878Reputation: 878Reputation: 878
Quote:
Originally Posted by indiancosmonaut View Post
I think one more thing that could be done is that we check for the file size at regular intervals of time. After two consecutive checks, if the file size is unchanged then we can recklessly assume that the file is copied 100% and so we can go ahead with the unzipping.

Kind regards,

indiancosmonaut
fixed.

Maybe another way would be to use lsof to check whether the file is opened by any programs. If it's not then it must be done downloading.
 
Old 04-29-2008, 03:26 AM   #9
indiancosmonaut
Member
 
Registered: Feb 2007
Posts: 65

Original Poster
Rep: Reputation: 15
Ok. The problem is that I do not have lsof installed. Thats the reason I was considering an alternative. But thanks anyway.

Kind regards,

indiancosmonaut
 
Old 04-29-2008, 03:33 AM   #10
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
A simpler way may be to download it as another name, then rename it when it's complete (assuming you're using a script to download it here, otherwise you can do this by hand also. rename/mv takes (almost) no time.
 
Old 05-01-2008, 12:48 PM   #11
schneidz
Senior Member
 
Registered: May 2005
Location: boston, usa
Distribution: fc-15/ fc-20-live-usb/ aix
Posts: 4,197

Rep: Reputation: 642Reputation: 642Reputation: 642Reputation: 642Reputation: 642Reputation: 642
how about every second checking for the existance of:
Code:
ps auxw | grep sftp <filename>
when it dissappears from the process list it should be done.


edit:
of course 'grep sftp <filename>' might show up in the process list so some magic has to be done to combat that.

Last edited by schneidz; 05-01-2008 at 12:49 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
File download says done and stops before file is completely downloaded in Firefox royeo Linux - General 2 12-24-2006 11:02 PM
SuperKaramba remove downloaded themes completely jimdaworm Linux - Software 8 09-21-2005 12:45 AM
create a self-extracting zip file with zip on solaris? samsolaris Solaris / OpenSolaris 3 10-15-2004 02:50 AM
Completely new to Linux, Downloaded Mandrake 10 and want to run both XP and Mandrake Scandal Mandriva 4 06-28-2004 11:03 PM
pcfg_openfile: unable to check htaccess file, ensure it is readable andy18 Linux - General 4 08-21-2003 02:29 PM


All times are GMT -5. The time now is 01:18 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration