mirroring ftp from RedHat 9.0 to SGI IRIX
 

Can someone recommend s/w to mirror a directory on a RedHat 9.0 system to a directory on an SGI IRIX system?

Something like wget or emirror, but the other way around. Note that I only have ftp access to the IRIX system.

Currently, I am using Transport2[1] on my Mac Laptop accessing the RH9 files via samba. Less than ideal, I think.

Max.

[1] http://www.panic.com/transmit/

lftp can mirror a remote site -> local or a local site -> remote via ftp.

I use this to keep my website up to date with:
$ lftp
~> open -u <username>,<password> <ftp_server>
~> lcd <source_directory>
~> cd <target_directory>
~> mirror -R -e
~> quit

You can put all the lftp commands into a file and do:
lftp -f <name_of_file>

The mirror command only transfers files which are newer on the source side, so it's not too bad with bandwidth. the -e option for mirror deletes files at the target directory that aren't present in the source directory

Dave

Awesome! Just what I was after! It even does symlinks correctly, which Transmit doesn't...

Thanks muchly.

Max.

directory structure and symlinks.
 

Hi again,

I want to make a command which will create specific files before taking on the task of uploading the big image files I have.

I want to create the directory structure and upload index.html files, as well as files which are symlinks.

I don't know how to do the directory structure, but I made a list (using find) of index.html files and uploaded them fairly easily, buy my attempts to use put to upload symbolic links fails - it complains about them being directories, when they only point to directories.

Does the put command follow symlinks? That is undesirable to me. I notice the mirror command does not follow symlinks (unless you specify the -L option).

Help?

Max.

Hi again.

Yes, the 'put' command does follow symlinks.

Symlinks probably won't work on the remote server, because symlinks hold the absolute address of the file they point to, and that address almost certainly won't be the same on the remote side. (I don't even know if symlinks are the same accross all versions of UNIX).

If you're using the SGI ftp system to just backup your files, then you may want to consider putting everything into a tar file. That will preserve symlinks, permissions, etc. If you need access to the files directly, then I think your only option is to create the symlinks on the SGI machine. Whether or not you can actually do this over ftp, I don't know. If you've got ssh or rlogin access, then you could do this through one of these protocols.

As for creating the directory structure, and creating files, could you be more specific?

Dave

Not if they're relative sym links :

lrwxrwxrwx 1 mwaterman user 14 Apr 1 00:42 ./chronological/2004/03/24/Dominatrix6 -> ../Dominatrix6

I'm not - I'm trying to upload this (it's partially there) :

http://reality.sgiweb.org/maxw/gallery

I don't think this is correct. lftp's mirror command recreates the sym links perfectly well, but it's 'put' command doesn't even try.

Well, lft's mirror command works fine, and will, eventually, produce what I want (I think). However, due to the large image files, it is taking ages. It is a photo gallery that I am attempting to create. I have created it on a local machine, and so I am now try to mirror that on the sgiweb server.

While it is in this interim stage, it looks really bad because not all the index.html files are there, and the small images that make up the decoration on the web pages do not work (because the sym links are not all in place).

I guess, ideally, I would like 'mirror' to have an option to upload the smallest files first. Alternatively, and this is what I was trying, I would like to make a list of files and have it upload them. The latter works for index.html files, but not for the sym links.

So, my only option seems to be to wait for 'mirror' to complete, or use a different ftp client that will upload sym links properly.

Any ideas?

Max.

OK, so your post lead me to check that the mirror command was creating the links, and, low and behold, it was not. It looks like it is actually following the local links and making many copies of what the links point to.

This isn't such a big deal, since what they point to is small, and the reason for them is more a maintenance issue, which is all done on the local machine anyway.

Looking at the man page, I see :

"
Note that when -R is used (reverse mirror), symbolic links are not cre-
ated on server, because ftp protocol cannot do it. To upload files the
links refer to, use `mirror -RL' command (treat symbolic links as
files).
"

Bummer.

Hrm, I did not use the '-L' options, but it still followed the links. Strange.

I guess the 'put' command doesn't work because it has the same problem. It can't create a symbolic link, because the ftp protocol doesn't support it, and when it tries to follow the link, it finds they point to directories, which it cannot copy (it only does files).

Hrm. I suppose I need to use the 'mirror -RL' command on the symlinks....I'll try that.

If you have any ideas on how to make the sym links on the remote machine (can an ssh client do it?), then I'm all ears :)

Thanks!

Max.

Huh. I've been using Linux for years and never knew symlinks could be relative. You learn something new every day.

If you've got an account on the SGI machine and sshd is running, then you can create the symlinks just like you would on your local machine. (That is if your account on the SGI machine can see the ftp space).

Dave

alternative to ftp?
 

Me too :D

I am not sure about the ssh account. I might be able to sort something out, but I wouldn't want to go in and create them all by hand anyway. I would want to be able to use something like lftp, that can do the mirror automatically, with a single command.

Is there such a thing for ssh? It's too much to hope that lftp can use ssh instead (the name suggests that it can't).

Max.

You might want to ftp into the SGI machine and see what commands are available. You might find something useful.

You can usually get a list of commands by doing either ? or help when you're in using ftp.

I don't hold out much hope. though.

It could be an option to tar and gzip the whole site and get someone on the SGI machine to untar it, or untar it yourself if you can get in through ssh. That would solve all your problems in one go.

Dave

P.S. It might be simpler just to write a script which changes the HTML to reference the files directly, without having symlinks at all. Just a thought....

Not many command made available to the ftp client, I'm afraid. Just help and chmod.

That would be fine the first time, but not for regular mirroring ... though, I suppose, the symlinks are not going to change much. I would have to be careful about maintaining the ownership of the files such that I can remove/change them with ftp - I've not been too successful at that in the past :(

Well, it's a script[1] which creates the html to start with :) It is done that way for a reason....it is because the files are not always in the same place - unless they are always the same relative to the current directory - ie ./Dominatrix6/ or ../Dominatrix6/

The whole problem started because I will have the gallery on 3 different web sites, each will likely have a different root, and will be in different parts of the world, not necessarily with access to the other.

:(

Max.

[1] http://marginalhacks.com/Hacks/album/

I think I'm out of ideas. I think symlinks and ftp just don't mix. Anyone else out there have any ideas??

If you find a solution, post back - I'd be interested to see how it's solved.

Good luck!

Dave

Thanks very much for your help!

One last question...I have tried, but can't quite seem to get it right.

How can I put this command into the background :

lftp -f themes.txt

where themes.txt contains :

"
open -u ... reality.sgiweb.org
mirror -L -c -R -e public_html/gallery/chronological/2001/02/Dominatrix6 gallery/chronological/2001/02/Dominatrix6
mirror -L -c -R -e public_html/gallery/chronological/2001/02/New_York/Dominatrix6 gallery/chronological/2001/02/New_York/Dominatrix6
mirror -L -c -R -e public_html/gallery/chronological/2001/02/New_York/tn/Dominatrix6 gallery/chronological/2001/02/New_York/tn/Dominatrix6
...
quit
"

I want them all to be run sequentially, but I don't want to have to be logged in while they're being run.

Thanks.

Max.

I got a reply from Alexander V.Lukyanov :

"
On Wed, Apr 07, 2004 at 12:44:13AM -0700, Max Waterman wrote:

>> I would like to be able to determine which file types
>> get copied first.
>> I would like it to create the directories so that the
>> directory
>> structure is there, then copy 'index.html', then copy
>> symlinks named
>> 'Dominatrix6' (which point to either a directory, or
>> another symlink).


There is a setting named mirror:order, which can be used to control
the order of files being transferred. Trailing slash means a directory.

So you should do before mirror:

set mirror:order "*/ index.html Dominatrix6 *"


>> I don't know how to make it create just the
>> directories.


Use -X option to exclude all files:

mirror -X *

"

Awesome :D

Max.

If you want to put a command into the background (demonise):

{ [command] > /dev/null 2>&1 < /dev/null & } &

So I would write a script with all the ftp commands in it and run it with, say;

{ uploadsite > /dev/null 2>&1 < /dev/null & } &

That disconnects lftp (or any other program) from the console (the /dev/null redirects), and makes it a grandchild process (the two &'s), so the shell it was launched from can't kill it.

Dave