Linux - SoftwareThis forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
lftp can mirror a remote site -> local or a local site -> remote via ftp.
I use this to keep my website up to date with:
$ lftp
~> open -u <username>,<password> <ftp_server>
~> lcd <source_directory>
~> cd <target_directory>
~> mirror -R -e
~> quit
You can put all the lftp commands into a file and do:
lftp -f <name_of_file>
The mirror command only transfers files which are newer on the source side, so it's not too bad with bandwidth. the -e option for mirror deletes files at the target directory that aren't present in the source directory
Originally posted by ilikejam lftp can mirror a remote site -> local or a local site -> remote via ftp.
I use this to keep my website up to date with:
$ lftp
~> open -u <username>,<password> <ftp_server>
~> lcd <source_directory>
~> cd <target_directory>
~> mirror -R -e
~> quit
You can put all the lftp commands into a file and do:
lftp -f <name_of_file>
The mirror command only transfers files which are newer on the source side, so it's not too bad with bandwidth. the -e option for mirror deletes files at the target directory that aren't present in the source directory
Dave
Awesome! Just what I was after! It even does symlinks correctly, which Transmit doesn't...
Originally posted by ilikejam lftp can mirror a remote site -> local or a local site -> remote via ftp.
snip
Dave
Hi again,
I want to make a command which will create specific files before taking on the task of uploading the big image files I have.
I want to create the directory structure and upload index.html files, as well as files which are symlinks.
I don't know how to do the directory structure, but I made a list (using find) of index.html files and uploaded them fairly easily, buy my attempts to use put to upload symbolic links fails - it complains about them being directories, when they only point to directories.
Does the put command follow symlinks? That is undesirable to me. I notice the mirror command does not follow symlinks (unless you specify the -L option).
Symlinks probably won't work on the remote server, because symlinks hold the absolute address of the file they point to, and that address almost certainly won't be the same on the remote side. (I don't even know if symlinks are the same accross all versions of UNIX).
If you're using the SGI ftp system to just backup your files, then you may want to consider putting everything into a tar file. That will preserve symlinks, permissions, etc. If you need access to the files directly, then I think your only option is to create the symlinks on the SGI machine. Whether or not you can actually do this over ftp, I don't know. If you've got ssh or rlogin access, then you could do this through one of these protocols.
As for creating the directory structure, and creating files, could you be more specific?
... then I think your only option is to create the symlinks on the SGI machine.
I don't think this is correct. lftp's mirror command recreates the sym links perfectly well, but it's 'put' command doesn't even try.
Quote:
As for creating the directory structure, and creating files, could you be more specific?
Dave
Well, lft's mirror command works fine, and will, eventually, produce what I want (I think). However, due to the large image files, it is taking ages. It is a photo gallery that I am attempting to create. I have created it on a local machine, and so I am now try to mirror that on the sgiweb server.
While it is in this interim stage, it looks really bad because not all the index.html files are there, and the small images that make up the decoration on the web pages do not work (because the sym links are not all in place).
I guess, ideally, I would like 'mirror' to have an option to upload the smallest files first. Alternatively, and this is what I was trying, I would like to make a list of files and have it upload them. The latter works for index.html files, but not for the sym links.
So, my only option seems to be to wait for 'mirror' to complete, or use a different ftp client that will upload sym links properly.
I don't think this is correct. lftp's mirror command recreates the sym links perfectly well, but it's 'put' command doesn't even try.
OK, so your post lead me to check that the mirror command was creating the links, and, low and behold, it was not. It looks like it is actually following the local links and making many copies of what the links point to.
This isn't such a big deal, since what they point to is small, and the reason for them is more a maintenance issue, which is all done on the local machine anyway.
Looking at the man page, I see :
"
Note that when -R is used (reverse mirror), symbolic links are not cre-
ated on server, because ftp protocol cannot do it. To upload files the
links refer to, use `mirror -RL' command (treat symbolic links as
files).
"
Bummer.
Hrm, I did not use the '-L' options, but it still followed the links. Strange.
I guess the 'put' command doesn't work because it has the same problem. It can't create a symbolic link, because the ftp protocol doesn't support it, and when it tries to follow the link, it finds they point to directories, which it cannot copy (it only does files).
Hrm. I suppose I need to use the 'mirror -RL' command on the symlinks....I'll try that.
If you have any ideas on how to make the sym links on the remote machine (can an ssh client do it?), then I'm all ears
Huh. I've been using Linux for years and never knew symlinks could be relative. You learn something new every day.
If you've got an account on the SGI machine and sshd is running, then you can create the symlinks just like you would on your local machine. (That is if your account on the SGI machine can see the ftp space).
Originally posted by ilikejam
Huh. I've been using Linux for years and never knew symlinks could be relative. You learn something new every day.
Me too
Quote:
If you've got an account on the SGI machine and sshd is running, then you can create the symlinks just like you would on your local machine. (That is if your account on the SGI machine can see the ftp space).
I am not sure about the ssh account. I might be able to sort something out, but I wouldn't want to go in and create them all by hand anyway. I would want to be able to use something like lftp, that can do the mirror automatically, with a single command.
Is there such a thing for ssh? It's too much to hope that lftp can use ssh instead (the name suggests that it can't).
You might want to ftp into the SGI machine and see what commands are available. You might find something useful.
You can usually get a list of commands by doing either ? or help when you're in using ftp.
I don't hold out much hope. though.
It could be an option to tar and gzip the whole site and get someone on the SGI machine to untar it, or untar it yourself if you can get in through ssh. That would solve all your problems in one go.
Dave
P.S. It might be simpler just to write a script which changes the HTML to reference the files directly, without having symlinks at all. Just a thought....
Originally posted by ilikejam
[B]You might want to ftp into the SGI machine and see what commands are available. You might find something useful.
You can usually get a list of commands by doing either ? or help when you're in using ftp.
I don't hold out much hope. though.
Not many command made available to the ftp client, I'm afraid. Just help and chmod.
Quote:
It could be an option to tar and gzip the whole site and get someone on the SGI machine to untar it, or untar it yourself if you can get in through ssh. That would solve all your problems in one go.
That would be fine the first time, but not for regular mirroring ... though, I suppose, the symlinks are not going to change much. I would have to be careful about maintaining the ownership of the files such that I can remove/change them with ftp - I've not been too successful at that in the past
Quote:
P.S. It might be simpler just to write a script which changes the HTML to reference the files directly, without having symlinks at all. Just a thought....
Well, it's a script[1] which creates the html to start with It is done that way for a reason....it is because the files are not always in the same place - unless they are always the same relative to the current directory - ie ./Dominatrix6/ or ../Dominatrix6/
The whole problem started because I will have the gallery on 3 different web sites, each will likely have a different root, and will be in different parts of the world, not necessarily with access to the other.
Originally posted by ilikejam I think I'm out of ideas. I think symlinks and ftp just don't mix. Anyone else out there have any ideas??
If you find a solution, post back - I'd be interested to see how it's solved.
Good luck!
Dave
I got a reply from Alexander V.Lukyanov :
"
On Wed, Apr 07, 2004 at 12:44:13AM -0700, Max Waterman wrote:
>> I would like to be able to determine which file types
>> get copied first.
>> I would like it to create the directories so that the
>> directory
>> structure is there, then copy 'index.html', then copy
>> symlinks named
>> 'Dominatrix6' (which point to either a directory, or
>> another symlink).
There is a setting named mirrorrder, which can be used to control
the order of files being transferred. Trailing slash means a directory.
So you should do before mirror:
set mirrorrder "*/ index.html Dominatrix6 *"
>> I don't know how to make it create just the
>> directories.
If you want to put a command into the background (demonise):
{ [command] > /dev/null 2>&1 < /dev/null & } &
So I would write a script with all the ftp commands in it and run it with, say;
{ uploadsite > /dev/null 2>&1 < /dev/null & } &
That disconnects lftp (or any other program) from the console (the /dev/null redirects), and makes it a grandchild process (the two &'s), so the shell it was launched from can't kill it.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.