LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 10-26-2011, 02:33 PM   #1
vagmztp
LQ Newbie
 
Registered: Oct 2011
Posts: 22

Rep: Reputation: Disabled
How do you create and maintain an actual file via script on Linux


I am a mainframe COBOL85 old time programmer and a new Unix/Linux C programmer. I want to modify a bazillion scripts that have ftp commands in them with the user id and password hard coded to call a 'generic' script, passing a key, that will access a file to pull out the information on whatever server they want ftp to. I was startled to hear that key-sequenced files aren't actually 'done' on Unix/Linux. A couple of my coworkers thought I could make a script that would have every possible user id/password and then maintain it with vi. This sounds pretty clunky to me! I was thinking of making a baby screen to maintain the file - and now I understand 'screens' aren't actually done on the Unix/Linux either! Does anyone have an idea of a generic script (and I'm currently better with the scripting than with C) that could be used to pull user ids and passwords from a file? And make the file maintainable?
Thanks! I've got until Friday to either come up with a plan or tell her it can't happen. Of COURSE it can happen, I just don't know how!
 
Old 10-26-2011, 03:04 PM   #2
macemoneta
Senior Member
 
Registered: Jan 2005
Location: Manalapan, NJ
Distribution: Fedora x86 and x86_64, Debian PPC and ARM, Android
Posts: 4,593
Blog Entries: 2

Rep: Reputation: 344Reputation: 344Reputation: 344Reputation: 344
There are so many ways to accomplish this, it's hard to know where to begin. The best way is to stop using insecure ftp, and use key-based scp/sftp instead. I assume you are already using ssh and have a key. If not, you can create a key with:

Code:
ssh-keygen
That only needs to be done once on each account. Just hit enter for the passphrase.

For each system that you need to scp/sftp to:

Code:
ssh-copy-id userid@host
After authentication, this will append the key to the authorized key list for the user@host. Again, this is a one-time operation. From now on, your scripts can perform secure transfers with:

Code:
scp /path/file user@host:path/file

or

scp user@host:path/file /path/file
 
1 members found this post helpful.
Old 10-26-2011, 03:18 PM   #3
vagmztp
LQ Newbie
 
Registered: Oct 2011
Posts: 22

Original Poster
Rep: Reputation: Disabled
I hear you

BUT this is corporate America and they want the scripts to change as little as possible - to save on costs. Some are already using sftp but not most. And there are literally hundreds of servers they ftp to - they pull files from this server, send them to that server, or send or receive from a service provider. All they have in common is (with the exception of the sftp ones) the user id and password are hardcoded in the script. I can't think there isn't some way to grab all those user ids and passwords, stick them somehow into a file and then call a script using a key to retrieve the info.

But you are right! With sftp you wouldn't have that problem. I can raise it as an alternative but you would not even believe the problems I had just getting my own logon to these boxes! And I'd have to involve some other department too. When you say each account - each server/userid& password do you mean? If we have 300, we'd have to have 300 keys? How about for 'exterior' servers - on other company sites. Would we have to work with them too? Would it work the same to retrieve files from other company sites? Right now, in these scripts, sftp is used seems to be only with a service provider.
Thanks for your quick response!
 
Old 10-26-2011, 03:29 PM   #4
macemoneta
Senior Member
 
Registered: Jan 2005
Location: Manalapan, NJ
Distribution: Fedora x86 and x86_64, Debian PPC and ARM, Android
Posts: 4,593
Blog Entries: 2

Rep: Reputation: 344Reputation: 344Reputation: 344Reputation: 344
You are going to change the ftp commands anyway, so you might as well simplify the code making it easier to maintain, more robust, and more secure while you do it. Putting passwords in a file is such a bad thing, it's impossible to justify. Just think how you would explain it to the media if there were a security violation. Do you think you would be able to convince a member of the board that you did the right thing?
 
Old 10-26-2011, 03:43 PM   #5
vagmztp
LQ Newbie
 
Registered: Oct 2011
Posts: 22

Original Poster
Rep: Reputation: Disabled
That's valid - but I still think it's better than having them hard coded in so many scripts! And when they (sometimes) change, it's a real hassle.

Is there a good source for information on sftp? I have Linux in a nutshell but I could use examples or steps to take. The boxes are Linux boxes (just in the last several months). If I make a plan to change the ftp to sftp, it ought to be an informed plan, I believe. Here's a snippet of one:

ftp -v -n xxxtest.ccs.xxx.com << _END! > $EDIH/mb/etc/etc/xxxtest_status
user "someuser" some password
ascii
prompt
cd /etc/etc/out
lcd $EDIH/mb/etc/etc/directory
ls
mget *.TXT
mdelete *.TXT
bye
_END!

And a snippet of one of the sftp to our service provider:
sftp -b $EDIH/etc/batchscript 700000006@nnn.nnn.nn.nn >$EDIH/interfact/util/etc.

The batchscript just has ftp commands like cd and put. The 70000 number isn't real, of course, and the nnn.nnn is the address of the server.
 
Old 10-26-2011, 03:54 PM   #6
macemoneta
Senior Member
 
Registered: Jan 2005
Location: Manalapan, NJ
Distribution: Fedora x86 and x86_64, Debian PPC and ARM, Android
Posts: 4,593
Blog Entries: 2

Rep: Reputation: 344Reputation: 344Reputation: 344Reputation: 344
The man pages (e.g., man scp or man sftp) contain the documentation. It looks like the above could be replaced by:

Code:
scp someuser@xxxtest.ccs.xxx.com:/etc/etc/out/*.TXT $EDIH/mb/etc/etc/directory/ && \
ssh someuser@xxxtest.ccs.xxx.com /bin/rm /etc/etc/out/*.TXT
 
Old 10-26-2011, 04:02 PM   #7
vagmztp
LQ Newbie
 
Registered: Oct 2011
Posts: 22

Original Poster
Rep: Reputation: Disabled
I will do that and see what I can find. I'm unfamiliar (duh!) with scp itself. And we just recently changed our putty configs to use ssh, so I'm new at that too. I would have to set up all these keys, though. Perhaps it'll tell me how to do that too. This sftp will work to Unix and IBM, I'm thinking.

Thanks for your help! I'll be back at it tomorrow...
 
Old 10-26-2011, 07:06 PM   #8
theNbomr
LQ 5k Club
 
Registered: Aug 2005
Distribution: OpenSuse, Fedora, Redhat, Debian
Posts: 5,399
Blog Entries: 2

Rep: Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908Reputation: 908
Quote:
Originally Posted by vagmztp View Post
I am a mainframe COBOL85 old time programmer and a new Unix/Linux C programmer. I want to modify a bazillion scripts that have ftp commands in them with the user id and password hard coded to call a 'generic' script, passing a key, that will access a file to pull out the information on whatever server they want ftp to. I was startled to hear that key-sequenced files aren't actually 'done' on Unix/Linux. A couple of my coworkers thought I could make a script that would have every possible user id/password and then maintain it with vi. This sounds pretty clunky to me! I was thinking of making a baby screen to maintain the file - and now I understand 'screens' aren't actually done on the Unix/Linux either! Does anyone have an idea of a generic script (and I'm currently better with the scripting than with C) that could be used to pull user ids and passwords from a file? And make the file maintainable?
Thanks! I've got until Friday to either come up with a plan or tell her it can't happen. Of COURSE it can happen, I just don't know how!
I guess I must know a little bit about how you feel about Linux, since none of those non-Linux things you mention don't mean anything to me. So, just for the sake of background information, all files in Linux are just a linear stream of 8-bit bytes. You write them sequentially, and then read them back in the same order. You get to seek to a specified offset within the file, arbitrarily, to commence or continue to read or write. Different tools treat the data differently, and file content depends on the purpose. You can open()/read()/write()/close() files of any type equally. Files whose primary purpose is to contain human-readable text are generally line-delimited by linefeed bytes.

So, as I understand your problem, you want to first mine all of the hard-coded user-specific content from a bunch of existing scripts (which we in Linux would interpret to be plain-text files). The mined data needs to be stored in a file, where an existing or to-be-created tool can be used to manage it. You also want to macro-ize the scripts, so that parameters can be passed into them at runtime, relieving a script maintenance issue. It follows that you would then need one more thing: a method/tool which knows how to extract the correct values from the database/file for use with a particular script. Perhaps each record in the database/file maps uniquely to a particular script. Since you have bazzillions of scripts, and since the data is going to be used by searching for and extracting particular records, maybe a DBMS (albeit small) is a good fit for your application. SQLite is nice and small, single-file based, and provides reasonable performance for low cost. The are probably existing FOSS GUI tools for generic access to SQLite databases, so your management tool is looked after, or at least a starting point.

The other side of the problem, the data mining, can probably be done fairly easily with some scripting language such as Perl, AWK, sed, Bash, etc, or some combination of those. Since it seems to be a once-and-done job, it doesn't have to be pretty.

--- rod.
 
1 members found this post helpful.
Old 10-31-2011, 09:15 AM   #9
vagmztp
LQ Newbie
 
Registered: Oct 2011
Posts: 22

Original Poster
Rep: Reputation: Disabled
Out sick for 2 days last week! Thought it was food poisoning and found out it was flu. Yuck.

My deadline passed and my boss is out for the week so I should be able to come up with something for this week.

Though I'm still thinking of the file/param stuff, I listened to macemoneta when he says it's criminal to put passwords in a file! Just as criminal in a bunch of scripts. Also these scripts are corporate critical stuff and what if the file, whether Linux or SQL (especially) was not available and the script wouldn't run? Can't really risk that by adding another layer, I figure. So I'm going to try and use sftp or scp and ssh to authenticate on a server. Haven't figured it out yet, though. I will persevere!

Thanks to you both for your help!
 
Old 10-31-2011, 08:54 PM   #10
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,348

Rep: Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749
Yeah, I'd go scp with Auth keys for that where possible. external sites you may have to settle for good old ftp...
Good idea to push the security angle where possible. More secure and makes you look good

Doing it the other way is possible if you have to for some sites.
Extracting the current names/passwds would be annoying, but then you can just create a simple file (or use a simple db) and create dedicated script/fn that other progs can call (or 'source' if its shell scripts).

Let us know how it goes.
Welcome to LQ
 
1 members found this post helpful.
Old 11-01-2011, 08:55 AM   #11
vagmztp
LQ Newbie
 
Registered: Oct 2011
Posts: 22

Original Poster
Rep: Reputation: Disabled
You guys are really helpful - and I have such dumb questions. Thanks for not making me feel foolish.

I find that I have to go to our Unix techie group - I don't have the permission to do any of ssh stuff. And lucky for me, there aren't as many server/user combinations as I supposed. So that is the route for me!

I will let you know how it goes! And you can be sure I'll post again with a different dumb question...
 
Old 11-01-2011, 10:31 AM   #12
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fedora-35
Posts: 5,313

Rep: Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918
Quote:
Originally Posted by macemoneta View Post
You are going to change the ftp commands anyway, so you might as well simplify the code making it easier to maintain, more robust, and more secure while you do it. Putting passwords in a file is such a bad thing, it's impossible to justify. Just think how you would explain it to the media if there were a security violation. Do you think you would be able to convince a member of the board that you did the right thing?
rite on, it's like writing your atm pin on the back of your card.
 
1 members found this post helpful.
Old 11-01-2011, 10:44 AM   #13
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fedora-35
Posts: 5,313

Rep: Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918
Quote:
Originally Posted by vagmztp View Post
BUT this is corporate America and they want the scripts to change as little as possible - to save on costs. Some are already using sftp but not most. And there are literally hundreds of servers they ftp to - they pull files from this server, send them to that server, or send or receive from a service provider. All they have in common is (with the exception of the sftp ones) the user id and password are hardcoded in the script. I can't think there isn't some way to grab all those user ids and passwords, stick them somehow into a file and then call a script using a key to retrieve the info.
it seems like some custom script would be able to grep out the necessary text. i dont know what the data looks like so if you provide a snippet maybe we can find a regex that would match your criteria ?
 
1 members found this post helpful.
Old 11-08-2011, 01:12 PM   #14
vagmztp
LQ Newbie
 
Registered: Oct 2011
Posts: 22

Original Poster
Rep: Reputation: Disabled
How it's going

So I find out that with 26 unique user/server combinations, only 2 of them can be changed to be sftp. I didn't realize that sftp is only on Linux - that even if installed on another box it may be different, etc., etc.! Now to get around this problem with the ones that can't use sftp, one of our Unix guys suggested I use cURL for authentication. Well, I have searched the internet (at least I tried) and I can't find a snippet of how to do this. I can find the man pages but I find those things to be pretty cryptic - at least to a newbie like me. And they don't provide examples. Can anyone out there give me an example of a script that uses cURL to do the ftp? I would want to pass it the filename (the netrc I think) where the userid and password would be found. I understand that it uses the HOME directive to find it.

Jest a leetle bitty script would help!

Thanks!
 
Old 11-10-2011, 07:23 AM   #15
schneidz
LQ Guru
 
Registered: May 2005
Location: boston, usa
Distribution: fedora-35
Posts: 5,313

Rep: Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918Reputation: 918
hi, i never use curl or ftp so i dont know but i typed in curl ftp into google and the 2nd page listed this:
Code:
To download a file from ftp server:
curl ftp://myftpsite.com/mp3/mozart_piano_sonata.zip --user myname:mypassword -o mozart_piano_sonata.zip

To upload a file to ftp server:
curl -T koc_dance.mp3 ftp://myftpsite.com/mp3/ --user myname:mypassword
this is horribly insecure obviousely since the user name and password are in plain text.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Create a script to display file name, Inode, and size of any file. Has to be a script JaxsunApex Linux - Newbie 7 01-29-2007 08:15 PM
how to create link into actual file? elmerliu Linux - Newbie 11 08-15-2004 11:40 AM
how to create a link into actual file? elmerliu Fedora 1 08-14-2004 08:56 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 09:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration