LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 01-20-2012, 08:54 AM   #1
sscn
LQ Newbie
 
Registered: Jan 2006
Location: China
Distribution: Fedora
Posts: 28

Rep: Reputation: 0
The Best Way to support 100 client PCs to transfer files to one Server?


Hi all,

I am evaluating couples of ways to improve the file transferring from about 100 client Linux Fedora PCs to one Windows or Linux Server via local network.

1. NFS
2. Samba
3. iSCSI
4. Others

Which way is the best model to support huge client PCs to transfer large datas to one Server at least 100pcs no problem though 100M network?

I got that Windows couldn't support NFS v4 for the moment. I can switch to Linux Server but what's the best way in practice not only in theory?

Thanks in advance
 
Old 01-20-2012, 09:08 AM   #2
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,417

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
I would be generally looking for a dynamic solution with simple network interface. Generically, I tend to try to use scp / sftp if possible. It's all on a single TCP port etc. I would not really want to use NFS if it's for scheduled jobs, as, without the additional hassle of automount, the shares will always need to be mounted etc, which might not be something you can rely on etc. And NFSv3 is pretty ugly / complicated traffic flow wise.

What are your concerns about doing it with 100 systems though? There are other factors about this you've not discussed. Is this a push or a pull? Do you have control over it being one or another? How often is this happening? What makes it happen? Do you need to poke each machine somehow to make it happen? how large are these files? 1x1gb or 100x10mb or 10000x100kb?

using standard tools sftp (with preshared keys) on a cron job would be a good fit in my book, but then I would be interested in managing the ongoing configuration of this setup using a tool like puppet, as it's excellent. If you want to trigger these things centrally, mcollective might also be interesting to you, which integrates with puppet very well.

Last edited by acid_kewpie; 01-20-2012 at 09:11 AM.
 
1 members found this post helpful.
Old 01-20-2012, 09:38 AM   #3
sscn
LQ Newbie
 
Registered: Jan 2006
Location: China
Distribution: Fedora
Posts: 28

Original Poster
Rep: Reputation: 0
Smile

Quote:
Originally Posted by acid_kewpie View Post
I would be generally looking for a dynamic solution with simple network interface. Generically, I tend to try to use scp / sftp if possible. It's all on a single TCP port etc. I would not really want to use NFS if it's for scheduled jobs, as, without the additional hassle of automount, the shares will always need to be mounted etc, which might not be something you can rely on etc. And NFSv3 is pretty ugly / complicated traffic flow wise.
Ummm, I got the painful experience to deal with 100 PCs to mount one Windows NFS Server so as to execute the same main program from the Server on each client PC. 20 ~ 30 PCs shall be fine but it would be stuck sometimes once the client PC reaches to around 100. Program is very slow to response and lots of time out issues occurred. Now I am thinking of copying the program to local client PC and check the sync with Server every time the main program executed on each client PC.

Quote:
Originally Posted by acid_kewpie View Post
What are your concerns about doing it with 100 systems though? There are other factors about this you've not discussed. Is this a push or a pull? Do you have control over it being one or another? How often is this happening? What makes it happen? Do you need to poke each machine somehow to make it happen? how large are these files? 1x1gb or 100x10mb or 10000x100kb?
Actually, the logs created on each PC are copied to the NFS Server as well via NFS way. The data is in 10000x100kb on each PC. It would be easy to use one script to mount the Server on each client PC before starting the main program from NFS Server. This way, logs are copied to the Server in real time. But it caused high network traffic and the log transfer model is heavy related to network stability.

Quote:
Originally Posted by acid_kewpie View Post
using standard tools sftp (with preshared keys) on a cron job would be a good fit in my book, but then I would be interested in managing the ongoing configuration of this setup using a tool like puppet, as it's excellent. If you want to trigger these things centrally, mcollective might also be interesting to you, which integrates with puppet very well.
I will check puppet to see whether it can meet my expectation to deal with log transferring process.

Thanks a lot
 
Old 01-20-2012, 09:45 AM   #4
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,417

Rep: Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981Reputation: 1981
well puppet won't help you transfer the files, it will let you deliver configuration to the clients, pull various strings etc, but it wouldn't be involved in the live running of the application, only putting it in place, managing possible cron jobs etc..
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
softwares that can be used to monitor downloads in ubuntu client PCs from server PC ifranklin Ubuntu 1 08-30-2011 05:31 PM
Client pcs cannot resolve IP from DHCP server shipon_97 Linux - Enterprise 1 08-09-2007 12:54 PM
fax transfer problem between Hylafax server in Linux and C# client of windows asifjavaidy2k Linux - Software 0 02-09-2007 06:32 AM
[most secure, open source, client/server file transfer program] Synesthesia Linux - Software 5 11-28-2005 08:20 PM
Why cant I transfer mail from server to client? artz Linux - Networking 10 08-09-2004 05:39 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 02:17 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration