LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices



Reply
 
Search this Thread
Old 04-29-2014, 09:00 AM   #1
arazjan
LQ Newbie
 
Registered: Apr 2014
Posts: 4

Rep: Reputation: Disabled
implement a Linux based network scenario


I need to generate the traffic, send it to the server, analyse/process the traffic using an application in Linux Debian, get the "Log" and see at what traffic rate the application actually is unable to function properly and needs to drop part of the traffic(packet loss/drop) because of the high load.

I need to understand how can I manage this implementation to have any results.

The idea is to compare the dedicated server to an application versus virtualised apps on the same server that VMs run on it with the same app running on each.

I highly appreciate your time and guide.
 
Old 04-29-2014, 02:53 PM   #2
rtmistler
Senior Member
 
Registered: Mar 2011
Location: Milford, MA. USA
Distribution: Angstrom, Debian, Ubuntu, MINT
Posts: 1,445
Blog Entries: 8

Rep: Reputation: 526Reputation: 526Reputation: 526Reputation: 526Reputation: 526Reputation: 526
Writing an application or script to generate traffic from one or several clients, to your server is one thing.

Writing an application or script, also modifying the server itself to analyze the incoming traffic is another thing.

Both of these things are concepts which seem applicable to what you're talking about here.

The question becomes one of: What levels of controls or alterations are applicable versus what existing (if any) things need to stay the same?

By which I mean:
  1. You have a server, if you intend it to run normally, servicing say HTTP, FTP, or other traffic as it always does and you wish to evaluate it's capabilities, specifically it's limitations; then you don't have flexibility to change much excepting you can monitor the CPU usage, the memory, the network usage, and perhaps the number of sessions or transactions which are passing through it and keep those metrics in one or several logs; some of which would be dependent on the available information from the server based daemons or applications their selfs.
  2. You have one or more clients; you can choose to make test applications which send very large amounts and very fast data of one or many types, or you intend to just increase standard application sessions by requesting many open sessions via a script using real data; like you telnet large files, FTP large files, or employ data intensive web sessions, streaming video, stuff like that
So that's where I'm going with my questions: How far do you intend or wish to go with this? What limitations will stop things from going further? Not - "I want to stop when the server is unable to function properly", but rather, "I can't write code" or "I can't, or shouldn't modify the standard server operations - I wish to test it in it's native behaviors, therefore I can keep metrics but need to stress it by varying what the clients do to it."

The simple way if the server has to stay the same is to try to add monitoring stuff at the server, employ the capabilities of the daemon logs for the various server-based apps/services - figure a script way to collaborate that and compute your servicing capabilities. Then grow that as you learn more about measuring your metrics. CPU, memory, disk, network usage statistics, gather those. Gather information on the number of sessions and the bandwidth being used by each client or session, or both.

From the client(s) perspective, it's either scripts to use standard transfer methods in iteration to simulate heavy real-world traffic. The more variations, the better. Perhaps choose from a table of tests and randomly select the table entry for each next test and run this on several clients, trying to ensure that they all have some level of pseudo-randomness as far as the data content and the choices of tests which they run. Say one test is FTP, do so from a specific directory, and do so in both directions - then have the contents of each client directory be different. Same thing for things like live video; maybe obtain a list of available video content on the server and randomly select to play a video, then have all clients do that when their random test selection is streaming video where they get or send a random video. Also test with forcible data; fast, iterative data no pauses, sending as fast as possible to stress test the server's capabilities. Get a large random file and send it repeatedly many times, maybe have a few clients do this, and do this with several sessions, several files; hammer away at the server. I'd do the stress testing and the random live testing at the same time. If you can't reach your server limits, then you need more clients, or more tests running from your clients, continue multiplying until you reach the limits.
 
1 members found this post helpful.
Old 04-30-2014, 05:42 AM   #3
arazjan
LQ Newbie
 
Registered: Apr 2014
Posts: 4

Original Poster
Rep: Reputation: Disabled
Thank you for your very broad and very very helpful answer.
I am not familiar with programming that much, specially when it comes to Linux kernel details.
I want to implement this scenario as a practical work for my thesis.
I have high hopes that someone has done similar implementation and I can use the help with less need to search and learn because I am really busy doing all works together plus studies and everything, albeit I'd love to learn and understand as much as possible since the subject is what I like so much.
 
Old 04-30-2014, 08:12 AM   #4
rtmistler
Senior Member
 
Registered: Mar 2011
Location: Milford, MA. USA
Distribution: Angstrom, Debian, Ubuntu, MINT
Posts: 1,445
Blog Entries: 8

Rep: Reputation: 526Reputation: 526Reputation: 526Reputation: 526Reputation: 526Reputation: 526
Quote:
Originally Posted by arazjan View Post
Thank you for your very broad and very very helpful answer.
I am not familiar with programming that much, specially when it comes to Linux kernel details.
I want to implement this scenario as a practical work for my thesis.
I have high hopes that someone has done similar implementation and I can use the help with less need to search and learn because I am really busy doing all works together plus studies and everything, albeit I'd love to learn and understand as much as possible since the subject is what I like so much.
I suggest you seek guidance from your advisor on this.

Do your own work. If there are things which you're not expected to develop as part of your project, then your advisor will give you the test resources. If you're expected to specify and create your own test resources, then you'd better get a lot more familiar with programming in some form. Realize that everyone's really busy, asking for help and then stating that you're too busy to do your own work comes across sounding very badly. But that is why I raise the point that if your main focus was not to develop tests, then your advisor can clarify that for you. If you are supposed to do that work, then actually do it and don't look for a shortcut. If this is a technical masters degree related to IT, then truly and honestly you are well advised to get familiar with programming in the form of scripting. It's not all about installing and configuring software, you'll have to develop utilities to accomplish many feats in that line of work.
 
1 members found this post helpful.
Old 04-30-2014, 06:17 PM   #5
Lantzvillian
Member
 
Registered: Oct 2007
Location: BC, Canada
Distribution: Fedora, Debian
Posts: 210

Rep: Reputation: 41
I'll add a few notes:
* Userspace logging is painful - but fairly fast if you write a good server application to handle NFLOG packets from iptables
* Libpcap is tooooooo slow (and you will miss packets).
* Distributing logging and trimming your input so you can handle processing.

There is alot of IEEE research and papers on this. You may also be able to find resources from Sans as well to assist you.

As Rtmistler stated - programming is a good skill when your trying to determine tests (or run them). Don't state your busy either, I am too If you are doing any degree, I recommend you do a through job and go above/beyond expectations... It will prepare you for the real world and add credibility to your degree program.

Also before I forget, there alot of things affecting logging performance:
* Hardware
* kernel/driver implementations/optimization
* Userspace schedulers (IMHO not deterministic)
* VM networking (which is flakey and not deterministic IMHO)

Last edited by Lantzvillian; 04-30-2014 at 06:19 PM.
 
1 members found this post helpful.
Old 05-08-2014, 06:45 AM   #6
arazjan
LQ Newbie
 
Registered: Apr 2014
Posts: 4

Original Poster
Rep: Reputation: Disabled
I have a question:
I need to send a traffic from one node to another node in the same LAN, to test an application like iPtables and log the performance.
What software do you suggest? There are plenty of applications when I search.

So I need to generate traffic, or replay a captured traffic (for example a pcap file) and send it to the destination ip.
 
Old 05-09-2014, 11:57 AM   #7
Lantzvillian
Member
 
Registered: Oct 2007
Location: BC, Canada
Distribution: Fedora, Debian
Posts: 210

Rep: Reputation: 41
What kind of performance are you looking to test?
* Latency
* Throughput
* Jitter
* Per protocol performance
* Max connections
* CPU load
* Dropped packets

You had better talk to your advisor about this - you don't simply replay packets or generate them... you need real hardware that is designed for network testing and its worth alot of money. Logging/accounting packets also will take a toll on your performance results as well!

Have you even performed a literature review or started writing a project/thesis proposal?
 
1 members found this post helpful.
Old 05-09-2014, 12:22 PM   #8
arazjan
LQ Newbie
 
Registered: Apr 2014
Posts: 4

Original Poster
Rep: Reputation: Disabled
Dear Lantzvillian, thank you very much for your answer,

I am doing my thesis step by step, of course your hints help save time significantly.

So I managed to do what I asked in previous posts and send/receive traffic in the network between nodes in desired speeds and with parameters we need.

Right now I am looking for solutions to use as a software/application that processes the traffic (requests/packets), and gives me statistics of latency and packet drop rate. The Latency and dropped packets are the performance items I need to look at.

Yes, I work under my professor's supervision, he looks at my works and gives hints to not go in wrong directions.

Thank you again for your kind answer.
 
Old 05-09-2014, 01:16 PM   #9
rtmistler
Senior Member
 
Registered: Mar 2011
Location: Milford, MA. USA
Distribution: Angstrom, Debian, Ubuntu, MINT
Posts: 1,445
Blog Entries: 8

Rep: Reputation: 526Reputation: 526Reputation: 526Reputation: 526Reputation: 526Reputation: 526
Arazjan,

At this point I think I might recommend that you try some of the software utilities which you've located with your web searching. First, evaluate which ones are most likely similar to what you feel you want, and then give them a try. Because I feel that the metrics which you're looking to see are not uncommon ones and therefore many of the tools you may find available on the web will provide you with metrics, or even similar metrics to the ones which you're looking to attain. Virtually any capture utility will attempt to place timestamps on the captured packets, many of them will display either from trace start time, or packet to packet time differences; the real issues there will likely be the precision which they give you those timestamps, as well as the accuracy of those timestamps. They can give you timestamps down to say milli-seconds of precision, but will they really be "accurate" to the milli-second? Good chance not unless it is an expensive piece of test equipment. You'll have to make some determinations as to whether or not a given utility is helpful.

I also believe if you give a few of these a try, evaluate the results from there and make some determination whether or not they're giving you good measurements versus not and then approach your advisor with that information; they may help guide you on some better avenues with your approach.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Transition from Windows-based network to Linux-based larynx1982 Linux - Software 8 02-19-2011 08:10 AM
How to implement Logrotate mechansim based on Size? shettigarac Linux - General 1 11-18-2009 06:33 AM
LXer: Implement a Linux Thin Client Network with New Book LXer Syndicated Linux News 0 08-23-2007 10:11 PM
Network Scenario aqman Linux - General 1 12-11-2004 03:20 PM
How to implement network protocol in Linux? jamespayne Programming 2 12-03-2004 02:29 PM


All times are GMT -5. The time now is 05:51 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration