implement a Linux based network scenario

I need to generate the traffic, send it to the server, analyse/process the traffic using an application in Linux Debian, get the "Log" and see at what traffic rate the application actually is unable to function properly and needs to drop part of the traffic(packet loss/drop) because of the high load.

I need to understand how can I manage this implementation to have any results.

The idea is to compare the dedicated server to an application versus virtualised apps on the same server that VMs run on it with the same app running on each.

I highly appreciate your time and guide.

Writing an application or script to generate traffic from one or several clients, to your server is one thing.

Writing an application or script, also modifying the server itself to analyze the incoming traffic is another thing.

Both of these things are concepts which seem applicable to what you're talking about here.

The question becomes one of: What levels of controls or alterations are applicable versus what existing (if any) things need to stay the same?

By which I mean:

1. You have a server, if you intend it to run normally, servicing say HTTP, FTP, or other traffic as it always does and you wish to evaluate it's capabilities, specifically it's limitations; then you don't have flexibility to change much excepting you can monitor the CPU usage, the memory, the network usage, and perhaps the number of sessions or transactions which are passing through it and keep those metrics in one or several logs; some of which would be dependent on the available information from the server based daemons or applications their selfs.
2. You have one or more clients; you can choose to make test applications which send very large amounts and very fast data of one or many types, or you intend to just increase standard application sessions by requesting many open sessions via a script using real data; like you telnet large files, FTP large files, or employ data intensive web sessions, streaming video, stuff like that

So that's where I'm going with my questions: How far do you intend or wish to go with this? What limitations will stop things from going further? Not - "I want to stop when the server is unable to function properly", but rather, "I can't write code" or "I can't, or shouldn't modify the standard server operations - I wish to test it in it's native behaviors, therefore I can keep metrics but need to stress it by varying what the clients do to it."

The simple way if the server has to stay the same is to try to add monitoring stuff at the server, employ the capabilities of the daemon logs for the various server-based apps/services - figure a script way to collaborate that and compute your servicing capabilities. Then grow that as you learn more about measuring your metrics. CPU, memory, disk, network usage statistics, gather those. Gather information on the number of sessions and the bandwidth being used by each client or session, or both.

From the client(s) perspective, it's either scripts to use standard transfer methods in iteration to simulate heavy real-world traffic. The more variations, the better. Perhaps choose from a table of tests and randomly select the table entry for each next test and run this on several clients, trying to ensure that they all have some level of pseudo-randomness as far as the data content and the choices of tests which they run. Say one test is FTP, do so from a specific directory, and do so in both directions - then have the contents of each client directory be different. Same thing for things like live video; maybe obtain a list of available video content on the server and randomly select to play a video, then have all clients do that when their random test selection is streaming video where they get or send a random video. Also test with forcible data; fast, iterative data no pauses, sending as fast as possible to stress test the server's capabilities. Get a large random file and send it repeatedly many times, maybe have a few clients do this, and do this with several sessions, several files; hammer away at the server. I'd do the stress testing and the random live testing at the same time. If you can't reach your server limits, then you need more clients, or more tests running from your clients, continue multiplying until you reach the limits.

Thank you for your very broad and very very helpful answer.
I am not familiar with programming that much, specially when it comes to Linux kernel details.
I want to implement this scenario as a practical work for my thesis.
I have high hopes that someone has done similar implementation and I can use the help with less need to search and learn because I am really busy doing all works together plus studies and everything, albeit I'd love to learn and understand as much as possible since the subject is what I like so much.

Originally Posted by arazjan Thank you for your very broad and very very helpful answer. I am not familiar with programming that much, specially when it comes to Linux kernel details. I want to implement this scenario as a practical work for my thesis. I have high hopes that someone has done similar implementation and I can use the help with less need to search and learn because I am really busy doing all works together plus studies and everything, albeit I'd love to learn and understand as much as possible since the subject is what I like so much.

I suggest you seek guidance from your advisor on this.

Do your own work. If there are things which you're not expected to develop as part of your project, then your advisor will give you the test resources. If you're expected to specify and create your own test resources, then you'd better get a lot more familiar with programming in some form. Realize that everyone's really busy, asking for help and then stating that you're too busy to do your own work comes across sounding very badly. But that is why I raise the point that if your main focus was not to develop tests, then your advisor can clarify that for you. If you are supposed to do that work, then actually do it and don't look for a shortcut. If this is a technical masters degree related to IT, then truly and honestly you are well advised to get familiar with programming in the form of scripting. It's not all about installing and configuring software, you'll have to develop utilities to accomplish many feats in that line of work.

I'll add a few notes:
* Userspace logging is painful - but fairly fast if you write a good server application to handle NFLOG packets from iptables
* Libpcap is tooooooo slow (and you will miss packets).
* Distributing logging and trimming your input so you can handle processing.

There is alot of IEEE research and papers on this. You may also be able to find resources from Sans as well to assist you.

As Rtmistler stated - programming is a good skill when your trying to determine tests (or run them). Don't state your busy either, I am too If you are doing any degree, I recommend you do a through job and go above/beyond expectations... It will prepare you for the real world and add credibility to your degree program.

Also before I forget, there alot of things affecting logging performance:
* Hardware
* kernel/driver implementations/optimization
* Userspace schedulers (IMHO not deterministic)
* VM networking (which is flakey and not deterministic IMHO)

I have a question:
I need to send a traffic from one node to another node in the same LAN, to test an application like iPtables and log the performance.
What software do you suggest? There are plenty of applications when I search.

So I need to generate traffic, or replay a captured traffic (for example a pcap file) and send it to the destination ip.

What kind of performance are you looking to test?
* Latency
* Throughput
* Jitter
* Per protocol performance
* Max connections
* CPU load
* Dropped packets

You had better talk to your advisor about this - you don't simply replay packets or generate them... you need real hardware that is designed for network testing and its worth alot of money. Logging/accounting packets also will take a toll on your performance results as well!

Have you even performed a literature review or started writing a project/thesis proposal?

Dear Lantzvillian, thank you very much for your answer,

I am doing my thesis step by step, of course your hints help save time significantly.

So I managed to do what I asked in previous posts and send/receive traffic in the network between nodes in desired speeds and with parameters we need.

Right now I am looking for solutions to use as a software/application that processes the traffic (requests/packets), and gives me statistics of latency and packet drop rate. The Latency and dropped packets are the performance items I need to look at.

Yes, I work under my professor's supervision, he looks at my works and gives hints to not go in wrong directions.

Thank you again for your kind answer.

Arazjan,

At this point I think I might recommend that you try some of the software utilities which you've located with your web searching. First, evaluate which ones are most likely similar to what you feel you want, and then give them a try. Because I feel that the metrics which you're looking to see are not uncommon ones and therefore many of the tools you may find available on the web will provide you with metrics, or even similar metrics to the ones which you're looking to attain. Virtually any capture utility will attempt to place timestamps on the captured packets, many of them will display either from trace start time, or packet to packet time differences; the real issues there will likely be the precision which they give you those timestamps, as well as the accuracy of those timestamps. They can give you timestamps down to say milli-seconds of precision, but will they really be "accurate" to the milli-second? Good chance not unless it is an expensive piece of test equipment. You'll have to make some determinations as to whether or not a given utility is helpful.

I also believe if you give a few of these a try, evaluate the results from there and make some determination whether or not they're giving you good measurements versus not and then approach your advisor with that information; they may help guide you on some better avenues with your approach.