LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 01-10-2010, 01:53 PM   #1
cigarette
LQ Newbie
 
Registered: Jan 2010
Posts: 9

Rep: Reputation: 0
Realtime bandwidth management?


Hello,

I'm a home user with a Laptop running Fedora and a Desktop running Debian.

The laptop is running 24/7 and I'm always downloading something from legaltorrents.com using rTorrent which is set to use all the available bandwidth. I don't wanna set a permanent limit on this.

The problem is that my connection is 512Kbps so it makes browsing difficult when torrents are sucking up all the bandwidth (that's normal, I know) but I'm not browsing 24/7.

Is there some sort of realtime bandwidth management apps (free or commercial) that will put a limit on rTorrent only when I'm browsing and then set it to use max bandwidth again when I finish browsing. It would be great if I could make it use all my available bandwidth when I load a youtube link and half the bandwidth when I visit other sites. The rest of the time, it can allow rtorrent to suck all the bandwidth.

It should also prioritize DNS (I'm already running dnsmasq for local caching)

I tried trickle (http://www.monkey.org/~marius/trickle/) but it doesn't work this way.
 
Old 01-10-2010, 02:02 PM   #2
nimnull22
Senior Member
 
Registered: Jul 2009
Distribution: OpenSuse 11.1, Fedora 14, Ubuntu 12.04/12.10, FreeBSD 9.0
Posts: 1,571

Rep: Reputation: 92
How your laptop and debian connected to internet? What through?
 
Old 01-10-2010, 02:10 PM   #3
cigarette
LQ Newbie
 
Registered: Jan 2010
Posts: 9

Original Poster
Rep: Reputation: 0
Fedora (laptop) is connected via WiFi to a wireless router.
Debian (Desktop) is connected via crossover cable to Laptop.

Last edited by cigarette; 01-10-2010 at 02:11 PM.
 
Old 01-10-2010, 02:36 PM   #4
nimnull22
Senior Member
 
Registered: Jul 2009
Distribution: OpenSuse 11.1, Fedora 14, Ubuntu 12.04/12.10, FreeBSD 9.0
Posts: 1,571

Rep: Reputation: 92
In your case, I think, only "traffic contol = tc" "man tc" can help.
It is not dynamic, but I believe can give to some traffic flow more priority.
You need to find good manual.
 
Old 01-10-2010, 02:45 PM   #5
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 556Reputation: 556Reputation: 556Reputation: 556Reputation: 556Reputation: 556
Yes, tc is the thing, along with maybe the CFQ queueing discipline (a kernel module).

The man page for tc was rather sparse last time I looked, and really good documentation was rare too, but there are some examples of using it, floating around the net.

As to it being "dynamic", I'm not sure what that might mean precisely in this context, but, like nimnull suggests, it can allow you to give priority to one traffic or another, for example:

: you have 512 units to deal with (kbits, mbits, whatever...)
: you allot a max of 512 kbits to bittorrent
: you allot a max of 256 kbits to http/https/whatever you use for browsing.
:: you give the browsing protocols priority

The above pseudo-example says:
A) "if I'm browsing, give me my <=256 kbits FIRST, before allotting whatever is left, to bittorrent"
B) "If I'm NOT browsing, bittorrent can have all the bandwidth."

This allows bittorrent to suck the network dry if it's the only thing using it, BUT once you start browsing, you get priority over the bandwidth, up to 256<units>, so bittorrent will still run, just not as fast.

Sorry for the vague pseudo code but if you want a script example, which I have <somewhere> just ask, then give me some time to locate it.

Sasha
 
Old 01-10-2010, 04:34 PM   #6
jeff_k
Member
 
Registered: Jan 2008
Location: San Diego, CA USA
Distribution: Debian / Ubuntu
Posts: 51

Rep: Reputation: 17
You can install wondershaper, which uses tc, and is pre-configured. You can edit the config file to optimize for various dynamic traffic scenarios.
http://lartc.org/wondershaper/


I used it, but found something I like even more:
wshaper.hfsc
http://sysphere.org/~anrxc/j/index-page4.html describes the script located at:
http://sysphere.org/~anrxc/local/sources/wshaper.hfsc
This script is also able to be tweaked for ip addresses, services, ports, etc. that you want to allocate bandwidth / low latency to.
Best Regards
 
Old 01-10-2010, 08:08 PM   #7
cigarette
LQ Newbie
 
Registered: Jan 2010
Posts: 9

Original Poster
Rep: Reputation: 0
Thanks for the tip, everyone. I'm running wshaper.hfsc right now. I'll report back after a day or two of using it and let you guys know how it's working out for me. I can notice the improvement on DNS already.

My rtorrent accepts connections on port 33333, but when it makes outgoing connections to other peers, it will connect to whatever port they're using. So it won't set the lowest priority on those connections but instead categorize it as DATA, and incoming connections categorized as P2P right? It's still an improvement, though.

@GrapefruiTgirl: Yes, please. It'll be great to look at your script too.
 
Old 01-10-2010, 08:13 PM   #8
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 556Reputation: 556Reputation: 556Reputation: 556Reputation: 556Reputation: 556
OK, since you ask, I'll dig it up and post it for you. I think it's on my firewall machine which currently has no net connection, so be patient and I'll post it a bit later on. Remember, it's not very sophisticated -- but you can examine it to at least see the syntax of how to use tc.

Back in a bit..
 
Old 01-10-2010, 09:14 PM   #9
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 556Reputation: 556Reputation: 556Reputation: 556Reputation: 556Reputation: 556
my tc qdisc script

I've left in the /proc stuff I also had this script doing, as well as the qdisc setup.
NOTE: I earlier mentioned the CFQ (Completely Fair Queueing) discipline, but I was mistaken (like I said (I think) I haven't used this in some time ) -- the qdisc (Queueing Discipline) I actually was using is the HTB module (Hierarchical Token Bucket) instead.

The way this script works, is I would have my firewall call it after the firewall was up. This script takes two arguments:

$1 is the external internet-connected interface on which to place the qdiscs, and...

$2 is a start/stop/restart argument.

So like:

shell# ./scriptname <nic> [start|stop|restart]

I used this with a dial-up connection mostly (ppp0) which had such low bandwidth that using this was almost useless -- keyword being "almost" -- because it DID actually help the two machines sharing the dial-up connection, to not starve each other.
Also, another key benefit of this setup, is that by regulating the consumed bandwidth on MY end such that it stayed *just* below the maximum possible bandwidth that our connection from the ISP could handle, it caused downloads (file transfers) to go at a nice steady rate from start to end, rather than that effect we often see where the transfer starts FAST, then gets slow-slow-slow, stops for a second, than goes FAST again..... It does this be 'teaching' the ISP that you only want traffic to arrive at a specified MAX rate and not to exceed that rate. This results in the ISP NOT overloading the connection, pausing, resending dropped packets, repeat....

Here it is:

OH, and P.S. for clarity -- the 192.168.0.88 address was my desktop machine, connected to eth0 on the firewall box on which this script ran, and 192.168.2.88 was for my roommates machine, connected to eth1 on the firewall box.

Code:
#!/bin/sh
# /etc/rc.d/rc.tcpip_proc_customize
#!/bin/bash
#
# to execute:
# <scriptname> <dev> <start|stop|restart>
# example: bash# ./script ppp0 start
#
configuration_start() {
echo "Setting /proc/ipv4 custom configuration, activating qdiscs."

  if [ -f /proc/sys/net/ipv4/ip_forward ]; then
    echo "Activating IPv4 packet forwarding."
    echo 1 > /proc/sys/net/ipv4/ip_forward
  fi

  if [ -f /proc/sys/net/ipv4/ip_dynaddr ]; then
    echo "Activating IPv4 dynamic ip address."
    echo 1 > /proc/sys/net/ipv4/ip_dynaddr
  fi

  if [ -f /proc/sys/net/ipv4/tcp_sack ]; then
    echo "Activating IPv4 selective ACK."
    echo 1 > /proc/sys/net/ipv4/tcp_sack
  fi

  if [ -f /proc/sys/net/ipv4/tcp_keepalive_time ]; then
    echo "Setting IPv4 TCP-keepalive sends to every 30 minutes."
    echo 1800 > /proc/sys/net/ipv4/tcp_keepalive_time
  fi

  if [ -f /proc/sys/net/ipv4/tcp_keepalive_intvl ]; then
    echo "Setting IPv4 TCP-keepalive re-send unACKeds to 30 second intervals."
    echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
  fi

  if [ -f /proc/sys/net/ipv4/tcp_keepalive_probes ]; then
    echo "Setting IPv4 TCP-keepalive repeat re-send unACKeds to 20 retries."
    echo 20 > /proc/sys/net/ipv4/tcp_keepalive_probes
  fi

  if [ -f /proc/sys/net/ipv4/tcp_slow_start_after_idle ]; then
    echo "Setting IPv4 TCP-slow start after idle, to false."
    echo 0 > /proc/sys/net/ipv4/tcp_slow_start_after_idle
  fi

# Begin HTB qdisc configuration:

# first define our qdiscs
# here is the main one, the root qdisk which is on ppp0
# the handle is 1:
tc qdisc add dev $1 root handle 1: htb 
# now, the parent class (handle 1:1) is the main class on qdisc 1: above
# and has available the full bandwidth
tc class add dev $1 parent 1:  classid 1:1  htb rate 2.5kbps burst 4k
# the 4 classids below are supposed to share the bw of the parent class
# eth0 classes:
# classid 1:10 has a lower priority 2  and lower avail bw; will be for incoming ftp
# classid 1:11 has higher priority 1, and higher avail bw; will be for other (http etc..) incoming
tc class add dev $1 parent 1:1 classid 1:10 htb rate 1.26kbps ceil 2.5kbps prio 3
tc class add dev $1 parent 1:1 classid 1:11 htb rate 1.26kbps ceil 2.5kbps prio 2
# eth1 classes, same idea as above on eth0
tc class add dev $1 parent 1:1 classid 1:12 htb rate 1.26kbps ceil 2.5kbps prio 3
tc class add dev $1 parent 1:1 classid 1:13 htb rate 1.26kbps ceil 2.5kbps prio 2



tc qdisc add dev $1 handle ffff: ingress
tc filter add dev $1 parent ffff: protocol ip prio 1 u32 \
	match ip src 0.0.0.0/0 police rate 2.5kbps burst 4k drop flowid :1


# all routes get stochastic fairness: (what does that mean??)
tc qdisc add dev $1 parent  1:10 handle 10: sfq perturb 10
tc qdisc add dev $1 parent  1:11 handle 11: sfq perturb 10
tc qdisc add dev $1 parent  1:12 handle 12: sfq perturb 10
tc qdisc add dev $1 parent  1:13 handle 13: sfq perturb 10

# prioritize ACK packets: put ACKs into the 'interactive' class
# which means they are given high priority; this increases responsiveness
# when using www or similar services. ACK traffic gets bw directly from the  main parent
# class (classid 1:1) hence the "flowid 1:1"
tc filter add dev $1 parent 1: protocol ip prio 1 u32 \
	match ip protocol 6 0xff \
	match u8 0x05 0x0f at 0 \
	match u16 0x0000 0xffc0 at 2 \
	match u8 0x10 0xff at 33 \
	flowid 1:1


# put eth0 ftp traffic traffic into lower priority eth0 category
tc filter add dev $1 protocol ip parent 1:0 prio 3 u32 \
	match ip dst 192.168.0.88 \
	match ip protocol 6 0xff \
	flowid 10:
# all other eth0 traffic into higher priority
tc filter add dev $1 protocol ip parent 1:0 prio 2 u32 \
	match ip dst 192.168.0.88 \
	flowid 11:
# and repeat same for eth1:
tc filter add dev $1 protocol ip parent 1:0 prio 3 u32 \
	match ip dst 192.168.2.88 \
	match ip protocol 6 0xff \
	flowid 12:
tc filter add dev $1 protocol ip parent 1:0 prio 2 u32 \
	match ip dst 192.168.2.88 \
	flowid 13:

# Ingress policing; This here accepts packets at near the maximum possible
# rate that our ppp0 connection can handle, and drops everything it cannot handle;
# meanwhile it tells the other end (the ISP) that we can only receive data at
# *this* rate, and so they should slow down their transmission and not try to queue
# so much data and jam it over the line to us because we will only drop it anyway. 
# This polices ALL inbound traffic on ppp0 and keeps it at or near our max
# receiving rate.

}


configuration_stop() {

echo "rc.optimize_tcpip: restoring default /proc/ipv4 settings, removing qdiscs."

  if [ -f /proc/sys/net/ipv4/ip_forward ]; then
    echo 0 > /proc/sys/net/ipv4/ip_forward
  fi

  if [ -f /proc/sys/net/ipv4/ip_dynaddr ]; then
    echo 0 > /proc/sys/net/ipv4/ip_dynaddr
  fi

  if [ -f /proc/sys/net/ipv4/tcp_sack ]; then
    echo 0 > /proc/sys/net/ipv4/tcp_sack
  fi

  if [ -f /proc/sys/net/ipv4/tcp_keepalive_time ]; then
    echo 7200 > /proc/sys/net/ipv4/tcp_keepalive_time
  fi

  if [ -f /proc/sys/net/ipv4/tcp_keepalive_intvl ]; then
    echo 75 > /proc/sys/net/ipv4/tcp_keepalive_intvl
  fi

  if [ -f /proc/sys/net/ipv4/tcp_keepalive_probes ]; then
    echo 9 > /proc/sys/net/ipv4/tcp_keepalive_probes
  fi

  if [ -f /proc/sys/net/ipv4/tcp_slow_start_after_idle ]; then
    echo 1 > /proc/sys/net/ipv4/tcp_slow_start_after_idle
  fi

# clean existing down- and uplink qdiscs, hide errors
tc qdisc del dev $1 root 2>&1 1> /dev/null
tc qdisc del dev $1 ingress 2>&1 1> /dev/null
}


# Reconfig the configuration:
configuration_restart() {
  configuration_stop $1
  sleep 1
  configuration_start $1
}


if [ ! "$1" -o ! "$2" ]; then
echo "rc.optimize_tcpip: missing argument-- please specify <dev> [start | stop | restart]"
exit 1
fi

case "$2" in
'start')
  configuration_start $1
  ;;
'stop')
  configuration_stop $1
  ;;
'restart')
  configuration_restart $1
  ;;
  *)
  echo "rc.optimize_tcpip: please specify $1 [ start | stop | restart ]"
;;
esac

#EOF
If you have questions, I'll do my best to answer them, but again, I haven't messed with this in quite a while, so I'm kinda rusty.

Sasha!

Last edited by GrapefruiTgirl; 01-10-2010 at 09:17 PM.
 
Old 01-11-2010, 01:16 PM   #10
jeff_k
Member
 
Registered: Jan 2008
Location: San Diego, CA USA
Distribution: Debian / Ubuntu
Posts: 51

Rep: Reputation: 17
Cigarette, regarding the wondershaper or wshaper.hfsc scripts, they need to be tweaked in order to work. The basic procedure is that you need to determine your download and upload speeds, so that you can enter them into the script. This is important, particularly so that the upload queue in your cable/DSL modem is never filled. If it is, it will exhibit the characteristics that Sasha describes. The way that you want to determine this is by NOT yet using the script, go to a download/upload internet speed test, and test your upload and download speeds. When you get this well-identified, then enter those into the scripts (or in the case of wondershaper, you need to include it on the command line). If you want to see the latency improvement that the scripts will provide, then before the scripts are active, open another shell and ping somewhere while the upload speed test is running. You will see horrible results. Now start up the wshaper.hfsc script. Run the upload speed test again; the upload speed, if you tweaked it correctly, will be slightly slower (maybe 80-90%), but the pings will be MUCH faster and more consistent. The purpose of the scripts is to slightly slow down your regular speeds, so that the modem's buffers are not filled, and so that you have control over which types of traffic/services in your LAN have access to the bandwidth. As you have already done, look for the ports, etc. that your services are running on and figure out which priority these services will get. If you want a bit more of a description of the hfsc discipline (hierarchical fair service curve), a good description is at:
http://linux-ip.net/articles/hfsc.en/ and a description of the various queueing approaches is at:
http://linux-ip.net/articles/Traffic...ul-qdiscs.html
Cigarette, even though you don't have control over the destination ports for your torrent peers, do you have control over the source port? If so, I believe you can specify that so that you can get the torrent traffic in the lowest priority category.

I got interested in implementing traffic control on my LAN when my son, who is an avid gamer, was complaining about the latency of his games, especially if I was doing a lot of downloading. Using hfsc with the proper tweaks, he now consistently hosts games (the host is the one that the internet game server determines has the best latency among the participants in an online game). There is no slowdown while several others in the family are surfing the net, or even if I am downloading a large file, or running torrents. It seems to run as advertised (believe me, I hear about it if the latency goes up).
Regards, Jeff
 
Old 01-13-2010, 06:58 PM   #11
cigarette
LQ Newbie
 
Registered: Jan 2010
Posts: 9

Original Poster
Rep: Reputation: 0
Hello,

I'd done that already, that is setting my upload and download speed in the script. Thanks for the explanation.

I'm not sure how to set the source port for outgoing connections in rtorrent. I'll try to find out.
 
Old 01-14-2010, 03:18 PM   #12
jeff_k
Member
 
Registered: Jan 2008
Location: San Diego, CA USA
Distribution: Debian / Ubuntu
Posts: 51

Rep: Reputation: 17
Cigarette, I did a little looking around at posts, and this post is from 2006, so I don't know if rtorrent has upgraded its capability to specify source ports, but:

http://osdir.com/ml/network.bit-torr.../msg00037.html
quoting the post:
"I don't think rtorrent has an option to do this. You can restrict which
port it _listens_ on for incoming connections by using the -p option
Viktor referred to, or port_range in the configuration file.

You can also have rtorrent bind _outgoing_ (as well as listening)
connections to a specific IP (-b, bind), so you might be able to
simulate the behaviour you're after by adding an additional IP to your
local system and getting rtorrent to use that; then, allow that IP to
make connections to the internet.

The Linux netfilter suite (iptables) can also permit/deny traffic based
on the local process ID amongst other factors; so you could potentially
set up the local firewall to allow the rtorrent process to connect to
whatever it wants."

If you want, you can specify a different IP address (for example, bind eth0:1 to another LAN address) and then add rules in wshaper.hfsc to lower the priority of that IP address.

If you want to, you can tag packets associated with the rtorrent PID using iptables for a specific TOS value, and the wshaper script can be amended to deal with packets of a given TOS value differently. You need the iptables OWNER match for that, described in the document located here:
http://www.frozentux.net/iptables-tu...tml#OWNERMATCH

The first question to answer is, is getting the rtorrent upload traffic moved from the data category to the lowest priority category important enough to work one of the above solutions? You still have general surfing and any required low latency applications getting priority now.
 
Old 01-14-2010, 03:21 PM   #13
GrapefruiTgirl
LQ Guru
 
Registered: Dec 2006
Location: underground
Distribution: Slackware64
Posts: 7,594

Rep: Reputation: 556Reputation: 556Reputation: 556Reputation: 556Reputation: 556Reputation: 556
Quote:
Originally Posted by jeff_k View Post
...
The Linux netfilter suite (iptables) can also permit/deny traffic based
on the local process ID amongst other factors; so you could potentially...
That gave me an idea.. Running rtorrent as a unique user (create a user/group account for it) would allow iptables to match traffic specific to that user.
 
Old 01-14-2010, 03:28 PM   #14
nimnull22
Senior Member
 
Registered: Jul 2009
Distribution: OpenSuse 11.1, Fedora 14, Ubuntu 12.04/12.10, FreeBSD 9.0
Posts: 1,571

Rep: Reputation: 92
Quote:
Originally Posted by jeff_k View Post

If you want to, you can tag packets associated with the rtorrent PID using iptables for a specific TOS value, and the wshaper script can be amended to deal with packets of a given TOS value differently. You need the iptables OWNER match for that, described in the document located here:
No, that was removed from IPTABLES, all you can do is:

owner
This module attempts to match various characteristics of the packet creator, for locally gener-
ated packets. This match is only valid in the OUTPUT and POSTROUTING chains. Forwarded packets
do not have any socket associated with them. Packets from kernel threads do have a socket, but
usually no owner.

[!] --uid-owner username

[!] --uid-owner userid[-userid]
Matches if the packet socket's file structure (if it has one) is owned by the given user.
You may also specify a numerical UID, or an UID range.

[!] --gid-owner groupname

[!] --gid-owner groupid[-groupid]
Matches if the packet socket's file structure is owned by the given group. You may also
specify a numerical GID, or a GID range.

[!] --socket-exists
Matches if the packet is associated with a socket.


It looks like kernel developers do not really think|care about security from "INSIDE".
 
Old 01-14-2010, 05:27 PM   #15
jeff_k
Member
 
Registered: Jan 2008
Location: San Diego, CA USA
Distribution: Debian / Ubuntu
Posts: 51

Rep: Reputation: 17
I wasn't aware that the OWNER match had been amended. Thanks

To summarize a couple of approaches that can work:

- have the rtorrent traffic run from a separate IP address, and within the wshaper script specify that traffic from this address gets lower priority;

- have the rtorrent processes run using a different UID/GID and then using the iptables OWNER match, tag the packets with a specific TOS value that can be handled as lower priority traffic within the script.

Last edited by jeff_k; 01-14-2010 at 05:28 PM. Reason: correction of spelling
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Bandwidth Management pHreak Linux - Newbie 6 01-05-2010 07:06 AM
Bandwidth management uwa45 Linux - Networking 2 06-15-2009 11:05 PM
IP bandwidth management RandomLinuxNewb Linux - Networking 1 07-23-2005 11:48 PM
bandwidth management ,, apenguinlinux Debian 1 02-18-2005 05:01 AM
bandwidth management . sam007 Linux - Distributions 0 12-16-2003 05:31 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 03:21 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration