[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Problem getting correct rates from CBQ...



 I have an interesting situation I'm trying to test out which simply
won't behave. The objective is to provide access to various web servers
on a local network at different set bitrates, so as to simulate access
from low-bandwidth connections. The technique I'm trying to employ is to
bind a second (and 3rd and 4th) IP subnet to an existing ethernet
interface on the router (a linux 2.2.12-20 machine, with IPSec and
MSChap patches compiled in) and then bind each of the web servers to
each of the subnets, thereby creating a route from any workstation to
any webserver which must loop through the router. The intent is then to
apply schedulers and routing table entries to provide each of the
different throttles... everything seems to work fine except:

 1)	The data rates won't adjust with any degree of grain.
Specifically the machine seems to not want to limit below about
300KBytes/sec and generally won't adjust in any predictable manner...

 2) 	The schedulers don't discard any queued packets, so the router
gets severly clogged with pending packets and the server transmits into
the router at full wirespeed. This is not a nice effect for the other
traffic running through that router.

My scheduler configuration is intended to be like this:

-1:0	[eth0/.48.251,.49.251,.50.251,.51.251]
  |
  1:1-\
  |    -TBF 10Mbit   <--Flow 16-- (traffic to/from 192.168.48.0/24)
  |
  1:2-\
  |    -TBF 56Kbit   <--Flow 17-- (traffic to/from 192.168.49.0/24) 
  |
  1:3-\
  |    -TBF 128Kbit  <--Flow 18-- (traffic to/from 192.168.50.0/24)
  |
  1:4-\
       -TBF 1.54Mbit <--Flow 19-- (traffic to/from 192.168.51.0/24)


and I'm using the following to set it up:


# First we attach additional subnet interfaces to eth0
/sbin/ip addr add 192.168.49.251/24 dev eth0 broadcast 192.168.49.255
scope global
/sbin/ip addr add 192.168.50.251/24 dev eth0 broadcast 192.168.50.255
scope global
/sbin/ip addr add 192.168.51.251/24 dev eth0 broadcast 192.168.51.255
scope global


# Now we attach the qdisc to the eth0 device
/sbin/tc qdisc add dev eth0 root handle 1: cbq bandwidth 10Mbit cell 8
avpkt 1000 mpu 64


# Next we drop in a root Class Based Queue processor (called 1:1) for
full 10Mbit traffic
/sbin/tc class add dev eth0 parent 1:0 classid 1:1 cbq bandwidth 10Mbit
rate 10Mbit \
	allot 1514 cell 8 weight 1Mbit prio 5 maxburst 20 avpkt 1000


# Then we add branches for traffic control (1:2, 1:3, 1:4) at each of
the various throttled bandwidths 
/sbin/tc class add dev eth0 parent 1:1 classid 1:2 cbq bandwidth 10Mbit
rate 56Kbit \
	allot 1514 cell 8 weight 5Kbit prio 5 bounded maxburst 20 avpkt
1000 
/sbin/tc class add dev eth0 parent 1:1 classid 1:3 cbq bandwidth 10Mbit
rate 128Kbit \
	allot 1514 cell 8 weight 13Kbit prio 5 bounded maxburst 20 avpkt
1000 
/sbin/tc class add dev eth0 parent 1:1 classid 1:4 cbq bandwidth 10Mbit
rate 1544Kbit \
	allot 1514 cell 8 weight 154Kbit prio 5 bounded maxburst 20
avpkt 1000


# ??????? Do we add leaf Token Bucket Filters to each branch?
/sbin/tc qdisc add dev eth0 parent 1:2 tbf rate 56Kbit buffer 3Kb/8
limit 4Kb
/sbin/tc qdisc add dev eth0 parent 1:3 tbf rate 128Kbit buffer 5Kb/8
limit 8Kb
/sbin/tc qdisc add dev eth0 parent 1:4 tbf rate 1544Kbit buffer 10Kb/8
limit 15Kb


# And we attach filters to allow traffic to be routed into each of the
leafs defined previously
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 100 route to 16
flowid 1:1
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 100 route to 17
flowid 1:2
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 100 route to 18
flowid 1:3
/sbin/tc filter add dev eth0 parent 1:0 protocol ip prio 100 route to 19
flowid 1:4


# Finally we add routing rules to move traffic into the appropriate
queue
/sbin/ip route add 192.168.48.253 via 192.168.48.251 realm 16
/sbin/ip route add 192.168.49.253 via 192.168.49.251 realm 17
/sbin/ip route add 192.168.50.253 via 192.168.50.251 realm 18
/sbin/ip route add 192.168.51.253 via 192.168.51.251 realm 19


Noting that eth0 is a 100Mbit NIC, and the machine is a P100, I get the
following results from running various checks during a transfer from a
web server to a workstation, using FTP... (these results may be slightly
out of wack. They were obtained on a different day to the configuration
above):

[root@phoenix kb1]# /sbin/tc -s qdisc show dev eth0
qdisc tbf 8004: rate 1544Kbit burst 10Kb lat 31.6ms
 Sent 253625 bytes 4649 pkts (dropped 0, overlimits 0)

 qdisc tbf 8003: rate 128Kbit burst 5Kb lat 228.9ms
 Sent 515383 bytes 9493 pkts (dropped 0, overlimits 0)

 qdisc tbf 8002: rate 56Kbit burst 3Kb lat 174.4ms
 Sent 1222421 bytes 22057 pkts (dropped 0, overlimits 12272)
 backlog 702b 13p

 qdisc tbf 8001: rate 100Mbit burst 3053b lat 96us
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

 qdisc cbq 1: rate 100Mbit (bounded,isolated) prio no-transmit
 Sent 28067233 bytes 98769 pkts (dropped 0, overlimits 115381)
 backlog 13p
  borrowed 0 overactions 0 avgidle 62 undertime 0


[root@phoenix kb1]# /sbin/tc -s class show dev eth0
class cbq 1: root rate 100Mbit (bounded,isolated) prio no-transmit
 Sent 28447558 bytes 65473 pkts (dropped 0, overlimits 0)
  borrowed 0 overactions 0 avgidle 62 undertime 0
class cbq 1:1 parent 1: leaf 8001: rate 100Mbit prio 5
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
  borrowed 0 overactions 0 avgidle 62 undertime 0
class cbq 1:2 parent 1: leaf 8002: rate 7bps (bounded,isolated) prio 5
 Sent 1222853 bytes 22065 pkts (dropped 0, overlimits 108301)
 backlog 19p
  borrowed 0 overactions 11730 avgidle -1.29679e+06 undertime 1.9834e+06
class cbq 1:3 parent 1: leaf 8003: rate 128Kbit (bounded,isolated) prio
5
 Sent 515383 bytes 9493 pkts (dropped 0, overlimits 30540)
  borrowed 0 overactions 2748 avgidle 1.37069e+06 undertime 0
class cbq 1:4 parent 1: leaf 8004: rate 1544Kbit (bounded,isolated) prio
5
 Sent 253625 bytes 4649 pkts (dropped 0, overlimits 3839)
  borrowed 0 overactions 153 avgidle 112059 undertime 0


[root@phoenix kb1]# /sbin/tc -s filter show dev eth0
filter parent 1: protocol ip pref 100 route
filter parent 1: protocol ip pref 100 route fh 0xffff0010 flowid 1:1 to
16
filter parent 1: protocol ip pref 100 route fh 0xffff0011 flowid 1:2 to
17
filter parent 1: protocol ip pref 100 route fh 0xffff0012 flowid 1:3 to
18
filter parent 1: protocol ip pref 100 route fh 0xffff0013 flowid 1:4 to
19


[root@phoenix kb1]# /sbin/ip route list
207.6.137.209 via 209.5.131.25 dev ipsec0
209.5.131.46 dev eth1  scope link
192.168.49.253 via 192.168.49.251 dev eth0 realm 17
192.168.51.253 via 192.168.51.251 dev eth0 realm 19
192.168.48.251 dev eth0  scope link
192.168.48.253 via 192.168.48.251 dev eth0 realm 16
192.168.50.253 via 192.168.50.251 dev eth0 realm 18
209.5.131.0/24 dev eth1  proto kernel  scope link  src 209.5.131.46
209.5.131.0/24 dev ipsec0  proto kernel  scope link  src 209.5.131.46
192.168.50.0/24 dev eth0  proto kernel  scope link  src 192.168.50.251
192.168.51.0/24 dev eth0  proto kernel  scope link  src 192.168.51.251
192.168.48.0/24 dev eth0  proto kernel  scope link  src 192.168.48.251
192.168.49.0/24 dev eth0  proto kernel  scope link  src 192.168.49.251
192.168.64.0/20 via 209.5.131.25 dev ipsec0
127.0.0.0/8 dev lo  scope link
default via 209.5.131.25 dev eth1


[root@phoenix kb1]# netstat -na

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address
State
tcp        0      0 192.168.48.251:2249     192.168.48.254:139
TIME_WAIT
tcp        0    138 209.5.131.46:23         209.5.131.51:62403
ESTABLISHED
tcp        0      0 192.168.48.251:2244     192.168.48.250:23
ESTABLISHED
tcp        0      0 209.5.131.46:23         209.5.131.51:62402
ESTABLISHED
tcp        0      0 0.0.0.0:1723            0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:139             0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:3128            0.0.0.0:*
LISTEN
tcp        0      0 127.0.0.1:1034          127.0.0.1:1035
ESTABLISHED
tcp        0      0 127.0.0.1:1035          127.0.0.1:1034
ESTABLISHED
tcp        0      0 127.0.0.1:1032          127.0.0.1:1033
ESTABLISHED
tcp        0      0 127.0.0.1:1033          127.0.0.1:1032
ESTABLISHED
tcp        0      0 127.0.0.1:1030          127.0.0.1:1031
ESTABLISHED
tcp        0      0 127.0.0.1:1031          127.0.0.1:1030
ESTABLISHED
tcp        0      0 127.0.0.1:1028          127.0.0.1:1029
ESTABLISHED
tcp        0      0 127.0.0.1:1029          127.0.0.1:1028
ESTABLISHED
tcp        0      0 127.0.0.1:1026          127.0.0.1:1027
ESTABLISHED
tcp        0      0 127.0.0.1:1027          127.0.0.1:1026
ESTABLISHED
tcp        0      0 0.0.0.0:80              0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:25              0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:515             0.0.0.0:*
LISTEN
tcp        0      0 192.168.48.251:53       0.0.0.0:*
LISTEN
tcp        0      0 127.0.0.1:53            0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:873             0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:98              0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:113             0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:79              0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:143             0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:110             0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:513             0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:514             0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:23              0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:21              0.0.0.0:*
LISTEN
tcp        0      0 0.0.0.0:111             0.0.0.0:*
LISTEN
udp        0      0 192.168.48.251:138      0.0.0.0:*
udp        0      0 192.168.48.251:137      0.0.0.0:*
udp        0      0 0.0.0.0:138             0.0.0.0:*
udp        0      0 0.0.0.0:137             0.0.0.0:*
udp        0      0 0.0.0.0:3401            0.0.0.0:*
udp        0      0 0.0.0.0:3130            0.0.0.0:*
udp     1552      0 0.0.0.0:3069            0.0.0.0:*
udp        0      0 0.0.0.0:53              0.0.0.0:*
udp        0      0 192.168.48.251:53       0.0.0.0:*
udp        0      0 127.0.0.1:53            0.0.0.0:*
udp        0      0 0.0.0.0:161             0.0.0.0:*
udp        0      0 0.0.0.0:518             0.0.0.0:*

udp        0      0 0.0.0.0:517             0.0.0.0:*
udp        0      0 209.5.131.46:500        0.0.0.0:*
udp        0      0 0.0.0.0:111             0.0.0.0:*
raw        0      0 0.0.0.0:1               0.0.0.0:*               7
raw        0      0 0.0.0.0:6               0.0.0.0:*               7
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  0      [ ACC ]     STREAM     LISTENING     874    /dev/printer
unix  9      [ ]         DGRAM                    514    /dev/log
unix  0      [ ACC ]     STREAM     LISTENING     612
/var/run/pluto.ctl
unix  0      [ ACC ]     STREAM     LISTENING     841    /var/run/ndc
unix  0      [ ]         STREAM     CONNECTED     220    @00000054
unix  0      [ ]         DGRAM                    13820
unix  0      [ ]         DGRAM                    13790
unix  0      [ ]         DGRAM                    1297
unix  0      [ ]         DGRAM                    1197
unix  0      [ ]         DGRAM                    1160
unix  0      [ ]         DGRAM                    928
unix  0      [ ]         DGRAM                    839
unix  0      [ ]         DGRAM                    611
unix  0      [ ]         DGRAM                    530


[root@phoenix kb1]# gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/egcs-2.91.66/specs
gcc version egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)


[root@phoenix kb1]# uname -a
Linux phoenix.secure.basis.org 2.2.12-20-ipsec-mschap-QoS #17 Tue May 2
23:12:19
 PDT 2000 i586 unknown


 All that said, does anyone have any pointers? Am I experiencing some
form of the egcs bug mentioned on linux-net back in July 1999? (Do a
power search for 'Re: Bug in CBQ'  in fa.linux.net on www.deja.com)

 Any hints or discussion would be appreciated...

kris.