Bandwidth issues in the Sprint network

[Brian Raaen]

I am currently having problems get upload bandwidth on a Sprint circuit. I am
using a full OC3 circuit. I am doing fine on downloading data, but uploading
data I can only get about 5Mbps with ftp or a speedtest. I have tested
against multiple networks and this has stayed the same. Monitoring Cacti
graphs and the router I do get about 30Mbps total traffic outbound, but
individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The assistance I
have gotten from Sprint up to this point is that they find no problems. Due
to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know
if I was overlooking something else.

[V***@vt.edu]

Post by Brian Raaen
have gotten from Sprint up to this point is that they find no problems. Due
to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know
if I was overlooking something else.

TCP window size tuning? I'd look there first...

[Scott Weeks]

--- ***@zcorum.com wrote:

I am currently having problems get upload bandwidth on a Sprint circuit. I am
using a full OC3 circuit. I am doing fine on downloading data, but uploading
data I can only get about 5Mbps with ftp or a speedtest. I have tested
against multiple networks and this has stayed the same. Monitoring Cacti
graphs and the router I do get about 30Mbps total traffic outbound, but
individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The assistance I
have gotten from Sprint up to this point is that they find no problems. Due
to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know
if I was overlooking something else.
------------------------------------


I would not use one FTP session to test bandwidth. The rate limiting may be in the FTP software or other area of the computer. Likewise, Speedtest servers (in my opinion) are more marketing tools than testing tools. Try several similarly configured (but separate boxes) FTP servers simultaneously. If you see it go up by a factor of three you've found the issue.

I have had to push my four OC-12s to Sprint to the max at times and get full BW. That's in Hawaii, but I imagine it's the same as other areas.

scott

[Robert D. Scott]

See if you can find a nother connector that can help with using iperf. Also,
make sure any system testing systems have tuned IP stacks. That info is also
linked from the iperf web page.

http://dast.nlanr.net/Projects/Iperf/

http://www.psc.edu/networking/projects/tcptune/

Robert D. Scott ***@ufl.edu
Senior Network Engineer 352-273-0113 Phone
CNS - Network Services 352-392-2061 CNS Receptionist
University of Florida 352-392-9440 FAX
Florida Lambda Rail 352-294-3571 FLR NOC
Gainesville, FL 32611

-----Original Message-----
From: owner-***@merit.edu [mailto:owner-***@merit.edu] On Behalf Of
Scott Weeks
Sent: Monday, April 07, 2008 5:24 PM
To: ***@merit.edu
Subject: Re: Bandwidth issues in the Sprint network




--- ***@zcorum.com wrote:

I am currently having problems get upload bandwidth on a Sprint circuit. I
am using a full OC3 circuit. I am doing fine on downloading data, but
uploading data I can only get about 5Mbps with ftp or a speedtest. I have
tested against multiple networks and this has stayed the same. Monitoring
Cacti graphs and the router I do get about 30Mbps total traffic outbound,
but individual (flows/ip?) test always seem limited. I would like to know
if anyone else sees anything similar, or where I can get help. The
assistance I have gotten from Sprint up to this point is that they find no
problems. Due to the consistency of 5Mbps I am suspecting rate limiting,
but wanted to know if I was overlooking something else.
------------------------------------


I would not use one FTP session to test bandwidth. The rate limiting may be
in the FTP software or other area of the computer. Likewise, Speedtest
servers (in my opinion) are more marketing tools than testing tools. Try
several similarly configured (but separate boxes) FTP servers
simultaneously. If you see it go up by a factor of three you've found the
issue.

I have had to push my four OC-12s to Sprint to the max at times and get full
BW. That's in Hawaii, but I imagine it's the same as other areas.

scott

[Scott Weeks]

------- ***@umn.edu wrote: -----------
I would like to second the recommendation and go one further. Internet2
has released a performance toolkit that is run from CD. I would like to
------------------------------------------

Post by Robert D. Scott
See if you can find a nother connector that can help with using iperf. Also,

-------------------------------------------


The thing to note about most tools like these is you need a box on both sides of the circuit using the same software. One could be 'out there' on the internet, but the further 'out there' your other box is, the less valid your test is.

scott

[Martin Hannigan]

Has this circuit ever run clean(normal)?


-M<

Post by Brian Raaen
I am currently having problems get upload bandwidth on a Sprint circuit. I am
using a full OC3 circuit. I am doing fine on downloading data, but uploading
data I can only get about 5Mbps with ftp or a speedtest. I have tested
against multiple networks and this has stayed the same. Monitoring Cacti
graphs and the router I do get about 30Mbps total traffic outbound, but
individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The assistance I
have gotten from Sprint up to this point is that they find no problems. Due
to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know
if I was overlooking something else.
--
Brian Raaen
Network Engineer

[Brian Raaen]

Currently there is not a proxy server in the network, although when using some
of the test on dslreports.com there is a message about compression being used
for the upload and to remove proxy settings. I have also been testing using
FTP on a *nix server as well. Both the server and PC are connect to a Cisco
2960 switch in the headend that is connected to the 7200 router. I can
transfer ftp at about 80Mbps between the PC and the server, so they are not
IO bound. The Site I am testing with is a ftp server located in a colo
facility that we use and has sufficient bandwidth. This circuit is clean in
the sense of not having CRC, framing or other errors but this is a new
circuit and we have never gotten more than 5Mbps out of a single session
(flow/ip) across the wan. I would have to double check the mtu, but it is
currently the default.
--
Brian Raaen
Network Engineer

Post by Brian Raaen
I am currently having problems get upload bandwidth on a Sprint circuit. I am
using a full OC3 circuit. I am doing fine on downloading data, but uploading
data I can only get about 5Mbps with ftp or a speedtest. I have tested
against multiple networks and this has stayed the same. Monitoring Cacti
graphs and the router I do get about 30Mbps total traffic outbound, but
individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The assistance I
have gotten from Sprint up to this point is that they find no problems. Due
to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know
if I was overlooking something else.
--
Brian Raaen
Network Engineer

[Sam Stickland]

Could be your TCP window size? A 17520 byte TCP window (Windows 2000)
will cause a single flow to top out at 5Mbps at about 50ms. What is the
latency on the link?

Try some figures here and see what limit you might be hitting:

http://www.wand.net.nz/~perry/max_download.php?bits_per_second=155000000&ack_size=40&no_delayed_acks=2&mss=1460&rtt=35&wsize=17520&ploss=0

Sam

Post by Brian Raaen
I am currently having problems get upload bandwidth on a Sprint circuit. I am
using a full OC3 circuit. I am doing fine on downloading data, but uploading
data I can only get about 5Mbps with ftp or a speedtest. I have tested
against multiple networks and this has stayed the same. Monitoring Cacti
graphs and the router I do get about 30Mbps total traffic outbound, but
individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The assistance I
have gotten from Sprint up to this point is that they find no problems. Due
to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know
if I was overlooking something else.

[Brian Raaen]

I have been using the Java based versions of the speed test.  At this point I
have had some Sprint people get in contact with me so I will see what they
find.  Thank you for all your help to everyone.
--
Brian Raaen
Network Engineer

Post by Brian Raaen
I am currently having problems get upload bandwidth on a Sprint circuit. I

am using a full OC3 circuit.  I am doing fine on downloading data, but
uploading data I can only get about 5Mbps with ftp or a speedtest.  I have
tested against multiple networks and this has stayed the same.  Monitoring
Cacti graphs and the router I do get about 30Mbps total traffic outbound, but
individual (flows/ip?) test always seem limited.  I would like to know if
anyone else sees anything similar, or where I can get help.  The assistance I
have gotten from Sprint up to this point is that they find no problems.  Due
to the consistency of 5Mbps I am suspecting rate limiting, but wanted to

Post by Brian Raaen
know if I was overlooking something else.
--
Brian Raaen
Network Engineer

[Mike Gonnason]

I have been using the Java based versions of the speed test. At this point I
have had some Sprint people get in contact with me so I will see what they
find. Thank you for all your help to everyone.
--
Brian Raaen
Network Engineer

Post by Brian Raaen
I am currently having problems get upload bandwidth on a Sprint circuit. I

am using a full OC3 circuit. I am doing fine on downloading data, but
uploading data I can only get about 5Mbps with ftp or a speedtest. I have
tested against multiple networks and this has stayed the same. Monitoring
Cacti graphs and the router I do get about 30Mbps total traffic outbound, but
individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The assistance I
have gotten from Sprint up to this point is that they find no problems. Due
to the consistency of 5Mbps I am suspecting rate limiting, but wanted to

Post by Brian Raaen
know if I was overlooking something else.
--
Brian Raaen
Network Engineer

Most of the speed test sites on the Internet basically issue a HTTP
GET request to a server and time the download. For upload they utilize
a HTTP POST via a CGI script and time that. The main issue I have with
these speed tests is that they only use a single TCP session for data
transfer, which is fine if you have a large or self adjusting TCP
window size and a relatively low latency link.

However for high capacity links, it is unlikely (but possible) that
you are planning to use a single TCP session and consume all the
available capacity. Realistically you will have a few dozen
server/applications/users and produce hundreds/thousands of TCP
sessions which will fully utilize the link.

For our PtP customers that have concerns regarding capacity, I
generally they suggest setup iperf at both ends and run a few tests
with multiple TCP sessions so they can independently verify. Hopefully
Sprint will take your concerns to heart and assist you with testing.

-Mike Gonnason

[Frank Bulk]

Does anyone know of bootable Linux CD with iperf on it?

Frank

-----Original Message-----
From: owner-***@merit.edu [mailto:owner-***@merit.edu] On Behalf Of Mike
Gonnason
Sent: Wednesday, April 09, 2008 9:05 AM
To: ***@merit.edu
Subject: Re: Bandwidth issues in the Sprint network

I have been using the Java based versions of the speed test. At this point I
have had some Sprint people get in contact with me so I will see what they
find. Thank you for all your help to everyone.
--
Brian Raaen
Network Engineer

Post by Brian Raaen
I am currently having problems get upload bandwidth on a Sprint circuit. I

am using a full OC3 circuit. I am doing fine on downloading data, but
uploading data I can only get about 5Mbps with ftp or a speedtest. I have
tested against multiple networks and this has stayed the same.

Monitoring

Cacti graphs and the router I do get about 30Mbps total traffic outbound, but
individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The assistance I
have gotten from Sprint up to this point is that they find no problems.

Due

to the consistency of 5Mbps I am suspecting rate limiting, but wanted to

Post by Brian Raaen
know if I was overlooking something else.
--
Brian Raaen
Network Engineer

Most of the speed test sites on the Internet basically issue a HTTP
GET request to a server and time the download. For upload they utilize
a HTTP POST via a CGI script and time that. The main issue I have with
these speed tests is that they only use a single TCP session for data
transfer, which is fine if you have a large or self adjusting TCP
window size and a relatively low latency link.

However for high capacity links, it is unlikely (but possible) that
you are planning to use a single TCP session and consume all the
available capacity. Realistically you will have a few dozen
server/applications/users and produce hundreds/thousands of TCP
sessions which will fully utilize the link.

For our PtP customers that have concerns regarding capacity, I
generally they suggest setup iperf at both ends and run a few tests
with multiple TCP sessions so they can independently verify. Hopefully
Sprint will take your concerns to heart and assist you with testing.

-Mike Gonnason

[Mike Gonnason]

A quick search comes up with Scientific Linux, but I cannot provide
any claims to suitability. I have never even heard of it before, but
it is provided as a LiveCD.

http://linux.web.psi.ch/livecd/software.html

-Mike Gonnason

Post by Frank Bulk
Does anyone know of bootable Linux CD with iperf on it?
Frank
-----Original Message-----
Gonnason
Sent: Wednesday, April 09, 2008 9:05 AM
Subject: Re: Bandwidth issues in the Sprint network

I have been using the Java based versions of the speed test. At this

point I

have had some Sprint people get in contact with me so I will see what

they

find. Thank you for all your help to everyone.
--
Brian Raaen
Network Engineer

Post by Brian Raaen
I am currently having problems get upload bandwidth on a Sprint

circuit. I

am using a full OC3 circuit. I am doing fine on downloading data, but
uploading data I can only get about 5Mbps with ftp or a speedtest. I

have

tested against multiple networks and this has stayed the same.

Monitoring

Cacti graphs and the router I do get about 30Mbps total traffic outbound,

but

individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The

assistance I

have gotten from Sprint up to this point is that they find no problems.

Due

to the consistency of 5Mbps I am suspecting rate limiting, but wanted to

Post by Brian Raaen
know if I was overlooking something else.
--
Brian Raaen
Network Engineer

Most of the speed test sites on the Internet basically issue a HTTP
GET request to a server and time the download. For upload they utilize
a HTTP POST via a CGI script and time that. The main issue I have with
these speed tests is that they only use a single TCP session for data
transfer, which is fine if you have a large or self adjusting TCP
window size and a relatively low latency link.
However for high capacity links, it is unlikely (but possible) that
you are planning to use a single TCP session and consume all the
available capacity. Realistically you will have a few dozen
server/applications/users and produce hundreds/thousands of TCP
sessions which will fully utilize the link.
For our PtP customers that have concerns regarding capacity, I
generally they suggest setup iperf at both ends and run a few tests
with multiple TCP sessions so they can independently verify. Hopefully
Sprint will take your concerns to heart and assist you with testing.
-Mike Gonnason

[Michael Holstein]

Post by Frank Bulk
Does anyone know of bootable Linux CD with iperf on it?

Knoppix STD (security tools distro)

http://www.knoppix-std.org/tools.html

Cheers,

Michael Holstein
Cleveland State University

[Murphy, William]

Not 100% sure about iperf but I2 has a nice Network Performance Toolkit
that runs on top of Knoppix and they have a downloadable ISO image...

Get the ISO here...
http://e2epi.internet2.edu/network-performance-toolkit.html

Interesting doc on configuring toolkit from SLAC...
http://confluence.slac.stanford.edu/display/IEPM/Network+Performance+Too
lkit


Bill Murphy
Senior Network Analyst
University of Texas Health Science Center - Houston

-----Original Message-----
From: owner-***@merit.edu [mailto:owner-***@merit.edu] On Behalf Of
Michael Holstein
Sent: Wednesday, April 09, 2008 10:00 AM
To: ***@iname.com
Cc: ***@merit.edu
Subject: Re: Bandwidth issues in the Sprint network

Post by Frank Bulk
Does anyone know of bootable Linux CD with iperf on it?

Knoppix STD (security tools distro)

http://www.knoppix-std.org/tools.html

Cheers,

Michael Holstein
Cleveland State University

[Frank Bulk]

I tried this on three laptops (two different models), and none of them would
fully boot. They would lock up at different points.

Unless someone has some workarounds, I think I'll be trying another ISO
package.

Regards,

Frank

-----Original Message-----
From: Tim Peiffer [mailto:***@umn.edu]
Sent: Wednesday, April 09, 2008 9:50 AM
To: ***@iname.com
Cc: ***@merit.edu
Subject: Re: Bandwidth issues in the Sprint network

http://e2epi.internet2.edu/network-performance-toolkit/network-performance-t
oolkit.iso

Post by Frank Bulk
Does anyone know of bootable Linux CD with iperf on it?
Frank
-----Original Message-----
Gonnason
Sent: Wednesday, April 09, 2008 9:05 AM
Subject: Re: Bandwidth issues in the Sprint network

I have been using the Java based versions of the speed test. At this

point I

have had some Sprint people get in contact with me so I will see what

they

find. Thank you for all your help to everyone.
--
Brian Raaen
Network Engineer

Post by Brian Raaen
I am currently having problems get upload bandwidth on a Sprint

circuit. I

am using a full OC3 circuit. I am doing fine on downloading data, but
uploading data I can only get about 5Mbps with ftp or a speedtest. I

have

tested against multiple networks and this has stayed the same.

Monitoring

Cacti graphs and the router I do get about 30Mbps total traffic

outbound,

Post by Frank Bulk
but

individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The

assistance I

have gotten from Sprint up to this point is that they find no problems.

Due

to the consistency of 5Mbps I am suspecting rate limiting, but wanted to

Post by Brian Raaen
know if I was overlooking something else.
--
Brian Raaen
Network Engineer

Most of the speed test sites on the Internet basically issue a HTTP
GET request to a server and time the download. For upload they utilize
a HTTP POST via a CGI script and time that. The main issue I have with
these speed tests is that they only use a single TCP session for data
transfer, which is fine if you have a large or self adjusting TCP
window size and a relatively low latency link.
However for high capacity links, it is unlikely (but possible) that
you are planning to use a single TCP session and consume all the
available capacity. Realistically you will have a few dozen
server/applications/users and produce hundreds/thousands of TCP
sessions which will fully utilize the link.
For our PtP customers that have concerns regarding capacity, I
generally they suggest setup iperf at both ends and run a few tests
with multiple TCP sessions so they can independently verify. Hopefully
Sprint will take your concerns to heart and assist you with testing.
-Mike Gonnason

[r***@rochester.rr.com]

You can also use ttcp from the command line, useful if its cisco on both ends of the circuit.

sr01#ttcp
transmit or receive [receive]: transmit
Target IP address: 1.1.1.1
calculate checksum during buffer write [y]:
perform tcp half close [n]:
send buflen [32768]:
send nbuf [2048]:
bufalign [16384]:
bufoffset [0]:
port [5001]:
sinkmode [y]:
buffering on writes [y]:
show tcp information at end [n]:

ttcp-t: buflen=32768, nbuf=2048, align=16384/0, port=5001 tcp -> 1.1.1.1

sr02#ttcp
transmit or receive [receive]:
receive packets asynchronously [n]:
perform tcp half close [n]:
receive buflen [32768]:
bufalign [16384]:
bufoffset [0]:
port [5001]:
sinkmode [y]:
rcvwndsize [32768]:
ack frequency [0]:
delayed ACK [y]:
show tcp information at end [n]:

ttcp-r: buflen=32768, align=16384/0, port=5001
rcvwndsize=32768, delayedack=yes tcp

Post by Michael Holstein

Post by Frank Bulk
Does anyone know of bootable Linux CD with iperf on it?

Knoppix STD (security tools distro)
http://www.knoppix-std.org/tools.html
Cheers,
Michael Holstein
Cleveland State University

[Frank Bulk]

Good idea, but the other side doesn't have a Cisco box.

Frank

-----Original Message-----
From: ***@rochester.rr.com [mailto:***@rochester.rr.com]
Sent: Wednesday, April 09, 2008 11:02 AM
To: Michael Holstein
Cc: ***@iname.com; ***@merit.edu
Subject: Re: Bandwidth issues in the Sprint network

You can also use ttcp from the command line, useful if its cisco on both
ends of the circuit.

sr01#ttcp
transmit or receive [receive]: transmit
Target IP address: 1.1.1.1
calculate checksum during buffer write [y]:
perform tcp half close [n]:
send buflen [32768]:
send nbuf [2048]:
bufalign [16384]:
bufoffset [0]:
port [5001]:
sinkmode [y]:
buffering on writes [y]:
show tcp information at end [n]:

ttcp-t: buflen=32768, nbuf=2048, align=16384/0, port=5001 tcp -> 1.1.1.1

sr02#ttcp
transmit or receive [receive]:
receive packets asynchronously [n]:
perform tcp half close [n]:
receive buflen [32768]:
bufalign [16384]:
bufoffset [0]:
port [5001]:
sinkmode [y]:
rcvwndsize [32768]:
ack frequency [0]:
delayed ACK [y]:
show tcp information at end [n]:

ttcp-r: buflen=32768, align=16384/0, port=5001
rcvwndsize=32768, delayedack=yes tcp

Post by Michael Holstein

Post by Frank Bulk
Does anyone know of bootable Linux CD with iperf on it?

Knoppix STD (security tools distro)
http://www.knoppix-std.org/tools.html
Cheers,
Michael Holstein
Cleveland State University

[Brian Raaen]

Some people wanted to know what I found the problem to be. I have discovered.
the problem for a fact is the TCP window size on uploads. I have a Linux box
that I changed the Window sizes to match and I still get 32k on a upload
window and 64k on a download window. With a ping time of 50ms I have a max
theoretical throughput of 5.2Mbps Which is about what I was getting. The
formula to calculate this is the following.

(((Ts/Tw)*Rtd)/1000)+((Ts*8)/(Lr*1000)))

Where the following are

Ts = Transfer size in Bytes
Tw = Tcp Window size in Bytes
Rtd = Round trip Delay in milliseconds
Lr = Line rate in bps

At this point I am still trying to locate the offending device that is
changing the window size. After I determine for sure whether the problem is
with my router, the sprint network, or another upstream system I will let
everybody know what I find.
--
Brian Raaen
Network Engineer

Post by Brian Raaen
I am currently having problems get upload bandwidth on a Sprint circuit. I am
using a full OC3 circuit. I am doing fine on downloading data, but uploading
data I can only get about 5Mbps with ftp or a speedtest. I have tested
against multiple networks and this has stayed the same. Monitoring Cacti
graphs and the router I do get about 30Mbps total traffic outbound, but
individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The assistance I
have gotten from Sprint up to this point is that they find no problems. Due
to the consistency of 5Mbps I am suspecting rate limiting, but wanted to know
if I was overlooking something else.
--
Brian Raaen
Network Engineer

[Lincoln Dale]

even with tuned TCP window sizes, make sure you don't have TCP syncookies
enabled on either endpoint.

many syncookie implementations have implications on supporting RFC1323 options.


cheers,

lincoln.

Post by Robert D. Scott
-----Original Message-----
Raaen
Sent: Friday, 18 April 2008 7:00 AM
Subject: Re: Bandwidth issues in the Sprint network
Some people wanted to know what I found the problem to be. I have discovered.
the problem for a fact is the TCP window size on uploads. I have a Linux box
that I changed the Window sizes to match and I still get 32k on a upload
window and 64k on a download window. With a ping time of 50ms I have a max
theoretical throughput of 5.2Mbps Which is about what I was getting. The
formula to calculate this is the following.
(((Ts/Tw)*Rtd)/1000)+((Ts*8)/(Lr*1000)))
Where the following are
Ts = Transfer size in Bytes
Tw = Tcp Window size in Bytes
Rtd = Round trip Delay in milliseconds
Lr = Line rate in bps
At this point I am still trying to locate the offending device that is
changing the window size. After I determine for sure whether the problem is
with my router, the sprint network, or another upstream system I will let
everybody know what I find.
--
Brian Raaen
Network Engineer

Post by Brian Raaen
I am currently having problems get upload bandwidth on a Sprint circuit. I

am

Post by Brian Raaen
using a full OC3 circuit. I am doing fine on downloading data, but

uploading

Post by Brian Raaen
data I can only get about 5Mbps with ftp or a speedtest. I have tested
against multiple networks and this has stayed the same. Monitoring Cacti
graphs and the router I do get about 30Mbps total traffic outbound, but
individual (flows/ip?) test always seem limited. I would like to know if
anyone else sees anything similar, or where I can get help. The assistance

I

Post by Brian Raaen
have gotten from Sprint up to this point is that they find no problems.

Due

Post by Brian Raaen
to the consistency of 5Mbps I am suspecting rate limiting, but wanted to

know

Post by Brian Raaen
if I was overlooking something else.

[Chris Adams]

Post by Lincoln Dale
even with tuned TCP window sizes, make sure you don't have TCP syncookies
enabled on either endpoint.

IIRC Linux (at least) syncookies only come into play when you are being
syn-flooded (i.e. when the kernel has to start dropping syns). Having
them enabled at other times has no impact, so there's rarely (if ever) a
reason to disable them.