10GE router resource

[user user]

Hi everybody!

I find myself in the market for some 10GE routers. As
I don't buy these everyday, I was wondering if any of
you guys had any good resources for evaluating
different vendors and models. I'm mainly thinking
about non-vendor resources as the vendorspeak sites
are not that hard to find.

Also I'd love to hear recommendatios for "budget" 10GE
routers. The "budget" router would be used to hook up
client networks through one 10GE interface and connect
to different transit providers through two 10GE
interfaces.

- Zed





____________________________________________________________________________________
Never miss a thing. Make Yahoo your home page.
http://www.yahoo.com/r/hs

[Mark Tinka]

Post by user user
Hi everybody!

Hello.

Post by user user
Also I'd love to hear recommendatios for "budget" 10GE
routers. The "budget" router would be used to hook up
client networks through one 10GE interface and connect
to different transit providers through two 10GE
interfaces.

Today, from Cisco, the smallest router you'll get a 10Gbps
Ethernet port on is the Cisco ASR1000 series. Mind you,
though, FCS for this box isn't until about May. Also, this
box is oversubscribed as the current switch fabric is
10Gbps.

From Juniper, the smallest M-series box you'll get the same
port on is the M120 platform.

You could also look at smaller switches from both vendors,
but if you plan on taking full BGP feeds from your upstream
providers, this might be an issue.

Cheers,

Mark.

[Kevin Oberman]

Date: Mon, 24 Mar 2008 19:12:57 +0800

Post by user user
Hi everybody!

Hello.

Post by user user
Also I'd love to hear recommendatios for "budget" 10GE
routers. The "budget" router would be used to hook up
client networks through one 10GE interface and connect
to different transit providers through two 10GE
interfaces.

Today, from Cisco, the smallest router you'll get a 10Gbps
Ethernet port on is the Cisco ASR1000 series. Mind you,
though, FCS for this box isn't until about May. Also, this
box is oversubscribed as the current switch fabric is
10Gbps.
From Juniper, the smallest M-series box you'll get the same
port on is the M120 platform.
You could also look at smaller switches from both vendors,
but if you plan on taking full BGP feeds from your upstream
providers, this might be an issue.

Depending on how the box will be used, Foundry is probably the cheapest,
followed by Force10. Since yo will be connecting to two transit
providers, you probably need the full routing table, but if you don't
need full routes, the new Juniper EX8200 looks like an option. It is
limited to about 12K routes in the FIB. It's not shipping at this time
and I don't know when FSR is scheduled.

Note that F10 does not do MPLS and neither F10 or Foundry has the
software stability of either C or J, so you will need to look closely at
exactly the features needed.

[Joel Snyder]

Post by user user
Also I'd love to hear recommendatios for "budget" 10GE
routers. The "budget" router would be used to hook up
client networks through one 10GE interface and connect
to different transit providers through two 10GE
interfaces.

If you don't need BGP-ish power, David Newman just published his test of
10GigE switches today in Network World. He was focusing mostly on
switching in the enterprise, but he has a variety of other performance
metrics and results which may be helpful:

http://www.networkworld.com/reviews/2008/032408-switch-test.html?t51hb

jms

[Justin Shore]

Post by Joel Snyder

Post by user user
Also I'd love to hear recommendatios for "budget" 10GE
routers. The "budget" router would be used to hook up
client networks through one 10GE interface and connect
to different transit providers through two 10GE
interfaces.

If you don't need BGP-ish power, David Newman just published his test of
10GigE switches today in Network World. He was focusing mostly on
switching in the enterprise, but he has a variety of other performance
http://www.networkworld.com/reviews/2008/032408-switch-test.html?t51hb

The author's specifications eliminated Cisco's 4900M from the
competition. That not unexpected though since it was a evaluation of
access switches w/ 10G uplinks. The 4900M has 8 on-board 10G interfaces
and expansion modules that can carry 8 more (not oversubscribed) or 16
(oversubscribed). It has has GigE support via TwinGig modules in the
expansion module bays. It also has a 320Gbps backplane and can handle
up to 200k v4 routes. It's an impressive little switch if you need 10G
aggregation. It can't handle a full table of course but it still has a
lot of use. No MPLS options. It's based on the 4500's Sup 6-E.

http://www.cisco.com/en/US/products/ps9310/index.html

The base unit starts at $16k.

Justin

[Greg VILLAIN]

Post by user user
Hi everybody!
I find myself in the market for some 10GE routers. As
I don't buy these everyday, I was wondering if any of
you guys had any good resources for evaluating
different vendors and models. I'm mainly thinking
about non-vendor resources as the vendorspeak sites
are not that hard to find.
Also I'd love to hear recommendatios for "budget" 10GE
routers. The "budget" router would be used to hook up
client networks through one 10GE interface and connect
to different transit providers through two 10GE
interfaces.
- Zed

Hiya,

When it comes to budget, force10 are good. I wouldn't be able to
confirm if they're worth performance-wise.
I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise
and performance-wise, provided you do not need rocket-science features.
MLX/XMR models will surely do the trick perfectly.

When it comes to router purchasing habits, we all tend to get
religious...
Bottom line is that most of the 'regular' vendors (namely Cisco,
Juniper, Foundry, Force10, Extreme, Riverstone) implement pretty much
the same set of features, which are all IETF/IEEE normalized, meaning
if you don't need proprietary features (and you'll wish you don't),
any router will be fine, the only difference will come from:
- the chassis being non-blocking or not (i.e. backplane design)
- the price per port
- the operating OS
- the feeling you'll get with the salesperson, and the reputation of
their Support Teams.
- vendor specific features such as Flow Sampling
To make it simple, most vendors have an IOS like OS, except Juniper
which has a really clever and elegant OS, but are very pricey.
Foundry and Force10 have the cheapest price per port
Cisco does only Netflow, Foundry & Force10 only SFlow (which is a true
standard) and I think Juniper does JFlow
Cisco's kits are packed with proprietary protocols (HSRP and GLBP
instead of VRRP, their own ethernet trunking, EIGRP as their own and
yet extremely efficient IGP, TCL scriptable CLI...) , some of them are
really good, some are crappy, but I suggest you'd stick with IEEE/IETF
protocol to avoid future trouble.

One thing: RSTP/802-1w is very (very, very, very) not often
interoperable between vendors who all have their own interpretation of
the norm and can quickly turn into a nightmare.
I'd strongly suggest try&buys if (R)STP interoperability is required,
but I'm a little paranoid :)

Greg VILLAIN
Independant Network & Telco Architecture Consultant

[Chris Grundemann]

Greg has laid out a great bit of information and I would like to add just
one possibility to the list of budget 10GE routers: Vyatta. According to a
recent press release from that company (
http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product
that is "2 to 3X higher performance at a cost savings of more than 75
percent" when compared to Cisco's 7200. Unfortunately I have not had the
opportunity to test or use the Vyatta routers yet; I have however
successfully used other open-source Linux based routers in the past with
great success. If you are looking for a truly budget 10GE router, they may
be worth adding to the list and looking into.

Post by Greg VILLAIN

Post by user user
Hi everybody!
I find myself in the market for some 10GE routers. As
I don't buy these everyday, I was wondering if any of
you guys had any good resources for evaluating
different vendors and models. I'm mainly thinking
about non-vendor resources as the vendorspeak sites
are not that hard to find.
Also I'd love to hear recommendatios for "budget" 10GE
routers. The "budget" router would be used to hook up
client networks through one 10GE interface and connect
to different transit providers through two 10GE
interfaces.
- Zed

Hiya,
When it comes to budget, force10 are good. I wouldn't be able to
confirm if they're worth performance-wise.
I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise
and performance-wise, provided you do not need rocket-science features.
MLX/XMR models will surely do the trick perfectly.
When it comes to router purchasing habits, we all tend to get
religious...
Bottom line is that most of the 'regular' vendors (namely Cisco,
Juniper, Foundry, Force10, Extreme, Riverstone) implement pretty much
the same set of features, which are all IETF/IEEE normalized, meaning
if you don't need proprietary features (and you'll wish you don't),
- the chassis being non-blocking or not (i.e. backplane design)
- the price per port
- the operating OS
- the feeling you'll get with the salesperson, and the reputation of
their Support Teams.
- vendor specific features such as Flow Sampling
To make it simple, most vendors have an IOS like OS, except Juniper
which has a really clever and elegant OS, but are very pricey.
Foundry and Force10 have the cheapest price per port
Cisco does only Netflow, Foundry & Force10 only SFlow (which is a true
standard) and I think Juniper does JFlow
Cisco's kits are packed with proprietary protocols (HSRP and GLBP
instead of VRRP, their own ethernet trunking, EIGRP as their own and
yet extremely efficient IGP, TCL scriptable CLI...) , some of them are
really good, some are crappy, but I suggest you'd stick with IEEE/IETF
protocol to avoid future trouble.
One thing: RSTP/802-1w is very (very, very, very) not often
interoperable between vendors who all have their own interpretation of
the norm and can quickly turn into a nightmare.
I'd strongly suggest try&buys if (R)STP interoperability is required,
but I'm a little paranoid :)
Greg VILLAIN
Independant Network & Telco Architecture Consultant

[Joel Jaeggli]

Post by Chris Grundemann
Greg has laid out a great bit of information and I would like to add
just one possibility to the list of budget 10GE routers: Vyatta.
According to a recent press release from that company
(http://www.vyatta.com/about/pressreleases.php?id=51) they offer a
product that is "2 to 3X higher performance at a cost savings of more
than 75 percent" when compared to Cisco's 7200. Unfortunately I have
not had the opportunity to test or use the Vyatta routers yet; I have
however successfully used other open-source Linux based routers in the
past with great success. If you are looking for a truly budget 10GE
router, they may be worth adding to the list and looking into.

Whether you can actually do 10Gb/s reasonably on a linux or freebsd
soft-switched router platform is going to depend a lot on your actual
pps rate.

800K pps which is 10Gb/s / 1500 bytes is feasible, but 19M pps which is
10Gb/s / 64 bytes is not.

Susceptibility to dos traffic at relatively low bit, but high pps rates
is a general issue with soft-switched platforms. and needs to be
accounted for in model deployments.

Post by Chris Grundemann

Post by user user
Hi everybody!
I find myself in the market for some 10GE routers. As
I don't buy these everyday, I was wondering if any of
you guys had any good resources for evaluating
different vendors and models. I'm mainly thinking
about non-vendor resources as the vendorspeak sites
are not that hard to find.
Also I'd love to hear recommendatios for "budget" 10GE
routers. The "budget" router would be used to hook up
client networks through one 10GE interface and connect
to different transit providers through two 10GE
interfaces.
- Zed

Hiya,
When it comes to budget, force10 are good. I wouldn't be able to
confirm if they're worth performance-wise.
I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise
and performance-wise, provided you do not need rocket-science features.
MLX/XMR models will surely do the trick perfectly.
When it comes to router purchasing habits, we all tend to get
religious...
Bottom line is that most of the 'regular' vendors (namely Cisco,
Juniper, Foundry, Force10, Extreme, Riverstone) implement pretty much
the same set of features, which are all IETF/IEEE normalized, meaning
if you don't need proprietary features (and you'll wish you don't),
- the chassis being non-blocking or not (i.e. backplane design)
- the price per port
- the operating OS
- the feeling you'll get with the salesperson, and the reputation of
their Support Teams.
- vendor specific features such as Flow Sampling
To make it simple, most vendors have an IOS like OS, except Juniper
which has a really clever and elegant OS, but are very pricey.
Foundry and Force10 have the cheapest price per port
Cisco does only Netflow, Foundry & Force10 only SFlow (which is a true
standard) and I think Juniper does JFlow
Cisco's kits are packed with proprietary protocols (HSRP and GLBP
instead of VRRP, their own ethernet trunking, EIGRP as their own and
yet extremely efficient IGP, TCL scriptable CLI...) , some of them are
really good, some are crappy, but I suggest you'd stick with IEEE/IETF
protocol to avoid future trouble.
One thing: RSTP/802-1w is very (very, very, very) not often
interoperable between vendors who all have their own interpretation of
the norm and can quickly turn into a nightmare.
I'd strongly suggest try&buys if (R)STP interoperability is required,
but I'm a little paranoid :)
Greg VILLAIN
Independant Network & Telco Architecture Consultant
--
"Those who do not create the future they want must endure the future
they get."
~Draper L. Kaufman, Jr.
--

[William Herrin]

Post by Chris Grundemann
Greg has laid out a great bit of information and I would like to add just
one possibility to the list of budget 10GE routers: Vyatta. According to a
recent press release from that company
(http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product
that is "2 to 3X higher performance at a cost savings of more than 75
percent" when compared to Cisco's 7200.

"Vyatta operates at Layer 3 wire speed across three Gigabit Ethernet
ports in full mesh when forwarding 512-byte frames or higher."

3x1 GE << 1x10 GE

Regards,
Bill Herrin

[Chris Grundemann]

On Tue, Mar 25, 2008 at 1:56 PM, William Herrin

Post by William Herrin

Post by Chris Grundemann
Greg has laid out a great bit of information and I would like to add just
one possibility to the list of budget 10GE routers: Vyatta. According to a
recent press release from that company
(http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product
that is "2 to 3X higher performance at a cost savings of more than 75
percent" when compared to Cisco's 7200.

"Vyatta operates at Layer 3 wire speed across three Gigabit Ethernet
ports in full mesh when forwarding 512-byte frames or higher."
3x1 GE << 1x10 GE

It appears that I put my foot in my mouth. I have read several claims
that the Vyatta software is scalable to 10G, most notably here:
http://www.networkworld.com/news/2008/031708-vyatta-open-source-router.html.
Upon further investigation, I have been unable to substantiate that
claim.

My experience is similar to those who have posted here, pps is the
limiting factor - usually somewhere between 500-800K. Apparently I
was over eager to believe that more had been achieved.

To Ann's question on resources; I have only used Linux routers with 1G
ports but have surpassed 10G total throughput (up+ down) using various
dual proc set ups, most often Intel Xeon in Dell servers. A gentlemen
by the name of Martin Pels wrote a good paper on the subject early
last year that can be found here:
http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at
700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and
2GB of RAM in a Dell PowerEdge 1950.

~Chris

Post by William Herrin
Regards,
Bill Herrin
--
3005 Crane Dr. ...................... Web: <http://bill.herrin.us/>
Falls Church, VA 22042-3004

[Adrian Chadd]

Post by Chris Grundemann
To Ann's question on resources; I have only used Linux routers with 1G
ports but have surpassed 10G total throughput (up+ down) using various
dual proc set ups, most often Intel Xeon in Dell servers. A gentlemen
by the name of Martin Pels wrote a good paper on the subject early
http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at
700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and
2GB of RAM in a Dell PowerEdge 1950.

Mike Tancsa did some benchmarking in late 2006:

http://www.tancsa.com/blast.html

I think things are slightly faster now but not because of a massive
change in software architecture.




Adrian

[Lamar Owen]

Post by Aaron Glenn

Post by Patrick Clochesy
Very interesting study I had not seen, and a bummer. That really puts a
cramp in my advocation of our CARP+pf load balancers/firewalls/gateways.
Than again, what's a PIX box capable of?

I'd rather tweak a whitebox than pay through the nose for a PIX.

But aren't PIXen whiteboxes internally? I know the PIX-like LocalDirector
that was donated to us makes a very nice nBox deployment for us.

Lots of these sorts of boxes are internally whiteboxes (I'm using that term
loosely to mean an Intel-based box that could potentially run something like
a Linux or *BSD). The second-hand Content Engine 565 I got on eBay that had
a fried power supply was just a Cisco-labeled IBM eServer xSeries 305, and
was loaded with WindowsXP when I got it. It's running CentOS 5 now, with a
new IBM power supply in the box. The two earlier Content Engines and two
even earlier Cache Engines I got second-hand are likewise custom Intel
hardware; PIII 800's, to be precise. Now, they DO use ECC RAM, which most
whiteboxes won't have. But otherwise they are customized whiteboxes, and
you're paying for the software and support.

But cisco is not alone in this. Nomadix gateways, to use one example, are
built on custom embedded x86 systems.

What I'm waiting on is someone to take a system like a Xilinx ML410 dev board
and use the FPGA to do hardware-accelerated forwarding/filtering. See
http://www.lynuxworks.com/board-support/xilinx/ml410.php for info on the
board.

As to PIXen performance, see the charts in
http://en.wikipedia.org/wiki/Cisco_PIX

[Lincoln Dale]

That said, it
is notable that Cisco is now running their latest announced hardware,
primarily the Nexus 7000's and ASR's, run a Linux kernel and IOS on top of
that.

Moore's Law may have helped software packet forwarding rates but there's still
2 to 3 orders of magnitude performance difference between hardware & software.

just to be clear about a few things:

in the case of Nexus 7K the control-plane runs atop of Linux, data-plane runs
entirely in custom packet forwarding ASICs distributed on the I/O (linecard)
modules. N7K never drops to "software forwarding". the first forwarding
engine in N7K does 60M PPS with all features enabled. i.e. you could be
performing ACLs on port, VLAN & routed on both ingress & egress, doing netflow,
policing, QoS, whatever - its still 60M PPS.

you'll see that pps numbers scale upwards as the product progresses through its
roadmap.


Cisco doesn't make any secret of N7K running atop of Linux, the reality is that
it doesn't have to be Linux, it could be any SMP/multi-threaded capable
POSIX-compliant kernel, it just so happens that Linux makes sense for a variety
of reasons.

Also, perhaps pedantic but just to be absolutely clear: N7K doesn't run on IOS,
it runs on NX-OS.


ASR is slightly different, it can perform packet processing in software (IOSd)
however that is really only meant for things that don't make sense to implement
in what is now called the QuantumFlow programmable processor. e.g. if you
needed your AppleTalk or Vines running at millions of packets/second, then i'd
argue you have bigger problems. :)


cheers,

lincoln.

[m***@bt.com]

Post by Chris Grundemann
http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a
wall at 700K pps and was using two dual core Intel Xeon 64bit
2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950.

Unless I am misreading this, he did not hit a wall. What he did was
test a design that was scalable to multiple cores and show that the
two core version could not go beyond 700k pps. The next logical question
is how much more can you push with larger numbers of cores. The key
thing is to use a recent Linux kernel that can share interrupts among
multiple cores and to run it on a CPU using MSI interrupts. Since this
was written up in January of 2007,

There are people who use Linux for load balancing who also are working
on finding how well it can cope with 10G of traffic and they have some
anecdotal evidence of 800k pps.

--Michael Dillon

[Greg VILLAIN]

Post by m***@bt.com

Post by Chris Grundemann
http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a
wall at 700K pps and was using two dual core Intel Xeon 64bit
2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950.

Unless I am misreading this, he did not hit a wall. What he did was
test a design that was scalable to multiple cores and show that the
two core version could not go beyond 700k pps. The next logical question
is how much more can you push with larger numbers of cores. The key
thing is to use a recent Linux kernel that can share interrupts among
multiple cores and to run it on a CPU using MSI interrupts. Since this
was written up in January of 2007,
There are people who use Linux for load balancing who also are working
on finding how well it can cope with 10G of traffic and they have some
anecdotal evidence of 800k pps.
--Michael Dillon

If I just may share my opinion on this whole Software Router debate.
Even if it is technically feasible to route traffic over a server, I
would not hesitate to sound old-fashioned and state that it is not a
server's main role, i.e. what it is designed for.
Mainly, I would assume that you'd get the same Network I/O issues with
small packets that Disk I/O you would notice in a strictly systems/
server environment.
Most of all, Routing Equipment manufacturers offer more than a
physical routing chassis, they offer Hardware and Software support and
that I say, is essential - if you want open source in your routing
devices, I'd suggest you pick Juniper, their OS is BSDdey - you'll
love it, plus they will provide you with support, which good or bad,
will be better than none in times where you'll be stuck with an
undocumented memory leak of your favorite open source software routers.
It is not about making it work, it is about having it work -all the
time-, even if it is more costly, even if YOU have failed
troubleshooting a crash, SOMEONE will be forced to help you, by
contract.
Risk assessment folks, risk assessment...

Greg VILLAIN
Independant Network & Telco Architecture Consultant




Greg VILLAIN
Independant Network & Telco Architecture Consultant
[phone]

[Mark Newton]

Post by Greg VILLAIN
It is not about making it work, it is about having it work -all the
time-,

Hey, that sounds fantastic. Can you let me know where I can get one
of these platforms that works -all the time-? Because the ones I have
now crash occasionally, which is inconvenient. And untidy.

Surely the thing that needs to work all the time is the network, not any
individual forwarding element within the network. Design so that a
software
or hardware fault in a commodity-OS router doesn't take down the whole
network,
then you can leave any serious outages until next business day.

That's how we're meant to do things. Right?

Considering that all major vendors are using open-source OS's as the
embedded microkernel of choice and running their "operating system" as
an
application (anyone have any ACE blades in 6500's?), I'm not convinced
by
any FUD that says open source OS's aren't suitable for routers. All
we're
really talking about here is the depth of the abstractions that
implement
the features we need; whether they're on dedicated silicon, custom-
designed
hardware, or a PC doesn't matter at all if they all meet customer-stated
requirements for performance and reliability.


- mark


--
Mark Newton Email: ***@internode.com.au
(W)
Network Engineer Email:
***@atdot.dotat.org (H)
Internode Systems Pty Ltd Desk: [phone]
"Network Man" - Anagram of "Mark Newton" Mobile: [phone]

[m***@bt.com]

Post by Greg VILLAIN
Even if it is technically feasible to route traffic over a
server,

A computer running Linux is not a server. If it's job is to
route and forward packets, then it is a router. And the fact
that people are pushing the envelope at 10G is because Linux
routers are already known to do a great job at 1G and below.

Post by Greg VILLAIN
It is not about making it work, it is about having it work
-all the time-,

You can't buy "...having it work -all the time-". The only way
to achieve that is by building in proper redundancy, probably
N+2 or more, and by paying attention to all the deployment and
operational nits that equipment vendors will not do for you.
Companies like Google have demonstrated that this can be very
effective with cheap components failing all the time.

Post by Greg VILLAIN
even if it is more costly, even if YOU have
failed troubleshooting a crash, SOMEONE will be forced to
help you, by contract.

Contracts never force anyone to help you. They only force the
vendor to pay a penalty when they fail to help you. When miracles
are needed, you had better design it into the system yourself, not
rely on a vendor to provide them.

Post by Greg VILLAIN
Risk assessment folks, risk assessment...

Indeed!
If you want to cover your behind, then buy the big brand names and
forget about the consequences. If you want to operate a network service
that functions all the time, then make sure you control and mitigate
all the risk elements. Don't assume that a big brand name will do it
for you. In particular, big brand names focus their effort on the
majority customers so if you want to do something a little bit
different,
you will find that the big brand name won't be able to help you until
next year, and that is assuming that their priorities don't shift your
request right off the list.

--Michael Dillon

[Justin Shore]

Post by m***@bt.com

Post by Greg VILLAIN
Even if it is technically feasible to route traffic over a
server,

A computer running Linux is not a server. If it's job is to
route and forward packets, then it is a router. And the fact
that people are pushing the envelope at 10G is because Linux
routers are already known to do a great job at 1G and below.

Just because you can do something doesn't mean that you should.

Loading Image...
Loading Image...
Loading Image...
Loading Image...

Justin

[Adrian Chadd]

Post by Justin Shore

Post by m***@bt.com
A computer running Linux is not a server. If it's job is to
route and forward packets, then it is a router. And the fact
that people are pushing the envelope at 10G is because Linux
routers are already known to do a great job at 1G and below.

Just because you can do something doesn't mean that you should.

Hands up those of you running Cat6500's in service provider
environments.




Adrian

[Tore Anderson]

* Justin Shore

Post by Justin Shore
Just because you can do something doesn't mean that you should.

* Adrian Chadd

Post by Justin Shore
Hands up those of you running Cat6500's in service provider
environments.

*hand*

Actually, not quite yet, but I'm considering purchasing a pair of
Cat6500's (with Sup720 PFC3CXL) for a new colo I'm setting up, bundling
them together with VSS. They'll terminate a few transit links and links
to other colos, in addition to functioning as distribution/access
switches for the data center itself.

Are you saying that there's something about the Cat6500's that makes
them unsuitable for such usage? I'd sure like to hear more about that
before I go ahead and buy them, if so.

Up until now I've been using whiteboxes with Linux and Quagga for the
layer 3 services. This setup has served us well, but it seems we're
starting to approach a performance limit at around 1 Gbps routed traffic
so we need new gear anyway - a good time to start doing routing in
hardware, I thought.

Regards

[Robert Boyle]

Post by Tore Anderson

Post by Adrian Chadd
Hands up those of you running Cat6500's in service provider
environments.

*hand*
Actually, not quite yet, but I'm considering purchasing a pair of
Cat6500's (with Sup720 PFC3CXL) for a new colo I'm setting up, bundling
them together with VSS. They'll terminate a few transit links and links
to other colos, in addition to functioning as distribution/access
switches for the data center itself.
Are you saying that there's something about the Cat6500's that makes
them unsuitable for such usage? I'd sure like to hear more about that
before I go ahead and buy them, if so.

Cisco wants you to pay 4 times as much for the 7600 which is the same
platform except the cards are vertical instead of horizontal. (If you
have a NEBS chassis, then that's not even a differentiator.) Oh,
there is also a ROM/PROM/Flash chip in the chassis which tells IOS
that you are on a Catalyst and not a 7600 so the newer 7600 IOS code
supposedly won't work. This is the "code split" which they did about
a year? ago. The Catalyst works great as a core router, but Cisco
says that's the job for a 7600, not a 6500. I don't know if there are
any other differences, FlexWAN card support? But for most of us, the
6500 works great and does everything we need. That's what the OP was
referring to I believe.

-Robert



Tellurian Networks - Global Hosting Solutions Since 1995
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin

[Justin Shore]

Post by Robert Boyle

Post by Tore Anderson

Post by Adrian Chadd
Hands up those of you running Cat6500's in service provider
environments.

*hand*
Actually, not quite yet, but I'm considering purchasing a pair of
Cat6500's (with Sup720 PFC3CXL) for a new colo I'm setting up, bundling
them together with VSS. They'll terminate a few transit links and links
to other colos, in addition to functioning as distribution/access
switches for the data center itself.
Are you saying that there's something about the Cat6500's that makes
them unsuitable for such usage? I'd sure like to hear more about that
before I go ahead and buy them, if so.

Cisco wants you to pay 4 times as much for the 7600 which is the same
platform except the cards are vertical instead of horizontal. (If you
have a NEBS chassis, then that's not even a differentiator.) Oh, there
is also a ROM/PROM/Flash chip in the chassis which tells IOS that you
are on a Catalyst and not a 7600 so the newer 7600 IOS code supposedly
won't work. This is the "code split" which they did about a year? ago.
The Catalyst works great as a core router, but Cisco says that's the job
for a 7600, not a 6500. I don't know if there are any other differences,
FlexWAN card support? But for most of us, the 6500 works great and does
everything we need. That's what the OP was referring to I believe.

6708 linecards aren't supported in 7600s (though I think that was
supposed to change in SRC, which I'm not running yet because its
reported to be buggy as hell). Support for the ES linecards is only
found in the 7600 chassis too which has certain implications (some
critical) if you're doing MPLS. One thing that the 6500 can't do and
never will be able to do is CALEA. If you're a SP and have to have
support for CALEA as required by law (and can't get it closer to the
edge) then you should put some more thought into the 7600s or some other
solution.

Yesterday Gert posted an interesting take to the C-NSP list on the
Business Unit split that formally separated the 6500s from the 7600s.

http://puck.nether.net/pipermail/cisco-nsp/2008-March/049082.html

I'm not advocating one over the other but I am advocating a thorough
examination of one's needs, wants and requirements before buying one or
the other. We (and every other US SP) are required by law to support
CALEA for all broadband subs. Since we couldn't do it on our edge we
were forced to do it in the core. That required us to run SR and SR
made us buy 7600s. Others may not have that need.

Justin

[Chris Marlatt]

Post by Greg VILLAIN
It is not about making it work, it is about having it work -all the
time-, even if it is more costly, even if YOU have failed
troubleshooting a crash, SOMEONE will be forced to help you, by contract.
Risk assessment folks, risk assessment...
Greg VILLAIN
Independant Network & Telco Architecture Consultant

In my experience this is almost entirely the opposite. They're only
forced to help you if you're about to buy more gear from them and you're
holding that over their head.

There have been times where I've been the one forced to find the
solution and present it to the vendor. They had no "on the books"
solution to what was, in the end, truly a fairly simple problem.

Moral of the story, you don't always get what you pay for.

Regards,

Chris

[Adrian Chadd]

Post by m***@bt.com

Post by Chris Grundemann
http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a
wall at 700K pps and was using two dual core Intel Xeon 64bit
2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950.

Unless I am misreading this, he did not hit a wall. What he did was
test a design that was scalable to multiple cores and show that the
two core version could not go beyond 700k pps. The next logical question
is how much more can you push with larger numbers of cores. The key
thing is to use a recent Linux kernel that can share interrupts among
multiple cores and to run it on a CPU using MSI interrupts. Since this
was written up in January of 2007,
There are people who use Linux for load balancing who also are working
on finding how well it can cope with 10G of traffic and they have some
anecdotal evidence of 800k pps.

I didn't think the hardware quite worked like that :)

The paper doesn't cover -why- he hit a limit on a single core and why two cores
are any faster. He didn't do any benchmarking, no oprofile traces, etc.

What would be much more interesting is to see where its running out of steam,
and why more L1 cache helps. The AMD/Intel difference could be due to how
the memory systems operate/differ, but its all conjecture from me at this point.
I haven't looked into it in depth.

Just a random datapoint, some FreeBSD related people working on commercial
systems have noted they were able to achieve 1mil pps on intel gige hardware.
Its just not in open source. :)




Adrian

[Ray Burkholder]

Is there a multiport card out there on to which some of the forwarding
responsibilities can be offloaded? Perhaps the CPU doesn't need to see
every packet that arrives on the machine.

-----Original Message-----
From: owner-***@merit.edu [mailto:owner-***@merit.edu] On Behalf Of
Adrian Chadd
Sent: Wednesday, March 26, 2008 09:30
To: ***@bt.com
Cc: ***@nanog.org
Subject: Re: 10GE router resource

Post by m***@bt.com

Post by Chris Grundemann
http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a
wall at 700K pps and was using two dual core Intel Xeon 64bit
2.33GHz CPUs and 2GB of RAM in a Dell PowerEdge 1950.

Unless I am misreading this, he did not hit a wall. What he did was
test a design that was scalable to multiple cores and show that the
two core version could not go beyond 700k pps. The next logical question
is how much more can you push with larger numbers of cores. The key
thing is to use a recent Linux kernel that can share interrupts among
multiple cores and to run it on a CPU using MSI interrupts. Since this
was written up in January of 2007,
There are people who use Linux for load balancing who also are working
on finding how well it can cope with 10G of traffic and they have some
anecdotal evidence of 800k pps.

I didn't think the hardware quite worked like that :)

The paper doesn't cover -why- he hit a limit on a single core and why two
cores
are any faster. He didn't do any benchmarking, no oprofile traces, etc.

What would be much more interesting is to see where its running out of
steam,
and why more L1 cache helps. The AMD/Intel difference could be due to how
the memory systems operate/differ, but its all conjecture from me at this
point.
I haven't looked into it in depth.

Just a random datapoint, some FreeBSD related people working on commercial
systems have noted they were able to achieve 1mil pps on intel gige
hardware.
Its just not in open source. :)




Adrian

[m***@bt.com]

Post by Ray Burkholder
Is there a multiport card out there on to which some of the
forwarding responsibilities can be offloaded? Perhaps the
CPU doesn't need to see every packet that arrives on the machine.

Am I the only person who has heard of Google?

It didn't take me long to find this wiki page
http://www.bro-ids.org/wiki/index.php/ClusterFrontends
for an Opensource Intrusion Detection System that lists
various 10G cards for Linux and a couple of FPGA cards
so that you can roll your own ASICs. Anyway, this one
http://www.lewiz.com/talon3220.html
has two ports and claims to reach 8.8 Gbps with 1500 byte
packets.

People rolling their own router are not the only ones who
want to do 10G on Linux.

--Michael Dillon

[Robert Boyle]

Post by m***@bt.com

Post by Ray Burkholder
Is there a multiport card out there on to which some of the
forwarding responsibilities can be offloaded? Perhaps the
CPU doesn't need to see every packet that arrives on the machine.

Am I the only person who has heard of Google?
It didn't take me long to find this wiki page
http://www.bro-ids.org/wiki/index.php/ClusterFrontends
for an Opensource Intrusion Detection System that lists
various 10G cards for Linux and a couple of FPGA cards
so that you can roll your own ASICs. Anyway, this one
http://www.lewiz.com/talon3220.html
has two ports and claims to reach 8.8 Gbps with 1500 byte
packets.
People rolling their own router are not the only ones who
want to do 10G on Linux.

Anyone who wants to roll your own more advanced apps on Linux without
reinventing the wheel may want to check out my friend's company:

http://www.bivio.net/products/bivio7000.htm

Even with their specialized hardware platform, bus, and extensive
tuning, they only get 10Gb/s throughput on the dual or quad 10G
modules. However you can do 100,000 line ACLs at that speed. It is
built for a different application than core routing. However, an XMR
or Sup720 will still be a lot cheaper and give better performance.

-Robert



Tellurian Networks - Global Hosting Solutions Since 1995
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin

[Mark Tinka]

Post by Robert Boyle
Even with their specialized hardware platform, bus, and
extensive tuning, they only get 10Gb/s throughput on the
dual or quad 10G modules. However you can do 100,000 line
ACLs at that speed. It is built for a different
application than core routing. However, an XMR or Sup720
will still be a lot cheaper...

The chassis and switch fabric would generally be affordable
(it'd normally be a bundle). It's the cost of the 10-Gig-E
line cards that is the enemy.

Mark.

[Paul Vixie]

Post by m***@bt.com
People rolling their own router are not the only ones who
want to do 10G on Linux.

speaking of which, has anybody run "xorp" in production? it looks as much
like JunOS as quagga/zebra looks like IOS. if "click" works on current
hardware and if the xorp/click integration is good, this could be a great
science fair project for smaller network operators who need big PPS.

[Peter Wohlers]

Post by Paul Vixie

Post by m***@bt.com
People rolling their own router are not the only ones who
want to do 10G on Linux.

speaking of which, has anybody run "xorp" in production? it looks as much
like JunOS as quagga/zebra looks like IOS. if "click" works on current
hardware and if the xorp/click integration is good, this could be a great
science fair project for smaller network operators who need big PPS.

Vyatta is built on top of xorp. You can download the bootable iso from
their site and take a low-commitment look:
http://www.vyatta.com/download/index.php

--Peter

[Robert Bays]

Actually the latest version of Vyatta uses Quagga. If anyone is
interested in discussing the differences in running the two in
production networks feel free to contact me off list.

In full disclosure, I work for Vyatta.

Cheers,
Robert.

Post by Peter Wohlers
Vyatta is built on top of xorp. You can download the bootable iso from
http://www.vyatta.com/download/index.php
--Peter

[Sargun Dhillon]

Actually, soon this will no longer be true. Vyatta's new platform,
Glendale, will be moving to Quagga. Quagga is much more stable, and
slow-moving compared to Xorp, which makes me slightly more comfortable
(less breakage between versions). There are some major features lacking
inside of the platform. For example, it lacks the ability to do BFD, BGP
over IPSec, Multicast, etc... This major lack of features makes this a
hard to deploy piece of software. I am sure with enough customers Vyatta
would be able to catch up to Cisco. Also, from a viewpoint of hardware,
x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with
a switch) 1 GigE ports in it. Though, the way that Linux works, it
cannot handle high packet rates. If you are planning on handling large
flows with mostly large packets, you are alright for the most part. Just
be warned.

Post by Peter Wohlers

Post by Paul Vixie

Post by m***@bt.com
People rolling their own router are not the only ones who
want to do 10G on Linux.

speaking of which, has anybody run "xorp" in production? it looks as much
like JunOS as quagga/zebra looks like IOS. if "click" works on current
hardware and if the xorp/click integration is good, this could be a great
science fair project for smaller network operators who need big PPS.

Vyatta is built on top of xorp. You can download the bootable iso from
http://www.vyatta.com/download/index.php
--Peter

[William Herrin]

Post by Sargun Dhillon
from a viewpoint of hardware,
x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with
a switch) 1 GigE ports in it. Though, the way that Linux works, it
cannot handle high packet rates.

Correction: The way DRAM works, it cannot handle high packet rates.
Also note that the PCI-X bus tops out in the 7 to 8 gbps range and
it's half-duplex.

High-rate routers try to keep the packets in an SRAM queue and instead
of looking up destinations in a DRAM-based radix tree, they use a
special memory device called a TCAM.

http://www.pagiamtzis.com/cam/camintro.html

Regards.
Bill Herrin

[Sargun Dhillon]

I wonder how difficult it would be to integrate such a device on to
an x86 board cheaply. Something like NetFPGA (http://netfpga.org/) would
be an interesting place to start. The board has on board SRAM, a bit of
DRAM, an FPGA, and 2 GigE interfaces.
I know it definitely isn't normal for Network Operators to fund
research like this, but it would still be fairly interesting if there
was an Open Router Consortium (something for Vyatta to start?) with
hardware acceleration to X86 routers. Possibly even making Quagga a
mainstream control plane. Right now Quagga is controlled by a few
engineers from Sun. This nearly produces a conflict on interest (Sun
used to have their own routing platform). Anyways, to end my rambling...
As network operators would you finance a low, medium end router with
decent ROI. The question for developers (Vyatta primarily), could you
do what Digium did for Asterisk--become business front, and provide
platforms for Asterisk deployment in the enterprise--for Quagga, Linux,
etc?

Post by William Herrin

Post by Sargun Dhillon
from a viewpoint of hardware,
x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with
a switch) 1 GigE ports in it. Though, the way that Linux works, it
cannot handle high packet rates.

Correction: The way DRAM works, it cannot handle high packet rates.
Also note that the PCI-X bus tops out in the 7 to 8 gbps range and
it's half-duplex.
High-rate routers try to keep the packets in an SRAM queue and instead
of looking up destinations in a DRAM-based radix tree, they use a
special memory device called a TCAM.
http://www.pagiamtzis.com/cam/camintro.html
Regards.
Bill Herrin

[William Herrin]

Post by Sargun Dhillon
I wonder how difficult it would be to integrate such a device on to
an x86 board cheaply. Something like NetFPGA (http://netfpga.org/) would
be an interesting place to start. The board has on board SRAM, a bit of
DRAM, an FPGA, and 2 GigE interfaces.

Hi Sargun,

SRAM != TCAM. With SRAM you can only access one word per cycle. The
coolness of the TCAM is that the entire memory is queried in one
cycle, spitting out the best match.

Nevertheless, there is some interesting hardware out there. The Endace
DAG card with the coprocessor has a TCAM on it, but it's not big
enough to handle a full BGP table.

Regards,
Bill Herrin

[m***@bt.com]

Post by William Herrin
High-rate routers try to keep the packets in an SRAM queue
and instead of looking up destinations in a DRAM-based radix
tree, they use a special memory device called a TCAM.

FPGAs can be used to do both SRAM and TCAMs. All that is needed
is an FPGA board with 10G or a 10G card with an FPGA on it.
Although NetFPGA and RiceNIC are both 1G devices, there is a
certain commercial market for programmable high-speed network cards
for things like Intrusion Detection and data-center/GRID type
applications.

Anyone seriously interested in this area should start hunting amongst
the developers (and researchers) of embedded systems. You might end
up working with a university student in the Czech Republic to put his
TCAM/FPGA implementation onto a 10G card because the Internet breaks
down the barriers that high-margin vendors have used to create lock-in.
Bleeding edge networks may not be able to do this type of deal
but then, they are only 1% or less of the network operators out there.

--Michael Dillon

[Buhrmaster, Gary]

Post by m***@bt.com
FPGAs can be used to do both SRAM and TCAMs. All that is needed
is an FPGA board with 10G or a 10G card with an FPGA on it.

The Xilinx Virtex family can already do 10G, if you
are into FPGA development (I seem to recall the
first Xilinx FPGA that could do 10G was 4-5 years
ago; forever in Moore's law). Other vendors have
equivalent parts. And the Xilinx family has an
available PowerPC core. I seem to recall a couple
of vendors making available a (micro)Linux kernel
for running on same. All the hardware you need
for building your own 10G router. Just add
FPGA development resources, some planar board
design, and software.

[Andrew C Burnette]

Post by William Herrin

Post by Sargun Dhillon
from a viewpoint of hardware,
x86 is a fairly decent platform. I can stuff 40 (4x10GigE multiplex with
a switch) 1 GigE ports in it. Though, the way that Linux works, it
cannot handle high packet rates.

Correction: The way DRAM works, it cannot handle high packet rates.
Also note that the PCI-X bus tops out in the 7 to 8 gbps range and
it's half-duplex.

Indeed. PCI-X is already an EOL'ed interface, if only cheap PCI-X cards
were available. Once you add extensive ACL's, there's loads more
[central] processing to be done than just packet routing (100k choices
versus 2 to 4 interfaces). System throughput gets slammed rather
quickly. Linux IPtables grumbles painfully at 100k line ACLs :) Not to
mention the options of what to do with a packet are very limited.

The AMD chips with extra L1 cache perform better on *bsd platforms as
the forwarding code is tight and likes to stay close to the CPU, and
context switching kills packet processing performance (thus the small
but notable increase in the multicore performance). The GP registers on
the AMD platform are also easy to deal with (and in 64 bit mode, you get
double the number for free) essentially working an end around a broken
stack architecture from a few decades ago....anyone recall the
simplicity of assembly language of the 6800 or the 6502? :-)

getting the latency down low enough for HPC clusters is a major hassle,
as the x86 PC design just doesn't have the bandwidth.

Of course, Intel makes some slick NPU's for custom work (e.g.
cloudshield.com). If you like starting at bit 0. (isn't that like slot
zero or port zero, it technically doesn't exist since zero is only a
placeholder in larger numbers if you mean anything greater than none? I
could swear back in the days of a SLC96, ports were 1-96, not 0-95 :-) )

http://developer.intel.com/design/network/products/npfamily/index.htm?iid=ncdcnav2+proc_netproc

too bad they [Intel] don't make a hypertransport capable version, or
you'd have one helluva multicore multiNPU system with no glue logic
required.

Fun to play around though.

regards,
andy

Post by William Herrin
High-rate routers try to keep the packets in an SRAM queue and instead
of looking up destinations in a DRAM-based radix tree, they use a
special memory device called a TCAM.
http://www.pagiamtzis.com/cam/camintro.html
Regards.
Bill Herrin

[Adrian Chadd]

Post by Andrew C Burnette
Indeed. PCI-X is already an EOL'ed interface, if only cheap PCI-X cards
were available. Once you add extensive ACL's, there's loads more
[central] processing to be done than just packet routing (100k choices
versus 2 to 4 interfaces). System throughput gets slammed rather
quickly. Linux IPtables grumbles painfully at 100k line ACLs :) Not to
mention the options of what to do with a packet are very limited.

I agree, and the rest of the discussion is interesting, but the iptables
deployments I've seen which do massive ACLs like this almost certainly end
up having ACLs you can collapse into a small number of set-lookup-and-act
rules.

Those set-lookup-and-act rules are much faster than the linear ACL lookups
which ipfw/iptables/ipf/pf/etc do by default (and all of them support
IP sets in some form or other); I did this trick recently to reduce the CPU
overhead on an old revision 2.8ghz P4 from 99% to <10% when routing 100mbit
of average-pps TCP.




Adrian

[Christopher Morrow]

Post by Chris Grundemann
Greg has laid out a great bit of information and I would like to add just
one possibility to the list of budget 10GE routers: Vyatta. According to a
recent press release from that company
(http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product
that is "2 to 3X higher performance at a cost savings of more than 75
percent" when compared to Cisco's 7200. Unfortunately I have not had the

when did the 7200 go 10ge?

[Robert Boyle]

On Tue, Mar 25, 2008 at 1:59 PM, Chris Grundemann

Post by Chris Grundemann
Greg has laid out a great bit of information and I would like to add just
one possibility to the list of budget 10GE routers: Vyatta. According to a
recent press release from that company
(http://www.vyatta.com/about/pressreleases.php?id=51) they offer a product
that is "2 to 3X higher performance at a cost savings of more than 75
percent" when compared to Cisco's 7200. Unfortunately I have not had the

when did the 7200 go 10ge?

Shh... It's a secret and hasn't been released yet. We have have a few
NPE-40Gs with four 10G XFP interfaces. ;) Nah... I'm just wishing...

-Robert


Tellurian Networks - Global Hosting Solutions Since 1995
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin

[Robert Boyle]

At 12:36 PM 3/25/2008, Greg VILLAIN wrote:
I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise

Post by Greg VILLAIN
and performance-wise, provided you do not need rocket-science features.
MLX/XMR models will surely do the trick perfectly.

I agree too. They still have a bit of development to do on the IPv6
side, but they are getting there. We are using them with Cat 65XXs
with SXF Sup720-3BXLs and XMRs. We run ISIS, BGP, and BFD. Everything
they say works really does. We have been very pleased. Definitely put
them on your short list. The price per port can't be beat and their
support is stellar. If you want to reliably route IPv4 and IPv6 at
wire speeds regardless of packet size or rate and optionally filter
at wire speed too on all ports then they make a great box.

-Robert



Tellurian Networks - Global Hosting Solutions Since 1995
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin

[Eddy Martinez]

Post by Greg VILLAIN
I'd strongly suggest Foundry, I'm a big fan of their kits, price-wise

Post by Greg VILLAIN
and performance-wise, provided you do not need rocket-science
features.
MLX/XMR models will surely do the trick perfectly.

I agree too. They still have a bit of development to do on the IPv6
side, but they are getting there. We are using them with Cat 65XXs
with SXF Sup720-3BXLs and XMRs. We run ISIS, BGP, and BFD.
Everything they say works really does. We have been very pleased.
Definitely put them on your short list. The price per port can't be
beat and their support is stellar. If you want to reliably route
IPv4 and IPv6 at wire speeds regardless of packet size or rate and
optionally filter at wire speed too on all ports then they make a
great box.
-Robert

Totally agree.
Foundry support is top notch and the boxes do deliver the promised
performance.

The headroom is impressive when the CPU is at 99%. Somehow *cough* we
(me) pegged
the CPU on the Server Irons and still had a very very responsive
console. Was able to find
the self inflicted error and fix the problem quickly. Out testers on
the outside say they did not
notice a performance degradation.

Foundry's performance and support make the price a clear value.

I've only experienced two flavors, Cisco and Foundry.

Eddy

[ann kok]

Hi Chris

Could you share your opensouce 10G info for me?

For the past 8 months, I have problem to use the 10G
in linux system. I have to continuous to upgrade the
hardwares...

my existing system is using the new CPU now, 4G
memory, 1 x 10G card plus several 1G NICs.
Intel 2 Ext CPU X9650 @ 3.00GHz
All CPU is in 100% used when it is in 4G totally
(download + upload).

thank you so much

Post by Chris Grundemann
Greg has laid out a great bit of information and I
would like to add just
Vyatta. According to a
recent press release from that company (
http://www.vyatta.com/about/pressreleases.php?id=51)
they offer a product
that is "2 to 3X higher performance at a cost
savings of more than 75
percent" when compared to Cisco's 7200.
Unfortunately I have not had the
opportunity to test or use the Vyatta routers yet; I
have however
successfully used other open-source Linux based
routers in the past with
great success. If you are looking for a truly
budget 10GE router, they may
be worth adding to the list and looking into.
On Tue, Mar 25, 2008 at 10:36 AM, Greg VILLAIN

Post by Greg VILLAIN

Post by user user
Hi everybody!
I find myself in the market for some 10GE

routers. As

Post by Greg VILLAIN

Post by user user
I don't buy these everyday, I was wondering if

any of

Post by Greg VILLAIN

Post by user user
you guys had any good resources for evaluating
different vendors and models. I'm mainly

thinking

Post by Greg VILLAIN

Post by user user
about non-vendor resources as the vendorspeak

sites

Post by Greg VILLAIN

Post by user user
are not that hard to find.
Also I'd love to hear recommendatios for

"budget" 10GE

Post by Greg VILLAIN

Post by user user
routers. The "budget" router would be used to

hook up

Post by Greg VILLAIN

Post by user user
client networks through one 10GE interface and

connect

Post by Greg VILLAIN

Post by user user
to different transit providers through two 10GE
interfaces.
- Zed

Hiya,
When it comes to budget, force10 are good. I

wouldn't be able to

Post by Greg VILLAIN
confirm if they're worth performance-wise.
I'd strongly suggest Foundry, I'm a big fan of

their kits, price-wise

Post by Greg VILLAIN
and performance-wise, provided you do not need

rocket-science features.

Post by Greg VILLAIN
MLX/XMR models will surely do the trick perfectly.
When it comes to router purchasing habits, we all

tend to get

Post by Greg VILLAIN
religious...
Bottom line is that most of the 'regular' vendors

(namely Cisco,

Post by Greg VILLAIN
Juniper, Foundry, Force10, Extreme, Riverstone)

implement pretty much

Post by Greg VILLAIN
the same set of features, which are all IETF/IEEE

normalized, meaning

Post by Greg VILLAIN
if you don't need proprietary features (and you'll

wish you don't),

Post by Greg VILLAIN
any router will be fine, the only difference will
- the chassis being non-blocking or not (i.e.

backplane design)

Post by Greg VILLAIN
- the price per port
- the operating OS
- the feeling you'll get with the salesperson, and

the reputation of

Post by Greg VILLAIN
their Support Teams.
- vendor specific features such as Flow Sampling
To make it simple, most vendors have an IOS like

OS, except Juniper

Post by Greg VILLAIN
which has a really clever and elegant OS, but are

very pricey.

Post by Greg VILLAIN
Foundry and Force10 have the cheapest price per

port

Post by Greg VILLAIN
Cisco does only Netflow, Foundry & Force10 only

SFlow (which is a true

Post by Greg VILLAIN
standard) and I think Juniper does JFlow
Cisco's kits are packed with proprietary protocols

(HSRP and GLBP

Post by Greg VILLAIN
instead of VRRP, their own ethernet trunking,

EIGRP as their own and

Post by Greg VILLAIN
yet extremely efficient IGP, TCL scriptable

CLI...) , some of them are

Post by Greg VILLAIN
really good, some are crappy, but I suggest you'd

stick with IEEE/IETF

Post by Greg VILLAIN
protocol to avoid future trouble.
One thing: RSTP/802-1w is very (very, very, very)

not often

Post by Greg VILLAIN
interoperable between vendors who all have their

own interpretation of

Post by Greg VILLAIN
the norm and can quickly turn into a nightmare.
I'd strongly suggest try&buys if (R)STP

interoperability is required,

Post by Greg VILLAIN
but I'm a little paranoid :)
Greg VILLAIN
Independant Network & Telco Architecture

Consultant
--
"Those who do not create the future they want must
endure the future they
get."
~Draper L. Kaufman, Jr.
--

____________________________________________________________________________________
Looking for last minute shopping deals?
Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping

[Patrick Clochesy]

Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?

I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.

AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct?

-Patrick

----- Original Message -----
From: "Adrian Chadd" <***@creative.net.au>
To: "Chris Grundemann" <***@gmail.com>
Cc: "William Herrin" <herrin-***@dirtside.com>, ***@nanog.org
Sent: Tuesday, March 25, 2008 6:02:03 PM (GMT-0800) America/Los_Angeles
Subject: Re: 10GE router resource

Post by Chris Grundemann
To Ann's question on resources; I have only used Linux routers with 1G
ports but have surpassed 10G total throughput (up+ down) using various
dual proc set ups, most often Intel Xeon in Dell servers. A gentlemen
by the name of Martin Pels wrote a good paper on the subject early
http://docs.rodecker.nl/10-GE_Routing_on_Linux.pdf. He hit a wall at
700K pps and was using two dual core Intel Xeon 64bit 2.33GHz CPUs and
2GB of RAM in a Dell PowerEdge 1950.

Mike Tancsa did some benchmarking in late 2006:

http://www.tancsa.com/blast.html

I think things are slightly faster now but not because of a massive
change in software architecture.




Adrian

[Aaron Glenn]

Post by Patrick Clochesy
Very interesting study I had not seen, and a bummer. That really puts a
cramp in my advocation of our CARP+pf load balancers/firewalls/gateways.
Than again, what's a PIX box capable of?

I'd rather tweak a whitebox than pay through the nose for a PIX.

Post by Patrick Clochesy
I also had to switch to OpenBSD as there was a fatal crash with the bridge
device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage
of the other 3 cores, correct?

Correct. There has been some great speed and efficiency improvements
in pf and other networking parts of OpenBSD; though from anecdotal
evidence, 10GbE is not ready for 'primetime' (for certain definitions
of 'primetime').

actually I'll just skip making an ass out of myself and hope henning@
chimes in, since I believe he reads NANOG as well.

aaron.glenn

[Patrick Giagnocavo]

Post by Aaron Glenn

Post by Patrick Clochesy
Very interesting study I had not seen, and a bummer. That really puts a
cramp in my advocation of our CARP+pf load balancers/firewalls/gateways.
Than again, what's a PIX box capable of?

I'd rather tweak a whitebox than pay through the nose for a PIX.

Curious if you or others have tried Solaris 10 or OpenSolaris, they
claim that they are approaching wire speed 10G with the right card
(possibly their own, which is about $995 list).

--Patrick

[Adrian Chadd]

Post by Patrick Giagnocavo

Post by Aaron Glenn

Post by Patrick Clochesy
Very interesting study I had not seen, and a bummer. That really puts a
cramp in my advocation of our CARP+pf load balancers/firewalls/gateways.
Than again, what's a PIX box capable of?

I'd rather tweak a whitebox than pay through the nose for a PIX.

Curious if you or others have tried Solaris 10 or OpenSolaris, they
claim that they are approaching wire speed 10G with the right card
(possibly their own, which is about $995 list).

FreeBSD is doing wirespeed 10GE.

Oh wait, do you mean forwarding, or just TCP? :)



Adrian

[Andy Dills]

Post by Aaron Glenn

Post by Patrick Clochesy
Very interesting study I had not seen, and a bummer. That really puts a
cramp in my advocation of our CARP+pf load balancers/firewalls/gateways.
Than again, what's a PIX box capable of?

I'd rather tweak a whitebox than pay through the nose for a PIX.

Post by Patrick Clochesy
I also had to switch to OpenBSD as there was a fatal crash with the bridge
device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage
of the other 3 cores, correct?

Correct. There has been some great speed and efficiency improvements
in pf and other networking parts of OpenBSD; though from anecdotal
evidence, 10GbE is not ready for 'primetime' (for certain definitions
of 'primetime').

Anybody who does any sort of home-brew routing NEEDS to read this post:

http://lists.freebsd.org/pipermail/freebsd-current/2008-January/082469.html

Quote:
---
Forwarding (routing between multiple interfaces) and filtering
(ipfw) IIRC with quad Intel e1000 NIC:

Dual Intel Xeon 2.8GHz: 240Kpps 12k L1 cache
Single Intel Xeon 2.8GHz: 380Kpps 12k L1 cache
Core 2 Duo 1.8Ghz: 420kpps 12k L1 cache
Single Pentium-M 1.8GHz: 550Kpps 32k L1 cache
Dual AMD opteron 2GHz: 890Kpps 64k L1 cache
Single AMD opteron 2GHz: 970Kpps 64k L1 cache

All these hosts had 255 vlan interfaces with about 3000 routes and
about 30000 firewall rules, with a good spread of packets between
the interfaces with polling and fastforwarding. I struggled to
generate enough packets to load the AMD routers.
---



Quite interesting data, no? Especially when you can now get 3GHz opterons
with 128k of L1 cache?

How sweet is a sub-$1k router that can do multiple gig-e's at 1.5mpps?
Sounds like a dynamite platform for high-end datacenter CPEs that are soft
on dynamic routing...and even the open-source dynamic routing is
reasonably solid these days...

Andy

---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---

[Alex Rubenstein]

Post by Andy Dills
How sweet is a sub-$1k router that can do multiple gig-e's at 1.5mpps?
Sounds like a dynamite platform for high-end datacenter CPEs that are soft
on dynamic routing...and even the open-source dynamic routing is
reasonably solid these days...

I can't believe I am about to ask this on a public mailing list, but..

Has anyone tested this in even a remotely production environment, while
running any sort of MPLS LDP as a LSR?

[Christopher Morrow]

Post by Alex Rubenstein

Post by Andy Dills
How sweet is a sub-$1k router that can do multiple gig-e's at 1.5mpps?
Sounds like a dynamite platform for high-end datacenter CPEs that are soft
on dynamic routing...and even the open-source dynamic routing is
reasonably solid these days...

I can't believe I am about to ask this on a public mailing list, but..
Has anyone tested this in even a remotely production environment, while
running any sort of MPLS LDP as a LSR?

bahahaah! oh, sorry...

also, how does all-small-packets performance and reasonable ACL
behaviour work? (reasonable for dos things let's keep under 1k acl
lines) What about IDB-type numbers? is this a 10-interfaces at
line-rate or 10k interfaces at line-rate (line-rate on say ... 8 10G
interfaces)?

Scaling a routing platform in software for high bandwidth services is
difficult... or seems to be at least.

-Chris

[Henning Brauer]

Post by Aaron Glenn

Post by Patrick Clochesy
Very interesting study I had not seen, and a bummer. That really puts a
cramp in my advocation of our CARP+pf load balancers/firewalls/gateways.
Than again, what's a PIX box capable of?

I'd rather tweak a whitebox than pay through the nose for a PIX.

Post by Patrick Clochesy
I also had to switch to OpenBSD as there was a fatal crash with the bridge
device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage
of the other 3 cores, correct?

Correct. There has been some great speed and efficiency improvements
in pf and other networking parts of OpenBSD; though from anecdotal
evidence, 10GbE is not ready for 'primetime' (for certain definitions
of 'primetime').
chimes in, since I believe he reads NANOG as well.

occasionally.

as with all other OSes constructed benchmarks would show 10GE to work at
wirespeed with reasonable hardware. I would not use it (yet) if I truly
need 10 GBit/s forwarding rate, and that goes for any OS.

[Adrian Chadd]

Post by Patrick Clochesy
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?

Well, you get what you pay for. If you're willing to blow $10k on a
firewall, maybe you'll be willing to blow $10k on a *BSD developer
to work on improving forwarding performance.

It'd only take ten or so people to make donations or sponsor work
of that size for the benefits to appear.

Post by Patrick Clochesy
I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.

Did you log a bug? :)

Post by Patrick Clochesy
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct?

Uhm, its not quite that simple. ithreads on FreeBSD at least will run on
one CPU at a time (unless you're running some hacked up russian-driven
intel gige driver, which runs multiple ithreads for the device to improve
performance under certain circumstances!) and these classes of cards and
busses wouldn't benefit from >1 core contending for one card/bus.

If you're running >1 card then you may find the ithreads run on different
CPUs, each doing lookups and forwarding, but I haven't sat down and looked
at that sort of forwarding performance under FreeBSD. My focus at the moment
is "tcp proxy on a stick" throughput with one interfaces and >1 core doing
userland processing.




Adrian

[Chris Marlatt]

Post by Patrick Clochesy
Very interesting study I had not seen, and a bummer. That really puts a cramp in my advocation of our CARP+pf load balancers/firewalls/gateways. Than again, what's a PIX box capable of?
I also had to switch to OpenBSD as there was a fatal crash with the bridge device in FreeBSD when used with my paticular OpenVPN/CARP/pf combination.
AFAIK pf/forwarding only takes place on one core and wouldn't take advantage of the other 3 cores, correct?
-Patrick

http://pf4freebsd.love2party.net/pflock/ is worth a quick read. 7.0
already supports some SMP networking but when the pflock changes are
done you'll likely see some pretty serious performance from those devices.

Regards,

Chris

[Henning Brauer]

Post by Patrick Clochesy
I also had to switch to OpenBSD

congrats

Post by Patrick Clochesy
AFAIK pf/forwarding only takes place on one core and wouldn't take
advantage of the other 3 cores, correct?

for the moment, yes.

[ann kok]

Hi

IMHO, people don't want to do this way when they have
the budget.

just Oracle VS Mysql

I admit open source is good. but just for employer

Post by Greg VILLAIN

Post by m***@bt.com

Post by Greg VILLAIN
Even if it is technically feasible to route

traffic over a

Post by m***@bt.com

Post by Greg VILLAIN
server,

A computer running Linux is not a server. If it's

job is to

Post by m***@bt.com
route and forward packets, then it is a router.

And the fact

Post by m***@bt.com
that people are pushing the envelope at 10G is

because Linux

Post by m***@bt.com
routers are already known to do a great job at 1G

and below.
Just because you can do something doesn't mean that
you should.

http://katcampbell.files.wordpress.com/2007/07/overload.jpg

Post by Greg VILLAIN
http://www.gpsa.co.za/Jokes/OVERLOAD.JPG
http://englishrussia.com/images/overload.jpg

http://www.tensionnot.com/images/images/Automobiles587.jpg

Post by Greg VILLAIN
Justin

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ