IOS rootkits

[Gadi Evron]

At the upcoming EusecWest Sebastian Muniz will apparently unveil an IOS
rootkit. skip below for the news item itself.

We've had discussions on this before, here and elsewhere. I've been
heavily attacked on the subject of considering router security as an issue
when compared to routing security.

I have a lot to say about this, looking into this threat for a
few years now and having engaged different organizations within Cisco on
the subject in the past. Due to what I refer to as an "NDA of
honour" I will just relay the following until it is "officially" public,
then consider what should be made public, including:

1. Current defense startegies possible with Cisco gear
2. Third party defense strategies (yes, they now exist)
2. Cisco response (no names or exact quotes will likely be given)
3. A bet on when such a rootkit would be public, and who won it
(participants are.. "relevant people").

From:
http://www.networkworld.com/news/2008/051408-hacker-writes-rootkit-for-ciscos.html

"A security researcher has developed malicious rootkit software for
Cisco's routers, a development that has placed increasing scrutiny on the
routers that carry the majority of the Internet's traffic.

Sebastian Muniz, a researcher with Core Security Technologies, developed
the software, which he will unveil on May 22 at the EuSecWest conference
in London. "

Gadi Evron.

[Paul Wall]

Gadi,

Please try to keep the self-promotion to a minimum, and come back when
you have meaningful data to share with operators.

Examples would include a list of affected platforms and code
revisions, as well as preventative measures.

Thank you,
Paul

Post by Gadi Evron
At the upcoming EusecWest Sebastian Muniz will apparently unveil an IOS
rootkit. skip below for the news item itself.
We've had discussions on this before, here and elsewhere. I've been
heavily attacked on the subject of considering router security as an issue
when compared to routing security.
I have a lot to say about this, looking into this threat for a
few years now and having engaged different organizations within Cisco on
the subject in the past. Due to what I refer to as an "NDA of
honour" I will just relay the following until it is "officially" public,
1. Current defense startegies possible with Cisco gear
2. Third party defense strategies (yes, they now exist)
2. Cisco response (no names or exact quotes will likely be given)
3. A bet on when such a rootkit would be public, and who won it
(participants are.. "relevant people").
http://www.networkworld.com/news/2008/051408-hacker-writes-rootkit-for-ciscos.html
"A security researcher has developed malicious rootkit software for
Cisco's routers, a development that has placed increasing scrutiny on the
routers that carry the majority of the Internet's traffic.
Sebastian Muniz, a researcher with Core Security Technologies, developed
the software, which he will unveil on May 22 at the EuSecWest conference
in London. "
Gadi Evron.
_______________________________________________
NANOG mailing list
http://mailman.nanog.org/mailman/listinfo/nanog

[Gadi Evron]

Post by Paul Wall
Gadi,
Please try to keep the self-promotion to a minimum, and come back when
you have meaningful data to share with operators.
Examples would include a list of affected platforms and code
revisions, as well as preventative measures.

Name on the door, money to be sent via paypal. I will sign my playgirl
cover for 5 USD each.

This is operational, and it is about me saying "na na na na na, na na na
na na na" to a discussion from two years ago. I have every intention to
gloat, but I will keep it to a minimum.

Yes?

Gadi.

Post by Paul Wall

Post by Gadi Evron
At the upcoming EusecWest Sebastian Muniz will apparently unveil an IOS
rootkit. skip below for the news item itself.
We've had discussions on this before, here and elsewhere. I've been
heavily attacked on the subject of considering router security as an issue
when compared to routing security.
I have a lot to say about this, looking into this threat for a
few years now and having engaged different organizations within Cisco on
the subject in the past. Due to what I refer to as an "NDA of
honour" I will just relay the following until it is "officially" public,
1. Current defense startegies possible with Cisco gear
2. Third party defense strategies (yes, they now exist)
2. Cisco response (no names or exact quotes will likely be given)
3. A bet on when such a rootkit would be public, and who won it
(participants are.. "relevant people").
http://www.networkworld.com/news/2008/051408-hacker-writes-rootkit-for-ciscos.html
"A security researcher has developed malicious rootkit software for
Cisco's routers, a development that has placed increasing scrutiny on the
routers that carry the majority of the Internet's traffic.
Sebastian Muniz, a researcher with Core Security Technologies, developed
the software, which he will unveil on May 22 at the EuSecWest conference
in London. "
Gadi Evron.
_______________________________________________
NANOG mailing list
http://mailman.nanog.org/mailman/listinfo/nanog

[Dragos Ruiu]

The question this presentation begs for me... is how many of the folks
on this list do integrity checking on their routers?

You can no longer say this isn't necessary :-).

I know FX and a few others are working on toolsets for this...

I'll probably have other comments after I see the presentation.
This development has all sort of implications for binary signing
requirements, etc...

cheers,
--dr

--
World Security Pros. Cutting Edge Training, Tools, and Techniques
London, U.K. May 21/22 - 2008 http://cansecwest.com
pgpkey http://dragos.com/ kyxpgp

[Deepak Jain]

Wouldn't this level of verification/authentication of running code be a
pretty trivial function via RANCID or similar tool?

I understand *why* we are worried about rootkits on individual servers.
On essentially "closed" platforms this isn't going to be rocket science.
It may seem odd by today's BCPs, but booting up from "golden" images via
write-protected hardware or TFTP or similar is pretty straightforward
-- especially for those of us who run large server farms.

A POP or node could certainly keep a few servers around that are a
permanent repository of these items for all the devices that get images.

If you can't trust the boot rom, well, that's an entirely separate matter.

I think the issue with rootkits whether server or embedded device is
more about infection vector than the maliciousness that could be caused
AFTER a compromise has occurred.

Deepak Jain

Post by Dragos Ruiu
The question this presentation begs for me... is how many of the folks
on this list do integrity checking on their routers?
You can no longer say this isn't necessary :-).
I know FX and a few others are working on toolsets for this...
I'll probably have other comments after I see the presentation.
This development has all sort of implications for binary signing
requirements, etc...
cheers,
--dr
--
World Security Pros. Cutting Edge Training, Tools, and Techniques
London, U.K. May 21/22 - 2008 http://cansecwest.com
pgpkey http://dragos.com/ kyxpgp
_______________________________________________
NANOG mailing list
http://mailman.nanog.org/mailman/listinfo/nanog

[Buhrmaster, Gary]

Post by Deepak Jain
I understand *why* we are worried about rootkits on
individual servers.
On essentially "closed" platforms this isn't going to be
rocket science.
It may seem odd by today's BCPs, but booting up from "golden"
images via
write-protected hardware or TFTP or similar is pretty
straightforward

Since todays bootstrap codes are in EEPROM (or
equivalent), if you get "root" once, you can
have "root" forever. Faking file system content
(and real time replacing of code) is the core
of any current (good) Linux/Mac/Windows rootkit.
Cisco/Juniper/Force10/whatever is just another
platform to do the same if you can replace the
bootstrap. Modular IOS might even make it
easier to do dynamic code insertion.

There are platforms (Xbox?, Tivo?, etc.) that try
to do cryptographic validation of the code they
are loading. Network devices are not yet doing
a true cryptograhic validation as far as I know,
although one could imagine that that might be a
next step to protect against that specific threat
(although I seem to recall that bypassing the Xbox
validations only took a few months, so it is harder
than it first appears to get right).

Gary

[Deepak Jain]

Post by Buhrmaster, Gary

Post by Deepak Jain
I understand *why* we are worried about rootkits on
individual servers.
On essentially "closed" platforms this isn't going to be
rocket science.
It may seem odd by today's BCPs, but booting up from "golden"
images via
write-protected hardware or TFTP or similar is pretty
straightforward

Since todays bootstrap codes are in EEPROM (or
equivalent), if you get "root" once, you can
have "root" forever. Faking file system content
(and real time replacing of code) is the core
of any current (good) Linux/Mac/Windows rootkit.
Cisco/Juniper/Force10/whatever is just another
platform to do the same if you can replace the
bootstrap. Modular IOS might even make it
easier to do dynamic code insertion.
There are platforms (Xbox?, Tivo?, etc.) that try
to do cryptographic validation of the code they
are loading. Network devices are not yet doing
a true cryptograhic validation as far as I know,
although one could imagine that that might be a
next step to protect against that specific threat
(although I seem to recall that bypassing the Xbox
validations only took a few months, so it is harder
than it first appears to get right).

I think that is exactly the point. Once a box has been thoroughly
compromised, its almost impossible to bring it back to a "known, good"
state without a complete (reformat). In the case of embedded HW, that
may include wiping/rewriting the EEPROMs to a known good state.

I don't think this is going to be outside of the purview of Network
Operators for very long, no matter what the case.

Anti-virii and such are somewhat interesting in the end-system model,
but when downtimes need to be scheduled significantly in advance for
network operations you either a) prevent infection by much tighter
controls at the get-go or b) provide a high-trust way to keep the
systems in a known good-state. This, of course, assumes true "bugs" are
kept to a minimum.

It does raise significant security concerns for those networks that have
employees/contractors/etc with turn-over that could leave a parting
"gift" in their respective networks. Changing passwords isn't really
sufficient anymore.

DJ

[Gadi Evron]

Post by Deepak Jain
Wouldn't this level of verification/authentication of running code be a
pretty trivial function via RANCID or similar tool?

Absolutely, and it actually makes sense. The problem though is that it is
one again an escalation war and counter-inventions keep happening. RANCID
will connect remotely and use the local tools to get results, these local
tools or their esults can be altered.

Post by Deepak Jain
I understand *why* we are worried about rootkits on individual servers. On
essentially "closed" platforms this isn't going to be rocket science.
It may seem odd by today's BCPs, but booting up from "golden" images via
write-protected hardware or TFTP or similar is pretty straightforward --
especially for those of us who run large server farms.

That is a neat idea, you mean something like a magic card?
Well, the rootkit could still hide in memory, or heck, on the video card
if it likes. While XR is not implemented your best bet is reflashing with
an updated version, screws up the memory allocations which is apparently a
difficult problem to overcome.

Post by Deepak Jain
A POP or node could certainly keep a few servers around that are a permanent
repository of these items for all the devices that get images.
If you can't trust the boot rom, well, that's an entirely separate matter.
I think the issue with rootkits whether server or embedded device is more
about infection vector than the maliciousness that could be caused AFTER a
compromise has occurred.

Here is very much disagree with you. Imagine what you can do with a Trojan
horse on a computer, say a server. You could, in effective terms, use it
as your own. You'd own it. The same is true for a router.

You could sniff the network, steal traffic, use it as a bridge to connect
to potnetially any part of your network, hide traffic, etc. The potential
for attackrs is almosy "cool".

Gadi.

Post by Deepak Jain
Deepak Jain

The question this presentation begs for me... is how many of the folks on
this list do integrity checking on their routers?
You can no longer say this isn't necessary :-).
I know FX and a few others are working on toolsets for this...
I'll probably have other comments after I see the presentation.
This development has all sort of implications for binary signing
requirements, etc...
cheers,
--dr
--
World Security Pros. Cutting Edge Training, Tools, and Techniques
London, U.K. May 21/22 - 2008 http://cansecwest.com
pgpkey http://dragos.com/ kyxpgp
_______________________________________________
NANOG mailing list
http://mailman.nanog.org/mailman/listinfo/nanog

[Paul Ferguson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Dragos Ruiu
The question this presentation begs for me... is how many of the folks
on this list do integrity checking on their routers?
You can no longer say this isn't necessary :-).
I know FX and a few others are working on toolsets for this...
I'll probably have other comments after I see the presentation.
This development has all sort of implications for binary signing
requirements, etc...

Yep -- I'd say just wait for the presentation (assuming Cisco
doesn't go after this guy like they did Mike Lynn) and then
determine the level of seriousness.

It would appear to have people very nervous, however. Including
Cisco. It will be interesting to see what develops.

- - ferg

-----BEGIN PGP SIGNATURE-----
Version: PGP Desktop 9.6.3 (Build 3017)

wj8DBQFILlgzq1pz9mNUZTMRAtmoAKC3bQLSqJzFDZklPMfdnkBX7fyccwCeN5mc
K1QQ9JnTqLmSfcNuj5JZ6Z8=
=W5F0
-----END PGP SIGNATURE-----


--
"Fergie", a.k.a. Paul Ferguson
Engineering Architecture for the Internet
fergdawg(at)netzero.net
ferg's tech blog: http://fergdawg.blogspot.com/

[Paul Wall]

What if some good comes from this "root kit"?

For instance, what if it lets us fix things like DOM on non-Cisco
XENPAKs and SFPs? Or lets us un-cripple our 6500 chassis to run the
code we want?

Of course, given the messenger, I'm sure it's just hype to help
bolster Gadi's security practice, and will prove to be no big deal.

Paul

[Matthew Moyle-Croft]

Post by Paul Wall
What if some good comes from this "root kit"?

I'm sure it'll be good for a number of security providers to hawk their
wares.

If the way of running this isn't out in the wild and it's actually
dangerous then a pox on anyone who releases it, especially to gain
publicity at the expensive of network operators sleep and well being.
May you never find a reliable route ever again.

MMC

[Simon Lockhart]

Post by Matthew Moyle-Croft

Post by Paul Wall
What if some good comes from this "root kit"?

I'm sure it'll be good for a number of security providers to hawk their
wares.

How long before we need to install Anti-virus / Anti-root-kit software on
our routers?

Simon

[Matthew Moyle-Croft]

Post by Simon Lockhart
How long before we need to install Anti-virus / Anti-root-kit software on
our routers?

Nah - we'll just replace them all with Macs. They don't need anti-virus ...

:-)

MMC

Post by Simon Lockhart
Simon

[Gadi Evron]

Post by Simon Lockhart

Post by Matthew Moyle-Croft

Post by Paul Wall
What if some good comes from this "root kit"?

I'm sure it'll be good for a number of security providers to hawk their
wares.

How long before we need to install Anti-virus / Anti-root-kit software on
our routers?

Very astute.

Sadly, this is already being done by a few people I know. No AV vendor has
such a tool to offer you, so don't bother asking them.

The question is, can you afford not to?

The answer may be yes, you can afford for your router to be a spying
machine for the enemy/competitor, and you can afford for it to be a bot
participating in DDoS (as currently, for example, many *nix routers are
known to be). The question is who can't afford for these things to happen...

Gadi.

Post by Simon Lockhart
Simon
--
Simon Lockhart | * Sun Server Colocation * ADSL * Domain Registration *
Director | * Domain & Web Hosting * Internet Consultancy *
_______________________________________________
NANOG mailing list
http://mailman.nanog.org/mailman/listinfo/nanog

[Gadi Evron]

Post by Gadi Evron
The question is who can't afford for these things to happen...
Gadi.

I can't help but feel you're pushing fear to further some other interest here
Gadi.

It is alright to have feelings.

Gadi.

[Mark Smith]

On Sat, 17 May 2008 07:03:58 -0500 (CDT)

Post by Gadi Evron

Post by Gadi Evron
The question is who can't afford for these things to happen...
Gadi.

I can't help but feel you're pushing fear to further some other interest here
Gadi.

It is alright to have feelings.

The rational thing to do is to move beyond fear.

[Matthew Moyle-Croft]

Post by Gadi Evron
It is alright to have feelings.
Gadi.

So I ask again, expecting nothing but another flippant answer:

Do you actually have live examples of this or able to demonstrate it or
are you just theorising about it all?

MMC

[Gadi Evron]

Post by Gadi Evron
It is alright to have feelings.
Gadi.

I will honour you flame-bait, but only once.

Do you actually have live examples of this or able to demonstrate it or are
you just theorising about it all?

Your question is irrelevant to our discussion, as I obviously base myself
on the first email in this thread discussing the poc (?) about to be
released, and my own statements from that first email in which I mention I
will not discuss my own experience on the subject of rootkit risks
and solutions until said poc (?) is released due to matters of honour.

MMC

[Matthew Moyle-Croft]

Post by Gadi Evron
The question is who can't afford for these things to happen...
Gadi.

I can't help but feel you're pushing fear to further some other interest
here Gadi.

Do you actually have live examples of this or able to demonstrate it or
are you just theorising about it all?

MMC

[Matthew Moyle-Croft]

I'd love to know what magical mystical protection your routers have that will
enable them to avoid the same fate as every other device and operating system
has. There's only one thing up there that doesn't have known rootkits
in the wild. Yet.

The question isn't IF routers have security vunerabilities, but whether
Gadi has an example he can demonstrate now of installing a root kit on
an IOS router NOW or not.

MMC

[Gadi Evron]

Post by Matthew Moyle-Croft

I'd love to know what magical mystical protection your routers have that will
enable them to avoid the same fate as every other device and operating system
has. There's only one thing up there that doesn't have known rootkits
in the wild. Yet.

The question isn't IF routers have security vunerabilities

Nope, the question is not about if routers have security vulnerabilities.
The question is how operators and organizations can defend their routers
against rootkits, and cisco's practices.

Post by Matthew Moyle-Croft
MMC

[Joel Jaeggli]

Post by Gadi Evron

Post by Matthew Moyle-Croft
The question isn't IF routers have security vunerabilities

Nope, the question is not about if routers have security vulnerabilities.
The question is how operators and organizations can defend their routers
against rootkits, and cisco's practices.

The existence proof of a root kit does little if anything to change how
one protects and secures the control plane.

[Jack Bates]

| Network administrators are not able to observe Lawful Intercept is
| enabled. No Lawful Intercept program messages or error messages are ever
| displayed on the console.
<http://www.cisco.com/en/US/docs/ios/12_0s/feature/guide/lawf_int.html>
This is a Sony-style rootkit, but it certainly demonstrate that the
concept is feasible (surprise).

Eh, it's a little misleading. Every Net admin knows when Lawful Intercept is
activated on their router. The processor utilization takes a major spike. What
it's doing might not be known, though umm, even intercept traffic itself can be
intercepted or redirected through portions of the network where it can be
intercepted. ;)

Jack

[Paul Wall]

It's the people who pop up and smear Gadi that I really wonder
about. There seems to be no good reason for this, unless possibly
they are blackhats of some sort. I remember a few years ago
when William Leibzon posted about his work which eventually
became completewhois.com and several blackhats popped up and
tried to smear him. So when people attack Gadi or anyone else
with no substantive facts to justify those attacks, I always
assume that they are part of the criminal gangs who drive network
abuse in the 21st century. Of course they may just be harmless
fools who think that they will become better network operators
if they can become part of the in group. Who knows...

Actually, Michael, folks who have problems with Gadi, William, and
certain other offenders are mainly annoyed with the quantity (high)
and quality (low) of their posts. That you seem to have a blind spot
in the direction of this particular explanation is dismaying but not
surprising.

Paul

[Suresh Ramasubramanian]

On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft

Post by Matthew Moyle-Croft
If the way of running this isn't out in the wild and it's actually
dangerous then a pox on anyone who releases it, especially to gain
publicity at the expensive of network operators sleep and well being.
May you never find a reliable route ever again.

This needs fixing. It doesnt need publicity at security conferences
till after cisco gets presented this stuff first and asked to release
an emergency patch.

--srs

[Jon Kibler]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Suresh Ramasubramanian
On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft

Post by Matthew Moyle-Croft
If the way of running this isn't out in the wild and it's actually
dangerous then a pox on anyone who releases it, especially to gain
publicity at the expensive of network operators sleep and well being.
May you never find a reliable route ever again.

This needs fixing. It doesnt need publicity at security conferences
till after cisco gets presented this stuff first and asked to release
an emergency patch.
--srs

According to Cisco, there is nothing to patch:
http://www.cisco.com/warp/public/707/cisco-sr-20080516-rootkits.shtml

Jon Kibler
- --
Jon R. Kibler
Chief Technical Officer
Advanced Systems Engineering Technology, Inc.
Charleston, SC USA
o: 843-849-8214
c: 843-224-2494
s: 843-564-4224

My PGP Fingerprint is:
BAA2 1F2C 5543 5D25 4636 A392 515C 5045 CF39 4253


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgusjEACgkQUVxQRc85QlO5kACfaZtij86HqIH540xeH+Uh/NyI
ccQAnjiRCMFnLxk/Ew9EuUKDzdLN6HQZ
=BCdw
-----END PGP SIGNATURE-----




==================================================
Filtered by: TRUSTEM.COM's Email Filtering Service
http://www.trustem.com/
No Spam. No Viruses. Just Good Clean Email.

[n3td3v]

On Sat, May 17, 2008 at 11:12 AM, Suresh Ramasubramanian

Post by Suresh Ramasubramanian
On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft

Post by Matthew Moyle-Croft
If the way of running this isn't out in the wild and it's actually
dangerous then a pox on anyone who releases it, especially to gain
publicity at the expensive of network operators sleep and well being.
May you never find a reliable route ever again.

This needs fixing. It doesnt need publicity at security conferences
till after cisco gets presented this stuff first and asked to release
an emergency patch.

Agreed,

You've got to remember though that a security conference is a
commercial venture, it makes business sense for this to be publically
announced at this security conference.

I think security conferences have become something that sucks as its
all become money making oriented and the people who run these things
don't really have security in mind, just the £ signs reflecting on
their eye balls.

Post by Suresh Ramasubramanian
--srs
--

All the best,

n3td3v

[Gadi Evron]

Post by Suresh Ramasubramanian
On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft

Post by Matthew Moyle-Croft
If the way of running this isn't out in the wild and it's actually
dangerous then a pox on anyone who releases it, especially to gain
publicity at the expensive of network operators sleep and well being.
May you never find a reliable route ever again.

This needs fixing. It doesnt need publicity at security conferences
till after cisco gets presented this stuff first and asked to release
an emergency patch.

I'd like to discuss:
1. What is it we are talking about.
2. Why it is serious.
3. What we can do to defend ourselves.

I'll be brief as this is not a briefing.

You are absolutely right on the sentiment, but miss the point on this
particular issue. I agree with you that in most cases, software
vulnerability issues should be resolved with the vendor first, especially
where critical infrastructure is involved. This is not only about
exploiting a vulnerability.

In this case it the the very realization that these issues exist
(namely being able to run Trojan horses on IOS systems AND/or hiding their
presense) is what we are discussing.

Router security as far as most operators are concerned includes the
following issues: software version (now update), configuration, ACL and
authentication (password) security. I include subjects such as BGP MD5 in
configuration.

These issues are indeed important and very neglected, after all, how many
"0wned" routers can be found that respond to cisco/cisco?

The main difference here is that we are now at a cross-roads where the
face of router security changes, It is that the realization that:

1. A router is not an hardware device, it is an embedded device with a
software operating system. As such it is as vulnerable to malware
(wide-spreading--worm, or targeted--Trojan horse) as a Windows machine
is.)

2. There are no real tools today for us to be able to detect such
malicious activity on a router, listing processes doesn't cut it.

3. What tools exist, which I hope to secure permission to discuss later
on, are only from third parties.

This is not about fear mongering, it's about facing reality how about how
Cisco handles security threats to their customer base before such an issue
becomes a public concern--namely, ignoring its very existence, at least as
far as the public can see.

The point is, I don't want to rely on third parties for my router's
security, even if I trust the said third party.

Gadi.

[Dragos Ruiu]

Post by Suresh Ramasubramanian
On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft

Post by Matthew Moyle-Croft
If the way of running this isn't out in the wild and it's actually
dangerous then a pox on anyone who releases it, especially to gain
publicity at the expensive of network operators sleep and well being.
May you never find a reliable route ever again.

This needs fixing. It doesnt need publicity at security conferences
till after cisco gets presented this stuff first and asked to release
an emergency patch.

Bullshit.

There is nothing to patch.

It needs to be presented at conferences, exactly because people will
play ostrich and stick their heads in the sand and pretend it can't
happen to them, and do nothing about it until someone shows them, "yes
it can happen" and here is how....

Which is exactly why we've accepted this talk. We've all known this is
a possibility for years, but I haven't seen significant motion forward
on this until we announced this talk. So in a fashion, this has
already helped make people more realistic about their infrastructure
devices. And the discussions, and idea interchange that will happen
between the smart folks at the conference will undoubtedly usher forth
other related issues and creative solutions. Problems don't get fixed
until you talk about them.

cheers,
--dr



--
World Security Pros. Cutting Edge Training, Tools, and Techniques
London, U.K. May 21/22 - 2008 http://cansecwest.com
pgpkey http://dragos.com/ kyxpgp

[Suresh Ramasubramanian]

Let's put it this way.

1. Yes there's nothing to patch, as such

2. It can be prevented by what's widely regarded as BCP on router
security, and has been covered at *nog, in cisco training material,
etc etc for quite some time now.

I am much less concerned about security conferences discussing this
than about the (highly uninformed) publicity that accompanies these
conferences.

Yes, this sounds a lot more like the bugtraq v/s full disclosure
discussion than I'm comfortable with, but I still think this could
have been handled a lot better.

--srs

Post by Dragos Ruiu
Bullshit.
There is nothing to patch.
It needs to be presented at conferences, exactly because people will play
ostrich and stick their heads in the sand and pretend it can't happen to
them, and do nothing about it until someone shows them, "yes it can happen"
and here is how....
Which is exactly why we've accepted this talk. We've all known this is a
possibility for years, but I haven't seen significant motion forward on this
until we announced this talk. So in a fashion, this has already helped make
people more realistic about their infrastructure devices. And the
discussions, and idea interchange that will happen between the smart folks
at the conference will undoubtedly usher forth other related issues and
creative solutions. Problems don't get fixed until you talk about them.
cheers,
--dr

[Gadi Evron]

Post by Suresh Ramasubramanian
Let's put it this way.
1. Yes there's nothing to patch, as such
2. It can be prevented by what's widely regarded as BCP on router
security, and has been covered at *nog, in cisco training material,
etc etc for quite some time now.
I am much less concerned about security conferences discussing this
than about the (highly uninformed) publicity that accompanies these
conferences.
Yes, this sounds a lot more like the bugtraq v/s full disclosure
discussion than I'm comfortable with, but I still think this could
have been handled a lot better.

It's easy to blame researchers for doing their studies, but the fact is,
if one whitehat researcher has done work on it, it is already exploited in
the wild.

Gadi.

Post by Suresh Ramasubramanian
--srs

Post by Dragos Ruiu
Bullshit.
There is nothing to patch.
It needs to be presented at conferences, exactly because people will play
ostrich and stick their heads in the sand and pretend it can't happen to
them, and do nothing about it until someone shows them, "yes it can happen"
and here is how....
Which is exactly why we've accepted this talk. We've all known this is a
possibility for years, but I haven't seen significant motion forward on this
until we announced this talk. So in a fashion, this has already helped make
people more realistic about their infrastructure devices. And the
discussions, and idea interchange that will happen between the smart folks
at the conference will undoubtedly usher forth other related issues and
creative solutions. Problems don't get fixed until you talk about them.
cheers,
--dr

_______________________________________________
NANOG mailing list
http://mailman.nanog.org/mailman/listinfo/nanog

[Dragos Ruiu]

Post by Suresh Ramasubramanian
2. It can be prevented by what's widely regarded as BCP on router
security, and has been covered at *nog, in cisco training material,
etc etc for quite some time now.
I am much less concerned about security conferences discussing this
than about the (highly uninformed) publicity that accompanies these
conferences.

I'm not going to touch the disclosure or not debate... it's been done.

But I will agree to disagree with you about the above two points.

First of all about prevention, I'm not at all sure about this being
covered by existing router security planning / BCP.
I don't believe most operators reflash their routers periodically, nor
check existing images (particularly because the tools for this
integrity verification don't even exist). If I'm wrong about this I
would love to be corrected with pointers to the tools.

Regarding the second point, I also lament the often liberal doses of
alarmism/FUD that get plastered over the popular media whenever
complicated technical issues are discussed - but unless we have some
have the discussions, and information dispersal, then the
misconceptions have no chance of being dispelled.
The threat of misinformed press does not seem to be sufficient to
justify censuring open discussion of the issues imho.

One of the thing I truly enjoy about the conferences we organize, is
seeing the synergism that occurs when multiple minds focus on these
security issues at the conferences. When the analysis is parallelized
over multiple brains, inevitably the creative solutions that occur
from the congregation of different viewpoints and ideas is pleasantly
surprising, and powerful. I've seen numerous examples of this: even
just last April I had a chance to be a fly on the wall at a discussion
between Jacob Appelbaum and Theo DeRaadt talking about the cold memory
attacks research Jacob started - the result of which was that during
the discussion it was realized that with the addition of about 30
lines of code in the power fail interrupt handler a large segment of
those attacks could be nullified, as they are now on OpenBSD. If the
discussion hadn't happened, the creative solution to it would have
never arisen. These kinds of "out of the box" solutions frequently
arise out of multi-person debate and free association that follows
discussions of serious issues - no-one has the whole picture and
adding other's viewpoints often brings superior solutions to problems
up.

So in my opinion the benefits of discussing serious issues at
conferences far outweigh the potential drawbacks of misguided media
coverage of them. What I infer from your post is that you are of the
opinion that issues such as this rootkit prototype should be reported
to CSIRT and then shuffled under a carpet. To which I respond that
that kind of attitude has led to what I currently consider to be an
inappropriate level of concern and awareness amongst service providers
of the seriousness of this threat. Cisco has some great guys, but
surely discussion of this threat amongst the wider security community
will lead to more and better solutions than Cisco operating in a
vacuum. And more importantly this issue is not a Cisco issue - the
basic threat vector should be a concern to other infrastructure
equipment manufacturers too. Until we talk about it, we cannot find
the right responses to the problem, and experts talking about it
usually leads to better and more comprehensive solutions than single
persons or smaller groups working in isolation.

cheers,
--dr

--
World Security Pros. Cutting Edge Training, Tools, and Techniques
London, U.K. May 21/22 - 2008 http://eusecwest.com
pgpkey http://dragos.com/ kyxpgp

[Joel Jaeggli]

Post by Dragos Ruiu
First of all about prevention, I'm not at all sure about this being
covered by existing router security planning / BCP.
I don't believe most operators reflash their routers periodically, nor
check existing images (particularly because the tools for this
integrity verification don't even exist). If I'm wrong about this I
would love to be corrected with pointers to the tools.

I have 6 years worth of rancid logs for every time the reported number
of blocks in use on my flash changes, I imagine others do as well.
That's hardly the silver bullet however.

We as I imagine others do expended a fair amount of cycles monitoring
who it is that our routers are talking to and protecting the integrity
of the communications channels that they use (bgp, ospf, ssh, tftp etc),
If a router has a tcp connection to someplace it shouldn't we'll
probably know about it. If it's announcing a prefix it shouldn't be,
we'll probably know about it, those are the easy ones though.

There are some things one might consider adding in terms of auditing,
comparing the running image more closely to the one in flash for
example, peroidic checksum of the on onflash image, after downloading to
another host would be another. I'm not sure that I'd trust the later
given the rooted box can I suppose hand you an unmodified version of the
subverted image.

In the end if you subvert a router, presumably you're doing it for a
purpose and given what the device does, that purpose is probably
detectable in a well instrumented network.

It is desirable I expect to insure that any locally stored security
credentials that might be subverted not be usable when connecting to
another router, that applies in a absence of root kits however.

Post by Dragos Ruiu
Regarding the second point, I also lament the often liberal doses of
alarmism/FUD that get plastered over the popular media whenever
complicated technical issues are discussed - but unless we have some
have the discussions, and information dispersal, then the
misconceptions have no chance of being dispelled.
The threat of misinformed press does not seem to be sufficient to
justify censuring open discussion of the issues imho.

[Gadi Evron]

Post by Joel Jaeggli

Post by Dragos Ruiu
First of all about prevention, I'm not at all sure about this being
covered by existing router security planning / BCP.
I don't believe most operators reflash their routers periodically, nor
check existing images (particularly because the tools for this
integrity verification don't even exist). If I'm wrong about this I
would love to be corrected with pointers to the tools.

I have 6 years worth of rancid logs for every time the reported number
of blocks in use on my flash changes, I imagine others do as well.
That's hardly the silver bullet however.
We as I imagine others do expended a fair amount of cycles monitoring
who it is that our routers are talking to and protecting the integrity
of the communications channels that they use (bgp, ospf, ssh, tftp etc),
If a router has a tcp connection to someplace it shouldn't we'll
probably know about it. If it's announcing a prefix it shouldn't be,
we'll probably know about it, those are the easy ones though.

I am very happy to hear you do these... very useful and will catch quite a
bit.

Post by Joel Jaeggli
There are some things one might consider adding in terms of auditing,
comparing the running image more closely to the one in flash for
example, peroidic checksum of the on onflash image, after downloading to
another host would be another. I'm not sure that I'd trust the later
given the rooted box can I suppose hand you an unmodified version of the
subverted image.

The result from your check can easily be modified, first thing I would
have changed is the checker. Say you did this from a usb stick--I'd just
hide the rootkit in memory.

Post by Joel Jaeggli
In the end if you subvert a router, presumably you're doing it for a
purpose and given what the device does, that purpose is probably
detectable in a well instrumented network.

Subversion may not be the goal. A router is perfect for faking outgoing
traffic. This traffic can contain stolen sniffed or relayed data.

Post by Joel Jaeggli
It is desirable I expect to insure that any locally stored security
credentials that might be subverted not be usable when connecting to
another router, that applies in a absence of root kits however.

[Joel Jaeggli]

Post by Gadi Evron

Post by Joel Jaeggli

Post by Dragos Ruiu
First of all about prevention, I'm not at all sure about this being
covered by existing router security planning / BCP.
I don't believe most operators reflash their routers periodically, nor
check existing images (particularly because the tools for this
integrity verification don't even exist). If I'm wrong about this I
would love to be corrected with pointers to the tools.

I have 6 years worth of rancid logs for every time the reported number
of blocks in use on my flash changes, I imagine others do as well.
That's hardly the silver bullet however.
We as I imagine others do expended a fair amount of cycles monitoring
who it is that our routers are talking to and protecting the integrity
of the communications channels that they use (bgp, ospf, ssh, tftp etc),
If a router has a tcp connection to someplace it shouldn't we'll
probably know about it. If it's announcing a prefix it shouldn't be,
we'll probably know about it, those are the easy ones though.

I am very happy to hear you do these... very useful and will catch quite
a bit.

Post by Joel Jaeggli
There are some things one might consider adding in terms of auditing,
comparing the running image more closely to the one in flash for
example, peroidic checksum of the on onflash image, after downloading to
another host would be another. I'm not sure that I'd trust the later
given the rooted box can I suppose hand you an unmodified version of the
subverted image.

The result from your check can easily be modified, first thing I would
have changed is the checker.

That is a normal thing to do with rootkits (return bogus results). Which
is part of the reason I suggested that method I did. Short of pulling
the flash you're not going to get a fully unbiased view of what's it on
it thusly the audit process has some limitations.

A TCPA style boot process would be a better approach. It's certainly not
a quick fix since it in general can't be retrofited to existing products.

Post by Gadi Evron
Say you did this from a usb stick--I'd just
hide the rootkit in memory.

Post by Joel Jaeggli
In the end if you subvert a router, presumably you're doing it for a
purpose and given what the device does, that purpose is probably
detectable in a well instrumented network.

Subversion may not be the goal. A router is perfect for faking outgoing
traffic. This traffic can contain stolen sniffed or relayed data.

If my device is now taking marching orders from a third party then by
definition it is subverted, regardless of agency or activity.

sub verte - turn from under

[Gadi Evron]

The result from your check can easily be modified, first thing I would have
changed is the checker.

That is a normal thing to do with rootkits (return bogus results). Which is
part of the reason I suggested that method I did. Short of pulling the flash
you're not going to get a fully unbiased view of what's it on it thusly the
audit process has some limitations.
A TCPA style boot process would be a better approach. It's certainly not a
quick fix since it in general can't be retrofited to existing products.

EuSecWest released this interview about the rootkit with its creator,
Sebastian Muniz of Core Security, it also mentions a third party product
to detect some of these issues. Thank whatever diety we like for FX's
work, as obviously Cisco isn't there yet.

http://eusecwest.com/sebastian-muniz-da-ios-rootkit.html

Say you did this from a usb stick--I'd just hide the rootkit in memory.

Post by Joel Jaeggli
In the end if you subvert a router, presumably you're doing it for a
purpose and given what the device does, that purpose is probably
detectable in a well instrumented network.

Subversion may not be the goal. A router is perfect for faking outgoing
traffic. This traffic can contain stolen sniffed or relayed data.

If my device is now taking marching orders from a third party then by
definition it is subverted, regardless of agency or activity.
sub verte - turn from under

[Marc Manthey]

Post by Gadi Evron
http://eusecwest.com/sebastian-muniz-da-ios-rootkit.html

its worth a digg...

<http://digg.com/security/Da_IOS_Rootkit>

regards


--

"Use your imagination not to scare yourself to death
but to inspire yourself to life."

Les enfants teribbles - research and deployment
Marc Manthey - head of research and innovation
Hildeboldplatz 1a D - 50672 Köln - Germany
Tel.:[phone]
Mobil:0049-1577-3329231
jabber :***@kgraff.net
blog : http://www.let.de
ipv6 http://www.ipsix.org
xing : https://www.xing.com/profile/Marc_Manthey

[Mark Smith]

On Sun, 18 May 2008 13:33:53 -0700

Post by Dragos Ruiu

Post by Suresh Ramasubramanian
2. It can be prevented by what's widely regarded as BCP on router
security, and has been covered at *nog, in cisco training material,
etc etc for quite some time now.
I am much less concerned about security conferences discussing this
than about the (highly uninformed) publicity that accompanies these
conferences.

I'm not going to touch the disclosure or not debate... it's been done.
But I will agree to disagree with you about the above two points.
First of all about prevention, I'm not at all sure about this being
covered by existing router security planning / BCP.
I don't believe most operators reflash their routers periodically, nor
check existing images (particularly because the tools for this
integrity verification don't even exist). If I'm wrong about this I
would love to be corrected with pointers to the tools.

<snip>

Cisco publish an MD5 sum (and BSD 'sum' IIRC) for the IOS image just
before you hit the download page, which you can record and then verify
after downloading (although they could make this a *lot* easier to do
by posting it after, or before and after the download).

There is also a /md5 option for the IOS verify command which can be
used to generate an MD5 sum for an IOS image stored on a router.

Regards,
Mark.

[Suresh Ramasubramanian]

So in my opinion the benefits of discussing serious issues at conferences
far outweigh the potential drawbacks of misguided media coverage of them.
What I infer from your post is that you are of the opinion that issues such

Well, there are any number of closed, no media, relevant people only
conferences, or communities like nsp-sec, that come in useful

Report to CSIRT by all means but that doesnt imply "brush it under
the carpet". Getting releases out and fixes (if only router
management bcp like in Joel Jaeggli's post) without various people
spreading FUD about it should certainly be an achievable goal?

srs

[Gadi Evron]

Post by Dragos Ruiu

Post by Suresh Ramasubramanian
On Sat, May 17, 2008 at 12:47 PM, Matthew Moyle-Croft

Post by Matthew Moyle-Croft
If the way of running this isn't out in the wild and it's actually
dangerous then a pox on anyone who releases it, especially to gain
publicity at the expensive of network operators sleep and well being.
May you never find a reliable route ever again.

This needs fixing. It doesnt need publicity at security conferences
till after cisco gets presented this stuff first and asked to release
an emergency patch.

Bullshit.
There is nothing to patch.
It needs to be presented at conferences, exactly because people will
play ostrich and stick their heads in the sand and pretend it can't
happen to them, and do nothing about it until someone shows them, "yes
it can happen" and here is how....
Which is exactly why we've accepted this talk. We've all known this is
a possibility for years, but I haven't seen significant motion forward
on this until we announced this talk. So in a fashion, this has
already helped make people more realistic about their infrastructure
devices. And the discussions, and idea interchange that will happen
between the smart folks at the conference will undoubtedly usher forth
other related issues and creative solutions. Problems don't get fixed
until you talk about them.

Dragus, while I hold full disclosure very close and it is dear to my
heart, I admit the fact that it can be harmful. Let me link that to
network operations.

People forget history. A few years back I had a chat with Aleph1 on the
first days of bugtraq. He reminded me how things are not always black and
white.

Full disclosure, while preferable in my ideology, is not the best solution
for all. One of the reasons bugtraq was created is because vendors did not
care about security, not to mention have a capability to handle security
issues, or avoid them to begin with.

Full disclosure made a lot of progress for us, and while still a useful
tool, with some vendors it has become far more useful to report to them
and let them provide with a solution first.

In the case of routers which are used for infrastructure as well as
critical infrastructure, it is my strong belief that full disclosure is,
at least at face value, a bad idea.

I'd like to think Cisco, which has shown capability in the past, is as
responsible as it should be on these issues. Experience tells me they have
a ways to go yet even if they do have good processes in place with good
people to employ them.

I'd also like to think tier-1 and tier-2 providers get patches first
before such releases. This used to somewhat be the case, last I checked it
no longer is -- for legitimate concerns by Cisco. has this changed?

So, if we don't patch the infrastructure up first, and clients don't know
of problems until they are public "for their own security" (an argument
that holds water only so much) perhaps it is the time for full disclosure
to be considered a viable alternative.

All that aside, this is a rootkit, not a vulnerability. There is no
inherent vulnerability to patch (unless it is very local). There is the
vulnerability of operators who don't so far even consider trojan horses
as a threat, and the fact tools don't exist for them to do something once
they do.

Gadi.

Post by Dragos Ruiu
cheers,
--dr
--
World Security Pros. Cutting Edge Training, Tools, and Techniques
London, U.K. May 21/22 - 2008 http://cansecwest.com
pgpkey http://dragos.com/ kyxpgp
_______________________________________________
NANOG mailing list
http://mailman.nanog.org/mailman/listinfo/nanog

[Gadi Evron]

Post by Paul Wall
What if some good comes from this "root kit"?
For instance, what if it lets us fix things like DOM on non-Cisco
XENPAKs and SFPs? Or lets us un-cripple our 6500 chassis to run the
code we want?
Of course, given the messenger, I'm sure it's just hype to help
bolster Gadi's security practice, and will prove to be no big deal.

A signed issue is now 25 bucks FOR YOU, Mister.

Post by Paul Wall
Paul
_______________________________________________
NANOG mailing list
http://mailman.nanog.org/mailman/listinfo/nanog

[Tony Varriale]

IIRC, the toolkit(s) can only be installed once having priv 15 on the
device.

If this is the case, the practicality of this is...well...not that
significant.

I do think the significance is that we are getting closer and closer to
treating infrastructure devices as end stations with respect to
susceptibility.

Looking forward to seeing all the details.

Gadi, have fun :)

tv
----- Original Message -----
From: "Gadi Evron" <***@linuxbox.org>
To: <***@merit.edu>
Sent: Friday, May 16, 2008 8:06 PM
Subject: [NANOG] IOS rootkits

Post by Gadi Evron
At the upcoming EusecWest Sebastian Muniz will apparently unveil an IOS
rootkit. skip below for the news item itself.
We've had discussions on this before, here and elsewhere. I've been
heavily attacked on the subject of considering router security as an issue
when compared to routing security.
I have a lot to say about this, looking into this threat for a
few years now and having engaged different organizations within Cisco on
the subject in the past. Due to what I refer to as an "NDA of
honour" I will just relay the following until it is "officially" public,
1. Current defense startegies possible with Cisco gear
2. Third party defense strategies (yes, they now exist)
2. Cisco response (no names or exact quotes will likely be given)
3. A bet on when such a rootkit would be public, and who won it
(participants are.. "relevant people").
http://www.networkworld.com/news/2008/051408-hacker-writes-rootkit-for-ciscos.html
"A security researcher has developed malicious rootkit software for
Cisco's routers, a development that has placed increasing scrutiny on the
routers that carry the majority of the Internet's traffic.
Sebastian Muniz, a researcher with Core Security Technologies, developed
the software, which he will unveil on May 22 at the EuSecWest conference
in London. "
Gadi Evron.
_______________________________________________
NANOG mailing list
http://mailman.nanog.org/mailman/listinfo/nanog

[Tuc at T-B-O-H.NET]

Post by Matthew Moyle-Croft

I'd love to know what magical mystical protection your routers have that will
enable them to avoid the same fate as every other device and operating system
has. There's only one thing up there that doesn't have known rootkits
in the wild. Yet.

The question isn't IF routers have security vunerabilities, but whether
Gadi has an example he can demonstrate now of installing a root kit on
an IOS router NOW or not.

Rootkit for 2500, 3000 and 4000...... Load this onto your router and you'll
have root and much more.

http://tinyurl.com/29duah

Tuc/TBOH