September 15, 2014

Cisco IOS Hints and Tricks

SIGS & Carrier’s Lunch DC Day: An Event Definitely worth Visiting

I spent last Tuesday in Bern attending the SIGS DC Day Event, and came back home extremely pleasantly surprised. The conference was nice and cozy, giving everyone plenty of opportunities to chat about data center technical challenges (thanks for all the wonderful conversations we had – you know who you are!).

Having the opportunity to meet fellow networking engineers and compare notes is great, but it’s even better to combine that with new knowledge, and that’s where the event really excelled.

Read more ...

by Ivan Pepelnjak ( at September 15, 2014 08:12 AM

XKCD Comics

September 12, 2014

Honest Networker
Networking Now (Juniper Blog)

Recap of VMworld 2014 USA - Juniper Style

This was the first year that I got to attend VMworld as a member of the Juniper family ( this was my fourth VMworld ). It was a great experience, we had our first lab in the Hands-on Lab which I personally think was a success and of course we had a booth. We received a lot of complements on the documentation for the lab and how it explained all the facets of the product. I had people fighting ( well not literally ) for the long sleeve shirts that we distributed to everyone who took the lab ( check it out below )



juniper lab shirt.JPG


It gave us a lot of visibility into our virtual security solutions and how they play in your VMware environment. The great thing, the fun isn't over…

A) VMware will make the labs available online in approximately 2 - 3 weeks so you can take them from the comfort of whatever you find comfortable, whenever you want to take it. The link is .
In the meantime, if you are interested in reading the lab that I wrote, it is available in PDF format and html format
2) We will be at VMworld 2014 Europe in Barcelona. Sadly we won't have shirts but the lab will be there and I promise to give you a hug if you take the lab. Hugs are better anyway.
The lab hours this year are : 
Monday / October 13 : 8:00 - 18:00
Tuesday / October 14 : 10:30 - 18:30
Wednesday / October 15 : 8:00 - 18:00
Thursday / October 16 : 8:00 - 18:00
I look forward to seeing you there!

by banksek at September 12, 2014 06:46 PM

Packet Pushers Blog/Podcast

Network Break 16

This week, EVO:RAIL & Converged thingies, Cisco's multiple SDN strategies, Don't be a precious snowflake and Congress open source project.

by Packet Pushers Podcast at September 12, 2014 05:22 PM

Renesys Blog

Sprint, Windstream: Latest ISPs to hijack foreign networks

Last year my colleague Jim Cowie broke a story about routing hijacks that resulted in Internet traffic being redirected through Iceland and Belarus. Unfortunately, little has changed since then and the phenomenon of BGP route hijacking continues unabated and on an almost daily basis.

In the past three days, the situation has gone from bad to downright strange as we have observed a flurry of this activity. Now, for the first time, we’re seeing major US carriers, Sprint and Windstream, involved in hijacking, along with the return of an operation out of Poland targeting Brazilian networks. We see router misconfigurations regularly in BGP data – could these incidents also be explained by simple command-line typos?

Route hijacking continues

In May my colleague, Earl Zmijewski gave a presentation about routing hijacks at the LINX 85 meeting, describing a comprehensive system that can be used to identify suspicious hijacks on a global basis and without any prior knowledge about the networks involved. While we now detect suspicious routing events on an almost daily basis, in the last couple of days we have witnessed a flurry of hijacks that really make you scratch your head.

To mention a few recent events, last month we tweeted about the Korea Meteorological Administration mistakenly hijacking a prefix from the US National Climatic Data Center, which hosts websites like and, thereby making these sites and many others unreachable to most of the Internet. And then we saw a company in South America trying to hijack back its address space from another entity that was squatting on it.

US carriers hijacking foreign networks

From 13:56 UTC on Tuesday (9-September) to 15:56 UTC on Wednesday (10-September), US wireless carrier Sprint started hijacking a prefix ( from Telesmart, an ISP in Macedonia. About 65% of our peers believed that Sprint was the origin of this prefix and so redirected Macedonian traffic for it to Sprint.

What was interesting was that once traffic arrived at Sprint, it continued onto Cogent and finally onto its intended destination at Telesmart in Skopje. Was this an accidental man-in-the-middle or something else? That is, probes from most of our servers around the world suddenly started going through Sprint to reach Internet resources hosted at Telesmart, such as news and municipal websites. Below is one of the more striking examples of the impact on traffic paths. Before the hijack, traffic from Sofia would reach Skopje in about 6ms, as these cites are fairly close.

trace from Sofia, Bulgaria to Telesmart, Skopje, Macedonia at 03:15 Sep 02, 2014
1 *
2    (Neterra, Sofia, Bulgaria)      0.339ms
3 (NTT Europe, Bulgaria Facility) 9.446ms
4 (NTT Europe, Bulgaria Facility) 6.883ms
5  (Telesmart, Skopje, Macedonia)  6.216ms

During the hijack, traffic took the scenic route through Frankfurt, Paris, Frankfurt (again), Munich, Vienna, Sofia (again), then on to Skopje and consequently took about 10 times longer to reach its intended destination.

trace from Sofia, Bulgaria to Telesmart, Skopje, Macedonia at 21:31 Sep 09, 2014
1 * 
2        (Neterra, Sofia, BG)                        10.682ms
3                   0.512ms
4                  34.056ms
5     (GTT, Frankfurt, DE)                        53.026ms
6          49.62ms
7         50.087ms
8      46.84ms
9   48.43ms
10       48.859ms
11       52.757ms
12       59.008ms
13       77.427ms
14        80.141ms
15     (Telesmart, Skopje, Macedonia)              48.916ms

For 26 hours, traffic to this Telesmart network in Skopje entered the Sprint network at various global locations (from Frankfurt to San Francisco) and was then handed off to Cogent in Paris en-route to Skopje. As we often are in these cases, we’re left wondering, what the heck was that? Sprint is a big network, but AS1239 doesn’t originate many prefixes outside the US and doesn’t originate anything starting with 95.x.x.x or 59.x.x.x or anything else that is typographically close to this prefix. Thus, this doesn’t look like an obvious and easily explained typo, something that we see all too often. Not only did traffic to this prefix get much slower, such diversion subjected a small amount of regional traffic to the risk of foreign interception — something on the minds of nearly everybody in the post-Snowden era.

But Sprint wasn’t the only US carrier to begin an unusual hijack on 9 September: US provider Windstream (AS7029) began announcing (SaudiNet), which is normally announced by Saudi Arabian incumbent, Saudi Telecom. Unlike the previous Sprint example, traceroutes to this prefix along the Windstream route died within Windstream, effectively knocking this network off the Internet for anyone accepting the bogus route. Then on Wednesday, Windstream announced a handful of strange routes for about 10 hours including one from Gaza, one from Iceland, and three from China — all more-specifics of existing routes, ensuring their global propagation and acceptance. These are listed below.

Hijack route    Organization label and location    Victim Route   Victim AS    Hadara Gaza BSA - 3ed subnet, PS    Hadara Technologies, AS15975 Advania hf., Reykjavik, IS Thor Data Center, AS50613 CHINANET Guangxi province network   China Telecom, AS4134 CHINANET Jiangxi province network  China Telecom, AS4134  CHINANET Huzhou node network  China Telecom, AS4134

Of the four examples above, only Advania of Iceland (AS30818) tried to get their traffic back by announcing the same more-specific as Windstream. Advania’s attempt to reclaim their network started at 15:06 UTC, at which point Windstream pulled their errant announcement. In all other cases, the hijacks continued until 15:49 UTC. Unlike the previous Sprint example, any traffic headed to computers in these IP address ranges would have been directed to Windstream during this time and then dropped, effectively knocking these networks off the Internet.

There is a potentially innocent explanation to this example. Perhaps, these address ranges were ones that Windstream deemed to be sources of bad traffic and so was “blackholing” them internally, a relatively common practice. In this scenario, we could have simply witnessed Windstream inadvertently leaking internal routes to the global Internet for 10 hours.

Polish hijacking of Brazilian IP address space

In another recent hijacking incident, starting at 03:16 UTC on 6 August, two different ISPs in Poland each started hijacking a prefix each from two different ISPs in Brazil. These hijacked routes were only circulated to a small number of other ISPs in Eastern Europe.

The two prefixes, and, were originated by INEA S.A. (AS33868) and INOTEL S.A. (AS44514), respectively. Once originated, they were both routed through the same upstream provider, another INEA S.A. ASN (AS13110) before going to Hurricane Electric (AS6939) and then onto a handful of providers in Eastern Europe. Routes took the following forms.

Normal route Hijacked route     … 3549 52770     … 6939 13110 33868 … 4230 28573 … 6939 13110 44514

Traceroutes from Eastern Europe show traffic paths redirected into INEA S.A. before continuing onto their proper destination.


trace from Warsaw, Poland to Revo Telecomunicações at 00:35 Aug 05, 2014
1 *
2   (GTS Poland Sp. z o.o., Warsaw, PL)             0.952
3     (GTS Czech s.r.o.,Prague, CZ)                   17.065
4 * 
5             17.222
6           25.666
7    (Revo Telecomunicações, Porto Alegre, BR)       248.358
8     (Revo Telecomunicações, Guarulhos, BR)          251.24
9      (Gm Telecom Ltda, Palmitinho, BR)               251.609

During (note the new INEA S.A. hops through Poznań, Poland):

trace from Warsaw, Poland to Revo Telecomunicações at 17:10 Aug 06, 2014
1  *
2  (GTS Poland Sp. z o.o., Warsaw)                 1.119ms
3   (GTS Poland Sp. z o.o., Warsaw)                 0.397ms
4                              5.407ms
5, Poznań, PL) 5.407ms
6, Poznań, PL)     5.608ms
7   (Level 3, Frankfurt, DE)                        27.76ms
8               27.74ms
9             27.65ms
10             27.743ms
11           27.854ms
12   (Revo Telecomunicações, Porto Alegre, BR)       249.385ms
13   (Revo Telecomunicações, Porto Alegre, BR)       248.138ms
14 *

At this year’s Defcon conference, Luca ‘kaeso’ Bruno and Mariano ‘emdel’ Graziano gave a talk about the vulnerabilities of some ISPs’ public looking glass utilities that would allow an attacker to remotely modify router configurations. INEA S.A. (AS33868) was on their list of vulnerable ISPs.

The hijacks described above ceased at the end of August. However, on Wednesday we saw some new suspicious activity from AS13110. At 00:14 UTC on 10 September, AS13110 hijacked a prefix from the US Department of Defense (, normally announced by AS27064). At 00:31 UTC the hijack had stopped, but then a few minutes later an ASN downstream of AS13110 began hijacking a new Brazilian prefix (, TecleNet Solucoes Tecnologicas) at 00:35 UTC. Was the DoD hijack an initial test before the real target was selected?

The above graph shows the percent of our peers accepting the bogus origin (AS57807) compared to the legitimate Brazilian origin (AS57807) over a day and a half of this activity. Then this hijack disappeared yesterday morning at 14:13 UTC while I was writing up this blog post.

And there is IP squatting to worry about

On 31 August, responding to an inquiry on the NANOG mail list, I wrote a description of a rather unique IP squatting operation out of Russia that briefly announces (mostly unused, sometimes not) IP address space assigned to others and originates these via two unused AS numbers. To be sure, IP squatting is not a new phenomenon. Nick Feamster and Anirudh Ramachandran from Georgia Tech presented on the use of IP squatting for spam operations at NANOG 36 in 2006.

Starting in late June, we have observed dozens of prefixes announced along AS paths of the following forms (origins are the rightmost ASN) each for no more than a couple hours at a time.

    … 39792 57756 {197329 or 43239}    prefix

    … 44050 57756 {197329 or 43239}    prefix

Since 30 August, these routes have taken the following form:

    … 44050 197598 {49121 197794 or 201910}    prefix

Below is a graph of the bogus route originations of this IP squatting operation on a recent day. Prefixes and origin ASNs are only up for a few hours at a time. AS201910 is the newest formerly unused ASN being used as a bogus origin. On Wednesday of this week, it announced three prefixes of unused IP address space:  Bundesministerium fuer Land- und Forstwirtschaft  AT  DuoDecad IT Services Luxembourg S.a r.l.  Luxembourg  Amadeus Data Processing GmbH  Bayern DE


While this IP squatting operation is rather unique in its methodology, there are far more examples of IP squatting that don’t involve brief prefix announcements from odd ASNs. Perhaps, an amusing on-going example is Bitcanal (AS197426) of Portugal, which originates unused IP address space from a number of locations ranging from Syria (, AYA Internet Service Provider) to Texas (, Office of the Attorney General, State of Texas). (Would someone please alert Texas Governor Rick Perry about that last one?)

The above graphs illustrate that the Syrian prefix has been globally routed out of Portugal since 25 August, while the new home of the Texas prefix has been seen by about 40% of our peers since 19 June.


So what is the story with all of this?

Well, for one, sloppiness abounds. Humans are at the controls and are always capable of mucking things up. For example, British Telecom Latin America (BT Latam, AS7908) has been globally announcing since April 2012. The route is technically a hijack of, which is announced by China Telecom but looks like a leaked internal route. Or take Beltelecom of Belarus, who has been globally announcing 13 prefixes within RFC6598 address space (such as and intended for internal carrier grade NAT operations since January of this year.

It is interesting to note that the Bitcoin hijack from earlier this year originated its bogus routes using two formerly unused ASNs with a common upstream AS path. The Russian IP squatting operation also typically uses two unused ASNs for its origination. Finally, the Polish hijacking of the Brazilian routes mentioned above also employed two different ASNs, although not unused this time. Perhaps it is a coincidence, or perhaps it is a known tactic: when hijacking multiple routes, it is better to spread out one’s bogus routes across multiple origins to lower the profile of any single misbehaving ASN.

Regardless of the cause of each of these incidents, the problem is a very real and growing one. Perhaps documenting these incidents will promote a greater understanding of the extent and nature of the problems around the trust-based Internet routing system in global use today.

The post Sprint, Windstream: Latest ISPs to hijack foreign networks appeared first on Renesys.

by Doug Madory at September 12, 2014 03:15 PM

Aaron's Worthless Words

FEMA and Your Business Continuity Plan

I passed the ROUTE exam a few days/weeks/months/something ago and decided to pursue certifications of another sort for a while. The wife and I are trying our best to help the community through our ham radio training, so I decided to go down that path a bit further. One thing I was interested in doing is to do EmComm during declared emergencies. That meant I had to take two FEMA courses online to be allowed in the EOC. I thought they would be terribly boring, but I found them to be quite familiar.

The first course was on the Incident Command System (ICS). The main idea is that, in the event of an emergency of any kind or size, an Incident Commander (IC) is assigned to be responsible for the recovery effort.  This mean analysis of the incident, generating an action plan (a very key component), and execution of said plan. If the IC can complete the action plan by himself, then off he goes. If he or she needs some additional resources like people or equipment, then he or she is empowered to draft that help from any entity that’s involved.

Another one of the big points of ICS is that a disaster response is distanced from day-to-day work.  That is, when you’re working in a response capacity, your day-job boss isn’t your boss (unless he happens to be your manager in the ICS).  ICS also gets rids of the whole dotted-line organizational structure.  This is called unity of command and supports a single objective work model.  No more being yelled at by 7 managers for not have a cover page on your TPS report.

Does this not sound like a great system reacting to a DR event? The IC in this case could be a product owner and that person would evaluate the situation, create a detailed action plan, draft personnel as needed, etc. It’s the IC’s job to figure out what needs to be done and what resources are need. If he or she calls you, you report to him or her today. Your day-to-day boss can’t pull you off to work on other things; this is a response situation that the business has blessed and is deemed a higher priority than your daily work. Very powerful stuff.

The second course I took was and introduction to the National Incident Management System (NIMS). The whole point of NIMS is to create a system that supports intra- and inter-departmental communication. At a very high level, this means using plain language and putting things in writing beforehand. We’ve all been told to “talk to Bill about that” and Bill tells you to call Mary.  NIMS dictates that we should have a document that says exactly what each is responsible for during a response. It tells us what Bill and Mary are supposed to be doing so we don’t have to guess. It’s better to know what everyone is doing than to guess or, even worse, assume what everyone is doing.

What if your product owner says we need to roll to the alternate data center? What does that really mean? Will most of your time be spent flopping around like a fish because you don’t know what’s going on? Do you have a list of outside entities (like your ISP or VAR) whose help will be needed during a DR incident? How about a definition of success? Do you have a plan to roll back when your recover from the disaster? Do you have a list of of tasks to get operations rolling at the alternate data center? Do you need to get your marketing department to send out messages to customers or update your website? Do you even have the criteria for declaring a disaster? Can I ask any more questions? NIMS tells us that we need to answer all those questions (and more) BEFORE the disaster happens. Let me say that again — BEFORE the disaster happens.

I recommend you take the ICS and NIMS training from FEMA. It’s obviously going to be focused on physical disasters, but the ideas apply pretty broadly. Each class takes 3 or 4 hours to finish and is broken up into sections, so you don’t have to do it all at once. There’s an exam at the end, but that’s obviously not a requirement (unless you need the cert like I did). The training is absolutely free and doesn’t require you to log in or anything.  And get your ham radio license.  :)

Send any new HTs questions my way.

by Aaron Conaway at September 12, 2014 02:04 PM

Cisco IOS Hints and Tricks

Tech Talks: The Essence of MPLS

Seamus Gilchrist sent me a fantastic list of MPLS- and MPLS-TE-related questions. Instead of starting an email exchange we agreed on something that should benefit a wider community: a lengthy whiteboard session discussing the basics of MPLS, MPLS-TE, load balancing and QoS in MPLS networks…

The first part of our conversation is already online: The Essence of MPLS.

by Ivan Pepelnjak ( at September 12, 2014 08:59 AM

XKCD Comics

September 11, 2014

Honest Networker
My Etherealmind
Honest Networker
Cisco IOS Hints and Tricks

Open-Source Hybrid Cloud Reference Architecture on Software Gone Wild

A while ago Rick Parker told me about his amazing project: he started a meetup group that will build a reference private/hybrid cloud heavily relying on virtualized network services, and publish all documentation related to their effort, from high-level architecture to device and software configurations, and wiring plans.

In Episode 8 of Software Gone Wild Rick told us more about his project, and we simply couldn’t avoid a long list of topics including:

Read more ...

by Ivan Pepelnjak ( at September 11, 2014 08:19 AM

September 10, 2014

Cisco IOS Hints and Tricks

IPv6 Neighbor Discovery (ND) and Multicast Listener Discovery (MLD) Challenges

A few days ago Garrett Wollman published his exasperating experience running IPv6 on large L2 subnets with Juniper Ex4200 switches, concluding that “… much in IPv6 design and implementation has been botched by protocol designers and vendors …” (some of us would forcefully agree) making IPv6 “…simply unsafe to run on a production network…

The resulting debate on Hacker News is quite interesting (and Andrew Yourtchenko is trying hard to keep it close to facts) and definitely worth reading… but is ND/MLD really as broken as some people claim it is?

Read more ...

by Ivan Pepelnjak ( at September 10, 2014 08:13 PM

Honest Networker
Networking Now (Juniper Blog)

September 2014 Microsoft Patch Tuesday Summary

Welcome to the September edition of Microsoft Patch Tuesday Summary. In this edition there are 4 updates; one is marked "Critical" and three are rated "Important". A total of 42 vulnerabilities were fixed over 4 bulletins this month. One of the Critical update MS14-052 is an all version Internet Explorer (IE 6 to 11) patch. This single update resolves 37 CVE's (Common Vulnerability and Exposure) including the publicly disclosed CVE-2013-7331


Here is a list of Security bulletins which were rolled out in today's Patch Tuesday release.

by prashantk at September 10, 2014 04:58 AM

XKCD Comics

September 09, 2014

Peter's CCIE Musings and Rants

vSwitch Updates

vSwitch Updates:

vMotion support for networks with less than 100ms RTT, vMotion between "Data Centers" (not literal data centers, just the VMWare construct called "Data center") and "officially" supporting vMotion over routed networks (which has been happening for a while but was never officially supported.)

by peter_revill ( at September 09, 2014 09:34 PM

Packet Pushers Blog/Podcast

Show 204 – Reducing Your Attack Surface with Avaya Stealth Networks – Sponsored

"The problem with ‘covering your tracks’ in network security is that your ‘covering’ becomes more conspicuous than your ‘tracks’," says Ed Koehler, Distinguished Engineer for Avaya’s Networking Division. Ed joins Greg Ferro and Ethan Banks for a ninja nerd-fest outlining a set of technologies that not only offer some innovative ways to set up your security architecture, but also simplifies the way that you do it. The Packet Pushers team specifically discusses with Ed how to segment for greater anomaly identification, how to streamline your firewall strategy, and how an ISID-VLAN-VRF combination can create truly independent stealth networks. Ed also outlines how customers are using this technology in healthcare, government, retail, and transportation to comply with industry regulations such as PCI and HIPAA. This show gets fairly nerdy, digging into L2 and L3 segmentation and exactly how that gets done with Shortest Path Bridging in the Avaya implementation. Ed is an unapologetic details guy, and the conversation gets into the weeds (in the best possible way) regarding just how frames traverse an SPB fabric in the context of "stealth." Links 802.1aq (Wikipedia) Ed's blog Ed's YouTube channel Avaya - Stealth Networks Overview (PPT - right-click and "Save Link As...") Avaya Technology Forum - Stealth PCI Networking Presentation (PPT - right-click and "Save Link As...")

by Packet Pushers Podcast at September 09, 2014 03:52 PM

Networking Now (Juniper Blog)

Security for the Cloud Data Center with Juniper Spotlight Secure Threat Intelligence Platform

In an earlier blog, I posed a few questions on security challenges that some Cloud Builders are facing today. Here, I offer some ideas for you to consider.

by skathuria at September 09, 2014 12:00 PM

Cisco IOS Hints and Tricks

vMotion Enhancements in vSphere 6

VMware announced several vMotion enhancements in vSphere 6, ranging from “finally” to “interesting”.

vMotion across virtual switches. Finally. The tricks you had to use previous were absolutely bizarre.

Read more ...

by Ivan Pepelnjak ( at September 09, 2014 08:40 AM

September 08, 2014

Packet Pushers Blog/Podcast

Network Admin in Cary NC

I’m helping a company (as a favor) that’s looking for a network administrator in the Cary, NC area. The company is moving from another area, and hence rebuilding their office and backend systems. They rely heavily on their IT “stuff,” as they’re essentially in the information business. Please send me an email if you’re interested […]

Author information

Russ White

Russ White
Principle Engineer at Ericsson

Russ White is a Network Architect who's scribbled a basket of books, penned a plethora of patents, written a raft of RFCs, taught a trencher of classes, and done a lot of other stuff you either already know about, or don't really care about. You want numbers and letters? Okay: CCIE 2635, CCDE 2007:001, CCAr, BSIT, MSIT (Network Design & Architecture, Capella University), MACM (Biblical Literature, Shepherds Theological Seminary). Russ is a Principal Engineer in the IPOS Team at Ericsson, where he works on lots of different stuff, serves on the Routing Area Directorate at the IETF, and is a cochair of the Internet Society Advisory Council. Russ will be speaking in November at the Ericsson Technology Day. he recently published The Art of Network Architecture, is currently working on a new book in the area of network complexity with Addison Wesley, a book on innovation from within a Christian worldview, and he blogs at on network engineering.

The post Network Admin in Cary NC appeared first on Packet Pushers Podcast and was written by Russ White.

by Russ White at September 08, 2014 07:23 PM

RFCs You Should Know: 6250

Most RFCs are deeply technical — and they follow the “Yaakov rule” for intelligibility (if you didn’t write it, or you didn’t sit with one of the authors in a bar someplace to talk about it, you can’t understand it), there are a few here and there every network engineer should know. RFC 6250 is […]

Author information

Russ White

Russ White
Principle Engineer at Ericsson

Russ White is a Network Architect who's scribbled a basket of books, penned a plethora of patents, written a raft of RFCs, taught a trencher of classes, and done a lot of other stuff you either already know about, or don't really care about. You want numbers and letters? Okay: CCIE 2635, CCDE 2007:001, CCAr, BSIT, MSIT (Network Design & Architecture, Capella University), MACM (Biblical Literature, Shepherds Theological Seminary). Russ is a Principal Engineer in the IPOS Team at Ericsson, where he works on lots of different stuff, serves on the Routing Area Directorate at the IETF, and is a cochair of the Internet Society Advisory Council. Russ will be speaking in November at the Ericsson Technology Day. he recently published The Art of Network Architecture, is currently working on a new book in the area of network complexity with Addison Wesley, a book on innovation from within a Christian worldview, and he blogs at on network engineering.

The post RFCs You Should Know: 6250 appeared first on Packet Pushers Podcast and was written by Russ White.

by Russ White at September 08, 2014 07:16 PM

Cisco IOS Hints and Tricks

Controller Cluster Is a Single Failure Domain

Some OpenFlow-focused startups are desperately trying to tell you how redundant their architecture is. Unfortunately all the whitepapers (and the prancing unicorns) cannot change a simple fact: an SDN controller (OpenFlow-based or otherwise) is in some aspects a single failure domain.

Read more ...

by Ivan Pepelnjak ( at September 08, 2014 07:19 PM

Packet Pushers Blog/Podcast

Common Network Design Concepts Part-2

In the first article of this series, reliability and resiliency has been explained. Every component and every device can and eventually will fail, thus system should be resilient enough to re converge/recover to a previous state. Resiliency can be achieved with redundancy. But how much redundancy is best for the resiliency is another consideration to […]

Author information

Orhan Ergun

Orhan Ergun, CCIE, CCDE, is a network architect mostly focused on service providers, data centers, virtualization and security.

He has more than 10 years in IT, and has worked on many network design and deployment projects.

In addition, Orhan is a:

Blogger at Network Computing.
Blogger and podcaster at Packet Pushers.
Manager of Google CCDE Group.
On Twitter @OrhanErgunCCDE

The post Common Network Design Concepts Part-2 appeared first on Packet Pushers Podcast and was written by Orhan Ergun.

by Orhan Ergun at September 08, 2014 11:00 AM

XKCD Comics

September 07, 2014

The Networking Nerd

An Educational SDN Use Case

During the VMUnderground Networking Panel, we had a great discussion about software defined networking (SDN) among other topics. Seems that SDN is a big unknown for many out there. One of the reasons for this is the lack of specific applications of the technology. OSPF and SQL are things that solve problems. Can the same be said of SDN? One specific question regarded how to use SDN in small-to-medium enterprise shops. I fired off an answer from my own experience:

Since then, I’ve had a few people using my example with regards to a great use case for SDN. I decided that I needed to develop it a bit more now that I’ve had time to think about it.

Schools are a great example of the kinds of “do more with less” organizations that are becoming more common. They have enterprise-class networks and needs and live off budgets that wouldn’t buy janitorial supplies. In fact, if it weren’t for E-Rate, most schools would have technology from the Stone Age. But all this new tech doesn’t help if you can’t find a way for it to be used to the fullest for the purposes of educating students.

In my example, I talked about the shift from written standardized testing to online assessments. Oklahoma and Indiana are leading the way in getting rid of Scantrons and #2 pencils in favor of keyboards and monitors. The process works well for the most part with proper planning. My old job saw lots of scrambling to prep laptops, tablets, and lab machines for the rigors of running the test. But no amount of pre-config could prepare for the day when it was time to go live. On those days, the network was squarely in the sights of the administration.

I’ve seen emails go around banning non-testing students from the computers. I’ve seen hard-coded DNS entries on testing machines while the rest of the school had DNS taken offline to keep them from surfing the web. Redundant circuits. QoS policies that would make voice engineers cry. All in the hope of keeping the online test bandwidth free to get things completed in the testing window. All the while, I was thinking to myself, “There has got to be an easier way to do this…”

Redefining with Software

Enter SDN. The original use case for SDN at Stanford was network slicing. The Next-Gen Network Team wanted to use the spare capacity of the network for testing without crashing the whole system. Being able to reconfigure the network on the fly is a huge leap forward. Pushing policy into devices without CLI cuts down on the resume-generating events (RGE) in production equipment. So how can we apply these network slicing principles to my example?

On the day of the test, have the configuration system push down a new policy that gives the testing machines a guaranteed amount of bandwidth. This reservation will ensure each machine is able to get what it needs without being starved out. With SDN, we can set this policy on a per-IP basis to ensure it is enforced. This slice will exist separate from the production network to ensure that no one starting a huge FTP transfer or video upload will disrupt testing. By leaving the remaining bandwidth intact for the rest of the school’s production network administrators can ensure that the rest of the student body isn’t impacted during testing. With the move toward flipped classrooms and online curriculum augmentation, having available bandwidth is crucial.

Could this be done via non-SDN means? Sure. Granted, you’ll have to plan the QoS policy ahead of time and find a way to classify your end-user workstations properly. You’ll also have to tune things to make sure no one is dominating the test machine pool. And you have to get it right on every switch you program. And remove it when you’re done. Unless you missed a student or a window, in which case you’ll need to reprogram everything again. SDN certainly makes this process much easier and less disruptive.

Tom’s Take

SDN isn’t a product. You can’t order a part number for SDN-001 and get a box labeled SDN. Instead, it’s a process. You apply SDN to existing environment and extend the capabilities through new processes. Those processes need use cases. Use cases drive business cases. Business cases provide buy in from the stakeholders. That’s why discussing cases like the one above are so important. When you can find a use for SDN, you can get people to accept it. And that’s half the battle.

by Tom Hollingsworth at September 07, 2014 11:44 PM

Cisco IOS Hints and Tricks

Is Anyone Using DMVPN-over-IPv6?

One of my readers sent me an interesting challenge: they’re deploying a new DMVPN WAN, and as they cannot expect all locations to have native (non-NAT) IPv4 access, they plan to build the new DMVPN over IPv6. He was wondering whether it would work.

Apart from “you’re definitely going in the right direction” all I could tell him was “looking at the documentation I couldn’t see why it wouldn’t work” Has anyone deployed DMVPN over IPv6 in a production network? Any hiccups? Please share your experience in the comments. Thank you!

by Ivan Pepelnjak ( at September 07, 2014 07:10 PM

See You in Bern on September 9th

TL;DR: I'll be in Bern on September 9th. If you'd like to drop by and discuss network design or automation challenges, read on…

Read more ...

by Ivan Pepelnjak ( at September 07, 2014 07:09 PM

Snabb Switch Deep Dive on Software Gone Wild

The pilot episode of Software Gone Wild podcast featuring Snabb Switch created plenty of additional queries (and thousands of downloads) – it was obviously time for another deep dive episode discussing the intricate innards of this interesting virtual switch.

During the deep dive Luke Gorrie, the mastermind behind the Snabb Switch, answered a long list of questions, including:

Read more ...

by Ivan Pepelnjak ( at September 07, 2014 05:15 PM

Pmacct: the Traffic Analysis Tool with Unpronounceable Name

SDN evangelists talking about centralized traffic engineering, flow steering or bandwidth calendaring sometimes tend to gloss over the first rule of successful traffic engineering: Know Thy Traffic.

In a world ruled by OpenFlow you’d expect the OpenFlow controller to know all the traffic; in more traditional networks we use technologies like NetFlow, sFlow or IPFIX to report the traffic statistics – but regardless of the underlying mechanism, you need a tool that will collect the statistics, aggregate them in a way that makes them usable to the network operators, report them, and potentially act on the deviations.

Read more ...

by Ivan Pepelnjak ( at September 07, 2014 11:03 AM

September 05, 2014

Cisco IOS Hints and Tricks

Scalability Enhancements in Cisco Nexus 1000V

The latest release of Cisco Nexus 1000V for vSphere can handle twice as many vSphere hosts as the previous one (250 instead of 128). Cisco probably did a lot of code polishing to improve Nexus 1000V scalability, but I’m positive most of the improvement comes from interesting architectural changes.

Read more ...

by Ivan Pepelnjak ( at September 05, 2014 08:01 AM

XKCD Comics

September 04, 2014

Networking Now (Juniper Blog)

Firefly for Software-defined data center (SDDC)

Background: In today’s world, data center virtualization has enabled the agility and elasticity which accelerates the delivery of infrastructure-networking, storage, and compute. However, the penetrable nature of the cloud also exposes the network to serious security issues. As services become more mobile and cloud focused, security services need to adopt to this dynamic environment to deliver security for communications within virtualized data center.


Security Issues In SDDC: SDDC (Software-defined data center) is a data center where all the infrastructure is virtualized and delivered as a service. In SDDC, the whole data center is controlled by a single layer of virtualization and all of the resources of data center are abstracted and automated. The processing capacity of each host is increased, processing loads are shared and moved among all hosts which significantly increases the amount of traffic. The traditional physical switching and routing devices create a tangled route that slows down the traffic and may not detect all the security issues within virtual infrastructure. So it is a wise idea to consider virtual appliance which will reduce latency and optimize performance.


Some of the key security challenges in SDDC are the lack of visibility into East-West (virtual machine to virtual machine) traffic, lack of dynamic security (Security not keeping pace with the rate of application provisioning). Other network security issues include undetected and uncontained malware outbreaks or insider attacks in the virtual environment and inability to enforce policies that isolate VMs, prevent VM sprawl.


Firefly for SDDC: Firefly addresses many of these security threats by providing next generation security features such as ant-virus/anti-spam, IDP, web filtering and intrusion prevention system which all are included in Unified Threat Management (UTM) solution. UTM solution allows an administrator to manage wide variety of security issues through a single management console. Junos Space Security Director supports centralized management and offers administrators a simple way to create series of security policies that will control the traffic from within and in between zones or even between VMs. These dynamic security policies understand the context of the virtual machines in the datacenter. Firefly also supports Junos Space Virtual Director, an intelligent, automated VM life cycle management application which easily scales VM to meet dynamic demand. Firefly provides rich connectivity features based on the powerful Junos foundation including routing, NAT and VPN.


SDDC Use Case for Firefly: A very common use case in SDDC is to segregate the guest VMs and provide advanced protection across tiers. Firefly fits into this use case as it can segment the VM and after the VMs are segmented, they are connected via VPN. Firefly also offers multiple layers of defense to protect from any kind of malware and other advanced security threats.


Conclusion: Juniper’s Firefly solution improves performance, lowers latency, and provides end-to-end security in virtualized data centers. Firefly is easily scalable to data centers of any size to ensure that organizations can attain full agility and efficiency of a data center.

by apattnaik at September 04, 2014 06:05 PM