November 22, 2014

Cisco IOS Hints and Tricks

Moving Workloads to the Clouds

David Spark published 16 tips for moving your workloads to the clouds. Contrary to the usual useless nonsense coming down from hybrid cloud evangelist (you know, the people who moved from “VMs following the sun” to “seamless hybrid cloud workload mobility”) some of the tips actually make sense, starting with “Have a real reason for the migration”. Enjoy!

by Ivan Pepelnjak ( at November 22, 2014 11:44 AM

November 21, 2014

Packet Pushers Blog/Podcast

APIs, APIs…a look at Arista’s eAPI

Arista switches have an API known as eAPI. In this article, I will discuss some of the basics of how eAPI operates, how to connect to it, and how to gather network information using it.   Basic eAPI operation eAPI uses JSON-RPC over HTTPS.  What this means in simpler terms is that the communication to and […]

Author information

Kirk Byers

Kirk Byers is the owner of Twin Bridges Technology–a bootstrapped technology business in San Francisco. He teaches Python courses for Network Engineers and writes about network automation at He is a long-time network engineer (CCIE #6243 emeritus), has extensive experience with *nix system administration, and is a Python programmer. He is interested in programming and networking and how to improve network engineering practices through automation.

The post APIs, APIs…a look at Arista’s eAPI appeared first on Packet Pushers Podcast and was written by Kirk Byers.

by Kirk Byers at November 21, 2014 11:52 PM

Network Break 22

Cisco Loves and Hates Net Neutrality, SDN WAN continues to grow and Analysts as AWS puppy dogs - drooling, licking themselves and barking at the AWS reinvent conference.

by Packet Pushers Podcast at November 21, 2014 08:00 PM

Honest Networker
My Etherealmind

Less Sales, Simpler Products, Easier Buying

Out of sheer frustration this week, I tweeted this and got a big response: The Back Story I’ve wasted about 60 hours of customers time working with resellers & vendors to get a quote for a relatively simple network upgrade. Neither the vendor staffers or the reseller employees knew how the product was licensed or […]

The post Less Sales, Simpler Products, Easier Buying appeared first on EtherealMind.

by Greg Ferro at November 21, 2014 04:00 PM

Cisco IOS Hints and Tricks

Open vSwitch Performance Revisited

A while ago I wrote about performance bottlenecks of Open vSwitch. In the meantime, the OVS team drastically improved OVS performance resulting in something that Andy Hill called Ludicrous Speed at the latest OpenStack summit (slide deck, video).

Let’s look at how impressive the performance improvements are.

Read more ...

by Ivan Pepelnjak ( at November 21, 2014 08:21 AM

XKCD Comics

November 20, 2014


Not Raising Red Flags While Looking For A New Job

A reader wrote to me, explaining that they were unhappy in their current job situation, and queried how they might be able look for a new job without raising any red flags with their existing employer. Tricky, but I have a few thoughts, having done this a time or two over my career. […]

by Ethan Banks at November 20, 2014 10:10 PM

Friday News Analysis: Open Networking Foundation, Dave Ward on Open

Open Networking Foundation Extends Conformance Testing to Non-Members “The Open Networking Foundation (ONF), a non-profit organization dedicated to accelerating the adoption of open Software-Defined Networking (SDN), today announced it has extended the ONF OpenFlow™ Conformance Testing Program to include testing for non-ONF members. Companies interested in receiving an OpenFlow Certificate of Conformance for […]

by Ethan Banks at November 20, 2014 08:13 PM

Cisco IOS Hints and Tricks
Potaroo blog

The Resolvers We Use

The Internet's Domain Name System is a modern day miracle. It may not represent the largest database that has ever been built, but nevertheless it's truly massive. The DNS is consulted every time we head to a web page, every time we send an email message, or in fact every time we initiate almost any transaction on the Internet. We assume a lot about the DNS. For example, content distribution networks are observed to make use of the location of the DNS resolver as being also the same location as the user. How robust is this assumption of co-locality of users and their resolvers? Are users always located "close" to their resolvers? More generally, what is the relationship between the end user, and the DNS resolvers that they use? Are they in fact closely related? Or is there widespread use of distant resolvers?

November 20, 2014 04:15 AM

Who's Watching?

It's been more than a year since Edward Snowden released material concerning the activities of US agencies in the area of cyber-intelligence gathering. A year later, and with allegations of various forms of cyber spying flying about, it's probably useful to ask some more questions. What is a reasonable expectation about privacy and the Internet? Should we now consider various forms of digital stalking to be "normal"? To what extent can we see information relating to individuals' activities online being passed to others?

November 20, 2014 04:15 AM

Networking Now (Juniper Blog)

Microsoft's Out-of-Band Fix for a Critical Kerberos Issue

Microsoft released an out-of-band update, MS14-068, yesterday to patch a critical bug in its Kerberos implementation. This bug could allow a remote, unprivileged, authenticated attacker to elevate their privileges to that of any other domain user. Such an attack could also enable the attacker to obtain domain administrator privileges and completely compromise the security restrictions enforced on the targeted domain.

by atyagi at November 20, 2014 01:15 AM

November 19, 2014

Packet Pushers Blog/Podcast

The changing of the guard: Telecom Style-Part 1

The sale of a incumbent local exchange carrier (ILEC) aka the local telephone company can be much more complicated than one might think, ordinary folks anyway. Networking & IT  professionals most likely have a different viewpoint as migrations are a fundamental part of the IT field. Such a transaction becomes more complicated when triple play […]

Author information

Mitchell Lewis

Mitchell is a young technology professional with an interest in networking & communications. He takes an interest in the ever changing telecom space as well as other networking technologies (datacenter etc). His main hobby project is overseeing the expansion of broadband into a co-operative campground, shares of which are owned by a family member.

Mitchell currently is a Cisco Certified Entry Level Tech (CCENT Certified) with intentions of obtaining higher as time permits.He graduated from the Connecticut Technical High School system with a focus in Information Systems Technology (Combination of Higher Ed MIS & Computer Science). While there he developed the production computing platform for his academic department( servers, networking, desktop).

He is currently pursing higher education from the UCONN School of Business & welcomes any opportunity to further advance his experience in the IT Field & professional knowledge.

The post The changing of the guard: Telecom Style-Part 1 appeared first on Packet Pushers Podcast and was written by Mitchell Lewis.

by Mitchell Lewis at November 19, 2014 10:26 PM

The Linux ip Command – An Ostensive Overview

It came to my attention and I was rather surprised to learn a while back that the Linux ifconfig command has been deprecated for quite some time by the Linux ip command set. The ip command isn’t new to me and I’ve recognised its advantages for some time but considering its ‘elevated’ status I thought […]

Author information

Steven Iveson

Steven Iveson

Steven Iveson, the last of four children of the seventies, was born in London and has never been too far from a shooting, bombing or riot. He's now grateful to live in a small town in East Yorkshire in the north east of England with his wife Sam and their four children.

He's worked in the IT industry for over 15 years in a variety of roles, predominantly in data centre environments. Working with switches and routers pretty much from the start he now also has a thirst for application delivery, SDN, virtualisation and related products and technologies. He's published a number of F5 Networks related books and is a regular contributor at DevCentral.

The post The Linux ip Command – An Ostensive Overview appeared first on Packet Pushers Podcast and was written by Steven Iveson.

by Steven Iveson at November 19, 2014 10:03 PM

Stop Doing Post Mortems & Root Cause Analysis With indeni

Indeni has technology that can predict known types of network failures using pre-mortem analysis.

Author information

Sponsored Blog Posts

The Packet Pushers work with our vendors to present a limited number of sponsored blog posts to our community. This is one. If you're a vendor and think you have some blog content you'd like to sponsor, contact us via

The post Stop Doing Post Mortems & Root Cause Analysis With indeni appeared first on Packet Pushers Podcast and was written by Sponsored Blog Posts.

by Sponsored Blog Posts at November 19, 2014 08:00 PM

Cisco IOS Hints and Tricks

Just Published: PCI DSS Videos

The edited videos of the fantastic PCI DSS webinar Michele Chubirka presented in early July have finally been published (yes, there’s a huge backlog that’s getting cleaned up). Enjoy!

by Ivan Pepelnjak ( at November 19, 2014 02:42 PM

Coping with Byzantine Routing Failures

One of my readers sent me an interesting challenge:

We have two MPLS providers sending us default routes and it seems like whenever we have problem with SP1 our failover is not happening properly and actually we have to go in manually and influence our traffic to forward via another path.

Welcome to the wondrous world of byzantine routing failures ;)

Read more ...

by Ivan Pepelnjak ( at November 19, 2014 08:41 AM

Packet Pushers Blog/Podcast

Show 213 – What’s Next for Avaya Enterprise Wireless – Sponsored

Unlike Gen Z’ers, who have never known a world without Wi-Fi (or Minecraft), some of us get to see technology come full circle. Join Alan Hase, VP of Avaya Networking, and the Packet Pushers as they outline (and relish and pontificate) how this phenomenon is playing out in WLAN and Mobility today. Alan highlights how modern WLAN solutions can combine the best of both worlds (the manageability/scale of controller based WLAN solutions with the resiliency/performance of wired solutions) to create a truly unified network that is optimized for today’s BYOD, application-driven, and 802.11ac world. This podcast gets into architecture discussions and is useful for anyone considering improving or expanding their wireless Enterprise footprint.

by Packet Pushers Podcast at November 19, 2014 04:00 AM

XKCD Comics

November 18, 2014

Honest Networker
The Networking Nerd

Wires Are The Exception


Last week I went to go talk to a group of vocational students about networking.  While I was there, I needed to send a couple of emails.  I prefer to write emails from my laptop, so I pulled it out of my bag between talks and did the first thing that came to mind: I asked for the wireless SSID and password.  Afterwards, I started thinking about how far we’ve come with connectivity.

I can still remember working with a wireless card back in 2001 trying to get the drivers to play nice with Windows 2000.  Now, wireless cards are the rule and wired ports are the exception.  My primary laptop needs a dongle to have a wired port.  My new Mac Mini is happily churning along halfway across the room connected to my network as a server over wireless.  It would appear that the user edge quietly became wireless and no tears were shed for the wire.

It’s also funny that a lot of the big security features like 802.1x and port security became less and less of an issue once open ports started disappearing in common areas.  802.1x for wired connections is barely even talked about now.  It’s more of an authentication mechanism for wireless now.  I’ve even heard some vendors of these solutions touting the advantages of using it with wireless and then throwing in the afterthought comment, “We also made it easy to configured for wired connections too.”

We still need wires, of course.  Access points have to connect to the infrastructure.  Power still can be delivered via microwave.  But the shift toward wireless has made ubiquitous cabling unnecessary.  I used to propose a minimum of four cable drops per room to provide connectivity in a school.  I would often argue for six in case a teacher wanted to later add an IP phone and a couple of student workstations.  Now, almost everything is wireless.  The single wire powers a desk phone and an antiquated desktop.  Progressive schools are replacing the phones with soft clients and the desktops with teach laptops.

The wire is not in any danger of becoming extinct.  But it is going to be relegated to the special purpose category.  Wires will only live behind the scenes in data centers and IDF closets.  They will be the thing that we throw in our bag for emergencies, like an extra console cable or a VGA adapter.

Wireless is the future.  People don’t walk into a coffee shop and ask, “Hey, where’s the Ethernet cable?” Users don’t crowd around wall plates with hubs to split the one network drop into four or eight so they can plug their tablets in.  Companies like Aruba Networks recognized this already when they started posing questions about all-wireless designs.  We even made a video about it:

While I don’t know that the all-wireless design is going to work, I can say with certainty that the only wires that will be running across your desktop soon will be power cables and the occasional USB cord.  Ethernet will be relegated to the same class as electrical wires connected to breaker boxes and water pipes.  Important and unseen.

by networkingnerd at November 18, 2014 08:11 PM

Honest Networker
Cisco IOS Hints and Tricks

Do We Have Too Many Knobs?

The last day of Interop New York found me sitting in the Speaker Center with a few friends pondering the hype and reality of SDN and brokenness of traditional network products. One of the remarks during that conversation was very familiar: “we have too many knobs to configure”, and I replied “and how many knobs do you think there are in Windows registry?" (or Linux kernel and configuration files).

Read more ...

by Ivan Pepelnjak ( at November 18, 2014 08:23 AM Blog

MAC Address Aggregation and Translation as an Alternative to L2 Overlays

Not so long ago, if you wanted to build a data center network, it was perfectly feasible to place your layer three edge on the top-of-rack switches and address each rack as its own subnet. You could leverage ECMP for simple load-sharing across uplinks to the aggregation layer. This made for an extremely efficient, easily managed data center network.

Then, server virtualization took off. Which was great, except now we had this requirement that a virtual machine might need to move from one rack to another. With our L3 edge resting at the top of the rack, this meant we'd need to re-address each VM as it was moved (which is apparently a big problem on the application side). So, now we have two options: We can either retract the L3 edge up a layer and have a giant L2 network spanning dozens of racks, or we could build a layer two overlay on top of our existing layer three infrastructure.

Most people opt for some form of the L2 overlay approach, because no one wants to maintain a flat L2 network with dozens or hundreds of thousands of end hosts, right? But why is that?

Continue reading · 10 comments

by Jeremy Stretch at November 18, 2014 02:05 AM

November 17, 2014

My Etherealmind

Cisco On Net Neutrality Isn’t What It Seems

Chambers pointed the finger at Net Neutrality for a slow down in purchasing by carriers in US markets and that he perceives it as damaging to Cisco business interests. I find it more credible that SDN/NFV is slowing capital investments than some far off political change.

The post Cisco On Net Neutrality Isn’t What It Seems appeared first on EtherealMind.

by Greg Ferro at November 17, 2014 06:00 PM

Cisco IOS Hints and Tricks

Just Published: Overlay Virtual Networks in Software Defined Data Centers

Overlay virtual networks are one of my favorite topics – it seems I wrote over a hundred blog posts describing various aspects of this emerging (or is it reinvented) technology since Cisco launched VXLAN in 2011.

During the summer of 2014 I organized my blog posts on overlay networks and SDDC into a digital book. I want to make this information as useful and as widely distributed as possible – for a limited time you can download the PDF free of charge.

Learn more about the book

by Ivan Pepelnjak ( at November 17, 2014 12:24 PM

Packet Pushers Blog/Podcast

BGPSEC: Basic Operation

I’m going to take a little break from my other two series to inject a short series on BGPSEC. I’ll return to HTIRW and RFCs you need to know shortly. BGPSEC is a set of standards currently under consideration in the IETF to secure BGP beyond the origin AS – in other words, to secure […]

Author information

Russ White

Principal Engineer at Ericsson

Russ White is a Network Architect who's scribbled a basket of books, penned a plethora of patents, written a raft of RFCs, taught a trencher of classes, and done a lot of other stuff you either already know about — or don't really care about. You can find Russ at 'net Work, the Internet Protocol Journal, and his author page on Amazon.

The post BGPSEC: Basic Operation appeared first on Packet Pushers Podcast and was written by Russ White.

by Russ White at November 17, 2014 08:00 AM

XKCD Comics

November 15, 2014

Packet Pushers Blog/Podcast

Network Break 21

IT Talent Shortage and Whiny CIOs, Podcasts Make Money, ACI vs NSX wobbles and Dell busts some moves at its conference.

by Packet Pushers Podcast at November 15, 2014 07:00 PM

Cisco IOS Hints and Tricks

Reinventing the wheel (or RFC 1925 sect 2.11)

Simon Wardley is another old-timer with low tolerance for people reinventing the broken wheels. I couldn’t resist sharing part of his blog post because it applies equally well to what we’re seeing in the SDN world:

No, I haven't read Gartner's recent research on this subject (I'm not a subscriber) and it seems weird to be reading "research" about stuff you've done in practice a decade ago (sounds familiar). Maybe they've found some magic juice? Experience however dictates that it'll be snake oil […]. I feel like the old car mechanic listening to the kid saying that his magic pill turns water into gas. I'm sure it doesn't ... maybe this time it will ... duh, suckered again.

Meanwhile the academics already talk about SDN 2.0.

by Ivan Pepelnjak ( at November 15, 2014 11:36 AM

November 14, 2014


Friday News Analysis: Arkin Net, Pica8, OpenDaylight, Plexxi, Juniper

Arkin Net Rises from Stealth with $7M in funding, to bring Google Search-like Simplicity to Software-Defined Datacenter Operations “Arkin Net is coming out of stealth mode with $7 million in funding, led by Nexus Venture Partners, to revolutionize the Software-Defined Networking (SDN) market – projected to be $8B by 2018. Arkin Net is […]

by Ethan Banks at November 14, 2014 02:44 PM

Packet Pushers Blog/Podcast

4 Inevitable Questions When Joining a Monitoring Group, Pt. 1

Leon Adato, Technical Product Marketing Manager with SolarWinds is our guest blogger today, with a sponsored post on the topic of alerting. The Four Questions For people who are interested in monitoring, there is a leap that you make when you go from watching systems that YOU care about, to monitoring systems that other people […]

Author information

Sponsored Blog Posts

The Packet Pushers work with our vendors to present a limited number of sponsored blog posts to our community. This is one. If you're a vendor and think you have some blog content you'd like to sponsor, contact us via

The post 4 Inevitable Questions When Joining a Monitoring Group, Pt. 1 appeared first on Packet Pushers Podcast and was written by Sponsored Blog Posts.

by Sponsored Blog Posts at November 14, 2014 02:00 PM

Cisco IOS Hints and Tricks

Viptela SEN: Hybrid WAN Connectivity with an SDN Twist

Like many of us Khalid Raza wasted countless hours sitting in meetings discussing hybrid WAN connectivity designs using a random combination of DMVPN, IPsec, PfR, and one or more routing protocols… and decided to try to create a better solution to the problem.

Viptela Secure Extensible Network (SEN) doesn’t try to solve every networking problem ever encountered, which is why it’s simpler to use in the use case it is designed to solve: multi-provider WAN connectivity.

Read more ...

by Ivan Pepelnjak ( at November 14, 2014 09:05 AM

XKCD Comics

November 13, 2014

Cisco IOS Hints and Tricks

Just Published: VXLAN 2.0 Videos

Last week I ran the second part of the updated (4-hour) VXLAN webinar. The raw videos are already online and cover these topics:

  • VXLAN-related technologies, including encapsulation, IP multicast use, unicast VXLAN, and VXLAN-over-EVPN;
  • VXLAN implementations, including Cisco Nexus 1000v, VMware vCNS, VMware NSX, Nuage VSP and Juniper Contrail;
  • VXLAN gateways, including Arista, Brocade, Cisco and Juniper;
  • Hardware VTEP integration with OVSDB and EVPN;
  • VXLAN-based data center fabrics, including Cisco’s ACI.

by Ivan Pepelnjak ( at November 13, 2014 01:04 PM

November 12, 2014

Packet Pushers Blog/Podcast

Show 211 – Should IT Engineers Get Fired For Production-Impacting Mistakes?

First off, apologies for the serialization error. We know, the last show was #212 in the title but #211 on the filename, when it should have been #211 through and through. We get it, and we're very sorry, especially to you OCD folks who are twitching uncontrollably right now. Don't fire us. Why didn't we make this show #213 and just skip #211 altogether? Because we have a production schedule that's planned several weeks in advance. Changing this one to #213 would mean that #213 would become #214, and so on for several weeks of lunacy and confusion. And probably crying. Yes, most definitely some crying involved. On a related note, Packet Pushers is poking at the idea of hiring a webmaster / content manager sort of person that would help us avoid these sorts of challenges. Let us know if you're interested in an exploratory conversation. Yes, seriously. Back to the podcast. So, changing numbers is actually sort of hard. Almost as hard as SDN, it would seem. In this show, Greg and Ethan discuss just how hard it is for both vendors and practitioners to get behind any one particular SDN strategy, in part because there's just so many different strategies out there. Has SDN become all about corner cases instead of transformative technology? Hmm. We talk about other things, too, such as whether or not IT people should get fired for making mistakes that cause an outage. Think about that, fellow engineers. Fat finger an interface ID. Cause an outage. Pack up and go home. And finally, Greg and Ethan have a bit of fun with a segment we've titled "Consultobabble." We hope it elicits at least a giggle. We certainly had fun recording the bit, so we'll do more unless you just hated it. In which case, we won't. Because we're all about you. Last but not least, please swing by our sponsors for this show, Auvik Networks and NetRounds. We appreciate them sponsoring Packet Pushers, and we hope you do, too.

by Packet Pushers Podcast at November 12, 2014 07:44 PM

Honest Networker
Renesys Blog

Use Protection if Peering Promiscuously


Last week, I wrote a blog post discussing the dangers of BGP routing leaks between peers, illustrating the problem using examples of recent snafus between China Telecom and Russia’s Vimpelcom.  This follow-up blog post provides three additional examples of misbehaving peers and further demonstrates the impact unmonitored routes can have on Internet performance and security.  Without monitoring, you are essentially trusting everyone on the Internet to route your traffic appropriately.

In the first two cases, an ISP globally announced routes from one of its peers, effectively inserting itself into the path of the peer’s international communications (i.e., becoming a transit provider rather than remaining a peer) for days on end.  The third example looks back at the China Telecom routing leak of April 2010 to see how a US academic backbone network prioritized bogus routes from one of its peers, China Telecom, to (briefly) redirect traffic from many US universities through China.

Recap: How this works

To recap the explanation from the previous blog (and to reuse the neat animations our graphics folks made), we first note that ISPs form settlement-free direct connections (peering) in order to save on the cost of sending traffic through a transit provider.  Suppose that ISP A and ISP B establish such a private link between their networks.  At the BGP routing level, ISP A will then send routes from its customers to its peer ISP B, who will in turn send these routes on to its customers.  As a result, the customers of ISP B will send traffic destined for ISP A through the newly established peering link, saving ISP B from having to pay its transit providers to carry the traffic.  This flow of routes and traffic is illustrated below.


The first way this can go wrong is for ISP B to announce the routes received from ISP A out to the global Internet (through its transit providers) or to ISP B’s other peers.  By doing this, ISP B inserts itself onto the path of incoming traffic to ISP A from outside ISP B’s own network, something ISP A certainly didn’t expect when it took on ISP B as a peer.


ISP B can also mess up by sending routes learned either from its transit providers or peers to ISP A.  If these routes are accepted by ISP A (and they typically will be), such errors put ISP B onto the path of outgoing traffic from ISP A to the networks erroneously announced along this peering link.


These two scenarios can happen independently.  As shown in our last blog, China Telecom leaked routes to and from Vimpelcom numerous times throughout the year.  Most of these incidents involved China Telecom leaking routes it learned from Vimpelcom out to the global Internet (scenario 1); however, on a few occasions, China Telecom also passed a full or partial routing table to Vimpelcom (scenario 2), altering how traffic flowed out of Vimpelcom.

Additional recent examples of peering leaks


logo_en             Yandex_Mail_Russia_Wide

Yandex is essentially the Russian version of Google.  It is the dominant Russian-language search engine and, like Google, Yandex has established a lot of peering links — although, for obvious reasons, with a greater emphasis on the Russian-speaking world.  Beltelecom is the incumbent telecom of Belarus and has become a recurring character in this blog (either for globally routing RFC6598 address space or MITM hijacks).  Beltelecom and Yandex have a peering relationship, as it makes a lot of sense for eyeball networks (Beltelecom) and content providers (Yandex) to try to save on transit costs by interconnecting.  However, for twelve days this year, Beltelecom announced routes it learned from Yandex to its transit provider Telecom Italia (i.e., leak scenario 1 from above).

The AS paths of the impacted routes took the following form:

… 6762 6697 13238 …

This AS path shows that routes from Yandex (AS13238) shared with its peer Beltelecom (AS6697) who leaked them to Telecom Italia (AS6762), a global Tier 1 provider. Normally no provider outside of Belarus would use Beltelecom to reach Yandex.

The result was that traffic destined for Yandex from customers around the world in Telecom Italia’s downstream cone was misdirected first to Beltelecom.  For Yandex’s networks in Russia (which shares a border with Belarus), the impact on latency was minor.  However, Yandex has networks outside of Russia (including some in the Netherlands and the United States) and, for those networks, the latency and paths were dramatically altered. For those receiving Yandex routes via Telecom Italia, Beltelecom inserted itself into Yandex-destined traffic from 22 May through 3 June of this year.

Consider the following example traceroute from Brazil to Yandex’s presence in Palo Alto, California before Beltelecom started leaking Yandex routes. The trace illustrates a typical traffic path, namely from Brazil to Miami and then on to New York and finally California.

trace from João Pessoa, Brazil to Yandex-Palo Alto at 05:46 May 20, 2014
1  *
2 ( Data Center, João Pessoa, Brazil) 0.259ms
3 (SITECNET INFORMÁTICA LTDA, João Pessoa, Brazil)   8.122ms
4                54.263ms
5 (Miami, US)            165.525ms
6                     119.060ms
7 (GTT, New York)                                 179.928ms
8 (New York, US)        199.109ms
9 (Las Vegas, US)          187.579ms
10   (Yandex, Palo Alto, US)                         179.933ms
11 (Palo Alto, US)  187.462ms

However, during the leak, traffic from the same server in Brazil to the same Yandex location in California was redirected to Beltelecom in Minsk, Belarus and then on to Yandex in Moscow, after which Yandex took the traffic to California on its internal backbone.

trace from João Pessoa, Brazil to Yandex-Palo Alto at 00:57 May 23, 2014
1  *
2 ( Data Center, João Pessoa, Brazil) 0.249ms
3 (SITECNET INFORMÁTICA LTDA, João Pessoa, Brazil)  54.175ms
4               54.932ms
5 (São Paulo, BR)       70.403ms
6 (São Paulo, BR) 54.192ms
7 (Frankfurt)   220.925ms
8 (Frankfurt) 404.091ms
9 (Minsk, BY)                   252.911ms
10 (Minsk, BY)                 254.373ms
11 (Minsk, BY)                251.801ms
12 (Minsk, BY)                       295.233ms
13 (Moscow, RU)         266.851ms
14 (Moscow, RU)                266.571ms
15 (Moscow, RU)                 266.611ms
16 (Moscow, RU)        276.847ms
17 (Yandex, Moscow, RU)                            309.937ms
18 (Germany)            281.413ms
19 (Asheville, US)   356.142ms
20 (Las Vegas, US)          356.994ms
21   (Yandex, Palo Alto, US)                         346.466ms
22 (Palo Alto, US)  356.994ms

Did Yandex finally notice this after nearly two weeks of poor performance due to misdirected traffic?  Did Beltelecom ultimately catch the error, after perhaps noticing a surge in traffic along its peering link with Yandex?  We may never know the answers to these questions, but we easily quantified the impact to Yandex’s Internet performance using our continuous global measurement and monitoring platform.


Rascom_Logo_trans        HpAccroche_01

Our next example of a peering relationship gone bad concerns Rascom’s leak of Telma’s routes this summer.  Telma is the incumbent telecom of the African island-nation of Madagascar, and Rascom is a Russian fixed-line operator with a network that extends throughout Europe.  While the previous example involved a very common type of peering relationship, between a content producer (Yandex) and a content consumer (Beltelecom), why on earth would a Russian ISP peer with an ISP from Madagascar?  How much traffic could possibly be passing between these two networks?  Could that traffic really justify a private connection to help to reduce transit costs?  Probably not.  The reason for this arrangement is likely due to the fact that both entities happen to be present at London Internet Exchange and decided to peer because … why not?

When a  provider from a faraway place like Africa or the Middle East establishes a presence at one of the European IXes, it often will establish peering relationships with anyone and everyone.  Once present at an IX, each additional connection carries little marginal cost, so you might as well connect with everybody there in the hopes of reducing your transit costs, if only slightly.  But as we’ll see in this example, each of your peers has the potential to screw up and alter the flow of your Internet traffic.  In other words, every relationship, no matter how seemingly insignificant, carries real risks.  There is no free lunch on the Internet.

Consider the following example of a normal traffic path from New York to Telma in Madagascar, the day before the routing leak.  Level 3 carries the traffic to London, where Telma picks it up and takes it first to Paris and then Madagascar.

trace from New York to Telma, Madagascar at 08:42 Aug 11, 2014
1  *
2             0.602ms
3             69.191ms
4           69.274ms
5            70.905ms
6           69.265ms
7           181.282ms
8               75.363ms
9      87.635ms
10 *
11 93.228ms
12  280.859ms
13      294.419ms
14 *
15 *
16 (Madagascar)           296.248ms


This next trace illustrates the impact of the routing leak on the path and latency between New York and Madagascar.  In this case, Tata takes the traffic to London and then Frankfurt before handing it off to Golden Telecom (Vimpelcom).  Golden takes the traffic to Moscow and delivers it to Rascom, who takes it straight back to London (at LINX), handing it off to Telma so it can continue its journey to Madagascar.  Wow!

trace from New York to Telma, Madagascar at 12:30 Aug 12, 2014
1  *
2   0.791ms
3   (Tata Communications, London)               85.475ms
4         85.733ms
5  *
6          85.654ms
7      85.369ms
8      85.666ms
9       85.332ms
10  (Tata Communications, Frankfurt, DE)        85.792ms
11                      130.005ms
12               131.454ms
13   (Rascom, Vyborg, RU)                       129.323ms
14 *
15      128.012ms
16   (London Internet Exchange (LINX))          126.734ms
17     133.410ms
18 *
19 153.67ms
20  356.605ms
21      348.541ms
22 *
23 *
24 (Madagascar)           350.924ms

We wouldn’t be surprised if the network engineers at both Rascom and Telma were completely unaware of this circuitous routing.  This level of monitoring is often overlooked.

China Telecom—National LambdaRail

Although our final example isn’t recent, it is worth mentioning in this discussion.  During the big China Telecom routing leak of April 2010 that caused an international stir, it is interesting to note where the bogus routes announced by China Telecom (AS23724) propagated the farthest. Before it ceased operations earlier this year, National LambdaRail (NLR) was a “high-speed national computer network owned and operated by the U.S. research and education community.”  NLR also had a peering relationship with China Telecom, the state telecom of China.  When NLR received the bogus origination announcements from its Chinese peer, it accepted them and routed traffic to China that was intended for numerous other locations around the world.

This is what can be most pernicious about routes received across peering links.  Routes from peers are typically prioritized over routes from providers to avoid transit costs.  While many, but certainly not all, transit providers filter the routes they receive from their customers in some manner, it is far less common for peers to do any filtering on the routes they exchange, largely due to the difficulty of determining appropriate routing behavior for an independent entity.  These prioritized and unfiltered peer routes have the potential to cause the performance and security problems we’ve outlined here.

In a 2012 paper entitled “A Case Study of the China Telecom Incident”, I assisted the authors by searching and analyzing traceroute data from the iPlane project for examples of traceroutes that were sucked into China Telecom during the routing leak.  Since much of iPlane’s data is generated using the networks of universities in the U.S. and many universities had a connection to NLR, there were many U.S. universities that had traffic redirected through China Telecom.  As is standard practice, NLR had prioritized routes from its peers—including China Telecom.  U.S. universities may have also prioritized routes from NLR over commercial transit links because NLR might have been a subsidized and therefore cheaper option. This provided a vector for those bogus routes to briefly (the entire incident lasted only 18 minutes) redirect traffic through China.

Here is an example traceroute pulled from iPlane traceroute data that illustrated the impact of the routing leak.  Starting in Norman, Oklahoma this trace goes out to to Internet2 and onto NLR’s routers on the west coast of the US.  There it hands the traffic off to China Telecom before returning it back to the US; it next appears in Cogent’s network in Chicago (ord) before making its way over to Boston.

0      (University of Oklahoma, Norman, US)        0.384ms
1   (RFC 1918)                                  0.287ms
2  (RFC 1918)                                158.051ms
3     (OneNet, Oklahoma City, US)                 0.364ms
4     (OneNet, Oklahoma City, US)                 0.875ms
5     (OneNet, Oklahoma City, US)                 3.025ms
6    (OneNet, Oklahoma City, US)                 3.057ms
7    (OneNet, Oklahoma City, US)                40.005ms
8    (Oklahoma Regents, Oklahoma City, US)      18.231ms
9        18.699ms
10    (National LambdaRail, Los Angeles, US)     71.529ms
11   (National LambdaRail, Los Angeles, US)     71.614ms
12  (National LambdaRail, Los Angeles, US)     71.606ms
13 *
14    (China Telecom, Guangzhou, CN)            280.357ms
15 *
16  296.328ms
17 294.124ms
18     298.383ms
19  315.434ms
20  328.007ms
21 335.061ms
22  340.852ms
23    339.829ms
24        (TA ASSOCIATES, Boston, US)               341.378ms

Next we provide a few examples of AS-level traceroutes (also based on the iPlane data) from U.S. universities impacted by the China Telecom routing leak, presented in a sequence-alignment style.  In each sequence, there is a trace that was redirected through China Telecom (AS4134) by way of NLR (AS11164) on 8 April 2010.  To illustrate the normal paths at that time, the errant one is sandwiched by AS-level traces seen on the previous and successive days.

Purdue University

AS path for ( to (Advertinet, US)
[04/07/10] 17 ----- ----- 209 12067 (19.18 ms)
[04/08/10] 17 11164  4134 209 12067 (106.47 ms)
[04/09/10] 17 ----- ----- 209 12067 (22.04 ms)

University of California, Santa Cruz

AS path for ( to (Spartan Stores Inc., US)
[04/07/10] 5739 2152 11164  286 19151 26554 33372 (67.57 ms)
[04/08/10] 5739 2152 11164 4134  3356 26554 33372 (237.99 ms)
[04/09/10] 5739 2152 11164  286 19151 26554 33372 (66.35 ms)

University of Massachusetts

AS path for ( to (SCOTTRADE, US)
[04/07/10] 1249 ----- -----  1239 3561 12221 (33.06 ms)
[04/08/10] 1249 22742 11164  4134 7018 12221 (247.83 ms)
[04/09/10] 1249 ----- -----  1239 3561 12221 (44.78 ms)

University of Florida

AS path for ( to (Bresnan Communications, LLC., US)
[04/07/10] 6356 ----- ----- 3356 7018 33588 (101.12 ms)
[04/08/10] 6356 11164  4134  174 7018 33588 (280.64 ms)
[04/09/10] 6356 ----- ----- 3356 7018 33588 (101.53 ms)

Oregon State

AS path for ( to (Secure-Netz, DE)
[04/07/10 00:00:00]  4201 -----  3701  3356 25074 (222.03 ms)
[04/08/10 00:00:00]  4201 11164  4134  3320 25074 (287.73 ms)
[04/09/10 00:00:00]  4201 -----  3701  3356 25074 (170.01 ms)


AS path for ( to (Copa Airlines, PA)
[04/07/10 00:00:00] 103 22335  3549 11556 26105 28031 (83.89 ms)
[04/08/10 00:00:00] 103 22335 11164  4134 26105 28031 (148.91 ms)
[04/09/10 00:00:00] 103 22335  3549 11556 26105 28031 (84.89 ms)

There were literally thousands more examples like these in the iPlane data.


In this blog post (and the last one), we don’t want to suggest we are somehow against peering.  Peering is an essential feature of Internet connectivity and will continue to be in the future.

The main takeaway is that if your network is going to prioritize routes from a peer over a transit provider, then your network engineers should also take the time to set up appropriate filtering and monitoring of these links to ensure you don’t accept and act on bogus routes.  As far as the routes you share with your peers, you need to monitor the paths traffic is taking to reach your network to determine if a peer is leaking your routes.  Additionally, if you don’t exchange much traffic with another entity, it may not be worth peering with them just because you are both present at the same Internet exchange point.  But if you must be promiscuous when peering, please use protection.

The post Use Protection if Peering Promiscuously appeared first on Dyn Research.

by Doug Madory at November 12, 2014 01:08 PM

Packet Pushers Blog/Podcast

Network Design Concepts Part-3

In the first article of the series, reliability and resiliency are covered. We should know that whatever device, link type or software you choose eventually they will fail. Thus designing resilient system is one of the most critical aspects of IT. I mentioned that one way of providing resiliency is redundancy. If we have redundant […]

Author information

Orhan Ergun

Orhan Ergun, CCIE, CCDE, is a network architect mostly focused on service providers, data centers, virtualization and security.

He has more than 10 years in IT, and has worked on many network design and deployment projects. Orhan works as a freelance network instructor, for training you can add ' Orhan Ergun ' on skype.

In addition, Orhan is a:
Blogger at Network Computing.
Blogger and podcaster at Packet Pushers.
Manager of Google CCDE Group.
On Twitter @OrhanErgunCCDE

The post Network Design Concepts Part-3 appeared first on Packet Pushers Podcast and was written by Orhan Ergun.

by Orhan Ergun at November 12, 2014 09:57 AM

Cisco IOS Hints and Tricks

Scaling the Cloud Security Groups

Most overlay virtual networking and cloud orchestration products support security groups more-or-less-statefulish ACLs inserted between VM NIC and virtual switch.

The lure of security groups is obvious: if you’re willing to change your network security paradigm, you can stop thinking in subnets and focus on specifying who can exchange what traffic (usually specified as TCP/UDP port#) with whom.

Read more ...

by Ivan Pepelnjak ( at November 12, 2014 09:33 AM

Networking Now (Juniper Blog)

November 2014 Microsoft Patch Tuesday Summary

Welcome to the November edition of Microsoft Patch Tuesday Summary. In this edition there are 14 updates; four are marked "Critical", eight are rated "Important" and two are rated "Moderate”. A total of 33 CVE's (Common Vulnerability and Exposure) were fixed over 14 bulletins this month. One of the Critical update MS14-064 addresses Sandworm related attack CVE-2014-6352 which was seen being exploited in the wild.


Here is a list of Security bulletins which were rolled out in today's Patch Tuesday release.



by prashantk at November 12, 2014 05:27 AM

XKCD Comics

November 11, 2014

Security to the Core | Arbor Networks Security

DDoS Activity in the Context of Hong Kong’s Pro-democracy Movement

In early August, we examined data demonstrating a striking correlation between real-world and online conflict [1], which ASERT tracks on a continual basis [2-7]. Recent political unrest provides another situation in which strong correlative indicators emerge when conducting time-series analysis of DDoS attack data.

The latest round of pro-democracy protests in Hong Kong began on September 22nd when “. . . Students from 25 schools and universities go ahead with a week-long boycott to protest Beijing’s decision to proceed with indirect elections for Hong Kong’s Chief Executive position.” [8]. The protests ramped up on September 28th when a larger pro-democracy group, Occupy Central with Love and Peace, combined forces with the student demonstrators [8-9]. On October 1st, protesters vowed to increased the level of civil disobedience if Hong Kong’s Chief Executive, Leung Chun-Ying, did not step down [10].  Since that time, tensions have increased, with police crackdowns, tear gas, barricades, skirmishes, shutdowns of government buildings and infrastructure, and heavy use of social media to promote both pro-and anti-protest sentiment.  By examining Arbor ATLAS Internet-wide attack visibility data we have identified DDoS attack activity in the APAC region which correlates strongly with the ebb and flow of protest activity in Hong Kong.

Arbor’s ATLAS Initiative

The DDoS information provided in the remainder of this report is derived from Arbor’s ATLAS Initiative. Arbor ATLAS receives anonymized Internet traffic and DDoS event data from over 290 ISPs worldwide which have deployed Arbor’s DDoS Mitigation solutions.  While many observed events are symptomatic of attacks during this period, it is important to note that we cannot definitively identify the motivations behind any given event.

Hong Kong as a Target of DDoS Attacks (September-October)

Number of Observed DDoS Attacks

The following graph illustrates that the number of observed DDoS attacks targeting Hong Kong-related online properties more than doubled between September and October, from 1,688 discrete attacks in September to 3,565 attacks in October:

Figure 1: Total Number of Attacks Targeting Hong Kong (September and October, 2014)

Figure 1: Total Number of Attacks Targeting Hong Kong (September and October, 2014)

Although the sheer number of DDoS attacks increased significantly from September to October, there was not a significant difference with respect to other attack attributes such as size or duration.  For example, the following charts break out the percentage of DDoS attacks within a given size range for both September and October, along with the raw number of DDoS attacks in that size range:

Figure 2: Percentage of Attacks within a given Size Range

Figure 2: Percentage of Attacks within a given Size Range

Overall, the percentage of DDoS attacks within a given size range remain fairly consistent from September to October, with the biggest difference being a relative 4% decrease in the number of DDoS  attacks within the 2gb/sec-to-5gb/sec range.

In summary, the analysis of the number and size of Hong Kong-related DDoS attacks depicted by Figures 1 and 2 above can be summed up by stating that “October saw more of the same – a lot more!

Size of Attacks and Related News Events

Figure 3 illustrates the largest DDoS attacks per day, in terms of bandwidth, targeting Hong Kong-related online properties during the month of October:

Figure 3: Peak Attack Sizes per Day (Gbps)

Figure 3: Peak Attack Sizes per Day (Gbps)

Three large DDoS attacks on October 14th (45.4gb/sec), 17th (38.3gb/sec), and 19th (45.6gb/sec) stand out. The total number of observed DDoS attacks targeting Hong Kong-related online properties (289, 419, and 427 respectively) also peaked on these days.  Since the vast majority of DDoS events reported via ATLAS are anonymized, it cannot be definitively determined how these specific DDoS attacks were related to the ongoing protests.  However, it appears that these attacks coincide with reports on Twitter and  by the Wall Street Journal of anti-protest crowds attempting to physically prevent pro-democracy newspaper publisher Apple Daily from distributing its newspapers. Specifically, the Journal noted that Apple Daily “simultaneously faced a cyberattack that brought down its email system for hours” [11]. On October 14th, Computerworld Hong Kong quoted an employee from Next Media (Apple Daily’s parent company), as follows: “The network was a total failure, affecting not just Apple Daily, but all the publications under Next Media” [12].

What’s Next?

Based on in-region DDoS attack statistics for the first week of November, continued DDoS attacks on Hong Kong-related Internet properties appear to be taking place. The following graph illustrates peak DDoS attack sizes in the 30gb/sec-plus range on four consecutive days (November 3rd – 6th):

Figure 4: Peak Attack Sizes per Day (Gbps) in the first week of November

Figure 4: Peak Attack Sizes per Day (Gbps) in the first week of November


While establishing definitive causal relationships and attribution are challenging  it is apparent that DDoS attacks have become the ‘new normal’ during periods of political unrest worldwide. In this case, we observed a 111% increase in the number of DDoS attacks targeting Hong Kong-related Internet properties when analyzing the months immediately before and after protester demands, on October 1st, for Hong Kong’s Chief Executive to step down. Additionally, large-scale DDoS attacks were observed targeting Hong Kong-related Internet properties that coincide with reports of debilitating disruptions of online media outlets sympathetic to the protest movement.





[4] ASERT Threat Intelligence Brief 2013-5: Observed Syrian Cyber Adversary Capabilities and Indicators. Available to Arbor customers upon request.

[5] ASERT Threat Intelligence Brief 2013-7: The Impact of Protest Activity on Internet Service in Thailand. Available to Arbor customers upon request.

[6] ASERT Threat Intelligence Brief 2013-4: Potential Targeted Attacks Against Various International Interests. TLP Amber. Available to Arbor customers upon request.

[7] ASERT Threat Intelligence Brief 2014-04: Counter Terrorism Expo and Bulgarian State Agency for National Security Cyber-Threat Alert. TLP Amber. Available to Arbor customers upon request.






by Kirk Soluk at November 11, 2014 07:09 PM

SNOsoft Research Team