July 16, 2018


How to create your own Docker image

I mentioned in my previous post that I’ll explain how to create your own Docker image and customize it however you’d like. While is great to just use an image from Docker Hub, it can be that you need some customized image to fit your needs. As said before, is not hard at all to … Continue reading How to create your own Docker image

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]

by Calin at July 16, 2018 05:03 AM

July 14, 2018

ipSpace.net Blog (Ivan Pepelnjak)

Updated Design on blog.ipspace.net

I synced the CSS used on blog.ipspace.net with the one used on the main web site. There should be no visible changes apart from a few minor fixes in color scheme and the main column being a bit narrower, but if you spot any errors please let me know.

During the summer break, I’m doing much-needed web site maintenance. Regular blog posts will return in autumn.

by Ivan Pepelnjak (noreply@blogger.com) at July 14, 2018 07:11 AM


New Ubuntu 18.04 Docker image – Python For Network Engineers

About one year ago I’ve created the Ubuntu 16.04 PFNE Docker image. It’s time for a new version of the Ubuntu PFNE Docker image to support Network engineers learn Python and test automation. Recently, Ubuntu announced that on the Ubuntu Docker Hub the 18.04 LTS version is using the minimal image. With this change when … Continue reading New Ubuntu 18.04 Docker image – Python For Network Engineers

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]

by Calin at July 14, 2018 05:09 AM

July 13, 2018

ipSpace.net Blog (Ivan Pepelnjak)

Review: Ansible for Networking Engineers

An engineer attending Ansible for Networking Engineers online course sent me this feedback:

This is a great place to learn Ansible and Network Automation from scratch. Starting with an emphasis on the fundamentals (YAML, JSON, Jinja2, how to group your network devices for automation, etc.) you progressively build up towards useful network automation.

He particularly liked the additional features that are part of any ipSpace.net online course:

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at July 13, 2018 12:22 PM

The Networking Nerd

Independence, Impartiality, and Perspective

In case you haven’t noticed recently, there are a lot of people that have been going to work for vendors and manufacturers of computer equipment. Microsoft has scored more than a few of them, along with Cohesity, Rubrik, and many others. This is something that I see frequently from my position at Tech Field Day. We typically hear the rumblings of a person looking to move on to a different position early on because we talk to a great number of companies. We also hear about it because it represents a big shift for those who are potential delegates for our events. Because going to a vendor means loss of their independence. But what does that really mean?

Undeclaring Independence

When people go to work for a manufacturer of a computing product, the necessarily lose their independence. But that’s not the only case where that happens. You can also not be truly independent if you work for reseller. If your company specializes in Cisco and EMC, are you truly independent when discussion Juniper and NetApp? If you make your money by selling one group of products you’re going to be unconsciously biased toward them. If you’ve been burned or had the rug pulled out from under you by a vendor, you may be biased against them.

Likewise, if you work yourself in an independent consulting business, are you honestly and truly independent? That would mean you’re making no money from any vendor as your customer. That means no drafting of whitepapers, no testing, and no interaction with them that involves compensation. This falls into the same line of thinking as the reseller employee. Strictly speaking, you aren’t totally independent.

Independence would really mean not even being involved in the market. A public defender is truly independent of IT. A plumber is independent of IT. They don’t have biases. They don’t care either way what type of computer or network solution they use. But you don’t consider a public defender or a plumber to be an expert on IT. Because in order to know anything about IT you have to understand the companies. You have to talk to them and get what they’re doing. And that means you’re going to start forming opinions about them. But, truthfully, real independence is not what you’re looking for here.

Partial Neutrality

Instead of independence, which is a tricky subject, what you should look for is impartiality. Being impartial means treating everything fairly. You accept that you have bias and you try to overcome it. For example, judges are impartial. They have their own opinions and thoughts about subjects. They rule based on the law. They aren’t independent of the justice system. Instead, they do their best to use logic and rules to decide matters.

Likewise, people in IT should strive to be impartial. Instead of forming an opinion about something without thought we should try to look at all aspects of the ideas before we form our opinions. I used to be a huge fan of IBM Thinkpad laptops. I carried one for many years at my old job. Today, I’m a huge fan of MacBooks. I use one every day. Does that mean that I don’t like Windows laptops any more? It means that I don’t use them enough to have a solid opinion. I may compare some of the features I have on my current laptop against the ones that I see, but I also recognize that I am making that comparison and that I need to take it into account when I arrive at my decision.

Extending this further, when making decisions about how we analyze a solution from a company, we have to take into account how impartial we’re going to be in every direction. When you work for a company that makes a thing, whether it be a switch or a laptop or a car, you’re going to be partial to the thing you make. That’s just how people are. We like things we are involved in making. Does that mean we are incapable of admiring other things? No, but it does mean that we have some things to overcome to truly be impartial.

The hardest part of trying to be impartial is realizing when you aren’t. Unconscious bias creeps in all the time. Thoughts about the one time you used a bad product or ate bad food in a restaurant make you immediately start thinking about buying something else quickly. Even when it’s unfounded we still do it. And we have to recognize when we’re doing it in order to counter it.

Being impartial isn’t easy on purpose. Being able to look at all sides of an issue, both good and bad, takes a lot of extra effort. We don’t always like seeing the good in something we dislike or finding the bad parts of an opinion we agree with. But in order to be good judges of things we need to find the balance.

Tom’s Take

I try my best to be impartial. I know it’s hard to do. I have the fortune to be independent. Sure, I’m still a CCIE so I have a lot of Cisco knowledge. I also have a lot of relationships with people in the industry. A lot of those same people work at companies that are my customers. In the end, I have to realize that I need to work hard to be impartial every day. That’s the only way I can say that I’m independent and capable of evaluating things on their own merits. It’s just a matter of having the right perspective.

by networkingnerd at July 13, 2018 04:51 AM

XKCD Comics

July 12, 2018


Goodbye firstdigest.com, welcome ipnet.xyz

I hope you’ve noticed that when you access firstdigest.com (or a link associated with this domain) you’re getting redirected to ipnet.xyz. No, I didn’t got my blog hacked. I’ve decided to go ahead with another domain for this blog. If you ever had to curiosity to read the About page (yes, it needs badly an … Continue reading Goodbye firstdigest.com, welcome ipnet.xyz

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]

by Calin at July 12, 2018 10:18 PM

My Etherealmind
Dyn Research (Was Renesys Blog)

Last Month In Internet Intelligence: June 2018

In June, we launched the Internet Intelligence microsite (home of this blog), featuring the new Internet Intelligence Map.  As the associated blog post noted, “This free site will help to democratize Internet analysis by exposing some of our internal capabilities to the general public in a single tool. …. And since major Internet outages (whether intentional or accidental) will be with us for the foreseeable future, we believe offering a self-serve capability for some of the insights we produce is a great way to move towards a healthier and more accountable Internet.”

While we will continue to share information about Internet disruptions and events as they occur via @InternetIntel, we also plan to provide a monthly roundup in a blog post, allowing readers to learn about Internet disruptions and events that they may have missed, while enabling us to provide additional context and insight beyond what fits within Twitter’s character limit.


In the past, countries including Iraq, Syria, and Ethiopia have implemented partial or complete national Internet shutdowns in an effort to prevent student cheating on exams. This past month saw Iraq implement yet another round of Internet shutdowns, and Algeria began a similar program as well.

The Internet Intelligence Map graphs shown above highlight Internet shutdowns that occurred in Iraq on June 21 and June 27. The national backbone was taken down from 4:00-6:00 UTC on the 21st, and from 3:00-5:00 UTC on the 27th to prevent student cheating as a second round of student exams began. An earlier round of shutdowns took place between May 27 and June 16, and this second round is expected to last until July 12. According to a published report, the shutdowns are being implemented at the request of the Ministry of Education.

A day prior, three separate brief disruptions to Internet connectivity occurred in Algeria, as shown in the figure above. According to a published report, “The Algerian Ministry of National Education announced that it will cut the Internet service across the entire country for an hour after the start of each High School Certificate Examination to avoid any exam leakage.” In addition to this Internet shutdown, additional measures were put into place in an effort to limit cheating, including banning mobile phones, tablets and other digital devices at exam locations.

The lower graph in the figure below shows the impact of these shutdowns on Telecom Algeria, the country’s state-owned telecommunications operator. Similar to the drops seen in the traceroute completion ratio in the figure above, three similar declines are seen in the number of completed traceroutes to endpoints within Telecom Algeria’s network on June 20.

A blog post from advocacy group SMEX indicated that Mauritania was also planning to implement a similar set of Internet shutdowns for exams between June 11 and June 21, and a Twitter post from them on June 19 highlighted a four-hour shutdown observed that day. However, there was no evidence of such shutdowns within the country seen in the Internet Intelligence Map on June 19, or over the broader time period. This may be because the shutdowns were more targeted in nature, affecting only mobile connectivity, according to a published report.

Fiber Cuts

Internet outages due to fiber cuts are not all that unusual, unfortunately, and occur fairly frequently on a local basis. However, sometimes these cuts have a wider impact, impacting Internet connectivity on a national basis. The Internet Intelligence team has used our measurement data in the past to illustrate the impact of cuts in the Ukraine, Egypt, Armenia, Chile, and Arizona.

On June 18, @Abdalla_Salmi posted the following Tweet:

<script async="async" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>

This Internet disruption in Libya can be seen clearly in the figure below starting late in the day GMT, lasting for just over 16 hours. The graphs show that it was not a complete national Internet outage, with none of the metrics dropping to/near zero, in line with the Tweeted statement that connectivity issues were only seen in some areas of the country.

According to a subsequent Facebook post from the Libyan Interim Government, the Internet disruption was due to a breakdown in a fiber optic cable – it was not due to fighting in the region. A published report included more details, explaining that the country’s General Authority for Communications and Informatics (GACI) said that the interruption was caused by a cut in the fiber optic cable in the Ghanema district near the city of Khoms in the western region, and that the services were restored gradually after maintenance work by Hatif Libya Company.

Just a few days earlier, on June 13, the Democratic Republic of the Congo experienced an Internet disruption that lasted for approximately half a day, as seen in the figure below. The country is no stranger to Internet disruptions, and has experienced issues in the past related to widespread political protests.

However, it appears that the problem this time was related to issues with a submarine cable, according to a Tweet from local telecommunications provider MTN Congo:

<script async="async" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>

According to TeleGeography’s submarinecablemap.com, the Congo is connected to the West African Cable System (WACS) and the Africa Coast to Europe (ACE) cable. A March 30 cut to the ACE cable impacted connectivity to 10 countries (not including the Congo), but it is unclear if it was the culprit in this disruption as well.

by David Belson at July 12, 2018 02:48 PM

July 11, 2018

My Etherealmind

Musing: The Short Life of Influencers

We have a seen a number of people in the influencer recently taking jobs at vendors. Keith Townsend is the latest and I wish him the very best. I'm not saying if you participate in @TechFieldDay you'll find a new job. But the correlation between the number of TFD delegates starting new jobs in the […]

by Greg Ferro at July 11, 2018 05:42 PM

XKCD Comics

July 10, 2018

Dyn Research (Was Renesys Blog)

Shutting down the BGP Hijack Factory

It started with a lengthy email to the NANOG mailing list on 25 June 2018: independent security researcher Ronald Guilmette detailed the suspicious routing activities of a company called Bitcanal, whom he referred to as a “Hijack Factory.”  In his post, Ronald detailed some of the Portuguese company’s most recent BGP hijacks and asked the question: why Bitcanal’s transit providers continue to carry its BGP hijacked routes on to the global internet?

This email kicked off a discussion that led to a concerted effort to kick this bad actor, who has hijacked with impunity for many years, off the internet.

Transit Providers

When presented with the most recent evidence of hijacks, transit providers GTT and Cogent, to their credit, immediately disconnected Bitcanal as a customer.  With the loss of international transit, Bitcanal briefly reconnected via Belgian telecom BICS before being disconnected once they were informed of their new customer’s reputation.

The following graphic illustrates a BGP hijack by Bitcanal via Cogent before Cogent disconnected them. Bitcanal’s announcement of (Beijing Jingdong 360 Degree E-commerce) was a more-specific hijack of, normally announced by AS131486 (Beijing Jingdong 360 Degree E-commerce).  The graphic on the right shows another prefix being initially transited by GTT (AS3257) and then briefly via BICS before those companies terminated service to Bitcanal.


Following the loss of these transit providers, these three prefixes (below), previously announced by Bitcanal, moved to a new home at Meerfarbig GmbH.  However, when Meerfarbig learned where their new customer had come from, Meerfarbig quickly disconnected them as well. Anna Dragun-Damian PL Anna Dragun-Damian PL Xantho Ltd LT

The loss of transit also disconnected Routed Solutions (AS39536), ostensibly a customer of Bitcanal, although Bitcanal is listed as its admin contact on its WHOIS registration.

The leftmost graphic below shows a prefix moving briefly to Meerfarbig after Bitcanal was cut off by major transit providers.  The rightmost graphic shows a prefix originated by AS39536 being disconnected when Bitcanal lost its transit, but returning to circulation via M247 (AS9009).

Internet Exchange Points (IXPs)

But Bitcanal didn’t only announce hijacked routes via transit providers, it has also extensively used Internet exchange points (IXPs) as a way to send hijacked routes directly to unsuspecting networks.  While the German IXP DE-CIX reportedly dropped Bitcanal last year for bad behavior, it took behind-the-scenes coordination in recent days to get Bitcanal booted from LINX and AMSIX, the major IXPs in London and Amsterdam, respectively.

Latest disconnections

In the past 24 hours, there have been two additional significant disconnections which greatly limit Bitcanal’s ability to announce its hijacks. At 16:46 UTC on 9 July 2018, Hurricane Electric (AS6939) de-peered Bitcanal (AS197426) (graphic on left). Earlier today at 11:40 UTC Portuguese transit provider IPTelecom terminated service to Bitcanal (graphic on right). While Bitcanal appears to remain connected (for the time being) at ESPANIX, with the loss of IPTelecom transit, Bitcanal is effectively cutoff from the global internet.

Bitcanal’s IPv6 route (2a00:4c80::/29) was also withdrawn at 16:04 UTC today. According to Spamhaus, it was also the source of large amounts of spam email and is listed on their IPv6 Drop list.

A Long Running Reputation

Longtime followers of this blog may recognize the name Bitcanal (retail name Ebony Horizon) as we have documented their numerous flagrant BGP hijacks in the past including the hijack of IP address space belonging to the State Attorney General of Texas ( back in 2014.  They warranted their own section (“Case 2”) in my 2015 blog post, The Vast World of Fraudulent Routing.

We’re not the only ones to have noticed something suspicious with Bitcanal: Spamhaus lists all of their ASNs (AS197426, AS3266, AS200775, and AS42229) on their ASN Droplist due to a history of originating massive amounts of spam email.

Lessons for IXPs

There are lessons to be learned from the past couple of weeks, specifically lessons for IXPs.

Bad actors like Bitcanal take advantage of IXPs to form myriad peering relationships for the purpose of injecting fraudulent routes.  These routes can be used to send spam and other malicious traffic.  These bad actors presume people don’t generally monitor the routes they receive from peers and by hijacking the IP space of others, they attempt to evade IP blacklists.

Based on the discussions with IXPs regarding this particular case, the following points are worthy of consideration.

1) Even if abuse didn’t take place across your exchange, you can still consider disconnection to mitigate future risk.  If it had been widely known that DECIX kicked out Bitcanal last year, might other IXes have disconnected them?  Or at least started scrutinizing their activity at the exchange?

2) IXPs are not just a neutral transport bus anymore. They facilitate a unique service that malicious actors can leverage.  Like it or not, this makes IXPs responsible too.

3) Ensure that you have monitoring and analysis capabilities in place.  Multiple IXPs contacted did not have MRT files of their route servers, or PCAP collection to verify any claim.  If an IXP has a policy of requiring evidence of bad behavior, it must also be collecting that evidence and, most importantly, a process to review that evidence when a reasonable inquiry is made.

The removal of this bad actor was accomplished with the work of a number of people in the internet community. I would especially like to thank Job Snijders of NTT for his assistance on this blog post.

by Doug Madory at July 10, 2018 05:41 PM

My Etherealmind

Looking to Meet, Podcast and Video in San Francisco/Silicon Valley next week

I’m travelling to Silicon Valley next week for Network Field Day 18 and Google Next the following week. I have extended my stay with a couple of free days . Get in touch if you would like to meet. Hoping to podcast, some video and some fun. Looking for people who listen to Packet Pushers who just want […]

by Greg Ferro at July 10, 2018 02:03 PM

July 09, 2018

My Etherealmind
ipSpace.net Blog (Ivan Pepelnjak)

Goodbye, content.ipspace.net

It turns out that while I cannot bring myself to writing or creating other content during the summer break, it feels perfectly fine to be a janitor and fix small things on the web site.

One of the long-outstanding items: get rid of the free content web site that never went where I wanted it to go… one can do only so much in 24 hours. All the features available on content.ipspace.net are now part of the main ipSpace.net web site including pointers to free content and list of free presentations.

During the summer break, I’m publishing blog posts about the projects I’m working on. Regular blog posts will return in autumn.

by Ivan Pepelnjak (noreply@blogger.com) at July 09, 2018 12:00 PM

XKCD Comics

July 06, 2018

The Networking Nerd

The Privacy Pickle

I recorded a fantastic episode of The Network Collective last night with some great friends from the industry. The topic was privacy. Originally I thought we were just going to discuss how NAT both was and wasn’t a form of privacy and how EUI-64 addressing wasn’t the end of days for people worried about being tracked. But as the show wore on, I realized a few things about privacy.

Booming In Peace

My mom is a Baby Boomer. We learn about them as a generation based on some of their characteristics, most notably their rejection of the values of their parents. One of things they hold most dear is their privacy. They grew up in a world where they could be private people. They weren’t living in a 1 or 2 room house with multiple siblings. They had the right of privacy. They could have a room all to themselves if they so chose.

Baby Boomers, like my mom, are intensely private adults. They marvel at the idea that targeted advertisements can work for them. When Amazon shows them an ad for something they just searched for they feel like it’s a form of dark magic. They also aren’t trusting of “new” things. I can still remember how shocked my mother was that I would actively get into someone else’s car instead of a taxi. When I explained that Uber and Lyft do a similar job of vetting their drivers it still took some convincing to make her realize that it was safe.

Likewise, the Boomer generation’s personal privacy doesn’t mesh well with today’s technology. While there are always exceptions to every rule, the number of people in their mid-50s and older that use Twitter and Snapchat are far, far less than the number that is the target demographic for each service. I used to wonder if it was because older people didn’t understand the technology. But over time I started to realize that it was more based on the fact that older people just don’t like sharing that kind of information about themselves. They’re not open books. Instead, Baby Boomers take a lot of studying to understand.

Zee Newest

On the opposite side of the spectrum is my son’s generation, Generation Z. GenZ is the opposite of the Boomer generation when it comes to privacy. They have grown up in a world that has never known anything but the ever-present connectivity of the Internet. They don’t understand that people can live a life without being watched by cameras and having everything they do uploaded to YouTube. Their idea of celebrity isn’t just TV and movie stars but also extends to video game streamers on Twitch or Instagram models.

Likewise, this generation is more open about their privacy. They understand that the world is built on data collection. They sign away their information. But they tend to be crafty about it. Rather than acting like previous generations that would fill out every detail of a form this generation only fills out the necessary pieces. And they have been known to put in totally incorrect information for no other reason than to throw people off.

GenZ realizes that the privacy genie is out of the bottle. They have to deal with the world they were born into, just like the Baby Boomers and the other generations that came before them. But the way that they choose to deal with it is not through legislation but instead through self-regulation. They choose what information they expose so as not to create a trail or a profile that big data consuming companies can use to fingerprint them. And in most cases, they don’t even realize they’re doing it! My son is twelve and he instinctively knows that you don’t share everything about yourself everywhere. He knows how to navigate his virtual neighborhood just a sure as I knew how to ride my bike around my physical one back when I was his age.

Tom’s Take

Where does that leave me and my generation? Well, we’re a weird mashup on Generation X and Generation Y/Millenials. We aren’t as private as our parents and we aren’t as open as our children. We’re cynical. We’re rebelling against what we see as our parent’s generation and their complete privacy. Likewise, just like our parents, we are almost aghast at the idea that our children could be so open. We’re coming to live in a world where Big Data is learning everything about us. And our children are growing up in that world. Their children, the generation after GenZ, will only know a world where everyone knows everything already. Will it be like Minority Report, where advertising works with retinal patterns? Or will it be a generation where we know everything but really know nothing because no one tells the whole truth about who they are?

by networkingnerd at July 06, 2018 05:11 PM

XKCD Comics

July 05, 2018


iNOG-10 & RIPE-Hackathon

In June 2018, I was lucky enough to attend the iNOG::10 session in Dublin, co-present a talk and also take part in the RIPE hackathon.

This post is a share on the experience. This isn’t because I’m running out of non-technical material, but this is to uncover both events for those that may want to attend, take part and experience what these kinds of sessions offer.


The iNOG Irish Network Operators community surfaced briefly with events in 2005 (originally as the IENOG) but fell silent and was reborn in 2015 as the organisation as it is today. Started by five returnees to Ireland and some economic migrants, the group has been seeing a high number of attendees to the events and over 700 members on Meetup! Not bad for something that came in on a started on a boat!!! (See below).

The group aims to deliver valuable content to the audience free of charge. Whilst ‘valuable’ has a variety of meanings depending on the audience, the general idea is to share experience of network based activities. As you can imagine, this is very wide ranging and just in the iNOG 10 session, talks were given on automation, data visualisation, data ethics and even zen like topics around the industry. Inclusion and diversity is a focal point and ALL are welcome. The atmosphere was super friendly, warm and welcoming. Whilst 35 minutes flew by for Mierdin and me, it was super fun to be there and I hope to return when the universe permits it.

<iframe allow="autoplay; encrypted-media" allowfullscreen="allowfullscreen" frameborder="0" height="315" src="https://www.youtube.com/embed/EFTrxDIBKC4" width="560"></iframe>


Over the weekend following iNOG::10, a hackathon held by RIPE and iNOG took place in a co-working space called DoSpace in Grand Canal Dock, Dublin.

For those wondering why such a thing happens and the reasons behind it, read on.

RIPE is the RIR for EMEA and anyone working in ISP / SP land knows what they do. For those that don’t, hit the link at the beginning of this paragraph for an overview. RIPE also happen to run the Atlas system, which consists of hardware probes and a data collection system. The sheer number of tests running is incredible and part of the hackathon was to find ways to re-use the data sets generated by the Atlas probes. However, I was pleased to see the team from RIPE promoted the ethics of Open Source and in reality wanted to bring people together to work on some common good. It created a warm fuzzy feeling and one that I was pleased to be part of over the weekend.

In terms of projects, members submitted an idea to the group via an elevator pitch vocally and it was left to the group to decide what they wanted to do. Check here for information on the projects. Once the teams were formed, we found a flock of tables and setup shop for the duration of the weekend. Over said weekend, each group defined tasks, created code repositories, organised pipelines and iterated towards success.

When it came to skill sets, everyone who attended brought different qualities with them. Some were new to software and came to learn as well as give back. Others were seasoned developers with leadership skills. It was great to see everyone work together irrelevant of skill gaps and close tasks off. Despite the skill gaps, everyone helped each other throughout the weekend without any serious disagreements or upset! In anything collaborative, healthy debate is always welcome, screaming and shouting is absolutely not. These events promote nothing but welcome and warmth, so without surprise, a good time was had by all and saying goodbye to the group was surprisingly quite emotional!

Special thanks to the iNOG organisers for not only iNOG::10 but also providing the food, drink and creature comforts required for something like the hackathon to happen. The team worked extremely hard without financial reward to ensure this event went as smooth as possible. More special thanks to the host location WorkDay for the great setting for iNOG::10 and DoSpace for the great setting and hospitality for the hackathon.


If you’re apprehensive about attending an event like this, the best thing you can do is reserve a place (these things get busy!) and just attend. A lot of effort is made by the organisers to create a welcoming environment and provide content of interest. Irrelevant of skill set, there is always something to learn and share.

For the sake of transparency, I have apprehensions about attending events like this from an inner monologue that goes: “I don’t really have any hardcore skills that anyone would find useful. I can’t do UIs and feel like a spare part around talented people”. It’s completely normal so pluck up the courage, take part and enjoy.

If you’re interested in anything iNOG, reach out to Donal O Duibhir and Cristian Sirbu.

I fell victim to not realising iNOG::10 was the 16th event. Thank you hex for hiding that information in plain sight.



Juniper paid for my travel to and from this event along with the hotel costs. Thank you to my employer for supporting me and this event indirectly.

Our team!

The post iNOG-10 & RIPE-Hackathon appeared first on ipengineer.net.

by David Gee at July 05, 2018 06:40 PM

ipSpace.net Blog (Ivan Pepelnjak)

Feedback: Data Center Infrastructure for Networking Engineers

When I created the Data Center Infrastructure for Networking Engineers webinar, I wanted to reach these goals:

  • Understand the data center acronym soup;
  • Build a conceptual framework of the data center technologies and solutions.

Every now and then I get feedback from a happy attendee telling me how the webinar helped them. Here’s what I got earlier this month:

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at July 05, 2018 12:24 PM

July 04, 2018

Aaron's Worthless Words

Cisco Live 2018 – Yes, I Went Too

It’s been a very busy month or so. June is always like that, it seems. There’s ARRL Field Day, which is always the last rainy weekend in June. This year, Cisco Live was in June, and that typically includes Tech Field Day activities. Right before that, we had the whole family in town for a family reunion. There was all sorts of stuff going on. Now that most of that has blown over, I’ve collected my thoughts and wanted to talk about Cisco Live this year.

Those who are of any importance in the networking world (LOL!) converged on Orlando this to attend the conference. Orlando brings back all sorts of memories — from Taverna Opa to Sizzler to LISP explained with plates — and we’re all familiar with the Orange County Convention Center. It’s a great facility with enough room to handle the largest of gatherings. I don’t think I saw the attendance numbers, but I would guess there were 30,000 attendees at Cisco Live this year. A typical crowd for the event, and the venue was more than adequate.

This year, I went on the Imagine Pass instead of the full conference pass. This pass included admission to the World of Solutions, the keynotes, the social media hub, the CAE, and meals. This sounds a whole lot like the Social Pass from the last 84882 years, right? Well, it is except for the food, which was expected to be of a much higher quality than the box meals we’ve been getting the last few years. Did that work out?  I really didn’t think the food could get any worse, but I was wrong. I would up only partaking only three times all week. It was that bad. Here’s an example.

That’s a barbecue pork sandwich and cole slaw. It tasted worse than it looks.  I wound up just throwing it away.  If the Imagine Pass was the same price as the old Social Pass, I wouldn’t think much of it.  The going rate was $400 MORE than the Social Pass, though.  This isn’t a $400 sandwich.

Since I wasn’t attending sessions all day, I was able to walk around the World of Solutions and talk to some vendors. It was exactly what I thought it would be. Was it mind-blowing? No. It met all my expectations and needs. That’s it.  Of course, the usual suspects were there doing their part — DevNet, the Technical Solutions Clinic, the Walk-In Self-Paced Labs. Of course, our favorite of all-time was there, too.

Cisco Tactical Operations. Look up those guys. Learn what they do. Be jealous that you don’t do that for a living (then figure out how to do this in your own tow, county, state, country).

For me, the biggest problem of the entire event was actually where the Social Media Hub was located. It was on the floor of the World of Solutions, which caused some problems. No one could access it until the World of Solutions was actually open. The team at the Social Media Hub was hard at work Saturday, Sunday, Monday, while the Twitterati were locked outside until the doors opened on Monday. Do we really need to actually be standing at the Hub to tweet? Of course not. The Hub, though, has always been a place where the social media participants can meet, get caught up with everything going on at the event, and fully interact with the team. Opportunities lost. Since the Hub was inside, we actually all got kicked out on the last day of the event as they began tearing down the World of Solutions. Not fun for us, but definitely not fun for the Social Media team who had to relocate unexpectedly. Next year, let’s just put the Hub outside, alright?

The Customer Appreciation Event (CAE) was at Universal Studios again. I don’t think I need to tell anyone that this is a great place (if you like that sort of thing), but it is Florida and it is outside. That means we got rained on. I got soaked! It was fine, though, since we got to catch a few songs from Cake. I’ve never even heard of the other acts, so I spent the rest of the time trying to dry off (don’t underestimate exactly how wet I actually was) and enjoying the cuisine.

Nonetheless, another great Wednesday at Cisco Live.

Will I go to Cisco Live again next year?  No doubt.  Will I go on the Imagine Pass?  Not even if someone else pays for it.

Send any 4th of July fireworks questions to me.

by Aaron Conaway at July 04, 2018 08:54 PM

XKCD Comics

July 03, 2018

ipSpace.net Blog (Ivan Pepelnjak)

IOS Adventures: Save the Princess

Want to become Captain Catalyst and save Princess Cattools from the Junipers tribe that invaded IOS Kingdom? Alexander Harsbo created an IOS Adventures game that will keep you busy should you get bored at the beach.

Enjoy ;)

by Ivan Pepelnjak (noreply@blogger.com) at July 03, 2018 06:00 AM

July 02, 2018

XKCD Comics

July 01, 2018


INE’s CCIE R&S v5 topology for EVE-NG using CSR1000v

In my previous blog post I’ve adapted the INE’s CCIE Routing and Switching topology to be used with EVE-NG using IOSv (or vIOS) L3 images for routers and L2 images for switches. Following the promise in that blog post, I’ve adapted the same topology using Cisco CSR1000v images for routers and IOSv L2 images for … Continue reading INE’s CCIE R&S v5 topology for EVE-NG using CSR1000v

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]

by Calin at July 01, 2018 03:37 AM

June 29, 2018

The Networking Nerd

Tale From The Trenches: The Debug Of Damocles

My good friend and colleague Rich Stroffolino (@MrAnthropology) is collecting Tales from the Trenches about times when we did things that we didn’t expect to cause problems. I wanted to share one of my own here about the time I knocked a school offline with a debug command.

I Got Your Number

The setup for this is pretty simple. I was deploying a CallManager setup for a multi-site school system deployment. I was using local gateways at every site to hook up fax lines and fire alarms with FXS/FXO ports for those systems to dial out. Everything else got backhauled to a voice gateway at the high school with a PRI running MGCP.

I was trying to figure out why the station IDs that were being send by the sites weren’t going out over caller ID. Everything was showing up as the high school number. I needed to figure out what was being sent. I was at the middle school location across town and trying to debug via telnet. I logged into the router and figured I would make a change, dial my cell phone from the VoIP phone next to me, and see what happened. Simple troubleshooting, right?

I did just that. My cell phone got the wrong caller ID. And my debug command didn’t show me anything. So I figured I must be debugging the wrong thing. Not debug isdn q931 after all. Maybe it’s a problem with MGCP. I’ll check that. But I just need to make sure I’m getting everything. I’ll just debug it all.

debug mgcp packet detail

Can You Hear Me Now?

Veterans of voice are probably screaming at me right now. For those who don’t know, debug anything detail generates a ton of messages. And I’m not consoled into the router. I’m remote. And I didn’t realize how big of a problem that was until my console started scrolling at 100 miles an hour. And then froze.

Turns out, when you overwhelm a router CPU with debug messages, it shuts off the telnet window. It also shuts off the console as well, but I wouldn’t have known that because I was way far way from that port. But I did starting hearing people down the hall saying, “Hello? Hey, are you still there? Weird, it just went dead.”

Guess what else a router isn’t doing when it’s processing a tidal wave of debug messages? It’s not processing calls. At all. For five school sites. I looked down at my watch. It was 2:00pm. That’s bad. Elementary schools get a ton of phone calls within the last hour of being in session. Parents calling to tell kids to wait in a pickup line or ride a certain bus home. Parents wanting to check kids out early. All kinds of things. That need phones.

I raced out of my back room. I acknowledged the receptionists comment about the phones not working. I jumped in my car and raced across town to the high school. I managed not to break any speed limits, but I also didn’t loiter one bit. I jumped out of my car and raced into the building. The look on my face must have warded off any comments about phone system issues because no one stopped me before I got to the physical location of the voice gateway.

I knew things were bad. I didn’t have time to console in and remove the debug command. I did what ever good CCIE has been taught since the beginning of time when they need to remove a bad configuration that broke their entire lab.

I pulled the power cable and cycled the whole thing.

I was already neck deep in it. It would have taken me at least five minutes to get my laptop ready and consoled in. In hindsight, that would have been five wasted minutes since the MGCP debugger would have locked out the console anyway. As the router was coming back up, I nervously looked at the terminal screen for a login prompt. Now that the debugger wasn’t running, everything looked normal. I waiting impatiently for the MGCP process to register with CallManager once more.

I kept repeating the same status CLI command while I refreshed the gateway page in CallManager over and over. After a few more tense minutes, everything was back to normal. I picked up a phone next to the rack and dialed my cell phone. It rang. I was happy. I walked back to the main high school office and told them that everything was back to normal.

Tom’s Take

My post-mortem was simple. I did dumb things. I shouldn’t have debugged remotely. I shouldn’t have used the detail keyword for something so simple. In fact, watching my screen fill up with five sites worth of phone calls in a fraction of a second told me there was too much going on behind the scenes for me to comprehend anyway.

That was the last time I ever debugged anything in detail. I made sure from that point forward to start out small and then build from there to find my answers. I also made sure that I did all my debugging from the console and not a remote access window. And the next couple of times I did it were always outside of production hours with a hand ready to yank the power cable just in case.

I didn’t save the day. At best, all I did was cover for my mistake. If it had been a support call center or a hospital I probably would have been fired for it. I made a bad decision and managed to get back to operational without costing money or safety.

Remember when you’re doing your job that you need to keep an eye on how your job will affect everything else. We don’t troubleshoot in a vacuum after all.

by networkingnerd at June 29, 2018 04:59 PM

XKCD Comics

June 28, 2018

Network Design and Architecture

3 people passed CCDE Practical/Lab exam on February 2018 from my class

3 people sent me their feedbacks after they pass February 2018 CCDE Practical/Lab exam.     Exam result was announced on May 2018 (2 to 3 months after according to new CCDE Practical exam result policy) but I couldn’t find a chance to share their results on the website.   I can’t count anymore how […]

The post 3 people passed CCDE Practical/Lab exam on February 2018 from my class appeared first on Cisco Network Design and Architecture | CCDE Bootcamp | orhanergun.net.

by Orhan Ergun at June 28, 2018 05:21 PM

My Etherealmind
ipSpace.net Blog (Ivan Pepelnjak)

Network Infrastructure as Code in Network Automation Online Course

In mid-May, I ran an onsite network automation workshop, and the manager organizing the workshop for his team invited me to a dinner with his peers. Not surprisingly, they wanted to hear about the topics covered in the workshop, and as soon as I mentioned Network-Infrastructure-as-Code several of them said “yes, that definitely needs to be covered.

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at June 28, 2018 06:53 AM

June 27, 2018

Ethan Banks on Technology

I Didn’t Even Know I Was Sick

This piece was originally published in the Packet Pushers’ Human Infrastructure Magazine, a publication about the human side of working in technology. HIM is sent every other week or so to Packet Pushers Ignition members. Sign up for free.

I recently tweeted…

I’ve become okay with only having so much time in my schedule. Would adding this { new | random | unexpected } thing to the mix stress me out? Yes? Then I can’t do it. Have to leave some space. Have to execute well on the things already on the list.

I grabbed a couple of replies that especially impacted me.

Cutting Things Loose Has A Cost

The hard part for me is deciding when to cut things loose in order to make room for new things that are more valuable. Sometimes it’s natural, like a job transition, but most of the time it’s not. I’d rather make intentional choices, not wait until I’m burned out. Of course, often the major problem with intentionally stopping a project is the social cost. Disappointing people is expensive for multiple reasons. And it’s very difficult to weigh that against the benefit of doing something new.


Benson crammed a whole lot of value into that reply that had me nodding my head in agreement and reflecting.

Cutting things loose. Identifying what’s most valuable is difficult. For me, the key is distinguishing what’s valuable to me versus what’s valuable to someone else. In my work, I’ve found that other people are happy for me do things for them that might not be the best use of my time.

My best work is found at the junction of projects I am uniquely capable of accomplishing and projects with the most substantial impact to others. The rest can be outsourced or cut.

Managers often use the desks of competent employees as a dumping ground. Respectfully pushing back can help. You’ll drown in an ocean of trivial tasks if you never speak up.

Not waiting for burn out. I recently realized that I’ve worked most of my career in a state of burnout. I’ve usually had one full-time job with an employer and one part-time job working for myself.

Technology work is stressful. Technology often works poorly, impacting businesses negatively. Add certifications to the mix, and it’s a formula for burn out. I’ve cycled through this formula repeatedly.

I don’t willingly live that way now. Although Packet Pushers has been my full-time job since 2015, travel became the side job. Since recently curtailing travel, I’m no longer constantly fretful or exhausted, i.e. burned out.

I didn’t realize I was living in this burned out state until I reduced my schedule and started feeling better. I was so used to feeling badly, that feeling good was a revelation. I didn’t even know I was sick.

Disappointing people comes with a cost. I am fearful that when I tell someone “no” or “no more” that I’ll alienate them. Sometimes, that’s exactly what happens. It’s a risk.

There is an upside to disappointing people, though. You get a manageable schedule. A manageable state of mind. A manageable life.

Being able to manage my life has become important. In recent cases where I’ve said “yes” to something I didn’t really want to, I regretted the time it took from my schedule and the energy it sapped from my life. I also resented the person that asked me to be involved. That’s not a productive mental state.

The Importance Of Margin

Margin is great as we go through different mountains and dips of life. Not always possible but I strive for it all the time.


Chris says it well. Margin. For the first time since being a college-bound teenager, I have had holes in my schedule where there was nothing I had nor needed to do.

Don’t misunderstand–my project list is full. I have over 100 tasks representing projects large and small. But there’s a difference between a task list that represents opportunity and one that represents obligation.

If I have the margin Chris strives for, I can chase opportunities. If I have no margin, my life is overfilled by obligations, leaving no space for the opportunities. By reducing the obligation load created by travel, I’ve capitalized on opportunities.

Where Does This End Up?

I’ll experience burnout in the future, I’m sure. I’d be lying to myself if I thought I was in such control of my life that I’d never again be overextended.

However, I’ve remembered what feeling good is like. I’m not eager to let that go.

by Ethan Banks at June 27, 2018 04:30 PM

XKCD Comics

June 26, 2018


Automation for Success

Businesses with high growth, complex tasks and repetition, tend to rely on or require automation to fulfill business challenges. Introducing automation is not without challenges and sometimes they can be quite significant. Identifying success is one of the early crucial activities that creates a business alignment. The identification exercise highlights justification for one more decisions and the removal of friction. Some of the decisions are not easy to make and friction is not easy to experience without applying pressure to various parts of an organization.

What follows is a number of scenarios with some reasoning around the kind of challenges that you’re already facing or likely to face.

If the absolute reasons are known, accepted and aligned against, you have just laid one of the foundational layers for success.

Challenge: High Growth

Great news and bad news. You’re in a business under stress from high growth! Lots of great challenges to solve and high pressure from not having them currently solved.

With high growth businesses, engineers or administrators are not under threat of being “automated out of their job”. If you identify as one of these people, you have an opportunity to learn new skills, be rewarded for finishing projects and also have opportunities to take part in a vibrant culture. This culture promotes sharing and education both internally to your business and externally at meet-ups and events.

This is always considered a golden era and memories of solving these challenges will remain with you for your entire career.

Processes are easy to implement because friction against acceptance is low. Trainings from Scrum and DevOps can be applied readily as the business is thirsty!

The most obvious danger here is being a naysayer and sticking ones head in the sand. You are mostly likely to be removed due to that act rather than being automated away.

Challenge: Repetition

Drudgery. Every day, week or month, you are repeating tasks and consume easy to provide data and are easy to validate with clear results. Some administrators and engineers like repetition! Small, byte (!) sized tasks provide closure and some mental reward. They can be started, executed and validated with some ease. The feeling is akin to being a small gear in a large clock. Mentally, it’s clear you’re part of something larger and the ease of the task provides mental safety. This is always true in environments facing high rates of technological change where new knowledge is required. Unfortunately for the “happiness in repetition” seekers, this area is a prime candidate for automation.

Simple repetitive tasks can be automated with high return yields when compared to the investment. As long as a source-of-truth exists so we can execute CRUD (Create, Read, Delete, Update) operations, a simple open or closed loop automated process offers a “build once, use many times” gain. If this was a financial investment, this would be a no brainer. If time gain is your goal, then these tasks are well worth looking at.

What do you if you do data entry or have mastered these repetitive tasks? It’s time to either do more of them using automation as your own accelerator, or find something else to do. One observation, if you’re in a high growth business with highly repetitive tasks, you’re not likely to be made redundant thanks to the business pressure of meeting growth! If you develop expertise to build these automations and maintain them (a largely disregarded area), you have a great opportunity. Ensure that your management team understand you are building and maintaining business knowledge through the power of workflows. One anecdote for hammering home the value here is that of pilots and automated planes. When the plane finds itself in a scenario that is unmanaged by automation, pilots that fly planes manually (think gliders and single engine aircraft) generally survive from the incident because they know what to do. Those who twist knobs and rely on the computer tend to not make it. It’s a horrid anecdote, but proven.

Challenge: Complexity

This is a composite challenge. These workflows return gains of reliability, repetition and remove a hero or two. Complex processes normally come with a high level of friction when trying to automate them. From engineers and administrators not being truthful about the process, data being harder to obtain and validate and the business gains not being completely clear. Again, a set of composite challenges.

If you are a hero right now, it’s time to be scared. You’ve put yourself in to an expensive position. The business will build flows around you and once that’s happened, it will be clear that you’re a bottleneck and need to be automated out of the way. High complexity challenges come with risks of low reliability when being done manually and once the merits of automation are clear, you’re a target. Your role could also be split up in to smaller less complex chunks.

Business gains

The hardest part of figuring out when something is worth automating or building. Business gains are always mostly hidden in politics. One departments saving is another departments demise. If one team is successful, the other team is fired for not being so.

In an ideal world, the business gains will be dictated from above, when accurate reporting and metrics of processes have been accurately reported from below. The directors and chiefs need to dictate what business gains are sought. A statement like “Automating stuff and winning more business” is clearly an uninformed one. “Saving on OpEX” is also terrible. It doesn’t tell us anything. How about these?

  • To remove syntax from service configuration
  • To remove variable collisions from service configuration
  • To triage data collection activities to speed up third line faults
  • To manage growth and predict PoP expansion to prevent order bottlenecks

These are clear and useable as planning heuristics for deciding what workflows to create. Notice how they’re not all just building configuration?

If a reduction in head count is the target, then what normally happens is an amalgamation of job function. Two or more job roles combine in to one and the number of levers a person operates increases. Event driven automation helps here so long as the correct signals can be formed in to events, which a system will react to and execute the workflow from the event trigger.

When organizations use automation to purely reduce headcount, it appears on the surface to help. When the business changes around the automated processes or worst case, the automation system fails, these approaches remind me of Humpty Dumpty and the job redundancy of the all the King’s horses and all the King’s men.

Ideal Challenge: Reliability

With NRE (Network Reliability Engineering), the bread and butter of the role is to increase reliability! Designing workflows with high reliability requires a “test driven development” approach for workflows which is similar to that of software.

Imagine a workflow that pulls a set of IP address from a source-of-truth for a direct internet access service. The workflow pulls a data structure containing all required variables, applies them to the various service templates and pushes each generated artifact (imagine a config file or some API call) to the correct service node via a control-plane. So far so good. We have the data, have generated the configs and pushed them out using a workflow that exits with code zero. When this happens, it means that our workflow was successful, not that the service is up and running. How can it? We only have visibility of our workflow’s exit code. Using reliability as the point of focus, we go deeper. Within the workflow, request information from the node with now modified state and validate it is what and where we expect it to be. That deals with the interface between the workflow and the network. Secondly, we must find a way to assert data from the actual service and not just the control-plane responsible for implementing it. Service based testing now happens, which tests the boundaries of the provided service. This could be hooking up a node pretending to be a customer at the PoP before hooking up the last mile in the case of direct internet access, or it could be bringing up a BGP peer and exchanging test traffic at an internet exchange. This not only validates the system is now in the mutated state, it also indicates reliably the chance of customer success when it comes to hooking them up!


Reliability is about removing hope. Not hope of things working, but hope from the equation of reliability. We want to be sure of things being reliable using instrumentation, metrics and statistics, not hoping that things work. If you’re going to test, do it at as many layers and points as possible. It’s better to know reliably than hope.

Reliability. Simples.

The post Automation for Success appeared first on ipengineer.net.

by David Gee at June 26, 2018 05:25 PM

ipSpace.net Blog (Ivan Pepelnjak)

Book: EVPN in Data Center

The EVPN in the Data Center book by Dinesh Dutt, the author of EVPN Technical Deep Dive webinar and member of ipSpace.net ExpertExpress team has finally been published. It’s currently kept safe behind Cumulus Networks regwall, but as O’Reilly published it, I would expect it to be available through other channels in the future.

by Ivan Pepelnjak (noreply@blogger.com) at June 26, 2018 05:57 AM

June 25, 2018

XKCD Comics

June 23, 2018

The Networking Nerd

Finding Value in Cisco Live 2018

The world famous Cisco Live Sign picture, 2018 edition

Another Cisco Live has come and gone. Overall it was a fun time for many. Catching up with friends. Meeting people for the first time. Enjoying the balmy Orlando weather. It was a chance to relive some great times for every one. But does Cisco Live 2018 dictate how the future of the event will go?

Packing The Schedule

Did you get a chance to attend any of the social events at Cisco Live? There were a ton. There were Tweetups and meet ups and special sessions galore. There was every opportunity to visit a lounge or area dedicated to social media presence, Boomerang videos, goofy pictures, or global outreach. Every twenty feet had something for you to do or some way for you to make an impact.

In fact, if you went to all of these things you probably didn’t have time for much else. Definitely not time for the four or five keynote addresses. Or a certification test. Or the classes and sessions. In fact, if you tried to do everything there was to do at Cisco Live, you’d probably not sleep the whole week. There’s almost as much stuff to do outside the conference sessions as there is to do in them.

But is it too much? Are the activities around the learning sessions taking away from the conference itself? Think about something like the Big Ideas theater this year. In theory, it’s a great way to get people to attend sessions that are not specifically related to tech. You can introduce new ideas, especially those that are focused more on changing the world. But you’re also competing for time away from sessions that are focused on new products or building better architectures.

Every booth in the World of Solutions is designed to draw you in and keep you there. For the sponsors of the event it’s important to have conversations about their products and solutions. For Cisco people, it’s almost like they’re competing with the sessions to give you different content or a chance to interview people. Is that how things should be? I can understand the desire of DevNet wanting to change the way people look at programmable networking, for example. But every other little booth like Cisco Advanced Services or the Emergency Response Vehicle? Those feel more like attractions designed to show off rather than educate.

Paying the Piper

And what does all this cost in the long run? Sure, I love having extra features around the conference as much as the next person. But to what end? Things don’t pay for themselves. Every conference has a budget. Every piece of entertainment and every showcase booth costs money in some way or another. And how does that all get paid for? By us, the attendees.

It’s no secret that attending conferences isn’t cheap. A full conference pass for Cisco Live is around $2,000. In the past, there were cheaper options for just attending for the people networking aspect of things. But, with the growth of DevNet and other “included” options at the conference, Cisco needed to find a way to pay for them this year.

I’m not going to spend a lot of time going into the Imagine Pass issue right now because I want to sit down and have an honest discussion with Cisco about the pros and cons of the approach. But it is very important that we examine what we’re getting for the increased cost. There has to be a significant value for people to want to be a part of the event if the costs go up. The way to do that is to create compelling reasons to want to be at Cisco Live.

The way not to do that is to lock the content behind gates. Some of the things at Cisco Live this year were placed in areas that were not easy to access. One of my personal pet peeves is the NetVet lounge. I’m going to start this off by saying that I was a NetVet for many years before I moved to Tech Field Day. I’m no longer a NetVet. However, until 2013 the NetVet lounge was one of the de facto social hangout places. Now, it’s another area where you can get coffee and snacks.

Why does the NetVet lounge bother me? Because of the placement. Front and center across the aisle from the on-site Cisco Store (which took the place of the Social Media hub from 2013). Why does the NetVet lounge get to be outside the World of Solutions? Aside from the historical reasons, I can’t think of a good reason. You need to have a full conference pass to achieve NetVet status. A full conference pass gets you into the World of Solutions. Why not have NetVets meet there?

The obvious reason is that the World of Solutions closes. Yet the NetVet lounge does too. And the hours are pretty similar. Why not move the NetVet lounge into the World of Solutions and give that space to the Social Media folks. There are no restrictions on getting into the Social Media Hub. Why not have them front and center? Again, aside from the “tradition” of having the NetVet lounge outside the World of Solutions I can’t think of a good reason.

Tom’s Take

I love Cisco Live. I realized this year that I’ve been to thirteen of them. Every year since 2006. The conference has changed and grown. The focus has shifted. But the people remain the same. With the changes in the way that the pass structure the people may not be there much longer. We, as IT professionals, need to decide what’s important and give some feedback. We need to make it constructive and honest. Point out what works and what doesn’t. Don’t whine, but offer direct criticism. We can only make the conference we want by telling the people what we need. That’s how you make Cisco Live a place to be for now and for the future.

by networkingnerd at June 23, 2018 04:47 AM

June 22, 2018

ipSpace.net Blog (Ivan Pepelnjak)

Time for a Summer Break

So many things have happened since I wrote “this is what we’re going to do in 2018” blog post. We ran

We also did a ton of webinars:

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at June 22, 2018 07:19 AM

Potaroo blog

Another 10 Years Later

The evolutionary path of any technology can often take strange and unanticipated turns and twists. At some points simplicity and minimalism can be replaced by complexity and ornamentation, while at other times a dramatic cut-through exposes the core concepts of the technology and removes layers of superfluous additions. The evolution of the Internet appears to be no exception and contains these same forms of unanticipated turns and twists. In thinking about the technology of the Internet over the last ten years, it appears that it’s been a very mixed story about what’s changed and what’s stayed the same.

June 22, 2018 05:00 AM

XKCD Comics

June 21, 2018


INE’s CCIE R&S v5 topology for EVE-NG

The last days I was working on adapting INE‘s lab topology, most specific the CCIE Routing and Switching v5 one, to be used in EVE-NG. In my opinion, INE offers some of the best training materials for Cisco and Juniper certifications. Along certification training you can find in their All Access Pass Subscription valuable learning … Continue reading INE’s CCIE R&S v5 topology for EVE-NG

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]

by Calin at June 21, 2018 09:15 PM

ipSpace.net Blog (Ivan Pepelnjak)

Upcoming Webinars and Events: Autumn 2018

On Tuesday I had the last webinar in spring 2018. One more online course session and it will be time for long summer break. In the meantime, we’re already planning the autumn events:

We also have the first webinars scheduled:

You can attend all these webinars with an ipSpace.net webinar subscription.

by Ivan Pepelnjak (noreply@blogger.com) at June 21, 2018 07:27 AM

Networking Now (Juniper Blog)

Windows Exploitation and AntiExploitation Evolution


Co-author:  Manoj Ahuje





Windows has been a target of hackers for a long time. The reason is simple: it is the OS with the largest deployment footprint around the world. Hackers discover vulnerabilities in Windows OS software as well as the various software that support Windows OS. Exploits as well as exploit mitigation techniques have evolved over the years. Over time, the targeted exploit has shifted from server applications to desktop applications. Adobe Flash has been a favorite target for the past seven to eight years. In this article we will talk about the exploits and their mitigation techniques on Windows.


Exploits are meant to compromise the system by taking advantage of a vulnerability in the program. Before moving ahead, let us first discuss how hackers find a vulnerability in an application.


Common vulnerability discovery techniques:

Fuzzing: This is a common way to test for vulnerabilities. The hackers provide a large range of inputs for the target software and observe for any unexpected behavior. For browsers, a lot of HTML files are created as an input. If the vulnerability has not been seen earlier, it is called a zero day vulnerability.


Patch diffing: This is a technique by which an update in software is compared with an old version. Windows used to update its software by means of DLLs. Hackers can compare the old and new DLL to find out what code is patched. After this, they can analyze it to find out if the old code is vulnerable. Hackers write the exploit to take advantage of the vulnerability assuming that not everybody would have applied the new software patch immediately. These kind of exploits are called one day exploits.



There are certain organizations that discover zero day vulnerabilities for commercial software testing. Hacking Team was one of these. Hacking Team exploits were leaked in 2015. Eternal Blue is also one of such infamous exploits that contributed to the spread of WannaCry and Petya malware



Types of Vulnerabilities and Windows countermeasures

Buffer Overflow vulnerabilities: These kinds of vulnerabilities occur when a program doesn’t check the boundary of user-supplied data. Stack overflow and heap overflow are the most common types of buffer overflow. Windows has patched almost all known and exploitable buffer overflow vulnerabilities and it is rare to see a new one discovered these days.

Other than patching the vulnerabilities in the software, Windows also developed some techniques to mitigate buffer overflows.


A Stack is a a very important data structure used in programming. It is used to store local variables and the function return address. Local variables can store user supplied data. As an example, HTTP server can store the HTTP request sent by the user in a variable. If the server does not check the size of the supplied HTTP request, it gets written to the stack where it can corrupt other data like the return address. The return address can be overwritten to redirect the control flow to a shellcode which is part of the user supplied variable value. A shellcode can pop a backdoor, launch a program or download another malware. Stack overflow exploitation works by overwriting the return address on the stack by a user supplied local variable.


Windows came up a with stack canaries to detect overwriting of the return address. The stack canaries feature was implemented in Windows XP SP2.  An exception was triggered when the return address was overwritten. Unfortunately, hackers bypassed this too using a technique called SEH Overwrite (Structured Exception Handler). The address of the exceptional handler needed to handle the return address overwrite exception was stored on the stack. Hence, the attacker was able to take control by overwriting this value.


Heap overflow also worked in a similar manner. User supplied data was able to corrupt important data structures on the heap to take control of the program. To track heap allocation, linked lists are used. The pointers used to connect node in the linked list are corrupted and overwritten by user controlled code.


Windows invented a technique called DEP (Data Execution Prevention) to prevent this kind of exploitation. The technique imposed a restriction that anything on the stack or heap should be treated as data and should not be executed.


                                                                                         DEP settings on windows






In order to bypass DEP, attackers created the shellcode by using codes in DLLs, which are supposed to execute. The shellcode is composed of addresses of instructions inside a DLL instead of only instructions. This technique is called ROP chain (return oriented programming). The ROP technique executes the code chunks that span across multiple DLLs to carry out the same functionality, which could have also been achieved by executing a contiguous chunk of shellcode placed in a heap or stack in absence of DEP. The code chunks from the DLLs used in ROP are called ROP gadgets. It was possible to create a ROP chain by using addresses inside a loaded DLL as the DLL addresses were fixed. To mitigate this ROP, Windows introduced another technique called ASLR (Address Space Layout Randomization). This technique randomized the base addresses of DLL with respect to the main executable each time the program started. This technique was quite effective with minimal flaws. The attacker used the non-randomized DLLs to create exploits. Attackers sometimes used another technique called information disclosure or information leak to find out the address of a DLL and then use the address in the exploit.


Now, stack and heap overflow are rarely seen in popular applications. It seems that most companies have done a good job on educating engineers about basic vulnerabilities. Use-After-Free has been the most recent popular vulnerability. It’s one of the most exploited vulnerabilities in the past five to six years.It was found in all browsers, Adobe Reader and Adobe Flash applications. Use-After-Free is a memory corruption vulnerability. The vulnerability is triggered when the program tries to access an object that has been freed. The object has been freed, but there was still a pointer that pointed to the memory location of the freed object. Trying to access the data pointed to the leads of the vulnerability. This kind of pointer is called a dangling pointer. Attackers can misuse the pointer to execute shellcode. Microsoft introduced a technique called isolated heap in mid-2014 to minimize use after free vulnerabilities. Isolated heap allocated separate heap for critical objects. The heap blocks are freed from user controlled data after the object is freed. Isolated heap helped to prevent exploitation of use-after-free vulnerabilities, but isolated heap was applied only to selective objects, not all objects. So, some objects might still be subject to risk. To further elevate the security, Microsoft added Protected Free or Deferred Free. In this technique, Microsoft does not free the object immediately. Instead, it frees it sometime later so that the attacker cannot predict the time when they can control the freed object.


Another very popular technique used to exploit browsers is heap spray. JavaScript used in browsers stores variables in the heap. The heap spray technique was used to fill the heap with lots of shellcode chunks. The advantage of this technique is that the attacker does not need to accurately predict the address of shellcode on heap. An address like 0xaaaaaaaa ,0xbbbbbbbbb is used, which most likely points to the heap and the shellcode would then probably lie around that. Chromium sandbox was one of Google Chrome’s innovations to counter all kinds of browser exploitation. Internet Explorer also came up with a similar solution.


But, still the DEP and ASLR was an issue. Hence, the novel technique JIT spray (Just In Time) was used. The technique was mostly used to exploit Flash-related vulnerabilities. JIT Engine is a native code generator, which is used by all modern browsers to speed up execution by parsing, optimizing and compiling bytecode to native code for the machine to run. The emitted code by JIT Engine is marked as executable in memory by default. This code can be sprayed into the heap by calculating the right size allocation of a page. After that, it’s a matter of jumping to a known heap address to get code execution bypassing DEP and ASLR all together. This bytecode can be generated in real time by JIT or it can be pre-generated and sent on the wire as a Flash/Java file on the internet. This technique was first used with Flash ActionScripts to spray heap, which used long XOR sequences to store shellcode and jumped onto a known heap address to get reliable code execution. After researchers were able to use the same JIT spray technique to exploit native JAVA and also used ASM.js recently for exploitation.


To keep up with the continuously evolving exploit landscape, Microsoft came up with EMET (Enhanced Mitigation Experience Toolkit) in 2009. It is an added layer of defense against code reuse attacks like ROP and provides better protection against Heap and JIT spray. Though it has to be manually installed by an administrator. Since EMET was not designed as an integral part of the OS, the exploit writers were able to bypass/disable EMET and achieve code execution in each of its versions. Though EMET led many security innovation in Windows product lines 7, 8 ,8.1, 10 and it’s Linux counterparts. As underlying OS changed, Microsoft decided to build in this security in OS as Windows Defender Exploit Guard, which will support future Windows versions after Windows 10, with many improvements. Microsoft decided to end life support for EMET. The latest version will be EOL’d on July 31, 2018.



Exploitation and Anti-Exploitation techniques is almost a cat and rat race. Security professionals always come up with an idea to combat Cyber criminals but at the same time criminals figure out a way to defeat it. Juniper Advanced threat detection product detects exploits to keep the customers safe.



by amohanta at June 21, 2018 04:06 AM

June 20, 2018

ipSpace.net Blog (Ivan Pepelnjak)

Worth Reading: Fake News in IT

Stumbled upon “Is Tech News Fake” article by Tom Nolle. Here’s the gist of his pretty verbose text:

When readers pay for news, they get news useful to readers.  When vendors pay, not only do the vendors get news they like, the rest of us get that same story.  It doesn’t mean that the story being told is a lie, but that it reflects the view of an interested party other than the reader.

High-quality content is not cheap, so always ask yourself: who’s paying for the content… and if it’s not you, you may be the product.

Full disclosure: ipSpace.net is funded exclusively with subscriptions and online courses. Some of our guest speakers work for networking vendors, but we always point that out, and never get paid for that.

by Ivan Pepelnjak (noreply@blogger.com) at June 20, 2018 09:20 AM

XKCD Comics

June 19, 2018

ipSpace.net Blog (Ivan Pepelnjak)

Presentation: Three Paths of Enterprise IT

During last week’s SIGS Technology Conference I had a keynote presentation about the three paths of enterprise IT.

Unfortunately, the event wasn’t recorded, but you can view the presentation here. Contact me if you have any questions, or Irena if you'd like to have a similar keynote for your event.

by Ivan Pepelnjak (noreply@blogger.com) at June 19, 2018 07:53 AM

June 18, 2018

My Etherealmind

Off to the Kubernetes – Networking in a Post VM world

Few people are using containers so why are all the vendors into it ?

by Greg Ferro at June 18, 2018 05:03 PM

The Networking Nerd

Conference Impostor Syndrome

In IT we’ve all heard of Impostor Syndrome by now. The feeling that you’re not just a lucky person that has no real skills or is skating by on the seat of their pants is a very real thing. I’ve felt it an many of my friends and fellow members of the community have felt it too. It’s easy to deal with when you have time to think or work on your own. However, when you take your show on the road it can creep up before you know it.

Conferences are a great place to meet people and learn about new ideas. It’s also a place where your ideas will be challenged and put on display. It’s not to difficult to imagine meeting a person for the first time at a place like Cisco Live or VMworld and not feeling little awe-inspired. After all, this could be a person whose works you’ve read for a long time. It could be a person you look up to or someone you would like to have mentor you.

For those in the position of being thrust into the limelight, it can be extremely difficult to push aside those feelings of Impostor Syndrome or even just a general level of anxiety. When people are coming up to you and thanking you for the content you create or even taking it to further extremes, like bowing for example, it can feel like you’re famous and admired for nothing at all.

What the members of the community have to realize is that these feelings are totally natural. You’re well within your rights to want to shy away from attention or be modest. This is doubly true for those of us that are introverts, which seems to happen in higher numbers in IT.

How can you fight these feelings?

Realize You Are Enough. I know it sounds silly to say it but you have to realize that you are enough. You are the person that does what they can every day to make the world a better place in every way you can. It might be something simple like tweeting about a problem you fixed. It may be as impressive as publishing your own network automation book. But you still have to stop and realize you are enough to accomplish your goals.

For those out there that want to tell their heroes and mentors in the community how awesome they are, remember that you’re also forcing them to look at themselves in a critical light sometimes. Some reassurances like, “I love the way you write” or “Your ability to keep the podcast going smoothly” are huge compliments that people appreciate. Because they represent skills that are honed and practiced.

Be The Best You That You Can Be. This one sounds harder than it might actually be. Now that you’ve admitted that you’re enough, you need to keep being the best person that you can be. Maybe that’s continuing to write great content. Maybe it’s something as simple as taking a hour out of your day to learn a new topic or interact with some new people on social media. It’s important that you take your skill set and use it to make things better overall for everyone.

For those out there that are amazed at the amount of content that someone can produce or the high technical quality of what they’re working on, remember that we’re all the same. We all have the same 24 hours in the day to do what we do. So the application of the time spent studying or learning about something is what separates leaders from the pack.

Build Up Others Slowly. This one is maybe the hardest of all. When you’re talking to people and building them up from nothing, you need to be sure to take your time in bringing them along. You can’t just swamp them with knowledge and minute details about their life that have gleaned from reading blogs or LinkedIn. You need, instead, to bring people along slowly and build them up from nothing into the greatest person that you know.

This works in reverse as well. Don’t walk up to someone and start listing off their requirements like a resume. Instead, give them some time to discuss it with you. Let the person you’re talking to dictate a portion of the conversation. Even though you may feel the need to overwhelm with information to justify the discussion you should let them come to their place when they are ready. That prevents the feeling of being overwhelmed and makes the conversation much, much easier.

Tom’s Take

It’s very easy to get lost in the world of feeling inadequate about what others think of you. It goes from adulation and excitement to an overwhelming sense of dread that you’re going to let people down. You have to fix that by realizing that you’re enough and doing the best you can with what you have. If you can say that emphatically about yourself then you are well on the way to ensuring that Conference Imposter Syndrome is something you won’t have to worry about.

by networkingnerd at June 18, 2018 11:53 AM

ipSpace.net Blog (Ivan Pepelnjak)

Vertical Integration Musings

One of my readers asked me a question that came up in his business strategy class:

Why did routers and switches end up being vertically integrated (the same person makes the hardware and the software)? Why didn't they go down the same horizontal path as compute (with Intel making chips, OEMs making systems and Microsoft providing the OS)? Why did this resemble the pre-Intel model of IBM, DEC, Sun…?

Simple answer: because nobody was interested in disaggregating them.

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at June 18, 2018 06:25 AM

XKCD Comics