December 14, 2018

The Networking Nerd

Some Random Thoughts From Security Field Day

I’m spending the week in some great company at Security Field Day with awesome people. They’re really making me think about security in some different ways. Between our conversations going to the presentations and the discussions we’re having after hours, I’m starting to see some things that I didn’t notice before.

  • Security is a hard thing to get into because it’s so different everywhere. Where everyone just sees one big security community, it is in fact a large collection of small communities. Thinking that there is just one security community would be much more like thinking enterprise networking, wireless networking, and service provider networking are the same space. They may all deal with packets flying across the wires but they are very different under the hood. Security is a lot of various communities with the name in common.
  • Security isn’t about tools. It’s not about software or hardware or a product you can buy. It’s about thinking differently. It’s about looking at the world through a different lens. How to protect something. How to attack something. How to figure all of that out. That’s not something you learn from a book or a course. It’s a way of adjusting your thinking to look at problems in a different way. It’s not unlike being in an escape room. Don’t look at the objects like you normally would. Instead, think about them with unique combinations that get you somewhere different than where you thought you needed to be.
  • Security is one of the only IT disciplines where failure is an acceptable outcome. If we can’t install a router or a wireless access point, it’s a bad job. However, in security if you fail to access something that should have been secured it was a success. That can lead to some very interesting situations that you can find yourself in. It’s important to realize that you also have to properly document your “failure” so people know what you tried to do to get there. Otherwise your success may just be a lack of proper failure.

Tom’s Take

I’m going to have some more thoughts from Security Field Day coming up another time. There’s just too much to digest at one time. Stay tuned for some more great discussions and highlights of my first real foray in the security community!

by networkingnerd at December 14, 2018 09:46 PM

My Etherealmind

Has Anyone Actually Sued Their Vendor ?

An oft-quoted reason for buying technology from a commercial vendor is legal recourse. If it fails, the company imagines that it could sue the supplier for negligence or reparations.

The post Has Anyone Actually Sued Their Vendor ? appeared first on EtherealMind.

by Greg Ferro at December 14, 2018 12:16 PM

Research: The Guidelines On Cyber Security Onboard Ships

This report is good reading and a useful template for writing your own IT security strategy.

The post Research: The Guidelines On Cyber Security Onboard Ships appeared first on EtherealMind.

by Greg Ferro at December 14, 2018 11:32 AM

Honest Networker

When overhearing “SDWAN will solve all your organisations edge-problems in a jiffy” for the third time in day.

LYEXTiz - Imgur

When overhearing “SDWAN will solve all your organisations edge-problems in a jiffy” for the third time in day.

by ohseuch4aeji4xar at December 14, 2018 11:28 AM

ipSpace.net Blog (Ivan Pepelnjak)

Zero-Touch Provisioning with Patrick Ogenstad

Zero-touch provisioning is always one of the big topics in the Building Network Automation Solutions online course, so we decided to invite Patrick Ogenstad (the author of excellent ZTP tutorial) to be a guest speaker in Spring 2019 course (register here).

In the meantime, enjoy his interview with Christoph Jaggi.

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at December 14, 2018 09:11 AM

XKCD Comics

December 13, 2018

ipSpace.net Blog (Ivan Pepelnjak)

Using Math in Networking on Software Gone Wild

We love to claim that we’re engineers and yet sometimes we have no clue how technology we use really works and what its limitations are… quite often because understanding those limitations would involve diving pretty deep into math (graphs, queuing and system reliability quickly come to mind).

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at December 13, 2018 07:54 AM

December 12, 2018

Honest Networker

When your co-worker starts to work as a SE for Cisco and sends the ACI sales-team your way.

cisco-se

When your co-worker starts to work as a SE for Cisco and sends the ACI sales-team your way.

Please come back firefly ;(

by ohseuch4aeji4xar at December 12, 2018 10:05 PM

Networking Now (Juniper Blog)

Juniper Networks and IoT Institute Survey: As IoT Deployments Collide with Multicloud Ecosystems, Where Does Security Stand?

According to Gartner, “by 2020, 75% of organizations will have deployed a multicloud or hybrid cloud model for their IT needs.”1 While we’ve known for some time that the future is multicloud, ensuring cybersecurity across diverse and non-traditional environments has mostly been an afterthought. Despite organizations weaving cloud-based ecosystems and Internet of Things (IoT) devices and applications into the fabric of their networks, they have not emphasized security at the same level. To learn more about what organizations are – and are not – doing to prepare and fortify their security postures, Juniper Networks partnered with the Internet of Things Institute to survey organizations implementing IoT projects.

by lpitt at December 12, 2018 02:00 PM

My Etherealmind

Video: Top 10 Misconceptions About 5G For Beginners

Breaks down the reality and practical considerations so that you can decode whether someone in marketing wrote those hypetastic 5G articles that you see everywhere.

The post Video: Top 10 Misconceptions About 5G For Beginners appeared first on EtherealMind.

by Greg Ferro at December 12, 2018 11:47 AM

Verizon Writes Down Oath by $4.8 Billion

And now they dream of selling SDWAN to Enterprises ? LOL

The post Verizon Writes Down Oath by $4.8 Billion appeared first on EtherealMind.

by Greg Ferro at December 12, 2018 11:03 AM

ipSpace.net Blog (Ivan Pepelnjak)

Can I Replace a Commercial Load Balancer with HAProxy?

A networking engineer attending the Building Next-Generation Data Centers online course sent me this question:

My client will migrate their data center, so they’re not interested in upgrading existing $vendor load balancers. Would HAProxy be a good alternative?

As you might be facing a similar challenge, here’s what I told him:

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at December 12, 2018 08:49 AM

XKCD Comics

December 11, 2018

My Etherealmind

Response: How Important is it to Understand Hardware Architecture?

This reddit post asked the question: For example, I’m going through a Cisco Live presentation on troubleshooting ASR routers, and the first 50 slides or so are completely dedicated to describing the Route Processors, Packet Engines, ASICs, Buffers, etc., and the different paths that packets can take through the hardware. While that’s all obviously important […]

The post Response: How Important is it to Understand Hardware Architecture? appeared first on EtherealMind.

by Greg Ferro at December 11, 2018 03:02 PM

Response: Super Micro says no implants on motherboards

I’m convinced that Bloomberg is wrong about the hardware implants. Until they retract the story they don’t have credibility to report on technology.  Supermicro commissioned 3rd party audit and found nothing which is confirmation of many other sources who also refute the claims. The ONLY people making the claim is Bloomberg and there is no […]

The post Response: Super Micro says no implants on motherboards appeared first on EtherealMind.

by Greg Ferro at December 11, 2018 02:52 PM

ipSpace.net Blog (Ivan Pepelnjak)

Building Your Own Virtual Lab

Last week we published Matt Oswalt’s thoughts on using virtual labs in training and testing. In the second part of his interview with Christoph Jaggi he talked about building a virtual lab.

Matt will cover the same topic in way more details in his guest speaker presentation in Spring 2019 Building Network Automation Solutions online course. Register here.

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at December 11, 2018 08:40 AM

About Networks

IT Blog Awards Finalist!

I have the honor to have my blog selected as a finalist in the Best Cert Study Journey category of the IT Blog Awards, hosted by Cisco. Voting is now open to everyone until January 4th, here!   Here, in my opinion, are my two best articles related to the category “Best Cert Study Journey”: My
Read More »

The post IT Blog Awards Finalist! appeared first on AboutNetworks.net.

by Jerome Tissieres at December 11, 2018 07:26 AM

December 10, 2018

ipSpace.net Blog (Ivan Pepelnjak)

Segment Routing Anyone?

One of my readers listened to a podcast where a $vendor described how they found another use case for source routing IPv6 segment routing (SR): 5G networks… and wondered whether SR made a comeback or is about to.

To figure out what segment routing is, watch the webinar we did with Jeff Tantsura a while ago.

I don’t know nearly enough about mobile networks to have an opinion, however…

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at December 10, 2018 08:17 AM

Honest Networker

ASR9000 fancy queuing strategy mechanics

IMG-4201

ASR9000 fancy queuing strategy mechanics

Credits: Aaron W.

by ohseuch4aeji4xar at December 10, 2018 07:40 AM

XKCD Comics

December 09, 2018

My Etherealmind

Lotus Bloody Notes/Domino Sold

For a time, I was the designated dumb person who managed a Lotus Bloody Notes deployment. This includes the server and desktop application plus a range of internal developed apps. But mostly it was an really poor attempt at an email solution.  The ‘bloody’ is my way of being polite  It was horrible. Everything about […]

The post Lotus Bloody Notes/Domino Sold appeared first on EtherealMind.

by Greg Ferro at December 09, 2018 03:47 PM

December 08, 2018

Network Design and Architecture

Routers Getting Routered – Silver Peak SD-WAN

Silver Peak SD-WAN. Routers Getting Routered – One of the Silver Peak’s Slogan for SD-WAN. First, let’s have a look at the video below  that the Slogan of “Routers Getting Routered” seems marketing, but actually it has a technical meaning behind it.     Video : Silver Peak Youtube Channel    I attended the last …

The post Routers Getting Routered – Silver Peak SD-WAN appeared first on Cisco Network Design and Architecture | CCDE Bootcamp | orhanergun.net.

by Orhan Ergun at December 08, 2018 02:15 PM

December 07, 2018

Network Design and Architecture

Hair-pining in a Wide Area Network – Simple Design Scenario

In this post we will discuss how hair-pining is occurring in some topologies in the Enterprise  branch sites, when connecting to 2 Service Providers, while the 2 Service Providers are not directly connected to each other and don’t have any MPLS/VPN Inter-AS Option (A,B,C,..) with each other. From the customer side, some of the remote …

The post Hair-pining in a Wide Area Network – Simple Design Scenario appeared first on Cisco Network Design and Architecture | CCDE Bootcamp | orhanergun.net.

by Orhan Ergun at December 07, 2018 03:10 PM

The Networking Nerd

It’s The Change Freeze Season

Everyone’s favorite time of the year is almost here! Is it because it’s the holiday season? Perhaps it’s the magic that happens at the end of the year? Or maybe, it’s because there’s an even better reason to get excited!

Change Freeze Season!

That’s right. Some of you reading this started jumping up and down like Buddy the Elf at the thought of having a change freeze. There’s something truly magical about laying down the law about not touching anything in the system until after the end-of-year reports are run and certified. For some, this means a total freeze of non-critical changes from the first of December all the way through the New Year until maybe even February. That’s a long time to have a frozen network? But why?

The Cold Shoulder

Change freezes are an easy thing to explain to the new admins. You simply don’t touch anything in the network during the freeze unless it’s broken. No tweaking. No experimenting. No improvements. Just critical break/fix changes only. There had better be a ticket. There should be someone yelling that something’s not right. Otherwise you’re in for it.

There are a ton of reasons for this. The first is something I remember from my VAR days as Boredom Repellent. When you find yourself at the end of year with nothing to do, you tend to get bored. After you’ve watched Die Hard for the fifteenth time this year you decide it’s time to clear out your project backlog. Or maybe you’ve been doing some learning modules instead. You find a great blog post from one of your favorite writers about a Great Awesome Amazing Feature That Will Save You Days Of Work If You Just Enable This One Simple Command!

In either case, the Boredom Repellent becomes like pheromones for problems. Those backlogged projects take more time than you expected. That simple feature you just need to enable isn’t so simple. It might even involve an entire code upgrade train to enable it. Pretty soon you find yourself buried in a CLI mess with people screaming about very real downtime. Now, instead of being bored you’re working until the wee hours of the night because of something you did.

The second reason for change freezes at the end of the year is management. You know, the people that call and scream at you as soon as their email appears to be running slow. The people that run reports once a month at 6:00pm and then call you because they get a funny warning message on their screen. Those folks. Guess what? End-of-year is their time to shine in all their glory.

This is usually the time they are under the most stress. Those reports have to be reprinted. All the financials from the year need to be consolidated and verified. The taxes will need to be paid. And all that paperwork and pressure adds up to stress. The kind of stress that makes any imperfections in the network seem ten times more important than before. Report screen not show success within 10 ms? Problem. Printer run out of yellow toner? Network problem. Laptop go to sleep while someone went to lunch and now the entire report is gone? Must be your problem. And guess who gets to work around the clock to solve it with someone bearing down on them from on high?

Don’t Let It Go

The fact is that we can’t have people doing things in the network without tracking those changes back to reasons. That applies for adventurous architects wanting to squeeze out the last ounce of performance from that amazing new switch. And it goes double for the CFO demanding you put his traffic into AF41 so it gets to the server faster so his reports don’t take six hours to print.

It all comes back to the simple fact that we have no way to track changes in our network and we have no way of knowing what will happen when we make one live. It feels an awful lot like this GIF:

Crazy, right? Yet every time we hit the Enter key, we are amazed at the results. Even for “modern” OSes with sanity checking, like Junos or IOS-XR, you have no way of knowing if a change you make on one device somewhere in the branch is going to crash OSPF or BGP for the entire organization. And even if there was a big loud warning popup that said, “ALERT: YOU ARE GOING TO BREAK EVERYTHING!!!”, odds are good we would just click past it.

Network automation and orchestration systems can prevent this. They can take the control of change management out of the hands of bored engineers and wrap it in process and policy. And if the policy says Change Freeze then that’s what you get. No changes. Likewise, if there is a critical need, like patching out a backdoor or something, that policy can be overridden and noted so that if there is a bug eight months from now in that code train that causes issues you can have documentation of the reason for the change when someone comes to chew you out.

Likewise, there are other solutions out there that try to prototype the entire network to figure out what will happen when you make a change. Companies like Forward Networks and Veriflow can prototype your network in a model that can assess the impact of a change before you commit to it. It’s the dream of a bored engineer because you can run simulations to your heart’s content to find out if two hours of code upgrades will really get you that 2% performance increase promised in that blog post. And for the CFO/CEO/CIO screaming at you to prioritize their traffic, these solutions can remind them that most of their traffic is Youtube and Spotify and having that at AF41 will cause massive issues for them.

What’s important is that you and the rest of the team realize that change freezes aren’t a solution to the problem of an unstable network. Instead, they are treating the symptoms that crop up from the underlying disease of the network not being a deterministic system. Unlike some other machines, networks run just fine at sub-optimal performance levels. You can make massive mistakes that will live in a network for years and never show their ugly face. That is, until you make a small change that upsets equilibrium and causes the whole system to fail, cascade style, and leave you holding the keyboard as it were.


Tom’s Take

I both love and hate Change Freeze season. I know it’s for the best because any changes that get made during this time will ultimately result in long hours at work undoing those changes. I also know that the temptation to experiment with things is very, very strong this time of year. But I feel like Change Freeze season will soon go the way of the aluminum Christmas tree when we get change management and deterministic network modeling systems in place to verify changes on a system-wide basis and not just sanity checking configs at a device level. Tracking, prototyping, and verification will solve our change freeze problems eventually. And that will make it the most wonderful time of the year all year long.

by networkingnerd at December 07, 2018 02:31 PM

ipSpace.net Blog (Ivan Pepelnjak)

Using Virtual Labs When Developing Network Automation Solutions

One of the fundamentals I always emphasize in introductory parts of my network automation workshops and online courses is the fact that we’re about to develop software that will control the most-mission-critical part of IT infrastructure, and should therefore use software development methodologies like version control, testing…

However, there’s a “small” glitch. While it’s perfectly possible to test most software in some virtual environment you can spin up on-the-fly using Vagrant, Docker, Jenkins, Travis, or some other CI/CD tool, testing a network automation solution requires access to network devices.

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at December 07, 2018 07:57 AM

XKCD Comics

December 06, 2018

Honest Networker

Leaking your Tier1 upstream routes to your Tier2 upstream that doesn’t do inbound filtering.

ozhm8ocq09221

Leaking your Tier1 upstream routes to your Tier2 upstream that doesn’t do inbound filtering.

by ohseuch4aeji4xar at December 06, 2018 07:14 PM

Dyn Research (Was Renesys Blog)

Last Month in Internet Intelligence: November 2018

In November, we saw fewer significant Internet disruptions in the Oracle Internet Intelligence Map as compared to prior months. As usual, there were hundreds of brief issues with limited impact and generally unknown causes, but the most notable issues last month were due to reported DDoS attacks, problems with terrestrial and submarine cables, and general network issues.

DDoS Attacks

On November 4 and 5, several Cambodian ISPs were targeted by DDoS attacks described as the “biggest attacks in Cambodian history.” Published reports indicated that ISPs including EZECOM, SINET, Telcotech, and Digi were targeted by DDoS attacks totaling nearly 150 Gbps, causing subscriber downtime lasting as much as half a day. Disruption from the attacks was visible in the Country Statistics view for Cambodia in the Internet Intelligence Map, as shown in the figure below. However, because Internet connectivity remained generally available (albeit impaired) across the country, the impact appears nominal in the graphs.

However, when viewed at a network level, the impact of the attacks appears to be more significant. SINET, one of the ISPs targeted by the DDoS attacks, posted a Tweet on November 5 letting users know that they were under attack, and followed up the next day with a longer explanation of what occurred, and the mitigation steps that were taken.

<script async="async" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>

<script async="async" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>

The Traffic Shifts figure below for SINET shows the DDoS attacks beginning to impact the network late in the day (GMT) on November 4, resulting in a significant decline in the number of completed traceroutes to endpoints in the network, along with a corresponding increase in latency. The problems lasted for approximately half a day before the metrics stabilized, though with fewer completed traceroutes at slightly higher combined latency.

Cable Concerns

On November 13, a brief disruption in Internet connectivity in the U.S. Virgin Islands was visible in the Oracle Internet Intelligence Map, as shown in the figure below. Subsequent research revealed that the observed issue was likely related to a fiber cut experienced by local ISP Viya. According to updates posted on the company’s website and Facebook page, “Viya’s fiber on St. Croix was accidentally cut by the Virgin Islands Water and Power Authority. As a result of the damage to the network, customers throughout St. Croix do not have internet, long distance and Cable TV service.” The impact of this fiber cut on Viya’s network can be seen in the Traffic Shifts figure below, with a clear drop in completed traceroutes to endpoints in the network and an associated increase in combined latency.

In April 2018, we saw damage to the ACE (Africa Coast to Europe) Submarine Cableimpact Internet connectivity for ten connected countries. On November 18, another issue with the cable once again disrupted connectivity across a number of countries connected to it. A Sierra Leone network provider reported that the issue originated at the landing station in Lisbon, Portugal.

<script async="async" charset="utf-8" src="https://platform.twitter.com/widgets.js"></script>

The figure below shows a matrix of eight countries connected to the ACE cable for which the Country Statistics views exhibited impacts from the reported technical failure. (The graphs are organized by distance from the Lisbon landing station where the problem reportedly occurred, with the closest appearing first.)

It is interesting to note that there is a range of impacts seen in the measurements, and that the observed disruptions don’t strictly align with (cable) distance from Lisbon or the number of submarine cable connections a given country has. The first six countries shown count the ACE cable as their sole submarine cable provider, while Benin connects to two cables, and Equatorial Guinea three, according to Telegeography’s Submarine Cable Map.

It is also interesting to observe how the ACE cable issue impacted individual network providers in the affected countries. While multiple examples were captured, there was a noticeable difference seen between two network providers in The Gambia. For AS25250 (Gamtel), the cable disruption caused a clear decline in the number of completed traceroutes to endpoints in the network, while combined latency spiked to approximately 3x normal levels. However, for AS37309 (QCell Limited) during the disruption period, it appears that traceroutes took alternate paths through Telecom Italia Sparkle and Orange when the path through Portugal Telecom became unavailable, with no significant impact to latency. While both networks have multiple upstream providers/peers, only one appeared to benefit from the redundancy.

In addition to the disruption discussed above, planned maintenance on the ACE cable was scheduled to take place between November 23-28, according to a Tweet from @Gamtel. The Traffic Shifts graphs in the figure below for AS25250 (Gamtel) show increases in combined latency on multiple days during the maintenance period, as well as an increase in the number of traceroutes reaching Gamtel through Sonatel when the path through Portugal Telecom became unavailable. In this case, the redundant connections appeared to minimize the disruption to Gamtel’s Internet connectivity.

Network Issues

Early in the morning (GMT) of November 2, the Cogent Communications Status Pagereported an issue on their network due to a double fiber cut, impacting regions including México City, Querétaro, and Guadalajara. While this outage was barely evident in the metrics shown for Mexico in the Country Statistics view of the Internet Intelligence Map, it was much more obvious in the Traffic Shifts graphs for Mexican network providers that have upstream connectivity from Cogent, as shown in the figures below. For AS14178 (MCM Telecom), traceroutes to endpoints within the network arrived via Level 3 (and PCCW Global to a lesser extent) during the Cogent outage, while at AS6503 (Axtel), Level 3 also picked up the slack, although the drop in traceroutes through Cogent was not as severe as was seen at MCM Telecom. The Cogent outage also resulted in nominal increases in latency for traceroutes to both networks.

Late in November, Internet disruptions were observed in both Burkina Faso and Somalia, as shown in the Country Statistics graphs in the figures below. While neither was a complete disruption in connectivity, further research showed a common point of failure.

In reviewing Traffic Shifts graphs for network providers in the two countries, we observed that the West Indian Ocean Cable Company (AS37662) [WIOCC] was the common chokepoint – that is, the issues seen were in networks downstream of WIOCC. As shown in the figures below, a disruption is clearly visible in AS37073 (Sahel Finance Burkina) on November 25, which is downstream of AS328316 (Point d’Atterissement Virtuel – Burkina Faso). The latter network relies on connectivity from WIOCC, and a coincident disruption is visible here as well. However, no disruption is evident when looking at the Traffic Shifts graph for WIOCC, indicating that the issue likely occurred in the connectivity between AS37662 and AS328316 and not upstream from WIOCC, although it isn’t clear if the root cause is related to some sort of configuration issue, or if it was due to physical infrastructure failure.

Similarly, the majority of traceroutes to Somali networks AS37371 (Hormuud Telecom Somalia) and AS327828 (Somali Optical Networks) (among others) also pass through WIOCC. As noted above, no disruptions were observed upstream of WIOCC, indicating that the issue likely occurred in the connectivity between it and these downstream networks.

Conclusion

Internet connectivity relies on a multi-tiered architecture: the control plane, the data plane, and the physical plane. We passively observe the control plane (collecting BGP updates), and actively probe/measure the data plane (through traceroutes and DNS usage data), but the physical plane (cables, satellites, power grid, etc.) remains largely invisible as it cannot be probed from afar. For those Internet disruptions that are not explained by “ground truth” (such as a Tweet, Facebook post, published article, status page update, etc.), we are forced to make educated inferences as to their cause, based on historical experience, factors including local geopolitical or weather activity, and other disruptions seen nearby and/or at the same time.

by David Belson at December 06, 2018 03:52 PM

ipSpace.net Blog (Ivan Pepelnjak)

Video: What Problem Are We Solving with SDDC?

Remember the Software-Defined Data Centers hype? While I covered SDDC concepts and technologies for years in my webinars and workshops, I never created an introductory webinar on the topic.

That omission has been fixed in late August – SDDC 101 webinar is available as part of free subscription, and as always I started with the seemingly simple question: What problem are we trying to solve?

by Ivan Pepelnjak (noreply@blogger.com) at December 06, 2018 08:07 AM

December 05, 2018

My Etherealmind

Thought: Face-to-face is effective, No One Talks About How much it costs

This HBR article from April 2017 highlights a researcher talking about their paper that face-to-face is more effective than email. Well, duh. However, prior to making their requests, we asked participants in each condition to predict how many of the 10 strangers they asked would agree to fill out the survey. Participants in the face-to-face […]

The post Thought: Face-to-face is effective, No One Talks About How much it costs appeared first on EtherealMind.

by Greg Ferro at December 05, 2018 06:00 PM

ipSpace.net Blog (Ivan Pepelnjak)

Odd Number of Spines in Leaf-and-Spine Fabrics

In the market overview section of the introductory part of data center fabric architectures webinar I made a recommendation to use larger number of fixed-configuration spine switches instead of two chassis-based spines when building a medium-sized leaf-and-spine fabric, and explained the reasoning behind it (increased availability, reduced impact of spine failure).

One of the attendees wondered about the “right” number of spine switches – does it has to be four, or could you have three or five spines. In his words:

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at December 05, 2018 07:37 AM

SNOsoft Research Team

The reality behind hospital and medical device security.

We recently presented at the DeviceTalks conference in Boston Ma about the vulnerabilities that affect hospitals and medical devices (insulin pumps, pacemakers, etc.).  The goal of our presentation wasn’t to instill fear but sometimes fear is a reasonable byproduct of the truth.  The truth is that of all the networks that we test, hospital networks are by far the easiest to breach.  Even more frightening is that the medical devices contained within hospital networks are equally if not more vulnerable than the networks that they are connected to.   It seems that the healthcare industry has spent so much time focusing on safety that they’ve all but lost sight of security.

The culprit behind this insecurity is mostly convenience.  Hospitals are generally run by healthcare experts with a limited understanding of Information Technology and an even more limited understanding of IT Security. It would be unreasonable to expect healthcare experts to also be IT security experts given the vast differences between both fields. When healthcare experts hire IT experts and IT Security experts they do it to support the needs of the hospital.  Those needs are defined by the doctors, nurses, and other medical professionals tasked with running the hospital.  Anything that introduces new complexity or significant changes will be slow to adopt or perhaps not adopted at all.  Unfortunately, good security is the antithesis of convenience and so good security often falls by the wayside despite best efforts by IT and security personnel.

Unfortunately, in many respects the IT security industry is making the situation worse with false advertising.  If antivirus solutions worked as well as they are advertised, then malware would be a thing of the past.  If Intrusion  Prevention Solutions worked as well as advertised, then intrusions would be a thing of the past.  This misrepresentation of the capabilities provided by security solutions produces a false sense of security.  We aren’t suggesting that these solutions are useless, but we are encouraging organizations to carefully test the performance and effectiveness of these solutions rather than simply trusting the word of the vendor.

After we breach a network there exists a 30-minute window of susceptibility to ejection from the network. Most malicious hackers have a similar or larger window of susceptibility.  If a breach is responded to within that window, then we will likely lose access to the network and be back to square one (successful damage prevention by the client).   If we are not detected before that window expires, then the chance of successful ejection from the network is close to zero.  Astonishingly, the average length of time it takes for most organizations to identify a breach is 191 days.  Rather than focusing on breach prevention (which is an impossibility) organizations should be focusing on breach detection and effective incident response (which is entirely attainable).  An effective incident response will prevent damage.

Within about 40 minutes of breaching a hospital network our team takes inventory.  This process involves identifying systems that are network connected and placing them into one of two categories.  Those are the medical device category and the IT systems category.  Contained within the IT systems category are things like domain controllers, switches, routers, firewalls and desktops.  Contained within the medical device category are things like imaging systems, computers used to program pacemakers, insulin pumps etc.  On average the systems in the medical device category run antiquated software and are easier to take control of than the IT devices.  This is where security and safety intersect and become synonymous.

These medical device vulnerabilities afford attackers the ability to alter the operation of life-critical systems.  More candidly, computer attackers can kill patients that depend on medical devices.  The reality of medical device vulnerability is nothing new and it doesn’t seem to be getting any better. This is clearly evidenced by the ever-increasing number of medical device recalls triggered by discovered cybersecurity vulnerabilities. These vulnerabilities exist because the security of the software being deployed on medical devices is not sufficiently robust to safeguard the lives of the patients that rely on them.

More frightening is that attackers don’t need to breach hospital networks to attack medical devices.  They can attack medical devices such as implants from afar using a laptop and a wireless antenna.  This was first demonstrated in 2011 by security researcher Barnaby Jack.   He proved the ability to wirelessly attack an insulin pump from a distance of 90 meters causing it to repeatedly deliver its maximum dose of 25 units until its full reservoir of 300 units was depleted.  In simple terms Barnaby demonstrated how easily an attacker could kill someone with a keyboard and make it look like a malfunction.  He also did the same thing with a pacemaker causing it to deliver a lethal 840-volt shock to its user.  Similar attacks are still viable today and affect a wide variety of life supporting devices.

To solve this problem two things needs to happen.  The first is that medical device manufacturers need to begin taking responsibility for the security of their devices.  They need to recognize that security is in many cases a fundamental requirement of safety.  They also need to begin taking a proactive approach to security rather than reactive.  In our experience medical device manufacturers are unfriendly when interfacing with vulnerability researchers.  They might want to reconsider and even offer bug bounties as a step in the right direction.

Hospitals also need to make some significant changes too.  They need to begin to put security above convenience when it has the potential to impact patient safety.  This might mean installing good password managers and enforcing strong passwords with two factor authentications, increasing security budgets, or even paying for good security training programs.  Most hospitals are patient safety focused but fail to recognize that IT security and patient safety are now synonymous.

The post The reality behind hospital and medical device security. appeared first on Netragard.

by Adriel Desautels at December 05, 2018 01:45 AM

XKCD Comics
Keeping It Classless

December 4 - NRE Labs Outage Post-Mortem

I awoke yesterday to a very crisp Tuesday morning in Portland, Oregon. I had just poured myself a nice glass of Stumptown Nitro cold brew coffee, and wandered upstairs to my office for an 8AM conference call. I joined the meeting, and started going through my usual routine - part of which includes looking at the day’s NRE Labs stats. Here’s what I saw: Well, that kind of sucks.

December 05, 2018 12:00 AM

December 04, 2018

Honest Networker
Networking Now (Juniper Blog)

JATP Appliances Deliver One-Touch Solution for Building a Stronger Security Posture

As 2018 comes to a close, it’s clear we need continued security innovations to keep our organizations better protected from cybercrime. Bad actors with limitless resources are hyper-innovative in finding and exploiting vulnerabilities. Meanwhile, our vulnerabilities are expanding with cloud environments and distributed, digital business models. We need new ideas to keep our security postures strong. Attacks like cryptojacking and new variants of ransomware are using revolutionary obfuscation tactics and automation to bypass traditional security solutions. The ongoing shortage of qualified security experts, combined with a flood of data from multiple security vendors and connections to multiple clouds, are driving the need for faster, more flexible and automated cybersecurity solutions.

 

by Amy James at December 04, 2018 03:06 PM

My Etherealmind

Virtual Design Clinic 3 at Packet Pushers

Attending conferences has always been expensive, time consuming and tiring. Most people don’t talk about this but prefer to highlight the positives of meeting people, sitting in high quality sessions and enjoy some time off from the repetitive activities of the $dayjob. If you have questions that you want answered but can’t attend, head over […]

The post Virtual Design Clinic 3 at Packet Pushers appeared first on EtherealMind.

by Greg Ferro at December 04, 2018 02:06 PM

Honest Networker
In Search of Tech

A Quick Guide To Configuring An Aerohive AP

Whether through a purchase or demo gear, you have your hands on an Aerohive AP. If you have never dealt with Aerohive before from a wireless perspective, you might be asking yourself how to configure the AP. This post is meant to serve as a starting point to take that AP and put some configuration on it so that you can start connecting clients.

This post is not meant to be an exhaustive reference, as there are MANY things that can be configured. Rather, it is meant to be a starting point in which the reader is introduced to the overall configuration mechanisms within HiveManager at a high level. At the time of publishing(December 2018), these are the steps needed to put in AP into operation. However, the nature of the rapid pace at which HiveManager is updated means that things could change that alter the configuration flow. It is also worth noting that this post utilizes the public cloud/Internet option for HiveManager. Although there are 3 different deployment options for HiveManager(public cloud, private cloud, single VM), this post utilizes the public cloud option. Configuration between the 3 different models is relatively the same.

Before You Plug That AP In

If you have experience with wireless vendors that use local forwarding from each AP, then you are already accustomed to having APs connected to switch ports configured as 802.1q links/trunks. If your only experience has been with controllers that were only configured with centralized forwarding, an Aerohive deployment is going to look a lot different as you will move from using access or single VLAN switch ports to using 802.1q trunk ports. Should you have some sort of requirement to tunnel traffic back to a central location for things like guest access, fear not. The Aerohive solution allows you to pick and choose which users need to be tunneled over the network to a particular location. Notice that it is particular users and not an entire SSID. The flexibility of the Aerohive solution allows to you be very granular with which traffic is tunneled and which traffic is switched locally.

You probably already have VLANs set aside for your various types of traffic. If you don’t already have one, you will want to create another VLAN for management purposes. Each Aerohive AP will have its own management IP, which will be used to talk to HiveManager, as well as any other Aerohive APs on the network. The management IP will also be used to make connections to RADIUS servers and any additional management systems for things like SNMP and syslog.

Plugging The AP In

Once you plug an Aerohive AP into your network, the AP is going to “phone home”. It can utilize a number of mechanisms to find where it needs to go in the same way that a controller based AP will. The typical method is going to be DNS. By default, the AP will come online, grab a DHCP lease, and try to connect to redirector.aerohive.com. In order to do that, it is going to have to have access to DNS servers, which will almost always be provided via the DHCP lease. However, in addition to DNS, the AP will attempt to set its internal clock with NTP. In order to make a secure connection to the redirector site or ultimately HiveManager, the AP will need to have its system clock set to a fairly current time so that the certificate on the Internet side of the connection is viewed as valid. There are certain defaults the AP can and will use for DNS and NTP, but if the outbound firewall blocks the ports for DNS and NTP, there is a chance the AP will fail to connect. There are ways to manually intervene via a console or SSH connection to the AP, but one of the benefits of cloud networking is making connections work right out of the box without having to perform staging operations on hardware. You should be able to just ship it to the desired location and plug it in.

Assuming the AP is plugged in, gets a DHCP lease, sets its timing, and can perform DNS resolution, it will connect to redirector.aerohive.com. It will sit there until you add the AP to HiveManager using its serial number. We’ll cover that aspect shortly. Once the serial number has been added to your virtual hive, the redirector system will then tell the AP where it should move to for management purposes. This all happens in the background once you add a device serial number to HiveManager.

You can plug in the AP right away, or plug it in after you have built out the initial configuration. You’ll know you have a secure connection to redirector or HiveManager if you see a white light displayed on the AP. Some models, like the wall plate AP(model 150W) will use a green light instead of a white light. Generally speaking, if you see anything other than a red/amber light, consider the device properly connected.

Initial HiveManager Setup

HiveManager is meant to be a self-service operation. As long as you have a license key(in lieu of this, you can trial HiveManager for 30 days with full feature functionality), you can provision your own environment without having to wait on Aerohive to build out your environment for you. This brings up another point. There are two flavors of HiveManager. The most common is referred to as HiveManager Select. This option requires an entitlement key for each of your devices(e.g. AP, switch, router). The second option is referred to as HiveManager Connect. With Connect, the management piece is free. All you have to buy is the hardware. The difference between Connect and Select is that with Connect, you have far fewer configuration and management options. You also will have to use the forums on the Aerohive HiveCommunity site for support. There are options to have phone support with HiveManager Connect if desired. With HiveManager Select, you have ALL options available in HiveManager as well as 24×7 web and phone support.

The first thing you want to do is open a web browser connection to https://cloud.aerohive.com. This is the site you will access HiveManager from for all future connections after registration, so you will want to bookmark or memorize the URL. Since you probably don’t have a login, you will want to click on the “REGISTER” link on the page. Fill in the appropriate information on the registration page. The only field you don’t have to fill out is the one that relates to sending a copy of the registration to a partner or Aerohive representative.


 

 

 

 

 

 

 

 

 

 

 

 

 

 

Once you click on “REGISTER”, you will receive an e-mail within a few minutes. Within that e-mail, there will be a link to set a password for the account you created.

If there are going to be multiple administrators within your HiveManager environment, only one person needs to fill out the initial registration information. You can add additional accounts once your virtual hive(VHM) is created through a much easier process. It is worth noting that once you click on “REGISTER”, another screen will appear with the following verbiage as well as an embedded HiveManager onboarding video:

 

 

 

 

 

 

 

 

 

 

 

 

This video will give you a general overview of the onboarding process as well as a basic overview of HiveManager. It is worthwhile to watch it if you are new to Aerohive, and in case you miss the link or just want to watch the video without going through the registration process, it is linked below:

<iframe allowfullscreen="allowfullscreen" frameborder="0" height="315" src="https://www.youtube.com/embed/FSWJD47WU2Y" width="560"></iframe>

Your password has been set, and now you see the following screen:

I am going to use the default 30 day trial option, but if you have an entitlement key, you can enter it here as well. Once I click on the “Get Started!” button in the bottom right, the following screen appears:

 

 

 

 

 

Click on the “Onboard Devices” button to get rid of this screen. You won’t see this screen again, and it is better if we learn how to configure things in the way in which we will normally do so within HiveManager. You should be looking at the “MONITOR” tab within HiveManager now. We’ll come back to this page in a bit. First though, we want to build an initial network policy, so click on the “CONFIGURE” tab at the top of the screen.

You should now see the following:

 

 

 

Note: The network policy is the container in which almost all configuration of a device is stored. Apart from device specific things like the hostname, almost everything you will do configuration-wise for all of your devices will live within the network policy. I’ll cover things like the network policy, user profile, and radio profiles in another post as each of those items require a much more detailed breakdown.

Click on “ADD NETWORK POLICY” and you will see the following screen:

In order to avoid configuration sprawl, we could configure wireless and wired devices under a single network policy. However, since all we are worried about for this particular post is wireless, uncheck the boxes that say “Switches” and Routing”. Give your network policy a name and then click “SAVE” at the bottom right of the screen.

You should see the following screen now:

Since we are only building a policy for wireless, you will notice that the “Router Settings” tab at the top is now grayed out. There is no point in going to that tab since we are not dealing with routing in this policy. What we are looking at now is the “Wireless Networks” tab. On this tab, we can create an SSID, or select an existing SSID if we had created one in another policy and wanted to re-use it. Since this is our first policy, click on the “ADD” button. You’ll notice that when you click on the button, two choices appear:

 

 

 

 

If we were creating a guest network, we could click on “Guest Access Network” and it would give us a streamlined way to configure guest type SSIDs. For this particular post though, we just want to create a simple WPA2 Personal SSID which uses a standard passphrase. Click on “All other Networks (standard)”.

You should now see the following screen:

Type in a name for your SSID. Notice there are two fields at the top. The “Name (SSID)” field is simply the name of the SSID object as it will appear in HiveManager. If you hit the tab key or simply click on the “Broadcast Name” field below, it will duplicate whatever you typed in on the “Name (SSID)” field. The “Broadcast Name” is what will actually be advertised over the air via a beacon. Under the SSID Usage section, you will see the various authentication types that can be used within this particular SSID. We want to create a basic WPA2 Personal SSID, so click on the “Personal” button. Private Pre-Shared Key(PPSK) is something similar to the standard WPA2 Personal authentication, but it uses multiple passphrases. See here for more info on PPSK.

Enter a passphrase in the “Key Value” field. If you want to make sure of what it will be set to, click on the “Show Password” box to display what you actually typed in the “Key Value” field.

Note: If at any time you are unsure of what you are looking at on the screen, there is always a question mark icon on the far right of the screen. If you click on that question mark, it will open up another tab and display the help documentation about that particular section or feature.

Click on “SAVE” at the bottom right. It should return you to the main “Wireless Networks” screen like the following:

 

Click on “NEXT” in the bottom right of the screen. Now, you will be looking at the “Device Templates” tab. Since we turned off switching and routing for this particular network policy, the only thing you see in the “Device Templates” tab is a section labeled “AP Templates”. If switching was turned on, it would display another section labeled “Switch Templates”.

We’re going to skip the “Device Templates” section in the interest of just getting an AP up and running. I’ll cover this tab in another post. Click “NEXT” at the bottom right and it should bring you to the “Additional Settings” tab as shown below:

We are going to skip this tab as well. It contains a bunch of different things that will be covered in another post. Click “NEXT” and it should bring you to the “Deploy Policy” tab which looks like this:

 

If we had any APs already added to HiveManager, this screen would show the APs in a list and then you could select the AP(s) you wanted to apply this policy to. However, we have not added an AP yet, so we are finished with the network policy configuration. Click “EXIT” on the bottom left of the screen.

Now you are taken back to the “Network Policies” screen. You should see the network policy we just created and it will look like this:

The basic network policy configuration is complete. The next step is to get the AP added to HiveManager and push the configuration to it.

Adding An AP To HiveManager

If you have not plugged in your Aerohive AP, go ahead and do that now. Eventually you should see a solid white light on the AP, or if it is one of the models like the AP 150W, it will be a green light.

Click on the “MONITOR” tab in HiveManager. Next, click on “ADD” and select “Advanced Onboarding”. You will see the following:

On this screen, add the serial number of the AP in the appropriate box. If you have multiple APs, you can add multiple serial numbers separated with a comma. Alternatively, you could use a CSV file to bring in a large number of APs at once. With the serial number(s) in place on the screen, click on “Next” at the bottom right. The following screen will appear next:

On this screen, you will be creating an “Organization” which directly correlates to the “MAPS” tab, which is where you can setup all of your different locations, buildings, and floors within a given building. When you fill in each of these lines, it uses the “City and State” to create a location, the “Street Address” to create a building, and then creates a default “Floor 1”. You can then assign each AP to a particular floor within a building, but you will have to add a floor plan first, which cannot be done from within this onboarding wizard. You can also go over to the “MAPS” tab and change the name of the location, building, etc. For now though, fill in each section and click on “Next” at the bottom right. The following screen will be displayed:

By default, the “Create Network Policy” screen will try and get you to make a new network policy for the device(s) you are onboarding. However, we previously created a network policy, so change the selection to “Use an existing network policy” and it should default to the name of the network policy you built previously. With that existing network policy selected, click on “Next” at the bottom right. The following screen will be displayed:

At this point, you should see the “Review Summary” screen. Click on “Finish” at the bottom right.

You will now be placed back in the main HiveManager interface in the “MONITOR” tab section. If your AP was recently powered on, it may look like the image below:

Once the AP makes a secure connection to HiveManager, the red hexagon on the far left under the “Status” column will turn green. The AP will push down the configuration from the network policy when it connects to HiveManager for the first time. Additionally, depending on the code level on the AP, it may update to a newer version. Expect it to take a few minutes for the AP to come up, update itself with configuration, and reboot before achieving a normal steady state. You may see the connection status go green, turn red again, and then go back to green.

When the AP has been updated with the network policy and is at a normal state to connect clients, it will look like the following:

Updating The Configuration

Your AP is now up and running in HiveManager, but the configuration was uploaded during the onboarding process. What I would like to do is show you what it looks like when you make a change to the network policy or device and need to push an updated configuration to it. Let’s change something simple like the hostname of the device.

Click on the hostname of the AP. By default, it will choose a hostname of “AH-” and the last 6 characters of its default MAC address.

Under the “CONFIGURATION” menu on the left side of the screen, click on “Device Configuration”. Modify the “Host Name” field to whatever you want to name the AP. Once that is done, scroll down and click on “SAVE” in the bottom right of the screen.

If you scroll back up to the top after clicking on “SAVE”, you will see a box that says “UPDATE NOW”. You could click on that and the changes will get pushed to the AP, but we are not going to do that in order to see what it will look like from the main MONITOR tab page. Click on “Back to Device List” at the top left instead.

Now you should see the main MONITOR tab screen with your AP, but notice there is a red exclamation point next to the green hexagon on the far left. That red exclamation point tells you that there is a difference in the configuration on HiveManager and the AP itself. We need to push this change to the AP before it will take effect.

You can click on the red exclamation point and see what those pending changes are. Go ahead and click on the exclamation point.

Another window opens up with 3 tabs. Since there is a pending change, it defaults you to the “Audit” tab.

The Audit tab shows you the general sections in which the configuration changed. This lets you know what changes were made from a high level. Click on the “Delta” tab.

When the Delta tab is displayed, you can see the exact CLI string that will be pushed to the AP. Note the “hostname” command I highlighted. If you were to click on the “Complete” tab, you would see the entire configuration that HiveManager shows for that particular AP. Click on the “X” at the top right of this window to close it. You could click on “CLOSE” at the bottom right as well if you wanted to.

Now that we know what changes are going to be pushed down to the AP, we want to initiate the push. Select the box to the far left of the line the AP is displayed on.

Next, click on the “Update Devices” button at the top right of the screen.

This new window that opens up gives you some options regarding pushing the configuration, updating the firmware on the device(Aerohive APs run HiveOS), or even scheduling when that updated configuration or firmware gets applied to the AP.

Note: There are two methods to push configuration to a device. Delta will only push the incremental changes and will not affect the AP to a great extent in terms of client connections unless you are changing something that is RF related. Complete will push the entire configuration to the AP, but in doing so, it will reboot the AP. You can use Delta pushes for the overwhelming majority of configuration changes to an AP. The system is smart enough to tell you if a complete push is required and will go so far as to make it to where you cannot select the delta option.

Since all we want to do is update the hostname, ensure that the “Delta Configuration Update” option is selected and then click on “PERFORM UPDATE” in the bottom right of the window.

After clicking on “PERFORM UPDATE”, HiveManager will start to push the delta update to the AP. It should take a few seconds to push it out. Once it is done, the red exclamation point will be replaced with a green check mark. When you see that green check mark, you know the configuration on HiveManager is in sync with the configuration on the AP.

You should be able to make a client connection to this particular AP using the passphrase you set when configuring the SSID.

That’s all there is to it! Well, that’s not ALL there is to it. There are a lot of other things you can configure in HiveManager, but those will be covered in follow-up posts. The goal here was to get some configuration loaded onto the AP so that you can connect some clients to it.

by Matthew Norwood at December 04, 2018 07:38 AM

ipSpace.net Blog (Ivan Pepelnjak)

Bifurcation of Knowledge

My friend Andrea Dainese (of the Route Reflector Labs fame) sent me this observation:

Because of lack of fundamental skills, I see two groups forming: junior guys with low salary (the bigger group), and a few experts (hopefully with higher salary). The middle group is disappearing. Intermediate-level engineers are either moving to the entry level (because the complexity is increasing and they are not keeping up with it) or to the upper level.

I call this phenomenon bifurcation of knowledge (I’m positive it has a formal name – would appreciate a comment with a set of pointers), and it’s a direct result of commoditization and the changing shape of the learning curve.

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at December 04, 2018 07:31 AM

December 03, 2018

ipSpace.net Blog (Ivan Pepelnjak)

David Gee on Security of Network Automation

One of the points David Gee, a guest speaker in Spring 2019 Building Networking Automation Solutions online course, and Christoph Jaggi touched on in their interview was the security of network automation solutions (see also: automated workflows and hygiene of network automation).

What are the security risks for automation?

Security is an approach, not an afterthought.

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at December 03, 2018 07:23 AM

XKCD Comics

November 30, 2018

Honest Networker
The Networking Nerd

The Magic of the CCIE

I stumbled across a great Reddit thread this week: Is the CCIE as impossible as it seems? There are a lot of great replies on that thread about people passing and the “good old days” of Banyan Vines, Appletalk, and more. It’s also a fascinating look into how the rest of the networking industry sees exams like the CCIE and JNCIE. Because those of us that have the numbers seem to be magicians to some.

Sleight of CLI Hand

Have you ever seen the cups and balls magic trick? Here’s an excellent example of it from the recently departed Ricky Jay:

<iframe allowfullscreen="true" class="youtube-player" height="329" src="https://www.youtube.com/embed/QwF1ec4Ji7Y?version=3&amp;rel=1&amp;fs=1&amp;autohide=2&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;wmode=transparent" style="border:0;" type="text/html" width="584"></iframe>

Impressive, right? It’s amazing to behold a master craftsman at work. Every time I watch that video I’m amazed. I know he’s doing sleight of hand. But I can’t catch it. Now, watch this same video but with annotations turned on. SPOILER ALERT – The annotations will tell you EXACTLY where the tricks are done:

<iframe allowfullscreen="true" class="youtube-player" height="329" src="https://www.youtube.com/embed/ukUZaYMcTOM?version=3&amp;rel=1&amp;fs=1&amp;autohide=2&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;wmode=transparent" style="border:0;" type="text/html" width="584"></iframe>

Is it more impressive now that you know how the tricks are done? Check out this demonstration from Penn and Teller that shows you exactly how they do the tricks as well:

<iframe allowfullscreen="true" class="youtube-player" height="329" src="https://www.youtube.com/embed/GmwT7L0hToQ?version=3&amp;rel=1&amp;fs=1&amp;autohide=2&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;wmode=transparent" style="border:0;" type="text/html" width="584"></iframe>

Okay, so it’s a little less mystifying now that you’ve seen how all the sleight of hand happens. But it’s still impressive because, as a professional, you can appreciate how the execute their tradecraft. Knowing that it’s not magic doesn’t mean it’s not an impressive feat. It must means you appreciate something different about the performance.

Let’s apply that to the CCIE. When you’re just starting out in networking, every piece of knowledge is new. Everything you learn is something you didn’t know before. Subnet masks, routing tables, and even just addressing an interface are new skills that you acquire and try to understand. It’s like learning how to take a coin from someone’s ear. It’s simple but it provides the building blocks for future tricks.

When you reach the level of studying for the CCIE lab, it does look like a daunting task. If you’ve followed Cisco’s guidelines you probably have your CCNP or equivalent knowledge. However, there is still a lot you don’t know. If you don’t believe that, go pick up Jeff Doyle’s Routing TCP/IP Volume 1 book. That book taught me I still had a lot to learn about networking.

But, as I slogged through the CCIE, I realized that I was acquiring skills. Just like the magicians that practice the cups and balls every day to get it right, I was picking up the ability to address interfaces quickly and see potential routing loops before I made them like I did in my first lab attempt. Each thing I learned and practiced not only made me a better engineer but also made the CCIE seem less like a mountain and more like a hill that could be climbed.

And I truly realized this when I was thumbing through a copy of the CCIE Official Exam guide. Someone had given me a copy to take a look at and I was happy with the depth of knowledge that I found. I wanted to pass it along to another junior engineer because, as I said to myself, “If only I had this book when I started! I could have skipped over all those other books!”

Practice, Practice, Practice!

That’s where I went wrong. Because I jumped right to the end goal instead of realizing the process. Magicians don’t start out making the Statue of Liberty disappear. They start out pulling coins from your ear and finding your card in a deck. They build their basic skills and then move on to harder things. But they most grand tricks in the magician’s top hat all still use the basic skills: sleight of hand, misdirection, and preparation. To neglect those is to court folly on stage.

CCIEs are no different. Every person that asks me about the test asks “How hard is it to pass?” I usually respond with something like “Not hard if you study.” Some of the people I talk to pick up on the “not hard” part and get crushed by the lab their first time out. They even end up with a $1,500 soda for their efforts. The other people, the ones that focus on “study” in my answer, they are the people who pass on the first attempt or the ones that get it right pretty quickly thereafter.

The CCIE isn’t a test. It’s a course in studying. It’s the culmination of teaching yourself the minutia of protocols and how they interact. The exam itself is almost perfunctory. It tests specific combinations of things you might see in the real world. And if you ask any CCIE, the real world is often ten time stranger than the lab. But the lab makes you think about the things you’ve already learned in new ways and apply that knowledge to find ways to solve problems. The lab isn’t hard because it’s easy. The lab becomes easier when you practice enough to not think the knowledge is hard any longer. I think Bruce Lee said it best:

I fear not the man who has practiced 10,000 kicks once, but I fear the man who has practiced one kick 10,000 times.

Most people would agree that Bruce Lee was one of the best martial artists of all times. And even he practiced until his fingers bled and he body was exhausted. Because he knew that being the best wasn’t about passing an exam for a belt or about showing off for people. It was about knowing what you needed to know and practicing it until it was second nature.


Tom’s Take

The CCIE has a certain magical aura for sure. But it’s not magical in and of itself. It’s a test designed to ensure that the people that pass know their skills at a deep level. It’s a test designed to make you look deeper at a problem and exhaust all your options before throwing in the towel. The CCIE isn’t impossible any more than sawing someone in half is impossible. It’s all about how your practice and prepare for the show that makes the trick seem impressive.

by networkingnerd at November 30, 2018 02:46 PM

ipSpace.net Blog (Ivan Pepelnjak)

Worth Reading: Back to Basics

I’m not the only one ranting about the need to get a firm grasp on fundamentals before doing the sexy stuff. Found an old blog post by Joel Spolsky (of the Law of Leaky Abstractions fame) on the exact same topic from programming perspective.

If you ever had to deal with a programming language, it’s definitely worth reading… but some of the details might make your head explode. You’ve been warned ;)

by Ivan Pepelnjak (noreply@blogger.com) at November 30, 2018 01:10 PM

Interview: Active-Active Data Centers with VXLAN and EVPN

Christoph Jaggi asked me a few questions about using VXLAN with EVPN to build data center fabrics and data center interconnects (including active/active data centers). The German version was published on Inside-IT, here’s the English version.

He started with an obvious one:

What is an active-active data center and why would I want to use an active-active data center?

Numerous organizations have multiple data centers for load sharing or disaster recovery purposes. They could use one of their data centers and have the other(s) as warm or cold standby (active/backup setup) or use all data centers at the same time (active/active).

Read more ...

by Ivan Pepelnjak (noreply@blogger.com) at November 30, 2018 01:10 PM

Honest Networker
XKCD Comics