Most of us are in some sort of lockdown (or quarantine or shelter-in-place or whatever it’s called) at the moment. Some have their hands full balancing work and homeschooling their kids (hang in there!), others are getting bored and looking for networking-related content (or you wouldn’t be reading this blog).

If you’re in the latter category you might want to browse some of the free ipSpace.net content: almost 3500 blog posts, dozens of articles, over a hundred podcast episodes, over 20 free webinars, and another 30+ webinars with sample videos that you can access with free subscription.

Need more? Standard subscription includes 260 hours of video content and if you go for Expert subscription and select the network automation course as part of the subscription, you’ll get another 60 hours of content plus hands-on exercises, support, access to Slack team… hopefully enough to last you way past the peak of the current pandemic.

Back during Networking Field Day 22, I was having a fun conversation with Phil Gervasi (@Network_Phil) and Carl Fugate (@CarlFugate) about SD-WAN and innovation. I mentioned that it was fascinating to see how SD-WAN companies kept innovating but that bigger, more established companies that had bought into SD-WAN seemed to be having issues catching up. As our conversation continued I realized that technical debt plays a huge role in startup culture in all factors, not just with SD-WAN. However, SD-WAN is a great example of technical debt to talk about here.

Any Color You Want In Black

Big companies have investments in supply chains. They have products that are designed in a certain way because it’s the least expensive way to develop the project or it involves using technology developed by the company that gives them a competitive advantage. Think about something like the Cisco Nexus 9000-series switches that launched with Cisco ACI. Every one of them came with the Insieme ASIC that was built to accelerate the policy component of ACI. Whether or not you wanted to use ACI or Insieme in your deployment, you were getting the ASIC in the switch.

Policies like this lead to unintentional constraints in development. Think back five years to Cisco’s IWAN solution. It was very much the precursor to SD-WAN. It was a collection of technologies like Performance Routing (PfR), Application Visibility Control (AVC), Policy Based Routing (PBR), and Network Based Application Recognition (NBAR). If that alphabet soup of acronyms makes you break in hives, you’re not alone. Cisco IWAN was a platform very much market by potential and complexity.

Let’s step back and ask ourselves an important question: “Why?” Why was IWAN so complicated? Why was IWAN hard to deploy? Why did IWAN fail to capture a lot of market share and ride the wave that eventually became SD-WAN? Looking back, a lot of the choices that were made that eventually doomed IWAN can come down to existing technical debt. Cisco is a company that makes design decisions based on what they’ve been doing for a while.

I’m sure that the design criteria for IWAN came down to two points:

1. It needs to run on IOS.
2. It needs to be an ISR router.

That doesn’t sound like much. But imagine the constraints you run into with just those two limitations. You have a hardware platform that may not be suited for the kind of work you want to do. Maybe you want to take advantage of x86 chipset acceleration. Too bad. You have to run what’s in the ISR. Which means it could be underpowered. Or incapable of doing things like crypto acceleration for VPNs, which is important for building a mesh of encrypted tunnels. Or maybe you need some flexibility to build a better detection platform for applications. Except you have to use IOS. Which uses NBAR. And anything you write to extend NBAR has to run on their platforms going forward. Which means you need to account for every possible permutation of hardware that IOS runs on. Which is problematic at best.

See how technical debt can creep in from the most simplistic of sources? All we wanted to do was build a platform to connect WANs together easily. Now we’re mired in a years-old hardware choice and an aging software platform that can’t help us do what needs to be done. Is it any wonder why IWAN didn’t succeed in the original form? Or why so many people involved with the first generation of SD-WAN startups were involved with IWAN, even if just tangentially?

Debt-Free Development

Now, let’s look at a startup like CloudGenix, who was a presenter at Networking Field Day 22 and was recently acquired by Palo Alto Networks. They started off on a different path when they founded the startup. They knew what they wanted to accomplish. They had a vision for what would later be called SD-WAN. But instead of shoehorning it into an existing platform, they had the freedom to build what they wanted.

No need to keep the ISR platform? Great. That means you can build on x86 hardware to make your software more universally deployable on a variety of boxes. Speaking of boxes, using commercial off-the-shelf (COTS) equipment means you can buy some very small devices to run the software. You don’t need a system designed to use ATM modules or T1 connections. If all you little system is ever going to use is Ethernet there’s no reason to include expansion at all. Maybe USB for something like a 4G/LTE modem. But those USB ports are baked into the board already.

A little side note here that came from Olivier Huynh Van of Gluware. You know the USB capabilities on a Cisco ISR? Yeah, the ISR chipset didn’t support USB natively. And it’s almost impossible to find USB that isn’t baked into an x86 board today. So Cisco had to add it to the ISR in a way that wasn’t 100% spec-supported. It’s essentially emulated in the OS. Which is why not every USB drive works in an ISR. Take that for what’s it’s worth.

Back to CloudGenix. Okay, so you have a platform you can build on. And you can build software that can run on any x86 device with Ethernet ports and USB devices. That means your software doesn’t need to do complicated things. It also means there are a lot of methods already out there for programming network operating systems for x86 hardware, such as Intel’s Data Plane Development Kit (DPDK). However CloudGenix chose to build their OS, they didn’t need to build everything completely from scratch. Even if they chose to do it there are still a ton of resources out there to help them get started. Which means you don’t have to restart your development every time you need to add a feature.

Also, the focus on building the functions you want into an OS you can bend to your needs means you don’t need to rely on other teams to build pieces of it. You can build your own GUI. You can make it look however you want. You can also make it operate in a manner that is easiest for your customer base. You don’t need to include every knob or button or bell and whistle. You can expose or hide functions as you wish. Don’t want customers to have tons of control over VPN creation or certificate authentication? You don’t need to worry about the GUI team exposing it without your permission. Simple and easy.

One other benefit of developing on platforms without technical debt? It’s easy to port your software from physical to virtual. CloudGenix was already successful in porting their software to run from physical hardware to the cloud thanks to CloudBlades. Could you imagine trying to get the original Cisco IWAN running in a cloud package for AWS or Azure? If those hives aren’t going crazy right now I’m sure you must have nerves or steel.

---

Tom’s Take

Technical debt is no joke. Every decision you make has consequences. And they may not be apparent for this generation of products. People you may never meet may have to live with your decisions as they try to build their vision. Sometimes you can work with those constraints. But more often than not brilliant people are going to jump ship and do it on their own. Not everyone is going to succeed. But for those that have the vision and drive and turn out something that works the rewards are legion. And that’s more than enough to pay off any debts, technical or not.

Context: https://radar.qrator.net/blog/how_you_deal_with_route_leaks

As I explained in How Networks Really Work and Upcoming Internet Challenges webinars, routing security, and BGP security in particular remain one of the unsolved challenges we’ve been facing for decades (see also: what makes BGP a hot mess).

Fortunately, due to enormous efforts of a few persistent individuals BGP RPKI is getting traction (NTT just went all-in), and Flavio Luciani and Tiziano Tofoni decided to do their part creating an excellent in-depth document describing BGP RPKI theory and configuration on Cisco- and Juniper routers.

There are only two things you have to do:

• Read the document;
• Implement RPKI in your network.

Thank you, the Internet will be grateful.

On March 23, 2020, Ned Bellavance and Ethan Banks had a discussion about AWS networking performance for the Day Two Cloud podcast. This is a complicated conversation to have, because making networking perform in AWS isn’t about speeds & feeds. It’s about services.You can’t just throw bandwidth at AWS and expect things to go faster. […]

The post AWS Networking Performance: Loadbalancers (Video) appeared first on Packet Pushers.

128 Technology offers a smarter way to support remote access. Your connections to the corporate network and cloud applications need to be Session Smart.

The post “Thou Shalt Not Bridge Networks!” – Why Traditional WFH Strategies Are Failing appeared first on Packet Pushers.

Zoom Meeting IDs are sequential ten digit numbers. People are randomly creating IDs and dropping into Zoom conferences often using abusive language, displaying pornagraphic images or worse.  Yes, Zoom has poor security posture generally. The design approach appears makes it easy to use as possible while compromising security. Compared to other conferencing platforms, it does […]

The post Tips to prevent Zoom Bombing, Security and Conference Hygience appeared first on EtherealMind.

Every time I’m complaining about the stupidities $vendors are trying to sell us, someone from vendorland saddles a high horse and starts telling me how I got it all wrong, for example:

It is a duty of a pre-sales, consultant, vendor representative to inform the customer about the risk.

When you stop laughing (maybe it was just April Fools’ joke ;), here’s how the reality of that process looks like (straight from one of my readers):

I remember when the VM guys and their managers were telling me (like they had discovered the solutions to all of ours problems) about “with VXLAN we can move a machine from one country to another, and keep having service with the same IP” … while looking at me with the “I’m so smart” face… and me thinking shit… I’m doomed :) … I don’t even want to start explaining … but in the long run I had to anyway.

One of the first hands-on exercises in our Networking in Public Cloud Deployments asks the attendees to automate something. They can choose the cloud provider they want to work with and the automation tool they prefer… but whatever they do has to be automated.

Most solutions include a simple CloudFormation, Azure Resource Manager, or Terraform template with a line or two of README.MD, but Erik Auerswald totally astonished me with a detailed and precise writeup. Enjoy!

In January 2019, my boss set out the challenge of Network Automation. I’d dabbled in Python  before, writing a beautiful expect script to scrap the number of users connected to a Cisco 5508 WLC and warn when it reached the 6000 client limit. I’d followed the learning labs on Cisco DevNet and thought “that’s interesting”. […]

The post A Journey in Model Driven Programmability for Network Engineers appeared first on Packet Pushers.

Random and hard to know

The post Uneven or Unbalanced Sound Levels on AirPods and Microphones on MacOS appeared first on EtherealMind.

With webinars being the only way to deliver training content these days, we’ll run one every week in April 2020:

• Starting on April 2nd I’ll talk about one of my favorite topics: switching, bridging and routing, covering almost everything ever invented from virtual circuits and source route bridging to so-called routing at layer-2 and IP forwarding based on host routes;

• I was planning to update the Introduction to Containers and Docker material for ages… but then had to move the December 2019 workshop to March 2020, only to cancel it a week before the coronavirus exploded for real in Switzerland. I hope I’ll manage to deliver the online version on April 9th ;)

• Dinesh Dutt is back on April 16th with an update of Network Automation Tools webinar, in which he’ll cover (among other things) the new network automation tools launched since we did the original webinar in 2016.

• On April 23rd Pete Lumbis plans to dive as deep into the intricacies of switching ASICs as he can without violating an NDA ;)

On March 23, 2020, Ned Bellavance and Ethan Banks had a discussion about AWS networking performance for the Day Two Cloud podcast. This is a complicated conversation to have, because making networking perform in AWS isn’t about speeds & feeds. It’s about services.You can’t just throw bandwidth at AWS and expect things to go faster. […]

The post AWS Networking Performance: EC2 Placement Groups – Video appeared first on Packet Pushers.

I’m a huge fan of video games. I love playing them, especially on my old consoles from my formative years. The original Nintendo consoles were my childhood friends as much as anything else. By the time I graduated from high school, everyone had started moving toward the Sony Playstation. I didn’t end up buying into that ecosystem as I started college. Instead, I just waited for my brother to pick up a new console and give me his old one.

This meant I was always behind the curve on getting to play the latest games. I was fine with that, since the games I wanted to play were on the old console. The new one didn’t have anything that interested me. And by the time the games that I wanted to play did come out it wouldn’t be long until my brother got a new one anyway. But one thing I kept hearing was that the Playstation was backwards compatible with the old generation of games. I could buy a current console and play most of the older games on it. I wondered how they managed to pull that off since Nintendo never did.

When I was older, I did some research into how they managed to build backwards compatibility into the old consoles. I always assumed it was some kind of translation engine or enhanced capabilities. Instead, I found out it was something much less complicated. For the PS2, the same controller chip from the PS1 was used, which ensured backwards compatibility. For the PS3, they essentially built the guts of a PS2 into the main board. It was about as elegant as you could get. However, later in the life of those consoles, system redesigns made them less compatible. Turns out that it isn’t easy to create backwards compatibility when you redesign things to remove the extra hardware you added.

Bringing It Back To The Old School

Cool story, but what does it have to do with enterprise technology? Well, the odds are good that you’re about to fight a backwards compatibility nightmare on two fronts. The first is with WPA3, the newest security protocol from the Wi-Fi Alliance. WPA3 fixes a lot of holes that were present in the ancient WPA2 and includes options to protect public traffic and secure systems from race conditions and key exchange exploits. You’d think it was designed to be more secure and would take a long time to break right? Well, you’d be wrong. That’s because WPA3 was exploited last year thanks to a vulnerability in the WPA3-Transition mode designed to enhance backwards compatibility.

WPA3-Transition Mode is designed to keep people from needing to upgrade their wireless cards and client software in one fell swoop. It can configure a WPA3 SSID with the ability for WPA2 clients to connect to it without all the new enhanced requirements. Practically, it means you don’t have to run two separate SSIDs for all your devices as you move from older to newer. But practical doesn’t cover the fact that security vulnerabilities exist in the transition mechanism. Enterprising attackers can exploit the weaknesses in the transition setup to crack your security.

It’s not unlike the old vulnerabilities in WPA when it used TKIP. TKIP was found to have a vulnerability that allowed for exploiting. People were advised to upgrade to WPA-AES as soon as possible to prevent this. But if you enabled older non-AES capable clients to connect to your SSIDs for compatibility reasons you invalidated all that extra security. Because AES had to operate in TKIP mode to connect the TKIP clients. And because the newer clients were happy to use TKIP over AES you were stuck using a vulnerable mode. The only real solution was to have a WPA-AES SSID to connect to for your newer secure clients and leave a WPA-TKIP SSID active for the clients that had to use it until they could be upgraded.

4Gs for the Price of 5

The second major area where we’re going see issues with backwards compatibility is with 5G networking. We’re hearing about the move to using 5G everywhere. We’ve no doubt heard by now that 5G is going to replace enterprise wireless or change the way we connect to things. Honestly, I’m not surprised someone has tried to make the claim that 5G can make waffles and coffee yet. But 5G is rife with the same backwards compatibility issues present in enterprise wireless too.

5G is an evolution of the 4G standards. Phones issued today are going to have 4G and 5G radios and the base stations are going to mix the radio types to ensure those phones can connect. Just like any new technology, they’re going to maximize the connectivity of the existing infrastructure and hope that it’s enough to keep things running as they build out the new setup. But by running devices with two radios or having a better connection from the older devices, you’re going to set yourself up to have your new protocol inherently insecure thanks to vulnerabilities in the old versions. It’s already projected that governments are going to take advantage of this for a variety of purposes.

We find ourselves in the same boat as we do with WPA3. Because we have to ensure maximum compatibility, we make sacrifices. We keep two different versions running at the same time, which increases complexity. We even mark a lot of necessary security upgrades as optional in order to keep people from refusing to implement them or fall behind because they don’t understand them¹.

The biggest failing for me is that we’re pushing for backwards compatibility and performance over security. We’re not willing to make the hard choices to reduce functionality in order to save our privacy and security. We want things to be backwards compatible so we can buy one device today and have it work on everything. We’ll just make the next one more secure. Or the one after that. Until we realize that we’re still running old 802.11 data rates in our newest protocols because no one bothered to remove them. We have to make hard choices sometimes and sacrifice some compatibility in order to ensure that we’re safe and secure with the newer technology.

---

Tom’s Take

Backwards compatibility is like the worst kind of nostalgia. I want the old thing but I want it on a new thing that runs faster. I want the glowing warmth of my youth but with the convenience of modern technology. It’s like buying an old sports car. Sure, you get all the look and feel of an old powerful engine. You also lose the safety features of the new body along with the comforts you’ve become accustomed to. You have to make a hard choice. Do you keep the old car original and lose out on what you like to get what you want? Or do you create some kind of hybrid that has exactly what you want and need but isn’t what you started with? It’s a tough choice to make. In the world of technology, there’s no right answer. But we need to remember that every compromise we make for performance can lead to compromises in security.

When I’ve seen my good friends Christopher Werny and Enno Rey talk about IPv6 security at RIPE78 meeting, another bit of one of my puzzles fell in place. I was planning to do an update of the IPv6 security webinar I’d done with Eric Vyncke, and always wanted to get it done by a security practitioner focused on enterprise networks, making Christopher a perfect fit.

As it was almost a decade since we did the original webinar, Christopher started with an overview of IPv6 security challenges (TL&DR: not much has changed).

A lot of people are confused about the roles of network layers (some more than others), the interactions between MAC addresses, IP addresses, and TCP/UDP port numbers, the differences between routing and bridging… and why it’s so bad to bridge across large distances (or in large networks).

I tried to explain most of those topic in How Networks Really Work webinar (next session coming on April 2nd), but as is usually the case someone did a much better job: you MUST READ the poetic and hilariously funny World in which IPv6 was a good design by Avery Pennarun.

Cato Networks has announced a new clientless remote access option called Instant Access. Remote workers use a browser to access approved applications from a company portal. Companies don’t need to install client software on end-user machines, which can speed employee access for companies trying to handle an influx of remote workers.

The post Cato Networks Rolls Out Clientless Remote Access appeared first on Packet Pushers.

Cancelled or postponed all conferences scheduled through August 2020.

The post Gartner Cancels Conferences losing >180MM in 2020 revenues appeared first on EtherealMind.

Cancelled or postponed all conferences scheduled through August 2020.

The post Gartner Cancels Conferences losing >260MM in 2020 revenues appeared first on Packet Pushers.

A reader of my blog was “blessed” with hands-on experience with SD-WAN offered by large service providers. Based on that experience he sent me his views on whether that makes sense. Enjoy ;)

---

We all have less-than-stellar opinion on service providers and their offerings. Its well known that those services are expensive and usually lacking quality, experience, or simply, knowledge. This applies to regular MPLS/BGP techniques as to - currently, the new challenge - SD-WAN.

Juniper Networks' Mist Systems has announced a new add-on service that can analyze and report on a year's worth of data from Mist APs, Juniper switches, and third-party network gear. The service targets network engineers, as well as lines of business that want location-based analytics for business insights.

The post Mist Systems’ Premium Analytics Combines Network Visibility And Business Insights appeared first on Packet Pushers.

One of the attendees of our Building Network Automation Solutions online course asked an interesting question in the course Slack team:

Has anyone wrote a playbook for putting a circuit into maintenance mode — i.e. adjusting metrics to drain traffic away from a circuit that is going to be taken down for maintenance?

As always, you have to figure out what you want to do before you can start to automating stuff.

We need a secure and trustable infrastructure. We need to be able to provide assurance that the service we are contacting is genuine, that the transaction is secured from eavesdroppers and that we leave no useful traces behind us. Why has our public key certificate system failed the Internet so badly?

Found this “gem” describing the differences between layer-2 and layer-3 on an unnamed $vendor web site.

Layer 2 is mainly concerned with the local delivery of data frames between network devices on the same network or local area network (LAN).

So far so good…

Years ago I figured out that I’d eventually have to migrate my blog from Blogger to something more independent, and based on my previous experience with Wordpress I wasn’t exactly enthusiastic to go down that path.

In 2015 I’ve seen Scott Lowe going from Wordpress to Jekyll and then to Hugo, and decided it might make sense to recreate ipSpace.net blog with a tool that generates static web pages… but never found the time to do it.

Defining service availability using the famous X nines (and all the hacks like “planned downtime doesn’t count”) is pretty useless in a highly distributed system where the only thing that really matters is the user experience, not ping response times. One should ask what precisely should we be measuring, and how could we make sure we can act on the measurements

More details in a concise analysis of the Meaningful Availability paper by the one-and-only The Morning Paper.

It’s been a crazy week. I know the curse is “May you live in interesting times,” but I’m more than ready for things to be less interesting for a while. It’s going to take some time to adjust to things. From a networking perspective, I have a few things that have sprung up.

• Video conferencing is now a big thing. Strangely, Cisco couldn’t make video the new phone. But when people are stuck at home now we need to do video again? I get that people have a need to see each other face-to-face. But having worked from home for almost seven years at this point I can tell you video isn’t a necessity. It’s a nice option, but you can get a lot accomplished with video calls and regular emails.
• Along side this is the fact that the push to put more video out there is causing applications to reach their breaking points. Zoom, which is fairing the best out of all of them so far, had some issues on Thursday morning. Tripling the amount of traffic that’s going out and making it very sensitive to delay and jitter is going expose a lot of flaws in the system.
• I applaud all of the companies in the last week that have chosen to step out and offer resources to help people work better from home. I also hope that employees and managers use them after this is over to help enable more remote work. Just remember that flexibility has a cost axis as well. Those VPNs and security services and CASBs aren’t going to be free forever. If it makes sense, use it. Otherwise, find something that does.
• Remember that this is a stressful time for everyone. I work from home all the time. And this week I have been totally exhausted. Try to find a way to keep your sanity. Step outside for air. Take a short break. Look for ways to keep yourself healthy. It’s going to take time for people to adjust to this. It’s going to take time even if you know how to work remotely too.

---

Tom’s Take

I’m not sure where this is all headed. We’re all still figuring it out. Things won’t look the same six months from now no matter what. But keep working where you can and improving what you do. The value in this shift comes from empowering us to do what we can. If that means cutting back on Netflix during working hours or spending some extra time learning a new skill make it happen and grow as much as you can. We’re going to need that.

An open videoconference to be nice to each other, blow the Corona blues away.

The post Invitation: Virtual Happy Hour The Second – TONIGHT 30 Mar 2020 2100BST/1600EST/1300PST appeared first on EtherealMind.

The tension between iteration and guarantees.

The post Opinion : Engineers, Solutions and Problems appeared first on EtherealMind.

After describing the FRRouting architecture, as well as recent performance optimizations and usability enhancements, Donald Sharp concluded the FRRouting webinar with detailed deployment guidelines.

With open networking from Cumulus and 400G Ethernet, you can upgrade from 10 to 40 to 100 to 400G devices without having to change your network OS or abandon the tools and processes you already use to run your network. Merchant silicon means you get abundant hardware choices, plus the option to use lower-cost optics, which can account for the better part of your hardware costs.

The post 400G Ethernet And Open Networking: A Powerful Combination appeared first on Packet Pushers.

One of the first roadblocks you’ll hit in your “let’s master Ansible” journey will be a weird error deep inside a Jinja2 template. Can we manage that complexity somehow… or as one of the participants in our Building Network Automation Solutions online course asked:

Is there any recommendation/best practices on Jinja templates size and/or complexity, when is it time to split single template into function portions, what do you guys do? And what is better in terms of where to put logic - into jinja or playbooks

One of my friends described the challenge as “Debugging Ansible is one of the most terrible experiences one can endure…” and debugging Jinja2 errors within Ansible playbooks is even worse, but there are still a few things you can do.

This blog post will focus on a topic I don’t usually dive into (Javascript and related tooling), but I felt like others might benefit from the solution to a problem I encountered while doing local development for antidote-web, the web front-end that powers NRE Labs. A quick aside on the architecture for the front-end code for the Antidote platform - the antidote-web project is the lynchpin for everything. It’s where the general structure of the front-end app is managed.

(Reading time: 9 minutes) Many of us in IT find ourselves with a little more time in the day now that our daily commutes have been…commuted, due to the recent global pandemic. I decided to use the extra time to create a simple tracker to scrape the Web pages of official healthcare sites and display […]

The post A Simple COVID-19 Web Tracker With Serverless Functions appeared first on Packet Pushers.

BGP Autonomous System Numbers (ASNs) were originally specified with 2 bytes, giving us a range of about 64K total numbers. As the Internet grew, we needed more ASNs than the 2-byte specification would allow. Poof! 4-byte ASNs were born. In this video, Ethan Banks reviews technical details of 4-byte ASNs including related RFCs and private […]

The post Packet Pushers Labs: BGP 4-Byte ASNs – Video appeared first on Packet Pushers.

As the last part of his network automation journey, Anne Baretta described the automation tools he used.

Notes

• We covered numerous automation tools (including the ones Anne used) in our Building Network Automation Solutions online course;
• For an overview of automation tools, watch the Network Automation Tools webinar.

A number of tech vendors are offering extended trials periods and delayed payments for software and tools. We're keeping a list here in case it's helpful to anyone.

The post COVID-19 Vendor Offers Tracking Page appeared first on Packet Pushers.