Distributed systems are complicated. Add networking to the mix, and you get traumatic challenges like the CAP theorem and Byzantine fault tolerance. Most of those challenges are unknown to engineers who have to suffer through the vendor marketing presentations, making it hard to determine whether the latest shiny gizmo works outside of PowerPoint.

I started collecting articles describing distributed-system gotchas years ago, wrote numerous blog posts on the topic in the heydays of the SDN Will Save the World lemming run, and organized them into the Distributed Systems Resources page.

One of the most common causes of Internet routing leaks is an undereducated end-customer configuring EBGP sessions with two (or more) upstream ISPs.

Without basic-level BGP knowledge or further guidance from the service providers, the customer network engineer¹ might start a BGP routing process and configure two EBGP sessions, similar to the following industry-standard CLI² configuration:

EIGRP routing updates have always contained the next hop field (similar to BGP updates), which was unused until Cisco IOS release 12.3 when the no ip next-hop-self eigrp AS-number interface configuration command was implemented.

EIGRP does not set the next hop field by default. An EIGRP router receiving a routing update thus assumes that the next hop of the received routes is the sending router. This behavior usually works well, but prevents site-to-site shortcuts to be established in DMVPN networks, and results in suboptimal routing in some route redistribution scenarios.

Brandon Hitzel published a detailed document describing various Internet WAN edge designs. Definitely worth reading and bookmarking.

Another phenomenal detective story published on Cloudflare blog: Unbounded memory usage by TCP for receive buffers, and how we fixed it.

TL&DR: Moving TCP window every time you acknowledge a segment doesn’t work well with scaled window sizes.

The interesting takeaways:

It’s been almost five years since I wrote about the challenges of project management and timing your work as an engineer. While most of that information is still very true even today I’ve recently had my own challenges with my son’s Eagle Scout project. He is of a mind that you can throw together a plan and just do a whole week of work in just a couple of days. I, having worked in the IT industry for years, have assured him that it absolutely doesn’t work like that. Why is there a disconnect between us? And how does that disconnect look to the rest of the world?

Time Taking You

The first problem that I often see when working with people that aren’t familiar with projects is that they vastly underestimate the amount of time it takes to get something done. You may recall from my last post that my project managers at my old VAR job had built in something they called Tom Time to every quote. That provided a way for my estimate to reflect reality once I arrived on site and found the things didn’t go according to plan.

Part of the reason why my estimates didn’t reflect reality was because there are a lot of things that go into a project that can’t quite be explained or calculated into the final estimate. For example, how long does it take for a switch to reboot? Some of them can be ready to pass traffic in a couple of minutes. Larger devices that need to test modules may take up to ten minutes to be ready to go. If you have to reboot that switch multiple times during your project how do you account for that time? Is there a line item for a hour’s worth of switch reboots? What about the project closeout meetings a paperwork? How do you build that into a project timeline?

People that underestimate the timeline of a project are almost always only focused on the work. They see that it should take them about five minutes to copy the config the switch and ten minutes to put it in the rack. Did they think about the time to unbox it? Cable it? Do a final test to ensure all configuration is correct and saved to the startup config? Each of these things sound trivial but they add time. Maybe you don’t do the final config test and hope for the best. But you can’t shave time on unboxing unless you have someone helping you do that. Which, of course, just adds time to the project in a different way.

The Price of Time

Does this mean that you just need to increase the amount of time that you put on a project? No, it doesn’t. One of the connectivity providers I worked with in the past had what they called a “foolproof method” of getting the right time estimate for a circuit. They doubled the number and increased to the next time unit. So two hours became four days. Three days became six weeks. And I became infuriated when I realized how much time something like this would take.

Part of the reasoning behind that thinking was that the project management overhead always took longer than expected. But the other thinking was that quoting much longer timelines gave them more room to cram in too much work for a single team. They could juggle deployments because they had enough hours in the quote that they could be more interrupt driven. Work on something until someone complains then move to that project and work on it until the complaining stops. You can see why providers like that quickly get a reputation for padding their projects.

Time costs money. Either someone is paying you to do the job or you’re paying for that resource to be unavailable for doing the job. You have to learn how to allocate your resources effectively. If you need to help your teams or your contractors understand the additional time that it takes to do a project you need to either package that time as a line item or educate them about what additional tasks you see. Accounting for that extra time is a better way to show value than just adding lots of extra wiggle room to a project so you don’t go over budget. The education aspect is especially important for talent that isn’t familiar with things from the outset. Teaching them how to look for those time sinks and making sure they’re tracked means their estimates will be much more accurate in the future.

Tom’s Take

My son is going to complete his project but he’s going to learn a lot about the way the world works in the process. Paint doesn’t dry overnight. It takes time to load and unload lumber. People need more than 24 hours notice to show up to work on something. These are all lessons I’ve learned over the years that I’m happy to teach. Time is important to us all because we don’t get any more of it. Every minute that goes by is a minute we can’t get back. Make the most of your time by tracking it appropriately and building those hidden things into your project estimates. That’s how you get time to be on your side for once.

After introducing the routing protocols and explaining the basics of link-state routing it was time for implementation considerations including:

• Collecting local endpoint reachability information
• Finding neighbors and exchanging the collected information (hint: a link-state topology database is just a distributed key-value store)
• Running the SPF algorithm (including partial SPF details) and installing the results

One of my readers sent me this (paraphrased) question:

What I have seen in my network are multicast packets with the IP source address set to 0.0.0.0 and source port set to 0. Is that considered acceptable? Could I use a multicast IP address as a source address?

TL&DR: **** NO!!!

It also seemed like a good question to test ChatGPT, and this time it did a pretty good job.

So far in this series we’ve discussed the history of the IETF, some of the tools you might want to use when building an IETF submission, and document formatting. There are other seemingly mystical concepts in the IETF process as well—for instance, what is a “document stream,” and what is a document’s “status?” Let’s look […]

The post Writing An IETF Draft: Document Streams And Document Status appeared first on Packet Pushers.

Years ago I’ve been involved in an interesting discussion focusing on NTP authentication and whether you can actually implement it reliably on Cisco IOS. What I got out of it (apart from a working example) was the feeling that NTP and it’s implementation in Cisco IOS was under-understood and under-documented, so I wrote an article about it. Of course the web version got lost in the mists of time but I keep my archives handy.

Last weekend I migrated that article to blog.ipSpace.net. I hope you’ll still find it useful; while it’s pretty old, the fundamentals haven’t changed in the meantime.

TL&DR: Installing an Ethernet NIC with two uplinks in a server is easy¹. Connecting those uplinks to two edge switches is common sense². Detecting physical link failure is trivial in Gigabit Ethernet world. Deciding between two independent uplinks or a link aggregation group is interesting. Detecting path failure and disabling the useless uplink that causes traffic blackholing is a living hell (more details in this Design Clinic question).

Want to know more? Let’s dive into the gory details.

I joined Twitter in October 2008 (after noticing everyone else was using it during a Networking Field Day event), and eventually figured out how to automate posting the links to my blog posts in case someone uses Twitter as their primary source of news – an IFTTT applet that read my RSS feed and posted links to new entries to Twitter.

This week, I got a nice email from IFTTT telling me they had to disable the post-to-Twitter applet. Twitter started charging for the API, and I was using their free service – obviously the math didn’t work out.

That left me with three options:

Before we managed to recover from the automation cargo cults, a tsunami wave of cargo cult AI washed over us as Edlyn V. Levine explained in an ACM Queue article. Enjoy ;)

Also, a bit of a historical perspective is never a bad thing:

Impressive progress in AI, including the recent sensation of ChatGPT, has been dominated by the success of a single, decades-old machine-learning approach called a multilayer (or deep) neural network. This approach was invented in the 1940s, and essentially all of the foundational concepts of neural networks and associated methods—including convolutional neural networks and backpropagation—were in place by the 1980s.

Bruce Schneier wrote an excellent essay explaining why we need trustworthy AI and why we won’t get it as long the AI solutions are created by large tech companies with you are a product business model.

Sometime last autumn, I was asked to create a short “network security challenges” presentation. Eventually, I turned it into a webinar, resulting in almost four hours of content describing the interesting gotchas I encountered in the past (plus a few recent vulnerabilities like turning WiFi into a thick yellow cable).

Each webinar section started with a short “This is why we have to deal with these stupidities” introduction. You’ll find all of them collected in the Root Causes video starting the Network Security Fallacies part of the How Networks Really Work webinar.

As technical people, we spend immense time and energy mastering the nuances of specific technologies. Esoteric knowledge is our currency, and we often measure our personal value against the yardstick of technical nuance. And sometimes (maybe lots of times) we gauge other people with the same yardstick, and dismiss those who don’t measure up. This […]

The post People Aren’t Stupid Just Because They Don’t Understand Tech appeared first on Packet Pushers.

Previous posts in this series covered numerous intricacies of DHCP relaying:

• DHCP relaying principles described the basics
• In Inter-VRFs relaying we figured out how a DHCP client reaches a DHCP server in another VRF without inter-VRF route leaking
• Relaying in VXLAN segments and relaying from EVPN VRF applied those lessons to VXLAN/EVPN environment.
• DHCP Relaying with Redundant DHCP Servers added relay- and server redundancy.

Now for the final bit of the puzzle: what if we want to do inter-VRF DHCP relaying with redundant DHCP servers?

Container Network Interfaces (CNIs) are plug-ins that enable networking capabilities. This video provides a brief overview of the Cillium CNI and the importance of network policies. https://www.youtube.com/watch?v=nzswIJpdPtY You can subscribe to the Packet Pushers’ YouTube channel for more videos as they are published. It’s a diverse a mix of content from Ethan and Greg, plus […]

The post Kubernetes Security And Networking 8: Loading The Cillium CNI – Video appeared first on Packet Pushers.

Sebastian described an interesting Cisco ACI quirk they had the privilege of chasing around:

We’ve encountered VM connectivity issues after VM movements from one vPC leaf pair to a different vPC leaf pair with ACI. The issue did not occur immediately (due to ACI’s bounce entries) and only sometimes, which made it very difficult to reproduce synthetically, but due to DRS and a large number of VMs it occurred frequently enough, that it was a serious problem for us.

Here’s what they figured out:

Last month I described how you can simplify your VLAN- or VRF lab topologies with VRF- and VLAN links, automatically setting vlan.access or vrf attribute on a set of links. Link groups allow you to do the same for any set of link attributes.

Sample Topology

Imagine you have a small network with three PE-routers connected to a central P-router:

Michele Chubirka published a must-read article on technology fallacies including this gem:

Technologists often assume that all problems can be beaten into submission with a technology hammer.

As I’ve been saying for ages (not that anyone would listen): all the technology in the world won’t save you unless you change the mentality and rearchitect broken processes.

I mentioned IP source address validation (SAV) as one of the MANRS-recommended actions in the Internet Routing Security webinar but did not go into any details (as the webinar deals with routing security, not data-plane security)… but I stumbled upon a wonderful companion article published by RIPE Labs: Why Is Source Address Validation Still a Problem?.

The article goes through the basics of SAV, best practices, and (most interesting) using free testing tools to detect non-compliant networks. Definitely worth reading!

Pete Lumbis concluded his ASICs for Networking Engineers presentation with a brief overview of types of switching ASICs and a wrap-up.

You can watch his entire 90-minute presentation (sliced into shorter videos) with Free ipSpace.net Subscription.

Tom Ammon sent me his thoughts on choosing the right level of abstraction in your network automation solution as a response to my What Is Intent-Based Networking blog post, and allowed me to publish them on ipspace.net.

I totally agree with your what vs how example with OSPF. I work on a NOS team where if we wanted, we could say, instead of “run OSPF on these links”, do this:

Messy RADIUS policies and misconfigurations may be allowing users to join personal devices to your network. Jennifer Minella provides a quick overview of RADIUS and 802.1x, common holes, and three options for filling them in this installment of her "Ask JJX" series.

The post Ask JJX: How Can I Stop Users From Joining Personal Devices To Our Network Using Their AD Credentials? appeared first on Packet Pushers.

There’s lot of places to focus on application security, but don’t forget to scan your Kubernetes manifests! This video takes you step-by-step through scanning your repository using Kubescape. https://www.youtube.com/watch?v=kwF-JoIQRTA You can subscribe to the Packet Pushers’ YouTube channel for more videos as they are published. It’s a diverse a mix of content from Ethan and […]

The post Kubernetes Security And Networking 7: Securing Kubernetes Manifests – Video appeared first on Packet Pushers.

The following sponsored blog post was written by Shankar Ramachandran at Palo Alto Networks. We thank Palo Alto Networks for being a sponsor. Internet of things (IoT) devices are now an integral part of any organization’s network. Smart lights, cameras, card readers, printers, etc., are critical to the day-to-day operations of branch offices and retail […]

The post Introducing SD-WAN With Integrated IoT appeared first on Packet Pushers.

Remember when Aruba was a wireless company? I know it sounds like something that happened 40 years ago but the idea that Aruba only really made wireless access points and some campus switches to support them isn’t as old as you think. The company, now known as HPE Aruba Networking (née Aruba, a Hewlett Packard Enterprise Company), makes more than just Wi-Fi gear. Yet the perception of the industry is that they’re still a wireless company looking to compete with the largest parts of the market.

Branching Out of Office

This year’s Aruba Atmopshere showed me that Aruba is trying to do more than just campus wireless. The industry has shifted away from just providing edge connectivity and is now focused on a holistic lineup of products that are user-focused. You don’t need to go much further than the technical keynote on the second day of the conference to see that. Or the Networking Field Day Experience videos linked above.

Do you know what Aruba wanted to showcase?

• Campus Switches
• Data Center Switches
• Private 5G/LTE
• SASE/SSE
• IoT
• Cloud-Enabled Management

You know what wasn’t on that list? Access points. For a “wireless” company that’s a pretty glaring omission, right? I think it’s actually a brilliant way to help people understand that HPE Aruba Networking is a growing part of the wider HPE business dedicated to connectivity.

It’s been discussed over the years that the HPE acquisition of Aruba was a “reverse acquisition”. That basically means that HPE gave Aruba control over their campus (and later data center) networking portfolio and let them run with it. It was successful and really helped highlight the needs that HPE had in that space. No one was talking about the dominance of Procurve switches. HPE was even reselling Arista gear at the time for the high end customers. Aruba not only was able to right the ship but help it grow over time and adopt home-grown offerings.

When you think of companies like Juniper and Cisco, do you see them as single product vendors? Juniper makes more than just service provider routers. Cisco makes more than just switches. They have distinct lines of business that provide offerings across the spectrum. They both sell firewalls and access points. They both have software divisions. Cisco sells servers and unified communications gear on top of everything else they do. There’s more to both of them than meets the eye.

Aruba needed to shed the wireless moniker in order to grow into a more competitive market segment. When you’re known as a single product vendor you tend to be left out of conversations. Would you call Palo Alto for switches or wireless? No, because they’re a firewall or SASE company. Yes, they make more than those products but they have a niche, as opposed to more diverse companies. I’m not saying Palo Alto isn’t diverse, just that they define their market segment pretty effectively. So much so that people don’t even call application firewalls by that name any longer. They’re “Palo Altos”, giving the company the same generic trademark distinction as Kleenex and Velcro.

User Face-to-Face

Aruba needs to develop the product lines that help get users connected. Wireless is an easy layup for them now so where do they expand? Switches are a logical extension so the CX lines were developed and continue to do well. The expansion into private LTE and security also help significantly, which are bolstered by their recent acquisitions.

Security is an easy one to figure out. Aruba has gone from SD-Branch, focused on people working in remote offices, to add on true SD-WAN functionality with the Silver Peak purchase, to now offering SSE with Axis Security being folded in to the mix. SSE is a growing market segment because the services offered are what users consume. SASE works great if you’re working from home all the time. In the middle of the pandemic that was a given. People had home offices and did their work there.

But now that restrictions are relaxed and people aren’t going into the office all the time. This hybrid work model means no hardware to do the inspection. Since SSE is not focused on hardware it’s a great fit for a mobile hybrid workforce. If you remember how much Aruba was touting the BYOD wireless-only office trend back in 2016 and 2017 you can see how SSE would have been a wonderful fit back then if it had existed. Given how the concept of a wireless-only BYOD office was realized through not having an office I’d say SSE is a perfect fit for the modern state of the enterprise.

Private 5G is a bit more complicated. Why would Aruba embrace a technology that effectively competes with its core business? I’d say that’s because they need to understand the impact that private cellular will have on their business. People aren’t dumping Wi-Fi and moving en masse to CBRS. We’ve reached a point where we’re considering what the requirements for private LTE deployments need to look like and where the real value lies for them. If you have a challenging RF environment and have devices capable of taking SIM cards it makes a lot of sense. Aruba having a native way of providing that kind of connectivity for users that are looking to offer it is also a huge win. It’s also important to note that Aruba wants to make sure it has complete control over the process, so what better way than acquiring a mature company that can integrate into their product lines?

Tom’s Take

I can’t take full credit for this idea. Avril Salter pointed it out during a briefing and I thought it was a wonderful point. Aruba isn’t a wireless company now because they’ve grown to become a true networking company. They offer more than just APs and devices that power them. There have a full line of products that address the needs of a modern user. The name change isn’t just a branding exercise. It represents a shift in the way people need to see the company. Growing beyond what you used to be isn’t a bad thing. It’s a sign of maturity.

In 2012, we saw the launch of Viptela, a pioneer in SDWAN network solutions. While they weren’t the first in SDWAN, I believe that badge goes to Talari; Viptela was the first company that caught my interest. I first saw what they were doing in 2015 on the Packet Pushers Podcast. Back then, the networking world was on fire with SDWAN offerings.

Sadly in 2017, Cisco purchased Viptela. I’m sure the leadership at Viptela was excited to be acquired by such a large networking company and hoped to develop Viptela to the pinnacle of their vision. But – despite calling themselves, and being repeatedly lamented for doing so, “a software company”, Cisco did what they so often do. They stopped platform development and tried to lower costs by integrating the vEdge software into their own. While I haven’t personally used the hybrid code, I haven’t heard good things from my coworkers. Eventually, we got the ISR1100 platform running Viptela code, but that was only a substitute for the vEdge 100 and 1000. We have still yet to see a replacement for the vEdge 2000.

May 9, 2023 – 6:57 AM UTC

certificate-status Installed
certificate-validity Not Valid – certificate has expired <<<<<<<<<<<<<<<

If you have a vEdge 100, vEdge 1000, or vEdge 2000 – you probably already know. But any of the following will result in a loss of service.

• Loss of connections to vSmart
• Loss of connections to vManage
• Port-Hop
• Control policy changes such as topology changes in the network
• Clear control connection
• Interface Flaps
• Device Reload

Cisco has published several workarounds, from increasing rekey values, to changing dates, and who knows what else. Of course, these are short-term solutions; only a certificate update (which will require new software) can fix this issue. Meanwhile, Cisco’s customers are left scrambling and trying to reach TAC. Internet rumors suggest that everyone working the front lines at TAC had at least 20-25 customers in their queue. I’m sure Cisco is glad they recently laid off significant numbers of support staff…

My take

Cisco should have learned its certificate lessons in 2015 when WLCs invalidated APs due to a 10-year certificate expiring. I’m pretty sure their workaround was disabling the security check… Interestingly, just two years after that, those certificates were extended to 20 years. Cisco increased the expiration again in 2019, extending them to 2099.

Cisco still needs to release an official RCA, but I have my own ideas. Viptela launched in 2012; one could surmise that these initial vEdge hardware platforms launched in 2013, Maybe around May 9th. – Yeah, a 10-year certificate mismanaged at Cisco (again). Talking with folks at Graphiant, it seems many of Viptela’s original leadership were quite disappointed in how their product was handled at Cisco post-acquisition. That certainly explains why most of the Viptela team left for other ventures.

First thing first; full disclosure; Readers should know I am an employee of Oracle cloud, although since my teams have nothing to do with the certification teams, this blog will demonstrate a candidate’s perspective. Last week, pretty much as soon as the first Oracle Cloud Infrastructure (OCI) multicloud exam became available, I was excited to […]

The post My Notes on OCI’s Multicloud Certification Exam appeared first on Packet Pushers.

This post originally appeared in the Packet Pushers’ Human Infrastructure newsletter. You can subscribe for free here. We never share or sell your details to anyone. One unforeseen event following the COVID-19 pandemic has been an uptick in attempts to organize workers. Starbucks and Amazon warehouse employees are two high-profile examples. Though private-sector union membership […]

The post The Case For IT Unionization appeared first on Packet Pushers.