I like using Cisco IOS for my routing protocol virtual labs¹. It uses a trivial amount of memory² and boots relatively fast. There was just one thing that kept annoying me: Cisco IOS release 15.x takes forever to install local routes in the BGP table and even longer to select the best routes and propagate them³.

I finally found the culprit: bgp update-delay nerd knob. Here’s what the documentation has to say about it:

Maybe it’s just me, but I always need a few extra devices in my virtual labs to have endpoints I could ping to/from or to have external routing information sources. We used VRF- and VLAN tricks in the days when we had to use physical devices to carve out a dozen hosts out of a single Cisco 2501, and life became much easier when you could spin up a few additional virtual machines in a virtual lab instead.

Unfortunately, those virtual machines eat precious resources. For example, netlab allocates 1GB to every Linux virtual machine when you only need bash and ping. Wouldn’t it be great if you could start that ping in a busybox container instead?

Robert Graham published a blog post describing how his IDS/IPS system handled 2 Mpps on a Pentium III CPU 20 years ago… and yet some people keep claiming that “Driving a 100 Gbps network at 80% utilization in both directions consumes 10–20 cores just in the networking stack” (in 2023). I guess a suboptimal-enough implementation can still consume all the CPU cycles it can get and then some.

As a network operator, I want to describe in plain language what I need a network to do, and the network is configured accordingly. Then I want the network to monitor itself, and when things aren’t going well, the network will repair itself with no involvement from me. Hey, daydreaming is fun.

In the real world, plain language describing my network requirements isn’t going to conjure a relevant network. I must perform hard work to create a network design that’s useful for a business. I have to think through issues like capacity needs under peak load, redundancy to survive a network failure, and resiliency to support business operations in the face of a catastrophic outage. I need to understand individual application requirements, and be sure the network can support those requirements. I have to consider modularity, repeatability, and supportability. I must work within a budget.

My design will translate into an arcane collection of devices, interfaces, interconnections, protocols, and topologies. I’ll rely on education, experience, and experimentation to fine-tune the design, and then I’ll put it into production. Depending on your personality, this arduous task likely falls somewhere between “fun” and “frightening” for you. But no matter who you are, there is an element of tedium that goes into complex network design & engineering.

That’s right. Tedium. Much detail goes into configuring a network to do much of anything. For me, getting details wrong is easy. An incorrect subnet mask. Forgetting to adjust MTU on an interface. Referencing a prefix list I haven’t actually created. Applying a QoS policy in the wrong direction. Setting a router ID but neglecting to restart the routing process.

If I bang away at the configs long enough, everything falls into place. By the time the design goes into production, it’s more or less right. Sure, maybe I need to do some cleanup, but the system is working as designed.

If I may be so bold, the process of network design and implementation I’m describing is The Old Way. As far as I’m concerned, the old way sucks. I want to automate as much of the old way as I can. While acknowledging that I will need to be in the loop to some degree (hey, a design has to happen, right?), I want to be out of the configuration loop as much as possible.

Configuration Is Just Details

A key idea I was taught many years ago was that configuration is just details. I didn’t grasp the importance of this at the time, because everything I’d been taught in a classroom was all about configuration. Configuration just details? All the vendor certification exams I’d passed suggested that configuration was the point. I chafed at the contradictory point the senior network designer was making.

What I was missing is that the strength of a network design is not in its configuration, but in its architecture. Put another way, the configuration that brings a design from the whiteboard and to a living, breathing architecture is merely a collection of details. Important details? Yes, but ask yourself this. Is the functioning of a network designed for a specific purpose any different because it came from Vendor C instead of Vendor A or Vendor J? Yes, the configuration will differ among the vendors, but the result will be the same.

I don’t want to have to care about configuration. I just want a network service to function as designed–and stay functioning.

Closed Loop Automation

In data networking, closed loop automation contains the following big ideas.

1. A network service is deployed in an automated way.
2. That network service is actively monitored.
3. The monitoring ensures that the service is meeting a set of service level agreements (SLAs).
4. Automation can update a network service if it is not meeting SLAs requirements, thus closing the loop.

In my view, closed loop automation moves me closer to my goal of not having to worry about configuration details.

Various vendors are offering closed loop automation tools today. Let’s take Anuta Networks’ Active Service Assurance (ASA). ASA is a relatively recent addition to Anuta’s ATOM platform. If you don’t know Anuta, they offer network automation aimed at service providers and large enterprises acting as service providers. Create a service, and ATOM will deploy it for you in a vendor agnostic way.

ASA adds network monitoring to know that a deployed service is or is not meeting the required service level agreement (SLA). If an SLA is not being met, ASA can modify the configuration to bring the service back into compliance.

Now, that all sounds lovely, if not downright magical. But the reality of what’s going on is more mundane and human-derived. For example, network services are defined by humans. The Anuta platform makes service definition a workflow-driven process and abstracts configuration details away, but a human is defining that workflow. The Anuta platform’s ASA will also monitor the service, but a human is describing the tests that need to be run. ASA will also attempt remediation, but the remediation actions that are possible are also defined by a human via an Anuta workflow.

This is not a criticism. This system is state of the art, and is among the best I’ve seen demonstrated in its flexibility and user interface. That flexibility also makes the platform non-trivial to make use of. I can’t just say, “Hey, Anuta. Make me an L3VPN,” and poof, magic happens. I have to define an L3VPN as a service. I have to establish the tests I care about that tell me an SLA is or is not being met. I must define what the options are such as rollback to restore a service to SLA compliance.

There is no network automation, closed loop or otherwise, I’ve seen that doesn’t require a significant upfront investment before that tool is useful. That’s certainly true for complex network configuration tasks with unique business requirements or specialized capabilities. I’m not suggesting there aren’t quick wins to be had with network automation tools. Many tools can help with things like NOS version management, config diffs, vulnerability assessments, and similar low hanging fruit shortly after network discovery is complete.

Closing The Loop Is What We’ve Been Missing All Along

Upfront investment notwithstanding, don’t think I’m dismissing closed loop automation as uninteresting. The idea is keenly interesting. Closed loop automation puts me, the network operator, one step further away from those pesky and easy to get wrong configuration details I don’t want to have to deal with. Importantly, closed loop automation also integrates monitoring into the network service.

Historically, networking has treated monitoring as a separate set of tools from configuration. In fact, I’ve always wanted a service to have self-monitoring as an integral part. I didn’t know that’s what I wanted. I’m used to building a service, then configuring a network management station, probably combined with application performance monitoring, to track service availability and performance. Combining the configuration & monitoring into one “closed loop automation” service means I’m going to be asking the right question.

The wrong question is, “Is the network service running?”

The right question is, “Is the network service running well?”

Frankly, having to design the tests that tell me with confidence that the service is running well is a big step in the right direction. Perhaps I can’t say, “Anuta, make me a network & make sure it’s running well.” But I can see the framework coming together where maybe, someday, I’ll be able to do just that.

For More Information

Anuta Networks presented at Networking Field Day 30, where I attended as a delegate. Find Anuta’s presentations in this YouTube playlist.

It’s Always the Network is a refrain that causes operations teams to shudder. No matter what your flavor of networking might be it’s always your fault. Even if the actual problem is DNS, a global BGP outage, or even some issue with the SaaS provider. Why do we always get blamed? And how can you prevent this from happening to you?

User Utopia

Users don’t know about the world outside of their devices. As soon as they click on something in a browser window they expect it to work. It’s a lot like ordering a package and having it delivered. It’s expected that the package arrives. You don’t concern yourself with the details of how it needs to be shipped, what routes it will take, and how factors that exist half a world away could cause disruptions to your schedule at home.

The network is the same to the users. If something doesn’t work with a website or a remote application it must be the “network” that is at fault. Because your users believe that everything not inside of their computer is the network. Networking is the way that stuff happens everywhere else. As professionals we know the differences between the LAN, the WAN, the cloud, and all points in between. However your users do not.

Have you heard about MTTI? It’s a humorous metric that we like to quote now that stands for “mean time to innocence”. Essentially we’re saying that we’re racing to find out why it’s not our problem. We check the connections, look at the services in our network, and then confidently explain to the user that someone else is at fault for what happened and we can’t fix that. Sounds great in theory, right?

Would you proudly proclaim that it’s someone else’s fault to the CEO? The Board of Directors? How about to the President of the US? Depending on how important the person you are speaking to might be you might change the way you present the information. Regular users might get “sorry, not my fault” but the CEO could get “here’s the issue and we need to open a ticket to fix it”. Why the varying service levels? Is it because regular users aren’t as important as the head of the company? Or is it because we don’t want to put in the extra effort for a knowledge workers that we would gladly do for the person that in ultimately going to approve a raise for us if we do extra work? Do you see the fallacy here?

Keeping Your Eyes Peeled

The answer, of course, is that every user at least deserves an explanation of where the problem is. You might be bent on justifying that it’s not in your network so you don’t have to do any more work but you also did enough work to place the blame outside of your area of control. Why not go one step further? You could check a dashboard for services on a cloud provider to see if they’re degraded. Or do a quick scan of news sites or social media to see if it’s a more widespread issue. Even checking a service to see if the site is down for more than just you is more information than just “not my problem”.

The habit of monitoring other services allows you to do two things very quickly. First is that it lowers the all-important MTTI metric. If you can quickly scan the likely culprits for outages you can then say with certainty that it’s out of your control. However, the biggest benefit to monitoring services outside of your organization that you rely on is that you’ll have forewarning for issues in your own organization. If your users come to you and say that they can’t get to Microsoft 365 and you already know that’s because they messed up a WAN router configuration you’ll head off needless blame. You’ll also know where to look next time to understand challenges.

If you’re already monitoring the infrastructure in your organization it only makes sense to monitor the infrastructure that your users need outside of it. Maybe you can’t get AWS to send you the SNMP strings you need to watch all their internal switches but you can easily pull information from their publicly available sources to help determine where the issues might lie. That helps you in the same way that monitoring switch ports in your organization helps you today. You can respond to issues before they get reported so you have a clear picture of what the impact will be.

Tom’s Take

When I worked for a VAR one of my biggest pet peeves was “not my problem” thinking. Yes, it may not be YOUR problem but it is still A problem for the user. Rather than trying to shift the blame let’s put our heads together to investigate where the problem lies and create a fix or a workaround for it. Because the user doesn’t care who is to blame. They just want the issue resolved. And resolution doesn’t come from passing the buck.

Matthias Luft concluded his part of Introduction to Cloud Computing webinar with a case study: how can you migrate an existing workload into a cloud environment?

The simplest way to implement layer-3 forwarding in a network fabric is to offload it to an external device¹, be it a WAN edge router, a firewall, a load balancer, or any other network appliance.

Routing at the (outer) edge of the fabric

There’s a trend to cloudify every aspect of IT. You might think that there’s no need to understand classical networking because “the cloud” solves problems related to routers and switches. The truth is that clouds have massive networks under the hood. Someone needs to connect cloud services and bring users to these clouds in a […]

The post A Journey To The AWS Advanced Networking Certification appeared first on Packet Pushers.

David Gee couldn’t resist making a few choice comments after I asked for his opinion of an early draft of the Network Automation Expert Beginners blog post, and allowed me to share them with you. Enjoy 😉

Network automation offers promises of reliability and efficiency, but it came without a warning label and health warnings. We seem to be perpetually stuck in a window display with sexily dressed mannequins.

There’s a lot of hype around both Kubernetes and edge computing, so it shouldn’t be a surprise that vendors and cloud providers are offering products and services that combine the two. But what is edge computing? And can you run Kubernetes at the edge?

The post Kubernetes At The Edge – Is It Really A Thing? appeared first on Packet Pushers.

I decided to stop caring about IPv6 when the protocol became old enough to buy its own beer (now even in US), but its second-system effects keep coming back to haunt us. Here’s a question I got for the February 2023 ipSpace.net Design Clinic:

How can we do IPv6 networking in a small/medium enterprise if we’re using multiple ISPs and don’t have our own IPv6 Provider Independent IPv6 allocation. I’ve brainstormed this with people far more knowledgeable than me on IPv6, and listened to IPv6 Buzz episodes discussing it, but I still can’t figure it out.

If you recently updated your Mac to Ventura 13.1 or 13.2, and you had installed Wireshark previously, then you may be having some trouble. If you open Wireshark, you will likey see the message “You don’t have permission to capture on local interfaces” and “You can fix this by installing ChmodBPF“. Even after installing this […]

The post MacOS Ventura 13.1 Breaks Wireshark appeared first on Packet Pushers.

netlab release 1.5.0 includes features that will help you start very large lab topologies (someone managed to run over 90 Mikrotik routers on a 24-core server):

• You can start libvirt virtual machines in batches to reduce the CPU overload that causes startup failures on large topologies.
• You can combine virtual machines and containers in the same lab, further reducing the memory footprint for devices available as true containers (Linux hosts, Cumulus/FRR routers, Arista cEOS)
• Use custom management network IP subnet if you’re running out of management IP addresses

To get more details and learn about additional features included in release 1.5.0, read the release notes. To upgrade, execute pip3 install --upgrade networklab.

New to netlab? Start with the Getting Started document and the installation guide.

Julia Evans published another fantastic must-read article: a debugging manifesto. Enjoy ;)

Did you ever wonder why it’s impossible to find good service company, why most software sucks, or why networking vendors can get away with selling crap? If you did, and found no good answer (apart from Sturgeon’s Law), it’s time to read Why is it so hard to buy things that work well? by Dan Luu.

Totally off-topic: his web site uses almost no CSS and looks in my browser like a relic of 1980s. Suggestions how to fix that (in Chrome) are most welcome.

The other day I was listening to an excellent episode of The Art of Network Engineering talking about technical marketing engineers (TME). The discussion was excellent and there was one line from Pete Lumbis in the episode that stuck with me. He said that one of the things that makes you good as a TME is being an “expert beginner”. That phrase resonates at lot with me.

Fresh Eyes on the Problem

I talked a bit about this last year when I talked about being a beginner and how exciting that it was to start over with something. As I compared that post to the AONE episode I realized that what Pete was talking about was a shift in mindset that gives you the energy and focus to pick things up quickly.

You may have heard the phrase “familiarity breeds contempt”. It’s a common phrase used to describe how we feel less impressed with things the more we learn about then. Our brains are wired to enjoy new things. We love new experiences, going to new places, or even meeting new people. The excitement and rush that we get from something unfamiliar causes our brain to devour things. It’s only once we become familiar with them that we feel the contempt set in.

It’s not always possible to avoid the contempt part of things. Think about something as dreary as your morning commute to work, whether it’s walking down the stairs or driving to the office. I used to joke that my car was practically on auto-pilot most mornings because I knew every bump in the road and every turn by heart. When I would go somewhere new I would have to focus more on road signs or directions.

Back to Basics

The beginner aspect of things is easier to deal with. That’s because we can trick ourselves into seeing something with fresh eyes. On a number of occasions I’ve mentioned my friend and mentor John Pross and his assertion that every upgrade or deployment that happened was like the first time doing it. He never took anything for granted. He always took things step-by-step. While this had the affect of him making sure everything was followed to the letter it also gave him the beginner aspect of looking for ways to improve or discovering new solutions to problems along the way.

Once the contempt or apathy sets in you’re going to get very good at just clicking through the steps to get to the end as fast as possible. If you don’t believe me think about how many times you’ve given directions that involve something like “Just click Next, Next, Next, Next and then you’re done”. Trust me, it sounds funnier when you say it out loud. But it speaks to the fact that we know the dialog boxes so well that we know they aren’t important. But what if they are?

If you want to understand what it feels like to be a beginner again and you’re having a hard time getting yourself in that mindset you should find a beginner and coach them through a task like a setup. Don’t just tell them what to do. Let them figure it out. Answer questions as they come up. Make them explain why they’re doing something. I bet you’ll learn a lot more as you have to help them understand why that configuration line is in there or why you always choose twice the amount of RAM in an instance. Once you see the process through the eyes of a beginner you have to learn it more completely in order to help them understand it.

In some roles, like a TME or a VAR engineer, the ability to be an expert beginner is critical to your job. You have to see a technology for the first time and pick up the basics quickly. I used to tell people that the excitement of being an engineer at a VAR was the variety of problems I’d be called on to solve. One day might be wireless clients. The next could be iSCSI storage arrays. Whatever the case may be I could count on finding myself in a new situation pretty regularly. It kept things exciting and made me realize I had to stay on my toes.

For those that work as product managers or on more specialized teams you need to make sure you’re taking time to approach things as a beginner. The “same old, same old” may not actually be the same any more. That kind of contempt and familiarity leads to the phrase “the way we’ve always done it” and doesn’t force you to challenge the process to understand how to improve it. Sometimes you need to step back and remember that you have to see everything for the first time.

Tom’s Take

Beginners shouldn’t feel like they’re a nuisance. In fact they should be celebrated for the energy and focus they bring to a task or project. For roles like a TME it’s important to bring the same kind of energy to new things. You can learn a lot when you allow your brain to soak up knowledge like a fresh sponge. More importantly, the ability to be a beginner helps you refine your knowledge base more and will ensure that you can explain a concept or process to someone with absolute certainty.

I had a lovely chat with David Gee on his built.fm podcast sometime in December. David switched jobs in the meantime, and so it took him a bit longer than expected to publish it. Chatting with David is always fun; hope you’ll enjoy our chat as much as I did.

This series started by discussing the history of the IETF and some of the tools you might use to build submissions to the IETF process. This, the second, post, will consider document formatting and two of the (sometimes) more difficult sections of an IETF draft to fill in. Formatting Just using one of the acceptable […]

The post Writing An IETF Draft: Formatting, Authorship, And Submissions appeared first on Packet Pushers.

A random tweet¹ pointed me to Vulnerability Note VU#855201 that documents four vulnerabilities exploiting a weird combination of LLC and VLAN headers can bypass layer-2 security on most network devices.

The security researcher who found the vulnerability also provided an excellent in-depth description focused on the way operating systems like Linux and Windows handle LLC-encapsulated IP packets. Here’s the CliffNotes version focused more on the hardware switches. Even though I tried to keep it simple, you might want to read the History of Ethernet Encapsulation before moving on.

Let’s see. We’ve queried stuff on Netbox and added stuff to Netbox. Now let’s update stuff.

Netbox, like all sources of truth, needs to be kept up-to-date if it’s going to be useful. Without doing some maintenance on the data, it will wind up being like that one Visio diagram that you give the auditors — it might have been accurate at one point but gets further and further from the truth every day. We’ll need to keep our stuff updated today in order to use it more effectively tomorrow.

As a warning to everyone, I am not a developer. I am a network engineer who is trying to do some automation stuff. Some of what I’m doing sounds logical to me, but I would not trust my own opinions for production work. I’m sure you can find a Slack channel or Mastodon instance with people who can tell you how to do things properly.

We’re going to again use Python and pynetbox for this (as the title says). Here’s the environment I’m working in.

Python         :  3.9.10 
Pynetbox       :  7.0.0  
Netbox version :  3.4.3 (Docker)

Remember when we loaded the data from the sites.yml file last time? We’re going to use that same file to run another script that will update existing information. This time, the script will check Netbox for some site values and updated it if it doesn’t match the YAML file. Here we go. As always, these scripts and YAML files are available in my Github repository.

### pynetbox_update_sites.py
import pynetbox
import yaml

ENV_FILE = "env.yml"
SITES_FILE = "sites.yml"

with open(ENV_FILE) as file:
    env_vars = yaml.safe_load(file)
    
with open(SITES_FILE) as file:
    sites_to_load = yaml.safe_load(file)
    
nb_conn = pynetbox.api(url=env_vars['netbox_url'])

token = nb_conn.create_token(env_vars['username'], env_vars['password'])

for site in sites_to_load:
    are_they_different = False
    print(f"Checking {site['name']} for updates...", end="")
    queried_site = nb_conn.dcim.sites.get(name=site['name'].upper())
    if not queried_site:
        print(f"Site {site['name']} does not exist. I'm choosing not to add it.")
        continue
    for key in site.keys():
        if site[key] != queried_site[key]:
            are_they_different = True
            print("looks like it's different. Will update.")
    if are_they_different:
        queried_site.update(site)
    else:
        print("seems to be the same.")
        continue
    
token.delete()

All the way down to line 15 should be pretty familiar already. Check out the last few posts to get caught up.

Line 17 goes through all the sites in the YAML so we can do stuff.

Line 18 sets a boolean variable called are_they_different to track if we need to do the update or not. We could just blindly update the object, but it seems a bit inefficient if the data is the same.

Lines 20 – 23 check to make sure the site actually exists. If it doesn’t, print a message and skip it. We’ll use that queried site object here in a bit to compare against the YAML.

I’m having trouble wording an explanation for lines 24 – 27. We first take the keys for the dictionary that we imported from YAML and go through each of them. If the value for that key in the Netbox object is different than the value for the same key in the YAML file, then we’ll set that boolean variable to True. If they’re the same, nothing will happen.

Lines 28 – 29 check to see if we need to do an update and then do it if needed. We’re done .all(), .get(), .filter(), and .create() (and even .delete() if you count the token thing), but this is the first time we’re doing an .update(). In this case, we’re taking the queried_site object and updating it with the data that came from the YAML. Any values that are different get updated.

Lines 30 – 32 tell the user nothing is happening since the values match.

Line 34 nukes the token we created in line 15.

Is this horrible code or what? We could probably take the YAML, do some value validation, then just update the object without all this frilly stuff. I mean, this isn’t our production database that’s taking 65k connection per second, so we’re probably not bogging down the Netbox server with additional updates. Also, the populate script and this update script should be one and the same. We would just load everything up from file, add things that needed to be added, and update things that needed to be updated. See again the note about me not knowing what I’m talking about. LOL

Watch out for keys in the YAML file. If you import a key that doesn’t match a valid Netbox field, then you’ll get an exception either from the comparison (the key doesn’t exist in the Netbox object, so KeyError) or the update (you can’t update the sreail_mun field, so RequestError). You also need to make sure the field is of the correct type; you can’t pass a string when Netbox is expecting an ID. You’ll need to do some validation to make sure you’re not going to get yourself in trouble later.

The script works. That’s fine, but all we’ve done is move the task of updating the data from Netbox to the YAML file. Someone still has to maintain the data no matter where it lives. It would be pretty cool if we had something to automagically go out into the network and get the data we need to update Netbox. We can definitely do that, but let’s start simple and just update serial numbers.

Where do the serial numbers live? Well, on the devices themselves. We’ll need to log into them — usually with SSH — to scrape that data. For SSH-enabled devices, we’ll use Netmiko to log in, run a command that shows the serial number, and update Netbox if needed. At home, the only device I have that runs SSH is a Mikrotik hAP AC3, so we’ll just act like this is the Internet router in Phoenix. If you’re interested in Netmiko and much-better Python than I would ever generate, make sure you take Kirk Byers course on Python for Network Engineers — this is very much worth your time if you’re just getting started in Python.

We have yet another YAML file with the IP information for the devices…and another one with the credentials to use to log in. This is pretty much the worst way to do this. The IP information should already be in Netbox, so just get it from there. The creds should be in a vault of some kind and not in a YAML file that you’ll wind up publishing on a public GitHub repo accidentally. This is a lab, though, so we’ll just do it this way for now. This sounds like more topics for later, doesn’t it?

The device YAML files contains a list of devices to check with name and mgmt_ip.

### devices_to_update.yml
- name: PHX-RTR01
  mgmt_ip: 172.22.0.1

The credentials YAML file is just username and password. I’m not going to publish my version for security’s sake.

Alright. Code.

### pynetbox_update_device_serial.py
import pynetbox
import yaml
from netmiko import ConnectHandler
import re

ENV_FILE = "env.yml"
DEVICES_FILE = "devices_to_update.yml"
DEVICE_CREDS_FILE = "device_creds.yml"

def load_env_vars():
    with open(ENV_FILE) as file:
        return yaml.safe_load(file)

def load_devices():
    with open(DEVICES_FILE) as file:
        return yaml.safe_load(file)
    
def load_device_creds():
    with open(DEVICE_CREDS_FILE) as file:
        return yaml.safe_load(file)

env_vars = load_env_vars()
devices_to_update = load_devices()
device_creds = load_device_creds()

nb_conn = pynetbox.api(url=env_vars['netbox_url'])
token = nb_conn.create_token(env_vars['username'], env_vars['password'])

for device in devices_to_update:
    print(f"Scraping {device['name']} for update.")
    # Build a dictionary for Netmiko to use to connect to the devices
    dev_conn = {
        'device_type': 'mikrotik_routeros',
        'host': device['mgmt_ip'],
        'username': device_creds['username'],
        'password': device_creds['password']
    }
    conn = ConnectHandler(**dev_conn)
    output = conn.send_command("/system/routerboard/print")
    conn.disconnect()
    
    scraped_info = {}
    
    lines = output.split("\n")
    
    for line in lines:
        m = re.match(".+serial-number: (\S+)", line)
        if m:
            scraped_info['serial'] = m.group(1)
            
    queried_device = nb_conn.dcim.devices.get(name=device['name'])
    if isinstance(queried_device, type(None)):
        print(f"The device {device['name']} doesn't exist. Skipping.")
        continue
    if queried_device['serial'] == scraped_info['serial']:
        print(f"The serials match. No changes.")
    else:
        print(f"Updating the serial number for {device['name']}.")
        queried_device.update({"serial": scraped_info['serial']})


token.delete()

The code is getting a bit out of hand without some comments. I’ll have to start including those from now on.

Lines 22 – 24 are calling local functions to load up the data from the YAML files. These are here just to show that I do indeed know how to use functions.

Line 29 goes through all the devices in our file. We only have one, so it shouldn’t take too long.

Lines 32 – 40 are the Netmiko stuff. First, we build up a dictionary that contains the connection information – host, username, password, and device type. This is the Netmiko device type and is used to figure out what prompts and login process to expect. Line 39 gets the output of the command /system/routerboard/print (a RouterOS command) and stores it in output. We’ll look at that again in a second.

Line 42 defines the dictionary we’ll send to Netbox if an update is needed.

Line 44 turns the value of output, which is a long string from the device, into a list of lines that are more usable. We’ll use those lines to do a regex match to find the serial number. Regex is its own beast, so do some reading & testing on your own.

Lines 46 – 49 are where the regex magic happens. Line 48 does the heavy lifting here; it finds a line that contains “serial-number: ” (yes, there’s a space in there at the end) and saves the characters after it. We use that value in line 49 (the m.group(1) thing) to set the serial number in the scraped_info dictionary.

Line 51 queries Netbox for the object we might need to update. The next few lines make sure it really exists before moving forward. We should probably do this before the SSH stuff so we don’t waste our time if the device isn’t already in Netbox.

Line 55 does the comparison of the scraped serial number versus the serial number in Netbox. If they don’t match, then we update like we did for the sites.

Updating serial numbers is nice, but that’s in the bottom 1% of the data you care about. You really care about subnets and addresses and interfaces and circuits and rack locations and more. Some things can be derived from the gear and others can’t. There’s always going to be some stuff you have to keep updated manually, but that data that can be updated automatically should be taken out of the hands of people. People make mistakes, get lazy, don’t read directions…that leads to something worse than no documentation — bad documentation.

Send any docker router images questions my way.

#python #netbox #pynetbox #automation #neteng

I usually post links to my blog posts to LinkedIn, and often get extraordinary comments. Unfortunately, those comments usually get lost in the mists of social media fog after a few weeks, so I’m trying to save them by reposting them as blog posts (always with original author’s permission). Here’s a comment David Sun left on my Network Automation Expert Beginners blog post

The most successful automation I’ve seen comes from orgs who start with proper software requirements specifications and more importantly, the proper organizational/leadership backing to document and support said infrastructure automation tooling.

Role-Based Access Control, or RBAC, lets you set permissions around who can access a system and at what level. RBAC is basic, but essential, security function. This video looks at RBAC for Kubernetes from two perspectives: in native Kubernetes and in platforms such as Azure Active Directory. Host Michael Levan brings his background in system […]

The post Kubernetes Security And Networking 2: Getting Started With Role-Based Access Control (RBAC) – Video appeared first on Packet Pushers.

It’s easy to get excited about what seems to be a new technology and conclude that it will forever change the way we do things. For example, I’ve seen claims that SmartNICs (also known as Data Processing Units – DPU) will forever change the network.

TL&DR: Of course they won’t.

Before we start discussing the details, it’s worth remembering what a DPU is: it’s another server with its own CPU, memory, and network interface card (NIC) that happens to have PCI hardware that emulates the host interface cards. It might also have dedicated FPGA or ASICs.

A friend of mine decided to use netlab to build a simple traditional data center fabric, and asked me a question along these lines:

How do I make all the ports be L2 by default i.e. not have IP address assigned to them?

Trying to answer his question way too late in the evening (I know, I shouldn’t be doing that), I focused on the “no IP addresses” part. To get there, you have to use the l2only pool or disable IPv4 prefixes in the built-in address pools, for example:

A long, long time ago, Mircea Ulinic (the author of Salt networking modules) wrote a long and thoughtful blog post on whether we need network automation (TL&DR spoiler: yes).

After reading the article, you might want to listen to the Salt and SaltStack podcast we did with Mircea a long while ago, and watch his presentation in Building Network Automation Solutions online course (also accessible with Expert Subscription).

Do you ever feel like we don’t have enough overlay networking technologies? Don’t worry, there’s always another one, for example Overlay Multilink Network Interface (OMNI) with Asymmetric Extended Route Optimization (AERO) services. Want to know more? Fred Templin described it in a series of overview articles on APNIC blog.

I’m wrapping up Networking Field Day 30 this week and as is always the case there was a lot of great discussion from both the presenters and the delegates outside of the presentations. It’s one of the reasons why I love doing this job even after almost ten years. I get to meet fun people and have an impact on so many things in the tech industry.

• Network-as-a-Service is coming. We recorded a roundtable discussion about it and I think the impact that it’s going to have on mid-sized businesses is massive. It’s going to be like cloud. Not just in operational capability. It’s also going to be a huge driver for what you can do with your network in support of applications. The snowflakes may melt under the weight of the cookies we make from the cookie cutter deployments.
• It feels like a lot of companies are trying to find what’s next. Part of that is coming from the ways that organizations are changing their outlook for what an office should be after the pandemic shutdowns. But still others are realizing they can’t use the same revenue stream for the next five years and hope to survive. This isn’t simply a game of trying to find an adjacent market to move into to drive growth to keep shareholders and investors happy. This is more about finding something that needs to be done because the alternative is no longer having a company.
• Sadly, it looks as though third party Twitter clients are gone for good. This is beyond irritating to me because of the way that I choose to consume Twitter. I’ve seen a lot of chatter in various comment threads about third parties making money from the good graces of Twitter but the fact is that those programs drive a LOT of the way that power users interact with the system. If the exodus wasn’t already accelerating I would imagine you’re going to see a lot more coming very, very soon.

Tom’s Take

Stay tuned for all the great info from Networking Field Day on the Tech Field Day YouTube channel and don’t forget to thank your networking team today. You may not need to call them today to tell them something is down but trust me they will appreciate you calling just to say you appreciate them.

David Bombal kindly invited me to have another chat talking about the future of networking in late 2022. The resulting (masterfully edited) video is already on YouTube. Hope you’ll enjoy it as much as I enjoyed chatting with David.

Sometimes it takes me years to answer interesting questions, like the one I got in a tweet in 2021:

Do you have a good article describing the one-to-one relation of layer-2 and layer-3 networks? Why should every VLAN contain one single L3 segment?

There is no mandatory relationship between multi-access layer-2 networks and layer-3 segments, and secondary IP addresses (and subnets) were available in Cisco IOS in early 1990s. The rules-of-thumb¹ claiming there should be a 1:1 relationship usually derive from the oft-forgotten underlying requirements. Let’s start with those.

While the pundits keeps telling me Docker is dead (looking at its documentation I would say they’re right) and Kubernetes it the way to go (yay!), some people still have to deal with Docker networking, and at least some of them found the Docker Networking Deep Dive webinar useful. Here’s a recent review:

You can scroll over internet pages as long as you can, you will rarely find this kind of specialized knowledge. This is the next level in term of knowledge about Docker.

If you belong to the “Kubernetes will rule the world” camp, we have you covered as well: Stuart Charlton created a phenomenal Kubernetes Networking Deep Dive webinar (approximately half of it is already accessible with free subscription).

This course looks at securing Kubernetes, with an emphasis, though not exclusively, on network security. Topics we’ll cover in the series include: Role-based access control Securing control planes and worker nodes API security and policy management Dealing with vulnerabilities Security capabilities of a service mesh Security-enabled Container Network Interfaces (CNIs) Securing the ingress controller Scanning […]

The post Kubernetes Security And Networking 1: Why Security Matters And Course Overview appeared first on Packet Pushers.

As a warning to everyone, I am not a developer. I am a network engineer who is trying to do some automation stuff. Some of what I’m doing sounds logical to me, but I would not trust my own opinions for production work. I’m sure you can find a Slack channel or Mastodon instance with people who can tell you how to do things properly.

I think there’s a theme in the last few posts. I can’t quite put my finger on it, though. We’ve talked about querying Netbox, but it’s pretty useless without data actually in it. Let’s look at how to get stuff in there using pynetbox.

Here’s the environment I’m running. All this code is in my Github repo.

Python         :  3.9.10 
Pynetbox       :  7.0.0  
Netbox version :  3.4.2  (Docker)

Adding sites is pretty logical first step in a new Netbox install. They don’t have any required fields that have to be created first, so let’s start there. I’ve got a YAML file called sites.yml that contains the site data I want to import. Here’s what that looks like.

### sites.yml
- name: NYC
  description: New York City
  physical_address: "123 Main Street\nNew York, NY 10001"
- name: CHI
  description: Chicago
  physical_address: "123 Main Street\nChicago, IL 60007"
- name: STL
  description: Saint Louis
  physical_address: "123 Main Street\nSaint Louis, MO 63101"
- name: DEN
  description: Denver
  physical_address: "123 Main Street\nDenver, CO 80014"
- name: PHX
  description: Phoenix
  physical_address: "123 Main Street\nPhoenix, AZ 73901"
- name: LAX
  description: Los Angeles
  physical_address: "123 Main Street\nLos Angeles, CA 90001"

This is a list of dictionaries – one for each site. Each site has a name, description, and physical address to use.

Here’s the code we’ll use to import that data. I will quickly admit that this code includes some very non-Pythonic methods. In my opinion, making code more easily readable is more important that doing it “the right way” in a lot of cases.

### pynetbox_populate_sites.py
import pynetbox
import yaml

ENV_FILE = "env.yml"
SITES_FILE = "sites.yml"

with open(ENV_FILE) as file:
    env_vars = yaml.safe_load(file)
    
with open(SITES_FILE) as file:
    sites_to_load = yaml.safe_load(file)
    
nb_conn = pynetbox.api(url=env_vars['netbox_url'])

token = nb_conn.create_token(env_vars['username'], env_vars['password'])

for site in sites_to_load:
    name = site['name'].upper()
    slug = site['name'].lower()
    queried_site = nb_conn.dcim.sites.get(name=name)
    if queried_site:
        print(f"Site {site['name']} already exists.")
        continue
    print(f"Adding {site['name']} to Netbox.")
    constructed_site = {"name": name, "slug": slug}
    if "description" in site.keys():
        constructed_site['description'] = site['description']
    if "physical_address" in site.keys():
        constructed_site['physical_address'] = site['physical_address']
    result = nb_conn.dcim.sites.create(constructed_site)
    
token.delete()

Lines 1 & 2 are the modules we want to use.

Lines 4 & 5 set the name of the files where some data will live.

Line 7 & 8 import the Netbox URL, username, password, etc., from a YAML file into a dictionary called env_vars. This post talks about that a bit.

Line 10 & 11 import the site data from a YAML file into a dictionary called sites_to_load.

Lines 13 – 15 and line 32 connects to Netbox, creates a token to use, then deletes it. See this post for more on that.

Line 17 goes through the sites from the YAML file to do the work.

Line 18 creates a variable called name with a value of the given site name in upper case. We’ll use this as the name of the site in Netbox. I just like to have the names of things that I configure in upper case. Total personal opinion.

Line 19 converts the name from the YAML file to lower case and saves it in a variable called slug. The slug is a URL-friendly version of the name that’s used by…heck, I don’t even know. It’s a required field, so something needs to be in there. I just feed it the name in lower case.

Line 20 start some checking. We don’t want to try and add a site that already exists, so let’s ask Netbox before trying to add it. The result is stored in queried_site.

Line 21 looks to see if queried_site has any value. If it does, that means the site name already exists in Netbox, so we need to skip it.

Line 22 & 23 prints an “already exists” message and continues to the next site in the list.

Line 25 start a new dictionary called constructed_site which we’ll use when it’s time to create the site. Name and slug are the required fields that we already know, so we’ll go ahead and add those.

Line 26 – 29 look to see if the optional fields for description and address exist. If they do, then add them to constructed_site for processing. If you want to add other fields to the YAML to import (region, ASN, timezone, tags, etc.), you can just add some lines to check that as well.

Line 30, of course, is where the magic happens. This uses .create() to — wait for it — create a site using the given dictionary. This returns the site object we created. We’re not doing anything with it, though we definitely should be checking the result to make sure it worked!

The output is pretty unremarkable. If the site exists, it says “Site X already exists.” If it get added, it says “Adding X to Netbox.”

What about some more-complex objects like devices? We can do that, too. To add a device, we need to pause a bit and take a look at the required fields, though. If you go into the GUI to add one manually, you’ll see device role (the function of the devices), device type (the make and model), site, and status are all required. They also are all objects that must already exist in Netbox, so we’ll have to check the given data before trying to add the device. If we don’t, we’ll get an exception somewhere down the line.

We’ll do another YAML file for the devices. This is what it looks like. There may or may not be some bad data in this one, so be on the lookout. **hint, hint**

### devices.yml
<SNIP>
- name: CHI-RTR01
  site: CHI
  type: GENERIC
  role: INET_ROUTER
- name: LAX-FRW01
  site: LAX
  type: GENERIC
  role: FIREWALL
<SNIP>
- name: ATL-FRW01
  site: ATL
  type: GENERIC
  role: INET_ROUTER
- name: PHX-RTR01
  site: PHX
  type: GENERIC
  role: INET_ROUTER
  status: planned
<SNIP>

The YAML contains a list of devices that include name, type, role, and status. It’s funny how that matches the required configuration, isn’t it? NOTE: To make things easier for us, I created a device type called “GENERIC” by hand. Every device here has this type, but you should put in the real makes and models in production. Someone will ask you for an inventory in the next few months, so I suggest you get serial number in there as well. Audit season is always around the corner.

Alright, here’s the long, long code. I’ll only mention the lines that are different than the code above.

### pynetbox_populate_devices.py
import pynetbox
import yaml

ENV_FILE = "env.yml"
DEVICES_FILE = "devices.yml"

with open(ENV_FILE) as file:
    env_vars = yaml.safe_load(file)
    
with open(DEVICES_FILE) as file:
    devices_to_load = yaml.safe_load(file)
    
nb_conn = pynetbox.api(url=env_vars['netbox_url'])

token = nb_conn.create_token(env_vars['username'], env_vars['password'])

valid_devices_status = []
for choice in nb_conn.dcim.devices.choices()['status']:
    valid_devices_status.append(choice['value'])

for device in devices_to_load:
    name = device['name'].upper()
    slug = device['name'].lower()
    
    # See if the device already exists
    queried_device = nb_conn.dcim.devices.get(name=name)
    if queried_device:
        print(f"The device {name} already exists. Skipping.")
        continue
    
    # See if the given device type exists
    dev_type = device['type'].upper()
    queried_type = nb_conn.dcim.device_types.get(model=dev_type)
    if isinstance(queried_type, type(None)):
        print(f"The type {dev_type} does not exist. Skipping.")
        continue
    
    # See if the given device role exists
    dev_role = device['role'].upper()
    queried_role = nb_conn.dcim.device_roles.get(name=dev_role)
    if isinstance(queried_role, type(None)):
        print(f"The role {dev_role} does not exist. Skipping.")
        continue
    
    # See if the given site exists
    site = device['site'].upper()
    queried_site = nb_conn.dcim.sites.get(name=site)
    if isinstance(queried_site, type(None)):
        print(f"The site {site} does not exist. Skipping.")
        continue
    
    
    constructed_device = {"name": name, "slug": slug, "site": queried_site.id, "device_role": queried_role.id, "device_type": queried_type.id}
    if "description" in device.keys():
        constructed_device['description'] = device['description']
    if "status" in device.keys():
        if device['status'] in valid_devices_status:
            constructed_device['status'] = device['status']
        else:
            print(f"The status of {device['status']} isn't valid. Skipping.")
            continue
    print(f"Adding {device['name']} to Netbox.")
    result = nb_conn.dcim.devices.create(constructed_device)
    
token.delete()

Lines 17 – 19 are interesting. Some of the fields in the Netbox GUI are dropdown boxes where you select a valid choice. You can’t just freehand the value; it has to be one of the valid choices available. You can use .choices() to get a full list of all valid choices, including the status field. Line 18 gets all the valid values for the status field and adds them to the list called valid_device_status so we can check them later. As homework, you should write code to get the choices for devices, prefixes, and device types and explore them a bit.

Lines 26 – 50 all do checking. Does the given device already exists? Does the given type exist? Does the given role exist? Does the given site exist? If they don’t, print an error message and go to the next device.

Lines 53 – 63 are basically the same as when we added the sites.

Line 57 is interesting. Remember the list of statuses we got in lines 16 – 18? This line checks the given status against that list to make sure they’re valid. If it’s not valid, print a message and move on. You can probably modify the script a bit to just default to “active” if you want.

Did you catch the bad data in there? One of the devices is for the Atlanta site, which doesn’t exist in Netbox. When you run the script, you’ll see this.

The site ATL does not exist. Skipping.

I guess some of that validation works. Not all of it, though. What if you put in a device without a role or type? This script would try to add a None as the value, which would cause a KeyError exception. This definitely needs more work, but it will get the job done.

Send any 18″ white oak logs questions to me.

#python #netbox #pynetbox

As a warning to everyone, I am not a developer. I am a network engineer who is trying to do some automation stuff. Some of what I’m doing sounds logical to me, but I would not trust my own opinions for production work. I’m sure you can find a Slack channel or Mastodon instance with people who can tell you how to do things properly.

A bit ago, we talked about getting information out of Netbox with Pynetbox. The example was very simple, but I’m afraid the real world dictates that querying every device every time is not very efficient or manageable. At some point, we’ll need to ask for a subset of everything, so let’s look at filtering.

We used .all() last time. It’s pretty obvious what that gives us. If we don’t want everything in the world returned, we can use .filter() along with some parameters to limit that result. Let’s get to an example.

We want to print a report of all devices with hostname and role. The devices should be grouped by site. This means we need to get a list of sites, go through that list, get the devices there, and print what we want. Here it goes.

Here’s the environment I’m running. All this code is in my Github repo.

Python         :  3.9.10 
Pynetbox       :  7.0.0  
Netbox version :  3.4.2  (Docker)

### pynetbox_query_filter_1.py
import pynetbox
import yaml

ENV_FILE = "env.yml"

with open(ENV_FILE) as file:
    env_vars = yaml.safe_load(file)

nb_conn = pynetbox.api(url=env_vars['netbox_url'])
token = nb_conn.create_token(env_vars['username'], env_vars['password'])

sites = nb_conn.dcim.sites.all()

for site in sites:
    site_header = f"\nDevices at site {site.name} ({site.description})"
    print(site_header)
    print("-" * len(site_header))
    devices = nb_conn.dcim.devices.filter(site_id=site.id)
    if len(devices) < 1:
        print("No devices.")
        continue
    for device in devices:
        print(f"{device.name:^20} {device.device_role.name:^20}")
        
token.delete()

Lines 1 & 2 are our imports. Basic Python stuff there.

Lines 4 – 10 and 25 are from a previous post about generating keys in pynetbox.

Line 12 gets all the sites.

Line 14 goes through each site to do the magic.

Lines 15 – 17 just print some header info. Line 17 is pretty cool trick for having the right number of underscores.

Line 18 is the one we care about right now. This asks Netbox to provide all the devices that have a site ID equal to the site we’re looking at. We’re using “site_id” as the argument here, but you can use any field you want to filter on. Status, rack ID, manufacturer, tags, create time..the list goes on. You can have more than one argument, too, which is pretty great.

Lines 19 – 21 check if we actually got devices for a site. If not, we just say “No devices.” and move on to the next site using continue.

Lines 22 & 23 go through the devices for this site and print the name and role. They use some fancy formatting to make it look nice.

Here’s the output from running this.

Devices at site CHI (Chicago)
------------------------------
     CHI-CSW01           CORE_SWITCH     
     CHI-RTR01           INET_ROUTER     

Devices at site DEN (Denver)
-----------------------------
     DEN-CSW01           CORE_SWITCH     
     DEN-RTR01            WAN_ROUTER     

Devices at site LAX (Los Angeles)
----------------------------------
     LAX-CSW01           CORE_SWITCH     
     LAX-FRW01             FIREWALL      
     LAX-RTR01            WAN_ROUTER

Devices at site NYC (New York City)
------------------------------------
     NYC-CSW01           CORE_SWITCH     
     NYC-FRW01             FIREWALL      

Devices at site PHX (Phoenix)
------------------------------
     PHX-CSW01           CORE_SWITCH     
     PHX-RTR01           INET_ROUTER     

Devices at site STL (Saint Louis)
----------------------------------
     STL-ASW01            ACC_SWITCH     
     STL-CSW01           CORE_SWITCH     
     STL-FRW01             FIREWALL

I think you can probably figure out how to do it, but check out pynetbox_query_filter_2.py in the repo to see a .filter() with more than one argument.

When you use .filter(), pynetbox returns a RecordSet (or None if there’s nothing to get), even if the query returns a single result. This means that you have to loop through the result each time you use filter(). If you want to get back a single Record, then use .get().

.get() takes the same arguments as .filter(), but the arguments must be specific enough for Netbox to return a single result. That is, the total sum of all the arguments must be unique across Netbox. If your arguments match more than one result, you get an error like this one.

ValueError: get() returned more than one result. Check that the kwarg(s) passed are valid for this endpoint or use filter() or all() instead.

You can keep stacking arguments until it’s unique (“device_role=”firewall”, site=”NYC”, rack=”RACK1″, position=14“, etc.), but that’s not very scalable or even worth your time to figure out if the query is unique enough. Because of that, I tend to only use .get() when I know the object ID (id=X). Since this is assigned by Netbox and can’t be reused., using it assures us that the query is specific enough.

.get() has its limitations but it’s still very useful, though. If a Netbox object has a reference to another Netbox object, the result will include some information about that referenced object. That’s a terrible sentence. Things might be clearer if we look at the result from a query.

{'airflow': None,
 'asset_tag': None,
 'cluster': None,
 'comments': '',
 'config_context': {},
 'created': '2023-01-16T14:43:40.208662Z',
 'custom_fields': {},
 'device_role': {'display': 'FIREWALL',
                 'id': 7,
                 'name': 'FIREWALL',
                 'slug': 'firewall',
                 'url': 'http://*.*.*.*/api/dcim/device-roles/7/'},
<SNIP>

This is a snip of a device record. You can see device_role isn’t just a string result; it’s got some information about the role for this device, including the ID of that role. Now we have a piece of information that we can use as a query for a specific device.

Here’s some code to get shipping information for the devices with a “planned” status. The real-world scenario is that you have configured these devices and need to ship them out to the right site for install.

### pynetbox_query_filter_3.py
import pynetbox
import yaml

ENV_FILE = "env.yml"

with open(ENV_FILE) as file:
    env_vars = yaml.safe_load(file)

nb_conn = pynetbox.api(url=env_vars['netbox_url'])
token = nb_conn.create_token(env_vars['username'], env_vars['password'])

devices = nb_conn.dcim.devices.filter(status='planned')

for device in devices:
    site = nb_conn.dcim.sites.get(id=device.site.id)
    print(f"Ship {device.name} to:\n{site.physical_address}\n")
        
token.delete()

Line 12 is a .filter() that retrieves only devices in a “planned” state. This is a RecordSet, so you have to iterate through to get anything useful.

Line 15 is the .get(). We get the site ID returned with the device (device.site.id), so we can use that in a .get() argument to get a single result. This is a Record, so you can use it directly.

The rest of the lines are pretty much the same as above, so I’ll skip the explanation. Here’s the output.

Ship LAX-RTR01 to:
123 Main Street
Los Angeles, CA 90001

Ship PHX-RTR01 to:
123 Main Street
Phoenix, AZ 73901

Ship STL-FRW01 to:
123 Main Street
Saint Louis, MO 63101

In summary, filtering is good. Carry on.

Send any soapmaking tips questions my way.