January 23, 2021

ipSpace.net Blog (Ivan Pepelnjak)

Worth Reading: Cloud Complexity Lies

Anyone who spent some time reading cloud providers' documentation instead of watching slide decks or vendor keynotes knows that setting up infrastructure in a public cloud is not much simpler than doing it on-premises. You will outsource hardware management (installations, upgrades, replacements…) and might deal with an orchestration system provisioning services instead of configuring individual devices, but you still have to make the same decisions, and take the same set of responsibilities.

Obviously that doesn’t look good in a vendor slide deck, so don’t expect them to tell you the gory details (and when they start talking about the power of declarative API you know you have a winner)… but every now and then someone decides to point out the state of emperor’s clothes, this time Gerben Wierda in his The many lies about reducing complexity part 2: Cloud.

For public cloud networking details, check out our cloud webinars and online course.

January 23, 2021 08:43 AM

January 22, 2021

Packet Pushers

VMware After Gelsinger: Integrating Fiefdoms For A Post-Hypervisor World

VMware's next CEO has two tasks: to construct a narrative about VMware's role and value as a company in a post-hypervisor world, and to integrate its various fiefdoms into a cohesive set of products that can provide greater utility when used together than when used individually.

The post VMware After Gelsinger: Integrating Fiefdoms For A Post-Hypervisor World appeared first on Packet Pushers.

by Drew Conry-Murray at January 22, 2021 10:01 PM

ipSpace.net Blog (Ivan Pepelnjak)

Podcast: IPv6 in the Cloud

In December 2020 Ed Horley invited me to a chat about IPv6 in the public cloud. While I usually don’t want to think about a protocol that’s old enough to buy its own beer in US, we nonetheless had interesting discussions (including the need for frequent RA messages in AWS VPC).

I also proved Clarke’s first law. Running IPv6 over IPv4 IPsec tunnel is a perfectly legal combo.

January 22, 2021 05:43 PM

The Networking Nerd

Planning For The Worst Case You Can’t Think Of

Remember that Slack outage earlier this month? The one that happened when we all got back from vacation and tried to jump on to share cat memes and emojis? We all chalked it up to gremlins and went on going through our pile of email until it came back up. The post-mortem came out yesterday and there were two things that were interesting to me. Both of them have implications on reliability planning and how we handle the worst-case scenarios we come up with.

It’s Out of Our Hands

The first thing that came up in the report was that the specific cause for the outage came from an AWS Transit Gateway not being able to scale fast enough to handle the demand spike that came when we all went back to work on the morning of January 4th. What, the cloud can’t scale?

The cloud is practically limitless when it comes to resources. We can create instances with massive CPU resources or storage allocations or even networking pipelines. However, we can’t create them instantly. No matter how much we need it takes time to do the basic provisioning to get it up and running. It’s the old story of eating an elephant. We do it one bite at a time. Normally we tell the story to talk about breaking a task down into smaller parts. In this case, it’s a reminder that even the biggest thing out there has to be dealt with in small pieces as well.

Slack learned this lesson the hard way. Why? Because they couldn’t foresee a time when their service was so popular that the amount of traffic rushing to their servers crushed the transit gateways. Other companies have had to learn this lesson the hard way too. Disney crawled on launch day because of demand. The release of a new game that requires downloading and patching on Day One also puts stress on servers. Even the lines outside of department stores on Black Friday (in less pandemic-driven years) are examples of what happens when capacity planning doesn’t meet demand.

When you plan for your worst case scenario, you have to think the unthinkable. Instead of asking yourself what might happen if everyone logs on at the same time you also need to ask when happens if they try and something goes wrong. I spent a lot of time in my former job thinking about simple little exercises like VDI boot storms, where office workers can push a storage system to the breaking point by simply turning all their machines on at the same time. It’s the equivalent of being on a shared network resource like a cable modem during the Super Bowl. There aren’t any resources available for you to use.

When we plan for capacity, we have to realize that even our most optimistic projections of usage are going to be conservative if we take off or go viral. Rather than guessing what that might be, take an hour every six months and readjust your projections. See how fast you’re growing. Plan for that crazy scenario where everyone decides to log on at the same time on a day where no one has had their coffee yet. And be ready for what happens when someone throws a wrench into the middle of the process.

What Happens When It Goes Wrong?

The second thing that came up in the Slack post-mortem that is just as worrisome was the behavior of the application when it realized there was a connection timeout. The app started waiting for the pathway to be open again. And guess what happened when AWS was able to scale the transit gateway? Slack clients started hammering the servers with connection requests. The result was something akin to a Distributed Denial-of-Service (DDoS) attack.

Why would the Slack client do this? I think it has something to do with the way the developers coded it and didn’t anticipate every client trying to reconnect all at once. It’s not entirely unsound thinking to be honest. How could we live in a world where every Slack user would be disconnected all at once? Yet, we do and it did and look what happened?

Ethernet figured this part out a long time ago. The CSMA/CD method for detecting collisions on a layer 2 connection has an ingenious solution for what happens when a collision is detected. Once it realizes that there was a problem on the wire it stops what is going on and calculates a random backoff timer based on the number of detected collisions. Once that timer has expired it attempts to transmit again. Because there has to be another station involved in a collision incident both stations do this. The random element of the timer calculation ensures that the likelihood of both stations choosing to transmit again at the same time is very, very low.

If Ethernet behaved like the Slack client did we would never resolve collisions. If every station on a layer 2 network immediately tried to retransmit without a backoff timer the bus would be jammed constantly. The architects of the protocol figured out that every station needs a cool off period to clear the wire before trying again. And it needs to be different for every station so there is no overlap.

Slack really needs to take this idea into account. Rather than pouncing on a connection as soon as it’s available there needs to be a backoff timer that prevents the servers from being swamped. Even a few hundred milliseconds per client could have prevented this outage from getting as big as it did. Slack didn’t plan beyond the worst case scenario because they never conceived of their worst case scenario coming to pass. How could it get worse than something we couldn’t imagine happening?


Tom’s Take

If you design systems or call yourself a reliability engineer, you need to develop a hobby of coming up with disastrous scenarios. Think of the worst possible way for something to fail. Now, imagine it getting worse. Assume that nothing will work properly when there is a recovery attempt. Plan for things to be so bad that you’re in a room on fire trying to put everything out while you’re also on fire. It sounds very dramatic but that’s how bad it can get. If you’re ready for that then nothing will surprise you. You also need to make sure you’re going back and thinking of new things all the time. You never know which piece is going to fail and how it will impact what you’re working on. But thinking through it sometimes gives you an idea of where to go when it all goes wrong.

by networkingnerd at January 22, 2021 05:28 PM

ipSpace.net Blog (Ivan Pepelnjak)

XML-to-JSON Information Loss, Cisco Nexus OS Edition

Last week I wrote about the interesting challenges you might encounter when using data generated by a Junos device in an Ansible playbook. Unfortunately it’s not just Junos – every system built around XML-based data structures might experience the same issues, including Cisco Nexus OS.

To be fair to Ansible developers: it’s not an Ansible problem, the problem is caused by fundamental incompatibility between XML and JSON encodings, and the naive use of standard XML Python libraries. It’s just that engineers who might stumble upon that problem commonly use Ansible.

January 22, 2021 04:50 PM

Packet Pushers

Stable: GNS3 2.2.17 + VMware Fusion 12.1.0 + macOS 11.1 (Build 20C69)

Even with minor caveats, I seem to be in a better place with macOS 11.1 Big Sur versus macOS 10.15.7 Catalina. Big Sur is not a flawless experience for me yet, but I have hope it will become so as software makers have time to adjust to all of Apple's changes. And I'll take being able to run GNS3 labs without kernel panics as a big win.

The post Stable: GNS3 2.2.17 + VMware Fusion 12.1.0 + macOS 11.1 (Build 20C69) appeared first on Packet Pushers.

by Ethan Banks at January 22, 2021 04:26 PM

XKCD Comics

January 21, 2021

My Etherealmind

Car Warranty vs Technology Service Contract

I’ve been struggling with the value of service contracts on IT equipment for some time now. As a rule of thumb, service contracts on IT infrastructure cost ~30% of the purchase price. This means that over three years of ownership you will pay the purchase price again. Which is quite a thing. I’m struggling to […]

by Greg Ferro at January 21, 2021 05:28 PM

ipSpace.net Blog (Ivan Pepelnjak)

How Important is BGP RPKI?

Corey Quinn mentioned me in a tweet linking to AWS announcement that they are the biggest user of BGP RPKI (by the size of signed address space) worldwide. Good for them – I’m sure it got their marketing excited. It’s also trivial to do once you have the infrastructure in place. Just saying…

On a more serious front: how important is RPKI and what misuses can it stop?

If you’ve never heard of RPKI, the AWS blog post is not too bad, Nick Matthews wrote a “look grandma, this is how it works” version in 280-character installments, and you should definitely spend some time exploring MANRS resources. Here’s a short version for differently-attentive ;))

January 21, 2021 07:28 AM

January 20, 2021

My Etherealmind

◎ Why Do Losers Watch Sportsball ?

Its bad for your brain because it teaches you the wrong things.

by Greg Ferro at January 20, 2021 07:20 PM

XKCD Comics

January 19, 2021

Packet Pushers

Is Computing A Right?

Recently, the idea of a cloud computing service delivered as a public utility was pitched to me. The idea was that computing power made available to those who would otherwise be unable to afford it would be a societal good. For example, imagine an academic group that needs compute for a research project. Or municipalities that would benefit their citizenry by leveraging a cloud-as-utility.

The post Is Computing A Right? appeared first on Packet Pushers.

by Ethan Banks at January 19, 2021 10:13 PM

ipSpace.net Blog (Ivan Pepelnjak)

Repost: VMware Fault Tolerance Woes

I always claimed that VMware Fault Tolerance makes no sense. After all, the only thing it does is protect a VM against a server hardware failure… in the world where software crashes are way more common, and fat fingers cause most of the outages.

But wait, it gets worse, the whole thing is incredibly complex – you might like this description Minh Ha left as a comment to my Fifty Shades of High Availability blog post.

January 19, 2021 07:55 AM

January 18, 2021

ipSpace.net Blog (Ivan Pepelnjak)

Build Virtual Lab Topology: Dual Stack Addressing, ArcOS and Junos Support

In mid-December I announced a set of tools that will help you build Vagrant-based remote labs much faster than writing Vagrantfiles and Ansible inventories by hand.

In early January I received a nice surprise: Dave Thelen not only decided to use the tool, he submitted a pull request with full-blown (and correctly implemented) ArcOS support. A few days later I managed to figure out what needs to be configured on vSRX to make it work, added Junos support, and thus increased the number of supported platforms to six (spanning five different operating systems).

January 18, 2021 07:32 AM

XKCD Comics

January 15, 2021

SNOsoft Research Team

Embedded Device Security Research: AXON Body 2 – Body Worn Cameras

AXON Body 2 – Body Worn Cameras

Introduction

Netragard performs regular vulnerability research against software and hardware. While most of this research is customer confidential, some of the research is intended for disclosure. The focus of our research for this article was the AXON Body 2 Worn Camera which plays a critical role in protecting civilians and police officers. Due to the sensitive nature of the evidence collected by the AXON Body 2 it is particularly important that the device successfully maintains the Confidentiality, Integrity and Availability of the data that it contains.

Netragard opted to focus on BWC’s because they have largely remained out of sight to the public. Despite the lack of public exposure Axon has been very active in terms of advancing its security posture. Axon provides Penetration Testing & Vulnerability Disclosure Guidelines and also offers a private bug bounty program through Hacker One. Additionally, the Axon product line has a documented methodology regarding product safeguards, considerations and recommendations. Finally, Axon maintains tight control over the distribution of its software and firmware which further helps to improve on security.

The second generation of the Axon BWC was redesigned and built on an Amberalla System-on-Chip (SoC). Then the Axon Body 3 (the third generation) was redesigned again and based on non-Ambarella system. It is important to note that the Axon BWC’s were not available for purchase until recently, and now can be found at online marketplaces, such as e-bay. It is fairly common for products like BWCs to make their way into second-hand markets as they are replaced by more current revisions. 

While Netragard did not find a useful life recommendation for Axon BWCs, the useful life recommendation for Axon’s TASER® product is five years. Generally, these guidelines are based on failure in the field metrics. The same useful life recommendation may very well apply to the Axon BWCs.

Netragard’s interest in the Axon BWCs was the product of an article where a researcher found an unencrypted video on an SD card that was extracted from the Axon Body-worn camera. Netragard decided to perform research against a more recent generation of the Axon BWC to get a better understanding of its security. This resulting in Netragard acquiring several AXON Body 2 devices that were configured with firmware 1.11.16 and in online mode. Its important to note that Netragard has not performed any research against the Axon Body 3 BWCs.

Finally, we would like to thank Axon for their cooperation and support during this project. Their positive and welcoming attitude towards security and security research is yet another example of how seriously they take the security of their products. As of the authoring of this article, Axon has produced patches for the issues disclosed herein.

What Are BWCs?

BWCs are camera systems designed to be worn by police. These devices record both audio and video with the intention of collecting incontrovertible evidence regarding cases and disputes between law-enforcement officers and the general public. In recent months, many jurisdictions have moved to purchase BWCs for their law enforcement officers and require their use. With their critical role in law enforcement and the evidence that they contain, the security of BWCs is of paramount importance. If evidence can be inappropriately accessed, modified, or removed from these cameras, then it could impact the results of legal proceedings and could put innocent people at risk.

Introduction to the Axon BWC

Even though the Axon BWC products have been on the market for approximately a decade, their proliferation into non Law Enforcement Agencies is remarkably low. A timeline of releases is provided below:

  • 2009: Axon Pro
  • 2012 and 2013: Axon Flex and Body
  • 2015 and 2016: Body 2 and Flex 2
  • 2019: Body 3

Netragard opted to perform research against the Body 2, which was released five years ago, but is still in regular active use today. The current version is the Axon Body 3.

Get Access: A classic embedded security example

Gaining access to the internals of the Body 2 camera system requires a screwdriver. There were no notable physical security systems in place like what one might encounter with a credit card reader for example. The physical security of any product designed to contain highly sensitive information is of the utmost importance. Key questions to ask when designing such a product are:

  • How much attack surface exists beneath the product casing and how can it be made tamper proof?
  • What hurdles and/or defenses can be built into the product that will hinder or stop a modern hacker from being able to dismantle and analyze the device?
  • Do your product design choices impact the post-launch security of the product?
A classic embedded security example

A Closer Look

Beneath the battery and below the foil Netragard found an Amberella SoC alongside other chip-level components.

When peeling back the foil Netragard discovered a BCM43340XKUBG.

Chipsip CT49488DD966C1, Ambarella A7LW35M, SK Hynix H26M78103CCR are all visible among various test pads.

When the PCB was removed from its case it would no longer power on. It turned out that the case contained a small metal clip that would complete a circuit on the PCB. To circumvent this Netragard soldered a yellow wire to the board where the clip would normally close the circuit. Once this was done Netragard was able to power the board on via its USB charging cable which is also used for synchronization with the Evidence Sync product.

Netragard also purchased bootleg versions of cables needed to connect the PCB to the Evidence Sync product from Amazon. Screenshots of these cables are provided below.

Axon sync cable

Connecting to Evidence Sync Software

Extracting evidence from the Printable Computer Board in the AXON Body 2 proved to be more challenging than expected. This is not because of additional security layers but instead because it is exceedingly difficult to find a copy of the Axon Evidence Sync software. While it took Netragard less than one hour of cumulative time to acquire all the hardware referenced above, it took several weeks to locate a usable copy of the software shown in the image below.

The AXON Body 2 does not mount like a traditional mass-storage device and requires both drivers and Evidence Sync software. Once connected to the Evidence Sync software Netragard discovered that the data stored on the AXON Body 2 was not encrypted at rest. If encryption were in use (it can be configured on the AXON Body 2 in online mode), then evidence extraction would be blocked without a key.

Connecting to Evidence Sync Software

The Various Modes

The AXON Body 2 supports three operating modes. A high-level overview of these modes is provided below while a more detailed overview is provided by Axon at the following URL: https://help.axon.com/hc/en-us/articles/221458387-Operating-modes

Online Mode

In the online mode, the Evidence Sync software uploads data from your Axon and TASER devices to your Evidence.com or Evidence.com lite account. CEW firing records are automatically uploaded to Evidence.com, but you will have to tell your Evidence Sync to upload TASER CAM and Axon videos to the Evidence.com website.

Offline Mode

In the offline mode, the Evidence Sync software downloads data from your CEW or recorder to your computer. If your organization does not use Evidence.com, you will always use Evidence Sync in the offline mode.

Note: Enabling Offline Mode requires users to accept a disclaimer acknowledging risks to the agency and data. This mode does not appear to provide the same level of integrity (in terms of chain of evidence) as Online mode. It also places data in the user’s PC which may or may not be properly protected.

MDT Mode

A mobile data terminal is a computer used in a police car. An MDT may also be called a Mobile Data Computer (MDC) or Mobile Computer Terminal (MCT). Evidence Sync has an operating mode for use with an MDT, called MDT Mode.

Note: In MDT mode (unlike Online or Offline mode) there is no way to remove video evidence from the device.

Mode Limitations

Some of the modes mentioned above come with restrictions as shown below. The Axon Evidence Sync software is mode aware, meaning it can detect which mode the AXON Body 2 is in.

<style type="text/css">.fusion-fullwidth.fusion-builder-row-1 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link) , .fusion-fullwidth.fusion-builder-row-1 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):before, .fusion-fullwidth.fusion-builder-row-1 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-1 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover, .fusion-fullwidth.fusion-builder-row-1 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:before, .fusion-fullwidth.fusion-builder-row-1 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-1 .pagination a.inactive:hover, .fusion-fullwidth.fusion-builder-row-1 .fusion-filters .fusion-filter.fusion-active a {border-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-1 .pagination .current {border-color: #f2b310; background-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-1 .fusion-filters .fusion-filter.fusion-active a, .fusion-fullwidth.fusion-builder-row-1 .fusion-date-and-formats .fusion-format-box, .fusion-fullwidth.fusion-builder-row-1 .fusion-popover, .fusion-fullwidth.fusion-builder-row-1 .tooltip-shortcode {color: #f2b310;}#main .fusion-fullwidth.fusion-builder-row-1 .post .blog-shortcode-post-title a:hover {color: #f2b310;}</style>

The Evidence Sync application is mindful of camera state, and limits access accordingly.

Online mode has a much more complex authentication and user management interface for evidence.

Without a proper agency login, the additional functionality to remove videos is not present for example.

<style type="text/css">.fusion-fullwidth.fusion-builder-row-2 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link) , .fusion-fullwidth.fusion-builder-row-2 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):before, .fusion-fullwidth.fusion-builder-row-2 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-2 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover, .fusion-fullwidth.fusion-builder-row-2 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:before, .fusion-fullwidth.fusion-builder-row-2 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-2 .pagination a.inactive:hover, .fusion-fullwidth.fusion-builder-row-2 .fusion-filters .fusion-filter.fusion-active a {border-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-2 .pagination .current {border-color: #f2b310; background-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-2 .fusion-filters .fusion-filter.fusion-active a, .fusion-fullwidth.fusion-builder-row-2 .fusion-date-and-formats .fusion-format-box, .fusion-fullwidth.fusion-builder-row-2 .fusion-popover, .fusion-fullwidth.fusion-builder-row-2 .tooltip-shortcode {color: #f2b310;}#main .fusion-fullwidth.fusion-builder-row-2 .post .blog-shortcode-post-title a:hover {color: #f2b310;}</style>

Potential Attack Vectors for Axon Body 2

With access to the Evidence Sync software, it is possible to access videos for devices in Offline mode (which is not the default) while videos on devices in Online mode remain inaccessible. When reverse engineering the Evidence Sync software, it appeared that it would be possible to create a modified version which enables an attacker to subvert the built-in security controls.

Software Reverse Engineering

<style type="text/css">.fusion-fullwidth.fusion-builder-row-3 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link) , .fusion-fullwidth.fusion-builder-row-3 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):before, .fusion-fullwidth.fusion-builder-row-3 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-3 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover, .fusion-fullwidth.fusion-builder-row-3 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:before, .fusion-fullwidth.fusion-builder-row-3 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-3 .pagination a.inactive:hover, .fusion-fullwidth.fusion-builder-row-3 .fusion-filters .fusion-filter.fusion-active a {border-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-3 .pagination .current {border-color: #f2b310; background-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-3 .fusion-filters .fusion-filter.fusion-active a, .fusion-fullwidth.fusion-builder-row-3 .fusion-date-and-formats .fusion-format-box, .fusion-fullwidth.fusion-builder-row-3 .fusion-popover, .fusion-fullwidth.fusion-builder-row-3 .tooltip-shortcode {color: #f2b310;}#main .fusion-fullwidth.fusion-builder-row-3 .post .blog-shortcode-post-title a:hover {color: #f2b310;}</style>

With access to the Evidence Sync software it is easy to learn how the underlying system works.

With the proper analysis and access, an attacker could develop their own Axon tools.

Targeting specific desirable administrator functionality would likely be the end-goal for an attacker.

<style type="text/css">.fusion-fullwidth.fusion-builder-row-4 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link) , .fusion-fullwidth.fusion-builder-row-4 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):before, .fusion-fullwidth.fusion-builder-row-4 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-4 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover, .fusion-fullwidth.fusion-builder-row-4 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:before, .fusion-fullwidth.fusion-builder-row-4 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-4 .pagination a.inactive:hover, .fusion-fullwidth.fusion-builder-row-4 .fusion-filters .fusion-filter.fusion-active a {border-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-4 .pagination .current {border-color: #f2b310; background-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-4 .fusion-filters .fusion-filter.fusion-active a, .fusion-fullwidth.fusion-builder-row-4 .fusion-date-and-formats .fusion-format-box, .fusion-fullwidth.fusion-builder-row-4 .fusion-popover, .fusion-fullwidth.fusion-builder-row-4 .tooltip-shortcode {color: #f2b310;}#main .fusion-fullwidth.fusion-builder-row-4 .post .blog-shortcode-post-title a:hover {color: #f2b310;}</style>

USB Sniffing

Evidence is transferred from a Body 2 to the Evidence Sync software via USB. This transfer uses the LibUSB protocol, making it trivial to observe and reverse-engineer. This could allow an attacker to snoop on the USB communications between a Body 2 camera and the Evidence Sync software, providing access to the videos.

The communication transport for Axon Body is based on LibUSB which makes it easy to observe. An attacker with sufficient time could use this to engineer a standalone program to extract any evidence off an Axon Body 2 BWC without needing to use the Axon Sync application.

Common USB Sniffing Options:

  • Wireshark
  • USBPcap
  • SnoopyPro
  • TotalPhase (hardware)
  • Ellisys (Hardware)
  • Vmware Vusb-Analyser

Broadcom Wireless SoC – BroadPWN

While disassembling the Axon BWC (Body 2), Netragard discovered a Broadcom WiFi BCM4334 chipset which is vulnerable to the BroadPWN vulnerability. BroadPWN is a Remote Code Execution (“RCE”) vulnerability that when exploited allows an attacker to execute arbitrary commands. Exploits for this vulnerability do exist in the wild but are generally designed for Android and iOS targets. Modifying the exploit to target and the BroadPWN vulnerability in the Axon BWC is not a trivial task. Additionally, Axon implemented mitigations for this vulnerability in the newer versions of its firmware. Netragard has not tested the efficacy of these mitigations.

BCM43340

The following image contains disclosure commentary about the BroadPWn vulnerability affecting BCM43340XKUBG.

BroadPWn vulnerability

Remote Detection

An attacker can easily detect Bluetooth Low Energy (“BTLE”) signals broadcast from Axon BWCs by searching for its private OUI which is 00:25:DF. Additionally, some AXON Body 2 devices broadcast a device name via their MAC address 12:20:13:03:33:05. While it is possible to disable wireless functionality, this may not be a viable solution for some parties. While detecting an AXON Body 2 does not introduce risk, it does enable an attacker to target the device which could have a significant impact. The following images show the various ways Kismet can detect these devices.

Conclusion

The Axon Body 2 BWC is a critical piece of hardware that ensures both civilians and law enforcement officers are protected. This is accomplished by recording both audio and video evidence. This evidence can mean the difference between freedom and imprisonment as demonstrated through various recent public incidents.

Axon was made aware of this research project well before the publication of this article. Axon was not only cooperative but provided support as needed and maintained an open and friendly communication channel with Netragard. Axon has addressed the vulnerabilities disclosed in this article and the fixes have already been pushed to customers.

<style>#main-internal-body h3, .main-internal-body h3 {margin-top: 3rem;}</style>
<style type="text/css">.fusion-fullwidth.fusion-builder-row-5 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link) , .fusion-fullwidth.fusion-builder-row-5 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):before, .fusion-fullwidth.fusion-builder-row-5 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-5 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover, .fusion-fullwidth.fusion-builder-row-5 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:before, .fusion-fullwidth.fusion-builder-row-5 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-5 .pagination a.inactive:hover, .fusion-fullwidth.fusion-builder-row-5 .fusion-filters .fusion-filter.fusion-active a {border-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-5 .pagination .current {border-color: #f2b310; background-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-5 .fusion-filters .fusion-filter.fusion-active a, .fusion-fullwidth.fusion-builder-row-5 .fusion-date-and-formats .fusion-format-box, .fusion-fullwidth.fusion-builder-row-5 .fusion-popover, .fusion-fullwidth.fusion-builder-row-5 .tooltip-shortcode {color: #f2b310;}#main .fusion-fullwidth.fusion-builder-row-5 .post .blog-shortcode-post-title a:hover {color: #f2b310;}</style>

The post Embedded Device Security Research: AXON Body 2 – Body Worn Cameras appeared first on Netragard.

by Adriel Desautels at January 15, 2021 09:13 PM

The Networking Nerd

Managing Leaders, Or Why Pat Gelsinger Is Awesome

In case you missed it, Intel CEO Bob Swan is stepping down from his role effective February 15 and will be replaced by current VMware CEO Pat Gelsinger. Gelsinger was the former CTO at Intel for a number of years before leaving to run EMC and VMware. His return is a bright spot in an otherwise dismal past few months for the chip giant.

Why is Gelsinger’s return such a cause for celebration? The analysts that have been interviewed say that Intel has been in need of a technical leader for a while now. Swan came from the office of the CFO to run Intel on an interim basis after the resignation of Brian Krzanich. The past year has been a rough one for Intel, with delays in their new smaller chip manufacturing process and competition heating up from long-time rival AMD but also from new threats like ARM being potentially sold to NVIDIA. It’s a challenging course for any company captain to sail. However, I think one key thing makes is nigh impossible for Swan.

Management Mentality

Swan is a manager. That’s not meant as a slight inasmuch as an accurate label. Managers are people that have things and look after them. Swan came from the financial side of the house where you have piles of resources and you do your best to account for them and justify their use. It’s Management 101. Managers make good CEOs for a variety of companies. They make sure that the moves are small and logical and will pay off in the future for the investors and eventually the workers as well. They are stewards first and foremost. When their background comes from something with inherent risk they are especially stewardly.

You know who else was a manager? John Sculley, the man who replaced Steve Jobs at Apple back in 1983. Sculley was seen as a moderating force to Jobs’ driving vision and sometimes reckless decision making skills. Sculley piloted the ship into calm waters at first but was ultimately sent packing because his decisions were starting to make less and less sense, such as exploring options to split Apple into separate companies and taking on IBM head-to-head on their turf.

Sculley was ousted and Jobs returned to Apple in 1993. It wasn’t easy at first but eventually the style of Jobs started producing results. Things like the iPod, iMac, and eventually the iPhone came from his vision. He’s a leader in that regard. Leaders are the ones that jump out and take risks to make big results. Leaders are people like John Kennedy that give a vision of going to the moon in a decade without the faintest idea how that might happen. Leadership is what drives companies.

Leaders, however, are a liability without managers. Leaders say “let’s go to the moon!” Managers sit down and figure out how to make that happen without breaking the budgets or losing too many people along the way. Managers are the grounded voices that guide leaders. Without someone telling a leader of the challenges to overcome they won’t see the roadblocks until the drive right into them.

Leaders without brakes on their vision have no reality to shape it. Every iMac has an Apple Lisa. Every iPod has the iPod Hi-Fi. Even the iPhone wasn’t the iPhone until the App Store came around against the original vision of Apple’s driving force. To put it another way, George Lucas is a visionary leader in filmmaking. However, when he was turned loose without management of his process we ended up with the messy prequel trilogy. Why was Empire Strikes Back such a good film? Because it had people like Lawrence Kasdan involved managing the process of Lucas creating art. They helped focus the drive of a leader and make the result something great.

Tech Leadership

Let’s bring this discussion back to Intel and Pat Gelsinger. I know he is the best person to lead Intel right now. I know that because Gelsinger is very much a tech leader. He has visions for how things need to be and he can see how to get there. He knows that reducing costs and reaving product lines at Intel isn’t going to make them a better company down the road no matter what the activist investors have to say on the matter. They may have wanted regime change when they petitioned the board back in December, but they may find the new king a bit harder to deal with.

Gelsinger is also a manager. Going from CTO to being COO at EMC and eventually CEO at VMware has tempered his technical chops. You can’t hope to run a company on crazy ideas and risky bets. Steve Jobs had people like Tim Cook in the background keeping him as grounded in reality as possible. Gelsinger picked up these skills in helming VMware and I think that’s going to pay off for him at Intel. Rather than running out to buy another company to augment capabilities that will never see the light of day, someone like him can see the direction that Intel needs to go and make it happen in a collected manner. No more FPGA acquisitions that never bear fruit. No more embarrassing sales of the mobile chip division because no one could capitalize on it.

Pat Gelsinger is the best kind of technical manager. I saw it in the one conversation I was involved in with him during an event. He stepped in to a talk between myself and a couple of analysts. He listened to them and to me and when he was asked for his opinion, he stopped for a moment to think. He asked a question to clarify and then gave his answer. That’s a tempered leader approach to things. He listened. He thought. He clarified. And then he made a decision. That means there is steel behind the fire. That means the driving factors of the decision-making process aren’t just “cool stuff” or “save as much money as we can”. What will happen is the fusion of the two that the company needs to stay relevant in a world that seems bent on passing it by.


Tom’s Take

I’ve worked for managers and I’ve worked for leaders. I don’t have a preference for one or the other. I’ve seen leaders sell half their assets to save their company. I’ve also seen them buy ridiculous stuff in an effort to build something that no one would buy. I’ve seen managers keep things calm in the middle of a chaotic mess. I’ve also seen them so wracked with indecision that the opportunities they needed to capitalize on sailed off into the sunset. If you want to be the best person to run a company as the CEO, whether it’s a hundred people or a hundred thousand, you should look to someone like Pat Gelsinger. He’s the best combination of a manager and leader that I’ve seen in a long time. In five years we will be talking about how he was the one to bring Intel back to the top of the mountain, both through his leadership and his management skills.

by networkingnerd at January 15, 2021 04:27 PM

ipSpace.net Blog (Ivan Pepelnjak)

Video: Multi-Layer Switching and Tunneling

After discussing the technology options one has when trying to get a packet across the network, we dived deep into two interesting topics:

  • How do you combine packet forwarding at multiple layers of OSI stack (multi-layer switching)?
  • What happens when you do layer-N forwarding over layer-M transport core where N <= M (example: IPv6 packets over IPv4 packets) aka tunneling?

You’ll find more details (including other hybrids like Loose Source Routing) in Multi-Layer Switching and Tunneling video.

The video is part of How Networks Really Work webinar and available with Free ipSpace.net Subscription.

January 15, 2021 07:38 AM

Potaroo blog

Addressing 2020

Time for another annual roundup from the world of IP addresses. Let’s see what has changed in the past 12 months in addressing the Internet and look at how IP address allocation information can inform us of the changing nature of the network itself.

January 15, 2021 06:00 AM

XKCD Comics

January 14, 2021

ipSpace.net Blog (Ivan Pepelnjak)

Imperative and Declarative API: Another Pile of Marketing Deja-Moo

Looks like some vendor marketers (you know, the same group of people who brought us the switching/routing/bridging stupidity) felt the need to go beyond the usual SDN and intent-based hype and started misusing the imperative versus declarative concepts. Unfortunately some networking engineers fell for the ploy; here’s a typical feedback along these lines I got from one of my readers:

I am frustrated by most people’s shallow understanding API’s, especially the differences between declarative (“what”) and imperative (“how”) API’s, and how that impacts one’s operations. Declarative APIs are the key pillar of what many vendors call “policy” or “intent-based” networking.

Let’s try to unravel that.

January 14, 2021 08:01 AM

Beware XML-to-JSON Information Loss (Junos with Ansible)

When you want to transport a complex data structure between components of a distributed system you’re usually using a platform-independent data encoding format like XML, YAML, or JSON.

XML was the hip encoding format in days when Junos and Cisco Nexus OS was designed and lost most of its popularity in the meantime due to its complexity (attributes, namespaces…) that makes it hard to deal with XML documents in most programming languages.

JSON is the new cool kid on the block. It’s less complex than XML, maps better into data structures supported by modern programming languages, and has decently fast parser implementations.

January 14, 2021 07:20 AM

January 13, 2021

SNOsoft Research Team

SolarWinds, SOX, and Corporate Responsibility for Cybersecurity

By now, most everyone has heard of the SolarWinds breach. Cybercriminals took advantage of SolarWinds’ poor cybersecurity practices to gain access to their network and implant malicious code within updates to their Orion network monitoring solution.

This Orion solution is widely used, and its compromise led to the attackers gaining access to the networks of many large enterprises and a significant percentage of US government agencies. As a result, intellectual property and sensitive government data has been compromised and much of it is being sold online. Investigations into the incident are still ongoing.

SolarWinds and SOX Disclosures

The SolarWinds breach has likely caused significant damage to the organization reputationally and financially. The damage caused by SolarWinds’ negligence is widespread, and the company will likely be the defendant in numerous lawsuits regarding the breach.

A recent class action lawsuit filed against the company’s leadership by SolarWinds shareholders demonstrates the potentially far-reaching impacts of such a breach. As a publicly-traded company, SolarWinds is subject to the Sarbanes-Oxley Act (SOX), which was passed in response to the Enron scandal to protect investors. Under SOX, a company’s CEO and CFO must sign an attestation that publicly-released statements regarding the company’s financial status are correct.

The lawsuit against SolarWinds focuses on a statement in SolarWinds’ 2019 10-K filing that acknowledges the risk of cyberattacks to the company. Based on this statement, the company acknowledges that this risk exists, that steps should be taken to mitigate this risk, and that any breach should be reported to shareholders.

SolarWinds was initially breached on September 4th, 2019, but the breach was not reported until December of the next year. Since the company has filed multiple 10-Q statements since with no reference to the breach, the plaintiffs in the SOX case allege that SolarWinds was negligent in managing its cybersecurity risk. Additionally, investigation into the incident revealed other instances of cybersecurity negligence, such as the use of a password solarwinds123 on the SolarWinds update server.

SolarWinds attack timeline
Source: Solarwinds

SOX Disclosures and the Cost of Poor Cybersecurity Due Diligence

Obviously, SolarWinds’ CEO and CFO are not directly responsible for detecting and remediating security incidents within their organization. However, they do hold overall responsibility, and the SOX Act allows them to be held personally responsible for misleading or false statements within SOX disclosures.

Any organization can suffer a security breach, but it is the responsibility of a company’s leadership to ensure that due diligence is performed to prevent incidents like the SolarWinds breach. SolarWinds failed to do their due diligence in two crucial ways:

  1. Internal Cybersecurity Failures: As SolarWinds mentions in their 10-K, it is impossible to fully protect against cybersecurity threats. However, the company failed to follow even the most basic cybersecurity best practices as demonstrated by the use of a blatantly insecure password (solarwinds123) on its update server.
  2. Failure to Perform Proper Security Testing: Passing a Penetration Test is not proof of strong cybersecurity, as demonstrated by Trustwave’s certification of Target before the 2013 breach. However, a Penetration Test should have detected the use of such a weak password on the update server. This oversight demonstrates a failure to perform proper due diligence on behalf of both SolarWinds and any organization that performed a Penetration Test for the company.

Taking Responsibility for Corporate Cybersecurity

The class action lawsuit against SolarWinds – if successful – creates a strong precedent for holding corporate executives personally responsible for their companies’ security failures. Under the SOX Act, executives can face 10 years in prison and a $1 million fine for signing off on misleading statements, and 20 years and $5 million if the deception was willful.

In cybersecurity, as in any field, mistakes can be made, and companies can be breached despite their best efforts. However, making a “good faith” effort toward strong corporate cybersecurity – including contracting regular Penetration Tests by a competent testing firm – is essential to earning forgiveness for cybersecurity failures. The appearance of good security isn’t the same as the real thing.

<style type="text/css">.fusion-fullwidth.fusion-builder-row-6 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link) , .fusion-fullwidth.fusion-builder-row-6 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):before, .fusion-fullwidth.fusion-builder-row-6 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-6 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover, .fusion-fullwidth.fusion-builder-row-6 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:before, .fusion-fullwidth.fusion-builder-row-6 a:not(.fusion-button):not(.fusion-builder-module-control):not(.fusion-social-network-icon):not(.fb-icon-element):not(.fusion-countdown-link):not(.fusion-rollover-link):not(.fusion-rollover-gallery):not(.fusion-button-bar):not(.add_to_cart_button):not(.show_details_button):not(.product_type_external):not(.fusion-quick-view):not(.fusion-rollover-title-link):not(.fusion-breadcrumb-link):hover:after {color: #f2b310;}.fusion-fullwidth.fusion-builder-row-6 .pagination a.inactive:hover, .fusion-fullwidth.fusion-builder-row-6 .fusion-filters .fusion-filter.fusion-active a {border-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-6 .pagination .current {border-color: #f2b310; background-color: #f2b310;}.fusion-fullwidth.fusion-builder-row-6 .fusion-filters .fusion-filter.fusion-active a, .fusion-fullwidth.fusion-builder-row-6 .fusion-date-and-formats .fusion-format-box, .fusion-fullwidth.fusion-builder-row-6 .fusion-popover, .fusion-fullwidth.fusion-builder-row-6 .tooltip-shortcode {color: #f2b310;}#main .fusion-fullwidth.fusion-builder-row-6 .post .blog-shortcode-post-title a:hover {color: #f2b310;}</style>

The post SolarWinds, SOX, and Corporate Responsibility for Cybersecurity appeared first on Netragard.

by Adriel Desautels at January 13, 2021 10:34 PM

Packet Pushers

Managers Must Prep Their Teams For Kubernetes – Video

If the boss demands a Kubernetes deployment, and flies in a team of consultants to get the project off the ground, what should the IT staff be prepared for once the consultants depart? That’s the question in this excerpt of Day Two Cloud podcast “Why Kubernetes Is Wrong For You.” You can listen to the […]

The post Managers Must Prep Their Teams For Kubernetes – Video appeared first on Packet Pushers.

by The Video Delivery at January 13, 2021 09:20 PM

XKCD Comics

January 12, 2021

Ethan Banks on Technology

The Attention Economy And The IT Talent Dearth

In IT operations, finding talent is difficult. For years, there has been a shortage of folks who are capable of maintaining complex infrastructure. To be sure, some of this is geographical. And certainly, the rate of technology change makes it difficult to find people with specific product skills. Hard to find a Kubernetes expert with ten years of experience. 🙄

But I suspect there’s a couple of other things going on that, when combined, make the talent dearth even worse.

The Brutality Of Complexity

When I was studying for Novell Netware 3 (before directory services) certifications decades ago, there was a lot to know. Networking with IPX. Architecture of x86 servers. NLMs. Storage strategies. Mail systems. Whatever else was in those red books many of us had on our shelves.

Pre-AD Microsoft certifications were similarly challenging. Domain controllers. Backup domain controllers. File & print systems. User permissions and design strategies. The GINA. Networking with IP, IPX, and NetBEUI. Mail systems. IIS. So much more.

That was before the addition of directory services to Novell and Microsoft operating systems. Directory services changed the game for file, print, email, and more back in the day, and it put a major burden on IT practitioners to skill up. Now, directory services is an authentication complexity we take for granted.

If you compare those early client/server operating systems and peripheral software, the complexity level is laughably simple compared to what an IP pro is meant to know today. Specialty certifications are deep and numerous. Operating infrastructure on-premises is still a need, but public cloud has added yet its own demands on IT professions. Public cloud is NOT outsourcing IT to Uncle Jeff, contrary to the lies told in executive suites.

I’ve yet to mention security, an utter failure on the part of the IT industry to get right. But even insufficient to protect our data as it is, cybersecurity adds an enormous complexity to application delivery infrastructure.

The state of the IT art is brutally complex. Even a basic comprehension of how a client/server transaction actually happens touches on physical networks, several protocols both standard and proprietary, encryption, authentication, deep packet inspection, server hardware, virtualization, operating systems, containers, certificates, tokens, databases, logging, and more. Much more, sadly.

An IT professional develops skills over a career, but it’s a career where there is never time to rest. It’s a career of churn and change. The next project. Implementing the next pile of crap a vendor sold their boss. The next changeover to the new shiny object.

Yes, many of the old skills translate to new ones. There are recognizable patterns in how computing problems are solved. (Buy the book I co-authored to study how this is true for networking.) But if I’m new to IT, how can I hope to comprehend what’s happening at a systems level? There is a mix of focused on-the-job training and diligent, committed, unceasing study required.

A Society Of Unfocused Attention

We are in a global society that, by and large, cherishes the short attention span. Amongst our tweets, Facebook posts, YouTube subscriptions, Netflix trending vapidity, and propaganda masquerading as news, we are a people who stare at screens, consuming mental garbage at maximum velocity–not merely at an alarming rate, although the rate is alarming. We take it in as fast as we possibly can.

We watch a 15 minute YouTube video and feel proud for having lasted that long. A 2+ hour superhero movie is only endurable because the images and scenes move so quickly that our brain stays bathed in happy chemicals.

Ads have become shorter and shorter so that we can’t hit the skip button quickly enough. A marketer can’t get a meaningful message out about their product anymore, and they no longer try. Entertainment is the order of the day, where geckos sell us insurance and athletes mug for fizzy sugar water, sharing the frame with sexy, gyrating humans.

Our brains are now trained to go to the next thing. And the next. And the next. And the next. Infinite scrolling. Amuse me. Entertain me. TikToks are only 30 seconds long. 30 seconds! Yet Tiktok made the news in December 2020 by experimenting with the unthinkably long time of 3 minutes.

Training The Next Generation

How are we as an IT industry going to properly train the next generation of IT engineers? This is a serious question. Shall we bait them with money? Perhaps. That’s a motivator for some. But what about when UBI becomes a reality, which is what the pandemic unemployment funds essentially were? Observational experience suggests that folks who are paid to do nothing…do nothing.

How will we take brutally complex application delivery systems and gain the attention required for people to become competent in operating them?

While pondering this issue with a friend, he pointed out that a major publisher is now limiting video lessons to 6 minutes–8 minutes being a hard maximum. Yes, the aggregated course material can be much longer. But if students have been deemed incapable of maintaining focus on a technical topic for longer than 6 minutes at a time, what does that say for the future of IT practice?

Technology mastery will be increasingly in the hands of the very few as a dwindling number of folks are willing, or perhaps even able, to create a mental state of focused learning. The application delivery stacks are enormously more complex than they were 25 years ago. Learning them requires a huge amount of focus over long periods of time.

Six Minutes Is Not A Solution

Is the answer really 6 minute lessons? Or is the answer for individuals to reject social media, reduce video consumption, and rediscover long-form learning? I vote for the latter. It’s up to us to take back control of whatever we’ve allowed to own our minds.

Let’s learn to read long, technically demanding books again. Let’s remember what it’s like to get into a lab exercise for a few hours at a time. Let’s practice doing One Thing™ for an uninterrupted span of time–no email, IMs, PMs, DMs, Slack adventures, badge notifications, browser tabs, or multitasking distracting us from the One Thing™.

What if we’re in the role of education? Writing blogs. Sharing educational videos. Delivering a presentation. Is the answer to entertain non-stop in order to drive engagement? I’m an instructor and not a comic, although I’ll share dad puns with reckless abandon. But how did we get to the point where how long someone paid attention (i.e. didn’t click away) was a success metric, versus what someone actually learned?

When I was a high school teacher, success was measured, in part, by the grades of my students. I surely worked hard not to be boring and to hold a classroom’s attention. Teaching, especially live teaching, is admittedly a sort of performance art. But I don’t think it’s possible to compete with the distraction economy. When teaching adults hard topics, they have to want to learn.

Going back to the problem of a dearth of IT talent, I believe this issue of attention might be getting to the root of it. Who wants to learn IT anymore? What’s their motivation? If their brains are already owned by social media, what’s left for the rest of their lives? How many folks that started reading this piece made it this far? Those that clicked away aren’t the sort that are going to be effective IT practitioners, staying current and learning lifelong.

I don’t have a clear solution to this dilemma, but I’m fairly sure the answer isn’t catering to those with attention deficits. I don’t want to accommodate the problem. I want to smash it.

by Ethan Banks at January 12, 2021 04:53 PM

Packet Pushers

Interconnecting GNS3 Virtual Machines – Video

GNS3 co-founder and developer Jeremy Grossman and networking instructor David Bombal talk with Ethan Banks about how separate GNS3 VMs communicate. You can listen to the full episode, “Heavy Networking 556: The State Of GNS3 For Network Labs,” by clicking this link. Heavy Networking is part of the Packet Pushers network of technical podcasts, including […]

The post Interconnecting GNS3 Virtual Machines – Video appeared first on Packet Pushers.

by The Video Delivery at January 12, 2021 01:00 PM

ipSpace.net Blog (Ivan Pepelnjak)

Automation Win: Chatops-Based Security

It’s amazing how quickly you can deploy new functionality once you have a solid foundation in place. In his latest blog post Adrian Giacometti described how he implemented a security solution that allows network operators to block source IP addresses (identified by security tools) across dozens of firewalls using a bot listening to a Slack channel.

Would you be surprised if I told you we covered similar topics in our automation course? 😇

January 12, 2021 06:42 AM

January 11, 2021

Packet Pushers

Give The Network Designer That Came Before You A Break

When you take over a network as a technical lead, you often run into design elements that make you do a spit-take. They did WHAT? Really? Were they...stupid? Clueless? Stupid AND clueless? Maybe they were, but I argue that you should give those humans that came before you a break. You weren't there. You don't know what constraints they were operating under. Since you don't know those things, it's hard to pass fair judgement. Unfair judgement? Oh, yeah. All day long, and you can even feel righteous while doing so. Super smug.

The post Give The Network Designer That Came Before You A Break appeared first on Packet Pushers.

by Ethan Banks at January 11, 2021 09:13 PM

Understanding GNS3 Appliances – Video

The labbing tool GNS3 has a capability called “appliances” but it may not mean what you think it means. GNS3 co-founder and developer Jeremy Grossman and networking instructor David Bombal talk with Ethan Banks about what appliances mean in the context of this software. You can listen to the full episode, “Heavy Networking 556: The […]

The post Understanding GNS3 Appliances – Video appeared first on Packet Pushers.

by The Video Delivery at January 11, 2021 08:30 PM

Automating Labs With The GNS3 API – Video

GNS3 co-founder and developer Jeremy Grossman and networking instructor David Bombal discuss the capabilities of GNS3’s API. GNS3 is a popular tool for creating virtual networks for labbing. You can listen to the full episode here: Heavy Networking 556: The State Of GNS3 For Network Labs. Heavy Networking is part of the Packet Pushers network […]

The post Automating Labs With The GNS3 API – Video appeared first on Packet Pushers.

by The Video Delivery at January 11, 2021 06:32 PM

ipSpace.net Blog (Ivan Pepelnjak)

Webinars in 2021

After deciding to take a slightly longer coffee break I went through the list of outstanding projects trying to figure out which ones I could complete in first half of 2021, which ones I’ll get to “eventually” and what’s a lost cause.

Guest Speakers

Irena is telling me that I should stop inviting guest speakers – our calendar is full until June 2021. Here’s what we have planned:

January 11, 2021 05:02 PM

Cioara's Cisco Blog

How to Re-Enable Backspace Key in Firefox

The Firefox web browser has enabled the backspace key to double as back button for many versions, however Mozilla recently decided to remove this functionality. If you are one of the many longtime Firefox users who rely on this feature and want to enable it again here is how you do it: Open a new […]

The post How to Re-Enable Backspace Key in Firefox appeared first on tekopolis.

by Adam at January 11, 2021 04:47 PM

My Etherealmind
XKCD Comics

January 09, 2021

ipSpace.net Blog (Ivan Pepelnjak)

Worth Reading: AI/ML/Space Predictions Scorecard, 2021 Edition

In January 2018 Rodney Brooks made a series of long-term predictions about self-driving cars, robotics, AI, ML, and space travel. Not surprisingly, his predictions were curmudgeonly and pessimistic when compared to the daily hype (or I wouldn’t be blogging about it)… but guess who was right ;)

He’s also the only predictor I’m aware of who is not afraid to compare what he wrote with how reality turned out years down the line. On January 1st he published the 2021 edition of the predictions scorecard and so far he hasn’t been too pessimistic yet. Keep that in mind the next time you’ll be listening to your favorite $vendor droning about the wonders of AI/ML.

January 09, 2021 04:23 PM

January 08, 2021

The Networking Nerd

Building Backdoors and Fixing Malfeasance

You might have seen the recent news this week that there is an exploitable backdoor in Zyxel hardware that has been discovered and is being exploited. The backdoor admin account with the clever name ‘zyfwp’ is not something that has been present in the devices forever. The account was put in during firmware version 4.60, which was released in Q4 2020.

Zyxel is rushing to patch the devices and remove the backdoor account. Users are being advised to disable remote administration until the accounts can be deactivated and proven to be removed. However, the bigger question in my mind relates to the addition of the user account in the first place. Why would you knowingly install a backdoor?

Hello, Joshua

Backdoors are nothing new in the computer world. I’d argue the most famous backdoor account in the history of computer hacking belongs to Joshua, the dormant login for the War Operations Programmed Response (WOPR) computer system in the 1983 movie Wargames. Joshua was an old login for the creator to access the system outside of the military chain of command. When the developer was removed from the project the account was forgotten about until a kid discovered it and kicked off the plot of the movie.

Joshua tells us a lot about developers and their desire to have access to the system. I’ll admit I’ve been in the same boat before. I’ve created my own logins to systems with elevated access to get tasks accomplished. I’ve also notified the users and administrators of those systems about my account and let them deal with it as needed. Most were okay with it being there. Some were hesitant and required it to be disabled after my work was done. Either way, I was up front about what was going on.

Joshua and zyfwp are examples of what happens when those systems are installed outside of the knowledge of the operators. What would have happened if the team in the Netherlands hand’t found the account? What if Zyxel devices were getting hacked and networks breached without anyone knowing the vector? I’m sure the account showed up in all the admin dashboards, right?

Easter Egg Hunts

Do you remember the Windows 3.1 Bear? It was a hidden reference in the credits to the development team’s mascot. You had to jump through a hoop to find it by holding down a keystroke combination and clicking a specific square in the Windows logo. People loved finding those little nuggets in the software all the way up to Windows 98.

What changed? Turns out, as part of Microsoft’s Trustworth Computing Initiative in 2002 they removed all undocumented features and code that could cause these kinds of things. It also might have had something to do with the antitrust investigations into Microsoft in the 1990s and how undocumented features in Windows and Office might have given the company a competitive advantage. Whatever the reason, Microsoft has committed to removing undocumented code.

Easter eggs are fun to find but represent the bright side of the dark issue above. What happens when the easter egg in question isn’t a credit roll but an undocumented account? What if the keystroke doesn’t bring up a teddy bear but instead gives the current user account full admin access? You scoff at the possibility but there’s nothing stopping a developer from making that happen.

These issues are part of the reason why all code and features need to be documented. We need to know what’s going on in the program and how it could impact us. This means no backdoors. If there is a way to access the system aside from the controls built in already it needs to be known and be able to be disabled if necessary. If it can’t be disabled then the users need to be aware of that fact and make the choice to not use the software because of security issues.

If you’re following along closely, you should have picked up on the fact that this same logic applies to backdoors that have been mandated by the government too. The current slate of US Senators seem to believe that we need to create keys that allow end-to-end encryption to be weakened and readable by law enforcement. However, as stated by companies like Apple for years, if you create a key for a lock that should only ever be opened under special circumstances you have still created a weakness that can be unlocked. We’ve seen the tools used by intelligence agencies stolen and used to create malware unlike anything we’ve ever seen before. What do you think might happen if they get the backdoor keys to go through encrypted messaging systems?


Tom’s Take

I don’t run Zyxel equipment in my home or anywhere I used to work. But if I did there would be a pile of it in the dumpster after this mess. Having a backdoor is one thing. Purposely making one is another. And having that backdoor discovered and exploited by the Internet is an entirely differently conversation. The only way to be sure that you’ve fixed your backdoor problem is to not have one in the first place. Joshua and zyfwp are what we need to get away from, not what we need to work toward. Malfeasance only stops when you don’t do it in the first place.

by networkingnerd at January 08, 2021 11:03 PM

My Etherealmind

Juniper has Self-Study Bundles

Not relying on third parties is probably good and reduces the grifters

by Greg Ferro at January 08, 2021 12:33 PM

ipSpace.net Blog (Ivan Pepelnjak)

Video: Cisco SD-WAN Policies and Centralized Magic

Right after Cisco SD-WAN devices are onboarded, how are the control and data plane tasks started? In this section, David Penaloza covers how Cisco SD-WAN solution makes the most of its SDN nature: single point of policy application and centralized management platform. The types of policies, the plane on which they act, their application and the actions that can performed are the main focus in this part of the series.

You need Free ipSpace.net Subscription to watch the video.

January 08, 2021 06:33 AM

About Networks

25 years as a Network Engineer!

Cisco-2501

In January 1996, I entered for the first time in the configuration of a Cisco 2501 router. This was the beginning of my career as a network engineer. That was just 25 years ago! Here’s a quick look back and a few tips for junior engineers who are at the beginning of their careers.   25 years as a Network Engineer! In 25 years, I had the opportunity to change several times my working environment and specialty as a network engineer: I went from network engineer and peering-manager for regional…

The post 25 years as a Network Engineer! appeared first on AboutNetworks.net.

by Jerome Tissieres at January 08, 2021 12:33 AM

XKCD Comics

January 07, 2021

My Etherealmind
ipSpace.net Blog (Ivan Pepelnjak)

Considerations for Host-based Firewalls (Part 2)

This is a guest blog post by Matthias Luft, Principal Platform Security Engineer @ Salesforce, and a regular ipSpace.net guest speaker.

A couple of months ago I had the pleasure to publish my first guest post here and, as to be expected from ipspace.net, it triggered some great discussion.

With this input and some open thoughts from the last post, I want to dive into a few more topics.

January 07, 2021 07:19 AM

January 06, 2021

My Etherealmind

How Change Review Boards Really Work to Fail

Its supposed to enable change but reality is about preventing change.

by Greg Ferro at January 06, 2021 03:34 PM

Packet Pushers

Zero Trust Networking: Too Hard? – Video

The concept of Zero Trust sounds good–scrutinize individual users or end points or application sessions and make access decisions based on a set of policies and attributes. It also sounds wicked hard. Mike (Zig) Zsiga discusses the notion of zero trust, and whether it’s achievable, in this excerpt of the Packet Pushers’ podcast episode “Heavy […]

The post Zero Trust Networking: Too Hard? – Video appeared first on Packet Pushers.

by The Video Delivery at January 06, 2021 10:30 AM

ipSpace.net Blog (Ivan Pepelnjak)

IBGP, IGP Metrics, and Administrative Distances

TL&DR: If you run multiple IGP protocols in your network, and add BGP on top of that, you might get the results you deserve. Even better, the results are platform-dependent.

One of my readers sent me a link to an interesting scenario described by Jeremy Filliben that results in totally unexpected behavior when using too many routing protocols in your network (no surprise there).

Imagine a network in which two edge routers advertise the same (external) BGP prefix. All other things being equal, it would make sense that other routers in the same autonomous system should use the better path out of the autonomous system. Welcome to the final tie-breaker in BGP route selection process: IGP metric.

January 06, 2021 06:39 AM

XKCD Comics

January 05, 2021

Packet Pushers

Where Network Design Fits Into Cloud Native – Video

Where do network engineers fit into the re-architecting of applications for the cloud? Do they just need to provide bandwidth, or is there more to it? Mike (Zig) Zsiga stops by the Packet Pushers’ Heavy Networking podcast to answer this and other questions about network design trends. You can listen to the full episode, Heavy […]

The post Where Network Design Fits Into Cloud Native – Video appeared first on Packet Pushers.

by The Video Delivery at January 05, 2021 10:20 PM

ipSpace.net Blog (Ivan Pepelnjak)

Planning the Next Extended Coffee Break

Long story short: ipSpace.net is going on an extended coffee break on June 24th 2021. You can stop reading; the rest of the blog post is full of details you probably don’t care about.

What exactly does that mean? Honestly, we don’t know yet… but we felt that it’s only fair to let engineers considering our subscriptions know months in advance what might happen.

Also, after investing two lifetimes into this project, and a few planned changes coming just before our regular summer hiatus (see below) it’s time for a longer break. ipSpace.net might be back to business-as-usual after a few months (unlikely), or it could be Ivan working on some interesting stuff (most likely) or ipSpace.net slowly disappearing into the sunset (not impossible).

January 05, 2021 08:44 AM