Hardware vendors are always making their silicon more complex and feature-rich. Is that a great idea or a disaster waiting to happen? We asked Luke Gorrie, the lead developer of Snabb Switch (an open-source user-land virtual switch written in Lua) about his opinions on the topic.

TL&DL version: Give me a dumb NIC, software can do everything else.

If you want to know more, listen to Episode 93 of Software Gone Wild.

As anyone starting their journey into AWS quickly discovers, cloud is different (or as I wrote in the description of my AWS workshop you feel like Alice in Wonderland). One of the gotchas: when you link multiple routing domains (Virtual Private Clouds – the other VPC) you have to create static routing table entries on both ends. Even worse, there’s no transit VPC – you have to build a full mesh of relationships.

The correct solution to this challenge is automation:

Read more ...

This is the third blog post in “thinking out loud while preparing Network Infrastructure as Code presentation for the network automation course” series. You might want to start with Network-Infrastructure-as-Code Is Nothing New and Adjusting System State blog posts.

As I described in the previous blog post, the hardest problem any infrastructure-as-code (IaC) tool must solve is “how to adjust current system state to desired state described in state definition file(s)”… preferably without restarting or rebuilding the system.

There are two approaches to adjusting system state:

Read more ...

Here’s a bit of good news for those of you scared of network automation replacing your jobs: even Elon Musk didn’t manage to pull it off, so I don’t think a networking vendor dabbling in intent will manage to do it (particularly considering the track record of networking vendors’ network management and orchestration systems).

Read more ...

You might have noticed that almost every BGP as Data Center IGP design uses the same AS number on all spine switches (there are exceptions coming from people who use BGP as RIP with AS-path length serving as hop count… but let’s not go there).

There are two reasons for that design choice:

Read more ...

Have you ever taken the opportunity to think about something from a completely different perspective? Or seen someone experience something you have seen through new eyes? It’s not easy for sure. But it is a very enlightening experience that can help you understand why people sometimes see things entirely differently even when presented with the same information.

Overcast Networking

The first time I saw this in action was with Aviatrix Systems. I first got to see them at Cisco Live 2018. They did a 1-hour presentation about their solution and gave everyone an overview of what it could do. For the networking people in the room it was pretty straightforward. Aviatrix did a lot of the things that networking should do. It was just in the cloud instead of in a data center. It’s not that Aviatrix wasn’t impressive. It’s the networking people have a very clear idea of what a networking platform should do.

Fast forward two months to Cloud Field Day 4. Aviatrix presents again, only this time to a group of cloud professionals. The message was a little more refined from their first presentation. They included some different topics to appeal more to a cloud audience, such as AWS encryption or egress security. The reception from the delegates was the differencue between night and day. Rather than just be satisfied with the message that Aviatrix put forward, the Cloud Field Day delegates were completely blown away! They loved everything that Aviatrix had to say. They loved the way that Aviatrix approached a problem they had seen and couldn’t quite understand. How to extend networking into the cloud and take control of it.

Did Aviatrix do something different? Why was the reaction between the two groups so stark? How did it happen this way? I think it is in part because networking people talk to a networking company and see networking. They find the things they expect to find and don’t look any deeper. But when the same company presents to an audience that doesn’t have networking on the brain for the entirety of their career it’s something entirely different. While a networking audience may understand the technology a cloud audience may understand how to make it work better for their needs because they can see the advantages. Perspective matters in this case because people exposed to new ideas find ways to make them work in ways that can only be seen with fresh eyes.

Letting Go of Wires

The second time I saw an example of perspective at play was at Mobility Field Day 3 with Arista Networks. Arista is a powerhouse in the data center networking space. They have gone up against Cisco and taken them head-to-head in a lot of deals. They have been gaining marketshare from Cisco in a narrow range of products focused on the data center. But they’re also now moving into campus switching as well as wireless with the acquisition of Mojo Networks.

When Arista stepped up to present at Mobility Field Day 3, the audience wasn’t a group of networking people that wanted to hear about CloudVision or 400GbE or even EOS. The audience of wireless and mobility professionals wanted to hear how Arista is going to integrate the Mojo product line into their existing infrastructure. The audience was waiting for a message that everything would work together and the way forward would be clear. I don’t know that they heard that message, but it wasn’t because of anything that Arista did on purpose.

Arista is very much trying to understand how they’re going to integrate Mojo Networks into what they do. They’re also very focused on the management and control plane of the access points. These are solved problems in the wireless world right now. When you talk to a wireless professional about centralized management of the device or a survivable control plane that can keep running if the management system is offline they’ll probably laugh. They’ve been able to experience this for the past several years so far. They know what SDN should look like because it’s the way that CAPWAP controllers have always operated. Wireless pros can tell you the flaws behind backhauling all your traffic through a controller and why there are much better options to keep from overwhelming the device.

Wireless pros have a different perspective from networking people right now. Things that networking pros are just now learning about are the past to wireless people. Wireless pros are focused more on the radio side of the equation than the routing and switching side. That perspective gives the wireless crowd a very narrow focus on solving some very hard problems but it does make them miss the point that their expertise can be invaluable to helping both networking pros and networking companies see how to take the best elements of wireless networking control mechanisms and implement them in such a way as to benefit everyone.

---

Tom’s Take

For me, the difficulty in seeing things differently doesn’t come from having an open mind. Instead, it comes from the fact that most people don’t have a conception of anything outside their frame of reference. We can’t really comprehend things we can’t conceive of. What you need to do to really understand what it feels like to be in someone else’s shoes is have someone show you what it looks like to be in them. Observe people learning something for the first time. Or see how they react to a topic you know well. Odds are good you might just find that you will know it better because they helped you understand it better.

Pradosh Mohapatra, the author of last week’s SD-WAN Overview webinar started his presentation with a seemingly simple question: What Is SD-WAN?

You need at least free ipSpace.net subscription to watch his answer.

This blog post was initially sent to the subscribers of my SDN and Network Automation mailing list. Subscribe here.

Saša Ratković (Apstra) published a must-read Intent-Based Networking Taxonomy which (not surprisingly) isn’t too far from what I had to say about the topic in a blog post and related webinar.

It’s also interesting to note that the first three levels of intent-based networking he described match closely what we’re discussing in Building Network Automation Solutions online course and what David Barroso described in Network Automation Use Cases webinar:

Read more ...

This blog post was initially sent to the subscribers of my SDN and Network Automation mailing list. Subscribe here.

Tom Limoncelli published a must-read article in ACM Queue describing GitOps – the idea of using Pull Requests together with CI/CD pipeline to give your users the ability to request changes to infrastructure configuration.

Using GitOps in networking is nothing new – Leslie Carr talked about this concept almost three years ago @ RIPE 71, and I described some of the workflows you could use in Network Automation 101 webinar.

Read more ...

This is the second blog post in “thinking out loud while preparing Network Infrastructure as Code presentation for the network automation course” series. If you stumbled upon it, you might want to start here.

An anonymous commenter to my previous blog post on the topic hit the crux of the infrastructure-as-code challenge when he wrote: “It's hard to do a declarative approach with Ansible and the nice network vendor APIs.” Let’s see what he was trying to tell us.

Read more ...

Last year’s experiment generated so much interest that I decided to repeat it this year: if you’re an undergraduate or Master's student and manage to persuade us that you’re motivated enough to automate the **** out of everything, you’ll get a free seat in Ansible for Networking Engineers online course.

Interested? Check out the details, and apply before October 1^st.

Too old? Please spread the word ;)

As I wrote on my recent post here, I was involved into a project to implement a Meraki MX into the Azure Cloud. This project also includes a migration phase with site-to-site VPN tunnels between Meraki MX and Cisco ASA. Even if the “Non-Meraki VPN peers” are supported on the Meraki MX, you may have
Read More »

The post Site-to-site VPN tunnels between Meraki MX and Cisco ASA appeared first on AboutNetworks.net.

Here’s a question I got from an attendee of my Building Next-Generation Data Center online course:

As far as I understood […] it is obsolete nowadays to build a new DC fabric with routing on the host using BGP, the proper way to go is to use IGP + SDN overlay. Is my understanding correct?

Ignoring for the moment the fact that nothing is ever obsolete in IT, the right answer is it depends… this time on answer(s) to two seemingly simple questions “what services are we offering?” and “what connectivity problem are we trying to solve?”.

Read more ...

Most blog posts generate the usual noise from the anonymous peanut gallery (if only they'd have at least a sliver of Statler and Waldorf in them), but every now and then there's a comment that's pure gold. The one made by Tony Przygienda (of RIFT fame) on Valley-Free Routing post is so good and relevant that I decided to republish it as a separate blog post. Enjoy!

Read more ...

About six months ago, I got fed up with my Meraki MR34 APs. They ran just fine, but they needed attention. They needed licenses. They needed me to pay for a dashboard I used rarely but yet had to keep up yearly. And that dashboard had most of the “advanced” features hidden away under lock and key. I was beyond frustrated. I happen to be at the Wireless LAN Professionals Conference (WLPC) and ran into Darrell DeRosia (@Darrell_DeRosia) about my plight. His response was pretty simple:

“Dude, you should check out Ubiquiti.”

Now, my understanding of Ubiquiti up to that point was practically nothing. I knew they sold into the SMB side of the market. They weren’t “enterprise grade” like Cisco or Aruba or even Meraki. I didn’t even know the specs on their APs. After a conversation with Darrell and some of the fine folks at Ubiquiti, I replaced my MR34s with a UniFI AP-AC-HD and an AP-AC-InWall-Pro. I also installed one of their UniFi Security Gateways to upgrade my existing Linksys connection device.

You may recall my issue with redundancy and my cable modem battery when I tried to install the UniFi Security Gateway for the first time. After I figured out how to really clear the ARP entries in my cable modem I got to work. I was able to install the gateway and get everything back up and running on the new Ubiquiti APs. How easy was it? Well, after renaming the SSID on the new APs to the same as the old one, I was able to connect all my devices without anyone in the house having to reconnect any of their devices. As far as they knew, nothing changed. Except for the slightly brighter blue light in my office.

I installed the controller software on a spare machine I had running. No more cloud controllers for me. I knew that I could replicate those features with a Ubiquiti Cloud Key, but my need to edit wireless settings away from home was pretty rare.

Edit: As pointed out by my fact checked Marko Milivojevic, you don’t need a Cloud Key for remote access. The Cloud Key functions more as a secure standalone controller instance that has remote access capabilities. You can still run the UniFi controller on lots of different servers, including dedicated rack-mount gear or a Mac Mini (like I have).

I logged into my new wireless dashboard for the first time:

It’s lovely! It gives me all the info I could want for my settings and my statistics. At a glance, I can see clients, devices, throughput, and even a quick speed test of my WAN connection. You’re probably saying to yourself right now “So what? This kind of info is tablestakes, right?” And you wouldn’t be wrong. But, the great thing about Ubiquiti is that its going to keep working after 366 days of installation without buying any additional licenses. It’s not going to start emailing me telling me it’s time to sink a few hundred dollars into keeping the lights on. That’s a big deal for me at home. Enterprises may be able to amortize license costs over the long haul but small businesses aren’t so lucky.

The Ubiquiti UniFi dashboard also has some other great things. Like a settings page:

Why is that such a huge deal? Well, Ubiquiti doesn’t remove functionality from the dashboard. They put it where you can find it. They make it easy to tweak settings without wishing on a star. They want you to use the wireless network the way you need to use it. If that means enabling or disabling features here and there to get things working, so be it. Those features aren’t locked away behind a support firewall that needs an act of Congress to access.

But the most ringing endorsement of Ubiquiti for me? Zero complaints in my house. Not once has anyone said anything about the wireless. It just “works”. With all the streaming and Youtube watching and online video game playing that goes on around here it’s pretty easy to saturate a network. But the Ubiquiti APs have kept up with all the things that have been thrown at them and more.

I also keep forgetting that I even have them installed. That’s a good thing. Because I don’t spend all my time tinkering with them they tend to fade away into the background of the house. Even the upstairs in-wall AP is chugging right along and serving clients with no issues. Small enough to fit into a wall box, powerful enough to feed Netflix for a whole family.

---

Tom’s Take

I must say that I’m very impressed by Ubiquiti. My impressions about their suitability for SMB/SME was all wrong. Thanks to Darrell I now know that Ubiquiti is capable of handling a lot of things that I considered “enterprise only” features. Even Lee Hutchinson at Are Technica is a fan of Ubiquiti at home. I also noticed that the school my kids attend installed Ubiquiti APs over the summer. It looks like Ubiquiti is making in-roads into SMB/SME and education. And it’s a very workable solution for what you need from a wireless system. Add in the fact that the software doesn’t require yearly upkeep and it makes all the sense in the world for someone that’s not ready to commit to the treadmill of constant licensing.

Johannes Weber was forced to stress-test the IPv6 networks are easy to renumber nonsense and documented his test results – a must-read for everyone deploying IPv6.

He found out that renumbering IPv6 in his lab required almost four times as many changes as renumbering (outside) IPv4 in the same lab.

My cynical take on that experience: “Now that you’ve documented everything that needs to be changed, make sure it’s automated the next time ;)”

 There is a recurring theme at Black Hat every year: security researchers come together and show the world how to hack into systems and things. This year was no exception. 

 

The sheer size of the cyber-attack surface becomes more daunting by the day. Networks now connect data centers and on-premise hardware to private and public clouds and IoT devices operating at the network edge, exposing more potential entry points and increasing vulnerability. The growing attack landscape places defenders and security teams at a disadvantage against cybercriminals. From the booth demonstrations to the keynote speeches and session presentations, this reality was inescapable at Black Hat.

 

Especially with IoT devices, the explosion of endpoints creates more vulnerabilities and exacerbates potential security risks. As consumers and enterprises witness the rapid proliferation of IoT – from smart watches and home security systems to medical devices and industrial equipment – security has widely been an afterthought. The majority of stock IoT devices are not designed or constructed with cybersecurity in mind, which is why we see PoCs compromise smart city systems and telecom gateways.

 

  

There were two sessions that I particularly enjoyed and that stood out to me, both centered around machine learning. The first session showed how supervised machine learning can lead a security solution down the wrong path when data scientists training the models are not paired with subject matter experts. The other session showed how deep neural networks can be used to mount a targeted attack where both the intended victim’s identity and the malicious payload are hidden within the ML models and are impossible to extricate by reverse engineering.

 

When touring the exhibit floor, a striking observation was that the majority of exhibiting vendors act as though they have solved the security problem for everyone. As an industry, it’s important we remember that our customers are people responsible for defending their networks against all kinds of attacks and in turn we have to help them achieve that goal, not swamp them with flashy screens that bypass the difficult use cases. Security vendors are for-profit companies and there is nothing wrong with that, but we need to make sure we are wowing customers with real capabilities that solve some of the tough problems they’re facing while at the same time clearly showing them where the gaps exist. No single vendor can solve the security challenge on their own – this is an already established reality. So, we need to help customers identify the gaps that they need to close with a handful of solutions. It comes down to the fact that we are all fighting the same fight, vendors and customers alike.

 

New security challenges are emerging from the explosive growth of connected devices, multicloud adoption and 5G, and they do not have clear solutions. Although the conference shed light on the challenges the security industry needs to address, unfortunately, solutions were not abundant because these challenges are complex and lack a silver bullet. There is no quick fix for securing the vast attack surface created by our networks. But, as a defender, I’m optimistic at the industry’s commitment to meet these challenges head-on. One initiative worth mentioning is the creation of the Cyber Threat Alliance in which major security vendors are sharing threat intelligence data on a daily basis to raise everyone’s customers’ security posture to a better level. Juniper Networks is a member of this alliance and strongly believes that defending all customers’ networks, not just those with Juniper security products, is a cause worth pursuing.

 

<iframe allow="autoplay" frameborder="no" height="166" scrolling="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/495150843&amp;color=%23ff5500&amp;auto_play=false&amp;hide_related=false&amp;show_comments=true&amp;show_user=true&amp;show_reposts=false&amp;show_teaser=true" width="100%"></iframe>

 

 

Reading academic articles about Internet-wide routing challenges you might stumble upon valley-free routing – a pretty important concept with applications in WAN and data center routing design.

If you’re interested in the academic discussions, you’ll find a pretty exhaustive list of papers on this topic in the Informative References section of RFC 7908; here’s the over-simplified version.

Read more ...

Following “if you can’t explain it, you don’t understand it” mantra I decided to use blog posts to organize my ideas while preparing my Networking Infrastructure as Code presentation for the Autumn 2018 Building Network Automation Solutions online course. Constructive feedback is highly appreciated.

Let’s start with a simple terminology question: what exactly is Infrastructure as Code that everyone is raving about? Here’s what Wikipedia has to say on the topic:

Read more ...

Remember the Token Ring!!

Evil CCIE left a lengthy comment on one of my blog posts including this interesting observation:

It's always interesting to hear all kind of reasons from people to deploy CLOS fabrics in DC in Enterprise segment typically that I deal with while they mostly don't have clue about why they should be doing it in first place. […] Usually a good justification is DC to support high amount of East-West Traffic....but really? […] Ask them if they even have any benchmarks or tools to measure that in first place :)

What he wrote proves that most networking practitioners never move beyond regurgitating vendor marketing (because that’s so much easier than making the first step toward becoming an engineer by figuring out how technology really works).

Read more ...

How do you see your network? Odds are good it looks like a big collection of devices and protocols that you use to connect everything. It doesn’t matter what those devices are. They’re just another source of packets that you have to deal with. Sometimes those devices are more needy than others. Maybe it’s a phone server that needs QoS. Or a storage device that needs a dedicated transport to guarantee that nothing is lost.

But what does the network look like to those developers?

Work Is A Highway

When is the last time you thought about how the network looks to people? Here’s a thought exercise for you:

Think about a highway. Think about all the engineering that goes into building a highway. How many companies are involved in building it. How many resources are required. Now, think of that every time you want to go to the store.

It’s a bit overwhelming. There are dozens, if not hundreds, of companies that are dedicated to building highways and other surface streets. Perhaps they are architects or construction crews or even just maintenance workers. But all of them have a function. All for the sake of letting us drive on roads to get places. To us, the road isn’t the primary thing. It’s just a way to go somewhere that we want to be. In fact, the only time we really notice the road is when it is in disrepair or under construction. We only see the road when it impacts our ability to do the things it enables.

Now, think about the network. Networking professionals spend their entire careers building bigger, faster networks. We take weeks to decide how best to handle routing decisions or agonize over which routing protocols are necessary to make things work the way we want. We make it our mission to build something that will stand the test of time and be the Eighth Wonder of the World. At least until it’s time to refresh it again for slightly faster hardware.

The difference between these two examples is the way that the creators approach their creation. Highway workers may be proud of their creation but they don’t spend hours each day extolling the virtues of the asphalt they used or the creative way they routed a particular curve. They don’t demand respect from drivers every time someone jumps on the highway to go to the department store.

Networking people have a visibility problem. They’re too close to their creation to have the vision to understand that it’s just another road to developers. Developers spend all their time worrying about memory allocation and UI design. They don’t care if the network actually works at 10GbE or 100 GbE. They want a service that transports packets back and forth to their destination.

The Old New Network

We’ve had discussion in the last few years about everything under the sun that is designed to make networking easier. VXLAN, NFV, Service Mesh, Analytics, ZTP, and on and on. But these things don’t make networking easier for users. They make networking easier for networking professionals. All of these constructs are really just designed to help us do our jobs a little faster and make things work just a little bit better than they did before.

Imagine all the work that goes into electrical power generation. Imagine the level of automation and monitoring necessary to make sure that power gets from the generation point to your house. It’s impressive. And yet, you don’t know anything about it. It’s all hidden away somewhere unimpressive. You don’t need to describe the operation of a transformer to be able to plug in a toaster. And no matter how much that technology changes it doesn’t impact your life until the power goes out.

Networking needs to be a utility. It needs to move away from the old methods of worrying about how we create VLANs and routing protocols and instead needs to focus on disappearing just like the power grid. We should be proud of what we build. But we shouldn’t make our pride the focus of hubris about what we do. Networking professionals are like highway workers or electrical company employees. We work hard behind the scenes to provide transport for services. The cloud has changed the way we look at the destination for those services. And it’s high time we examine our role in things as well.

---

Tom’s Take

Telco workers. COBOL programmers. Data entry specialists. All of these people used to be the kings and queens of their field. They were the people with the respect of hundreds. They were the gatekeepers because their technology and job roles were critical. Until they weren’t any more. Networking is getting there quickly. We’ve been so focused on making our job easy to do that we’ve missed the point. We need to be invisible. Just like a well built road or a functioning electrical grid. We are not the goal of infrastructure. We’re just a part of it. And the sooner we realize that and get out of our own way, we’re going to find that the world is a much better place for everyone involved in IT, from developers to users.

In recent Software Gone Wild episodes we explored emerging routing protocols trying to address the specific needs of highly-meshed data center fabrics – RIFT and OpenFabric. In Episode 92 with Dinesh Dutt we decided to revisit the basics trying to answer a seemingly simple question: do we really need new routing protocols?

Read more ...

One of my subscribers wondered whether it would make sense to build a traditional leaf-and-spine fabric or go for Cisco ACI. He started his email with:

One option is a "standalone" Spine/Leaf VXLAN-with EVPN deployment based on Nexus equipment. This approach could probably be accompanied by some kind of automation like Ansible to ease operation/maintenance of the network.

This is what I would do these days if the customer feels comfortable investing at least the minimum amount of work into an automation solution. Having simpler technology + well-understood automation solution is (in my biased opinion) better than having a complex black box.

Read more ...

The summer break is over, and we’ve already scheduled a half-dozen events and webinars in August and September:

• We’ll start with Introduction to Software-Defined Data Centers webinar on August 28^th.
• There are still a few places left in the Lean Start in Network Automation workshop in Zurich, Switzerland, on August 30^th. Register here.

We’ll run an event or webinar in almost every single week in September:

Read more ...

As promised, here’s the second part of my Benefits of Network Automation interview with Christoph Jaggi published in German on Inside-IT last Friday (part 1 is here).

What are some of the challenges?

The biggest challenge everyone faces when starting the network automation is the snowflake nature of most enterprise networks and the million one-off exceptions we had to make in the past to cope with badly-designed applications or unrealistic user requirements. Remember: you cannot automate what you cannot describe in enough details.

Read more ...

It’s no surprise that Twitter’s developers are messing around with the platform. Again. This time, it’s the implementation of changes announced back in May. Twitter is finally cutting off access to their API that third party clients have been using for the past few years. They’re forcing these clients to use their new API structure for things like notifications and removing support for streaming. This new API structure also has a hefty price tag. For 250 users it’s almost $3,000/month.

You can imagine the feedback that Twitter has gotten. Users of popular programs like Tweetbot and Twitterific were forced to degrade client functionality thanks to the implementation of these changes. Twitter power users have been voicing their opinions with the hashtag #BreakingMyTwitter. I’m among the people that are frustrated that Twitter is chasing the dollar instead of the users.

Breaking The Bank

Twitter is beholden to a harsh mistress. Wall Street doesn’t care about user interface or API accessibility. They care about money. They care are results and profit. And if you aren’t turning a profit you’re a loser that people will abandon. So Twitter has to make money somehow. And how is Twitter supposed to make money in today’s climate?

Users.

Users are the coin of Twitter’s realm. The more users they have active the more eyeballs they can get on their ads and sponsored tweets and hashtags. Twitter wants to court celebrities with huge followings that want to put sponsored tweets in their faces. Sadly for Twitter, those celebrities are moving to platforms like Instagram as Twitter becomes overrun with bots and loses the ability to have discourse about topics.

Twitter needs real users looking at ads and sponsored things to get customers to pay for them. They need to get people to the Twitter website where these things can be displayed. And that means choking off third party clients. But it’s not just a war on Tweetbot and Twitterific. They’ve already killed off their Mac client. They have left Tweetdeck in a state that’s barely usable, positioning it for power users. Yet, power users prefer other clients.

How can Twitter live in a world where no one wants to use their tools but can’t use the tools they like because access to the vital APIs that run them are choked off behind a paywall that no one wants to pay for? How can us poor users continue to use a service that sucks when used through the preferred web portal?

You probably heard my rant on the Gestalt IT Rundown this week. If not, here you go:

<iframe allowfullscreen="true" class="youtube-player" height="329" src="https://www.youtube.com/embed/qAkYHBdN8-8?version=3&amp;rel=1&amp;fs=1&amp;autohide=2&amp;showsearch=0&amp;showinfo=1&amp;iv_load_policy=1&amp;start=795&amp;wmode=transparent" style="border:0;" type="text/html" width="584"></iframe>

I was a little animated because I’m tired of getting screwed by developers that don’t use Twitter the way that I use it. I came up with a huge list of things I didn’t like. But I wanted to take a moment to talk about some things that I think Twitter should do to get their power users back on their side.

1. Restore API Access to Third Party Clients – This is a no-brainer for Twitter. If you don’t want to maintain the old code, then give API access to these third party developers at the rates they used to have it. Don’t force the developers working hard to make your service usable to foot the bills that you think you should be getting. If you want people to continue to develop good features that you’ll “borrow” later, you need to give them access to your client.
2. Enforce Ads on Third Party Clients – I hate this idea, but if it’s what it takes to restore functionality, so be it. Give API access to Tweetbot and Twitterific, but in order to qualify for a reduced rate they have to start displaying ads and promoted tweets from Twitter. It’s going to clog our timeline but it would also finance a usable client. Sometimes we have to put up with the noise to keep the signal.
3. Let Users Customize Their Experience – If you’re going to drive me to the website, let me choose how I view my tweets. I don’t want to see what my followers liked on someone else’s timeline. I don’t want to see interesting tweets from people I don’t follow. I want to get a simple timeline with conversations that don’t expand until I click on them. I want to be able to scroll the way I want, not the way you want me to use your platform. Customizability is why power users use tools like third party clients. If you want to win those users back, you need to investigate letting power users use the web platform in the same way.
4. Buy A Third Party Client and Don’t Kill Them Off – This one’s kind of hard for Twitter. Tweetie. The original Tweetdeck. There’s a graveyard of clients that Twitter has bought and caused to fail through inattention and inability to capitalize on their usability. I’m sure Loren Britcher is happy to know that his most popular app is now sitting on a scrap heap somewhere. Twitter needs to pick up a third party developer, let them develop their client in peace without interference internally at Twitter, and then not get fired for producing.
5. Listen To Your Users, Not Your Investors – Let’s be honest. If you don’t have users on Twitter, you don’t have investors. Rather than chasing the dollars every quarter and trying to keep Wall Street happy, you should instead listen to the people that use your platform and implement the changes their asking for. Some are simple, like group DMs in third party clients. Or polls that are visible. Some are harder, like robust reporting mechanisms or the ability to remove accounts that are causing issues. But if Twitter keeps ignoring their user base in favor of their flighty investors they’re going to be sitting on a pile of nothing very soon.

---

Tom’s Take

I use Twitter all the time. It’s my job. It’s my hobby. It’s a place where I can talk to smart people and learn things. But it’s not easy to do that when the company that builds the platform tries as hard as possible to make it difficult for me to use it the way that I want. Instead of trying to shut down things I actively use and am happy with, perhaps Twitter can do some soul searching and find a way to appeal to the people that use the platform all the time. That’s the only way to fix this mess before you’re in the same boat as Orkut and MySpace.

This post acts as the introduction for two other posts, which cover Junos data collection tools for Kafka and InfluxDB. The code is open-sourced and licensed under MIT. Both applications are ready for release and I’ve spent considerable spare time building and testing both pieces of software.

To Go or not to Go

Back in yesteryear, I used to be a C developer and enthusiast. Thanks to the infamous K&R C book, it made perfect sense when I needed a language that provided syntax what I thought of at the time as one level higher than assembly.
Roll the clock forwards a decade and Go has become my ‘go to’ (multi-pun intended) language. It’s powerful from its simplicity, easy to debug and super easy to observe when things aren’t going as you planned. The concurrency capabilities seem to make perfect sense and development cycles are short thanks to the powerful “batteries included” tool-chain. Building binaries couldn’t be easier and building containers for the likes of Docker is a piece of cake. Thanks also to the language’s popularity, tools like Travis-CI are easy to work with. Powerful enough to do almost anything, easy enough to learn in days and offers grace when handling concurrent logic flows.

For example, Junos runs FreeBSD and building binaries to run on Junos is as easy as one can imagine. Whether you develop on OSX, Linux or Windows, the process is identical:

GOOS=freebsd go build

. Aint that simple?

Just one note here, on veriexec enabled Junos it isn’t enough to just build the binaries; it needs packaging too!

Inspiration

Working for vendors (at the time of writing, Juniper networks) I get a fair amount of insight to the real problems network operators (SP/enterprise/telcos/data centres) face daily simultaneously whilst trying to evolve their strategy. Thanks to the current automation craze, any set of data applied to the network mutating its state, comes from a decision to make the change (the when), the desired graph end state (the what) and input mechanisms that trigger mutation (the how).
This post acts as an umbrella for two other posts based on the “when” and how to correlate what when is.
The problem with “when” is that of multi-dimensions. Time moves on whilst you’re in the past and looking forwards. A different way of looking at this would be, correlation tasks over time, become a squeeze-box sliding window of observation. Real-time distributed systems apply back pressure on to correlation systems as state-machines hold in various positions waiting for signals that trigger transitional shifts to the end-state. The only reasonable means to go forwards is make the data pipelines as fluid as possible. Kafka and InfluxDB are perfect weapons for this challenge and treat correlation as one or more components that do not lock up these data systems.

From a “signal-to-event” correlation process, I’ve began to think correlation is best served from a micro-services architecture, with each correlation being it’s own application. Currently the trend is to use a platform that holds a lot of state whilst correlating. That means if it crashes, all of that is lost too. I’ll write something about that soon! For now and the purpose of this post, the focus is squarely on extracting the data from Junos.

Exploring the Process

Kafka and InfluxDB have their own requirements for the insertion of data.

With respect to data, Kafka requires that the producer writes data to a topic. InfluxDB requires the producer knows the database name, tag and field sets.

I’ve seen frustration around InfluxDB when there isn’t enough tag or field information to discern data sources, leading to data overwrites when the time stamp is the same. You do not have this issue with Kafka due to Kafka working with offsets within a topic within a partition within a cluster (unless you forcibly overwrite the offset!).

When it came to the design of the collector, I followed the IP Engineer learning charter and discovered most of the work was done for me. Here’s the UML diagram of what I needed to do.

// Here is the YUML that created the diagram above
// Here is the YUML that created the diagram above
// {type: activity}
(start) -> (Start Kafka & ZooKeeper) -> (Kafka & ZooKeeper)
(start) -> (Start InfluxDB) -> (InfluxDB)
(start) -> (Start Chronograf)
(note: For admin of InfluxDB and checking) -> (Start Chronograf)
(start) -> (Collect specific Junos metrics over NETCONF) -> (Transform data to Go variables) -> (Go variables)
(Go variables) -> (Extract name for Kafka topic) -> (Insert data to topic) -> (Kafka & ZooKeeper)
(Go variables) -> (Extract data for InfluxDB tag and field sets) -> (Insert data with timestamp) -> (InfluxDB)

After a short hunt around using my Google powers, I found Daniel Czerwonk’s Junos exporter for Prometheus. This appeared to do all of the hard work of data acquisition and all I needed to do was replace the Prometheus code with Kafka and InfluxDB client function calls and do some handling for clean exit.

What I did next was:

1. Used my fork of go-netconf to acquire the data from Junos.
2. Added the Influx & Kafka clients to Daniel’s code and removed traces of Prometheus.
3. Wrapped the code with channels for starting and stopping routines and sharing data between Go routines
4. Added command line arguments to each application and the ability to use SSH keys as well as plain text user/password credentials

Both applications were relatively easy to put together. I had a concern of memory leaks for a short while, but after using ‘gops’ for a few days and monitoring usage of memory with InfluxDB and Chronograf, I put those worries out the back of my brain. Any application that launches multiple Go routines and is truly asynchronous, will behave in a way that feels organic. As tasks are dealt with by routines and as context switching happens, it can feel very fluid with waves of growth and shrinkage over time.

For instructions on using both the Junos Kafka and InfluxDB applications, use the links below. Each application is command line argument driven and is easy to build and install. Each application also comes with Dockerfiles for ease of use.

Exporter idea

Modern operational processes demand that metrics like device, interface, process and service be obtained from the network nodes for a variety of uses like machine-learning based prediction and event-driven automation. Junos offers a solid NETCONF interface that can be used to obtain operational and configuration information as well as newer gRPC based telemetry subsystems. On Junos systems that do not offer a telemetry subsystem, data can be reliably obtained from NETCONF, transformed and published in a required format to time series databases and streaming platforms. Junos as of yet cannot do this today without some off-box help. This is where applications like the below come in to play.

Applications listed in the following sections can be run in Docker containers or as foreground applications, connect to the NETCONF service on a Junos instance via username/password credential pairs or SSH keys and connect to the target data producer APIs which at the time of writing are InfluxDB and Kafka.

One might be wondering about the use of Go here and the answer is simple and comes in the form of bullet points:

• Go is simple
• Go does not have version 2.x or 3.x differences that will break things
• Go compiles in to a target OS specific build, exceptionally easily
• Go empowers the user to create custom collectors and compile them in to the binary
• Go offers powerful concurrency right out of the box without thinking about dodgy parallelism libraries

Collectors are very easy to create and included in the build. Take a look inside the

collectors

directories of each project to figure out how to create and customize your own. A number are included covering the major topics.

jinfluxexporter

InfluxDB & Junos link

Note: Requires that InfluxDB has been built with uint64 support

jkafkaexporter

Kafka & Junos link

Close

I’ve spent considerable time working on both applications and will support them best effort. Pull requests are welcomed for bug fixes, feature additions and general help!

Disclaimer

Please note, each of these applications have been published by me directly and are by no means supported by Juniper Networks. They will be supported best effort through GitHub. You use them squarely at your own risk.

The post Junos Kafka & InfluxDB Exporters appeared first on ipengineer.net.

There is a saying, attributed to Abraham Maslow, that when all you have is a hammer then everything looks like a nail. A variation is that when all you have is a hammer, then all you can do it hit things! For a legislative body, when all you can do is enact legislation, then that’s all you do! Even when it’s pretty clear that the underlying issues do not appear to be all that amenable to legislative measures, some legislatures will boldly step forward into the uncertain morass and legislate where wiser heads may have taken a more cautious and considered stance.

In this article I'd like to look at the roles of Security Extensions for the DNS (DNSSEC) and DNS over Transport Layer Security (DoT) and question DoT could conceivably replace DNSSEC in the DNS.

You should definitely watch this amazing video from Ben Sigelman of LightStep that was recorded at Cloud Field Day 4. The good stuff comes right up front.

In less than five minutes, he takes apart crazy notions that we have in the world today. I like the observation that you can’t build a system more than three or four orders of magnitude. Yes, you really shouldn’t be using Hadoop for simple things. And Machine Learning is not a magic wand that fixes every problem.

However, my favorite thing was the quick mention of how emulating Google for the sake of using their tools for every solution is folly. Ben should know, because he is an ex-Googler. I think I can sum up this entire discussion in less than a minute of his talk here:

Google’s solutions were built for scale that basically doesn’t exist outside of a maybe a handful of companies with a trillion dollar valuation. It’s foolish to assume that their solutions are better. They’re just more scalable. But they are actually very feature-poor. There’s a tradeoff there. We should not be imitating what Google did without thinking about why they did it. Sometimes the “whys” will apply to us, sometimes they won’t.

Gee, where have I heard something like this before? Oh yeah. How about this post. Or maybe this one on OCP. If I had a microphone I would have handed it to Ben so he could drop it.

Building a Laser Moustrap

We’ve reached the point in networking and other IT disciplines where we have built cargo cults around Facebook and Google. We practically worship every tool they release into the wild and try to emulate that style in our own networks. And it’s not just the tools we use, either. We also keep trying to emulate the service provider style of Facebook and Google where they treated their primary users and consumers of services like your ISP treats you. That architectural style is being lauded by so many analysts and forward-thinking firms that you’re probably sick of hearing about it.

Guess what? You are not Google. Or Facebook. Or LinkedIn. You are not solving massive problems at the scale that they are solving them. Your 50-person office does not need Cassandra or Hadoop or TensorFlow. Why?

• Google Has Massive Scale – Ben mentioned it in the video above. The published scale of Google is massive, and even it’s on the low side of the number. The real numbers could even be an order of magnitude higher than what we realize. When you have to start quoting throughput numbers in “Library of Congress” numbers to make sense to normal people, you’re in a class by yourself.
• Google Builds Solutions For Their Problems – It’s all well and good that Google has built a ton of tools to solve their issues. It’s even nice of them to have shared those tools with the community through open source. But realistically speaking, when are you really going to use Cassandra to solve all but the most complicated and complex database issues? It’s like a guy that goes out to buy a pneumatic impact wrench to fix the training wheels on his daughter’s bike. Sure, it will get the job done. But it’s going to be way overpowered and cause more problems than it solves.
• Google’s Tools Don’t Solve Your Problems – This is the crux of Ben’s argument above. Google’s tools aren’t designed to solve a small flow issue in an SME network. They’re designed to keep the lights on in an organization that maps the world and provides video content to billions of people. Google tools are purpose built. And they aren’t flexible outside that purpose. They are built to be scalable, not flexible.

Down To Earth

Since Google’s scale numbers are hard to comprehend, let’s look at a better example from days gone by. I’m talking about the Cisco Aironet-to-LWAPP Upgrade Tool:

I used this a lot back in the day to upgrade autonomous APs to LWAPP controller-based APs. It was a very simple tool. It did exactly what it said in the title. And it didn’t do much more than that. You fed it an image and pointed it at an AP and it did the rest. There was some magic on the backend of removing and installing certificates and other necessary things to pave the way for the upgrade, but it was essentially a batch TFTP server.

It was simple. It didn’t check that you had the right image for the AP. It didn’t throw out good error codes when you blew something up. It only ran on a maximum of 5 APs at a time. And you had to close the tool every three or four uses because it had a memory leak! But, it was a still a better choice than trying to upgrade those APs by hand through the CLI.

This tool is over ten years old at this point and is still available for download on Cisco’s site. Why? Because you may still need it. It doesn’t scale to 1,000 APs. It doesn’t give you any other functionality other than upgrading 5 Aironet APs at a time to LWAPP (or CAPWAP) images. That’s it. That’s the purpose of the tool. And it’s still useful.

Tools like this aren’t built to be the ultimate solution to every problem. They don’t try to pack in every possible feature to be a “single pane of glass” problem solver. Instead, they focus on one problem and solve it better than anything else. Now, imagine that tool running at a scale your mind can’t comprehend. And you’ll know now why Google builds their tools the way they do.

---

Tom’s Take

I have a constant discussion on Twitter about the phrase “begs the question”. Begging the question is a logical fallacy. Almost every time the speaker really means “raises the question”. Likewise, every time you think you need to use a Google tool to solve a problem, you’re almost always wrong. You’re not operating at the scale necessary to need that solution. Instead, the majority of people looking to implement Google solutions in their networks are like people that put chrome everything on a car. They’re looking to show off instead of get things done. It’s time to retire the Google Cargo Cult and instead ask ourselves what problems we’re really trying to solve, as Ben Sigelman mentions above. I think we’ll end up much happier in the long run and find our work lives much less complicated.