Another Cisco Live has come and gone. Overall it was a fun time for many. Catching up with friends. Meeting people for the first time. Enjoying the balmy Orlando weather. It was a chance to relive some great times for every one. But does Cisco Live 2018 dictate how the future of the event will go?

Packing The Schedule

Did you get a chance to attend any of the social events at Cisco Live? There were a ton. There were Tweetups and meet ups and special sessions galore. There was every opportunity to visit a lounge or area dedicated to social media presence, Boomerang videos, goofy pictures, or global outreach. Every twenty feet had something for you to do or some way for you to make an impact.

In fact, if you went to all of these things you probably didn’t have time for much else. Definitely not time for the four or five keynote addresses. Or a certification test. Or the classes and sessions. In fact, if you tried to do everything there was to do at Cisco Live, you’d probably not sleep the whole week. There’s almost as much stuff to do outside the conference sessions as there is to do in them.

But is it too much? Are the activities around the learning sessions taking away from the conference itself? Think about something like the Big Ideas theater this year. In theory, it’s a great way to get people to attend sessions that are not specifically related to tech. You can introduce new ideas, especially those that are focused more on changing the world. But you’re also competing for time away from sessions that are focused on new products or building better architectures.

Every booth in the World of Solutions is designed to draw you in and keep you there. For the sponsors of the event it’s important to have conversations about their products and solutions. For Cisco people, it’s almost like they’re competing with the sessions to give you different content or a chance to interview people. Is that how things should be? I can understand the desire of DevNet wanting to change the way people look at programmable networking, for example. But every other little booth like Cisco Advanced Services or the Emergency Response Vehicle? Those feel more like attractions designed to show off rather than educate.

Paying the Piper

And what does all this cost in the long run? Sure, I love having extra features around the conference as much as the next person. But to what end? Things don’t pay for themselves. Every conference has a budget. Every piece of entertainment and every showcase booth costs money in some way or another. And how does that all get paid for? By us, the attendees.

It’s no secret that attending conferences isn’t cheap. A full conference pass for Cisco Live is around $2,000. In the past, there were cheaper options for just attending for the people networking aspect of things. But, with the growth of DevNet and other “included” options at the conference, Cisco needed to find a way to pay for them this year.

I’m not going to spend a lot of time going into the Imagine Pass issue right now because I want to sit down and have an honest discussion with Cisco about the pros and cons of the approach. But it is very important that we examine what we’re getting for the increased cost. There has to be a significant value for people to want to be a part of the event if the costs go up. The way to do that is to create compelling reasons to want to be at Cisco Live.

The way not to do that is to lock the content behind gates. Some of the things at Cisco Live this year were placed in areas that were not easy to access. One of my personal pet peeves is the NetVet lounge. I’m going to start this off by saying that I was a NetVet for many years before I moved to Tech Field Day. I’m no longer a NetVet. However, until 2013 the NetVet lounge was one of the de facto social hangout places. Now, it’s another area where you can get coffee and snacks.

Why does the NetVet lounge bother me? Because of the placement. Front and center across the aisle from the on-site Cisco Store (which took the place of the Social Media hub from 2013). Why does the NetVet lounge get to be outside the World of Solutions? Aside from the historical reasons, I can’t think of a good reason. You need to have a full conference pass to achieve NetVet status. A full conference pass gets you into the World of Solutions. Why not have NetVets meet there?

The obvious reason is that the World of Solutions closes. Yet the NetVet lounge does too. And the hours are pretty similar. Why not move the NetVet lounge into the World of Solutions and give that space to the Social Media folks. There are no restrictions on getting into the Social Media Hub. Why not have them front and center? Again, aside from the “tradition” of having the NetVet lounge outside the World of Solutions I can’t think of a good reason.

---

Tom’s Take

I love Cisco Live. I realized this year that I’ve been to thirteen of them. Every year since 2006. The conference has changed and grown. The focus has shifted. But the people remain the same. With the changes in the way that the pass structure the people may not be there much longer. We, as IT professionals, need to decide what’s important and give some feedback. We need to make it constructive and honest. Point out what works and what doesn’t. Don’t whine, but offer direct criticism. We can only make the conference we want by telling the people what we need. That’s how you make Cisco Live a place to be for now and for the future.

So many things have happened since I wrote “this is what we’re going to do in 2018” blog post. We ran

• An automation course and a data center course, resulting in three new modules in the automation course, and over 20 hours of fresh content
• Three workshops, two of them brand new and resulting in material for VMware NSX Deep Dive and AWS Networking webinars (both coming in early autumn).

We also did a ton of webinars:

Read more ...

The evolutionary path of any technology can often take strange and unanticipated turns and twists. At some points simplicity and minimalism can be replaced by complexity and ornamentation, while at other times a dramatic cut-through exposes the core concepts of the technology and removes layers of superfluous additions. The evolution of the Internet appears to be no exception and contains these same forms of unanticipated turns and twists. In thinking about the technology of the Internet over the last ten years, it appears that it’s been a very mixed story about what’s changed and what’s stayed the same.

The last days I was working on adapting INE‘s lab topology, most specific the CCIE Routing and Switching v5 one, to be used in EVE-NG. In my opinion, INE offers some of the best training materials for Cisco and Juniper certifications. Along certification training you can find in their All Access Pass Subscription valuable learning … Continue reading INE’s CCIE R&S v5 topology for EVE-NG

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]

On Tuesday I had the last webinar in spring 2018. One more online course session and it will be time for long summer break. In the meantime, we’re already planning the autumn events:

• We’ll start with Lean Start in Network Automation workshop in Zurich, Switzerland, on August 30^th. Register here.
• Building Network Automation Solutions online course starts on September 20^th.

We also have the first webinars scheduled:

• We’ll start with two introductory webinars: SDDC 101 on August 28^th and SD-WAN Overview on September 4^th.
• The first session of technical deep dive into VMware NSX will be on September 11^th.
• Amazon Web Service Networking workshop was a huge success, and we’ll turn it into a webinar in early October.
• There will be an interesting math-focused webinar on October 8^th;
• Dinesh Dutt will talk about fabric troubleshooting in mid-November;

You can attend all these webinars with an ipSpace.net webinar subscription.

 

Co-author:  Manoj Ahuje

 

 

 

Windows has been a target of hackers for a long time. The reason is simple: it is the OS with the largest deployment footprint around the world. Hackers discover vulnerabilities in Windows OS software as well as the various software that support Windows OS. Exploits as well as exploit mitigation techniques have evolved over the years. Over time, the targeted exploit has shifted from server applications to desktop applications. Adobe Flash has been a favorite target for the past seven to eight years. In this article we will talk about the exploits and their mitigation techniques on Windows.

 

Exploits are meant to compromise the system by taking advantage of a vulnerability in the program. Before moving ahead, let us first discuss how hackers find a vulnerability in an application.

 

Common vulnerability discovery techniques:

Fuzzing: This is a common way to test for vulnerabilities. The hackers provide a large range of inputs for the target software and observe for any unexpected behavior. For browsers, a lot of HTML files are created as an input. If the vulnerability has not been seen earlier, it is called a zero day vulnerability.

 

Patch diffing: This is a technique by which an update in software is compared with an old version. Windows used to update its software by means of DLLs. Hackers can compare the old and new DLL to find out what code is patched. After this, they can analyze it to find out if the old code is vulnerable. Hackers write the exploit to take advantage of the vulnerability assuming that not everybody would have applied the new software patch immediately. These kind of exploits are called one day exploits.

 

Leaks:

There are certain organizations that discover zero day vulnerabilities for commercial software testing. Hacking Team was one of these. Hacking Team exploits were leaked in 2015. Eternal Blue is also one of such infamous exploits that contributed to the spread of WannaCry and Petya malware

 

 

Types of Vulnerabilities and Windows countermeasures

Buffer Overflow vulnerabilities: These kinds of vulnerabilities occur when a program doesn’t check the boundary of user-supplied data. Stack overflow and heap overflow are the most common types of buffer overflow. Windows has patched almost all known and exploitable buffer overflow vulnerabilities and it is rare to see a new one discovered these days.

Other than patching the vulnerabilities in the software, Windows also developed some techniques to mitigate buffer overflows.

 

A Stack is a a very important data structure used in programming. It is used to store local variables and the function return address. Local variables can store user supplied data. As an example, HTTP server can store the HTTP request sent by the user in a variable. If the server does not check the size of the supplied HTTP request, it gets written to the stack where it can corrupt other data like the return address. The return address can be overwritten to redirect the control flow to a shellcode which is part of the user supplied variable value. A shellcode can pop a backdoor, launch a program or download another malware. Stack overflow exploitation works by overwriting the return address on the stack by a user supplied local variable.

 

Windows came up a with stack canaries to detect overwriting of the return address. The stack canaries feature was implemented in Windows XP SP2.  An exception was triggered when the return address was overwritten. Unfortunately, hackers bypassed this too using a technique called SEH Overwrite (Structured Exception Handler). The address of the exceptional handler needed to handle the return address overwrite exception was stored on the stack. Hence, the attacker was able to take control by overwriting this value.

 

Heap overflow also worked in a similar manner. User supplied data was able to corrupt important data structures on the heap to take control of the program. To track heap allocation, linked lists are used. The pointers used to connect node in the linked list are corrupted and overwritten by user controlled code.

 

Windows invented a technique called DEP (Data Execution Prevention) to prevent this kind of exploitation. The technique imposed a restriction that anything on the stack or heap should be treated as data and should not be executed.

 

                                                                                         DEP settings on windows

 

 

 

 

In order to bypass DEP, attackers created the shellcode by using codes in DLLs, which are supposed to execute. The shellcode is composed of addresses of instructions inside a DLL instead of only instructions. This technique is called ROP chain (return oriented programming). The ROP technique executes the code chunks that span across multiple DLLs to carry out the same functionality, which could have also been achieved by executing a contiguous chunk of shellcode placed in a heap or stack in absence of DEP. The code chunks from the DLLs used in ROP are called ROP gadgets. It was possible to create a ROP chain by using addresses inside a loaded DLL as the DLL addresses were fixed. To mitigate this ROP, Windows introduced another technique called ASLR (Address Space Layout Randomization). This technique randomized the base addresses of DLL with respect to the main executable each time the program started. This technique was quite effective with minimal flaws. The attacker used the non-randomized DLLs to create exploits. Attackers sometimes used another technique called information disclosure or information leak to find out the address of a DLL and then use the address in the exploit.

 

Now, stack and heap overflow are rarely seen in popular applications. It seems that most companies have done a good job on educating engineers about basic vulnerabilities. Use-After-Free has been the most recent popular vulnerability. It’s one of the most exploited vulnerabilities in the past five to six years.It was found in all browsers, Adobe Reader and Adobe Flash applications. Use-After-Free is a memory corruption vulnerability. The vulnerability is triggered when the program tries to access an object that has been freed. The object has been freed, but there was still a pointer that pointed to the memory location of the freed object. Trying to access the data pointed to the leads of the vulnerability. This kind of pointer is called a dangling pointer. Attackers can misuse the pointer to execute shellcode. Microsoft introduced a technique called isolated heap in mid-2014 to minimize use after free vulnerabilities. Isolated heap allocated separate heap for critical objects. The heap blocks are freed from user controlled data after the object is freed. Isolated heap helped to prevent exploitation of use-after-free vulnerabilities, but isolated heap was applied only to selective objects, not all objects. So, some objects might still be subject to risk. To further elevate the security, Microsoft added Protected Free or Deferred Free. In this technique, Microsoft does not free the object immediately. Instead, it frees it sometime later so that the attacker cannot predict the time when they can control the freed object.

 

Another very popular technique used to exploit browsers is heap spray. JavaScript used in browsers stores variables in the heap. The heap spray technique was used to fill the heap with lots of shellcode chunks. The advantage of this technique is that the attacker does not need to accurately predict the address of shellcode on heap. An address like 0xaaaaaaaa ,0xbbbbbbbbb is used, which most likely points to the heap and the shellcode would then probably lie around that. Chromium sandbox was one of Google Chrome’s innovations to counter all kinds of browser exploitation. Internet Explorer also came up with a similar solution.

 

But, still the DEP and ASLR was an issue. Hence, the novel technique JIT spray (Just In Time) was used. The technique was mostly used to exploit Flash-related vulnerabilities. JIT Engine is a native code generator, which is used by all modern browsers to speed up execution by parsing, optimizing and compiling bytecode to native code for the machine to run. The emitted code by JIT Engine is marked as executable in memory by default. This code can be sprayed into the heap by calculating the right size allocation of a page. After that, it’s a matter of jumping to a known heap address to get code execution bypassing DEP and ASLR all together. This bytecode can be generated in real time by JIT or it can be pre-generated and sent on the wire as a Flash/Java file on the internet. This technique was first used with Flash ActionScripts to spray heap, which used long XOR sequences to store shellcode and jumped onto a known heap address to get reliable code execution. After researchers were able to use the same JIT spray technique to exploit native JAVA and also used ASM.js recently for exploitation.

 

To keep up with the continuously evolving exploit landscape, Microsoft came up with EMET (Enhanced Mitigation Experience Toolkit) in 2009. It is an added layer of defense against code reuse attacks like ROP and provides better protection against Heap and JIT spray. Though it has to be manually installed by an administrator. Since EMET was not designed as an integral part of the OS, the exploit writers were able to bypass/disable EMET and achieve code execution in each of its versions. Though EMET led many security innovation in Windows product lines 7, 8 ,8.1, 10 and it’s Linux counterparts. As underlying OS changed, Microsoft decided to build in this security in OS as Windows Defender Exploit Guard, which will support future Windows versions after Windows 10, with many improvements. Microsoft decided to end life support for EMET. The latest version will be EOL’d on July 31, 2018.

 

CONCLUSION

Exploitation and Anti-Exploitation techniques is almost a cat and rat race. Security professionals always come up with an idea to combat Cyber criminals but at the same time criminals figure out a way to defeat it. Juniper Advanced threat detection product detects exploits to keep the customers safe.

 

 

Stumbled upon “Is Tech News Fake” article by Tom Nolle. Here’s the gist of his pretty verbose text:

When readers pay for news, they get news useful to readers.  When vendors pay, not only do the vendors get news they like, the rest of us get that same story.  It doesn’t mean that the story being told is a lie, but that it reflects the view of an interested party other than the reader.

High-quality content is not cheap, so always ask yourself: who’s paying for the content… and if it’s not you, you may be the product.

Full disclosure: ipSpace.net is funded exclusively with subscriptions and online courses. Some of our guest speakers work for networking vendors, but we always point that out, and never get paid for that.

During last week’s SIGS Technology Conference I had a keynote presentation about the three paths of enterprise IT.

Unfortunately, the event wasn’t recorded, but you can view the presentation here. Contact me if you have any questions, or Irena if you'd like to have a similar keynote for your event.

Few people are using containers so why are all the vendors into it ?

In IT we’ve all heard of Impostor Syndrome by now. The feeling that you’re not just a lucky person that has no real skills or is skating by on the seat of their pants is a very real thing. I’ve felt it an many of my friends and fellow members of the community have felt it too. It’s easy to deal with when you have time to think or work on your own. However, when you take your show on the road it can creep up before you know it.

Conferences are a great place to meet people and learn about new ideas. It’s also a place where your ideas will be challenged and put on display. It’s not to difficult to imagine meeting a person for the first time at a place like Cisco Live or VMworld and not feeling little awe-inspired. After all, this could be a person whose works you’ve read for a long time. It could be a person you look up to or someone you would like to have mentor you.

For those in the position of being thrust into the limelight, it can be extremely difficult to push aside those feelings of Impostor Syndrome or even just a general level of anxiety. When people are coming up to you and thanking you for the content you create or even taking it to further extremes, like bowing for example, it can feel like you’re famous and admired for nothing at all.

What the members of the community have to realize is that these feelings are totally natural. You’re well within your rights to want to shy away from attention or be modest. This is doubly true for those of us that are introverts, which seems to happen in higher numbers in IT.

How can you fight these feelings?

Realize You Are Enough. I know it sounds silly to say it but you have to realize that you are enough. You are the person that does what they can every day to make the world a better place in every way you can. It might be something simple like tweeting about a problem you fixed. It may be as impressive as publishing your own network automation book. But you still have to stop and realize you are enough to accomplish your goals.

For those out there that want to tell their heroes and mentors in the community how awesome they are, remember that you’re also forcing them to look at themselves in a critical light sometimes. Some reassurances like, “I love the way you write” or “Your ability to keep the podcast going smoothly” are huge compliments that people appreciate. Because they represent skills that are honed and practiced.

Be The Best You That You Can Be. This one sounds harder than it might actually be. Now that you’ve admitted that you’re enough, you need to keep being the best person that you can be. Maybe that’s continuing to write great content. Maybe it’s something as simple as taking a hour out of your day to learn a new topic or interact with some new people on social media. It’s important that you take your skill set and use it to make things better overall for everyone.

For those out there that are amazed at the amount of content that someone can produce or the high technical quality of what they’re working on, remember that we’re all the same. We all have the same 24 hours in the day to do what we do. So the application of the time spent studying or learning about something is what separates leaders from the pack.

Build Up Others Slowly. This one is maybe the hardest of all. When you’re talking to people and building them up from nothing, you need to be sure to take your time in bringing them along. You can’t just swamp them with knowledge and minute details about their life that have gleaned from reading blogs or LinkedIn. You need, instead, to bring people along slowly and build them up from nothing into the greatest person that you know.

This works in reverse as well. Don’t walk up to someone and start listing off their requirements like a resume. Instead, give them some time to discuss it with you. Let the person you’re talking to dictate a portion of the conversation. Even though you may feel the need to overwhelm with information to justify the discussion you should let them come to their place when they are ready. That prevents the feeling of being overwhelmed and makes the conversation much, much easier.

---

Tom’s Take

It’s very easy to get lost in the world of feeling inadequate about what others think of you. It goes from adulation and excitement to an overwhelming sense of dread that you’re going to let people down. You have to fix that by realizing that you’re enough and doing the best you can with what you have. If you can say that emphatically about yourself then you are well on the way to ensuring that Conference Imposter Syndrome is something you won’t have to worry about.

One of my readers asked me a question that came up in his business strategy class:

Why did routers and switches end up being vertically integrated (the same person makes the hardware and the software)? Why didn't they go down the same horizontal path as compute (with Intel making chips, OEMs making systems and Microsoft providing the OS)? Why did this resemble the pre-Intel model of IBM, DEC, Sun…?

Simple answer: because nobody was interested in disaggregating them.

Read more ...

Lately I’ve started working more and more with EVE-NG to test various network scenarios, automation and in general to try and learn something everyday. If you’re familiar with EVE-NG, you know where to find various Linux images which you can download and install . Very helpful indeed, however all of them are coming without any … Continue reading Ubuntu image for EVE-NG – Python for network engineers

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]

A while ago I found an interesting analysis of HTTP/2 behavior under adverse network conditions. Not surprisingly:

When there is packet loss on the network, congestion controls at the TCP layer will throttle the HTTP/2 streams that are multiplexed within fewer TCP connections. Additionally, because of TCP retry logic, packet loss affecting a single TCP connection will simultaneously impact several HTTP/2 streams while retries occur. In other words, head-of-line blocking has effectively moved from layer 7 of the network stack down to layer 4.

What exactly did anyone expect? We discovered the same problems running TCP/IP over SSH a long while ago, but then too many people insist on ignoring history and learning from their own experience.

I DO NOT CARE. IT JUST HAS TO WORK. NO REALLY. I JUST DONT CARE ABOUT CABLING.

Statistics says, the more often you do something, the higher the chances of a negative event occurring when you do it.

Applying this revelation, if you fly regularly, the chances increase of a delayed flight, or being involved in an incident or accident. A somewhat macabre reference perhaps.

Let’s take something real which happened to me this week (11th June 2018). Whilst working out of one of Juniper’s regional offices, I returned back to the hotel room to carry on working whilst putting my feet up. Something felt strange in the room but I couldn’t put my finger on the weirdness. After a couple of hours, I realised that all of my belongings were gone from the room. Everything! Thanks to a mix-up with the house keeping system, the maids threw my collection of travel items in to some bags ready for disposal. Thanks to a procedure that the hotel operates, for my items to be thrown to the garbage, a manager is required to sign off on the request. A process saved my belongings and I’m thankful that the managers knew this process and also knew where my stuff was likely to be. Before my items were returned, I had already been equipped with emergency toiletries and was settling down for the evening as a knock on the door came. This is a great frame for an automation story.

In the scope of automation, assume a workflow is composed of solid components (i.e. they do one job well, consume structured inputs and return meaningful error codes and structured outputs). These solid components will communicate with what we can assume is an abstract layer or a network device. If we change the device code, we can assume at some point a failure will occur in one or more of these components thanks to interface mismatches and feature creep. Workflows that have been built with “design thinking” methodologies will test for failure scenarios. These failsafe tests are designed to prevent catastrophic failure derived from badly behaved workflows or faulty underlying components. Using the flight example made earlier, it’s true that only so many catastrophic scenarios can be mitigated against. On the other end of the spectrum, a pilot observing flashing warning lights would not attempt to take off. A meteor striking your aircraft like a well-honed pint swilling dart player hitting a bullseye would be something you cannot mitigate against. Using a term that you will get used to very quickly this year, “aim to increase reliability” in both the outcome of your workflow and the processing of it. It’s an NRE (Network Reliability Engineering) tenet. Even if the workflow burns and crashes in the depths of computing hell, it must part of your design process to alert a human of non-recoverable failure. ChatOps is a great candidate for this.

With my hotel experience, the maids were acting autonomously without much thought. They checked their work sheet and it said what it said. No logic that says “Hey, this person’s stuff is here” seemed to be apparent. You could argue here that housekeeping is a well-oiled autonomous process and the hotel’s overarching processes wrapped reliability gains closely to the simple automations of said housekeeping. The outcome was to prevent guest’s items from being accidentally trashed and it worked. They engineered for reliability.

Successful automation is 90/10 in the favour of composition vs tools. Design in the name of reliability eats tooling for breakfast.

The post Automation for Reliability appeared first on ipengineer.net.

This blog post was initially sent to the subscribers of my SDN and Network Automation mailing list. Subscribe here.

Whenever someone mentions intent-based networking I try to figure out what exactly they’re talking about. Not surprisingly, I get a different answer every single time. Confused by all that, I tried to find a good definition, but all I could find was vendor marketing along the lines of “Intent-based networking captures and translates business intent so that it can be applied across the network,” or industry press articles regurgitating vendor white papers.

Read more ...

Like we do with Slack today.

This is the feedback I got from someone who used ExpertExpress to discuss the evolution of their data center:

The session has greatly simplified what had appeared to be a complex and difficult undertaking for us. Great to get fresh ideas on how we could best approach our requirements and with the existing equipment we have. Very much looking forward to putting into practice what we discussed.

And here’s what Nicola Modena (the expert working with the customer) replied:

As I told you, the problem is usually to map the architectures and solutions that are found in books, whitepapers, and validated designs into customer’s own reality, then to divide the architecture into independent functional layers, and most importantly to always start from requirements and not technology.

A really good summary of what ipSpace.net is all about ;) Thank you, Nicola!

I got this design improvement suggestion after publishing When Is BGP No Better than OSPF blog post:

Putting all the leafs in the same ASN and filtering routes sent down to the leafs (sending just a default) are potential enhancements that make BGP a nice option.

Tony Przygienda quickly wrote a one-line rebuttal: “unless links break ;-)”

Read more ...

Working or Not Working. There is no such as a good network, just a working network.

In 2014, we did a series of podcasts on Snabb Switch (Snabb Switch and OpenStack, Deep Dive), a software-only switch delivering 10-20 Gbps of forwarded bandwidth per x86 core. In the meantime, Snabb community slowly expanded, optimized the switching code, built a number of solutions on top of the packet forwarding core, and even forked a just-in-time Lua compiler to get better performance.

To find out the details, listen to Episode 91 of Software Gone Wild in which Luke Gorrie explained how far the Snabb project has progressed in the last four years.

I had a very interesting conversation today with some friends about predictive wireless surveys. The question was really more of a confirmation: Do you need to draw your walls in the survey plan when deciding where to put your access points? Now, before you all run screaming to the comments to remind me that “YES YOU DO!!!”, there were some other interesting things that were offered that I wanted to expound upon here.

Don’t Trust, Verify

One of the most important parts of the wall question is material. Rather than just assuming that every wall in the building is made from gypsum or from wood, you need to actually go to the site or have someone go and tell you what the building material is made from. Don’t guess about the construction material.

Why? Because not everyone uses the same framing for buildings. Wood beams may be popular in one type of building, but steel reinforcement is used in other kinds. And you don’t want to base your predictive survey on one only to find out it’s the other.

Likewise, you need to make sure that the wall itself is actually made of what you think it is. Find out what kind of sheetrock they used. Make sure it’s not actually something like stucco plastered over chicken wire. Chicken wire as the structure of a plaster wall is a guaranteed Faraday Cage.

Another fun thing to run across is old buildings. One site survey I did for a wireless bid involved making sure that a couple of buildings on the outer campus were covered as well. When I asked about the buildings and when they were made, I found out they had been built in the 1950s and were constructed like bomb shelters. Thick concrete walls everywhere. Reinforcement all throughout. Once I learned this, the number of APs went up and the client had to get an explanation of why all the previous efforts to cover the buildings with antennas hadn’t worked out so well.

X-Ray Vision

Speaking of which, you also need to make sure to verify the structures underneath the walls. Not just the reinforcement. But the services behind the walls. For example, water pipes go everywhere in a building. They tend to be concentrated in certain areas but they can run the entire length of a floor or across many floors in a high rise.

Why are water pipes bad for wireless? Well, it turns out that the resonant frequency of water is the same as 802.11b/g/n – 2.4GHz. It’s how microwaves operate. And water loves to absorb radiation in that spectral range. Which means water pipes love to absorb wireless signals. So you need to know where they are in the building.

Architectural diagrams are a great way to find out these little details. Don’t just assume that walking through a building and staring at a wall is going to give you every bit of info you need. You need to research plans, blueprints, and diagrams about things. You need to understand how these things are laid out in order to know where to locate access points and how to correct predictive surveys when they do something unexpected.

Lastly, don’t forget to take into account the movement and placement of things. We often wish we could get involved in a predictive survey at the beginning of the project. A greenfield building is a great time to figure out the best place to put APs so we don’t have to go crawling over bookcases. However, you shouldn’t discount the chaos that can occur when an office is furnished or when people start moving things around. Things like plants don’t matter as much as when someone moves the kitchen microwave across the room or decides to install a new microphone system in the conference room without telling anyone.

---

Tom’s Take

Wireless engineers usually find out when the take the job that it involves being part radio engineer, part networking engineer, part artist, and part construction general contractor. You need to know a little bit about how buildings are made in order to make the invisible network operate optimally. Sure, traditional networking guys have it easy. They can just avoid running cables by florescent lights or interference sources and be good. But wireless engineers need to know if the very material of the wall is going to cause problems for them.

A quick and dirty post on running the Juniper vQFX on VMWare ESXi.

You might be wondering why ESXi seeing as we’re all cloudy types. ESXi is purely a case of laziness. Each server in my control has ESXi 6.5 installed. This becomes tin management at the most basic level.

Part of my home network has a DMZ which has several public IP addresses and I expose systems and VNFs externally over the internet. More recently thanks to the IP fabric craze, part of what I’m exploring is easy integration and feature enhancement on Juniper vQFX instances. Two choices exist:

• Install vQFX on servers with KVM
• Install on ESXi

I went for the easy ground (because why make it harder than it has to be?) Turns out, it wasn’t as straight forward as it should be, although not difficult. Just a niggle.

Installation Process

Grab yourself the RE and PFE images from the Juniper download site:
https://www.juniper.net/support/downloads/?p=vqfxeval I Grabbed the 18.1 RE and the 17.4 PFE image.

Next, extract the two

.vmdk

files from the

.box

files. You can use the trusty tar tool to extract the files required. Below are two files I downloaded from the Juniper site and renamed for clarity.

Run the

tar -xzf <file>

in each directory. Your outcome will look like below:

├── PFE
│   ├── Vagrantfile
│   ├── box.ovf
│   ├── metadata.json
│   ├── packer-virtualbox-ovf-1520879272-disk001.vmdk
│   └── vqfx-pfe-virtualbox.box
└── RE
    ├── Vagrantfile
    ├── box.ovf
    ├── metadata.json
    ├── packer-virtualbox-ovf-1524541301-disk001.vmdk
    └── vqfx-re-virtualbox.box

One requires step here is to run the

.vmdk

file through the

vmkfstools

tool on an EXI machine. Here’s the command I used for that. This tool converts the disk back to being thin provisioned.

vmkfstools -i <input_filename> <output_filename> -d thin

If you do not do this step, the VMs still boot, however I also saw some write errors to disk. I can’t comment any further beyond seeing some error messages due to time investigation time constraints.

Next, to try and figure out what each VM needs and what operating system it runs on, take a sneaky peak inside each of the

box.ovf

files. By looking inside the RE OVF file we know it’s a FreeBSD Operating system with an IDE disk controller. Ok. We can also see what see the requirements for vCPU and RAM.

Here’s the resulting data needed to build those VMs.

RE:
    CPU:        1
    RAM:        1024 MB
    OS:         FreeBSD
    DiskCtl:    IDE, Controller0 Master
    NICs:       Management, Internal
    Type:       VMvware 5.5 (type 11)
PFE:
    CPU:        1
    RAM:        2048
    OS:         Ubuntu x64
    Diskctl:    IDE, Controller0 Master
    NICs:       Management, Internal, Revenue(n)...
    Type:       VMvware 5.5 (type 11)

Now you can use the WebUI or vmtools CLI and build virtual machines. I had issues running these VMs as version 6.5, but I can confirm they work just fine as 5.5!

Here’s another showing how to set up the disk controller, CPU and RAM.

One final set of warnings. When launching the VMs, sometimes the NIC order changes. Ensure the ordering of NICs:

1. Management NIC
2. Internal NIC (on own vSwitch with 9000 byte MTU)
3. For PFE port three onwards are revenue ports

Also, the availability of the PFE for 18.1 isn’t there yet, so I used the 17.4 PFE. It is normal to have semantic version mismatches between the RE and PFE virtual machines, so don’t worry!

Finally finally, this is a community supported project, so please do not moan at Juniper or your account manager!

Enjoy.

The post Juniper vQFX10K on ESXi 6.5 appeared first on ipengineer.net.

This blog post was initially sent to the subscribers of my SDN and Network Automation mailing list. Subscribe here.

A while ago I complained how the GUI- or API-based orchestration (or intent-based) systems make it hard to figure out what exactly has been configured because they can’t give you a single text configuration file that you could track with version-control software.

Dirk Feldhaus found the situation so ridiculous that he decided to create an Ansible playbook that collects and dumps tenant parameters configured on a Cisco ACI tenant as a homework assignment in the Building Network Automation Solutions online course. As he explained the problem:

Read more ...

Qn from the audience, answered.

Done with phone calls.

After figuring out how packet forwarding really works within AWS VPC (here’s an overview, the slide deck is already available to ipSpace.net subscribers) the next obvious question should be: “and how do I integrate a network services device like a next-generation firewall I have to use because $securityPolicy into that environment?”

Please don’t get me started on whether that makes sense, that’s a different discussion.

Christer Swartz, an old-time CCIE and occasional guest on Software Gone Wild podcast will show you how to do it with a Palo Alto firewall during my Amazon Web Services Networking Deep Dive workshop on June 13^th in Zurich, Switzerland (register here).

I feel like a teenage girl with a fashion blog who hasn’t posted in 6 months and comes back with “I know I haven’t posted in a while…”  Sigh.  It’s been right at a year since I actually published a post, so I figured I would give everyone an update.

I’ve had some personal things going on lately, and those have taken all of my energy.  We’ve made it through those rough times, so my energy is coming back.  I’m feeling better every day, and I hope I can get back to producing some content.  And, let me tell you…I’ve got some stuff to talk about.

*insert star wipe here*

We got a new director-level dude at the office, and he’s really mixing things up for us.  His philosophy includes changing the way we do everything that we do.  Like literally everything.  He ran a report for me on my ticket queue and showed me that 60% of my ticket count was on stupid stuff that’s below my pay grade.  His advice : Make somebody else do it.  So I did.  I taught myself some more Python (not hard since I’ve done coding stuff throughout my career and college), learned some Ansible, wrote a couple modules, put some playbooks in Rundeck, and taught our support center how to use them.  Now I don’t have to worry about that noise any more.  Inspired by my resounding success, I looked at other tasks I was doing.  Firewall rule updates, simple routing changes, packet capture setups…now all done through some sort of automation tool.  The whole team is involved, too.  On a regular basis, my teammates and I get calls from other groups asking for help automating their own tasks.  Proud moments for us.  More on that later.

We’re not only changing the way we do things day-to-day, but we’re also thinking about longer-term architectural changes.  We’re bringing in whitebox switches where we can to save capital while still providing functionality and stability. In our first journey down the whitebox road, we went with Cumulus Networks running on top of EdgeCore switches to form a pretty little EVPN fabric in one of our smaller data centers.  I’ll write more on this later, but it was a great experience doing something other than the normal IOS and NX-OS configurations.  If we put our new-found enthusiasm for automation together with this Debian-based OS on the switches, we get a nifty little system that’s functional, scalable, stable, flexible, manageable…I’m running out of Ls here.  I’m really super-excited to be part of this.

One thing that hasn’t changed, though, is attending Cisco Live.  I will be there next week with bells on.  This year, I’m going on the Imagine pass, so let’s see what happens.  As usual, my schedule will be pretty booked.  The Saturday Adventure is at Kennedy Space Center.  I’ll be at Tech Field Day Extra on Monday and Wednesday.  World of Solutions is on the schedule somewhere.  There are parties.  There are tweetups.  There are get-togethers.  A full week of building the real social network (you know…like eyeball contact) and catching up with both people and technology.  All make for a great experience that I wouldn’t miss for anything.  I’m sure I’ll see a lot of you next week in Orlando.

Send any new blog tags questions to me.

Money can buy you cool friends and build your platform.

Using EBGP instead of an IGP (OSPF or IS-IS) in leaf-and-spine data center fabrics is becoming a best practice (read: thing to do when you have no clue what you’re doing).

The usual argument defending this design choice is “BGP scales better than OSPF or IS-IS”. That’s usually true (see also: Internet), and so far, EBGP is the only reasonable choice in very large leaf-and-spine fabrics… but does it really scale better than a link-state IGP in smaller fabrics?

Read more ...

Ivan Pepelnjak has an interesting post up today about MacGyver-ing in the network. He and Simon Milhomme are right that most small-to-medium sized networks are pretty much non-reference architectures and really, really difficult to manage and maintain properly on the best of days. On the worst of days, they’re a nightmare that make you want to run screaming into the night. But why?

One Size Never Fits All

Part of the issue is that reference architectures and cookie-cutter designs aren’t made for SMEs. Sure, the large enterprise and cloud providers have their own special snowflakes. But so too do small IT shops that have been handed a pile of parts and told to make it work.

People like Greg Ferro and Peyton Maynard-Koran believe this is due to vendors and VARs pushing hardware and sales cycles like crazy. I have attributed it to the lack of real training and knowledge about networking. But, it also has a lot to do with the way that people see IT as a cost center. We don’t provide value like marketing. We don’t collect checks like accounting. At best, we’re no different than the utility companies. We’re here because we have to be.

Likewise, when IT is forced into making decisions based on some kind of rebate or sale we tend to get stuck with the blame when things don’t work correctly. People that wouldn’t bat an eye at buying a new Rolex or BMW get downright frugal when they’re deciding which switches to buy to run their operation for the next 10 years. So, they compromise. Surely you all can make this switch work? It only has half the throughput as the one you originally specced but it was on sale!

So, stuck with bad decisions and no real documentation or reference points, we IT wizards do what we can with what we have. Just like Angus MacGyver. Sure, those networks are a pain to manage. They’re a huge issue when it’s time to upgrade to the next discount switch. And given the number of fires that we have to fight on a daily basis we never get a chance to go back and fix when we nailed up in the first place. Nothing is more permanent than a temporary fix. And no install lasts longer than one that was prefaced with “let me just get this working for the next week”.

Bespoke Off The Rack

So, how do we fix the MacGyver-ing? It’s not going to be easy. It’s going to be painful and require us to step out of our comfort zones.

1. Shut Up And Do As I Say. This one is hard. Instead of having management breathing down your neck to get something installed or working on their schedule, you need to push back. You need to tell people that rushing is only going to complicate things. You need to fire back when someone hands you equipment that doesn’t meet your spec. You need to paraphrase the famous Shigeru Miyamoto – “A delayed network is eventually good, but a rushed installation is bad forever.”
2. Document Like It Will Be Read Aloud In Court. Documentation isn’t just a suggestion. It’s a necessity. We’ve heard about the “hit by a bus” test. It’s more than just that, though. You need to not only be able to replace yourself but also to be able to have references for why you made the decisions you did. MacGyver-ing happens when we can’t remember why we made the decisions we did. It happens when we need to come up with a last minute solution to an impossible problem created by other people (NSFW language). Better to have that solution documented in full so you know what’s going on three years from now.
3. Plan Like It’s Happening Tomorrow. Want to be ready for a refresh? Plan it today. Come back to your plan. Have a replacement strategy for every piece of equipment in your organization. Refresh the plan every few months. When someone comes to you with a new idea or a new thing, write up a plan. Yes, it’s going to be a lot of spinning your wheels. but it’s also going to be a better solution when someone comes to you and says that it’s time to make a project happen. Then, all you need to do is pull out your plan and make it happen. Imagine it like the opposite of a DR plan – Instead of the worst case scenario this is your chance to make it the best case.

---

Tom’s Take

There’s a reason the ringtone on my phone has been the theme from MacGyver for the last 15 years. I’m good at building something out of nothing. Making things work when they shouldn’t. And, as bad as it sounds, sometimes that’s what’s needed. Especially in the VAR world. But it shouldn’t be standard operating procedure. Instead, make your plan, execute it when the time comes, and make sure no one pushes you into a situation where MacGyver-ing is the last option you have.

Here’s a response I got from Simon Milhomme on my Why Is Network Automation So Hard article:

Read more ...

Traditional networking engineers, or virtualization engineers familiar with vSphere or VMware NSX, often feel like Alice in Wonderland when entering the world of Amazon Web Services. Everything looks and sounds familiar, and yet it all feels a bit different

I decided to create a half-day workshop (first delivery: June 13^th in Zurich, Switzerland) to make it easier to grasp the fundamentals of AWS networking, and will publish high-level summaries as a series of blog posts. Let’s start with an overview of what’s different:

Read more ...

As discussed in a previous blog post, IETF designed EVPN to be next-generation BGP-based VPN technology providing scalable layer-2 and layer-3 VPN functionality. EVPN was initially designed to be used with MPLS data plane and was later extended to use numerous data plane encapsulations, VXLAN being the most common one.

Design Requirements

Like any other BGP-based solution, EVPN uses BGP to transport endpoint reachability information (customer MAC and IP addresses and prefixes, flooding trees, and multi-attached segments), and relies on an underlying routing protocol to provide BGP next-hop reachability information.

Read more ...

David Barroso was sick-and-tired of using ZX Spectrum of Network Automation and decided to create an alternative with similar functionality but a proper programming language instead of YAML dictionaries masquerading as one. The result: Nornir, an interesting network automation tool formerly known as Brigade we discussed in Episode 90 of Software Gone Wild.

Notes:

• If you want to test-drive Nornir, start here;
• To get the references at the end of our chat, watch this and this;
• David will talk about Nornir in autumn 2018 session of Building Network Automation Solutions online course.

Silicon Valley has rules about shipping faulty, incomplete products for the first version. If you are not embarrassed by the first version of your product, you’ve launched too late. – Reid Hoffman Many companies, including Enterprise IT vendors, take this as a way to charge high prices while they finish the product. Executives readily convince […]

Being an introvert at school is about survival until you can escape.