During the Think Tank that I participated in at Dell World, the topic of conversation turned to startups.  Specifically, how do startups drive innovation?  As I listened to the folks around the table like Justin Warren (@JPWarren) and Bob Plankers (@Plankers) talk about the advantages that startups enjoy when it comes to agility, I started to wonder if some startups today are hurting the process more than they are helping it.

Exit Strategy

The entire point of creating a business is to make money.  You do that by creating a product that you can sell to someone.  It doesn’t have to be a runaway success.  So long as you structure the business correctly you can make money for a good long while.  The key is that you must structure the business to pay off in the long run.

Startups seem to have this idea that the most important part of the equation is to build something quickly and get it onto the market.  The business comes second.  That only works if you are playing a very short game.  The bad decisions you make in the foundation of your business will come back to bite you down the road.

Startups that don’t have a business plan only have one other option – an exit strategy.  In far too many cases, the business plan for a startup is to build something so awesome that another larger company is going to want to buy them.  As I’ve said before talking about innovation, buying your way into a new product line does work to a point.  For the large vendor, it is dependent on the available cash on hand.  For the startup, the idea is that you need to have enough capital on hand to survive long enough to be bought.

Looking For A Buyer For The End Of The World

There’s nothing more awkward than a company that’s obviously seeking a buyout from a large vendor that hasn’t received it yet.  I look at the example of Violin Memory.  Violin makes flash storage cards for servers to accelerate caching for workloads.  They were a strategic parter of HP for a long time.  Eventually, HP decided to build those cards themselves rather than rely on Violin as a supplier.  This put Violin in a very difficult position.  In hindsight, I’m sure that Violin wanted to be bought by HP and become a division inside the server organization.  Instead, they were forced to look elsewhere for funds.  They chose to file an initial public offering (IPO) that hit the initial target.  After that, the parts of the business that weren’t so great started dragging the stock price down, angering the investors to the point where executives are starting to leave and lawsuits look likely.

Where did Violin go wrong?  If they had built a solid business in the first place they might have been able to keep selling along even though HP had decided not to buy them.  They might have been able to stay afloat long enough to find another buyer or file an IPO when they have a more stable situation with earnings or expenses.  They might have been able to make money instead of losing it hand over fist.

The Standoff

The idea that startups are just looking for a payday from a larger company is hurting innovation.  Startups just want to get the idea formed enough to get some interest from buyers.  Who cares if we can make it work in reality?  We just have to get someone to bite on the prototype.  That means that development is key.  Things like payroll and operating expenses come second.

In return, companies are becoming skittish of buying a startup.  Why take a chance on a company that may not be around next week?  I’d rather spend my time on internal innovation.  Or, better yet buy that failed startup for pennies on the dollar when they liquidate due to inability to manage a business.  Larger companies are going to shy away from startups that want millions (or billions) for ideas.  That lengthens the time that it takes to innovate, either because companies must invest time internally or spend countless hours negotiating purchases of intellectual property.

---

Tom’s Take

Obviously, not all startups have these issues.  Look at companies like Nimble Storage.  They are a business first.  They make money.  They don’t have out-of-control expenses.  They filed for an IPO because it was the right time, not because they needed more money to keep the lights on.  That’s the key to proper innovation.  Build a company that just happens to make something.  Don’t build a product that just happens to have a business around it.  That means you can continue to improve and innovate on your product as time goes on.  It means you don’t have to look for an exit strategy as soon as the funding starts running dry.  Then your strategy looks more like a plan.

It’s easy to say “SDN is the physical separation of the network control plane from the forwarding plane, and where a control plane controls several devices,” handwave over the details, and let someone else figure them out. Implementing that concept in a reliable manner is a totally different undertaking.

Read more ...

The rise in Bitcoin values seems to have caused an equal increase of Bitcoin spam as malware authors attempt to make money off the many new market participants. One site that was spammed to me three times in one day is bitcoin-alarm.net. I ignored it the first two times, but they must have really wanted me to look at it, so who am I not to oblidge.

The site promises a tool to notify you of market changes by SMS, without ever mentioning any nefarious behaviour. YouTube videos teach you what Bitcoin is, and how to install this free tool.  They even provide a link so you can donate to the author, although it appears no one has chosen to do so. This I have to download.

The download BitcoinAlarm.exe (MD5: edfa12d4a454b0eb786bbe92050ab88a) had just 1 hit on VirusTotal when I first scanned it (from Kaspersky). Is it a false positive on a nice free tool? Lets dig deeper.

The download is an installer. A quick strings didn’t turn up anything interesting, so lets try binwalk:

I carved out this RAR archive to see what it contains:

dd if=BitcoinAlarm.exe.virus of=out.rar bs=1 skip=756224
mkdir ext
unrar x out.rar ext/

There’s an SFX script run, lets see what it does:

A quick check of winupdate.exe with VirusTotal shows that it’s the valid (and non-malicious) AutoIt executable. AutoIt is a great little scripting language for Windows, it’s especially useful for automating GUI related tasks. So if winupdate.exe is AutoIt that would make 5943564.IFW an AutoIt script. It looks like it was obfuscated somewhat though:

Run it through

sed -e '/^;[0-9]/d'

to clean up the garbage and we end up with this script. It starts by checking if Avast is running and if so it sleeps for 20 seconds. I guess this is long enough for Avast to get bored and go look at something else:

Well, that’s certainly not a good sign. It’s a pretty solid chance that if software is checking for an antivirus engine that it’s up to no good. A scan of the rest of the file contains other interesting methods like “disable_uac”, “anti_hook”, “persistence”, “botkiller”, “downloader”, “disable_syste_restore”. It’s starting to look like Kaspersky was right, congrats on being the 1/49 to detect this.

I see a lot of calls to IniRead(), and they’re all reading 65901.PPZ. It looks like this is the configuration file. In contains:

[6404000]
6662859=9455413
[2244034]
6224525=3244993
[3206254]
5598349=4588436
[5378250]
6296134=4064234
[1109091]
1109091=asvep

Matching these to the script we see find the sections are:

# 6404000 == disable_uac()
# 2244034 == AdlibRegister("anti_hook", 500)
# 3206254 == AdlibRegister("persistence", 500)
# 5378250 == startup()
# 1109021 == $sKey

This crypto key is used in Main to decrypt and run the file 20070.RQT:

The easiest way to decrypt this file was to simply let the script do the work. There’s a lot of code outside of functions though, so care has to be taken to remove everything non-crypto related. Remove the _RunPE() and replace it with

FileWrite($uniscriptdir & "\DECRYPTED", $sArquive)

The decrypted file had 30/48 hits of VirusTotal when I scanned it (MD5: 224c73f8172123e5ddca2302425664a6). It’s called NetWiredRC and is a remote access trojan made for stealing login information, and likely in this case being used to steal Bitcoins. It connect to bitcoins.dd-dns.de on port 3360.

Some choice credential related strings from the decrypted malware:

%s\Thunderbird\profiles.ini
select *  from moz_logins
%s\.purple\accounts.xml
Software\Microsoft\Internet Explorer\IntelliForms\Storage2

This free utility is nothing more than malware with very low detection rate being spammed to anyone that might have a Bitcoin sitting around. When I checked the domain with urlvoid it had zero ‘bad’ reports and was not blacklisted. I’ve since submitted the domain to multiple scanners and it’s now detected by Scumware.

On a recheck BitcoinAlarm.exe’s detection is up to 14 of 49 scanners, and the download link appears to return 404. bitcoins.dd-dns.de is no longer answering on port 3360.

Never before has it been so easy to leave cash accessible from the Internet, so expect more malware to make off with your Bitcoin wallet. Bitcoins that are not in use should be moved off into cold storage, or donated to the human fund at 136K8a5Mb8uDguFb7RnoXz7gzBSe2xaEED (ahem, worth a shot right?).

In late November I got a perfect excuse for visiting South Africa – I was invited to be a guest speaker at CCIE Club Africa meeting talking about the obvious topics: SDN, OpenFlow and Network Function Virtualization (NFV).

In the end, I delivered three SDN presentations in two days, all of them for engineers focusing primarily on Cisco, and got pleasantly surprised by their keen interest in the basics of these new technologies.

Read more ...

If the motivation behind the effort behind securing BGP was to allow any BGP speaker to distinguish between routing updates that contained “genuine” routing information and routing updates that contained contrived or false information, then these two reports point out that we’ve fallen short of that target. What’s gone wrong? Why are certain forms of routing Man-In-The-Middle attacks all but undetectable for the RPKI-enabled BGPSEC framework?

 I'm reading a book by Michael Lewis called Boomerang.  It was written about a year and a half ago in the aftermath of the stock market crash and the U.S. subprime mortgage mess. The book goes through the impact of the financial crash from the perspective of four different countries (Iceland, Ireland, Greece and Germany) and a couple of small local U.S. governments (San Jose and Vallejo).  It's a fascinating read and, like his other books, take a complex subject and turns it into something understandable. 

 

If you haven't read his other book on this topic, The Big Short, it's worth a weekend to understand what happened in sub-prime.  But this blog isn't meant to be an advertisement for you to buy Michael Lewis books; it's about a thought that occurred to me while reading that relates to security. 

Introduction We’re in one of the most exciting times in data networking.  While I’m sure we’re all sick of vendors co-opting technologies in their infancy, there is a lot of good work going on to change the fundamentals of moving data (I shudder to call this a paradigm shift; I’ll save that term for life […]

As an avid reader of RFCs and RFC drafts, I’m always running across little bits of knowledge I either already knew and forgot (I forget a lot of things), or things I didn’t know and wouldn’t have expected. RFC5942, published way back in 2010 (a long time in network engineering terms), discusses a topic I […]

This sponsored blog post was written by Clark Zoeller, CCIE #13760, Sales Engineer with ActionPacked Networks. Quality of Service (QoS) is a suite of technologies used to manage bandwidth usage as data crosses computer networks. Its most common use is for protection of real-time and high priority data applications. QoS technologies, or tools, each have […]

Lots of talk about simplicity recently and the fact that networking is too complicated and inflexible. There are a number of choices that you deliver simplicity in Overlay Networks. Least Functionality  - One choice is to reduce the network to most minimal and least functionality. But we tried that with L2 ECMP protocols like TRILL […]

When it comes to employee-owned devices entering the network, finding a balance between company security and end-user productivity is a major challenge. That delicate balance is really the major challenge when it comes to “Bring Your Own Devices” (BYOD). The activities of provisioning BYOD devices to ensure secure access to the network as well as enforcing policy and protecting the enterprise is typically a complex multi-vendor pain.  Many times these solutions, while working towards the same goals, don’t necessarily play nicely together. Try telling that to your CEO, who wants to just work from any device, any time and from anywhere.

The Organisation for Economic Co-operation and Development, the OECD, is a widely referenced and respected source of objective economic data and comparative studies of national economies and economic performance. The organization has a very impressive track record of high quality research and a justified reputation of excellence in its publications, even with its evident preference for advocating economic reform through open markets and their associated competitive rigors. OECD activities in the past have proved to be instrumental in facilitating change in governmental approaches to common issues that have broad economic and social dimensions. So how does IPv6 fit into this picture of OECD activities?

A good friend of mine sent me an interesting question:

When I configure mpls ip on an interface, will all packets on that interface be labeled, or just the MPLS/VPN packets received through VRFs? I always assumed that stuff in the global routing table just got forwarded as IP packets without any labels.

Well, that’s not how MPLS works (at least not in its default incarnation on Cisco IOS).

Read more ...

Allow me to introduce another of our Security & Mobility Now bloggers.

 

Please meet Ashur Kanoon, senior manager, technical marketing.

Allow me to introduce another of our Security & Mobility Now bloggers.

 

Please meet Kyle Adams, our chief software architect for Junos WebApp Secure.

Packet forwarding behavior of VMware NSX and Hyper-V Network Virtualization is well documented; no such documentation exists for Amazon VPC. However, even though Amazon uses a proprietary solution (heavily modified Xen hypervisor with homemade virtual switch), it’s pretty easy to figure out the basics from the observed network behavior and extensive user documentation.

Read more ...

Getting a CCIE is considered to be the pinnacle of a person’s networking career.  It is the culmination of hundreds (if not thousands) of hours of study.  People pass the lab and celebrate with the relief that can only come from completing a milestone in life.  But it’s important for newly-minted CCIEs to realize that getting your number doesn’t mean you obtained hubris with it.

A great article that talks about something similar comes from Hunter Walk.  It’s Fine To Get an MBA, But Don’t Be An MBA shows many of the things I’m talking about.  With the MBA, it’s a bit different.  The MBA is a pure book learning environment with very little practical experience.  The CCIE is a totally practical exam that requires demonstration of knowledge.  However, both of these things share something in common.  People get very hung up on the knowledge from the certification and forget to keep an open mind about other ideas.  In essence, someone that is “Being a CCIE” is using their certification incorrectly.

Here are some points:

Get A CCIE to further your knowledge about networking and learn how system work. Don’t Be A CCIE and think that you’ve learned everything there is to know about networking.

Get A CCIE and work with your coworkers and peers to solve problems.  Don’t Be A CCIE and ignore everyone because you think you’re smarter than they are.

Get A CCIE and contribute to the community with knowledge and experience.  Don’t Be A CCIE and refuse to share because you can’t be bothered.

Get A CCIE and help your company to take on bigger and better networking projects.  Don’t Be A CCIE and assume you are indispensable.

Get A CCIE because you want to.  Don’t Be A CCIE and assume you’ve always been one.

A CCIE doesn’t change who you are.  It just serves to show people how dedicated you can be.  Don’t let five little numbers turn you into a bully or a know-it-all.  Realize you still have much to learn.  Understand that your position is now at the forefront of where networking is going, not where it has been.  When you know that being a CCIE is more than just a piece of paper, then you will have truly gotten your CCIE.

Cisco FabricPath is a TRILL-based layer 2 forwarding technology that can take the place of spanning-tree. Allowing a fully-meshed layer 2 network to forward traffic across all links, FabricPath helps customers to make the most of their expensive 10GbE and 40GbE interconnects. In this show, Jamie Caesar, Colby Glass, and Ed Diaz discuss real-world FabricPath design and deployment with host Ethan Banks. There’s no one from Cisco on this show – just end users sharing their real-life FabricPath experiences. This is part 2 on FabricPath. You should probably listen to part 1 before listening to part 2. Enjoy! Links Show 152 - Nexus Announcements From Cisco Live 2013 With Ron Fuller – Sponsored < Includes a discussion about Nexus Validation Testing, which Ethan mentioned in this show. This show was sponsored in part by INE.com. Please give them a visit. Excellent training from some of the very best in the business.

More than 15 years ago the cover story of ACM netWorker magazine discussed the dawn of the stupid network – an architecture with smart edge nodes and simple packet forwarding code. Obviously we learned nothing in all those years – we’re still having the same discussions.

Here are a few juicy quotes from that article (taken completely out of context solely for your enjoyment).

Read more ...

The Cisco Product Security Incident Response Team (PSIRT) has published three important vulnerability advisories: Cisco TelePresence VX Clinical Assistant Administrative Password Reset Vulnerability Cisco WAAS Mobile Remote Code Execution Vulnerability Cisco IOS Software Session Initiation Protocol Denial of Service Vulnerability Cisco TelePresence VX Clinical Assistant Administrative Password Reset Vulnerability The vulnerability is due to a coding error that resets the password for the admin user to a blank password on every reboot. An attacker could exploit this vulnerability by logging in to the administrative interface as the admin user with a blank password. Vulnerable Products Cisco TelePresence VX Clinical Assistant […]

As I approach the end of the calendar year & prepare for a holiday break, I've been reflecting on what I've learned and done this year. Of course, this also leads to what I need to learn next year.I'm making a number of changes to my career next year and as part of my planning process I wrote a list of eight things I think I've learned in the last 20 years.

Github project that checks a DNS entry against over 1500 DNS servers worldwide. For those times when changing the IP address of the mail server or website and you want to see how the DNS propagation in progressing. Very useful if you haven’t been able to reduce the DNS TTL time prior to the change […]

Gateways between overlay virtual world and (VLAN-based) physical reality are a crucial component in every design using overlay virtual networks. Ideally one could use virtual appliances, but sometimes the users keep asking for layer-2 gateways.

The VMware NSX Layer-2 Gateways video from the VMware NSX Architecture webinar describes the use cases for layer-2 gateways and the VMware NSX implementations.

The only thing I would say to anyone who will take the new version is don’t fear any change, or addition to the updated version of the lab. When version 4 was announced there was so much fear over the troubleshooting section, and MPLS being added. Change brings fear, unnecessary fear. I ended up failing version 3 of the lab two weeks before version 4 came out. I wasn’t ready for it. So I took two more years before making another attempt (to be fair we had another child between attempts…) Then I passed version 4 on my first attempt. Simply because I was ready to take it. INE, Narbik, and IPexpert will have their material updated to get you through the new version. Study hard, and run with it…

Yea I know there is not much here, but I don’t want to give my long drawn out opinion on a version of the lab I will never have to take. I don’t think it is fair to anyone reading this that will take the new version I would strongly pay attention to what the trainers have to say though.

The new MDM integration with your Juniper SSL VPN appliance is the biggest advance in endpoint posturing since … well, forever.  And, now's the time to re-look at an old favorite feature - HostChecker - and see how integration with Junos Pulse can eliminate the dependence on ActiveX and Java on connecting endpoints.

Tomorrow, December 6th 2013, at 10:00 PST (GMT 18:00) I will be running a free live online session on Introduction to DMVPN for CCIE R&S v5 Candidates.  You can sign-up for this seminar here.  Additionally the link to attend is available at the top of the dashboard when you login to the INE Members Site.

This session is the first of many to help candidates transition from the current CCIE R&S v4 Blueprint to the recently announced CCIE R&S v5 Blueprint that goes live on June 4^th 2014. We will continue to run additional sessions in the future on new topics that have been added to the CCIE R&S v5 Blueprint, such as IPv6 First Hop Security, IPsec LAN-to-LAN tunnels, GET VPN, IGP Convergence & Scalability, and BGP Convergence & Scalability, just to name a few.  These sessions are not only applicable to CCIE R&S v5 candidates, but also to those pursuing the CCNA, CCNP, or CCIE Security tracks, as well as for everyday engineers looking to apply these technologies in their production environments.

Tomorrow’s session will focus on the theory of what Dynamic Multipoint VPN (DMVPN) is, what problems it was designed to solve, and where it fits in the overall network design as compared to other technologies such as MPLS Virtual Private LAN Service (VPLS) or MPLS Layer 3 VPNs.  The session will also include live implementation examples of DMVPN on the Cisco IOS CLI.  Expect this session to run somewhere around 2 – 3 hours in length.

I hope to see you there!

Last week I explained how layer-2 and layer-3 packet forwarding works in VMware NSX – a solution that closely emulates traditional L2 and L3 networks. Hyper-V Network Virtualization (HNV) is different – it’s almost a layer-3-only solution with only a few ties to layer-2.

Read more ...

This is the Cisco ASA ethernet information leak exploit that leverages the vulnerability noted in CVE-2003-0001. Versions prior to 8.4.4.6 and 8.2.5.32 are affected. Multiple platform ethernet Network Interface Card (NIC) device drivers incorrectly handle frame padding, allowing an attacker to view slices of previously transmitted packets or portions of kernel memory. This vulnerability is the result of incorrect implementations of RFC requirements and poor programming practices, the combination of which results in several variations of this information leakage vulnerability. The simplest attack using this vulnerability would be to send ICMP echo messages to a machine with a vulnerable ethernet […]

Initial release of Hyper-V Network Virtualization (HNV) was an add-on to the Hyper-V Extensible Switch, resulting in an interesting mixture of bridging and routing. In Windows Server 2012 R2 the two components became tightly integrated, resulting in a pure layer-3 solution.

Read more ...

Cisco announced this week that they are upgrading the venerable CCIE certification to version five.  It’s been about three years since Cisco last refreshed the exam and several thousand people have gotten their digits.  However, technology marches on.  Cisco talked to several subject matter experts (SMEs) and decided that some changes were in order.  Here are a few of the ones that I found the most interesting.

Time Is On My Side

The v5 lab exam has two pacing changes that reflect reality a bit better.  The first is the ability to take some extra time on the troubleshooting section.  One of my biggest peeves about the TS section was the hard 2-hour time limit.  One of my failing attempts had me right on the verge of solving an issue when the time limit slammed shut on me.  If I only had five more minutes, I could have solved that problem.  Now, I can take those five minutes.

The TS section has an available 30 minute overflow window that can be used to extend your time.  Be aware that time has to come from somewhere, since the overall exam is still eight hours.  You’re borrowing time from the configuration section.  Be sure you aren’t doing yourself a disservice at the beginning.  In many cases, the candidates know the lab config cold.  It’s the troubleshooting the need a little more time with.  This is a welcome change in my eyes.

Diagnostics

The biggest addition is the new 30-minute Diagnostic section.  Rather than focusing on problem solving, this section is more about problem determination.  There’s no CLI.  Only a set of artifacts from a system with a problem: emails, log files, etc.  The idea is that the CCIE candidate should be an expert at figuring out what is wrong, not just how to fix it.  This is more in line with the troubleshooting sections in the Voice and Security labs.  Parsing log files for errors is a much larger part of my time than implementing routing.  Teaching candidates what to look for will prevent problems in the future with newly minted CCIEs that can diagnose issues in front of customers.

Some are wondering if the Diagnostic section is going to be the new “weed out” addition, like the Open Ended Questions (OEQs) from v3 and early v4.  I see the Diagnostic section as an attempt to temper the CCIE with more real world needs.  While the exam has never been a test of ideal design, knowing how to fix a non-ideal design when problems occur is important.  Knowing how to find out what’s screwed up is the first step.  It’s high time people learned how to do that.

Be Careful What You Wish For

The CCIE v5 is seeing a lot of technology changes.  The written exam is getting a new section, Network Principles.  This serves to refocus candidates away from Cisco specific solutions and more toward making sure they are experts in networking.  There’s a lot of opportunity to reinforce networking here and not idle trivia about config minimums and maximums.  Let’s hope this pays off.

The content of the written is also being updated.  Cisco is going to make sure candidates know the difference between IOS and IOS XE.  Cisco Express Forwarding is going to get a focus, as is ISIS (again).  Given that ISIS is important in TRILL this could be an indication of where FabricPath development is headed.  The written is also getting more IPv6 topics.  I’ll cover IPv6 in just a bit.

The biggest change in content is the complete removal of frame relay.  It’s been banished to the same pile as ATM and ISDN.  No written, no lab.  In it’s place, we get Dynamic Multipoint VPN (DMVPN).  I’ve talked about why Frame Relay is on the lab before.  People still complained about it.  Now, you get your wish.  DMVPN with OSPF serves the same purpose as Frame Relay with OSPF.  It’s all about Stupid Router Tricks.  Using OSPF with DMVPN requires use of mGRE, which is a Non-Broadcast Multi-Access (NBMA) network.  Just like Frame Relay.  The fact that almost every guide today recommends you use EIGRP with DMVPN should tell you how hard it is to do.  And now you’re forced to use OSPF to simulate NBMA instead of Frame Relay.  Hope all you candidates are happy now.

vCCIE

The lab is also 100% virtual now.  No physical equipment in either the TS or lab config sections.  This is a big change.  Cisco wants to reduce the amount of equipment that needs to be physically present to build a lab.  They also want to be able to offer the lab in more places than San Jose and RTP.  Now, with everything being software, they could offer the lab at any secured PearsonVUE testing center.  They’ve tried in the past, but the access requirements caused some disaster.  Now, it’s all delivered in a browser window.  This will make remote labs possible.  I can see a huge expansion of the testing sites around the time of the launch.

This also means that hardware-specific questions are out.  Like layer 2 QoS on switches.  The last reason to have a physical switch (WRR and SRR queueing) is gone.  Now, all you are going to get quizzed on is software functionality.  Which probably means the loss of a few easy points.  With the removal of Frame Relay and L2 QoS, I bet that services section of the lab is going to be really fun now.

IPv6 Is Real

Now, for my favorite part.  The JNCIE has had a robust IPv6 section for years.  All routing protocols need to be configured for IPv4 and IPv6.  The CCIE has always had a separate IPv6 section.  Not any more.  Going forward in version 5, all routing tasks will be configured for v4 and v6.  Given that RIPng has been retired to the written exam only (finally), it’s a safe bet that you’re going to love working with OSPFv3 and EIGRP for IPv6.

I think it’s great that Cisco has finally caught up to the reality of the world.  If CCIEs are well versed in IPv6, we should start seeing adoption numbers rise significantly.  Ensuring that engineers know to configure v4 and v6 simultaneously means dual stack is going to be the preferred transition method.  The only IPv6-related thing that worries me is the inclusion of an item on the written exam: IPv6 Network Address Translation.  You all know I’m a huge fan of NAT.  Especially NAT66, which is what I’ve been told will be the tested knowledge.

Um, why?!? 

You’ve removed RIPng to the trivia section.  You collapsed multicast into the main routing portions.  You’re moving forward with IPv6 and making it a critical topic on the test.  And now you’re dredging up NAT?!? We don’t NAT IPv6.  Especially to another IPv6 address.  Unique Local Addresses (ULA) is about the only thing I could see using NAT66.  Ed Horley (@EHorley) thinks it’s a bad idea.  Ivan Pepelnjak (@IOSHints) doesn’t think fondly of it either, but admits it may have a use in SMBs.  And you want CCIEs and enterprise network engineers to understand it?  Why not use LISP instead?  Or maybe a better network design for enterprises that doesn’t need NAT66?  Next time you need an IPv6 SME to tell you how bad this idea is, call me.  I’ve got a list of people.

---

Tom’s Take

I’m glad to see the CCIE update.  Getting rid of Frame Relay and adding more IPv6 is a great thing.  I’m curious to see how the Diagnostic section will play out.  The flexible time for the TS section is way overdue.  The CCIE v5 looks to be pretty solid on paper.  People are going to start complaining about DMVPN.  Or the lack of SDN-related content.  Or the fact that EIGRP is still tested.  But overall, this update should carry the CCIE far enough into the future that we’ll see CCIE 60,000 before it’s refreshed again.

More CCIE v5 Coverage:

Bob McCouch (@BobMcCouch) – Some Thoughts on CCIE R&S v5

Anthony Burke (@Pandom_) – Cisco CCIE v5

Daniel Dib (@DanielDibSWE) – RS v5 – My Thoughts

INE – CCIE R&S Version 5 Updates Now Official

IPExpert – The CCIE Routing and Switching (R&S) 5.0 Lab Is FINALLY Here!

This new strong man of browser technologies is making Java, Flash, Active-X, and SilverLight look as old as grandma's stationwagon, but also helping make for a new and great remote access experience.

Every time I talk about small (per-application) virtual appliances, someone inevitably cries “And who will manage thousands of appliances?” Guess what – I’ve heard similar cries from the mainframe engineers when we started introducing Windows and Unix servers. In the meantime, some sysadmins manage more than 10.000 servers, and we’re still discussing the “benefits” of humongous monolithic firewalls.

Read more ...

There is no such thing as career path.

I wrote this several years ago. And I believe it's still true. For those who don't understand why I wrote such thing, please spend few minutes to read that blog post before leaving nasty comment. (This means: you still can leave nasty comment after reading that post :))

Allow me to share my secret: patience is not my virtue.

Every time I want to change to new position, or to new job title, I move to new company. Some of my previous employers offered me promotion the moment I gave them my resignation letter. Some of them simply didn't care and just let me go. For those who offered me promotion, I never accepted the offer. I thought they should have offered that while I was still with them, not at my last moment in the company when I usually had decided to leave.

There was a time I even worked as independent contractor. Had to deal with the customer directly, defined the scope by myself, set the performance index, and delivered end-to-end solution to customer. No job title. No career. Hmm, good old days. Even it was only for several months before I got back into corporate job.

Eventually I managed to get promotion in one company. And the truth is, when I was given the new job title I didn't feel like it was a promotion. Because I've been doing the scope of work and assuming the responsibility of that new title for years, long before the company really made it happen.

So here is the complete statement: there is no such thing as career path for technical people, who still believe the only way to go up is by working hard in their current position. Most techies believe if they work hard and be good on what they do, somebody will eventually notice and give the reward.

I don't believe such thing exists anymore.

You may disagree as the experience is different for each individual. But based on my own experience, I can suggest you the following to get the promotion or target position you always dream of:

1. Define the target position you want to be
2. Assume responsibility and work with the scope of that target position, regardless of your current position or job title
3. Get the right attention from the right people

What is my next target position, you may ask?
I want to be able to put the title in my business card just as Mark's.












How about you?

Today Cisco posted their official announcement on the upcoming changes for CCIE Routing & Switching Version 5.  The majority of the announcement is along the same lines as previously rumored changes, except for the official launch date, which is now scheduled for June 4^th 2014.  This should bring a great sigh of relief to you if you’re currently nearing the end of your CCIE R&S v4 preparation, as you now have a 6 month window to pass the v4 lab exam before the change to v5 occurs.

Specifically the announcement details changes to technical topics covered both in the written and lab exams, the equipment used, as well as the exam format, as follows:

Technical Topic Changes

New Lab Topics:

• Interpreting Packet Captures
• Bidirectional Forwarding Detection (BFD)
• Multi Address Family (AF) EIGRP
• Dynamic Multipoint VPN (DMVPN)
• IPsec
• IPv6 First Hop Security

Of the new topics announced, the big ones are DMVPN and IPsec.  These are specifically listed as DMVPN Single Hub and IPsec with Pre-Shared Keys, so the scope is not nearly as large as the CCIE Security.  If you don’t yet know what any of these terms mean, don’t worry, you soon will

Topics moved from the Lab to the Written:

• IPv6 Multicast
• RIPng
• IPv6 Tunneling
• IOS AAA with TACACS+ and RADIUS
• 802.1x
• Layer 2 QoS
• Performance Routing (PfR)

Topics completely removed:

• Flexlinks
• ISL
• Layer 2 Protocol Tunneling
• Frame-Relay
• WCCP
• IOS Firewall
• IOS IPS
• RITE
• RMON
• RGMP
• RSVP QoS
• WRR/SRR

For topics removed, there are three killer areas here: Frame Relay, PfR, and Layer 2 QoS.  Frame Relay’s removal is no surprise, as Ethernet based last mile access solutions such as Metro Ethernet and Virtual Private LAN Services (VPLS) have exploded in the past few years and have eclipsed legacy methods such as DS3 Frame Relay.  From a technology design point of view though, a lot of the Frame Relay theory transfers directly over to DMVPN, as DMVPN could be thought of as a way to emulate legacy hub-and-spoke network designs over a public transport.

As for PfR’s removal, this one is a bit of a surprise, and I can already hear Brian Dennis’s screams of agony:

While the general idea of PfR is great, I’ve never seen it implemented other than in very small scale environments due to the management complexity.  You have to give Cisco credit though, as PfR is essentially SDN version 1.0, and now a very large portion of the industry is focused on this type of application.

The other large change here is the removal of Layer 2 QoS.  While this is still a very important topic, the problem with L2 QoS is that it is highly platform dependent, and the way that Catalyst 29xx/35xx/45xx/65xx implement L2 QoS is generally unique to each.  Therefore in the interest of platform independence and virtualization, L2 QoS gets the axe.  This brings us to our next topic, which is the hardware changes in the new blueprint.

Equipment Changes

As previously rumored, the new CCIE R&S v5 equipment is going all virtual.  As CCIE R&S v4 had already been using virtual IOS for the troubleshooting section of the exam, this should come as no surprise. The biggest implication of this change is that the size of the topology is now arbitrary.  I wouldn’t be surprised going into the exam and seeing a configuration section with 20+ routers in the topology.

The other implication of this change is that certain features can no longer be tested on, as they’re not supported in the virtual IOS.  Those topics that can’t be tested, such as Layer 2 QoS or Flexlinks, are now explicitly excluded from the topic scope of the exam.

Format Changes

Last but not least, a new testing section has been introduced into the R&S v5 lab exam format.  While the written exam format stays the same, the lab now includes a “diagnostic” section, which focuses on the diagnosis and resolution of network issues from a more high level point of view.

This new section won’t use equipment, but instead will present the candidate with information such as network diagrams, CLI outputs, log outputs, traffic captures, and email exchanges, based on which they will be expected to diagnose a presented network problem.  Based on the description in the announcement, I would assume that this format is going to be similar to the CCDE Practical Exam testing format, which tests analytical skills without the need of access to actual devices CLI.

Another minor change to the exam is how the timing of sections works.  In the v4 format, candidates had a maximum of 2 hours to complete the troubleshooting section, and a minimum of 6 hours for the configuration section.  If the candidate used less than 2 hours in troubleshooting, the extra time rolled over to the configuration section.  In the v5 format this changes along with the addition of the diagnostic section.

In v5, candidates will have a maximum of 2.5 hours to complete troubleshooting, a fixed 30 minutes for the diagnostic section, and the rest to complete configuration.  Any time less than 2.5 hours used in troubleshooting will be credited towards configuration.  For example if a candidate uses only 1.5 hours in troubleshooting then the configuration section would be 6 hours, which along with the .5 hour of diagnostic adds up to a total of 8 hours for the exam.

How Does This Affect Me As An INE Customer?

The good news is that if you’ve purchased and of the R&S v4 products from INE, you’re covered for the v5 products.  You won’t have to pay anything to upgrade to the v5 products, including the Bootcamps.  If you already attended a v4 bootcamp and want to resit a v5 bootcamp, there’s no charge for it.

As it’s no secret that Cisco’s blueprint changes have been in the works for quite some time, as have INE’s plans for the v5 update.  We have a bunch of new exciting product updates and more importantly new product features that we’re going to be launching along with the v5 product updates.  More information will be available about these updates in the coming weeks.

In the short term I’m going to be running a free online class this Friday – December 6^th 2013 – at 10:00 PST (GMT –8) on Introduction to DMVPN for CCIE R&S Candidates.  I’ll post another blog update tomorrow with more information on this.

 

 

It’s rumored that the announcement for the R&S CCIE v5 update should be coming soon (November timeframe) and the switch over for the lab sometime around March 2014. Cisco Live Europe has a R&S v5 Technical Breakout scheduled for anyone attending. The update to version 5 is rumored to be a 100% virtual lab environment similar to how the troubleshooting section of the lab is done now. The major benefit of the lab going virtual is that the topics covered will be platform independent. You will not need to buy 2911′s or 3750x’s to prepare and can use any relatively newer router or switch to prepare or use a virtualized environment (IOL/IOU/VIRL, GNS3, CSR). The goal of the v5 appears to be to focus on the technologies themselves and less on the hardware and a specific topology. This is the best move Cisco has made for the R&S CCIE program in years as candidates will need to focus more on the technologies themselves and not worry about IOS versions, hardware platforms, physical topologies, etc.

Allegedly the R&S CCIE v5 blueprint will see legacy topics like Frame-Relay removed. Additionally it’s possible some of the more lesser used features of the IOS like Zone-Based Firewall, WCCP, IPv6 multicast, and PfR could be removed from the lab. A few of the topics we could see added are IPSec, DMVPN, and Embedded Packet Capture. We may see ISIS added to the written at least if not the lab. This could be the last version of the R&S lab that isn’t IOS XE based so we could see it added to the written.

Currently the lab has a 2 hour troubleshooting section and a 6 hour configuration section. The new lab may contain, in addition to the troubleshooting section, a new diagnostic section. This means the lab could have a troubleshooting, diagnostic and configuration section. I would assume the points for this new section would come from the configuration section and the troubleshooting section would remain the same or possibly even slightly higher in points.

So what does this mean for someone currently preparing for the R&S v4 blueprint? If you feel you are close to taking the lab but do not have to scheduled, you should schedule a date ASAP. Once the official v5 announcement comes out from Cisco, it will be hard to schedule a lab date. If you have a date scheduled before March 2014 then you should be fine. If your date is after February 2014 then I would recommend you move it up ASAP. The longer we go into November the more likely the new blueprint date has been pushed back by Cisco.

Additional v4 bootcamps will be added to the schedule before the March 2014 changeover. We will start transitioning the current bootcamps over to version 5 around the first of the year. For the self-paced products we will start releasing labs and videos covering the new blueprint in November. 90% of the material from the version 4 blueprint will carry over to the version 5 lab blueprint. Topics you can skip for the routers will be Frame-Relay, PfR, WCCP, Zone-Based Firewall along with technologies that are not supported in IOU L2/IOL L2. Here is a list of features we may not see for the layer 2 section since the switches will be virtual.

1) QinQ Tunneling
2) ISL trunks
3) DHCP Snooping
4) Layer 3 Port Channel
5) Private VLANs
6) SPAN/RSPAN/ERSPAN

Post any questions you have about the new blueprint changes and I’ll start creating a FAQ below:

Q) I purchased the version 4 self-paced material so will I be covered for the version 5 products?
A) Yes.

Q) Will I have to pay anything for get the new version 5 material?
A) No.

Q) I attended an R&S CCIE v4 bootcamp so can I resit a v5 bootcamp for free?
A) Yes.

Q) Will INE offer racks for the new blueprint?
A) Yes. Although the lab is virtual it is still good to spend part of your preparation using real hardware as that is what you use in your day-to-day job. Towards the end of your preparation you can hone your skills using virtualized environment. We will be using the CSR along with real switches for the virtualized environments.

Q) What about my tokens?
A) Your tokens will carry over.

Q) I’m currently schedule for the lab after the v5 update. Can I still take the v4 lab?
A) No.

Q) Do you feel strongly that the announcement will come out in November?
A) I do feel confident that the announcement will be in November but it could slip since they are trying to align the Cisco 360 update to the lab release.

Inside Recent Point-of-Sale Malware Campaign Activities

Curt Wilson, Dave Loftus, Matt Bing

An active Point of Sale (PoS) compromise campaign designed to steal credit and debit card data using the Dexter and Project Hook malware has been detected. Indicators of compromise will be provided for mitigation and detection purposes. Prior to the publication of this Threat Intelligence document (embedded at the end of this post), members of the FS-ISAC, major Credit Card vendors and law enforcement were notified.

It appears that there are at least three distinct versions of Dexter:

1. Stardust (looks to be an older version, perhaps version 1)
2. Millenium (note spelling)
3. Revelation (two observed malware samples; has the capability to use FTP to exfiltrate data)

In early November 2013, ASERT researchers discovered two servers hosting Dexter and other POS malware to include Project Hook.  The Dexter campaign looks more active, especially in the Eastern Hemisphere and therefore shall be the main focus herein. Dexter, first documented by Seculert in December 2012, is a Windows-based malware used to steal credit card data from PoS systems. The exact method of compromise is not currently known, however PoS systems suffer from the same security challenges that any other Windows-based deployment does. Network and host-based vulnerabilities (such as default or weak credentials accessible over Remote Desktop and open wireless networks that include a PoS machine), misuse, social engineering and physical access are likely candidates for infection. Additionally, potential brittleness and obvious criticality of PoS systems may be a factor in the reportedly slow patch deployment process on PoS machines, which increases risk. Smaller businesses are likely an easier target due to reduced security. While the attackers may receive less card data from smaller retailers, infections may be more numerous and last longer due to the lack of security reporting and security staff in such environments.

Figure 1: Dexter (Purple) and Project Hook (Orange) infections in the Eastern Hemisphere

Figure 2: Dexter (Purple) and Project Hook (Orange) infections in the western hemisphere

For the full document to include a list of various compromise indicators and information about the back-end infrastructure, please download the full public report -

Dexter and Project Hook Break the Bank

 

In the first article here, I walked through importing netflow data into a single Hadoop instance (pseudonode) and mentioned a progression of the project to add multiple nodes. The ability to do distributed storage and distributed processing of data is ultimately the benefit of using Hadoop/HDFS. So, let’s expand on the project and add one or […]

This week it's Greg was configuring spanning tree in the data centre and had a problem with a switch cluster that didn't work proper. How much networking do you need in a data centre ? Lets say you purchases 2 x 32 port 40GbE switches (common Trident2 configuration) for USD$30K and you use QSFP breakouts to give you 4 x 10GbE per 40GbE port. For easier maths lets assume that you use 25 ports of 40GbE to give you 100 x 10GbE port in total. That's enough ports for 50 blade chassis with 2 x 10GbE chassis. Each chassis holds eight servers for a total of 400 physical servers. At a low density of 20 VMs per physical servers, that makes a total of 20 x 400 or 8000 VMs. You don't need even need a network to make that work, just two switches. Nexus 9000 is really cheap. So how will Cisco make it's revenue ? What will be the actual price of ACI licenses ? For that matter what about VMware NSX ? Both companies have paid around a billion dollars to build these products and will expect to recover that cost. Will customers pay for expensive software ? The Nexus 9000 has limited L2 and L3 features and may not be the right solution to replace your existing switches. It's not clear whether they do MLAG etc. Nexus 7000 and ACI/APIC integration is likely to have some limitations or specific criteria . Will it need new line cards ? What about Supervisor upgrades ? And will the NX-OS Plus arrive on time and in good, reliable condition ? Noting the similarities between Juniper QFabric and Cisco ACI and vindication of Juniper's strategy. Concerns about the lack of standards in SDN and specifically the lack of certainty around interoperability between ACI, NSX, OpenDaylight and many others. Talking about the fact that the data centre is not one single network, it's many small networks connected together.

Once you decide to use BGP as the routing protocol in your DMVPN network, you face a few more design choices:

• Should you use IBGP or EBGP?
• Should you use a unique AS number for every DMVPN site, or the same AS number on all spoke sites?

The BGP Routing in DMVPN Access Networks ExpertExpress case study describes these dilemmas in more details; if you face a similar problem and would like me to review your design, get in touch.

table.padded-table td { padding:0px 20px 0px 0px }

Tensions in Thailand are high as a week of anti-government protests have turned violent and continue unabated. However, in an apparent reversal of a common theme with anti-government protests in recent years, multiple news sources have reported that protestors temporarily cut off a large portion of Internet service to their country. For example, The New York Times reported yesterday that “protesters also raided a state-owned telecommunications office, temporarily cutting Internet service to thousands of people on Saturday.”

CAT Telecom Outage

We can confirm that state-owned Thai telecom Communications Authority of Thailand (AS4651), commonly referred to as CAT Telecom, was completely offline for over three hours on Saturday during the government’s crackdown on the protestors. In a series of three outages, CAT stopped providing Internet transit to 204 routes at 8:30 UTC, 207 routes at 8:48 UTC and finally lost service to 906 routes at 8:52 UTC. In total, the outages left Thailand with 32% of its Internet paths down until restoration occurred at 12:11 UTC.

With CAT’s routes down, all Internet communication to and through them came to a halt. This is illustrated by our latency measurements, shown on the right. However, although CAT was unavailable for this period of time, alternative international gateways JasTel, TOT and True Internet were either unaffected or experienced surges in traffic as multi-homed customers shifted traffic away from CAT.

Alternatives Unaffected

During the CAT outage, we observed a large amount of traffic re-routed to True Internet (AS38082) pictured to the right. CAT’s rival, state-owned telecom TOT (AS38040) also experienced an increased in traffic when Internet Thailand shifted traffic away from CAT pictured directly below. The unavailability of CAT transit had negligible impact on the service of private operator JasTel (AS45629), pictured below right.

Corporate Impact

Due to Internet transit diversity among corporations in Thailand, we can see that many enterprises were able to ride out the outage by immediately shifting transit to an alternative provider. For example, the Siam Commercial Bank (AS45806) was able to use service from True Internet during the CAT Telecom outage as illustrated on the right. The image below illustrates how Siam Commercial Bank connects to the global Internet. Note the secondary connectivity available through True Internet.

Conclusion

If it is true that the anti-government protestors caused the 3-hour CAT Telecom outage on Saturday, this development raises the question: will national telecommunications infrastructure become a high-priority target for protestors?

Improvements in Internet diversity are politically neutral — they help keep the Internet connected, regardless of who’s trying to shut off the Internet, or for what purpose. From the perspective of Thailand’s financial sector and others that rely on the Internet, this is a good thing.

The post Protests Lead to Outage in Thailand appeared first on Renesys.

This is a useful looking utility on making the network on a windows system perform badly so that developers can test their applications over poor network connection. I think I can use this to make application performance the developer’s problem (instead of mine after the app is deployed). Leveraging the awesome WinDivert library, clumsy stops […]

During the recent Storage Field Day 4, I was listening to Howard Marks (@DeepStorageNet) talking about storage Quality of Service (QoS).  He was describing something he wanted his storage array to do that sounded suspiciously similar to strict priority queuing (or Low Latency Queuing if you speak Cisco).

As I explained how we solved the problem of allowing a specific amount of priority for a given packet stream, it finally dawned on me how software defined networking had the ability to increase productivity in organizations.  It’s not via automation, although that is a very alluring feature.  It’s because SDN can act as a language interpreter for those that don’t speak the right syntax.

It’s said that a picture is worth a thousand words.  As true as that is I would also argue that a few words can paint a thousand pictures.  It’s all in the interpretation.  If I told you I wanted a picture of a “meadow at sunrise”, you’ve probably come up with an idea in your head of what that would look like.  And the odds are good that it’s totally different from my picture.  These simple words are open to interpretation on a wide scale.  Now, let’s constrain the pictures a bit based on the recipient.  Someone that lives in France would have a totally different idea of a meadow than someone that lives in San Francisco.  Each view is totally valid, but the construction of the picture will be dictated by the thought process of the person.  They each know what the meadow looks like, they’re just a bit different.

Painting with SDN

Extend this metaphor to software.  Specifically to networking and storage.  I have a concept that I need to implement.  User A needs guaranteed access to a specific resource for a period of time that should not exceed a given threshold.  A phone may need priority queue access to a slow WAN link.  A backup client may need guaranteed access to a target server without overrunning the link bandwidth.  Two totally different use cases that can be described in the same general language.  Today, I would need two people to implement those instructions on different systems.  Programming a storage array is much different that programming a router or a switch.

But the abstraction of SDN allows my to input a given command and have it executed on dissimilar hardware via automation and integration.  The SDN management system doesn’t care that I want LLQ on a router or strict priority QoS on a backup client.  It knows that there should be an implementation of the given commands on a set of attached systems.  It’s up to the API integration with the systems to determine the syntax needed to execute the commands.  As a network engineer, I don’t need to know the commands to create the QoS construct on the storage array.  I just need to know what I want to accomplish.  The software takes care of the rest.

SDN could usher in a new age for natural language programming.  I often hear my friends in the CCIE community complaining that people only learn the commands to make something happen.  They ignore the basic concepts behind why something works the way it does.  If I can’t explain the concept to my 8-year old son, I don’t know it very well.  I can’t abstract it to the point where a simpler mind can understand.  What if that simple mind had the capability to translate those instructions into actionable programming?  What if I just had to state what I wanted to do?  ”Ensure that all traffic on Link A is given priority treatment, except if it is for the backup server.”  The system can create API calls to program the access control lists and storage arrays to take care of all that without needing to fire up a CLI session or a browser.

This is going to require a lot of programming on the front end.  The management system needs to be able to interpret keywords in the instruction set to pick out the right execution items.  Then the system needs to know where to dispatch the commands to make them land on the right APIs.  In the case of the above example, the system would need to know to send two different sets of commands – one to the storage array to provide metered access during backup windows and another set to the networking gear to ensure the right QoS policies were enforced during the given time window.  Oh, and you’re going to want to have the system go back and clean up those ACLs when it’s time for production hours to start again.

---

Tom’s Take

This isn’t going to be easy by any means.  But adding all the value into the front end management system means that any other system attached on the backend via API is going to benefit.  I’d rather be doing the work in the right places as opposed to spending all our time on the backend and neglecting the interface to the whole system.  Engineers are notorious for writing terrible GUIs.  Let’s take the time to abstract the bad things away and instead make something that’s really going to change the way we program systems.

The report shows strong evidence that very specific prefixes were hijacked and diverted to countries where legal jurisdiction could be reasonably assumed to be weak. Renesys does not say which prefixes were hijacked but consider hijacking a corporate PI space and capturing a copy of all the email to & from a large company (email […]

There are very few technologies in that data center that have had as significant of an impact of VMware’s vMotion. It allowed us to de-couple operating system and server operations. We could maintain, update, and upgrade the underlying compute layer without disturbing the VMs they ran on. We can write web applications in the same […]