During his Network Programmability 101 webinar Matt Oswalt described three phases of network programmability. The first level in the pyramid of programmable awesomeness (his words, not mine) is described in today’s video.
During his Network Programmability 101 webinar Matt Oswalt described three phases of network programmability. The first level in the pyramid of programmable awesomeness (his words, not mine) is described in today’s video.
I’ve been researching four different and distinct types of networking in the last few weeks. I’m finding that the cognition required to jump between technologies is making my head hurt. Here is a summary of four technology areas that interest me this week. Optical Networking As part of research project I have been getting deep […]
The discussion in the comments to my LAG versus ECMP post took a totally unexpected turn when someone mentioned BFD failure detection over port channels (link aggregation groups – LAGs).
What’s the big deal?Read more ...
Want to know what the difference between Virtual Chassis and Virtual Chassis Fabric is? How Local Link Bias works? How ISSU on QFX 5100 works even though the box doesn’t have two supervisor boards? You’ll find answers to all these questions in new videos describing Juniper data center switches.
Been researching HTTP2 protocol on the basis that is will, more or less, be the dominant protocol on the Internet and everywhere else. Aside from the sense of excitement I get from looking at solving old problems, HTTP2 is a huge change for networking and this site has the best explanation I’ve found so far. Check […]
People talking about long-distance workload mobility and cloudbursting often forget the physical reality documented in the fallacies of distributed computing. Today we’ll focus on bandwidth, in a follow-up blog post we’ll deal with its ugly cousin latency.
TL&DR summary: If you plan to spread application components across the network without understanding their network requirements, you’ll get the results you deserve.Read more ...
Recently I've been on a search for a 'better' font to use in terminals. In an unrelated coincidence, I learned about anti-aliasing, I still don't understand it but it makes a difference.
“Sometimes my head is a bit of an idiot” is something my daughter might say and that happens to me too, if that time is today and this article, let me know. If you don’t get the Cabbage Patch reference and its juxtaposition to automation, see here. I’ve tried to avoid sarcasm (and arrogance) but have […]
He's worked in the IT industry for over 15 years in a variety of roles, predominantly in data centre environments. Working with switches and routers pretty much from the start he now also has a thirst for application delivery, SDN, virtualisation and related products and technologies. He's published a number of F5 Networks related books and is a regular contributor at DevCentral.
On Cisco IOS, this is a very useful command "show tcp vty xx" to show TCP statistics of the VTY session. If you think your terminal is running slow because of packet loss or delay then this command will provide visibility. The other cause is the CPU/Memory running slow if you don't see any errors on the TCP (as you can see below).
Several SDN solutions that coexist with the traditional control- and data planes instead of ripping them out and replacing them with the new awesomesauce use BGP to modify the network’s forwarding behavior.
Enjoy the show (this time in video format).
Very, very funny quote in the Pew Research Report: How could people benefit from a gigabit network? One expert in this study, David Weinberger, a senior researcher at Harvard’s Berkman Center for Internet & Society, predicted, “There will be full, always-on, 360-degree environmental awareness, a semantic overlay on the real world, and full-presence massive open […]
The post Killer Apps in the Gigabit Age | Pew Research Center’s Internet & American Life Project appeared first on EtherealMind.
SRX platforms complete the Trifecta
Earlier this month, many of the world’s biggest cloud-service providers quietly cooperated to update the open-source Xen hypervisor software. What wasn’t publicly revealed until after the update was safely completed, however, was that it actually was a carefully coordinated operation intended to head off a major security breach, as identified in the Xen patch advisory.
Before we get into the how, let’s talk about the why. According to the CIDR Report, the global IPv4 routing table sits at about 525,000 routes, it has doubled in size since mid 2008 and continues to press upwards at an accelerated rate. This momentum, which in my estimate started around 2006, will most likely never slow down. As network engineers, what are we to do? Sure, memory is as plentiful as we could ask for, but what of TCAM? On certain platforms, like the 7600/6500 on the Sup720 and even some of the ASR1ks we have already surpassed the limits of what they can handle (~512k routes in the FIB). While it is possible to increase the TCAM available for routing information, there are other solutions that don’t include replacing hardware just yet.
As far as I know, adjusting TCAM partitioning on the ASR1000 is not possible at this time.
Before I get too deep into this, I should clarify as many of you (yes, I’m looking at you Fry) are asking yourselves why is an ISP running BGP on a 6500… Many of my customers are small ISPs or data centers that have little to no networking experience. They are the small guys attempting to provide high speed service to rural areas that truly need it. Most of these guys are 3-4 person shops that have a ton of people wearing multiple hats, and after spending the last decade working with them, I have to respect that. /soapbox
My favorite solution to this problem has been to filter out routes that have long AS Paths. This works particularly well if you’re receiving full tables + a default from your upstream providers. My thoughts have always been, less ensure path optimization for very short AS Paths, and for anything above 3 networks… who cares!? The example below uses AS path filtering and local preference to always ensure that we’re sending traffic, to destinations 3 networks or less away, out the best path that we have.
ip as-path access-list 100 permit ^[0-9]*$ ip as-path access-list 200 permit ^[0-9]*_[0-9]*$ ip as-path access-list 300 permit ^[0-9]*_[0-9]*_[0-9]*$ ! ip prefix-list any seq 5 permit 0.0.0.0/0 le 32 ! route-map ebgp-in permit 10 match as-path 100 set local-preference 193 ! route-map ebgp-in permit 20 match as-path 200 set local-preference 192 ! route-map ebgp-in permit 20 match as-path 300 set local-preference 191 ! route-map ebgp-in deny 99 match ip address any ! router bgp 65100 bgp log-neighbor-changes neighbor 18.104.22.168 remote-as 65011 neighbor 22.214.171.124 route-map ebgp-in in neighbor 126.96.36.199 remote-as 65022 neighbor 188.8.131.52 route-map ebgp-in in !
As you can see, we’re using a route-map to filter updates from our peer. Inside our first statement we’re using a match statement on AS-Path ACL 100, which has a regular expression to match updates with a single AS number in the AS-Path. Our set statement is used to modify the local-preference on those routes well above the default 100. While the BGP best path selection algorithm would certainly prefer these routes according to their AS-Path, personally I like overriding all local-preference settings throughout my configs to suit the needs of the business. I also typically set BGP Communities on these prefixes to aide in identification of applied policy. But I digress. This continues on in the next statement, matching an AS-Path length of 2, and setting a slightly lower local-preference. And again in the third statement, until we reach statement 99, which is configured to deny any other routes from being learned.
In addition to the routing table limitations, the sheer amount of load that running BGP adds to the CPU in your 6500/7600 series is going to be the last nail in the coffin, and I completely understand and agree. And because I understand many of you that are still on those platforms need an affordable option, I have good news for you. The ASR 9001 has a scaled down 60gbps build that comes in at a rather reasonable price, which should be rather affordable after you factor in trade-in value on your legacy platform. Not only will the ASR 9k completely blow the doors off your 7600 right out of the box, but it should last you a rather long time, as it is scalable to 120gbps. As for it’s routing abilities, it shares the same IOS-XR platform as the larger ASR 9ks, and has plenty of memory to support millions of routes.
As a Chief Information Security Officer, I get a lot of questions about the cyber security threats and what worries me most. I field questions about Anonymous, geo-political hackers, cyber-extortionists, malware, and the like.
I was listening to the Packet Pushers show #203 – an interesting high-level discussion of policies (if you happen to be interested in those things) – and unavoidably someone had to mention how the networking is all broken because different devices implement the same functionality in different ways and use different CLI/API syntax.Read more ...
It’s Microsoft Patch Tuesday! In the October edition there are 8 updates; three are marked "Critical" and five are rated "Important". A total of 24 vulnerabilities were fixed over 8 bulletins this month. One of the Critical update MS14-056 is an all version Internet Explorer (IE 6 to 11) patch. This single update resolves 14 CVE's (Common Vulnerability and Exposure).
It’s been far too long since the last MindshaRE post, so I decided to share a technique I’ve been playing around with to pull C2 and other configuration information out of malware that does not store all of its configuration information in a set structure or in the resource section (for a nice set of publicly available decoders check out KevTheHermit’s RATDecoders repository on GitHub). Being able to statically extract this information becomes important in the event that the malware does not run properly in your sandbox, the C2s are down or you don’t have the time / sandbox bandwidth to manually run and extract the information from network indicators.
To find C2 info, one could always just extract all hostname-/IP-/URI-/URL-like elements via string regex matching, but it’s entirely possible to end up false positives or in some cases multiple hostname and URI combinations and potentially mismatch the information. In addition to that issue, there are known families of malware that will include benign or junk hostnames in their disassembly that may never get referenced or only referenced to make false phone-homes. Manually locating references and then disassembling using a disassembler (in my case, Capstone Engine) can help to verify that you have found the correct information and avoid any of the junk inserted to throw your analysis off.
For those not familiar, Capstone Engine is a disassembler written by Nguyen Anh Quynh that was first released in 2013. The engine has seen a significant amount of development in that short amount of time and has a good track record of handling some tricky disassembly. Most importantly, it supports most popular programming languages, including Python – my current programming language of choice. One complaint I have with using an on-the-fly disassembler is the lack of symbols, but that can be gotten around by taking the list of imports and addresses from pefile and then checking any memory references against it. All of the PoCs presented expect an image base of 0×400000, but for any production use the actual image base should be parsed out and replaced.
Backoff is a recently discovered PoS malware family. I noticed that many of the times the malware was sandboxed, it would not communicate with a C2, but I could see the C2 info in plain-text in the binary or other times when the C2 was down.
In an attempt to “correctly” locate the C2 information and utilize some Capstone-fu, I crafted a function that first locates hostname- or IP-like strings in the binary, looks for a “mov [register+offset]/<addr> addr” pattern, and then uses capstone to disassemble to obtain the other configuration elements.
This ends up being useful, since the argument order is not necessarily the same. This doesn’t work for all versions, but does work for most – I have encountered a number that are using a VisualBasic injector or are using an array structure to store the config so the below code will not work. This can be coupled with another piece of code that searches for version-like strings and then disassembles to find the additional campaign name attached to the binary. The code should check to see if a) host,port, URI are defined after the loop and b) if the number of mov instructions encountered before the call was 3. The number of mov’s ends up being important since my code starts with the hostname and the arguments are not always encountered in the same order. If the mov’s are less than 3, then I jump back the appropriate number of mov’s via regex search and then walk the disassembly again to see if I encounter the expected configuration data. This will also help find the backup domains and URLs that are embedded in the malware that may not be seen during a sandbox run even if there is successful communication to the C2. The code is quick and dirty and can easily be improved by validating some common instructions seen in between, but is presented as-is for this example:
md = Cs(CS_ARCH_X86, CS_MODE_32) md.detail = True movs = 0 host = None uri = None port = None for insn in md.disasm(code, 0x1000): if insn.mnemonic == 'mov': movs += 1 if insn.operands.type == X86_OP_IMM: v = insn.operands.value.imm.real if v < 65536: port = v else: x = self.get_string(file,v-0x400000) if URI_REGEX.match(x): uri = x elif DOMAIN_REGEX.match(x): host = x elif IP_REGEX.match(x): host = x elif insn.mnemonic == 'call': break if movs == 3: break
Alina is a PoS malware family that has been around for awhile. Similar to Backoff, I noticed that many of the sandbox runs did not successfully communicate with the malware when the configuration was viewable.
I used a similar process to what I did with Backoff to first locate potential C2 candidates and then search for XREFs and disassemble with capstone. Many times the C2 is stored is pushed onto the stack followed by instructions setting local variables and then a subroutine call. Prior to the push of the C2 and the URI, there is another push that represents the length of the string and can also be used to validate the sequence. Once again, this is a great place to utilize capstone to make sure that anything that is extracted matches up with what is desired.
This sequence of pushes and calls always seems to be preceded by a call to InitializeCriticalSection, so I first look for that, using a dict built from loading the binary into pefile to get at the import table.. The order that the hostname and the c2 occur in the binary can be flip-flopped, so I allow for that. I do make sure that the next push after the strlen is a string The code can be extended further to validate that the strlen matches the string I extract from the binary, but this is just a PoC
for i in md.disasm(CODE, push_len_addr): if instr_cnt == 0: # check for InitializeCriticalSection if i.mnemonic == 'call' and \ impts.get(i.operands.mem.disp,'') == 'InitializeCriticalSection': print "On the right track..." else: break elif i.mnemonic == 'push' and i.operands.imm < 0x100: strlen = i.operands.imm str_instr = instr_cnt + 1 print "Found the strlen push",i.mnemonic,i.op_str elif strlen and str_instr == instr_cnt and i.mnemonic == 'push': addr = i.operands.imm if addr == 0x400000+file.find(s): print 'found hostname push' hostname = get_string(file,addr-0x400000) print hostname else: uri = get_string(file,addr-0x400000) if URI_REGEX.match(uri): print uri instr_cnt += 1
My last example involves a more complex example. Drive stores its most interesting strings in an encrypted format and does not decrypt all those strings in the same function (for more information see my previous blog post here), instead scattering the calls throughout the binary. In this example, I use the encrypted install name – it always starts with the same characters – to help us locate the decryption function. The decryption function is the function called right after the call that Xrefs the encrypted install name.
With the address of the decryption function known, I use the “k=” string used in the phone-home to help locate the network communication function. This function is where the C2 information is first decrypted and the C2 and the URI are the first two things decrypted in this function. The code can then be walked further down to locate the C2 port, but that code is not shown here.
Here’s the first piece of code used to locate the decryption function:
mov_addr = '\xb8'+struct.pack("<I",0x400000+file.find(s)) instr_addr = 0x400000+file.find(mov_addr) if instr_addr <= 0x400000: mov_addr = '\xba'+struct.pack("<I",0x400000+file.find(s)) instr_addr = 0x400000+file.find(mov_addr) # looks for PUSH EBP; MOV EBP, ESP func_start = file[:instr_addr-0x400000].rfind('\x55\x8b\xec') code = file[func_start:func_start+0x200] md = Cs(CS_ARCH_X86, CS_MODE_32) md.detail = True decrypt_func_next = False calls = 0 for i in md.disasm(code, func_start+0x400000): # looking for mov eax, if i.mnemonic == 'mov' and len(i.operands) == 2 \ and i.operands.type == X86_OP_REG and i.operands.reg == X86_REG_EAX \ and i.operands.type == X86_OP_IMM and i.operands.imm >= 0x400000 \ and i.operands.imm <= 0x500000: d = decrypt_drive(get_string(file,i.operands.imm-0x400000)) # validate that this is indeed the install name if d.endswith('.exe'): config['install_name'] = d decrypt_func_next = True # check for the next call after the install name call elif decrypt_func_next and 'install_name' in config \ and i.mnemonic == 'call' and calls == 1: config['decrypt_func'] = i.operands.imm break elif 'install_name' in config and i.mnemonic == 'call': calls += 1
Now that the decryption function has been located, the desired C2 information can now be located.
mov_inst = '\xba'+struct.pack("<I",0x400000+file.find('k=')) mov_k_addr = 0x400000+file.find(mov_inst) # look for PUSH EBP; MOV EBP, ESP func_start = file[:instr_addr-0x400000].rfind('\x55\x8b\xec') code = file[func_start:func_start+0x200] md = Cs(CS_ARCH_X86, CS_MODE_32) md.detail = True calls = 0 d = None for i in md.disasm(code, func_start + 0x400000): # look for mov edx, <addr> if i.mnemonic == 'mov' and len(i.operands) == 2 \ and i.operands.type == X86_OP_REG and i.operands.reg == X86_REG_EDX \ and i.operands.type == X86_OP_IMM and i.operands.imm >= 0x400000 \ and i.operands.imm <= 0x500000: d = get_string(file,i.operands.imm-0x400000) # if call decrypt_func, then decrypt(d) elif i.mnemonic == 'call' and i.operands.imm == config['decrypt_func'] and d: # first call is the c2 host/ip if calls == 0: config['host'] = decrypt_drive(d) d = None calls += 1 # 2nd call is the URI elif calls == 1: config['uri'] = decrypt_drive(d) d = None break
Capstone is a useful tool to have in your toolbox and hopefully the PoC code presented in this post will aid others in the future. For my own future work, I plan to tighten up the code presented and work on getting code for other interesting malware families into something that will be suitable to push out for public release.
It seems as though the entire tech world is splitting up. HP announced they are splitting the Personal Systems Group into HP, Inc and the rest of the Enterprise group in HP Enterprise. Symantec is forming Veritas into a separate company as it focuses on security and leaves the backup and storage pieces to the new group. IBM completed the sale of their x86 server business to Lenovo. There are calls for EMC and Cisco to split as well. It’s like the entire tech world is breaking up right before the prom.
The Great Tech Reaving is a logical conclusion to the acquisition rush that has been going on throughout the industry for the past few years. Companies have been amassing smaller companies like trading cards. Some of the acquisitions have been strategic. Buying a company that focuses on a line of work similar to the one you are working on makes a lot of sense. For instance, EMC buying XtremIO to help bolster flash storage.
Other acquisitions look a bit strange. Cisco buying Flip Video. Yahoo buying Tumblr. There’s always talk around these left field mergers. Is the CEO looking for synergy? Is there a hidden play that we’re unaware of? Sometimes that kind of thinking pays off. Other times you end up with Zimbra. More often than not, the company ends up writing down the assets of the acquired company and taking very little from the purchase. Maybe not as big as the Autonomy write down, but even getting rid of Flip can make waves.
It makes a person wonder what the point of an acquisition is if it’s just going to wind up being an accounting charge later. Is it a tax shelter? A way to use up outstanding cash? Maybe even a way to buy a particularly good developer and fold them into your organization to keep them out of a competitor’s hands? The reasons are myriad but it appears that the fever is dying down. And that might end up hurting innovation in the long term.
This Is Not An Exit Strategy
Think about the startup out there making a hot new technology. They had their heart set on getting bought by a bigger company in the market. Now, they just watched that company split off half of their business into a new company. Cash is hard to find for a new acquisition. Now the startup has to find a different way to monetize things. Should we redouble our efforts to market the product? Get new investors? Go public?
I’ve said before that pinning your hopes on getting purchased isn’t the best way to run a business. It’s like betting all your hopes on getting the winning numbers in the lottery. It might happen, but the odds are against it. Perhaps the end result of a market full of split companies will be a reevaluation of the idea of an exit strategy. Rather than building a business for the sole purpose of being bought entrepreneurs will start building businesses to make products and sell them. It’s a radical idea, but not so radical as to be unbelievable. Just ask Hewlett and Packard. Or Jobs and Wozniak. Or anyone else that didn’t have an exit strategy instead of a business plan.
Companies can be too big. IBM has sold off most of what made it IBM. Symantec and HP are in the process. The next domino to fall will be EMC. Then Cisco. After that, the landscape will look much different. But in a good way. It’s like a stock split. The same amount of knowledge is out there. It’s just held differently. That’s good for the industry because it forces the status quo to change. New alliances, new partnerships, and new synergies can be found by upsetting the apple cart now and then.
Since the dawn of time people have skirted best practice and banged together networks, putting the proverbial square peg in the esoteric round hole. For example, new vendor XYZ’s solution has brought in new requirements for deployment. While it may seem easier for to throw together a new firewall, a switch, and maybe some additional routes, and of course Tom‘s favorite… NAT — but where does it stop!? As you continue to pile layer upon layer into your uninspired network design you will soon realize that your “beautiful network” has become the ugly duckling that you need help fixing.
That leads me to my first point. Complex systems are expensive, not only in CAPEX, but in OPEX. When you design and build a network, you have to ensure that you are not building something that no one else has dreamed up, or else your problems will also be unique. And without the additional money to hire top tier engineers, you could be short staffed, or worse yet, facing the problem on your own. The more complex your network becomes, the more likely it is to fail. As I’m often quoted as saying, “The complexity required for robustness, often goes again robustness…”, and those systems are often replaced.
As you build upon your complex design, your entire network, once agile because ridged and unable to adapt to changes. While you have to learn to understand that no single design can last for ever, the simpler designs tend to be more flexible and adaptable into your ever changing needs. You have to remember that your network is not just there to serve the end users, or systems. Your network is in the middle of everything your company does and has to be able to mold itself to fit the businesses ecosystem.
Design flexibility starts with simplicity, but also requires adding complexity when it comes to redundancy. Without redundancy upgrades and maintenance impact core services, those impacts could force bad policy into place making it impossible for you to do you maintenance. I’ve worked on far too many enterprise networks that suffer from lack up maintenance windows, which only ends up making the problems worse.
Last, but certainly least I want to talk about testing. One of the biggest things I learned at my last job is that no matter how meticulously you designed your system, no matter how much redundancy you think you have, all of that has to be tested on a regular basis. Changes happen, and it always seems that no matter how much documentation you have, something is going to be left out. The only thing that is going to find these problems is real life, end to end testing… LDAP connections for your VPN, DNS issues, vendor configuration issues, everything that is critical for your business to function needs to be well documented and tested.
Following the breakups of IBM and HP as they divest the low profit divisions and EMC under a some pressure to disband the Federation, the same question is often raised about Cisco but what could go ?
The moment personal photos of Jennifer Lawrence, Kim Kardashian and other celebrities were leaked from iCloud it became global breaking news and suddenly everyone had questions and opinions about cloud security.
This was the first year that I got to attend VMworld as a member of the Juniper family ( this was my fourth VMworld ). It was a great experience, we had our first lab in the Hands-on Lab which I personally think was a success and of course we had a booth. We received a lot of complements on the documentation for the lab and how it explained all the facets of the product. I had people fighting ( well not literally ) for the long sleeve shirts that we distributed to everyone who took the lab ( check it out below )
It gave us a lot of visibility into our virtual security solutions and how they play in your VMware environment. The great thing, the fun isn't over…
My “Was it bufferbloat?” blog post generated an unexpected amount of responses, most of them focusing on a side note saying “it looks like there really are service providers out there that are clueless enough to reorder packets within a TCP session”. Let’s walk through them.Read more ...
Over at CircleID, Geoff Huston has a long’ish article on Title II regulation of the Internet, and the ideals of “net neutrality.” The reasoning is tight and strong — his conclusion a simple one: At its heart, the Internet access business really is a common carrier business. So my advice to the FCC is to […]
In recent network designs, the big, hot and heavy chassis switch has become the last option for a number of reasons. Switch Performance and Capacity. Port Density In the past, the most common decision for buying a chassis has been port density. A chassis backplane provides a high speed connection for the line cards to […]
Michael Church wrote an interesting answer on Quora, describing a logarithmic scale of programming skills and (even more importantly) hints to follow to get from n00b into the top N% (for some small value of N):
Replace “programmer” with “networking engineer” and read the whole answer ;)