All Posts in “Security”

To patch or not to patch

It’s a funny thing, having worked as a compliance and security admin I had the ominous job of running scans on network environments and providing their local IT teams with lists of vulnerabilities. I got that gig after years of working on the other side and actually taking the vulnerability assessment reports and implementing the security guidance, then writing PoAMs (Plans of Action or Mitigation) for when the patch couldn’t be applied.

This is an on going struggle for many institutions that actually care about security, which sadly I am told isn’t as many as you would like to think. The thing with patches though, they come from developers and not all patches are equal. Let’s just look at the biggest culprit of patch releases in the enterprise, Microsoft, which is so well known for their patch release cycle that “Patch Tuesday” even gets branding. But if you look closely at what patches are being released its a mix of security patches, driver updates, and new feature releases.

Some of which actually change your security posture for the worse. So how do you determine what is really a necessary patch and what is just a new Silverlight plug-in for Cindy’s cat videos? Well the basics are ensure your Anti-virus is always patched and the vulnerability library is renewed as frequently as possible. Secondly know that the more often you use an application the higher it’s potential threat to your environment can be, so key or critical applications should always have their security patches applied.

Critical or High importance flagged patches are a no brainer, beyond that risk assessment tools can help to determine the right sets of mandatory patches and which can be mitigated. Remember when I talked about managing risk? Well patching is just another log in that fire, you have to determine if an application vulnerability that can’t be patched due to loss of productivity or capability is worth the risk you run if it opens the system up to an exploit. Well design systems and networks will have a series of gates and checks to ensure that these open windows don’t get to far out of hand. But you as an admin must be vigilant to ensure you have closed as many of them as possible leaving only the bare minimum open.

I get this is common sense, and maybe  you work through patching and mitigation all the time. But the ease of exploiting unpatched systems is something I have witnessed first hand and the aftermath can be devastating, and even resume generating. Contrarily I have also patched systems with “critical updates” that cause system wide outages. There was a 16-bit code bug running around in 2002-2003 time frame and we had to kick off a registry change to disable 16-bit code execution. This was one of about 100 IA vulnerabilities released for the month, and I scripted most of them to be executed through a javascript that was set with a restart. The patch window hits, we restart our servers and everything comes up fine, I validated the necessary changes had been implemented and we are in business. I go home late and get some sleep.

Next day I am back in the office and everything is humming along fine until noon. Then our environment goes belly up. We can’t figure out why, initially so I start combing through logs with the team. Turns out there was a licensing error for one of our core applications. We had a couple hundred thousand users hitting this app every day so this was not a good situation. We started looking at all of the changes that were implemented. Everything had run fine in the test environment so we start replicating our production back to test. Turns out the core app uses 16-bit code to authenticate the license with the client servers. A quick script and push later and boom we are back in business, and I am writing an after action and a PoAM. We lost a day of production, which stunk because of a patch that tested fine because our config was slightly different on the test than on the prod side.

IA came back and told me that the patch was needed and we would have to fix the core code, which was commercially obtained. A ticket was put in with the vendor and I left 2 years later without ever seeing a license code update release. But it’s just one example of how not every patch, even a critical one need be applied.

Want to share your patch nightmare?

Blog Security

Son of a … ok talking security again I guess. I started getting worried because it appeared that my blog perma-links were redirecting. Clearly a hi-jack or so I thought, so enter reading up on word press security.

First use a hosting service for this blog, I do not try to run it out of my house, because that’s cray cray. It’s a basic WP site, with a custom theme, no biggie. I have a couple of plugins for managing comments and tracking statistics. When I found this permalink issue out I decided it was time to dig in on what security for WP was available. I settled on BulletProof Security plug-in by AITpro software.

My concern was around the ease of getting into the admin console and changing the .htaccess file. BulletProof allows you to lock the .htaccess and even replace it with a secure .htaccess. It has other benefits as well such as providing caching, securing login’s and PHP source code. Honestly it was really easy to configure and I have been happy with the locked down results.

But ironically it didn’t solve my permalink problem. What the hell? I’ve locked down everything on the blog why are the permalink’s redirecting. Then I found a link to an ancient manuscript, ok it was just an old wordpress support forum, that mentioned that sometimes permalinks get hosed up and simply changing to a custom permalink structure will fix the issue. Boom its fixed like FM from the sky.

Well then … sometimes it’s the simple stuff I suppose security made relatively easy. What are your tips and tricks for securing your blog?

NSX and Securing Multi-tenancy Policy

Discussing security and multi-tenant cloud environments both public and private consumes so much of my time. This time though let’s get into how micro-segmentation and the coolness that is VMware’s NSX. At least it’s a different angle so I don’t get bored writing about it J.

From an Operations perspective I want my environment to be as simple as possible, and consistently deployed. As I add tenants to my infrastructure this becomes even more imperative, right? Think about it this way if you are a car mechanic you can be a mechanic at a dealership and see similar vehicles all day let’s say Audi for example. Or you can work at a boutique shop and see every kind of vehicle. The benefit of the dealership is they can roll through more cars per day on average than a boutique shop because all of the cars are roughly the same lay out and use the same size nuts and bolts etc. Essentially the mechanic has the blueprints for the cars they are servicing.

In Operations if we have the same management stack across everything we have the blueprint and it’s simple. Enter the security team …

In security simple is also important, BUT, we also need to have gates and gate keepers. In a multi-tenant environment regardless of the M&O be it vRA or XStream we need to ensure that one tenant’s data can not corrupt another’s. Equally important we have to make sure that if\when a tenant is compromised (get your head out of the gutter) that compromise doesn’t trickle to my core infrastructure or to other tenants.

“Obvi Mike!”

Yeah I get it but that’s not as obvious to everyone, because think about what this means. First it means you need separate attack surfaces. In VMware terms this means two vCenters, one to manage the core infrastructure and another to manage the tenant infrastructure. This ensures that in the event that you get attacked and your tenants get PWND you still have a layer between the tenants and the core infrastructure that manages it all. It also means you have a stretched network for management. This creates a vulnerability in and of itself because it stretches across the core infrastructure and tenant architectures. However, we do not provide access to this management network from tenants directly rather we create jump servers or dual homed transitional boxes.

In NSX terms: We have controllers spanned across the environment, each tenant has it’s own NSX Edge and it’s own set of VXLANs. The Management network is linked off of whatever the M&O service is and provides visibility into the tenant for management but does not allow tenant traffic to jump across to other tenants. This is done by creating a distributed Firewall with ACLs for specific traffic on one of the M&O VM’s vNICs. It would be possible to also NAT this traffic across the Edge Gateway from tenant to management for the necessary monitoring and orchestration traffic.

If that all sounds good, then your next question is how does this work? Rather than reinventing the wheel I would recommend you read Matt Berry & Anthony Burke’s post on zero trust solution architecture with NSX. I think it best captures the right way to segment environments for security.

What do you think? Is this a little to security crazy or is this the way you would architect your multi-tenant environments?