It’s a funny thing, having worked as a compliance and security admin I had the ominous job of running scans on network environments and providing their local IT teams with lists of vulnerabilities. I got that gig after years of working on the other side and actually taking the vulnerability assessment reports and implementing the security guidance, then writing PoAMs (Plans of Action or Mitigation) for when the patch couldn’t be applied.

This is an on going struggle for many institutions that actually care about security, which sadly I am told isn’t as many as you would like to think. The thing with patches though, they come from developers and not all patches are equal. Let’s just look at the biggest culprit of patch releases in the enterprise, Microsoft, which is so well known for their patch release cycle that “Patch Tuesday” even gets branding. But if you look closely at what patches are being released its a mix of security patches, driver updates, and new feature releases.

Some of which actually change your security posture for the worse. So how do you determine what is really a necessary patch and what is just a new Silverlight plug-in for Cindy’s cat videos? Well the basics are ensure your Anti-virus is always patched and the vulnerability library is renewed as frequently as possible. Secondly know that the more often you use an application the higher it’s potential threat to your environment can be, so key or critical applications should always have their security patches applied.

Critical or High importance flagged patches are a no brainer, beyond that risk assessment tools can help to determine the right sets of mandatory patches and which can be mitigated. Remember when I talked about managing risk? Well patching is just another log in that fire, you have to determine if an application vulnerability that can’t be patched due to loss of productivity or capability is worth the risk you run if it opens the system up to an exploit. Well design systems and networks will have a series of gates and checks to ensure that these open windows don’t get to far out of hand. But you as an admin must be vigilant to ensure you have closed as many of them as possible leaving only the bare minimum open.

I get this is common sense, and maybe  you work through patching and mitigation all the time. But the ease of exploiting unpatched systems is something I have witnessed first hand and the aftermath can be devastating, and even resume generating. Contrarily I have also patched systems with “critical updates” that cause system wide outages. There was a 16-bit code bug running around in 2002-2003 time frame and we had to kick off a registry change to disable 16-bit code execution. This was one of about 100 IA vulnerabilities released for the month, and I scripted most of them to be executed through a javascript that was set with a restart. The patch window hits, we restart our servers and everything comes up fine, I validated the necessary changes had been implemented and we are in business. I go home late and get some sleep.

Next day I am back in the office and everything is humming along fine until noon. Then our environment goes belly up. We can’t figure out why, initially so I start combing through logs with the team. Turns out there was a licensing error for one of our core applications. We had a couple hundred thousand users hitting this app every day so this was not a good situation. We started looking at all of the changes that were implemented. Everything had run fine in the test environment so we start replicating our production back to test. Turns out the core app uses 16-bit code to authenticate the license with the client servers. A quick script and push later and boom we are back in business, and I am writing an after action and a PoAM. We lost a day of production, which stunk because of a patch that tested fine because our config was slightly different on the test than on the prod side.

IA came back and told me that the patch was needed and we would have to fix the core code, which was commercially obtained. A ticket was put in with the vendor and I left 2 years later without ever seeing a license code update release. But it’s just one example of how not every patch, even a critical one need be applied.

Want to share your patch nightmare?