In-Depth

We Have a Software Quality Problem

"I'm an MSP, providing IT services to a handful of small businesses, and the software supply chain brings a lot of risks to my clients."

• By Paul Schnackenburg
• 08/26/2024

At the recent Black Hat conference, Jen Easterly, head of the Cybersecurity and Infrastructure Security Agency (CISA), said:

"We don't have a cybersecurity problem. We have a software quality problem."

This is true, and it also got me thinking about something that's been bothering me for quite some time. I'm an MSP, providing IT services to a handful of small businesses, and the software supply chain brings a lot of risks to my clients.

In this article I'll outline the risks we all take, enterprise and SMB alike when we introduce software applications and hardware (firewalls or networking gear) into our infrastructure, how Software-as-a-Service (SaaS) changes the calculation of the risk, why cloud computing is both a boon and a curse, and make some humble suggestions as to steps we can take as an industry to improve our resiliency. Throughout I'll use several practical examples based on my clients.

Every App Is an Added Risk
It's interesting how subconscious thoughts and attitudes change, and then one day you suddenly realize how much your stance has changed on a particular issue, when you're asked to make a decision. In my case this came a few months ago when I got an inquiry from one of my clients. They're a small, independent school here on the Sunshine Coast of Australia, about 25 staff and 90 students, covering year 1 to 12. Every student from year 4 onwards has a Windows 11 laptop, with the young ones using shared iPads, and every teacher also has their own laptop. A long time ago I decided that both staff and students were only given ordinary user accounts, not local administrators on their devices, providing alternative administrative accounts for two staff members to assist with the occasional task that required local admin access.

A teacher reached out and asked me about an application that he wanted to deploy to all laptops in his class. I investigated the app and its requirements and instead suggested an alternative web application, not because there was anything suspicious about the desktop app, but simply because introducing an additional application adds risk. This isn't news to anyone, particularly those of you who work in large, especially heavily regulated businesses, you have processes to quantify this risk. But in SMB land, we don't have that luxury, nor the in-house / outsourcing resources to test software to see how bad it actually is. The risks in an educational application like this are many. The code could be malicious or tracking student activity in ways that aren't appropriate. Or the code could be so poor that it's opening an attack vector into our infrastructure. The application could also be fine today, but sold to another developer next week, who has malicious intentions.

Or the developer might be totally above board and trying to do the right thing, but their application is relying on many open-source components (which in turn relies on other OSS building blocks), any one of which could turn out to be malicious or vulnerable. Any of these risks this could get my client pwned. So, my new mantra is: Any application introduced into the environment must justify through educational or business requirements their existence before being deployed.

Note that SMBs aren't the only ones facing these challenges. Larger organizations might have an even larger set of challenges, as shown in the graphic below.

The Analyst Approach
For as long as I can remember, one way to narrow down which application / service to consider when a business has an IT need is relying on Gartner, Forrester and other analysts. But their approach is also flawed -- they're focusing on "leaders" who can execute on new features and innovate. When in fact, the most important feature of any software or cloud service that we introduce in business isn't the shiny new features -- it's reliability and good support. Because once the salespeople have done their job and the POC has been signed off, the act of operationalizing the application falls to the IT department or outsourced MSP. And no Gartner magic quadrant is going to help with that process, which is actually where the business realizes the ROI on the investment.

I do like Peerspot as an alternative to the analysts, it gathers end user reviews of applications and services and is a good place to start your investigation if you're evaluating an application.

Cybersecurity Software -- Worst of Them All?
Business software might have quality and supportability issues, but at least cybersecurity software is better -- right? Right?

Well, it turns out, not so much. The number of businesses getting pwned because their "secure" VPN solution isn't, or their firewall has an issue is staggering (again, looking mostly at the SMB space now). Turns out writing secure security software is hard, taking a lot of resources, time and focus, and this doesn't equate to shiny new features being added quickly -- and slipping in next year's Gartner ranking isn't an option.

For what it's worth, I've settled on Checkpoint for physical client firewalls, and they've had one single, high-impact flaw that needed prompt patching in the last 7-plus years.

And the mantra of patching to improve security? Turns out that for most software, statistically speaking, updating it introduces more flaws than in the older versions. The exception to this is browsers and major OSes -- the organizations behind these dedicate the resources to fix vulnerabilities and improve code quality and cyber resiliency overall. But for everything else, patching adds more vulnerabilities than it fixes. Obviously, if there are publicly known flaws that can be exploited, patching anything is still important; just don't assume that this is going to make your infrastructure overall more secure in the long term. And this brings us back to where we started -- we have a software quality problem.

I listened to a recent podcast interview with Michael Howard of the "Writing Secure Code" book fame. In it he mentioned that the developer graduates that they're hiring really don't know how to write secure software. It's just not (enough) of a part of the curriculum, which means tech companies have to train them on this when they're hired. This is equivalent to training engineers at university to build bridges but not worrying too much about the security aspects of making sure the bridge doesn't collapse under heavy load. This isn't a reflection on the many developers I know who take security very seriously and are doing their best to improve whatever code they're working on, but we need a fundamental shift in how we educate developers (which frankly involves training the lecturers, because it seems they don't as a rule know how to write secure code either).

And those blinky-light hardware firewalls and VPN connection servers -- they're often not quite as secure as the glossy brochure depicts them to be. And some companies (looking at Fortinet here) expressly block any investigation into the quality of the software running on their devices, something that doesn't really inspire confidence.

Finally, another big challenge with cybersecurity software is point solutions -- individual services or applications designed to solve one particular problem. The issue is that this offloads the integration of these disparate solutions onto users. As a defender I need all of my defensive solutions to talk to each other seamlessly, because attackers don't just compromise one application, they move like water into any cracks, and I need visibility across all of it. My choice for my clients is Microsoft Defender XDR, combined with Microsoft Sentinel.

Applications vs. Cloud Services
The result of the above reasoning is to avoid installing local applications or on-premises applications as much as possible, with SaaS services accessed in a browser being the better option. Overall, this is true, but it definitely depends on the cloud service provider (CSP) you're using. The big ones are good and getting better. But many smaller outfits now have the opportunity to offer up a lucrative service with questionable backend infrastructure, something that you might get away with in public cloud (for a while) but which would be quickly noticed in an on-premises application.

As a real-world example, Microsoft's support for M365 has been a real hit and miss for me. I probably open a case every couple of months on average, and sometimes I've had some really excellent people who quickly and efficiently tracked down the issue. Other times it's been days or weeks of slow progress, clearly attributable to a lack of knowledge. It's a tough job being in support, especially for cloud services that change so quickly, but excellent support should be table stakes today.

I'm sure everyone who's reading this has worked at a business where the façade presented to current and potential customers was very shiny, and you simply hoped the often dusty and less than stellar reality in the back room never came to light. This doesn't work for long in IT however, as criminal threat actors have a way of providing a postpaid "unsolicited penetration test" -- also known as ransomware -- wherever there's money to be made.

Another real-world example is the school management system called FACTS at the aforementioned school. Today it's an on-premises Windows Server 2016 VM, with SQL Server 2016 as the underlying database. At the end of the year, that on-premises system will no longer be supported, and a forced migration to the cloud is on the cards. And that cloud application is quite expensive, because essentially, they're just running two VMs in Azure for you, on top of still having to pay licensing for the software itself. That's a long way from a cloud-native, micro services architecture, distributed infrastructure behind a SaaS service offering true consumption-based licensing on a per-user, per-month basis.

I am heartened that Microsoft Azure is now publishing CVE's for vulnerabilities in their cloud, even if no customer action is required and that AWS is following suit. This amount of insight into the inner workings of cloud platforms is important to build customer confidence, and this transparency will overall improve trust.

Conclusion
Even if all graduating software developers learned how to code securely, assumed that their code / system was going to be attacked and used in ways that they didn't intend and thus wrote more resilient code, we'd still be a long way from improving all the existing code in businesses worldwide. And the capitalist / IT industry's insatiable hunger and drive for the latest features will not slow down, until (perhaps) businesses demand better quality products. This, plus regulation might finally take the sting out of those licensing agreements that basically tells you that the software might not do anything you think it'll do, and there's nothing you can do about it, and instead put some responsibility back on the tech companies.

After all, many of them have signed the recent Secure by Design pledge, which might move the needle on code quality going forward. We live in hope.