In-Depth

Microsoft, Do Better

• By Paul Schnackenburg
• 09/05/2023

It's high time for Microsoft to have a rebirth -- the death spiral of its attitude to security needs to change. In this article I'll look at the last few years of security blunders, Microsoft's lackluster responses to these, the inherent conflict in being a purveyor of both the platforms and the security tools on top of those platforms, the curse of legacy software, and give a list of actions that should be taken (IMHO).

Acknowledgement
I learned a long time ago to start with the good first, acknowledging strengths before pointing out weaknesses, so here goes.

Microsoft makes some amazing platforms. My SMB clients would not be as effective as they are in running their businesses (one financial services firm, one engineering consultancy and two educational institutions) if they weren't running on Microsoft 365. And for their security needs (one on M365 Business Premium, the others on M365 E5 or A5), Defender for Endpoint, Defender for Identity, Defender for Office on the XDR side, combined with Microsoft Sentinel and Defender for Cloud Apps is a fantastic, integrated solution that make it possible for me to run a one-man SOC for for them. I literally could not provide these SMBs with the same level of protection without a lot of manual integration of disparate security tools.

Also, Microsoft's work in Ukraine since Russia's illegal invasion has been nothing short of outstanding. Amazing, dedicated cyber warriors fighting to defend democracy against war criminals for very little recognition -- add to that the level of Microsoft's financial support with cloud services and so on. (Also, shout out to Swedish (where I was born) company Yubico donating 20,000+ Yubi keys to Ukraine through Hideez).

And I follow many excellent security researchers that work at Microsoft, and they share their great work so anything that follows is not a slight on the 8,500 cybersecurity professionals in the trenches.

But there's a problem at Microsoft in regard to security and it needs serious attention by the leadership, something that seems not to have been happening over the last few years.

The World's Biggest Elephant in the Room
On Jan. 15, 2002, Bill Gates wrote a memo to everyone at Microsoft -- of Trustworthy computing fame. And Microsoft did shift then in a big way, and some years ago I would have described Microsoft as a leader in how to do security right, but it seems to me that over the last few years it's lost its way.

Here are some examples of security situations where Microsoft's response wasn't up to snuff.

ChaosDB -- August 2021

CosmosDB is Azure's NoSQL database that provides several different APIs to the same data, and automatic deployment to different regions and synchronization between different copies at five different consistency models to match your application's needs. It also had a flaw in that you could use its Jupyter notebooks feature to steal primary keys from other customers' Cosmos DB databases and thus access (or alter) the data stored there, discovered by Wiz. Microsoft fixed this one within 48 hours of being notified, but how was this not found by internal testing prior to the notebook feature being rolled out?

OMIGOD -- September 2021

Linux VMs in Azure deployed on Infrastructure-as-a-Service (IaaS) with certain management services enabled had Microsoft's Open Management Infrastructure (OMI, the equivalent to WMI on Windows), agent silently installed on their virtual machines (VMs). Four separate flaws were found by Wiz that would allow an attacker to become root on the VM. One flaw, with a CVSS score of 9.8 of 10, simply required sending a single packet with the authentication header removed.

Microsoft patched the flaws in August 2021, and the vulnerabilities were disclosed mid-September, but many organizations didn't even know that they had to update the agent they didn't know was installed.

Here's the official article, but Microsoft was cagey in their response to this one, and should definitely have been more transparent about the agent being silently installed. Also, these flaws should have been found by their internal red team, particularly the one that was super easy to exploit.

Sunburst / Solarwinds -- December 2020

There has been many a keyboard worn out by pundits writing about this supply chain attack by the Russians against SolarWinds Orion network monitoring tool that initially compromised 18,000 different companies worldwide. The attackers then carefully chose specific organizations to further compromise, using Microsoft 365 and vulnerable Active Directory Federation Services (AD FS) deployments, specifically ones where their root keys were not stored in Hardware Security Module (HSMs). For this article there are two takeaways: first that Microsoft strongly recommends storing sensitive keys in HSMs (see below), and second, many U.S. government organizations had trouble identifying if their Microsoft 365 tenants had been compromised, because they weren't using the highest (most expensive) tier of licensing and thus didn't have access to "premium logging" (also see below).

Exchange Server -- Ongoing

I'm not going to break down each individual flaw found in Exchange Server on-premises over the last few years; suffice to say that there have been many, and often they have resulted in large scale compromise of the victims. Notable ones are ProxyLogon, ProxyShell and ProxyNotShell. However, none of these flaws have been exploited in Exchange Online as part of Microsoft 365.

If Microsoft wasn't running Exchange Online, and Exchange Server was still their flagship email and collaboration platform, it's hard to imagine that they wouldn't assign more resources to fixing these flaws before the bad guys find them. As it is, it seems like it's a supported product, but only by a few guys in a back office, and that's not sufficient against today's determined attackers.

Unauthorized Access to Cross-Tenant Applications in Microsoft Power Platform -- July 2023

This one doesn't have a catchy name, but in essence the Microsoft Power Platform (Power Apps, Power Automation) can use custom connectors that uses Azure function hosts to connect to various data sources. Those hosts are managed by Microsoft, but the API endpoint is unauthenticated, meaning that if you knew the hostname you could interact with it.

Microsoft was notified on March 30, 2023, and initially released a patch on the Aug. 3, which only fixed the issue for newly deployed applications. The company was originally going to wait until Sept. 28, 2023, to release the patch for all customers, but after Tenable's limited advisory, they released the patch for all customers.

Understandably Amit Yoran, CEO of Tenable, was very disappointed in Microsoft's response and wrote about it in a LinkedIn post.

What was lacking in the response here was transparency. Cloud customers (including at least one bank) didn't know that their sensitive data was at risk and didn't have information to take mitigation actions. Also, only patching for new applications (initially) wasn't sufficient.

This one was a doozy, where an attacker could compromise any Azure AD (now Entra ID) registered multi-tenant application that wasn't configured to properly check that the user trying to sign in was from a tenant that should have access. Approximately 25 percent of all multi-tenant apps were susceptible to this attack at the time of publication by Wiz. The demonstration by Wiz of the attack was to change the most popular soundtrack results on Bing.com to have the music from "Hackers" (1995 movie) as No. 1.

The main takeaway from this one is the complexity of the cloud, especially Entra ID, and how easy it is for a small misconfiguration to have huge implications.

nOAuth -- June 2023

This recent flaw -- also in OAUTH in Entra ID -- allowed complete account takeover if an application allowed sign-in with a Microsoft (consumer) account and was discovered by Descope. This is less of a flaw as Microsoft specifically warns against using email as an identifier for these claims, but still demonstrates the point that configuring application registration and multi-tenancy in Entra ID is complex and small mistakes can have large consequences.

Definitely the most serious issue of all on this list is this recent attack. In summary, Chinese adversaries "acquired" an expired (in 2021) private signing key, only meant to work for Microsoft consumer accounts. It was used to mint working access tokens for Exchange Online (and possibly other services). These were then used to compromise at least 25 U.S. organizations, including some government departments.

This wasn't even discovered by Microsoft themselves (which is embarrassing), but one of their clients who had the aforementioned "premium" logging enabled.

The circumstances around this attack are still not clear, although the Cyber Security Review Board (CSRB) will make this their focus for their third investigation (the two previous ones covered Log4J and the Lapsus$ gang). Microsoft's official blog post leaves a lot of questions (at the time of writing) such as:

• Why did an expired private signing key still work?
• How did the attackers obtain the key?
• Why wasn't the key stored in an HSM (as Microsoft recommended to their customers after Sunburst)?
• Why did a key for consumer accounts work for signing M365 business access tokens?
• Why wasn't this caught by Microsoft's own telemetry (rather than a customer)?
• Why wasn't this found by internal red team probing? It seems like the first place to look -- in the identity plane.

Steps to Take
These are CEO Satya Nadella's words back in January 2021:

"We're investing $20 billion-plus to advance our security solutions over the next five years."

And:

"One of the things about Microsoft's investment is really not just about the security investment -- which is substantial when you look at Defender or Sentinel or Azure Active Directory and the list goes on -- but it is that ability to take all of our investments in AI and data and infrastructure and bring it to bear for security. And that I think is what makes us pretty unique."

That was explaining the $10 billion in revenue from security services in 2020 and establishing Microsoft as a serious cybersecurity vendor, but in hindsight I think there's a mismatch between words and actions. You cannot be a successful cybersecurity vendor (for long) without being laser focused on security internally.

As Microsoft's technologies underpin so much of our modern world, their internal security must be top notch, not just OK. Specifically, here are my recommendations for Microsoft:

1. Change the company culture from feature first, to secure feature first. Yes, everything is moving at cloud speed, and yes, adding new features to cloud services is easy, and might get you promoted. But it needs to be underpinned by the same mentality that a large airline manufacturer might have -- if there's a smell of a security risk, that takes precedence, and the assembly line is stopped. It looks to me like that's not happening today.
2. Transparency in reporting. For traditional software we have CVEs and responsible disclosure that allow businesses to evaluate their risk and see if a flaw is found in software they're using and take appropriate mitigation measures until a patch is available. In cloud software, no such transparency exists, hence we must blindly trust, with no ability to "peek behind the curtain. And exec Brad Smith and his legions of lawyers must be nowhere near those write ups -- they must be as transparent as possible. At the moment it feels like you've lost our trust; to regain it you must be the most upfront about your failures, with no weasel words to be found.
3. Currently Microsoft is in a very strong financial position, which is probably going to be even more boosted by all these (expensive) Copilots popping up in different M365 and Azure services. Use some of it to really fix the underlying, systemic, cultural issues that have led to the breaches listed. Double the budget for MSRC. Double the bug bounty program payouts.
4. Invite external security researchers to investigate Entra ID (Azure AD), Microsoft 365, Azure and your other cloud services internal workings (this is already happening in limited capacity). This is in addition to doubling the size of your internal red teams, and also focus them on the internal workings of the cloud.
5. Many of the breaches and issues listed above arise from the inherent complexity of modern public clouds. Initially when we moved IaaS VMs from on-premises to the cloud, the security paradigm was familiar and we knew how to monitor, implement host-based firewalls, EDR and monitoring and so on. The Platform-as-a-Service (PaaS), SaaS and functions/container/Kubernetes world is much more complex, and complexity is the enemy of security, as is the constant change as new features are added at cloud speed. Microsoft must take a leading role here, help educate developers on this complexity and most importantly, design UIs and APIs to make the secure configuration the default and make it very hard to pick the insecure option.
6. Adding "premium" logging in mid-2023, three years after the issues around the SolarWinds breaches, and extending storage from 90 to 180 days is too little, too late. You need to make sure every M365 customer has access to all the logs they'll need to investigate a breach, for at least one year.
7. Balance between being the platform vendor as well as selling the add-on security services on top. This is a difficult proposition, but my suggestion would be that more resources be devoted to fixing flaws in Azure, M365, Windows and so on, which will benefit all customers, rather than focusing too many resources on the advanced security features.

Data Privacy Legislation
On a related note, the U.S. really needs a federal data privacy law. In the absence of it, 12 states have now enacted (slightly different) legislation, the problem being that 11 of those show signs of being influenced by lobbyists. This Malicious Life podcast episode covers this extensively, and this Politico article does as well, and both show the tech giants behind this watering down of the legislation, putting big tech and profits over protecting ordinary people. I understand why organizations such as Google and Meta do -- they literally wouldn't exist without surveillance capitalism -- but Microsoft, you don't. You sell great cloud services, and you can be a champion for data privacy, not just in marketing material, but also in reality.

Conclusion
These are of course not just my opinions. Many people in and outside the cybersecurity industry are drawing attention to this growing problem and begging Microsoft to make the required changes. Here is J. A. Guerrero-Saade, senior director of Sentinel Labs at SentinelOne and his thread. There are also many others.

It'll be interesting to see the findings of the CSRB, and if any other government actions will focus on Microsoft. Time will tell if Microsoft has a deep realization and makes the required cultural changes -- we'll all be watching.