The Cranky Admin

Trust as a Service

Trusting the wrong service provider or hosting company can be disastrous. Due diligence is required.

• By Trevor Pott
• 04/25/2017

In the modern, cloudy-as-a-service world, on-premises IT teams are both nerdy overlords and lowly users. Balancing administrative requirements with providing excellence in customer service is increasingly complicated by our need to act as filters for upstream service providers.

Most organizations in the western world are at this point entirely dependent upon IT to function. Take away their computers and they simply wouldn't be able to do business; some critical aspect, even if it's "only" bookkeeping and taxes, is being automated by an application. In many cases it has been this way for so long, and become so normal, that prices have been reduced and margins shrunk such that replacing those computer programs with people would not be financially viable.

Being able to wrap your arms around the servers, workstations and switches upon which you depend can be psychologically important, especially to the risk averse. Having an actual, physical throat to choke when it goes wrong can be equally comforting.

A Lifetime To Earn; a Moment To Lose
Despite an unending stream of marketing over the past decade proclaiming how much more reliable and secure public cloud services are, the very idea of hosted computing, either with a service provider or a public cloud provider, makes many people nervous. IT staff are often amongst the most nervous about hosted services. We're trained to be risk averse to the point of paranoia. Regardless of where workloads are hosted, when something goes awry the finger of blame points at us.

Because of this, trust is a very big part of the equation when some or all of the IT we depend upon is outside of our control. Trust can take a lifetime to earn, and it can be lost in an instant. It's also something IT vendors simply aren't very good at.

We're used to designing against the failure of on-premises IT failure. Disaster-Recovery-as-a-Service (DRaaS) is one of the major use cases for hosted IT. We aren't, however, used to thinking or talking about on-premises IT as just another IT provider, one that's not fundamentally different from a service provider or public cloud provider.

Marketing has taught us to think of hosted providers as "different." This mindset is so pervasive that admins can forget about even designing for the possibility that the hosted provider will fail. A self-service portal is, apparently, all it takes for many of us to forget how to architect a datacenter; we mentally avoid thinking of hosted providers as the outsourced additional premises that they are. This is dangerous.

Ask yourself for a moment which companies you really trust; companies you trust not only with your own livelihood, but with the livelihoods of everyone you work with. Think of everyone who depends on your organization to remain solvent and functional: you, your co-workers, spouses, offspring… the whole chain. Now ask yourself, do you really trust your vendors enough to place the fate of all those people in their hands?

This is the fundamental problem with hosted IT.

Dire Consequences
I live in Canada. If the major utilities fail, people die. As I type this, there is a foot of snow on the ground and more falling. If my utility providers failed me in the middle of the night, my pets could die. My family could die. I could die. It only takes a few hours, and one can simply drift away and succumb to the cold while asleep.

Despite this, I trust my utility providers every day. If an ice storm takes out the power lines, they will do everything humanly possible to get those lines back up. The same is true for my natural gas provider. These are heavily-regulated utilities subject to massive fines and potentially even jail time for failure. With a few notable exceptions, the system has worked for millions of Canadians for decades.

Hosted IT providers are not similarly regulated. Nobody comes around and slaps them on the wrist for failure. Instead, the burden and risks of ensuring that all is well rests on the customer. If something important goes sideways, it isn't the cloud provider that will end up in front of a judge. It's you.

Using cloud services, then, can only be accomplished with a great deal of trust, or a great deal of paranoia. One can simply trust the vendor's claims about the quality of their service -- and that it will continue at that quality and price for years to come -- or they can treat hosted IT providers as untrusted, and design for failure.

Mistrusting Microsoft
Microsoft is my canonical example of a hosted IT provider I will probably never be able to trust. Without even getting into Microsoft's efforts to kill its own channel, I find that its efforts to remove both end user and administrative control over our own operating systems to be an alarming and disturbing breach of trust. Personally, I can't trust Microsoft with cloud computing when they prove so readily that they care very little about trust in other aspects of their organization.

Even if lives or livelihoods don't depend on the IT I choose to entrust to a hosted provider, whatever IT I am entrusting to a hosted provider will impact my customers. Those customers will probably be internal to the organization I serve, but they will also likely be customers of that organization.

Many organizations provide downstream IT services. Even the smallest SMBs I have worked with make some portion of their custom middleware available to customers for order submission, order tracking or logistics.

Those middleware IT services drive pretty much all business processes internal to the organization. I can recount many cases where customers have designed their IT and even their entire business processes around these sorts of IT services, and who then go on to present IT services to their customers, sometimes with the chain being five or six layers deep.

The Domino Effect
Without designing for provider resiliency, an outage by a hosted IT provider could take out the app at the core of my organization; which in turn paralyzes our customers' organizations; and their customers' organizations; and so on, all the way down the line. Our whole society works like this today: a rat's nest of interconnectivity and interdependency. We are all of us in the business of consuming and selling trust.

We cannot forget about the consequences of the choices we make. We must pay attention to whom we trust, and be able to justify why we chose to have placed the trust that we did with the vendors and providers we did.

After all, if it does go sideways, we may well be the ones having to explain ourselves to a judge.