The Infrastruggle

The Rise, Fall, and Rise, of Virtual Tape Libraries

VTLs have a storied history. Are they about to make a comeback?

• By Jon William Toigo
• 03/11/2016

Readers of my work know that I am something of a tape evangelist. Simply put, the zettabytes of data that are amassing in the digital universe over the next few years will require tape storage: there is just not enough combined manufacturing capacity in either the disk or solid state industries to handle more than a ZB-and-a-half per year, and analysts are projecting between 10 and 60 ZBs of new data requiring storage over the next five years.

So there's that.

My research has also found tape to be the only storage media choice for an economically-valid archive strategy. This position has made me keep the Active Archive Alliance at arm's length and made me reluctant to add my full-throated support for some of the disk-based archive kits entering the market recently.

And, to be honest, my bias toward tape has also made me suspicious of Virtual Tape Libraries (VTLs) that use (typically) a disk-based repository as a surrogate for tape in a backup role. Let me back into this one, since I am striving to formulate a more balanced view.

Evolution Of The Virtual Tape Library
VTLs have been around a long time, as shown in Figure 1. When tape was slow and backup windows were shrinking in mainframe datacenters three decades ago, somebody smart came up with the idea of "front-ending" the tape system with a cache of disk drives. You could use this cache as a "virtual" tape library, enabling backup data to be written to the cache very rapidly and allowing the production system to get back to work.

Another advantage was that tape jobs could be "stacked" on the VTL, prior to writing the data to tape. This addressed a problem in mainframes that resulted in tape media being used very inefficiently, enabling write jobs to use the entire reel or cartridge rather than writing data in dribs and drabs across many pieces of media.

Once data was copied to the VTL disk, as an "off-line" or "near-line" process, the cached data would then be written to the tape system where it could be used to make local and off-site copies of the backup data. This multi-stage process leveraged VTLs to eliminate what we now call "data friction": production latencies brought about by slow data copies. It is worth noting that the original VTLs were software-based. That is, software running on the mainframe was used to designate some of the disk storage (DASD) for use as the VTL cache. Later, vendors introduced disk arrays that were explicitly designed to be used as VTLs, but the first VTLs, like storage infrastructure generally, were software-defined.

The second wave of VTLs were targeted at the needs of the distributed computing environment, where they sought to address several challenges. For one, the bandwidth of early distributed LANs was too constrained to handle backup data traffic from many servers simultaneously, and scheduling multiple backups was problematic, even with the best backup software.

Reinventing VTL
Thus the concept of the VTL was re-invented. VTL appliances could be installed in the same subnetwork as the servers whose data they would back up. Plus, in addition to serving as a write cache for a tape library, each VTL appliance became a full-fledged emulation of a physical tape environment, capable of representing multiple "virtual" robots and tape drives to expedite data capture, whether or not the actual hardware existed in the physical tape environment.

Behind the scenes, the idea was to do something with a VTL appliance that could not be done simply with backup software. By the late 1990s, tape had fallen into disfavor both as a function of disinformation by disk vendors and of a mismatch between tape capacities and data growth rates. Smart backup software vendors, following the lead of server operating system vendors, began to support "backup-to-disk" functionality in their products. To differentiate VTLs from this generic implementation of disk-based backup caching, vendors added functionality like drive emulation and, later, deduplication, to their appliances.

Even then, the ultimate strategy was to unload data cached on the VTL onto tape to capitalize on tape's portability, the "air gap" it provided between accessible (and prone to corruption, deletion or unauthorized disclosure) production data and a tape-based safety copy, and the ability of tape-based data to be restored to any target, regardless of the vendor brand-name on the kit (a key limitation of proprietary disk and flash storage arrays). A VTL was a stop-gap, resolving certain transport issues, but not a replacement for tape, in the final analysis (Figure 2).

It is worth noting that, in the early 2000s, the chief reason that companies were giving to explain their interest in SAN technology was to "enable the sharing of enterprise tape libraries" with distributed systems in their environments. VTLs had not eliminated tape; they had become complimentary.

Deduplicating VTLs Disrupt the Traditional Storage Model
However, the rise of de-duplicating arrays in the mid-2000s seemed to change the outlook. For a long time, the industry had been fairly adept at characterizing storage platforms in three basic flavors (Figure 3). Tier 1 or "primary storage" arrays were expensive, proprietary, low capacity and high performance platforms supposedly targeted at data that was frequently accessed and modified. Tier 2 or "secondary storage" arrays were somewhat less costly and comprised high capacity with fairly nimble accessibility. Tier 3 or "tertiary" storage encompassed VTLs, tape systems, optical disc, and other storage aimed at hosting data very inexpensively that did not require fast access or frequent updates. By the mid-2000s, however, vendors were blurring the lines between these archetypes.

Some primary storage vendors began offering array products that combined fast storage and capacity storage, with tiering software built onto array controllers. But the real assault on the storage order came from vendors of de-duplicating arrays and arrays featuring drive spin-down (for lower energy costs -- see Figure 4), who sought to substitute their products, which were built mainly on Tier 2 components, for tertiary storage altogether, eliminating the third storage category completely.

The curious thing about this model was the huge increase in acquisition cost for de-duplicating VTLs in particular. One vendor proffered a de-duplicating VTL comprised of a dedicated disk subsystem with VTL and de-duplication software running on a proprietary controller for an MSRP of about $410,000.

Peeling back the onion, one could see that the chassis, disk drives and other hardware components of the system, if purchased directly from their vendors, totaled about $3K. That meant that the vendor was charging $407K for the software, claiming that de-duplication made every drive in the system deliver storage capacity equivalent to 70 disk drives, and that this 70:1 reduction ratio justified the steep sticker price.

Moreover, while the initial products in the de-duplicating VTL appliance (Figure 5) supported a "tape back-store" for customers who wanted to make a safety copy of their data on the VTL, later iterations dropped support for this feature and encouraged consumers to de-commission tape altogether. A tchotchke from the vendor's booth at trade shows was a bumper sticker that exclaimed, "Tape Sucks! Move on."

The fact that this vendor (and many competitors) were unable to deliver anything like the promised data reduction ratios, plus the economic downturn in the late 2000s, and the dramatic increase in the capacity and performance of tape technology helped to quash this trend. However, the concept of personalizing generic low cost disk to provide a backup buffer remained a valid one, especially with the advent of so-called software-defined storage.

'Software-Defined' Meets VTL
Software-defined storage (SDS) has many roots, with one clear path tracing directly back to System Managed Storage (SMS) in IBM mainframes. Beginning in the late 1970s/early 1980s, direct attached storage devices (DASD) connected to the backplane of the mainframe, and were mainly controlled by SMS software running in the mainframe OS environment. Eventually, part of SMS control was ceded to on-array controllers, but the approach to separating the management and provisioning of storage from the underlying hardware was on full display in the mainframe datacenter.

Ironically, it was another vendor, seeking absolute control of the datacenter a la IBM circa 1977, who reinvigorated the SMS idea in the late 2000s, and called it SDS. That was VMware, whose rationale for usurping on-array management and provisioning control was actually based less on architecture than on marketecture.

In 2005, VMware was advising analyst houses that the server consolidation enabled under its hypervisor software would bend the CAPEX spending curve. Applications would be hosted on fewer arrays, administered by fewer operators, and delivering the same or better application access to end users.

However, that did not occur. In many firms, application performance suffered after being virtualized. VMware blamed "legacy" storage for bottlenecking I/O throughput and villainized SANs and NAS storage for its high cost and lack of flexibility.

In truth, storage I/O had nothing to do with virtual machine (VM) performance in most cases, a fact that could be clearly established by examining storage queue depth on underperforming systems. A low queue depth suggested that reads and writes were not being bottlenecked or delayed to any great extent, but that did not stop the hypervisor maker from citing a litany of complaints about overpriced proprietary storage to bolster its case for a "rip and replace" strategy.

In place of proprietary arrays and SANs, VMware and others posited the use of internal or directly attached (DAS) storage components with each virtual server. To reduce hardware costs, storage componentry would be generic hardware or "white box," with management and provisioning functionality delivered by a software layer running under the auspices of a hypervisor, rather than loaded and operating on a proprietary array controller.

While this architecture seemed sound enough, the devil was in the details. Each hypervisor vendor began defining, certifying and providing drivers for select hardware. Choosing gear that was not on a supported hardware list could create major problems.

Additionally, hypervisor vendors made clear that they did not want the hardware that they were supporting with their hypervisor SDS stack to be usable by any other hypervisor's workload. In effect, their strategy entailed walling off data into proprietary silos that were accessible only to workloads and their data operating under the particular hypervisor. Islands of Storage

The creation of isolated islands of storage was exactly the problem that SAN vendors had sought to address by aggregating storage into a common fabric, where either centralized or (mostly) on-array software could be used for management and provisioning. The gains of doing so would be to turn storage into a shared infrastructure resource available to any workload.

As consumers became more savvy about the intentions of hypervisor vendors and the constraints that their models were imposing, many looked to independent software vendors (ISVs) with hypervisor- and hardware-agnostic SDS stacks, opening the door to a more robust and diversified SDS model. To stay alive, small ISVs made deals with server hardware vendors, who themselves were challenged by the perception of all server gear as a "commodity" kit and who saw ISV wares as a potential opportunity to create "hyperconverged infrastructure" in an appliance footprint.

For the past two years, hyperconverged infrastructure has been front of mind for many companies. Ideally, the pre-integrated combination of hypervisor software, ISV SDS stacks, potentially ISV Software-Defined Network stacks, and server, networking and storage hardware could be delivered to market as "atomic building blocks" of infrastructure. Provided that all systems availed themselves of common orchestration and management, preferably via an open protocol such as the World Wide Web Consortium's REST, a new and highly simplified methodology for building agile infrastructure could be enabled.

The work on hyperconvergence is ongoing, but the delivery of HCI appliances is happening from both leading server makers and from secondary market server and storage gear resellers (offering HCI appliances built on off-lease or refurbished server equipment). An underexplored opportunity in the HCI space is the customization of appliances to perform discrete or specialized functions.

Enter Starwind Software's VTL
Starwind Software, one of the top ten ISVs in the SDS market, has been working with both hypervisor vendors and hardware vendors to deliver virtualization- and kit-agnostic SDS stacks for use both as alternatives to hypervisor proprietary SDS solutions ("virtual SANs") and as core ingredients of pre-integrated HCI appliances.

Recently, the company began shipping a VTL that is also software-defined. Reminiscent in some respects of software-based VTLs of the past, the Starwind VTL is marketed as a means to shorten backup times by enabling accelerated replication of snapshot backups from primary storage to high capacity disk. It also provides Starwind Tape Redirector, to facilitate the recording of backups to physical tape "for better safety" or to meet the requirements of regulations that do not allow for tape backup to be eliminated.

What makes the Starwind solution different from competitors is that it is available either as a fully-built appliance or as a software component of Starwind's Virtual SAN SDS solution that can run on a customer's existing hardware. The product also works across iSCSI for distance replication and supports SLAs for data protection in various configurations.

The Starwind VTL, according to the vendor, provides an additional level of data protection in their SDS technology, which already enables synchronous mirroring, asynchronous replication, deduplication and compression services, and snapshot technology. They also provide an ability that may come into greater future use:  the ability to off-load virtual tape images from on-site repositories directly to cloud storage providers. They are working with a major cloud service company now on a simple-to-use snap up service that will be announced shortly.

If Starwind Software's VTL achieves some uptake in the market, its success will likely usher in a new era of hyperconverged infrastructure appliances customized to specific functions, including data protection. Watch this space.