Dan's Take

Parallel Processing With Docker

Univa updates Grid Engine to work with containers.

• By Dan Kusnetzky
• 09/18/2015

The folks at Univa have enhanced Grid Engine. The company's goal was making its parallel processing software play nice with Docker containers, along with all of the other types of physical and virtual processing solutions it already supports.

This new version of the software now has the ability to monitor and manage parallel workloads executing in Docker containers. The company also had to make sure that Docker container-based workloads would work alongside workloads executing directly on many types of physical systems running many different operating systems, or running inside many different types of virtual machine (VM) software.

Scaling Up and Out
Univa announced that its Univa Grid Engine Container Edition product fully incorporates Docker containers. The Container Edition features the ability to run containers at scale and blend containers with other workloads; it also supports heterogeneous applications and technology environments.

When Docker containers are run in a Grid Engine cluster, Grid Engine can manage scheduling decisions and resource allocation for Docker containers alongside other workloads. It includes cluster-wide full job control with pre-emption, termination and run-time statistics. The Grid Engine Container Edition already works with any infrastructure, any operating system and thousands of applications and frameworks.

Grid Engine allows a user or administrator to schedule Docker containers so that business-critical containers are prioritized over other workloads, thus maximizing the use of shared resources.

Dan's Take: Don't See Every Problem as a Nail
I've followed Grid Engine for quite some time, from when it was a Sun product, to when it was acquired by Oracle, to when it was offered as open source software. It's an excellent example of parallel processing technology, one of the five forms of virtual processing software.

Unfortunately for suppliers of this type of technology, IT developers and decision makers have a tendency to follow trends, rather than staying with technology that's been proven to work. Decision makers and developers in the recent past have become overly enamored of VM software combined with VM movement software. This reminds me of the famous phrase: "If you only have a hammer, you tend to see every problem as a nail."

Developers of technical or analytical software often were directed to -- or chose to -- simply decompose complex applications, encapsulate them in VMs and have them execute on a cluster of systems being managed by a workload management system, rather than decomposing the complex applications into individual functions and using a parallel processing monitor, such as Univa's Grid Engine or IBM's Platform LSF.

Masters of Illusion
While this approach was uniform and much simpler to understand, it often failed to maximize the utilization of the underlying physical resources. A parallel processing monitor can make it possible to harness the power of tens, hundreds or even thousands of computers and make them work on a large single problem or many smaller ones in a manageable, intelligent way.

In my book 2011 O'Reilly Media book, "Virtualization: A Manager's Guide," virtual processing technology is described in the following way:

Taken as an aggregate, virtual processing software creates an illusion in which it is possible for processing capabilities of systems to be utilized in new ways. Using virtual machine software or operating system virtualization and partitioning software, a single system can be utilized as if it was many individual systems. Using parallel processing software, workload management software and clustering software, multiple machines can be seen as a single computing resource.

You Say You Want an Evolution?
The suppliers of parallel processing technology, such as Univa and IBM, have continued to add capabilities to their products and integrate them into digital content creation, technical applications, architectural design, non-destructive testing, pharmaceutical research, big data and other workloads. Univa's extension of Grid Engine to be able to manage Docker applications is a good example of this type of parallel processing evolution.

If your organization has a need to harness the power of a huge number of systems to address its computational needs, learning more about Univa's Grid Engine would be useful.