Hammering Next Gen Hybrid Clouds

[Preamble: I have been invited by  GestaltIT as a delegate to their TechFieldDay from Oct 17-19, 2018 in the Silicon Valley USA. My expenses, travel and accommodation are paid by GestaltIT, the organizer and I was not obligated to blog or promote their technologies presented at this event. The content of this blog is of my own opinions and views]

Hammerspace came out of stealth 2 days ago. Their objective? To rule the world of data for hybrid clouds and multi-clouds, and provide “unstructured data anywhere on-demand“. That is a bold statement, for a company that is relatively unknown, except for its deep ties with the now defunct Primary Data. Primary Data’s Chairman, David Flynn, is the head honcho at Hammerspace.

The Hammerspace technology has come the right time in my opinion because the entire cloud, multi-cloud and hybrid cloud stories have become fractured, siloed. The very thing that cloud computing touted to fix has brought back the same set of problems. At the same time, not every application was developed for the cloud. Applications rely on block storage services, or NAS protocols, or the de facto S3 protocols for storage repositories. However, the integration and communication between applications break down when these on-premises applications are moving to the cloud, or when applications residing the cloud are moved back to on-premises for throughput delivery, or even applications residing at the edge.

What if you can deliver just the required data to the right applications, anywhere in the clouds or on-premises, without changing the application structure or interface, or the data formats? What if you know all these requirements of each applications through its metadata, and the platform intelligently moves the data to applications anywhere for performance, security, compliance, costs and other business reasons. That is what Hammerspace is about. In a nutshell, Hammerspace, to me, is Metadata-as-a-Service along with many other Data Services, in which it is a Control Plane that virtualizes the metadata and data into a single global namespace.

Given the pedigree of Hammerspace in NFS, the thoughts of Parallel NFS (pNFS) came up quickly, even before the session started. pNFS splits the metadata and the data, and the pNFS client consults a metadata server for the layout or the map of the data.

My usual punditry is to look at the Hammerspace architecture and make commentary. I screenshotted the diagram below from the Tech Field Day video.

Hammerspace is an on-premises VM or it could be an AMI (AWS Machine Image) in the cloud. From the diagram lower left corner, NAS technology such as NetApp FAS and DellEMC Isilon is read via NFS by the Hammerspace VM. Filesystem metadata, telemetry metadata, and applications intent in its SLO (service level objectives) form the trinity consolidation of metadata. This enhanced metadata is aggressively shared across all Hammerspace instances across on-premises VMs and the clouds, and forms a single global “filesystem” namespace. This near synchronous replication (without breaking the law of physics as David puts it) is their Active-Active Geospanning namespace.

Within the enhanced metadata space, File-level Attributes are meshed with Tags, Labels, and Keywords. The matrix of these file characteristics and others details are continuously fed into Hammerspace Machine Learning Engine, in a “Feedback Control Loop“, optimizing the data and promote or demote the Value of the files in the global namespace. This was described on the whiteboard by David, and I hope what I have written has captured the gist of their unique metadata-based technology.

Of course what I have described above is just a small but important piece of Hammerspace. The “rainbow” diagram below was brought up several times to describe the full capability of Hammerspace, starting with the Data Virtualization Layer, to Hybrid Cloud Data Management, and lastly Metadata-as-a-Service in the outer most layer of the rainbow hemisphere.

As they were presenting, my thoughts were on the “Data Locality, Data Mobility” design thinking which I usually bring up in my consulting work. This technology is solving a big part of that design requirement by putting the data closest to where it is required, and only moving what is required with the right business and operational requirements.

After their presentation, there were much chatter of Hammerspace among the delegates. They are obviously more in tune with the past development of Primary Data than me (since we don’t get much coffee shop tech talks in Malaysia) , and they spoke their views. Listening to the banter, I learned that Hammerspace has acquired the IP crown jewel of Primary Data, and I like what I have heard, seen and consumed so far.

The realization of Hammerspace’s global namespace requires the development and cooperation of partners, both on-premises and in the cloud. Hammerspace, on its day of coming out of stealth, already has strong partners in its stable. Amazon Web Services and Microsoft (probably included Azure cloud as well) are already on-board, whilst perennial storage stalwarts NetApp and Western Digital are too, in their partner roster. Completing the picture are RedHat and Cloudian, and I am pretty sure there will be more partners in the months to come.

What I have written in my blog does not do Hammerspace justice. There are so much to cover from a technology perspective, and I barely covered its architecture. There are so many things that could  potential and possibilities are far reaching. There wasn’t enough time to cover the key topics in Hammerspace and they intend to do more sessions for the Tech Field Day 17 delegates. I am sure we will see more and more of Hammerspace

FUNNY SIDE NOTE: If you Google Hammerspace, it is describe as this extradimensional space with unlimited storage. Popular in cartoons and anime, one can pull out almost anything from Hammerspace. One of the delegates, Ethan Banks, put up this cartoon in the Slack channel.