Magic happening

[Preamble: I am a delegate of Storage Field Day 15 from Mar 7-9, 2018. My expenses, travel and accommodation are paid for by GestaltIT, the organizer and I am not obligated to blog or promote the technologies presented at this event. The content of this blog is of my own opinions and views]

The magic is happening.

Dropbox, the magical disruptor, is going IPO.

When Dropbox first entered into the market which eventually termed as BYOD (Bring your Own Device), it was a phenomenon. There was nothing else that matched its simplicity and ease-of-use. A file uploaded into the cloud was instantaneously available on the tablets and smart phones. It was on every storage vendor’s presentation slides, using Dropbox as the perennial name dropping tactic to get end users buy-in.

Dropbox was more than that, and it went on to define a whole new market segment known as Enterprise File Synchronization and Sharing (EFSS), together with everybody else such as Box, Easishare (they are here in South East Asia), and just about everybody else. And the executive team at Dropbox knew they were special too, so much so that they rejected a buyout attempt by Apple in 2011.

Today, Dropbox is beyond BYOD and EFSS. They are a full fledged collaboration platform that includes project management, project workflow, file versioning, secure file transfer, smart file synchronization and Dropbox Paper. And they offer comprehensive plans from Basic, Plus and Professional to Business and Enterprise. Their upcoming IPO, I am sure, will give them far greater capital to expand, and realize their full potential as the foremost content-based collaboration platform in the world.

Dropbox began their exodus from AWS a couple of years ago. They wanted to control their destiny and have moved more than 500PB into their own private data center for their customer data. That was half-an-exabyte, people! And two years later, they saved $75million of operating costs after they exited AWS. Today, they have more than 1 Exabyte of customer data! That is just incredible.

And Dropbox’s storage architecture started with a simple foundational design called “Magic Pocket“. Magic Pocket is a “fixed-length, immutable” block storage layer.

The block size is fixed at 4MB chunks (for parallel performance and service resumption reasons), compressed and deduped (for capacity savings reasons), encrypted (for security reasons) and replicated (for high availability reasons).

The blocks contains the user data but the metadata to the blocks are separately stored in order to easily scale to the multi-exabyte realms. The metadata is stored in a FileJournal, an external key-value store system disassociated from Magic Pocket. This gives Dropbox architects and engineers the flexibility to keep the immutable blocks in the lower storage ranks but allows mutable, variable-length bytes files to be written, modified, appended and deleted without the dependency of the block storage beneath it. (Refer to the layers above)

As we move forward to understand the Dropbox storage architecture and the data services, we see how blocks are gathered into buckets, buckets into volumes, volumes into cells, and cells into zones. The diagram below consolidates the culmination of all the layers, each providing the foundation for the layer above it.

In this Field Day session, the Dropbox team did not do a very deep dive on Magic Pocket. Much of the information I wrote in this blog came from their blog on the Magic Pocket technology. But I was excited that they got the Storage Field Day delegates fixated with their philosophy and reasons why they went to their flexible cloud storage architecture. The move from AWS S3 to the Magic Pocket architecture has benefited Dropbox. According to James Cowling, one of the key architects of Dropbox for this project, the performance gain was 3x-5x compared to their past services when they were on AWS S3.

At the end of the day, we were given a tour of the Dropbox facility here in the Silicon Valley and we got to see their modular gear, Diskotech.

Dropbox is entering into a magical moment now. Their upcoming IPO will give them a tremendous boost to their expanding their visions. And it will be the Magic Pocket storage foundation’s scaling prowess that will get them there. We wish them good fortune and good future ahead.