Beyond the WORM with MinIO object storage

I find the terminology of WORM (Write Once Read Many) coming back into the IT speak in recent years. In the era of rip and burn, WORM was a natural thing where many of us “youngsters” used to copy files to a blank CD or DVD. I got know about how WORM worked when I learned that the laser in the CD burning process alters the chemical compound in a segment on the plastic disc of the CD, rendering the “burned” segment unwritable once it was written but it could be read many times.

At the enterprise level, I got to know about WORM while working with tape drives and tape libraries in the mid-90s. The objective of WORM is to save and archive the data and files in a non-rewritable format for compliance reasons. And it was the data compliance and data protection parts that got me interested into data management. WORM is a big deal in many heavily regulated industries such as finance and banking, insurance, oil and gas, transportation and more.

Obviously things have changed. WORM, while very much alive in the ageless tape industry, has another up-and-coming medium in Object Storage. The new generation of data infrastructure and data management specialists are starting to take notice.

Worm Storage – Image from Hubstor (https://www.hubstor.net/blog/write-read-many-worm-compliant-storage/)

I take this opportunity to take MinIO object storage for a spin in creating WORM buckets which can be easily architected as data compliance repositories with many applications across regulated industries. Here are some relevant steps.

[ Note: I am using WORM and object locking interchangeably here because this is for MinIO object storage. Object locking in cloud native storage is the mechanism that can enable WORM but the mechanisms of WORM are aplenty in other medium types such as tapes, optical discs, and probably DNA storage (I haven’t learned about this yet).

WORM planning

The initial step is NOT to create the WORM bucket. Data management prudence must apply first before putting the data in a WORM state. Questions about:

• What data to lock
• What type of locking – Compliance, Governance, Legal Hold
• Retention Period
• Versioning

All these questions must be answered prior to the creation of the WORM bucket, because once the MinIO bucket is locked, there are not many options to change the settings again until the validity of the settings is over.

Creating a MinIO WORM bucket

The screenshot below is straightforward. In the MinIO console, create a bucket. Provide a name to identify the bucket.

Slide the Object Locking from Off to On. By default, the Versioning is automatically turned On as well. If you want to WORM objects to have an expiry date, slide the Retention to On. The Validity can be set days and years where the objects locked are kept locked until an expiry date is reached.

Creating an object lock bucket in MinIO – a WORM feature

Compliance and Governance mode settings

The 2 modes seen are Compliance and Governance. They deserve a separate section to explain them.

• Governance Mode – Objects in the bucket or the entire bucket are prevented from normal users. Privileged users with the right permission can still alter the retention settings and delete the objects.
• Compliance Mode – Objects in the bucket or the entire bucket cannot be deleted by all users until the retention period has expired. Even privileged users cannot modify the retention period to bypass the lock.

A object also can be placed in Legal Hold which has no retention period and expiry date. This puts the lock on the object indefinitely.

Data tampering and ransomware

Ransomware definitely played a big role in nudging WORM into the attention again. But the initial intention was simpler where industries and businesses wanted to preserve the data for long term and prevent data tampering. Regardless, the immutability feature is now a must in many organizations looking at data protection in the face of a pandemic threat, digitally, and I am not talking about Covid-19.

Thus the design of the data management scope around data immutability involves data protection, data security, data compliance, data privacy and even data preservation and data sovereignty as well.

Considerations and cost

In the era of cloud computing, s3 storage has become the de facto standard, and object storage is underpinning the distributed ways to store and share data via buckets. However, one very important mindset is to make sure that the right sets of data are given the right labeling of usage when it comes to keep data locked in buckets. The labeling I am referring to here can be tied to the AAA (Authentication, Authorization, Audit) data management mindset to make data in the buckets secure and ensure that they are protected.

Mislabeling of usage can also lead to complications and costs. Once the data is locked in a WORM bucket, the data is intentionally not modifiable and in the more restrictive settings of compliance and legal hold, the objects in the bucket is not deletable as well for a selected period of time. In the case of legal hold, it is forever. These of course, leads to a longer term capacity cost consideration as well.

Thus, WORM is a feature in the larger part of data management ecosystem. Organizations can take advantage of the modernized version of WORM with object storage, and in my books, MinIO is top notch.

MinIO is a native s3 object storage service in iXsystems™ TrueNAS® CORE and Enterprise. It can also architected as a scale-out distributed object storage cluster with TrueNAS® SCALE.