While TrueNAS® CORE and TrueNAS® Enterprise are more well known for its NAS (network attached storage) prowess, many organizations are also confidently placing their enterprise applications such as hypervisors and databases on TrueNAS® via SANs (storage area networks) as well. Both iSCSI and Fibre Channel™ (selected TrueNAS® Enterprise storage models) protocols are supported well.
To reliably protect these block-based applications via the SAN protocols, ZFS snapshot is the key technology that can be dependent upon to restore the enterprise applications quickly. However, there are still some confusions when it comes to the state of recovery from the ZFS snapshots. On that matter, this situations are not unique to the ZFS environments because as with many other storage technologies, the confusion often stem from the (mis)understanding of the consistency state of the data in the backups and in the snapshots.
Crash Consistency vs Application Consistency
To dispel this misunderstanding, we must first begin with the understanding of a generic filesystem agnostic snapshot. It is a point-in-time copy, just like a data copy on the tape or in the disks or in the cloud backup. It is a complete image of the data and the state of the data at the storage layer at the time the storage snapshot was taken. This means that the data and metadata in this snapshot copy/version has a consistent state at that point in time. This state is frozen for this particular snapshot version, and therefore it is often labeled as “crash consistent“.
In the event of a subsystem (application, compute, storage, rack, site, etc) failure or a power loss, data recovery can be initiated using the last known “crash consistent” state, i.e. restoring from the last good backup or snapshot copy. Depending on applications, operating systems, hypervisors, filesystems and the subsystems (journals, transaction logs, protocol resiliency primitives etc) that are aligned with them, some workloads will just continue from where it stopped. It may already have some recovery mechanisms or these workloads can accept data loss without data corruption and inconsistencies.
Some applications, especially databases, are more sensitive to data and state consistencies. That is because of how these applications are designed. Take for instance, the Oracle® database. When an Oracle® database instance is online, there is an SGA (system global area) which handles all the running mechanics of the database. SGA exists in the memory of the compute along with transaction logs, tablespaces, and open files that represent the Oracle® database instance. From time to time, often measured in seconds, the state of the Oracle® instance and the data it is processing have to be synched to non-volatile, persistent storage. This commit is important to ensure the integrity of the data at all times.
However, there is a gap between the consistency state in the memory and the consistency state in the persistent storage while the database is running. This gap is often defined by time in seconds, and can also defined by newer transactions and updates that have not been committed to persistent storage yet. Therefore the consistency state of the database structures in memory and the consistency state of database files in the persistent storage have to be coordinated to ensure that when the backup or snapshot is triggered, the copy is in an “application consistent” state.
This coordination, in SQL speak*, basically performs ‘ALTER DATABASE BEGIN BACKUP’, <snapshot triggered>, ‘ALTER DATABASE ONLINE’.
[ * Note: I have not done this for more than 20 years, so my SQL syntax could be entirely wrong! ]
The above puts the database in a quiesce mode for a very brief period. Clients and users interactions with the database continue as usual but the database at that moment is just accepting requests but not processing them. In other words, the database is “frozen” or paused momentarily to allow the storage snapshot to be triggered. The storage snapshot is started, also done in seconds, and the SQL script will put the database back into normal operations after the snapshot instance.
The workflow I described is well depicted in the sequence below:
Application consistent snapshot workflow from N2WS blog
How a crash consistent snapshot done in TrueNAS®
In this blog, we are going to discuss about crash consistent snapshots, and not the application consistent*** which is the TrueNAS® VM snapshot. That is because I have limited resources in my home lab** to perform the application consistent recovery part.
[ ** Note: My lab is just 1 x Dell Optiplex with a 2nd generation Intel® i5 CPU, 32GB RAM, a couple of Adata 120GB SSDs, 1GbE network with 2 other laptops. I actually do almost everything with Virtualbox™. I also got a TrueNAS® Mini X from the company a few months back but I haven’t done anything with it yet ]
[ *** Note: For application consistent snapshots for VMware®, TrueNAS® has the VMware® snapshot feature ]
These steps are specifically for recovering ZFS zvol over iSCSI from a crash consistent snapshot in TrueNAS®.
First of all, make sure there is a snapshot for the zvol that is being served as an iSCSI volume to Windows. Choose the most relevant snapshot for the recovery. Select the “>” sign on the right of the selected snapshot, and click ‘Clone to New Dataset’. This converts the read only snapshot to a clone of the snapshot with read and write permission. This clone is ready to be provisioned through the TrueNAS® iSCSI volume provisioning steps. The cloning process in TrueNAS is shown below:
Cloning a snapshot in TrueNAS
At the Windows iSCSI initiator, make sure that this new iSCSI volume from the cloning process is discovered and connected as shown below:
Windows iSCSI initiator to connect to the iSCSI volume clone from TrueNAS
Once connected, go the Control Panel > Administrative Tools > Computer Management > Storage > Disk Management and find the new iSCSI volume. This logical disk will appear offline. Turn this online and convert this to a disk using the simple volume creation wizard.
Windows Disk Management Volume Management
This will appear as a new drive letter. In the example, Drive J: is in the File Explorer as a new disk volume from TrueNAS®. The recovery from the storage perspective is complete and done in minutes. The drive is ready to use.
Drive J: in my Windows 7 workstation
Rolling further forward, TrueNAS® can create application consistent snapshots with VMware®. The VMware snapshot feature in TrueNAS (as shown below) can coordinate with the VMware® VSS (Volume Shadow Copy Services) component for a particular datastore to create a snapshot that is more consistent with the said application in the VM.
TrueNAS VMware snapshot
There are also several application consistent backup solution in the market for ZFS. Here is one for MySQL and MariaDB databases on ZFS volumes called ZRM (ZFS Recovery Manager) from Zmanda.
Data Loss and RPO & RTO
Crash consistent snapshots and backups will have data loss. Organizations must work out how much data they can afford to lose, and how much downtime they can tolerate to bring the application service back online. Thus, the conversation of RPO (recovery point objectives) and RTO (recovery time objectives) plays a part in defining what services are critical to the business of the organization.
Then IT must work out the procedures and the process to recover the data to using the most relevant crash consistent data copy of the snapshot to resuscitate and/or to rebuild the application services to a point of business resumption.
VSS (Volume Shadow Copy Services)
At the heart of this, in the application consistent backup scenarios, in the Microsoft® Windows world, this coordination and workflow framework is handled by the VSS (Volume Shadow Copy Services). VSS plays an immensely important role in enabling Windows-based application consistent snapshots and backups.
Overview of the Microsoft Volume Shadow Copy Service
There is a great Microsoft article that describes about VSS mechanics in depth here. I won’t be describing VSS in details here because many have written about it but from a TrueNAS® and ZFS perspective, the integration with the VSS framework has made the backups and the snapshots of data more resilient and recovery to restore to a consistency level that is synonymous to the RPO of an organization. Confidence in the business is restored.
