Tag Archives: backup

Backup – Lest we forget

By | | , , , , , , , , , , , , , , , , , , , , , , ,
Leave a comment

World Backup Day – March 31st

Last week was World Backup Day. It is on March 31st every year so that you don’t lose your data and become an April’s Fool the next day.

Amidst the growing awareness of the importance of backup, no thanks to the ever growing destructive nature of ransomware, it is important to look into other aspects of data protection – both a data backup/recovery and a data security –  point of view as well.

3-2-1 Rule, A-B-C and Air Gaps

I highlighted the basic 3-2-1 rule before. This must always be paired with a set of practised processes and policies to cultivate all stakeholders (aka the people) in the organization to understand the importance of protecting the data and ensuring data recoverability.

The A-B-C is to look at the production dataset and decide if the data should be stored in the Tier 1 storage. In most cases, the data becomes less active and these datasets may be good candidates to be archived. Once archived, the production dataset is smaller and data backup operations become lighter, faster and have positive causation as well.

Air gaps have returned to prominence since the heightened threats on data in recent years. The threats have pushed organizations to consider doing data offsite and offline with air gaps. Cost considerations and speed of recovery can be of concerns, and logical air gaps are also gaining style as an acceptable extra layer of data. protection.

Backup is not total Data Protection cyberdefence

If we view data protection more holistically and comprehensively, backup (and recovery) is not the total data protection solution. We must ignore the fancy rhetorics of the technology marketers that backup is the solution to ensure data protection because there is much more than that.

The well respected NIST (National Institute of Standards and Technology) Cybersecurity Framework places Recovery (along with backup) as the last pillar of its framework.

NIST Cybersecurity Framework

Continue reading →

Please cultivate 3-2-1 and A-B-C of Data Management

By | | , , , , , , , , , , , , , , , , , , , , , ,
Leave a comment

My Sunday morning was muddled 2 weeks ago. There was a frenetic call from someone whom I knew a while back and he needed some advice. Turned out that his company’s files were encrypted and the “backups” (more on this later) were gone. With some detective work, I found that their files were stored in a Synology® NAS, often accessed via QuickConnect remotely, and “backed up” to Microsoft® Azure. I put “Backup” in inverted commas because their definition of “backup” was using Synology®’s Cloud Sync to Azure. It is not a true backup but a file synchronization service that often mislabeled as a data protection backup service.

All of his company’s projects files were encrypted and there were no backups to recover from. It was a typical ransomware cluster F crime scene.

I would have gloated because many of small medium businesses like his take a very poor and lackadaisical attitude towards good data management practices. No use crying over spilled milk when prevention is better than cure. But instead of investing early in the prevention, the cure would likely be 3x more expensive. And in this case, he wanted to use Deloitte® recovery services, which I did not know existed. Good luck with the recovery was all I said to him after my Sunday morning was made topsy turvy of sorts.

NAS is the ransomware goldmine

I have said it before and I am saying it again. NAS devices, especially the consumer and prosumer brands, are easy pickings because there was little attention paid to implement a good data management practice either by the respective vendor or the end users themselves. 2 years ago I was already seeing a consistent pattern of the heightened ransomware attacks on NAS devices, especially the NAS devices that proliferated the small medium businesses market segment.

The WFH (work from home) practice trigged by the Covid-19 pandemic has made NAS devices essential for businesses. NAS are the workhorses of many businesses after all.  The ease of connecting from anywhere with features similar to the Synology® QuickConnect I mentioned earlier, or through VPNs (virtual private networks), or a self created port forwarding (for those who wants to save a quick buck [ sarcasm ]), opened the doors to bad actors and easy ransomware incursions. Good data management practices are often sidestepped or ignored in exchange for simplicity, convenience, and trying to save foolish dollars. Until ….

Continue reading →

Crash consistent data recovery for ZFS volumes

By | | , , , , , , , , , , , , , , , , , , , , , , ,
Leave a comment

While TrueNAS® CORE and TrueNAS® Enterprise are more well known for its NAS (network attached storage) prowess, many organizations are also confidently placing their enterprise applications such as hypervisors and databases on TrueNAS® via SANs (storage area networks) as well. Both iSCSI and Fibre Channel™ (selected TrueNAS® Enterprise storage models) protocols are supported well.

To reliably protect these block-based applications via the SAN protocols, ZFS snapshot is the key technology that can be dependent upon to restore the enterprise applications quickly. However, there are still some confusions when it comes to the state of recovery from the ZFS snapshots. On that matter, this situations are not unique to the ZFS environments because as with many other storage technologies, the confusion often stem from the (mis)understanding of the consistency state of the data in the backups and in the snapshots.

Crash Consistency vs Application Consistency

To dispel this misunderstanding, we must first begin with the understanding of a generic filesystem agnostic snapshot. It is a point-in-time copy, just like a data copy on the tape or in the disks or in the cloud backup. It is a complete image of the data and the state of the data at the storage layer at the time the storage snapshot was taken. This means that the data and metadata in this snapshot copy/version has a consistent state at that point in time. This state is frozen for this particular snapshot version, and therefore it is often labeled as “crash consistent“.

In the event of a subsystem (application, compute, storage, rack, site, etc) failure or a power loss, data recovery can be initiated using the last known “crash consistent” state, i.e. restoring from the last good backup or snapshot copy. Depending on applications, operating systems, hypervisors, filesystems and the subsystems (journals, transaction logs, protocol resiliency primitives etc) that are aligned with them, some workloads will just continue from where it stopped. It may already have some recovery mechanisms or these workloads can accept data loss without data corruption and inconsistencies.

Some applications, especially databases, are more sensitive to data and state consistencies. That is because of how these applications are designed. Take for instance, the Oracle® database. When an Oracle® database instance is online, there is an SGA (system global area) which handles all the running mechanics of the database. SGA exists in the memory of the compute along with transaction logs, tablespaces, and open files that represent the Oracle® database instance. From time to time, often measured in seconds, the state of the Oracle® instance and the data it is processing have to be synched to non-volatile, persistent storage. This commit is important to ensure the integrity of the data at all times.

Continue reading →

NAS is the next Ransomware goldmine

By | | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
2 Comments

I get an email like this almost every day:

It is from one of my FreeNAS customers daily security run logs, emailed to our support@katanalogic.com alias. It is attempting a brute force attack trying to crack the authentication barrier via the exposed SSH port.

Just days after the installation was completed months ago, a bot has been doing IP port scans on our system, and found the SSH port open. (We used it for remote support). It has been trying every since, and we have been observing the source IP addresses.

The new Ransomware attack vector

This is not surprising to me. Ransomware has become more sophisticated and more damaging than ever because the monetary returns from the ransomware are far more effective and lucrative than other cybersecurity threats so far. And the easiest preys are the weakest link in the People, Process and Technology chain. Phishing breaches through social engineering, emails are the most common attack vectors, but there are vhishing (via voicemail) and smshing (via SMS) out there too. Of course, we do not discount other attack vectors such as mal-advertising sites, or exploits and so on. Anything to deliver the ransomware payload.

The new attack vector via NAS (Network Attached Storage) and it is easy to understand why.

Continue reading →

Commvault big bet

By | | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
1 Comment

I woke up at 2.59am in the morning of Sept 5th morning, a bit discombobulated and quickly jumped into the Commvault call. The damn alarm rang and I slept through it, but I got up just in time for the 3am call.

As I was going through the motion of getting onto UberConference, organized by GestaltIT, I was already sensing something big. In the call, Commvault was acquiring Hedvig and it hit me. My drowsy self centered to the big news. And I saw a few guys from Veritas and Cohesity on my social media group making gestures about the acquisition.

I spent the rest of the week thinking about the acquisition. What is good? What is bad? How is Commvault going to move forward? This is at pressing against the stark background from the rumour mill here in South Asia, just a week before this acquisition news, where I heard that the entire Commvault teams in Malaysia and Asia Pacific were released. I couldn’t confirm the news in Asia Pacific, but the source of the news coming from Malaysia was strong and a reliable one.

What is good?

It is a big win for Hedvig. Nestled among several scale-out primary storage vendors and little competitive differentiation, this Commvault acquisition is Hedvig’s pay day.

Continue reading →

Perils of avoiding BC and DR

By | | , , , , , , , , , , ,
Leave a comment

News in recent months have been unfavourable, even to the point of poignancy. Maybe I didn’t have all the details to place my opinion, but it has appeared that these recent events have neglected the practice of  BC (business continuity) and DR (disaster recovery).

The recent bad news

The most recent is one close to home. The KLIA (Kuala Lumpur International Airport) and KLIA2 operations were disrupted quite significantly for 4 days due to “network switch” failure. I followed the news and comments quite intently in those bad days, and I did not see any single comment discussing about BC or DR. If BC and DR were present at the airports, the airport operations would have been restored within minutes or hours, not days. Investigations are still on-going to find out what really happened in the KLIA/KLIA2 incident.

Continue reading →

Quantum Corp should spin off Stornext

By | | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
1 Comment

What’s happening at Quantum Corporation?

I picked up the latest development news about Quantum Corporation. Last month, in December 2018, they secured a USD210 million financial lifeline to support their deflating business and their debts. And if you follow their development, they are with their 3rd CEO in the past 12 months, which is quite extraordinary. What is happening at Quantum Corp?

Quantum Logo (PRNewsFoto/Quantum Corp.)

Stornext – The Swiss Army knife of Data Management

I have known Quantum since 2000, very focused on the DLT tape library business. At that time, prior to the coming of LTO, DLT and its successor, SuperDLT dominated the tape market together with IBM. In 2006, they acquired ADIC, another tape vendor and became one of the largest tape library vendors in the world. From the ADIC acquisition, Quantum also got their rights on Stornext, a high performance scale out file system. I was deeply impressed with Stornext, and I once called it the Swiss Army knife of Data Management. The versatility of Stornext addressed many of the required functions within the data management lifecycle and workflows, and thus it has made its name in the Media and Entertainment space.

Jack of all trades, master of none

However, Quantum has never reached great heights in my opinion. They are everything to everybody, like a Jack of all trades, master of none. They are backup with their tape libraries and DXi series, archive and tiering with the Lattus, hybrid storage with QXS, and file system and scale-out with Stornext. If they have good business run rates and a healthy pipeline, having a broad product line is fine and dandy. But Quantum has been having CEO changes like turning a turnstile, and amid “a few” accounting missteps and a 2018 CEO who only lasted 5 months, they better steady their rocking boat quickly. Continue reading →

Storage and Data Management Planning crucial for Malaysian SMBs

By | | , , , , , , , , , , , , , , , , , , ,
Leave a comment

Hybrid IT for 2019 and beyond

2019 is here.

I am especially buoyed by the strong network storage industry footing in 2018, reported by The Register last week. 2018 was certainly a blowout year for storage infrastructure and storage software, both for on-premises and the cloud computing platforms. The AWS Outposts announcement over a month ago also just affirmed that the new world is Hybrid IT. And there is plenty to look forward to in 2019.

Malaysian Economic Doldrums

Things are not as rosy for the Malaysia economy in 2019. It will be a challenging 2019 as reported by the Edge, a local business publication. The GDP (gross domestic product) of the first half of 2018 shrunk, from 5.9% in 2017, to 4.65%, and it is estimated to be 4.9% in 2019. With an inexperienced new government, a weak currency, and more competitive economies emerging in ASEAN, Malaysia small and medium businesses (SMBs) could be challenged.

The knee jerk reaction would be to cut the IT spending and revert to buying on price. This has happened too often, because there are always other operating costs that may be more pressing. Furthermore, many of the SMBs are still aimless when it comes to transforming their businesses into the digital data era, groping in the dark and sputtering to get its worth with their IT investments. Often, many are misinformed and stumbled, resulting in much higher wastage and costs.

There is a local saying here:

“Good thing No Cheap; Cheap thing No Good“

And the saying is very apt to describe that there is value in investing well, and the price factor should not always be the main determinant criteria of buying IT infrastructure, software and services.

Many of these SMBs also lack experienced IT staff to manage their IT environment. There is also a hurried urgency to modernize IT, because a well-planned and executed IT strategy and operations would definitely increase their Competitive Advantage. Continue reading →

Swiss army of data management

By | | , , , ,
3 Comments

Back in 2000, before I joined NetApp, I bought one of my first storage technology books. It was “The Holy Grail of Data Storage Management” by Jon William Toigo. The book served me very well, because it opened up my eyes about the storage networking and data management world.

I mean, I have been doing storage for 7 years before the year 2000, but I was an implementation and support engineer. I installed my first storage arrays in 1993, the trusty but sometimes odd, SPARCstorage Array 1000. These “antiques” were running 0.25Gbps Fibre Channel, and that nationwide bank project gave me my first taste and insights of SAN. Point-to-point, but nonetheless SAN.

Then at Sun from 1997-2000, I was implementing the old Storage Disk Packs with FastWide SCSI, moving on to the A5000 Photons (remember these guys?) and was trained on the A7000, Sun’s acquisition of Encore way back in the late nineties. Then there was “Purple”, the T300s which I believe came from the acquisition of MaxStrat.

The implementation and support experience was good but my world opened up when I joined NetApp in mid-2000. And from the Jon Toigo’s book, I learned one of the most important lessons that I have carried with me till this day – “Data Storage Management is 3x more expensive that the data storage equipment itself“. Given the complexity of the data today compared to the early 2000s, I would say that it is likely to be 4-5x more expensive.

And yet, I am still perplexed that many customers and prospects still cannot see the importance and the gravity of data storage management, and more precisely, data management itself.

A couple of months ago, I had to opportunity to work on an RFP for project in Singapore. The customer had thousands of tapes storing digital media files in addition to tens of TBs running on IBM N-series storage (translated to a NetApp FAS3xxx). They wanted to revamp their architecture, and invited several vendors in Singapore to propose. I was working for a friend, who is an EMC reseller. But when I saw that tapes figured heavily in their environment, and the other resellers were proposing EMC Isilon and NetApp C-Mode, I thought that these resellers were just trying to stuff a square peg into a round hole. They had not addressed the customer’s issues and problems at all, and was just merely proposing storage for the sake of storage capacity. Sure, EMC Isilon is great for the media and entertainment business, but EMC Isilon is not the data management solution for this customer’s situation. Neither was NetApp with the C-Mode solution.

What the customer needed to solve was a data management solution, one that involved

  • Single namespace for video editors and programmers, regardless of online disk storage or archived tape storage
  • Transparent and automated storage tiering and addressing the value of the data to the storage media
  • A backup tier which kept a minimum 2 recent copies for file restoration in case of disasters
  • An archived tier which they could share with their counterparts in other regions
  • A transparent replication tier which would allow them to implement a simplified disaster recovery mechanism with their counterparts in Japan and China

And these were the key issues that needed to be addressed, not the scale-out, usual snapshot mechanism. These features are good for a primary, production storage infrastructure, but this customer’s business operations had about 70-80% data and files which were offline in tapes. I took the liberty to advise my friend to look into Quantum StorNext, because the solution could solve the business problem NOT solving it from an IT point of view. Continue reading →

Whitewashing Cloudsh*t

By | |
1 Comment

Pardon my French but I just had about enough of it!

I was invited to attend the Internet Alliance Association‘s event today at OneWorld Hotel. It was aptly titled “Global Trends on Cloud Technology”. I don’t know much about the Internet Alliance but I was intrigued by the event because I wanted to know what the Malaysian hosting and service providers are doing on the cloud. I was not in touch with the hosting providers landscape for a few years now, so I was like an eager-beaver, raring to learn more.

After registration, I quickly went to the first booth behind the front counter. He said he was a cloud consultant, so I asked what his company does. He said they provide IaaS, PaaS and so on. I asked him if I could purchase IaaS with a credit card and what was the turnaround time to get a normal server with Windows 2008 running.

He obliged with a yes. They accept credit card purchases. But the turnaround to have the virtual server ready is 1 day. It would take 24-hours before I get a virtual server running Windows. So, I assumed the entire process was manual and I told him that. He assured me that the whole process is automatic. At the back of my mind, if this was automatic, will it take 24-hours? Reality set in when I realized I am dealing with a Malaysian company. Ah, I see.

A few more sentences were exchanged. He told me that they are hosted at AIMS, a popular choice. I inquired about their Disaster Recovery. They don’t have a disaster recovery. More perplexity for me. Hmmm …

In the end, I was kinda turned off by his “story” about how great they are, better than Freenet and AIMS and so on. If they are better than AIMS, why host their cloud at AIMS?

I went to another booth which had a sign call “1-Nimbus”. The number “1” is the usual 1-Malaysia Logo with the word “Nimbus” next to it. Here’s that “1” logo below.

It was the word “Nimbus” that capture my attention. I thought, “Wow, is this really Nimbus?” Apparently not. Probably some Malaysian company borrowed that name .. we are smart that way. “1-Nimbus, Cloud Backup”, it read. I asked the chap (another consultant), who gave me the brochure, “How does it work?” “Does it require any agent?”

“Err, actually, I am not really technical. Let me refer you to my colleague”. A bespectacled chap popped over and introduced himself as a technical guy. I asked again, “How does this cloud backup work?”. His reply … “Err, it’s not really our product. Go check out the website”, and gave me another brochure.  Damn!

From then on, there were more excuses as I kept repeating the same questions from one booth to another – tell me what you do in the cloud? Right now, I decided to do a pie chart of how I assessed the exhibition lobby floor.

 

I went on. There were about 15 booths. With exception of Falconstor, only one booth managed to tell me some decent stuff. They were KumoWorks and the guy spoke well about their Cloud Desktop with Citrix and iGel thin client. And they are from Singapore. It figures!

I cannot but to feel nauseated by most of the booths at the OneWorld Hotel exhibition lobby. If this is the state our “Cloud Service Providers”, I think we are in deep sh*t. Whitewashing aside and over using the word “Cloud” everywhere is one thing. These guys don’t even know what they are talking about. It is about time we admit that the Singaporeans are better than us. Even they might not know their stuff well, at least they know how to package the whole thing and BS to me intelligently!

And I learned a new “as-a-Service” today. One cloud consultant introduced me to “Application-as-a-Service”. I was so tempted to call it “Ass“.