Yes, Yes, I have been inactive for almost 2 months. There were many things I had to do to put my business back into shape again, and hence my lack of activities in my blog.
Yes, Yes, I have a lot of catching up to do, but first I would like to report that one of the more prominent web hosting companies (many of who frequently brand themselves as “Cloud” companies) in Malaysia have been hacked.
I got the news at about 8.00am on September 28th morning and I was in Bangalore, India. Friend of mine buzzed me on Facebook Messenger, and shared with me the following:
Thursday, September 27, 2012 1:46 AM Date: 27th Sep 2012 Time: 6.01PM GMT +0800 We have an intrusion incident that happened early this morning around 12midnight of 27th September 2012. About 50 customers’ Virtual Machines hosted on our CLOUD were deleted from the cloud server. When we spotted the abnormal behavior, we managed to stop the intruder from causing more damages to our system. From our initial investigation, we suspect one of our employees who will leave the company at this month end logged into one of our control panels and deleted some Virtual Machines. The backup was terminated at the same time when the Virtual Machines were deleted. At this point of time, our team is working relentlessly on restoring the affected virtual machines and customer data. In the mean time, my COO is lodging a police report and my manager is lodging a report to MyCERT while I am writing this email. We are truly sorry about the whole incident as it has caused a great deal of inconvenience to our customers and their end customers as well. Please also be rest assured that our CLOUD is truly secured; this incident was not a successful hacking attempt but rather sabotage via an ordinary login. Detailed investigation reports will be compiled and sent to our customers. Sincerely, Chan Kee Siak Founder and CEO =================================== Summary / History of issues: =================================== 27th Sep 2012, 1.00am: - We detected several virtual machines on the cloud were throwing warning signals. - Technical Managers were immediately informed. 01.30am: - We found out that an intruder was attempting to delete some of the virtual machines on our CLOUD cluster. - The intruder was using a valid login to access our CLOUD control panel. - COO was informed, signed in to co-ordinate. - The access of the intruder has been disabled to prevent further damage. - We posted an announcement at: https://support.exabytes.com.my/News/2248/c...aintenance.aspx 02.00am: - CEO was informed. - We found out that the intruder was using the login ID and password which belonged to one of the staff members whom we had recently sent out termination notice. The last working day of this staff was end of this month. - Around 50++ Virtual Machines / VPS were affected. - We started to inform affected customers. 02.30am: - Rebuild and restoration of virtual machines began. 10.00am: - Some Virtual Machines were Restored. The rest were still pending, on going. - For Virtual machines without extra R1Soft Backup, we have recreated blank virtual machines with Operating System. 12:30pm: - Attempted to recover the deleted backup on the CLOUD Backup server via data recovery tool. No guarantee and no ETA yet, we were doing our very best. 5.39pm: - 80% of virtual machines were recreated. However, some were without the latest backup of data. - Our engineers were attempting to recover the Cloud Backup Hard Drive with the use of recovery tool. However, as the size was huge, it might take few more hours. Damage: - The CLOUD Accounts, Virtual Machines and CLOUD Backup of affected clients were deleted. Only client with additional R1Soft backup still has the recent backup. ================================= Date: 27th September 2012 Time: 1:55 AM GMT+8 Maintenance Details: We have been alert by our monitoring system that certain Cloud VM has been found to be inaccessible. Our senior admin engineers are now working to resolve the issues. Maintenance effect: VMs affected isolated under MY-CLOUD-02 Zone. We regret for any inconveniences caused. Best regards, Support team ------------------ Technical Support Department.
My friend said there were loss of data and there was no backup. Whoa! No backup? No freaking backup?
First thing I told my friend was to check the Service Level Agreement (SLA). What did the SLA said? Well, … here is what my friend shared with me and apparently from Exabytes website.
"Circumstances beyond our reasonable control, including, without limitation, acts of God, acts of any governmental body, war, insurrection, terrorism, SABOTAGE, armed conflict, embargo, fire, flood, strike or other labor disturbance, interruption of or delay in transportation, unavailability of or interruption or delay in telecommunications or third party services, virus attacks or hackers, failure of third party software (including, without limitation, ecommerce software, payment gateways, chat, statistics or free scripts) or inability to obtain raw materials, supplies, or power used in or equipment needed for provision of this SLA"
Well, well, well. I am not going to scrutinize the SLA. I will let the legal eagles (aka lawyers) do their best to interpret it but to me, that pretty much indemnify the web hosting company from everything, doesn’t it?
At the flick of the mind, my friend might be the victim and that goes for pretty everybody else who is affected by the disaster. But from another point of view, it is really a case of “willing buyer, willing seller”. I am pretty sure most customers of Exabytes signed up because they were attracted by the “cheap” Exabytes offerings, but didn’t take the time to read the fine print (aka the SLA). Therefore, when they sign on as a customer of Exabytes, they are ignorantly and definitely in the most apathy fashion, accepted the risk prescribed by the SLA. In Malay, this attitude is called “tidak apa” or “don’t fucking care“.
Yes, that attitude is pretty much with many of us Malaysians because we take many things for granted … UNTIL … disaster strikes. Then it is a case of trying to find blame, often lacking the intelligence and gumption to point the finger back at ourselves.
But there are 2 sides to the coin. If you dissect the SLA above in a logical manner, there is also a lack of responsibility and duty of the provider to the customer. And I believe this may be true with the other “cloud” providers in Malaysia as well because there lacks definite guidelines, standard operating procedures, regulations and enforcement to this sort of things.
Who can the “victimized” customer turn to? My first thought was to MCMC, the Malaysia Communications and Multimedia Commission but I was only guessing. The other thought was the Internet Alliance in Malaysia, but then again, I don’t really know what they do except that it is being headed by the boss of IPServer One.
All I can for now is as digital customers in this whole new whole, we have to demand higher standards, demand better protection, demand greater professionalism in everything that affects us, especially in the new Cloud Economy. If we continue the “tidak apa” attitude, view and accept things as they are with apathy, all I can say is “Good luck and good riddance”.
(Note: I want to set the record that this post is not intend to put Exabytes, the web hosting company in a bad light. This could happen to anyone. It is the intention of this post to let the readers know and learn from the disaster, and use the incident as a platform to set higher standards, higher goals to improve data protection, and reduce data loss)
One last advice. It’s OK to be cheap as long as we know the risks and we do our homework to mitigate the risks. It’s NOT OK to be cheap when you are dumb, and blame others. Good luck!
It was not hacked. It was internal sabotage. Please do not simply misused the word “hack”
Dear sir/madam behind the name anonymous,
The definition of hacking can mean different things to different people. I acknowledged you have a different definition of the “hack” in your perspective. Perhaps you can define your meaning of “hack” to share your version.
The security of the company in question may have been breached from an operational process perspective and a disgruntled employee (or ex-employee) had the means (i.e. the right authentication and authorization) to renege his/her professionalism to the company’s data. That action(s), has in fact, violated the in-place security process (or lack of) and that’s in my definition, is hacking with the malicious intent.
You have the right not to accept my version or view. That is my response to you to agree to disagree.
Thank you
/Chin-Fah