[ Disclosure: I was invited by GestaltIT as a delegate to their Storage Field Day 19 event from Jan 22-24, 2020 in the Silicon Valley USA. My expenses, travel, accommodation and conference fees were covered by GestaltIT, the organizer and I was not obligated to blog or promote the vendors’ technologies presented at the event. The content of this blog is of my own opinions and views ]
“Cheap and deep“, “Race to Zero” were some of the less storied calls I have come across when discussing about object storage, and it was really de-valuing the merits of object storage as vendors touted their superficial glory of being in the IDC Marketscape for Object-based Storage 2019.
Almost every single conversation I had in the past 3 years was either explaining what object storage is or “That is cheap storage right?”
[ Disclosure: I was invited by GestaltIT as a delegate to their Storage Field Day 19 event from Jan 22-24, 2020 in the Silicon Valley USA. My expenses, travel, accommodation and conference fees were covered by GestaltIT, the organizer and I was not obligated to blog or promote the vendors’ technologies presented at this event. The content of this blog is of my own opinions and views ]
Cloud computing will have challenges processing data at the outer reach of its tentacles. Edge Computing, as it melds with the Internet of Things (IoT), needs a different approach to data processing and data storage. Data generated at source has to be processed at source, to respond to the event or events which have happened. Cloud Computing, even with 5G networks, has latency that is not sufficient to how an autonomous vehicle react to pedestrians on the road at speed or how a sprinkler system is activated in a fire, or even a fraud detection system to signal money laundering activities as they occur.
Furthermore, not all sensors, devices, and IoT end-points are connected to the cloud at all times. To understand this new way of data processing and data storage, have a look at this video by Jay Kreps, CEO of Confluent for Kafka® to view this new perspective.
Data is continuously and infinitely generated at source, and this data has to be compiled, controlled and consolidated with nanosecond precision. At Storage Field Day 19, an interesting open source project, Pravega, was introduced to the delegates by DellEMC. Pravega is an open source storage framework for streaming data and is part of Project Nautilus.
The data generated at source (end-points, sensors, devices) is serialized, timestamped (as event occurs), continuous and infinite. These are the properties of a time series data stream, and to make sense of the streaming data, new data formats such as Avro, Parquet, Orc pepper the landscape along with the more mature JSON and XML, each with its own strengths and weaknesses.
You can learn more about these data formats in the 2 links below:
Many time series projects started as DIY projects in many organizations. And many of them are still DIY projects in production systems as well. They depend on tribal knowledge, and these databases are tied to an unmanaged storage which is not congruent to the properties of streaming data.
At the storage end, the technologies today still rely on the SAN and NAS protocols, and in recent years, S3, with object storage. Block, file and object storage introduce layers of abstraction which may not be a good fit for streaming data.
A quick reaction leans towards a strange one. Microsoft of all people, buying a chip company? Does it make sense? However, leaning deeper, it starts to make some sense. And I believe the desire is spurred by Amazon Web Services announcement of their Graviton processor at AWS re:Invent last month.
AWS acquired Annapurna Labs in early 2015. From the sources, Annapurna was working on low powered, high performance networking chips for the mid-range market. The key words – lower powered, high performance, mid-range – are certainly the musical notes to the AWS opus. And that would mean the ability for AWS to control their destiny, even at the edge. Continue reading →
It has been on my mind for a long time and I have been avoiding it too. But it is time to face the inevitable and just talk about it. After all, the more open the discussions, the more answers (and questions) will arise, and that is a good thing.
Yes, it is the big elephant in the room called Data Security. And the concern is going to get much worse as the proliferation of edge devices and fog computing, and IoT technobabble goes nuclear.
I have been involved in numerous discussions on IoT (Internet of Things) and Industrial Revolution 4.0. I have been in a consortium for the past 10 months, discussing with several experts of their field to face future with IR4.0. Malaysia just announced its National Policy for Industry 4.0 last week, known as Industry4WRD. Whilst the policy is a policy, there are many thoughts for implementation of IoT devices, edge and fog computing. And the thing that has been bugging me is related to of course, storage, most notably storage and data security.
Storage on the edge devices are likely to be ephemeral, and the data in these storage, transient. We can discuss about persistence in storage at the edge another day, because what I would like to address in the data security in these storage components. That’s the Big Elephant in the room I was relating to.
The more I work with IoT devices and the different frameworks (there are so many of them), I became further enlightened by the need to address data security. The proliferation and exponential multiplication of IoT devices at present and in the coming future have increased the attack vectors many folds. Many of the IoT devices are simplified components lacking the guards of data security and are easily exposed. These components are designed for simplicity and efficiency in mind. Things such as I/O performance, storage management and data security are probably the least important factors, because every single manufacturer and every single vendor are slogging to make their mark and presence in this wild, wild west world.
Picture from https://fcw.com/articles/2018/08/07/comment-iot-physical-risk.aspx
I didn’t get a chance to attend Pure Accelerate event last month. From the blogs and tweets of my friends, Pure Accelerate was an awesome event. When I got the email invitation for the localized Pure Live! event in Kuala Lumpur, I told myself that I have to attend the event.
The event was yesterday, and I was not disappointed. Coming off a strong fiscal Q1 2018, it has appeared that Pure Storage has gotten many things together, chugging full steam at all fronts.
When Pure Storage first come out, I was one of the early bloggers who took a fancy of them. My 2011 blog mentioned the storage luminaries in their team. Since then, they have come a long way. And it was apt that on the same morning yesterday, the latest Gartner Magic Quadrant for Solid State Arrays 2017 was released.
Almost every single conversation I had in the past 3 years was either explaining what object storage is or “That is cheap storage right?”
