AoE – All about Ethernet!

This is long overdue.

A reader of my blog asked if I could do a piece on Coraid. Coraid who?

This name is probably a name not many people heard of in Malaysia. Even most the storage guys that I talk to never heard of it.

I have known about Coraid for a few years now (thanks to my incessant reading habits), looking at it from nonchalant point of view.  But when the reader asked about Coraid, I contacted Kevin Brown, CEO of Coraid, whom I am not exactly sure how I was connected through LinkedIn. Kevin was very responsive and got one of their Directors to contact me. Kaushik Shirhatti was his name and he was very passionate to share their Coraid technology with me. Thanks Kevin and Kaushik!

That was months ago but the thought of writing this blog post has been lingering. I had to scratch the itch. 😉

So, what’s up with Coraid? I can tell that they are different but seems to me that their entire storage architecture is so simple that it takes a bit of time for even storage guys to wrap their head around it. Why do I say that?

For storage guys (like me), we are used to layers. One of the memorable movie quotes I recalled was from Shrek: “Orges are like onions! Onions have layers!“.

When we dissect the storage networking topology, we have the RAID layer, the abstraction into LUNs layer, the network layer, the session layer, the zoning layer, the masking and mapping layers, and so on, and so forth. You know what I mean. Those are the constructs of a typical SAN architecture. When iSCSI came along more than a decade ago, that layering gotten thicker with more layers, but somehow the customers adopted it. iSCSI, love it or loathe it, has been eating up the Fibre Channel SAN market in the past few years.

Why? I believe the familiarity of “local disk” kept the customers feeling good, within their comfort zone. “It looks and feels like a D: drive”. iSCSI runs on the Ethernet network, not that complex Fibre Channel stuff.

And this is helped by applications such as databases which want and demand local ownership of the storage space. So, the local disk phenomenon continues, despite our usual grouches and complaints about iSCSI performance and security, among others.

But those layers bothers us. Between the compute layer and the storage layer, there are just too many “layers” of ensuring data delivery, consistency, protection, etc, etc. The storage layer is too far apart from the compute layer and in this era of virtualization and cloud, the storage layer must be as close as possible to the compute layer. No wonder we storage guys get blamed for not delivering fast I/O to the compute layer. We have too many layers in between. Maybe we should rope in the network guys to be blamed as well because that’s the layer in between!

No wonder some vendors (VMware included with its VSA architecture) are beginning to reignite the DAS (direct-attached storage) architecture again, coupling it with scale-out and global namespace features.

Well, Coraid is a very good fit into the requirements SAN guys might love. Here are a few I can think of:

  • Lesser layers resulting in higher data delivery throughput and lower latency in response time
  • Higher security because IP (Internet Protocol) is less involved. We all know the proliferation of IP, but it is also where the exposure to routing that is the bane of security concerns when it comes to networked storage
  • Removing traditional SAN implementation such as multipathing, zoning, firewalls etc will inadvertently reduce cost of managing the storage architecture
  • Local disk flavour reduces fear and still dances within the customers’ comfort zone

Coraid unique technology is AoE (ATA-over-Ethernet). They are the inventor of the technology and basically it is what it is. It has the ability to deliver ATA and SCSI protocols over the Ethernet communication at Layer 2. It does not use IP.

Here’s a screenshot I took from Wikipedia about the structure of AoE.

AoE Ethernet type is 0x88A2.

The whole Ethernet SAN architecture, as prescribed by Coraid, is still pretty much the initiator and target concepts, with no architectural change to the Ethernet network. Coraid supplies the HBA (which I was informed is a reflashed Gigabit Ethernet NIC), the AoE driver (easy to install and available for Windows and Linux platforms), and the Coraid EtherDrive SRX storage array. The SRX supports SATA, SAS and SSD drives mixed within the appliance’s chassis.

We know well enough we should never bet against Ethernet, so the future looks bright as the Ethernet speed is moving towards 40Gbit/sec and 100Gbit/sec mainstream is not too far away.

There is no need for multipathing as the AoE driver at the host and OS level installs the AoE mapping layer, which essentially dices all data blocks into 8KB chunks and tracks the sending, the acknowledgment and the receiving of these chunks between the AoE HBA and the SRX storage array. It uses “configuration strings” to ensure consistency and implements a “host-based cooperative locking“.

According to the description from Wikipedia, when more than 1 initiator is accessing the AoE target (the SRX appliance), the AoE target acts as an arbitrator to record and keeps track of which host is accessing the storage blocks at a particular point in time. If there is a conflict, only one access is accepted and the other host is informed of the conflict. The AoE mapping layer will then handle the retransmit if required.

Each segmented 8KB blocks are sent over multiple Ethernet frames, through all available AoE HBA ports as a unicast link. This method ensures the highest possible I/O throughput to the SRX storage array. Coraid claims that their SRX appliance can deliver up to 1,800 MB/sec throughput and 200,000 IOPS in a single 4U appliance.

But my reader wanted to know about Coraid replication technology. Coraid implements remote data copy (which can be viewed as a replication or mirroring technology) in 2 possible ways (excluding host-based replication/mirroring) at their technology’s level.

One of the methods is to introduce the Coraid EtherDrive VSX. Here’s a high level look at the VSX architecture implemented with SRX (courtesy of Coraid):

The EtherDrive VSX appliance integrates multiple EtherDrive SRX appliances and allows block-level synchronous mirroring and asynchronous remote replication. In addition to that, the VSX is also responsible for implementing storage virtualization, snapshots, cloning and thin provisioning.

Hence, their architecture is rather different than other more traditional storage vendors where all these technologies such as replication, snapshots and thin provisioning are all integrated into their storage array. By separating the VSX and the SRX, I can only assume that this will give Coraid the ability to adapt their architecture in a much more flexible manner, leaving the EtherDrive SRX appliance to do what it does best – which is to deliver local drive storage in the high performing manner with the lowest latency through the Ethernet architecture.

The world is changing and the demands of delivering unified storage is on everybody’s lips. So, in July 2012, if I am not wrong about the date, Coraid has licensed ZFS to deliver high-performance NAS with their Coraid ZX appliance. Here’s a high level diagram I got from Coraid’s website.

As you know, I am big fan of ZFS. (read my blog post here). I think by introducing ZFS into their offering, Coraid has completed their portfolio of storage solutions. Of course, ZFS also adds another data replication method. This is the second way that Coraid can deliver remote replication in a disaster recovery situation. Kaushik also shared this slide with me:

Coupled with the acquisition of Yunteq in late 2011, Coraid is integrating their Storage Manager management interface into EtherCloud, their all-encompassing storage management interface. According to IDC, in which I quote:

Coraid EtherCloud is designed to make every aspect of storage management and
control programmable using a REST API. Using the API, administrators can easily
automate workflows related to storage provisioning and management, and integrate
the management of storage with the rest of the infrastructure.

EtherCloud consists of a policy engine that allows storage to be dynamically allocated
and managed according to application requirements and business policies with endto-
end visibility and control. This allows IT administrators to reduce infrastructure
provisioning times significantly and provide a programmable platform to integrate
storage resources with networking and computing resources. EtherCloud also allows
application owners to deploy storage with one-click provisioning in a multitenant
environment without having to deal with the complexity of underlying storage

The overall Coraid architecture is shown below:

Coraid is indeed evolving moving into quite a complete storage solution. As Kevin Brown, Coraid CEO once quoted “We use raw Ethernet for a SAN”.

Indeed AoE is All About Ethernet.

p/s: I just want it clear that I am not paid write this. I do not owe any favour to anyone but I wanted to share with my readers (especially in Malaysia) that there is an interesting storage technology and we should not fear learning about it. 

Tagged , , , , , , . Bookmark the permalink.

About cfheoh

I am a technology blogger with 30 years of IT experience. I write heavily on technologies related to storage networking and data management because those are my areas of interest and expertise. I introduce technologies with the objectives to get readers to know the facts and use that knowledge to cut through the marketing hypes, FUD (fear, uncertainty and doubt) and other fancy stuff. Only then, there will be progress. I am involved in SNIA (Storage Networking Industry Association) and between 2013-2015, I was SNIA South Asia & SNIA Malaysia non-voting representation to SNIA Technical Council. I currently employed at iXsystems as their General Manager for Asia Pacific Japan.

16 Responses to AoE – All about Ethernet!

  1. Tracy Reed says:

    I love AoE for the simplicity and have deployed a lot of it over the years since around 2005. I don’t use it so much anymore because I have been pretty much forced over to iSCSI. Nobody trusts ATA/IDE, however irrational that mistrust is. Everyone thinks SCSI=Enterprise. Plus AoE hasn’t integrated very well with the Linux distros. It is certainly fast and easy though.

    • cfheoh says:

      Hi Tracy

      Thanks for sharing your experience.

      Out of curiosity, in your part of the world, how is the momentum going for Coraid and AoE? I don’t think it will be out there challenging the iSCSI and FC market, but is there a strong market interest?

      I know that Coraid had recently opened an office in Sydney and they may be considering South East Asia, where I am at. I am keen to work with/for them if something good is going to happen.

      Looking forward to your reply. Thank you


      • Tracy Reed says:

        Unfortunately, I see no momentum for Coraid/AoE in the US. I have never met anyone else who used AoE. Coraid says they are having success in niche industries which I am not involved in. I think AoE has lots of potential but Coraid has utterly failed to capitalize on it.

        A few months ago I wrote an email to the AoE mailing list explaining some of my difficulties with AoE. Ed Cashin of Coraid replied. You will probably find it interesting:

        • cfheoh says:

          Hi Tracy

          Thanks for sharing your experience and frustration.

          Some guy from Jakarta, Indonesia said he’s been using AoE target for years and loving it. He has 2,000 clients over 100 offices. Like you said in your threads, the licensing model is iffy, because many could built the storage target themselves. Also, friend of mine told me there was a Coraid requirement in a government tender in Brunei a couple of years back.

          So, Coraid’s existence is spotty and shows up in unexpected places. And it is hard to convince people when the technology is not mainstream and different.

          These information are useful next time Coraid speaks with me.

          Have a great weekend! Thanks

  2. ynchow says:

    Chin Fah, I am probably one of the exception. I get to know about Coraid two years back when I saw AoE in on of the tender spec for the Brunei government. Just like what you had predicted, we responded with iSCSI storage.

    • cfheoh says:

      Hi Chow

      Nice to hear from you. Didn’t know you were reading my blog post. 🙂

      Yes, Coraid is not well known in this part of the world. I know they are expanding into this region, and had set up an office in Australia about 6 months ago. I have gotten to know their Sales Director over the last few concalls, and he was asking around for resellers/partners etc. Would you be interested?

      Please share my blog to your friends, partners and associates. I hope to have the blog as a chockful of information for everyone interested.

      Thank you

  3. Jgan says:

    Simple is good 🙂

  4. Halim says:

    Just to share, I used AoE technology since 2007 until now on all Linux servers serving Windows & Linux clients.
    We did have deployed AoE to over than 100 offices for over than 2000 workstations throughout Indonesia, and some smaller numbers on other countries.
    No doubt at all, very fast, works very well, robust and low cost.
    Thumbs up !

    • cfheoh says:

      Hi Halim,

      Glad that you are using AoE technology in Indonesia and in other countries, albeit a smaller scale. Thanks for sharing your experience with us. I presume that you have built the AoE target on the Linux servers yourself.

      All the best to you and thank you

  5. Hi Tracy,

    IMHO and base on your post… I think it could have been fair if you say you “ONLY” used software implementation of AoE target and not any Coraid hardware appliances… (main subject of this post is focus on Coraid goods and not on opensource contribs).

    Anyway, regarding AoE initiator and Coraid or community contribs please check commits for a year : huge work from Coraid folks on Linux mainline.

    Best Regards,
    Yacine @Alyseo

  6. garegin says:

    AoE uses SCSI now. They are not using the ATA command set. You can use SCSI disks, SATA disks, it doesn’t matter.

    • cfheoh says:

      Hello Garegin

      Thank you for the comment.

      The presence of SCSI (do you mean SAS disks, since parallel SCSI disks are pretty much gone) and SATA disks should not matter because these are pretty much the backend hardware that handles the circuitry. AoE is relevant in the front-facing part in the Ethernet network and would be directly competing to the iSCSI protocol.

      In fact, AoE is definitely capable carrying SCSI-3 common command sets. The definition of AoE header frame format allows different payloads, hence SCSI should not be a problem.

      BTW, are you a user of AoE? Where are you based? I am interested to know your experience in AoE. Do share!

      Thank you

      • garegin says:

        I watched the presentation of AoE by Coraid on youtube and the guy says that the disks are presented as SCSI to the OS. You obviously can use any type of disk you want.
        No I have not used AoE. Would if I needed to.

  7. Dan says:

    Hi All,

    We use Coraid kit here, we have been since about 2010 before they ventured into the UK.
    We spoke directly with the guys in the US when they were very small and purchased it directly with them.
    We wanted to virtualise our infrastructure and were almost going to buy EMC kit then Coraid popped up. We took a look, bought a second hand unit from eBay, done some tests and decided to go with Coraid for our production virtualisation project.

    We have 2 x SRX3200’s and 2 x VSX appliances that synchronously mirror over our 10GbE fibre link to another building. We use a no single point of failure scenario.

    We have around 50 servers virtualized on this kit including SQL, Exchange and have an additional SRX2800 with some SSD’s on which we are running 80 Windows 7 VDI’s on.

    It was easy to set up, it has good fault tolerance with the VSX appliances and is really speedy, it seems as good as having the disks connected locally to each server in terms of speed and configuration.

    We are a 6th form college based in Grays in the UK and was a big gamble for us with such a new technology but the gamble paid off and saved us a fortune compared to the big boys of storage.


    • cfheoh says:

      Hi Dan

      Thank you for sharing you Coraid experience with us. I am really glad that it turned out well.

      I had several conversations with Coraid in the past and wanted to represent them in Asia but it did not work out. Unfortunately Coraid does not have the type of funding and marketing like the other start-ups.

      Their technology is clean, and more importantly lightweight compared to the more popular iSCSI. And coupled with their ZFS solution, Coraid is more resilient. Mind you, I have built enterprise storage systems with OpenIndiana ZFS on Supermicro servers and they run well. Therefore, I know that Coraid will do well technically.

      Unfortunately, lots of people are still pretty skeptical about Coraid because of their lack of knowledge of their solution and technology. Your confident with Coraid speaks volume about their ability.

      I wish you all the best and thank you.


  8. jjjjkemp says:

    FYI, April 2015, Coraid went bust, $100M+ lost. !,700 customers told they are now high and dry, on their own. Net result is big win for the Open Source community. Software code is now free, and runs on commodity hardware. See

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.