One of the great parts about being at the frontier of technology is being able to rollout some pretty unique gear. We have always invested heavily in good technology, which in turn helps deliver fast and reliable systems.
As part of this, we have been testing and planning a clustered storage platform. This clustered storage is based on Parallels Cloud Storage (PCS), which allows the use of off-the-shelf servers as a distributed storage platform. Traditionally, large scale file systems used to be implemented via a Storage Area Network (SAN) which although effective it heavily relies on vendor lockin and is extremely costly.
Currently, Conetix utilises a combination of SAN and Direct Attached Storage (DAS) to meet our current storage requirements. By migrating our storage to PCS, we will achieve greater speeds and greater redundancy yet at a lower cost then expanding our existing SAN. As a customer, this means that you will have greater protection against disk failure as well as maintaining our high performance.
As another advantage, it allows many options for High Availability (HA) for our running Virtual Private Servers (VPS). We can move VPS’s between physical servers in under 10 seconds with no downtime, allowing for rapid recovery and load balancing.
System Design
As per the Parallels recommendations, we’re using a three replica based design. This means that every block of data is distributed to a minimum of three other storage nodes. What this provides is the ability to lose up to two nodes in the cluster yet the storage will still remain online.
This is separate to our backups of course, which again adds another layer of data integrity to ensure your data isn’t lost. We run daily backups of all systems, which is stored on a separate storage network.
The other important part of the rollout is ensuring the network infrastructure is up to the task. As anyone who’s worked with large networks will know, there’s a lot of complexity that can quickly bring things undone in a hurry. Adding redundancy and “stacking” the switches then adds yet another layer on top of this. Like the servers, we have stuck with Dell kit here to ensure compatibility and we’re familiar with the Dell configuration and performance.
Testing
Before rolling out any new platform, we run through a number of tests and benchmarks to ensure the stability and performance remain as expected. Testing a server failure in a test environment over and over using different scenarios has given us some great confidence in the system. It’s also meant that we have been able to document our procedures for disaster recovery so that in the event of a real failure, we have proven methods of recovery.
Initial benchmarks look very healthy, we’ve run a few tests and so far we’re able to peak around 2.5GB/s write, with acceptable latency around 2GB/s. This is with a 64 thread read / write workload, which is far more than we expect to see in typical scenarios.
Even when deliberately degrading the cluster by switching a node off, we’re not seeing any significant change in performance. This is is exactly what we wanted to see!
Monitoring
Lastly, we need to keep a very close eye on how the system performs via some internal monitoring systems. We use a combination of commercial software (such as PRTG) as well as our own custom developed system. This gives us not only fault tolerant monitoring but differing data to help track and narrow down if an issue occurs.
Rest assured, with 3 replicas of your data on our cluster combined with 4 hour warranty response from Dell we can ensure the cluster does not go offline. Our alerting has multiple escalation points as well as notifying multiple staff members. Monitoring has always been a strong focus at Conetix, we always strive to ensure any faults or performance variances are quickly dealt with.
Conclusion
We’re really excited by this system and the opportunities that it will offer our clients. Combining high performance Dell systems along with high levels of redundancy is a “no compromise” solution that has previously been very costly to implement.
Progressively over 2014, we’ll be rolling both new and existing nodes into our cluster to increase performance and reliability even further. We’re also testing Solid State Drive (SSD) acceleration, which will allow us to reach beyond 100,000 IOPS.
