Redis¶
As an architect on the Internet of Things Red Hat ITCM project I was a big supporter of heavyweight AMQP message queue clusters across numerous data centers for achieving true 5 9’s of uptime while focused on no messages getting lost. Since this was a massive safety system used by all railroads, there was a true business need to prevent message loss and mitigate downtime. With a focus on safety + high-availability, the overhead of implementing a massive message queue (MQ) system like that had direct implications on development speed and budgetary requirements to build the solution.
After spending years setting up routemaps and implementing publisher-subscriber (pub-sub) messaging patterns, I have found Redis to serve a great dual purpose of not only caching but also as a very simple publisher-subscriber queueing system. Knowing how expensive it is to build and support highly available 99.999% AMQP solutions, I have created, supported and open sourced multiple variations of highly-available Redis clusters that run in Docker containers that can be distributed across multiple nodes just like RabbitMQ or MRG-M AMQP clusters. I use these Redis clusters for caching and publisher-subsriber messaging every single day.
Redis is the backbone and first lego block I would choose when laying the foundation for a new distributed real-time stack. For the past two years, my Redis servers are utilizing over 30+ TB of data transfer in a month. In May 2016, my AWS bill recorded the data transfer in at 47,567.947 GB. I have tuned and optimized my Redis clusters to never die while achieving a good throughput somewhere around +1 TB/day.
I am indebted to Antirez for his Redis caching technology and Todd Lucas for introducing me to Redis as a means to efficiency cache data.
Load-Balanced, Self-Healing Redis Cluster¶
I created this repository for housing my investment platform’s self-healing real-time stock and news data pipelines. I use it for caching and messaging. It is a load-balanced HAProxy for connectivity across a multi-host 3-node Redis Cluster (no sharding)
To date the only crash has been due to overrunning memory and filling up the hard drive…I find the docker deployment tools to be a fantastic fit for message queue technologies like Redis and RabbitMQ.
6-Node Sharded Redis Cluster¶
I built this repository as a means to distribute a 6-node Redis cluster with sharding. It is the scaled out HAProxy version mentioned above: