WimPi: In-Memory Analytics on a “Wimpy” Cluster
In the data management research community, it is common practice to use the latest and greatest hardware in order to demonstrate orders-of-magnitude performance improvements while ignoring the (often very hefty) associated price tag. Real-world deployments follow suit, using expensive setups that cost even more to power and cool.
We challenge the traditional wisdom that high-end hardware is absolutely necessary for state-of-the-art performance and instead propose a radically different architecture based on low-end micro-servers. While others have previously explored similar ideas for less data-intensive tasks (e.g., key-value stores), common belief is that the typical features of this hardware (e.g., less powerful CPUs, smaller RAM) are not suitable for in-memory analytics workloads.
To demonstrate the feasibility of our approach, we built WimPi, a prototype micro-server cluster that consists solely of Raspberry Pi 3B+ nodes. We believe that this novel architecture has the potential to offer competitive performance with orders-of-magnitude cost savings when compared to traditional setups.