Rhino: Efficient Management of Very Large Distributed State for Stream Processing Engines

Bonaventura Del Monte, Steffen Zeuch, Tilmann Rabl, Volker Markl

Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD 2020) | June 2020

Abstract:

Scale-out stream processing engines (SPEs) are powering large big data applications on high velocity data streams. Industrial setups require SPEs to sustain outages, varying data rates, and low-latency processing. SPEs need to transparently reconfigure stateful queries during runtime. However, state-of-the-art SPEs are not ready yet to handle on-the-fly reconfigurations of queries with terabytes of state due to three problems. These are network overhead for state migration, consistency, and overhead on data processing. In this paper, we propose Rhino, a library for efficient reconfigurations of running queries in the presence of very large distributed state. Rhino provides a handover protocol and a state migration protocol to consistently and efficiently migrate stream processing among servers. Overall, our evaluation shows that Rhino scales with state sizes of up to TBs, reconfigures a running query 15 times faster than the state-of-the-art, and reduces latency by three orders of magnitude upon a reconfiguration.

Bibtex:

 
@inproceedings{delmonte_rhino_sigmod2020,
    author = {Del Monte, Bonaventura and Zeuch, Steffen and Rabl, Tilmann and Markl, Volker},
    title = {Rhino: Efficient Management of Very Large Distributed State for Stream Processing Engines},
    year = {2020},
    isbn = {9781450367356},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3318464.3389723},
    doi = {10.1145/3318464.3389723},
    abstract = {Scale-out stream processing engines (SPEs) are powering large big data applications on high velocity data streams. Industrial setups require SPEs to sustain outages, varying data rates, and low-latency processing. SPEs need to transparently reconfigure stateful queries during runtime. However, state-of-the-art SPEs are not ready yet to handle on-the-fly reconfigurations of queries with terabytes of state due to three problems. These are network overhead for state migration, consistency, and overhead on data processing. In this paper, we propose Rhino, a library for efficient reconfigurations of running queries in the presence of very large distributed state. Rhino provides a handover protocol and a state migration protocol to consistently and efficiently migrate stream processing among servers. Overall, our evaluation shows that Rhino scales with state sizes of up to TBs, reconfigures a running query 15 times faster than the state-of-the-art, and reduces latency by three orders of magnitude upon a reconfiguration.},
    booktitle = {Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data},
    pages = {2471–2486},
    numpages = {16},
    keywords = {stateful stream processing, distributed and parallel databases},
    location = {Portland, OR, USA},
    series = {SIGMOD '20}
}

View Paper