Grizzly: Efficient Stream Processing Through Adaptive Query Compilation

Philipp M. Grulich, Sebastian Breß, Steffen Zeuch, Jonas Traub, Janis von Bleichert, Zongxiong Chen, Tilmann Rabl, Volker Markl

Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD 2020) | June 2020

Abstract:

Stream Processing Engines (SPEs) execute long-running queries on unbounded data streams. They rely on managed runtimes, an interpretation-based processing model, and do not perform runtime optimizations. Recent research states that this limits the utilization of modern hardware and neglects changing data characteristics at runtime. In this paper, we present Grizzly, a novel adaptive querycompilation-based SPE to enable highly efficient query execution on modern hardware. We extend query-compilation and task-based parallelization for the unique requirements of stream processing and apply adaptive compilation to enable runtime re-optimizations. The combination of light-weight statistic gathering with just-in-time compilation enables Grizzly to dynamically adjust to changing data-characteristics at runtime. Our experiments show that Grizzly achieves up to an order of magnitude higher throughput and lower latency compared to state-of-the-art interpretation-based SPEs.

Bibtex:

 
@inproceedings{grulich2020grizzly,
  title={Grizzly: Efficient Stream Processing Through Adaptive Query Compilation},
  author={
    Grulich, Philipp M and Bre{\ss}, Sebastian and Zeuch, 
    Steffen and Traub, Jonas and Bleichert, Janis von and Chen, Zongxiong and 
    Rabl, Tilmann and Markl, Volker
  },
  booktitle = {Proceedings of the ACM SIGMOD
     International Conference on Management of Data (SIGMOD 2020)},
  year = {2020},
  month = {June},
  publisher = {ACM SIGMOD},
  edition = {SIGMOD},
}

View Paper Code