Enabling efficient and general subpopulation analytics in multidimensional data streams
Files
Published version
Date
2022-09-29
Authors
Manousis, Antonis
Cheng, Zhuo
Ben Basat, Ran
Liu, Zaoxing
Sekar, Vyas
Version
Published version
OA Version
Citation
A. Manousis, Z. Cheng, R. Ben Basat, Z. Liu, V. Sekar. 2022. "Enabling Efficient and General Subpopulation Analytics In Multidimensional Data Streams" Proceedings of the VLDB Endowment, Volume 15, Issue 11, pp.3249-3262. https://doi.org/10.14778/3551793.3551867
Abstract
Today's large-scale services (
e.g.
, video streaming platforms, data centers, sensor grids) need diverse real-time summary statistics across multiple subpopulations of multidimensional datasets. However, state-of-the-art frameworks do not offer general and accurate analytics in real time at reasonable costs. The root cause is the combinatorial explosion of data subpopulations and the diversity of summary statistics we need to monitor simultaneously. We present Hydra, an efficient framework for multidimensional analytics that presents a novel combination of using a "sketch of sketches" to avoid the overhead of monitoring exponentially-many subpopulations and universal sketching to ensure accurate estimates for multiple statistics. We build Hydra as an Apache Spark plugin and address practical system challenges to minimize overheads at scale. Across multiple real-world and synthetic multidimensional datasets, we show that Hydra can achieve robust error bounds and is an order of magnitude more efficient in terms of operational cost and memory footprint than existing frameworks (e.g., Spark, Druid) while ensuring interactive estimation times.
Description
License
This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment.