Dataset-On-Demand: Automatic View Search and Presentation for Data Discovery

Abstract

Many data problems are solved when the right view of a combination of datasets is identified. Finding such a view is challenging because of the many tables spread across many databases, data lakes, and cloud storage in modern organizations. Finding relevant tables, and identifying how to combine them is a difficult and time-consuming process that hampers users' productivity. In this paper, we describe Dataset-On-Demand (DoD), a system that lets users specify the schema of the view they want, and have the system find views for them. With many underlying data sources, the number of resulting views for any given query is high, and the burden of choosing the right one is onerous to users. DoD uses a new method, 4C, to reduce the size of the view choice space for users. 4C classifies views into 4 classes: compatible views are exactly the same, contained views present a subsumption relationship, complementary views are unionable, and contradictory views have incompatible values that indicate fundamental differences between views. These 4 classes permit different presentation strategies to reduce the total number of views a user must consider. We evaluate DoD on two key metrics of interest: its ability to reduce the size of the choice space, and the end to end performance. DoD finds all views within minutes, and reduces the number of views presented to users by 2-10x.