AI Summary: This feature request proposes adding a new dataset-level check function to validate data ingestion processes. The primary goal is to ensure that the number of records ingested accurately matches the upstream source, addressing a critical need in ingestion scenarios. This validation mechanism would support verifying metrics like row counts or executing custom SQL logic against upstream tables.
Databricks framework to validate Data Quality of pySpark DataFrames and Tables