Regulatory compliance-readiness in the AI Supply Chain: examining datasets in Hugging Face

Abstract

A very large number of datasets are made available via Hugging Face (HF). These datasets are used to train a vast number of AI models, also available via HF. There are regulatory issues that might arise in the AI supply chain when those datasets do not follow, for instance, data privacy practices, or do not disclose their data sources, cleaning and preparation processes. In this preliminary work, regulatory compliance-readiness is examined as a quality attribute in the framework of HF datasets, with a main focus on data privacy (e.g. GDPR, CCPA). Towards this direction, an analysis on regulatory compliance (e.g. with privacy laws) of the dataset using the dataset card and the dataset structure overview as starting points has been performed. We collected 11,682 HF datasets that have a dataset card, are not gated and have at least 500 downloads and analyzed the datasets using automated techniques and manual analysis on a sample of the dataset. The results show that a very small number of datasets are explicit on the datasets creation processes and mention data sources, while some make mentions to Personally Identifiable Information (PII) or other sensitive data in their data schemas. These results show the need for more detailed examinations of regulatory compliance in AI datasets and models, and research on tools that provide guidance to practitioners and automate the compliance process.