WEIRD Audits? Research Trends, Linguistic and Geographical Disparities in the Algorithm Audits of Online Platforms -- A Systematic Literature Review

Abstract

The increasing reliance on complex algorithmic systems by online platforms has sparked a growing need for algorithm auditing, a methodology evaluating these systems' functionality and impact. In this paper, we systematically review 176 peer-reviewed online platform-focused algorithm auditing studies and identify trends in their methodological approaches, the geographic distribution of authors, and the selection of platforms, languages, geographies, and group-based attributes in the focus of the reviewed research. We find a significant skew of research focus towards few online platforms, Western contexts, particularly the US, and English language data. Additionally, our analysis indicates a tendency to focus on a narrow set of group-based attributes, often operationalized in simplified ways, which might obscure more nuanced aspects of algorithmic bias and discrimination. We provide a clearer understanding of the current state of the online platform-focused algorithm auditing and identify gaps to be addressed for a more inclusive and representative research landscape.