The Impact of Question Framing on the Performance of Automatic Occupation Coding

Abstract

Occupational data play a vital role in research, official statistics, and policymaking, yet their collection and accurate classification remain a challenge. This study investigates the effects of occupational question wording on data variability and the performance of automatic coding tools. We conducted and replicated a split-ballot survey experiment in Germany using two common occupational question formats: one focusing on "job title" (Berufsbezeichnung) and another on "berufliche T\"atigkeit" (loosely translated as occupation or occupational task). Our analysis reveals that automatic coding tools, such as CASCOT and OccuCoDe, exhibit sensitivity to the form and origin of the data. Specifically, these tools were more efficient when coding responses to the job title question format than the occupational task format, suggesting a potential way to improve the respective questions for many German surveys. In a subsequent "detailed tasks and duties" question, providing a guiding example prompted respondents to give longer answers without broadening the range of unique words they used. These findings highlight the importance of harmonising survey questions and and ensuring that automatic coding tools are robust to differences in question wording. Further research is needed to optimise question design and coding tools for greater accuracy and applicability in occupational data collection.