Center for Social Data Analytics Colloquium-Yaoyao Dai and Ben Radford, UNC Charlotte

Yaoyao Dai and Ben Radford

Talk title: "Large Language Models for Measuring Contested and
Multi-dimensional Concepts"

Abstract:
Large language models (LLMs) are deep neural network models pre-trained on massive amounts of text
data. These models have demonstrated remarkable success in transferring linguistic knowledge learned in
training to various downstream applications. However, one common concern when directly applying
LLMs to the political science domain is the often complex and contested definitions of the target concepts,
especially when the scholarly understanding of a concept may differ from the general audience writings
that those LLMs were trained on. In this paper, we evaluate five popular zero-shot LLMs’ performances
in measuring populism, one of the most contested and popular concepts in academic and general public
discussion in the past two decades. We compare this zero-shot approach with expert coding and a finetuned classifier. We find that ChatGPT 4o overall gives comparable results to expert coding and supervised
learning. Predicting populism directly using LLMs can yield a measure with low face validity. However,
with careful prompting, LLMs are able to provide measures better aligned with the scholarly understanding
of a contested concept.