The use of LLMs to annotate data in management research: Foundational guidelines and warnings

Deskripsi

The emergence of large language models (LLMs) offers new opportunities for AI integration in research, particularly for data annotation and text classification. However, researchers lack guidance on implementation best practices, as the benefits and risks of these tools remain poorly understood. We develop a foundational framework for effective LLM implementation in management research, providing structured guidance on key decisions throughout the research process. We illustrate this framework through an empirical application: classifying sustainability claims in crowdfunding projects. While LLMs can match or exceed traditional methods' performance at lower cost, we find that variations in prompt design can significantly affect results and downstream analyses. We develop procedures for sensitivity analysis and provide detailed documentation to help researchers implement robustness while maintaining methodological integrity.