pipette
ESEspañol

Inferring L1 Backgrounds from L2 English with LLM Token Probabilities

Ahmet Yavuz Uluslu, Mark Gales, Kate Knill, Gerold Schneider

Preprint

In the authors' words

Native language identification (NLI) infers a non-native speaker's first language (L1) from their second language (L2) production. We investigate a token probability approach that scores each text sample as a continuation of a prompt prefix specifying a candidate L1, then predicts the candidate with the highest score. We evaluate our approach using Llama 3.1 8B and 70B on three datasets: TOEFL9, Write and Improve 2024, and BULATS-L1. We compare its performance with previous prompting-based classification approaches. Token-probability scoring substantially improves on direct classification, particularly with the 8B model, while the 70B model achieves state-of-the-art performance across all datasets. Our results indicate that pretrained LLMs' token probabilities contain information useful for distinguishing L1 backgrounds from L2 English production.

Main resultThe abstract does not state a limitation.

Appeared: Monday, October 19. arXiv. Preprint, not yet peer-reviewed.