CITATION.cff

cff-version: 1.2.0
message: "If you use this work, please cite it as below."
title: "On the alignment of LM language generation and human language comprehension"
authors:
  - family-names: Bolliger
    given-names: Lena Sophia
  - family-names: Haller
    given-names: Patrick
  - family-names: Jäger
    given-names: Lena Ann
editors:
  - family-names: Belinkov
    given-names: Yonatan
  - family-names: Kim
    given-names: Najoung
  - family-names: Jumelet
    given-names: Jaap
  - family-names: Mohebbi
    given-names: Hosein
  - family-names: Mueller
    given-names: Aaron
  - family-names: Chen
    given-names: Hanjie
date-released: 2024-11-01
conference: "Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP"
location: "Miami, Florida, US"
publisher: "Association for Computational Linguistics"
pages: "217-231"
url: "https://aclanthology.org/2024.blackboxnlp-1.14"
abstract: "Previous research on the predictive power (PP) of surprisal and entropy has focused on determining which language models (LMs) generate estimates with the highest PP on reading times, and examining for which populations the PP is strongest. In this study, we leverage eye movement data on texts that were generated using a range of decoding strategies with different LMs. We then extract the transition scores that reflect the models' production rather than comprehension effort. This allows us to investigate the alignment of LM language production and human language comprehension. Our findings reveal that there are differences in the strength of the alignment between reading behavior and certain LM decoding strategies and that this alignment further reflects different stages of language understanding (early, late, or global processes). Although we find lower PP of transition-based measures compared to surprisal and entropy for most decoding strategies, our results provide valuable insights into which decoding strategies impose less processing effort for readers. Our code is available via https://github.com/DiLi-Lab/LM-human-alignment."