publications

2024

1. NAACL

   Llama meets EU: Investigating the European Political Spectrum through the Lens of LLMs

   Chalkidis I.*, and Brandl S.*

   2024

   Abs PDF TL;DR

   Instruction-finetuned Large Language Models inherit clear political leanings that have been shown to influence downstream task performance. We expand this line of research beyond the two-party system in the US and audit Llama Chat in the context of EU politics in various settings to analyze the model’s political knowledge and its ability to reason in context. We adapt, i.e., further fine-tune, Llama Chat on speeches of individual euro-parties from debates in the European Parliament to reevaluate its political leaning based on the EUandI questionnaire. Llama Chat shows considerable knowledge of national parties’ positions and is capable of reasoning in context. The adapted, party-specific, models are substantially re-aligned towards respective positions which we see as a starting point for using chat-based LLMs as data-driven conversational engines to assist research in political science.

2. LREC-COLING

   Evaluating Webcam-based Gaze Data as an Alternative for Human Rationale Annotations

   Brandl S., Eberle O., Ribeiro T., Søgaard A., and Hollenstein N.

   2024

   Abs PDF TL;DR

   Rationales in the form of manually annotated input spans usually serve as ground truth when evaluating explainability methods in NLP. They are, however, time-consuming and often biased by the annotation process. In this paper, we debate whether human gaze, in the form of webcam-based eye-tracking recordings, poses a valid alternative when evaluating importance scores. We evaluate the additional information provided by gaze data, such as total reading times, gaze entropy, and decoding accuracy with respect to human rationale annotations. We compare WebQAmGaze, a multilingual dataset for information-seeking QA, with attention and explainability-based importance scores for 4 different multilingual Transformer-based language models (mBERT, distil-mBERT, XLMR, and XLMR-L) and 3 languages (English, Spanish, and German). Our pipeline can easily be applied to other tasks and languages. Our findings suggest that gaze data offers valuable linguistic insights that could be leveraged to infer task difficulty and further show a comparable ranking of explainability methods to that of human rationales.

2023

1. EMNLP

   Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models

   Cabello L., Bugliarello E., Brandl S., and Elliott D.

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Dec 2023

   Abs PDF Code Poster

   Pretrained machine learning models are known to perpetuate and even amplify existing biases in data, which can result in unfair outcomes that ultimately impact user experience. Therefore, it is crucial to understand the mechanisms behind those prejudicial biases to ensure that model performance does not result in discriminatory behaviour toward certain groups or populations. In this work, we define gender bias as our case study. We quantify bias amplification in pretraining and after fine-tuning on three families of vision-and-language models. We investigate the connection, if any, between the two learning stages, and evaluate how bias amplification reflects on model performance. Overall, we find that bias amplification in pretraining and after fine-tuning are independent. We then examine the effect of continued pretraining on gender-neutral data, finding that this reduces group disparities, i.e., promotes fairness, on VQAv2 and retrieval tasks without significantly compromising task performance.

2. arxiv

   On the Interplay between Fairness and Explainability

   Brandl S., Bugliarello E., and Chalkidis I.

   Dec 2023

   Abs PDF TL;DR

   In order to build reliable and trustworthy NLP applications, models need to be both fair across different demographics and explainable. Usually these two objectives, fairness and explainability, are optimized and/or examined independently of each other. Instead, we argue that forthcoming, trustworthy NLP systems should consider both. In this work, we perform a first study to understand how they influence each other: do fair(er) models rely on more plausible rationales? and vice versa. To this end, we conduct experiments on two English multi-class text classification datasets, BIOS and ECtHR, that provide information on gender and nationality, respectively, as well as human-annotated rationales. We fine-tune pre-trained language models with several methods for (i) bias mitigation, which aims to improve fairness; (ii) rationale extraction, which aims to produce plausible explanations. We find that bias mitigation algorithms do not always lead to fairer models. Moreover, we discover that empirical fairness and explainability are orthogonal.

3. EMNLP

   Rather a Nurse than a Physician – Contrastive Explanations under Investigation

   Eberle O.*, Chalkidis I.*, Cabello L., and Brandl S.

   In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Dec 2023

   Abs PDF TL;DR Code Poster

   Contrastive explanations, where one decision is explained in contrast to another, are supposed to be closer to how humans explain a decision than non-contrastive explanations, where the decision is not necessarily referenced to an alternative. This claim has never been empirically validated. We analyze four English text-classification datasets (SST2, DynaSent, BIOS and DBpedia-Animals). We fine-tune and extract explanations from three different models (RoBERTa, GTP-2, and T5), each in three different sizes and apply three post-hoc explainability methods (LRP, GradientxInput, GradNorm). We furthermore collect and release human rationale annotations for a subset of 100 samples from the BIOS dataset for contrastive and non-contrastive settings. A cross-comparison between model-based rationales and human annotations, both in contrastive and non-contrastive settings, yields a high agreement between the two settings for models as well as for humans. Moreover, model-based explanations computed in both settings align equally well with human rationales. Thus, we empirically find that humans do not necessarily explain in a contrastive manner.

4. arxiv

   WebQAmGaze: A Multilingual Webcam Eye-Tracking-While-Reading Dataset

   Ribeiro T., Brandl S., Søgaard A., and Hollenstein N.

   Dec 2023

   Abs PDF

   We create WebQAmGaze, a multilingual low-cost eye-tracking-while-reading dataset, designed to support the development of fair and transparent NLP models. WebQAmGaze includes webcam eye-tracking data from 332 participants naturally reading English, Spanish, and German texts. Each participant performs two reading tasks composed of five texts, a normal reading and an information-seeking task. After preprocessing the data, we find that fixations on relevant spans seem to indicate correctness when answering the comprehension questions. Additionally, we perform a comparative analysis of the data collected to high-quality eye-tracking data. The results show a moderate correlation between the features obtained with the webcam-ET compared to those of a commercial ET device. We believe this data can advance webcam-based reading studies and open a way to cheaper and more accessible data collection. WebQAmGaze is useful to learn about the cognitive processes behind question answering (QA) and to apply these insights to computational models of language understanding.

5. TACL

   Domain-Specific Word Embeddings with Structure Prediction

   Lassner D.*, Brandl S.*, Baillot A., and Nakajima S.

   Transactions of the Association for Computational Linguistics Mar 2023

   Abs PDF TL;DR Code Poster

   Complementary to finding good general word embeddings, an important question for representation learning is to find dynamic word embeddings, for example, across time or domain. Current methods do not offer a way to use or predict information on structure between sub-corpora, time or domain and dynamic embeddings can only be compared after post-alignment. We propose novel word embedding methods that provide general word representations for the whole corpus, domain- specific representations for each sub-corpus, sub-corpus structure, and embedding alignment simultaneously. We present an empirical evaluation on New York Times articles and two English Wikipedia datasets with articles on science and philosophy. Our method, called Word2Vec with Structure Prediction (W2VPred), provides better performance than baselines in terms of the general analogy tests, domain-specific analogy tests, and multiple specific word embedding evaluations as well as structure prediction performance when no structure is given a priori. As a use case in the field of Digital Humanities we demonstrate how to raise novel research questions for high literature from the German Text Archive.

2022

1. AACL

   Every word counts: A multilingual analysis of individual human alignment with model attention

   Brandl S., and Hollenstein N.

   In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) 2022

   Abs PDF TL;DR Code

   Human fixation patterns have been shown to correlate strongly with Transformer-based attention. Those correlation analyses are usually carried out without taking into account individual differences between participants and are mostly done on monolingual datasets making it difficult to generalise findings. In this paper, we analyse eye-tracking data from speakers of 13 different languages reading both in their native language (L1) and in English as language learners (L2). We find considerable differences between languages but also that individual reading behaviour such as skipping rate, total reading time and vocabulary knowledge (LexTALE) influence the alignment between humans and models to an extent that should be considered in future studies.

2. CLASP

   A Cross-lingual Comparison of Human and Model Relative Word Importance

   Morger F., Brandl S., Beinborn L., and Hollenstein N.

   In Proceedings of the 2022 CLASP Conference on (Dis)embodiment Sep 2022

   Abs Code

   Relative word importance is a key metric for natural language processing. In this work, we compare human and model relative word importance to investigate if pretrained neural language models focus on the same words as humans cross-lingually. We perform an extensive study using several importance metrics (gradient-based saliency and attention-based) in monolingual and multilingual models, including eye-tracking corpora from four languages (German, Dutch, English, and Russian). We find that gradient-based saliency, first-layer attention, and attention flow correlate strongly with human eye-tracking data across all four languages. We further analyze the role of word length and word frequency in determining relative importance and find that it strongly correlates with length and frequency, however, the mechanisms behind these non-linear relations remain elusive. We obtain a cross-lingual approximation of the similarity between human and computational language processing and insights into the usability of several importance metrics.

3. ICWSM

   Evaluating Deep Taylor Decomposition for Reliability Assessment in the Wild

   Brandl S., Hershcovich D., and Søgaard A.

   In Proceedings of the International AAAI Conference on Web and Social Media Jun 2022

   TL;DR Code
4. NAACL

   How Conservative are Language Models? Adapting to the Introduction of Gender-Neutral Pronouns

   Brandl S., Cui R., and Søgaard A.

   In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Jul 2022

   Abs PDF TL;DR Code

   Gender-neutral pronouns have recently been introduced in many languages to a) include non-binary people and b) as a generic singular. Recent results from psycholinguistics suggest that gender-neutral pronouns (in Swedish) are not associated with human processing difficulties. This, we show, is in sharp contrast with automated processing. We show that gender-neutral pronouns in Danish, English, and Swedish are associated with higher perplexity, more dispersed attention patterns, and worse downstream performance. We argue that such conservativity in language models may limit widespread adoption of gender-neutral pronouns and must therefore be resolved.

5. ACL

   Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?

   Eberle O.*, Brandl S.*, Pilot J., and Søgaard A.

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) May 2022

   Abs PDF Code Poster

   Learned self-attention functions in state-of-the-art NLP models often correlate with human attention. We investigate whether self-attention in large-scale pre-trained language models is as predictive of human eye fixation patterns during task-reading as classical cognitive models of human attention. We compare attention functions across two task-specific reading datasets for sentiment analysis and relation extraction. We find the predictiveness of large-scale pre-trained self-attention for human attention depends on ‘what is in the tail’, e.g., the syntactic nature of rare contexts.Further, we observe that task-specific fine-tuning does not increase the correlation with human task-specific reading. Through an input reduction experiment we give complementary insights on the sparsity and fidelity trade-off, showing that lower-entropy attention vectors are more faithful.

6. ACL

   Challenges and Strategies in Cross-Cultural NLP

   Hershcovich D., Frank S., Lent H., Lhoneux M., Abdou M., Brandl S., Bugliarello E., Cabello Piqueras L., Chalkidis I., Cui R., Fierro C., Margatina K., Rust P., and Søgaard A.

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) May 2022

   Abs PDF TL;DR

   Various efforts in the Natural Language Processing (NLP) community have been made to accommodate linguistic diversity and serve speakers of many different languages. However, it is important to acknowledge that speakers and the content they produce and require, vary not just by language, but also by culture. Although language and culture are tightly linked, there are important differences. Analogous to cross-lingual and multilingual NLP, cross-cultural and multicultural NLP considers these differences in order to better serve users of NLP systems. We propose a principled framework to frame these efforts, and survey existing and potential strategies.

2021

1. Fourier SPoC: A customised machine-learning analysis pipeline for auditory beat-based entrainment in the MEG\makebox[0pt][l]\phantom

   Brandl Stephanie, Haumann Niels Trusbak, Radloff Simjon, Dähne Sven, Bonetti Leonardo, Vuust Peter, Brattico Elvira, and Grube Manon

   BioRxiv 2021

2020

1. Motor Imagery Under Distraction—An Open Access BCI Dataset

   Brandl Stephanie, and Blankertz Benjamin

   Frontiers in neuroscience 2020
2. Balancing the composition of word embeddings across heterogenous data sets

   Brandl Stephanie, Lassner David, and Alber Maximilian

   arXiv preprint arXiv:2001.04693 2020
3. Early Corona Twitter Dataset

   Brandl Stephanie, and Lassner David

   HAL preprint hal-02861167 2020
4. Corona Twitter Dataset: 16 February 2020 - 03 March 2020

   Brandl Stephanie, and Lassner David

   2020

2019

1. Times Are Changing: Investigating the Pace of Language Change in Diachronic Word Embeddings

   Brandl Stephanie, and Lassner David

   In Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change 2019

2015

1. Bringing BCI into everyday life: Motor imagery in a pseudo realistic environment

   Brandl Stephanie, Höhne Johannes, Müller Klaus-Robert, and Samek Wojciech

   In 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER) 2015
2. Robust common spatial patterns based on Bhattacharyya distance and Gamma divergence

   Brandl Stephanie, Müller Klaus-Robert, and Samek Wojciech

   In The 3rd International Winter Conference on Brain-Computer Interface 2015
3. BCI under distraction: Motor imagery in a pseudo realistic environment

   Brandl Stephanie

   2015