Custom Colloquia
Abstract:
This colloquium is about probing and interpretability in encoder-decoder models. Neural networks now match or exceed human performance on many tasks, yet we rarely know what a model represents internally or how it reaches an answer. The session has three aims: to introduce what interpretability research is and why it matters; to walk through different probing and interpretability methods; and to open a discussion that gathers ideas for taking the work further.
We present our ongoing work: probing encoder-decoder transformers trained on a small, fully controlled linguistic task to trace what a network encodes internally.
The discussion involves: whether a model actually uses the information it encodes, what makes an interpretability claim convincing, and whether small-scale findings transfer to large models.
Abstract:
Human values have been studied in social sciences for decades to
understand and compare individual and societal attitudes and behavior.
The Schwartz value theory \[1\] has formed the basis for computational
analysis of human values in web data. The latest human values
detection models have proven effective on news articles, arguments,
and historical documents. Applying human values detection to large
scale data enables exploring research questions that cannot be
possible with small and static datasets. For example, studying the
change between individuals’ security and self-direction values in
tweets on COVID-19 (e.g., TweetsCOV19 \[[2](https://data.gesis.org/tweetscov19/)\] having 41million tweets from Oct. 2019 to Aug.
2022) in response to real-world events. Moreover, the latest human
values detection models are not transparent, have lower
explainability, are not grounded in human values theory, and may not be reproducible.
The motivation of my work, of which I want to present the status at
the colloquium, is to develop a lexicon-based method to detect human
values with good accuracy and very high computational efficiency. Our
method extends the existing human values lexicon with i) more words
for better coverage ii) term weights to discriminate terms based on
distinguishing values, and iii) to cater for context in complex
sentence structures where only the terms may not be sufficient to
detect values. Our approach is grounded in human values theory, is
transparent with higher explainability, and is reproducible. For
evaluation, we compare our model with baseline models on
Touché23-ValueEval \[[3](http://www.gesis.org)\] and Touché24-ValueEval \[[4](https://zenodo.org/records/13283288)\] datasets. In the
colloquium, we aim to discuss our methodological choices, particularly
the lexicon design and contextual sensitivity.
\[1\] Schwartz, S. H. (1994). Are there universal aspects in the
structure and contents of human values?. Journal of social issues,
50(4), 19-45.
\[2\] <https://data.gesis.org/tweetscov19/>
\[[3](http://www.gesis.org)\] Mirzakhmedova, N., Kiesel, J., Alshomary, M., Heinrich, M.,
Handke, N., Cai, X., ... & Stein, B. (2024, May). The
touché23-valueeval dataset for identifying human values behind
arguments. In Proceedings of the 2024 Joint International Conference
on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (pp.
16121-16134).
\[4\] <https://zenodo.org/records/13283288>
Abstract:
Artificial intelligence (AI) is rapidly transforming the healthcare sector by optimizing clinical decision-making, improving diagnostic accuracy and supporting personalized patient care. Recent advances in machine learning, natural language processing, and generative AI have accelerated the integration of AI-powered tools into routine clinical practice. Applications range from medical image analysis and risk prediction to clinical documentation, patient monitoring and decision support systems.
Despite these promising benefits, the use of AI in healthcare continues to face significant challenges. Concerns regarding data privacy, algorithmic bias and transparency continue to limit its widespread adoption. The reliability and generalizability of AI models can be compromised by the quality and representativeness of the training data, potentially leading to performance differences across different patient populations. Furthermore, regulatory frameworks, ethical considerations, and the need for clinical validation remain critical factors for safe and effective use.
In today’s healthcare landscape, AI is increasingly viewed as a supportive technology rather than a replacement for healthcare professionals. While AI systems can complement clinical expertise and streamline routine tasks, human oversight remains essential for interpreting results, managing complex cases, and maintaining patient trust. The successful integration of AI into clinical workflows depends on balancing technological innovation with ethical responsibility and evidence-based evaluation. Given growing demands and staffing shortages in healthcare, AI offers significant opportunities to improve the efficiency and quality of care, provided its limitations and risks are carefully weighed.
Motivation:
The manual documentation of doctor letters and curation of clinical registries is time-consuming, resource-intensive, and a significant bottleneck for effective real-world evidence generation. While regex-based methods fail to capture the complexity of unstructured clinical documents, Large Language Models (LLMs) offer a promising approach for automation by leveraging contextual information.
Methods:
We developed a multi-stage pipeline to extract structured data for the EMCL registry from unstructured clinical documents. Using Google Gemma4, the LLM first identifies clinical/treatment periods and compiles a consistent clinical patient history from all available documents
Results:
In a proof-of-concept on 12 patients from the University Hospital Düsseldorf, some with up to 31 heterogeneous documents, 88% of clinical/treatment periods and 95% of extracted parameters across 4 tables matched a human reference. The results demonstrate that off-the-shelf open-source models can be leveraged for this task, though a final human expert check remains necessary. A larger multi-centric validation cohort is currently being assembled.
Open Questions & Discussion:
A core challenge is handling errors that originate not from the model, but from the source documents themselves. If a typo in a single document incorrectly shifts a treatment/observation period, the model may extract two separate periods where only one exists. We would like to discuss with the audience how such document-level errors can be detected and handled systematically. A second question concerns the reliability of extracted values more broadly. We would welcome input from both clinicians and NLP researchers on whether and how a traffic-light system for human documentarians could be implemented in practice.