Artificial Intelligence Large Language Models
in Digital Humanities

Christof Schöch
Trier Center for Digital Humanities
Trier University, Germany

CODHES-2024
Conference on Digital Humanities
and Environmental Sustainability

31 Oct 2024

Introduction

• Welcome everyone to this talk.
• I’m very happy to be here.
• I’m sorry I can’t be there in person with you.
• And I feel honored to have the opportunity to speak at your conference.
• Many thanks to the organizers for making this possible.
• Before I dive into the topic of my talk, please allow me a few words of introduction, so that my perspective on the topic of “AI and DH”
• By the way, you can follow along with the slides! See the link here.

Slides: dhtrier.quarto.pub/codhes2024

My background

• My Education
  • Master’s Degree: French, English, Psychology (Freiburg U)
  • Ph.D.: Eighteenth-Century French Literature (Kassel and Paris)
  • Post Doc: Scholarly Digital Editions, Research Infrastructrure, Text and Data Mining (Würzburg U)
• Professor of Digital Humanities (2017–, Trier U)
  • Primary field: Computational Literary Studies
  • Co-editor of the Journal of Computational Literary Studies (JCLS)
  • Co-director, Trier Center for Digital Humanities
  • Coordinator of a Master of Science in Digital Humanities
  • President of the Alliance of Digital Humanities Organizations (ADHO, 2023-2024)

dh.uni-trier.de

• (Describe education)
• (Describe role in Trier)
• So, I am not a computer scientist
• I have a rather particular specialisation in CLS
• But I am involved in many ways in DH beyond “just literary studies”
• My perspective on the topic of “AI in DH”:
• that of someone practising CLS
• using the recent affordances of LLMs
• I also teach ML and LLMs in our Master’s programme

Overview

1. What do we mean when we say “AI”?
2. AI LLMs in DH today
3. Use cases of AI LLMs in DH
   1. Multilingual stylometry
   2. Direct speech recognition
   3. Historical wine labels
4. Conclusion: affordances and concerns

• My aim today is to show how AI has already transformed DH
• Specifically, I want to look at several cases where we have used AI
• And also try to derive some more general lessons about AI from them
• We will see that AI has changed the field quite strongly
• And I expect that change to continue, and intensify
  
• But there are also concerns with this ubiquity of AI; I will address these as well.
• But first of all, let’s talk a little bit about what we mean by the phrase “AI”

What do we mean when we say AI?

AI between hype and doom

• Big questions these days: Will AI save the world, or not?
• Will AI save the energy crisis, and climate change, or just make things worse?

AI > ML > DL (e.g. Chollet)

• In this view, AI is more general than ML or DL
• Essentially, all automation is in some way AI
• That makes AI a very broad cover term
• That might be useful for marketing
• Including self-marketing, by the way:
• I can say: “Ah, AI. I’ve done that for more than a decade!”
• But I think the terms is too broad, and too loaded, to be useful for understanding

CS > ML > DL > LLMs | AI

• I prefer the opposite view, where the cover term is CS
• Here, ML is just one part of CS; and AI an even more specific part of CS
• “Traditional ML” is not AI, it’s statistical classification
• Deep Learning is not AI, it’s just neural networks doing classification
• LLMs are not AI, they are just contextual word embeddings
• Even generative LLMs are not AI, they are just glorified chatbots
• To me, only autonomous agents seeking out occasions to learn, interacting with their environment, and having many different capabilities, are AI.
• They need ML, DL and LLMs, but those are just building bricks for AI, not AI itself.
• In that perspective, I don’t really do AI.
• What I do, is I use LLMs, including generative LLMs, in my research
• This is what I can report on today
• I will do this by talking about several projects where we have used, or are using, LLMs.
• And I will try to derive some lessons from them, each time.

AI LLMs in DH today

Progress is impressive

• This visual from 2018 is now completedly outdated,
• and was already so after less than 5 years
• LLMs (both BERT type and GPTs) have boosted performance in NLP
• Multimodal LLMs have boosted capabilities in computer vision
• Digital Humanities is likely to follow a similar path
• We should try to learn from what happened in other fields, and anticipate this.

(Dan Jurafsky 2018)

LLMs are now everywhere in DH

• Papers at many venues routinely use them
  • Digital Humanities Conference
  • Computational Humanities Research
  • Journal of Computational Literary Studies
• Wide range of issue are addressed using LLMs
  • NLP tasks supporting DH research
  • Complex annotation tasks
  • Text encoding / information extraction
  • Text and image analysis
  • Support when programming

But there are also problematic issues

• Explainability for ML/AI
• Reproducibility / Open Science
• Sustainability

• In order not to approach these issues in abstract manner,
• But rather based on some concrete research,
• I would now like to turn to several cases where we used AI in Trier in recent times.
• After presenting these studies, we will be able to come back to the issues.

Use Case: Multilingual Stylometry

• This use case study concerns computational methods of text analysis
• One of the oldest paradigms in this area is stylometric authorship attribution
• Essentially, the idea that the author of an anonymous text can be identified based on a comparison of the word frequency profiles of the disputed text and texts by multiple candidate authors.

(See: Schöch et al. 2024,
CLS INFRA: https://clsinfra.io)

High-profile cases of stylometric authorship attribution

William Shakespeare:
Craig and Kinney (2009)

Molière and Corneille:
Cafiero and Camps (2019)

Elena Ferrante:
Tuzzi and Cortelazzo (2018)

Galbraith / Rowling:
Juola (2015)

• Some high-profile cases of authorship attribution in recent times
• These make stylometry a highly-contested methodology
• Of course, the long-standing investigations of Shakespeare
• In France, there has been a “Molière-Corneille”-controversy, almost a “feud”
• Recent high-profile cases have involved best-selling pseudonomous author Elena Ferrante
• And the detective novels written by Joanne K. Rowling, the Harry Potter author, under a pseudonym.
• BUT: These are almost always single-language authorship questions.

Multilingual stylometry?

translation ↦ ↧ original	eng	fra	ukr	hun
eng	eng-eng	eng-fra	eng-ukr	eng-hun
fra	fra-eng	fra-fra	fra-ukr	fra-hun
ukr	ukr-eng	ukr-fra	ukr-ukr	ukr-hun
hun	hun-eng	hun-fra	hun-ukr	hun-hun

• Using corpora from the European Literary Text Collection (ELTeC)
• 4 different languages, 30 novels each, translation into the other three languages
• 360 novels to translate => use DeepL to perform machine translation

• In our project, the question has been addressed from a multilingual perspective
• That is, we want to know: How does language influence stylometric performance?
• But also: how does corpus composition influence it?
• To answer such questions, we need to decouple language from corpus composition
• Work with corpora in four languages: eng, fra, hun, ukr
• Using original versions and translations into all three other languages (using DeepL machine translation)
• We’ve build an interactive showcase to allow us and others to investigate this dataset.

https://showcases.clsinfra.io/stylometry

Multilingual Stylometry: Overview

• Settings: word unigrams, full novels, 100-500 MFW
• Average accuracy at a reasonable setting, for each corpus
• Hungarian is the easiest corpus
• Ukrainian is the most difficult corpus
• Performance is always better in the original language (the diagonal from top left to bottom right)
• Performance drops for the translated corpora (to some extent, because the translator is always the same)

https://showcases.clsinfra.io/stylometry

Lessons learned

• Affordances
  • Using DeepL was a game-changer for this research
  • Interesting new insights into stylometric methods
  • Impossible without using LLM-based machine translation
• Limitations
  • The translations are only of “acceptable” quality
  • The main part of the research is just “normal” ML
  • Each translation pair performed by one “agent”: unrealistic

Use Case: Direct Speech Recognition

(CLS INFRA: https://clsinfra.io)

A typical CLS problem: Speech and Thought Representation

• Fundamental distinction for analysis of fictional narrative
  • Narrator vs. character speech (and thoughts)
  • Direct, indirect, free indirect character speech
• For many reasons, it might be relevant to separate the two
  • How much has the author given an individual voice to each character?
  • Is there a correlation between subgenres and proportion of character speech?

Our first attemps at this (2016)

• Corpus of French novels ca. 1840 to 1920
• Focus on direct character speech (sentence-level)
• Traditional ML approach
  • Feature engineering: 80 features!
  • Train and test a classifier (SVM)
  • Resulting F1-Score: 0.924

(See: Schöch et al. 2016)

LLM-based approach (2021)

• Würzburg/Mannheim team
• Corpus of German narrative texts
• Focus on various types of character speech (token-level)
• LLM-based approach
  • Pre-trained LLMs
    (BERT and Flair)
  • Fine-tuning with extensive training data
  • Resulting F1-Scores: ~0.87

(See: Brunner et al. 2020)

Approach using SetFit (2024)

• Focus on direct character speech (sentence-based)
• Using SetFit (Sentence Transformers Fine-Tuning)
  • BERT-based sentence transformers
  • Few-shot learning: small amount of training data
  • Resulting F1-Scores: 0.89-0.93 (depending on scenario)

(Publication in preparation)

Lessons learned

• Progress is indeed impressive!
  • Not so much in performance (remains similar)
  • But in time/effort needed: days instead of months/years
  • Notably: no hand-crafted features required anymore
• A big concern: Explainability
  • Insight into direct speech through feature weights
  • Trivial in a traditional ML approach
  • Much more involved in a DL/LLM approach

Use Case: Historical Wine Labels

(See: Weis and Schöch 2024,
LODiNG: https://loding.uni-trier.de)

Collections of historical wine labels

Knowledge Graph about the wine labels

• Knowledge Graph using Wikibase (Linked Open Data paradigm)
• One focus: mapping of vineyards, wine makers, villages, labels
• Another focus: Objects depicted on the labels
• Goal: Use LLMs to annotate our collection and feed the KG

First tests with ChatGPT (Nov. 2023)

• ChatGPT 4 with ADA plugin
• Prompt: “What objects, things, animals, text and design elements to you recognize in this image?”
• Very impressive OCR and object detection
• Some interesting inferences (“likely the Mosel”)
• But: unstructured prose, difficult to work with

• Very impressive quality of the annotation
• Note the very amateurish prompt
• But: we don’t want prose, we need semi-structured data for analysis
• So we need to do some prompt engineering, custom GPTs, with controlled vocabularies
• Possible with ChatGPT, but the next step is to do this using an open source local system

Using a Custom GPT (Oct. 2024, ChatGPT 4o)

• Changes
  • Detailed instructions (more precise prompt)
  • Controlled vocabulary of depicted objects (~100 items)
  • JSON output as default
• Improvements
  • Less conversational, more useful results
  • Items mentioned are part of the vocabulary
• Persisting issues
  • Instructions applied inconsistently
  • QIDs are incorrect
  • And: our results are not reproducible

Lessons learned

• Affordances
  • Compared to systems from about 5 years ago, progress is impressive
  • Accessibility is very high: Easy to create Custom GPT
  • A good way to feed information into our Wikibase
• Limitations
  • Constraining ChatGPT to formalized knowledge is hard
  • Obtaining the same performance with open, desktop solutions is hard
• Way forward
  • Use ChatGPT for free prose descriptions of labels
  • Use traditional “object detection” to feed the KG

Conclusion: Avantages and Challenges of LLMs in DH

Advantages: Impressive progress!

• Higher performance in some tasks
• Lower development time on other tasks
• No hand-crafted features required anymore
• Better representation of meaning in context
• Context spans will continue to grow: sentences, paragraphs, novels
• Promising approach:
  • DH: Using LLMs for Information Extraction that feeds into Knowledge Graphs;
  • NLP: Use Knowledge Graphs (e.g. via KG embeddings) to feed factual knowledge into LLMs and improve IE

Challenges

• Fairness: Very high hardware requirements (= high cost)
• Sustainability: Very high energy needs (use LLM only when needed)
• Explainability: Much harder in DL/LLMs than in ML
• Open Science: Most LLMs not open source, not reproducible, not transparent

Final words: learn from the past

• Big data in DH (~2010)
  
• Topic Modeling in DH (~2012)
  
• Word Embeddings in DH (~2015)
  
• ChatGPT in DH (~2023)

• With “Big Data” I think to some extent, DH fell into the trap: Oh, CS has Big Data, we need that too.
• A bit later, with Topic Modeling, same thing: we need that too.
• Similarly with Word Embeddings (word2vec)
• But Topic Modeling, and word2vec, was designed to adress the challenge of when you have a lot of data!
• We find ourselves with corpora typical of DH, they are too small!
• Then we need solutions for that, like pre-trained topic models (BERTopics).
• But for less-resources languages, orhistorical languages, there are no corpora to pretrain a BERTopic model. So then we go into few-shot learning.
• Solve tech problems with more tech.
• That’s not a good approach.
• Let’s not make the same mistake with generative LLMs.
• Yes, non-generative LLMs like BERT (contextual word embeddings) are fundamental tools for Digital humanities, in order to represent and work with the meaning of words and texts.
• But with generative LLMs, we may face a similar challenge: They are designed and optimized for a task that is not entirely compatible with the goals of DH, and of research more generally:
• in fundamental research, it’s not about optimizing F1-Scores, but about better understanding our domain.
• A black box will never solve that for us.

Many thanks! 😺

References

Brunner, Annelen, Ngoc Duyen Tanja Tu, Lukas Weimer, and Fotis Jannidis. 2020. “To BERT or Not to BERT-Comparing Contextual Embeddings in a Deep Learning Architecture for the Automatic Recognition of Four Types of Speech, Thought and Writing Representation.” In SwissText/KONVENS. https://www.academia.edu/download/101289662/paper5.pdf.

Cafiero, Florian, and Jean-Baptiste Camps. 2019. “Why Molière Most Likely Did Write His Plays.” Science Advances 5 (11): eaax5489. https://doi.org/10.1126/sciadv.aax5489.

Craig, Hugh, and Arthur F. Kinney. 2009. Shakespeare, Computers, and the Mystery of Authorship. Cambridge University Press.

Juola, Patrick. 2015. “The Rowling Case: A Proposed Standard Protocol for Authorship Attribution.” Digital Scholarship in the Humanities 30 (suppl. 1): 100–113. https://doi.org/10.1093/llc/fqv040.

Schöch, Christof, Julia Dudar, Evgeniia Fileva, and Artjoms Šeļa. 2024. “Multilingual Stylometry: The Influence of Language on the Performance of Authorship Attribution Using Corpora from the European Literary Text Collection.” In Proceedings of the Computational Humanities Research Conference 2024. Aarhus: CEUR. https://doi.org/10.5281/zenodo.13995613.

Schöch, Christof, Daniel Schlör, Stefanie Popp, Annelen Brunner, Ulrike Henny, and José Calvo Tello. 2016. “Straight Talk! Automatic Recognition of Direct Speech in Nineteenth-Century French Novels.” In Digital Humanities: Conference Abstracts, 346–53. Kraków: Jagiellonian University & Pedagogical University. http://dh2016.adho.org/abstracts/31.

Tuzzi, Arjuna, and Michele A. Cortelazzo, eds. 2018. Drawing Elena Ferrante’s Profile: Workshop Proceedings. Padova: Padova UP.

Weis, Joëlle, and Christof Schöch. 2024. “Vom Perler Hasenberg Zur Lehmener Würzlay – Weinetiketten Digital Erschließen.” In Digital Ist Besser? Sammlungsforschung Im Digitalen Zeitalter, edited by Katharina Günther and Stefan Alschner. Göttingen: Wallstein. https://www.wallstein-verlag.de/9783835356153-002.html.