Keyness in Computational Literary Studies: History, Definitions and Evaluation

Christof Schöch
With Keli Du, Julia Dudar, Julia Röttgermann, Julian Schröter
(Trier University, Germany)

Untangling Associations: Advances in keyword and collocation analysis
Université Paul Valéry, Montpellier, 22 Sept. 2023.

• Hello everyone, my name is Christof Schöch
• It’s a pleasure and an honor to have the occasion to speak here
• I guess I am here to represent a marginal actor in keyness research, the field of Computational Literary Studies ;-)
• The work I report on here has been performed in the context of a project called “Zeta and Company: Measures of Distinctiveness for Computational Literary Studies”
• This is therefore collaborative work, which I have undertaken together with Keli Du, Julia Dudar, Julia Röttgerman and Julian Schröter

Presentation: https://dhtrier.quarto.pub/montpellier

Overview

• # 1 – History: Keyness in CLS
• # 2 – Definitions: Keyness / Distinctiveness
• # 3 – Evaluation: Classification Task
• # 4 – Conclusion: Findings and Outlook

• As you can see here, I’d like to proceed in three steps:
• First, recap the recent history of keyness in CLS, which is mostly the story of Zeta, a keyness measure introduced by John Burrows
• Second, I’d like to present some conceptual work we have recently done in a project in Trier, on what “keyness” could mean.
• And finally, I’d like to summarize some evaluation results we have obtained in that

History: Keyness in CLS

• So, let’s start off with some recent history of keyness, or Zeta, in CLS research.
• The two are almost identical in this context.

Burrows’ Zeta in Authorship Attribution

• The key publication is: John Burrows, “All the way through” (Burrows 2007)
• He proposed to use Zeta in the context of authorship attribution
• Zeta is calculated as the difference of the document frequencies of a feature in two contrasting sets of documents, where the documents are segments of full texts.¹
  
• Zeta: “A simple measure of [an author’s] consistency in the use of each word-type.” (=> dispersion)
• Focuses “on a single author and seek[s] to identify which of many texts are most likely to be his or hers.” (=> authorship attribution)

• The story of Zeta starts quite recently, with a paper by John Burrows.
• He’s a pioneer of CLS with his 1987 book on Style in Jane Austen.
• (Then go through the points.)

1. Slides with details (link).

Zeta for Authorship Attribution: Shakespeare (Craig and Kinney 2009)

• As an example, this is from work by Hugh Craig and Arthur Kinney on Shakespeare
• They use a rather typical scenario of using Zeta for authorship attribution
• Keywords for Shakespeare and other authors are established using Zeta
• Each segment is plotted as a function of both keyword groups’ proportion in each segment
• A text of disputed authorship is plotted there as well
• Depending on the clarity of separation, and where the disputed segments fall, an authorship attribution is made.

Further uses and discussion of Keyness

• Uses and discussion of Zeta:
  • Hoover (2010)
  • Weidman and O’Sullivan (2018)
  • Rizvi (2019b)
  • Rizvi (2019a)
  • Barber (2021)
  • Rizvi (2022)
  • Hoover (2022)
• Uses of other keyness measures:
  • using Antconc (log-likelihood)
  • or TXM (spécificité)

• This is more as a reference, so I’ll skip that slide.
• Maybe just two words:
  • It is remarkable that there is a strong focus on Zeta, mostly for AA
  • Often using stylo, but just as often own implementations
  • And there is work using other measures, using other tools, but that is less visible.
  • As an aside: It is also remarkable that none of the work on Zeta has ever been noticed by the CCL crowd.

Keyness for gender (Weidman and O’Sullivan 2018)

Scatterplot of segments by male and female authors, by percentage of markers and anti-markers, for three literary periods.

• Soon, the potential of Zeta for a broader range of contrastive scenarios was discovered
• What you see here is an example of uses of Zeta for gender and gendered language
• The tool used here is stylo, mostly used in AA
• To be honest, I think the authors misread their own plot when commenting on it (mistaking the green and red markers for words, when they are in fact segments)
• What this shows, in my opinion, is the following:
  • there is a peak of gendered literary language in the modernist period
  • in the middle plot, texts from either group use quite a distinct use of vocabulary;
    
  • before and after, overlap in vocabulary use is much greater.

Keyness for Genre (Schöch 2018)

PCA plot using 50 Zeta-based keywords. Comedies in red, tragedies in blue, tragi-comedies in green.

• In a similar spirit of “stylometry beyond authorship”, I started to use Zeta for contrastive analyses of literary subgenres
• Literary studies compares things all the time, so keyness seems more generally relevant than just for authorship attribution
• The figure shown here concerns French comedies, tragedies and tragi-comedies
• The aim was o settle the question of whether tragi-comedies are a special type of comedies, of tragedies, or something in between.
  
• 50 words identified by Zeta to be distinctive of comedies or of tragedies were identified
• Their frequencies were then used for a PCA on texts of the three genres, so including “tragi-comedies”
• One can see that comedies (in red, on the left) use quite a distinct vocabulary;
  
• Tragedies (in blue, on the right) are also quite distinct;
• But tragi-comedies (in green), overlap more with tragedies than with comedies.

Zeta and Company (Schöch et al. 2018)

• Project funded by the German Research Foundation (DFG, 2020-2023)
• Domain of application: popular subgenres of the 20th-century French novel
• Inspirations: John Burrows (Burrows 2007), Jeffrey Lijffijt (Lijffijt et al. 2014), MOTIFS (e.g. Kraif and Tutin 2017), Phraséorom (e.g. Diwersy et al. 2021), dispersion (Gries 2008)
• Fundamental aim: Enable scholars in CLS to make educated choices about what keyness measure to use
• Also: Bridge the gap between CCL and CLS
• Activities: modeling, implementing, evaluating and using statistical measures of comparison of two groups of texts.

• This work led to a proposal for a stand-along project on keyness
• Modeling: Try to establish a systematic understanding allowing us to describe all measures of distinctiveness using the same descriptive elements
• Implementing: Developing a Python package that provides multiple measures in the same framework
• Evaluating: Using quantitative methods (e.g. downstream classification task) and qualitative methods (application study)
• Using: Trying to answer the question of what we can find out about subgenres using keyness measures
• In the following, I will just focus on two areas: conceptual work and one evaluation task.

Definitions: Keyness / Distinctiveness

• Let’s start with a short recap of some of our conceptual work
• This was focused on trying to understand what “keyness” really is;
• and what it means to evaluate a keyness measure.

Traditional definition of keyness

• Purely quantitative sense: A keyword is “a word which occurs with unusual frequency […] [in a document or corpus] by comparison with a reference corpus”. (Scott 1997)

• We find this kind of definition rather problematic
• First reason: It is kind of circular or self-fulfilling;
• Also: It needs to be changed if the basis of calculation is changed;
• For example: Many people appear to think nowadays that dispersion should play a role in the calculation of keyness, along with frequency;
• So apparently, dispersion is relevant to what keyness really is;
• That can’t be true if keyness is defined by frequency.
• Therefore: We need to define keyness independently of its statistical operationalization.
• We consider that a separate definition of keyness is necessary, because with a purely quantitative definition that is identical to a measure, evaluation remains circular
• This set us off to think about what keyness (or distinctiveness) could mean, qualitatively

Source: User Hyacinth, Source: Wikimedia Commons, Licence: CC BY-SA.

What is Distinctiveness? (Schröter et al. 2021)

• (A) Logical vs. statistical sense
  • Purely logical: A feature is distinctive of corpus A if its presence in a document D is a necessary and sufficient condition for D to belong to A and not to B.
  • Statistical: A feature is distinctive of corpus A if it is true that, the higher its keyness in document D, the higher the probability that D is an instance of A and not of B.
• (B) Salient vs. agnostic
  • Salient: A feature is distinctive iff it is noticed by readers (for confirming or violating their expectations)
  • Agnostic: A feature can be distinctive without being salient in the above-mentioned sense.
• (C) Qualitative vs. no qualitative content
  • Qualitative content: A feature is distinctive iff it expresses e.g. aboutness or stylistic character (=> interpretability)
  • No qualitative content: A feature can be key regardless of qualitative content (=> discriminatory power)

• We came up with three dimensions of a notion of keyness (or distinctiveness, as we prefer to say)
• (Explain the three dimensions, then following comments.)
• There is a certain fit between traditional genre theory (and likewise: philological authorship attribution), the logical concept of keyness, the assumption of salient features, and the assumption of a qualitative content of key features.
• Similarly, there is a connection between modern theories of genre (hybrid genres, hybrid texts); the statistical sense of keyness; the agnostic sense; and the assumption of no necessary qualitative content.
• Except that: in CLS, we aim for a statistical and agnostic notion combined with qualitative content of the distinctive features! Tricky.
• Also: if there are multiple legitimate conceptions of keyness, or aspects of keyness, we may need multiple measures supporting these different understandings.

Measures in Zeta and Company (Du et al. 2022)

• TF-IDF
• Ratio of relative frequencies
• Chi-squared test
• Log-likelihood ratio test
• Welch’s t-test
• Wilcoxon rank-sum test
• Burrows Zeta – (Burrows 2007)
• Log-Zeta – (Schöch et al. 2018)
• Eta (difference of DP) – (Gries 2008; Du et al. 2021)

• That led us to looking at a range of keyness measures
• Listed here are the measures we have implemendet in our Python package “pydistinto”
• Several of them are clearly based on corpus-level frequency only: for example RRF, LLR
• Others use various strategies to take dispersion into account: for example Zeta, Log-Zeta, Eta.
• Still others rely on distribution within the corpus, but not necessarily just dispersion: for example Welch, Wilcoxon.
• So what do these measures capture? How do they perform in the context of CLS-related tasks?

Evaluation: Classification Task

• This is what we try to find out in our various evaluation tasks.
• One of them is focused on the “discriminative power” of the measures (statistical, agnostic notion of keyness)
• It is based on a downstream classification task involving literary subgenres
• We have other evaluation strategies, but I’ll focus on this one today.

Evaluation Task: Genre Classification (Du, Dudar, and Schöch 2022)

• Downstream classification task: “How reliably can a machine learning classifier, based on words identified using a given measure of distinctiveness, identify the subgenre of a novel when provided only with a short segment of that novel?”
• Basic setup
  • 4 classifiers
  • Different numbers of keywords (N)
  • Textual units are 5000-word segments
  • 10-fold-cross validation (90/10 split of segments)
  • Baseline: random selection of N words

• Why a classification task? Because annotating a gold standard for keyness is not possible (because target and comparison corpus as a whole need to be considered, a task that is cognitively impossible for humans)
• And because evaluating the appropriateness of a list of keywords for the category it is supposed to represent is difficult to formalize.
• However, readers can identify the subgenre a novel belongs to rather quickly, after reading just a few paragraphs or pages.
• Therefore: Can a machine learning classifier mirror this capacity?
• And: What measures provide the most useful features for this?
• Note that we are here in the statistical and agnostic paradigm, and so primarily in the meaning of keywords as words with “discriminatory power”
• However, we are interested also in the interpretability of lists of distinctive words

Corpus: French contemporary popular fiction

• Genres: 4 groups (crime, sentimental, science-fiction and general narrative fiction)
• Period: 2 groups (1980s, 1990s)
• Size: 40 novels for 8 groups = 320 novels, 17.3 million words
• Published as derived text format – 10.5281/zenodo.7111522
• (Today: 50 novels per decade and genre, 1970-2000 = 600 novels)

Results #1 (Du, Dudar, and Schöch 2022)

Classification performance on the French corpus (1980s) with four classifiers, depending on the measure of distinctiveness and the setting of 𝑁.

• If we vary N, the larger N, the better the performance and the smaller the differences between the classifiers
• That makes sense, of course, as the classifiers just get more information to work with
• Does this mean the measure doesn’t matter for discriminatory power?
• Not really:
  • in a CLS context where interpretation of keyword lists matters as much as discriminatory performance
  • if the most useful keywords are also to be inspected qualitatively, their number needs to remain limited.
• Therefore, the measure matters and dispersion-based measures do better

Results #2 (Du, Dudar, and Schöch 2022)

Distribution of classification performance on the 1980s French corpus with N = 10 using Multinomial Naive Bayes

• These are our detailed results for small N: 10 keywords and 10 negative keywords per group were used, for a total of 80 words
• There is clearly a top group, made up of the Zetas, Eta and TF-IDF
• There is a median group, with Wilcoxon and Welch
• And there is a low-performing group, with LLR, Chi-Squared and RRF
• Tested also for seven different ELTeC corpora, confirming the overall result.
• So for small numbers of features, dispersion-based measures provide words with stronger discriminatory power than other measures.
  
• Such measures also tend to select medium-frequency words that are highly-interpretable (=> salient, qualitative)

Conclusion: Findings and Outloook

• So, time to wrap up.

What have we found out so far?

• Definition
  • Keyness or distinctiveness, as a concept, can be defined in different ways
  • A match between a certain understanding of distinctiveness and a specific statistical operationalization can be established using a suitable method of evaluation.
• Evaluation
  • Dispersion-based keyness measures show best performance in a subgenre classification task, especially when the number of features is small
  • Such measures also tend to select medium-frequency words that are highly-interpretable (=> salient, qualitative)

What are the next steps?

• Perform further experiments, using synthetic texts and test tokens with pre-determined frequency- and/or dispersion-based contrasts
• Perform an application study that aims to match keywords to generic traits derived from research on popular subgenres (qualitative reference for qualitative understanding of distinctiveness)
• Add measures: dispersion + LLR (Egbert and Biber 2019), measure based on DP_nofreq (Gries 2021), LRC (Evert 2022)
• Move on to more complex features: multi-word expressions and semantic features (next project phase, 2024-2026)
• Find a strategy for how to handle a multi-dimensional approach to keyness (multiple meanings, multiple measures), e.g. along the lines proposed in (Gries 2019)

• Present our future work.
• To conclude:
  • I realize that I may not have been able to contribute much to your debates;
  • But I hope that I was able, at least, to illustrate to you where research into keyness stands in Computational Literary Studies.

Thank you!

• Presentation: https://dhtrier.quarto.pub/montpellier
• Contact: schoech@uni-trier.de

References

Barber, Ros. 2021. “Big Data or Not Enough? Zeta Test Reliability and the Attribution of Henry VI.” Digital Scholarship in the Humanities 36 (3): 542–64. https://doi.org/10.1093/llc/fqaa041.

Burrows, John. 2007. “All the Way Through: Testing for Authorship in Different Frequency Strata.” Literary and Linguistic Computing 22 (1): 27–47. https://doi.org/10.1093/llc/fqi067.

Craig, Hugh, and Arthur F. Kinney. 2009. Shakespeare, Computers, and the Mystery of Authorship. Cambridge University Press.

Diwersy, Sascha, Laetitia Gonon, Vannina Goossens, Olivier Kraif, Iva Novakova, Julie Sorba, and Ilaria Vidotto. 2021. “La phraséologie du roman contemporain dans les corpus et les applications de la PhraseoBase.” Corpus, no. 22. https://doi.org/10.4000/corpus.6101.

Du, Keli, Julia Dudar, Cora Rok, and Christof Schöch. 2021. “Zeta & Eta: An Exploration and Evaluation of Two Dispersion-based Measures of Distinctiveness.” In Proceedings of the Conference on Computational Humanities Research 2021, edited by Maud Ehrmann, Folgert Karsdorp, Melvin Wevers, Tara Lee Andrews, Manuel Burghardt, Mike Kestemont, Enrique Manjavacas, Michael Piotrowski, and Joris van Zundert, 2989:181–94. CEUR Workshop Proceedings. Amsterdam, the Netherlands: CEUR.

———. 2022. “Kontrastive Textanalyse mit pydistinto - Ein Python-Paket zur Nutzung unterschiedlicher Distinktivitätsmaße.” Potsdam: Zenodo. https://doi.org/10.5281/zenodo.6327967.

Du, Keli, Julia Dudar, and Christof Schöch. 2022. “Evaluation of Measures of Distinctiveness. Classification of Literary Texts on the Basis of Distinctive Words.” Journal of Computational Literary Studies 1 (1). https://doi.org/10.48694/jcls.102.

Egbert, Jesse, and Doug Biber. 2019. “Incorporating Text Dispersion into Keyword Analyses.” Corpora 14 (1): 77–104. https://doi.org/10.3366/cor.2019.0162.

Evert, Stephanie. 2022. “Measuring Keyness.” In Book of Abstracts of the Digital Humanities 2022. Tokyo: ADHO. https://doi.org/10.17605/OSF.IO/CY6MW.

Gries, Stefan Th. 2008. “Dispersions and Adjusted Frequencies in Corpora.” International Journal of Corpus Linguistics 13 (4): 403–37. https://doi.org/10.1075/ijcl.13.4.02gri.

———. 2019. “15 Years of Collostructions: Some Long Overdue Additions/Corrections (to/of Actually All Sorts of Corpus-Linguistics Measures).” International Journal of Corpus Linguistics 24 (3): 385–412. https://doi.org/10.1075/ijcl.00011.gri.

———. 2021. “What Do (Most of) Our Dispersion Measures Measure (Most)? Dispersion?” November. https://doi.org/10.1075/jsls.21029.gri.

Hoover, David L. 2010. “Teasing Out Authorship and Style with t-Tests and Zeta.” In Digital Humanities Conference. London.

———. 2022. “Zeta Revisited.” Digital Scholarship in the Humanities 37 (4): 1002–21. https://doi.org/10.1093/llc/fqab095.

Kraif, Olivier, and Agnès Tutin. 2017. “Des motifs séquentiels aux motifs hiérarchiques : l’apport des arbres lexico-syntaxiques récurrents pour le repérage des routines discursives.” Corpus, no. 17 (January). https://doi.org/10.4000/corpus.2889.

Lijffijt, Jefrey, Terttu Nevalainen, Tanja Säily, Panagiotis Papapetrou, Kai Puolamäki, and Heikki Mannila. 2014. “Significance Testing of Word Frequencies in Corpora.” Digital Scholarship in the Humanities 31 (2): 374–97. https://doi.org/10.1093/llc/fqu064.

Rizvi, Pervez. 2019a. “An Improvement to Zeta.” Digital Scholarship in the Humanities 34 (2): 419–22. https://doi.org/10.1093/llc/fqy039.

———. 2019b. “The Interpretation of Zeta Test Results.” Digital Scholarship in the Humanities 34 (2): 401–18. https://doi.org/10.1093/llc/fqy038.

———. 2022. “The Interpretation of Zeta Test Results: A Supplement.” Digital Scholarship in the Humanities 37 (4): 1172–78. https://doi.org/10.1093/llc/fqac011.

Schöch, Christof. 2018. “Zeta für die kontrastive Analyse literarischer Texte. Theorie, Implementierung, Fallstudie.” In Quantitative Ansätze in den Literatur- und Geisteswissenschaften. Systematische und historische Perspektiven, edited by Toni Bernhart, Sandra Richter, Marcus Lepper, Marcus Willand, and Andrea Albrecht, 77–94. Berlin: de Gruyter.

Schöch, Christof, Daniel Schlör, Albin Zehe, Henning Gebhard, Martin Becker, and Andreas Hotho. 2018. “Burrows : Exploring and Evaluating Variants and .” In Book of Abstracts of the Digital Humanities Conference. Mexico City: ADHO.

Schröter, Julian, Keli Du, Julia Dudar, Cora Rok, and Christof Schöch. 2021. “From Keyness to Distinctiveness Triangulation and Evaluation in Computational Literary Studies.” Journal of Literary Theory 15 (1-2): 81–108. https://doi.org/10.1515/jlt-2021-2011.

Scott, Mike. 1997. “PC Analysis of Key Words And Key Key Words.” System 25 (2): 233–45. https://doi.org/10.1016/S0346-251X(97)00011-0.

Weidman, Sean G., and James O’Sullivan. 2018. “The Limits of Distinctive Words: Re-evaluating Literature’s Gender Marker Debate.” Digital Scholarship in the Humanities 33 (2): 374–90.

Bonus slides

Measures with references (Du, Dudar, and Schöch 2022)

All corpora (Du, Dudar, and Schöch 2022)

Correlation between measures (Du, Dudar, and Schöch 2022)

Keyness in stylo: genre (A.C. Doyle)

   

• Here, a simple comparison of Doyle’s detective novels agains other novels
• In the left plot, the focus is on the keywords
  • words “preferred” by detectve fiction on the right
  • words “avoided” by detective fiction on the left
• In the right plot, the focus is on text segments
  • segments from detective fiction are in red
  • segments from other fiction are in green
  • their position is determined by the percentage of marker and antimarker words
• In any case, the shift from using Zeta for authorship attribution to using Zeta for text group comparison clearly happened.