Comparing Comparisons

Evaluating Measures of Keyness or Distinctiveness in Computational Literary Studies




Christof Schöch
(Trier University, Germany)

Kyungpook National University
Daegu, South Korea

22 May 2024

Introduction

Thanks



Sujin Kang and Heejin Kim of Kyungpook National University, as well as KADH (Korean Association for Digital Humanities).



Funding from the German Research Foundation (DFG), who has been funding this research (Zeta and Company, 2020-2023, Beyond Words, 2024-2026)


Thanks to all the project contributors: Keli Du, Julia Dudar, Cora Rok, Julia Röttgermann, Julian Schröter.

Overview

  • 1 – Introduction
  • 2 – What is Keyness all about?
  • 3 – Definitions: Keyness / Distinctiveness
  • 4 – History: Keyness in Computational Literary Studies
  • 5 – Quantitative Evaluation: Classification task
  • 6 – Qualitative Evaluation: Genre profile task
  • 7 – Conclusion: Findings and outlook
  • As you can see here, I’d like to proceed in several steps:
  • First, explain what keyness is and why it is more fundamental than you might think.
  • Then, recap the recent history of keyness in CLS, which is mostly the story of Zeta, a keyness measure introduced by John Burrows
  • Then, I’d like to present some conceptual work we have recently done in a project in Trier, on what “keyness” could mean.
  • And finally, I’d like to summarize some evaluation results we have obtained in that project, in two ways: quantitatively and qualitatively.

What is keyness all about?

Some recent findings in my work in CLS1

  • The average sentence length in English-language novels in the period 1920–1939 is 18.9 words.
  • In adventure novels by Arthur Conan Doyle, the ‘hunting’-topic has a probability of 0.38.
  • The average word frequency of the word “et” (and) in French crime fiction author Maurice Leblanc is 2.65.





  • Present these as interesting recent findings
  • Ask: That’s interesting, isn’t it?
  • Then explain: Of course, these numbers don’t mean anything without relating them to other, comparable values.
  • Numbers only make sense on a known scale, in comparison to other values.

  1. Computational Literary Studies

Sentence length in English-language novels

Source: dragonfly.hypotheses.org/1152

  • Only with a comparison can we answer the question:
  • Is 18.9 words per sentence a lot?
  • Actually it’s not, relatively speaking: it was much higher in earlier periods.

Topics in Arthur Conan Doyle

Source: dragonfly.hypotheses.org/1219

  • Same for the topic probabilities:
  • Only by looking at other probabilities, can we see that 0.38 is actually very high.

Word-frequency in Maurice Leblanc

Source: dragonfly.hypotheses.org/745

  • And just quickly: 2.65 relative frequency for the word “et” is more than in other authors, in this particular comparison.
  • Meaning: In a Bayesian setting, and in the context of stylometric authorship attribution, observing a high frequency of “et” might contribute some evidence towards the hypothesis that the text has been written by Maurice Leblanc.
  • Keyness measures also answer that question:
    • Is this value high or low?
    • Is it unusual or expected?
  • But they do this in a particular context, that of the comparison of groups of texts.
  • Let’s see more closely how keyness can be defined, then.

Definitions: Keyness / Distinctiveness

  • Let’s start with a short recap of some of our conceptual work
  • This was focused on trying to understand what “keyness” really is;
  • and what it means to evaluate a keyness measure.

Traditional definition of keyness

  • Purely quantitative sense: A keyword is “a word which occurs with unusual frequency […] [in a document or corpus] by comparison with a reference corpus”. (Scott 1997)
  • We find this kind of definition rather problematic
  • First reason: It is kind of circular or self-fulfilling;
  • Also: It needs to be changed if the basis of calculation is changed;
  • For example: Many people appear to think nowadays that dispersion should play a role in the calculation of keyness, along with frequency;
  • So apparently, dispersion is relevant to what keyness really is;
  • That can’t be true if keyness is defined by frequency.
  • Therefore: We need to define keyness independently of its statistical operationalization.
  • We consider that a separate definition of keyness is necessary, because with a purely quantitative definition that is identical to a measure, evaluation remains circular
  • This set us off to think about what keyness (or distinctiveness) could mean, qualitatively

Source: User Hyacinth, Source: Wikimedia Commons, Licence: CC BY-SA.

What is Distinctiveness? (Schröter et al. 2021)

  • (A) Logical vs. statistical sense
    • Purely logical: A feature is distinctive of corpus A if its presence in a document D is a necessary and sufficient condition for D to belong to A and not to B.
    • Statistical: A feature is distinctive of corpus A if it is true that, the higher its keyness in document D, the higher the probability that D is an instance of A and not of B.
  • (B) Salient vs. agnostic
    • Salient: A feature is distinctive iff it is noticed by readers (for confirming or violating their expectations)
    • Agnostic: A feature can be distinctive without being salient in the above-mentioned sense.
  • (C) Qualitative vs. no qualitative content
    • Qualitative content: A feature is distinctive iff it expresses e.g. aboutness or stylistic character (=> interpretability)
    • No qualitative content: A feature can be key regardless of qualitative content (=> discriminatory power)
  • We came up with three dimensions of a notion of keyness (or distinctiveness, as we prefer to say)
  • (Explain the three dimensions, then following comments.)
  • There is a certain fit between traditional genre theory (and likewise: philological authorship attribution), the logical concept of keyness, the assumption of salient features, and the assumption of a qualitative content of key features.
  • Similarly, there is a connection between modern theories of genre (hybrid genres, hybrid texts); the statistical sense of keyness; the agnostic sense; and the assumption of no necessary qualitative content.
  • Except that: in CLS, we aim for a statistical and agnostic notion combined with qualitative content of the distinctive features! Tricky.
  • Also: if there are multiple legitimate conceptions of keyness, or aspects of keyness, we may need multiple measures supporting these different understandings.

Various keyness measures (Du et al. 2022)

  • Ratio of relative frequencies
  • TF-IDF
  • Chi-squared test
  • Log-likelihood ratio test
  • LLR effect size – (Evert 2022)
  • Welch’s t-test
  • Wilcoxon rank-sum test
  • Burrows Zeta – (Burrows 2007)
  • Log-Zeta – (Schöch et al. 2018)
  • Eta (difference of DP) – (Gries 2008; Du et al. 2021)
  • That led us to looking at a range of keyness measures
  • Listed here are the measures we have implemendet in our Python package “pydistinto”
  • Several of them are clearly based on corpus-level frequency only: for example RRF, LLR
  • Others use various strategies to take dispersion into account: for example Zeta, Log-Zeta, Eta.
  • Still others rely on distribution within the corpus, but not necessarily just dispersion: for example Welch, Wilcoxon.
  • So what do these measures capture? How do they perform in the context of CLS-related tasks?
  • And what is special about keyness in CLS? We’ll look at keyness in CLS first, before adressing the question of performance and evaluation.

History: Keyness in CLS

  • So, let’s start off with some recent history of keyness, or Zeta, in CLS research.
  • The two are almost identical in this context.

Burrows’ Zeta in Authorship Attribution

  • The key publication is: John Burrows, “All the way through” (Burrows 2007)
  • He proposed to use Zeta in the context of authorship attribution
  • Zeta is calculated as the difference of the document frequencies of a feature in two contrasting sets of documents, where the documents are segments of full texts.
  • Zeta: “A simple measure of [an author’s] consistency in the use of each word-type.” (=> dispersion)
  • Focuses “on a single author and seek[s] to identify which of many texts are most likely to be his or hers.” (=> authorship attribution)
  • The story of Zeta starts quite recently, with a paper by John Burrows.
  • He’s a pioneer of CLS with his 1987 book on Style in Jane Austen.
  • (Then go through the points.)

Zeta for Authorship Attribution: Shakespeare (Craig and Kinney 2009)

  • As an example, this is from work by Hugh Craig and Arthur Kinney on Shakespeare
  • They use a rather typical scenario of using Zeta for authorship attribution
  • Keywords for Shakespeare and other authors are established using Zeta
  • Each segment is plotted as a function of both keyword groups’ proportion in each segment
  • A text of disputed authorship is plotted there as well
  • Depending on the clarity of separation, and where the disputed segments fall, an authorship attribution is made.

Further uses and discussion of Keyness

  • Uses and discussion of Zeta:
    • Hoover (2010)
    • Weidman and O’Sullivan (2018)
    • Rizvi (2019b)
    • Rizvi (2019a)
    • Barber (2021)
    • Rizvi (2022)
    • Hoover (2022)
  • Uses of other keyness measures:
    • using Antconc (log-likelihood)
    • or TXM (spécificité)
  • This is more as a reference, so I’ll skip that slide.
  • Maybe just two words:
    • It is remarkable that there is a strong focus on Zeta, mostly for AA
    • Often using stylo, but just as often own implementations
    • And there is work using other measures, using other tools, but that is less visible.
    • As an aside: It is also remarkable that none of the work on Zeta has ever been noticed by the CCL crowd.

Keyness for gender (Weidman and O’Sullivan 2018)

Scatterplot of segments by male and female authors, by percentage of markers and anti-markers, for three literary periods.

  • Soon, the potential of Zeta for a broader range of contrastive scenarios was discovered
  • What you see here is an example of uses of Zeta for gender and gendered language
  • The tool used here is stylo, mostly used in AA
  • To be honest, I think the authors misread their own plot when commenting on it (mistaking the green and red markers for words, when they are in fact segments)
  • What this shows, in my opinion, is the following:
    • there is a peak of gendered literary language in the modernist period
    • in the middle plot, texts from either group use quite a distinct use of vocabulary;
    • before and after, overlap in vocabulary use is much greater.

Keyness for Genre (Schöch 2018)

PCA plot using 50 Zeta-based keywords. Comedies in red, tragedies in blue, tragi-comedies in green.

  • In a similar spirit of “stylometry beyond authorship”, I started to use Zeta for contrastive analyses of literary subgenres
  • Literary studies compares things all the time, so keyness seems more generally relevant than just for authorship attribution
  • The figure shown here concerns French comedies, tragedies and tragi-comedies
  • The aim was o settle the question of whether tragi-comedies are a special type of comedies, of tragedies, or something in between.
  • 50 words identified by Zeta to be distinctive of comedies or of tragedies were identified
  • Their frequencies were then used for a PCA on texts of the three genres, so including “tragi-comedies”
  • One can see that comedies (in red, on the left) use quite a distinct vocabulary;
  • Tragedies (in blue, on the right) are also quite distinct;
  • But tragi-comedies (in green), overlap more with tragedies than with comedies.

Zeta and Company / Beyond words

  • Projects funded by the German Research Foundation (DFG, 2020-2023, 2024-2026)
  • Domain of application: popular subgenres of the 20th-century French novel
  • Inspirations: John Burrows (Burrows 2007), Jeffrey Lijffijt (Lijffijt et al. 2014), MOTIFS (e.g. Kraif and Tutin 2017), Phraséorom (e.g. Diwersy et al. 2021), dispersion (Gries 2008)
  • Fundamental aim: Enable scholars in CLS to make educated choices about what keyness measure to use
  • Also: Bridge the gap between Computational Linguistics and Computational Literary Studies
  • Activities: modeling, implementing, evaluating and using statistical measures of comparison of two groups of texts.
  • This work led to a proposal for a stand-along project on keyness
  • Modeling: Try to establish a systematic understanding allowing us to describe all measures of distinctiveness using the same descriptive elements
  • Implementing: Developing a Python package that provides multiple measures in the same framework
  • Evaluating: Using quantitative methods (e.g. downstream classification task) and qualitative methods (application study)
  • Using: Trying to answer the question of what we can find out about subgenres using keyness measures
  • In the following, I will just focus on two areas: conceptual work and one evaluation task.

Quantitative Evaluation:
Classification Task

  • This is what we try to find out in our various evaluation tasks.
  • One of them is focused on the “discriminative power” of the measures (statistical, agnostic notion of keyness)
  • It is based on a downstream classification task involving literary subgenres
  • We have other evaluation strategies, but I’ll focus on this one first.

Data: Corpus of French contemporary fiction


  • 320 novels in 8 groups (4 subgenres x 2 decades)
  • Published as derived text format – 10.5281/zenodo.7111522
  • Genres: 4 groups (crime, sentimental, science-fiction and general narrative fiction)
  • Period: 2 groups (1980s, 1990s)
  • Size: 40 novels for 8 groups = 320 novels, 17.3 million words
  • (Today: 50 novels per decade and genre, 1970-2000 = 600 novels)

Evaluation Task: Genre Classification

  • Downstream classification task: “How reliably can a machine learning classifier, based on words identified using a given measure of distinctiveness, identify the subgenre of a novel when provided only with a short segment of that novel?”
  • Basic setup
    • 4 classifiers
    • Different numbers of keywords (N)
    • Textual units are 5000-word segments
    • 10-fold-cross validation (90/10 split of segments)
    • Baseline: random selection of N words
  • Why a classification task? Because annotating a gold standard for keyness is not possible (because target and comparison corpus as a whole need to be considered, a task that is cognitively impossible for humans)
  • And because evaluating the appropriateness of a list of keywords for the category it is supposed to represent is difficult to formalize.
  • However, readers can identify the subgenre a novel belongs to rather quickly, after reading just a few paragraphs or pages.
  • Therefore: Can a machine learning classifier mirror this capacity?
  • And: What measures provide the most useful features for this?
  • Note that we are here in the statistical and agnostic paradigm, and so primarily in the meaning of keywords as words with “discriminatory power”
  • However, we are interested also in the interpretability of lists of distinctive words

Reference: (Du, Dudar, and Schöch 2022)

Results #1

Classification performance on the French corpus (1980s) with four classifiers, depending on the measure of distinctiveness and the setting of 𝑁.

  • If we vary N, the larger N, the better the performance and the smaller the differences between the classifiers
  • That makes sense, of course, as the classifiers just get more information to work with
  • Does this mean the measure doesn’t matter for discriminatory power?
  • Not really:
    • in a CLS context where interpretation of keyword lists matters as much as discriminatory performance
    • if the most useful keywords are also to be inspected qualitatively, their number needs to remain limited.
  • Therefore, the measure matters and dispersion-based measures do better

Results #2

Distribution of classification performance on the 1980s French corpus with N = 10 using Multinomial Naive Bayes

  • These are our detailed results for small N: 10 keywords and 10 negative keywords per group were used, for a total of 80 words
  • There is clearly a top group, made up of the Zetas, Eta and TF-IDF
  • There is a median group, with Wilcoxon and Welch
  • And there is a low-performing group, with LLR, Chi-Squared and RRF
  • Tested also for seven different ELTeC corpora, confirming the overall result.
  • So for small numbers of features, dispersion-based measures provide words with stronger discriminatory power than other measures.
  • Such measures also tend to select medium-frequency words that are highly-interpretable (=> salient, qualitative)

Qualitative Evaluation:
subgenre profiles

Why qualitative evaluation?

  • Different kinds of evaluation test different aspects of a measure
    • Quantitative evaluation: checks discriminative power
    • Qualitative evaluation: focuses on interpretability and aboutness

Our approach: match keywords with sugenre profiles

  • Establish ‘subgenre profiles’
    • Based on scholarly literature
    • Systematically describe subgenres:
      setting, characters, themes, narrative form, language, etc.
  • Create lists of keywords for each subgenre
    • For three measures: Log-likelihood ratio, Zeta, Welch
    • Annotate the lists wrt to relevance to subgenre profiles

Reference: (Röttgermann et al. submitted)

Example: ‘setting’ in science fiction

Science fiction Zeta LLR Welch
Space / Setting: Solitary settings are typical, e.g. space, desert or the arctic. Vast and imaginative array of settings, e.g. space, alternate universes, hallucinatory landscapes, the moon, Mars. planet, space, surface, universe, ground, star, zone, outside, spatial planet, space, universe planet, ground, space, surface, outside, universe, zone
9/50 3/50 7/50

Results: Interpretability of measures

  • Interannotator agreement (Cohen’s kappa): κ=0.33 (low!)
  • General interpretability (left)
    • high: Zeta and Welch
    • low: Log-likelihood ratio
  • Matches of keywords with subgenre profile categories (right)
    • Highly variable, somewhat inconclusive

  • For crime fiction: language patterns and others
  • for scifi: theme, language patterns, thematic concepts
  • For sentimental: thematic concepts, narrative form
  • For literary fiction: mostly thematic concepts

Conclusion: Findings and Outloook

  • So, time to wrap up.

What have we found out so far?

  • Definition
    • Keyness should not be defined as ‘unusual frequency’; aspects like discriminatory power, salience or aboutness are important.
    • A match between a certain understanding of keyness and a specific measure can be established using a suitable method of evaluation.
  • Evaluation
    • Dispersion-based keyness measures show best performance in a subgenre classification task, especially when the number of features is small
    • Dispersion and distribution-based measures tend to select medium-frequency words that are highly-interpretable and fare well in a qualitative evaluation
  • Zeta
    • Zeta is a fully competitive keyness measure, and often preferable to Log-likelihood ratio in the context of CLS

What are the next steps?

  • Perform further experiments, using synthetic texts and test tokens with pre-determined frequency- and/or dispersion-based contrasts
  • Consider additional measures:
    • measure based on DPnofreq (Gries 2021),
    • LRC / effect size (Evert 2022);
    • Fisher’s exact test (Lebart and Salem 1994)
  • Move on to more complex features: multi-word expressions and semantic features (Beyond Words, 2024-2026)
  • Find a strategy for how to handle a multi-dimensional approach to keyness (multiple meanings, multiple measures), e.g. along the lines proposed in (Gries 2019)
  • Present our future work.
  • I hope I was able to show why keyness is an interesting topic.
  • And I hope that I was able to illustrate to you where research into keyness stands in CLS.




Thank you for your kind attention!

  • Presentation: https://dhtrier.quarto.pub/knu
  • Contact: schoech@uni-trier.de

References

Barber, Ros. 2021. “Big Data or Not Enough? Zeta Test Reliability and the Attribution of Henry VI.” Digital Scholarship in the Humanities 36 (3): 542–64. https://doi.org/10.1093/llc/fqaa041.
Burrows, John. 2007. “All the Way Through: Testing for Authorship in Different Frequency Strata.” Literary and Linguistic Computing 22 (1): 27–47. https://doi.org/10.1093/llc/fqi067.
Craig, Hugh, and Arthur F. Kinney. 2009. Shakespeare, Computers, and the Mystery of Authorship. Cambridge University Press.
Diwersy, Sascha, Laetitia Gonon, Vannina Goossens, Olivier Kraif, Iva Novakova, Julie Sorba, and Ilaria Vidotto. 2021. “La phraséologie du roman contemporain dans les corpus et les applications de la PhraseoBase.” Corpus, no. 22. https://doi.org/10.4000/corpus.6101.
Du, Keli, Julia Dudar, Cora Rok, and Christof Schöch. 2021. “Zeta & Eta: An Exploration and Evaluation of Two Dispersion-based Measures of Distinctiveness.” In Proceedings of the Conference on Computational Humanities Research 2021, edited by Maud Ehrmann, Folgert Karsdorp, Melvin Wevers, Tara Lee Andrews, Manuel Burghardt, Mike Kestemont, Enrique Manjavacas, Michael Piotrowski, and Joris van Zundert, 2989:181–94. CEUR Workshop Proceedings. Amsterdam, the Netherlands: CEUR.
———. 2022. “Kontrastive Textanalyse mit pydistinto - Ein Python-Paket zur Nutzung unterschiedlicher Distinktivitätsmaße.” Potsdam: Zenodo. https://doi.org/10.5281/zenodo.6327967.
Du, Keli, Julia Dudar, and Christof Schöch. 2022. “Evaluation of Measures of Distinctiveness. Classification of Literary Texts on the Basis of Distinctive Words.” Journal of Computational Literary Studies 1 (1). https://doi.org/10.48694/jcls.102.
Evert, Stephanie. 2022. “Measuring Keyness.” In Book of Abstracts of the Digital Humanities 2022. Tokyo: ADHO. https://doi.org/10.17605/OSF.IO/CY6MW.
Gries, Stefan Th. 2008. “Dispersions and Adjusted Frequencies in Corpora.” International Journal of Corpus Linguistics 13 (4): 403–37. https://doi.org/10.1075/ijcl.13.4.02gri.
———. 2019. “15 Years of Collostructions: Some Long Overdue Additions/Corrections (to/of Actually All Sorts of Corpus-Linguistics Measures).” International Journal of Corpus Linguistics 24 (3): 385–412. https://doi.org/10.1075/ijcl.00011.gri.
———. 2021. “What Do (Most of) Our Dispersion Measures Measure (Most)? Dispersion?” https://doi.org/10.1075/jsls.21029.gri.
Hoover, David L. 2010. “Teasing Out Authorship and Style with t-Tests and Zeta.” In Digital Humanities Conference. London.
———. 2022. “Zeta Revisited.” Digital Scholarship in the Humanities 37 (4): 1002–21. https://doi.org/10.1093/llc/fqab095.
Kraif, Olivier, and Agnès Tutin. 2017. “Des motifs séquentiels aux motifs hiérarchiques : l’apport des arbres lexico-syntaxiques récurrents pour le repérage des routines discursives.” Corpus, no. 17. https://doi.org/10.4000/corpus.2889.
Lebart, Ludovic, and André Salem. 1994. Statistique Textuelle. Dunod.
Lijffijt, Jefrey, Terttu Nevalainen, Tanja Säily, Panagiotis Papapetrou, Kai Puolamäki, and Heikki Mannila. 2014. “Significance Testing of Word Frequencies in Corpora.” Digital Scholarship in the Humanities 31 (2): 374–97. https://doi.org/10.1093/llc/fqu064.
Rizvi, Pervez. 2019a. “An Improvement to Zeta.” Digital Scholarship in the Humanities 34 (2): 419–22. https://doi.org/10.1093/llc/fqy039.
———. 2019b. “The Interpretation of Zeta Test Results.” Digital Scholarship in the Humanities 34 (2): 401–18. https://doi.org/10.1093/llc/fqy038.
———. 2022. “The Interpretation of Zeta Test Results: A Supplement.” Digital Scholarship in the Humanities 37 (4): 1172–78. https://doi.org/10.1093/llc/fqac011.
Röttgermann, Julia, Keli Du, Julia Dudar, and Christof Schöch. submitted. “Expertise vs. Statistics: A Qualitative Evaluation of Three Keyness Measures Applied to Subgenres of the French Novel.” DHQ, submitted.
Schöch, Christof. 2018. “Zeta für die kontrastive Analyse literarischer Texte. Theorie, Implementierung, Fallstudie.” In Quantitative Ansätze in den Literatur- und Geisteswissenschaften. Systematische und historische Perspektiven, edited by Toni Bernhart, Sandra Richter, Marcus Lepper, Marcus Willand, and Andrea Albrecht, 77–94. Berlin: de Gruyter.
Schöch, Christof, Daniel Schlör, Albin Zehe, Henning Gebhard, Martin Becker, and Andreas Hotho. 2018. “Burrows : Exploring and Evaluating Variants and .” In Book of Abstracts of the Digital Humanities Conference. Mexico City: ADHO.
Schröter, Julian, Keli Du, Julia Dudar, Cora Rok, and Christof Schöch. 2021. “From Keyness to Distinctiveness Triangulation and Evaluation in Computational Literary Studies.” Journal of Literary Theory 15 (1-2): 81–108. https://doi.org/10.1515/jlt-2021-2011.
Scott, Mike. 1997. “PC Analysis of Key Words And Key Key Words.” System 25 (2): 233–45. https://doi.org/10.1016/S0346-251X(97)00011-0.
Weidman, Sean G., and James O’Sullivan. 2018. “The Limits of Distinctive Words: Re-evaluating Literature’s Gender Marker Debate.” Digital Scholarship in the Humanities 33 (2): 374–90.

Bonus slides

Measures with references (Du, Dudar, and Schöch 2022)

All corpora (Du, Dudar, and Schöch 2022)

Correlation between measures (Du, Dudar, and Schöch 2022)

Keyness in stylo: genre (A.C. Doyle)

   

  • Here, a simple comparison of Doyle’s detective novels agains other novels
  • In the left plot, the focus is on the keywords
    • words “preferred” by detectve fiction on the right
    • words “avoided” by detective fiction on the left
  • In the right plot, the focus is on text segments
    • segments from detective fiction are in red
    • segments from other fiction are in green
    • their position is determined by the percentage of marker and antimarker words
  • In any case, the shift from using Zeta for authorship attribution to using Zeta for text group comparison clearly happened.

https://dhtrier.quarto.pub/knu – CC BY

1
Comparing Comparisons Evaluating Measures of Keyness or Distinctiveness in Computational Literary Studies Christof Schöch (Trier University, Germany) Kyungpook National University Daegu, South Korea 22 May 2024

  1. Slides

  2. Tools

  3. Close
  • Comparing Comparisons
  • Introduction
  • Thanks
  • Overview
  • What is keyness all about?
  • Some recent findings in my work in CLS1
  • Sentence length in English-language novels
  • Topics in Arthur Conan Doyle
  • Word-frequency in Maurice Leblanc
  • Definitions: Keyness / Distinctiveness
  • Traditional definition of keyness
  • What is Distinctiveness? (Schröter et al. 2021)
  • Various keyness measures (Du et al. 2022)
  • History: Keyness in CLS
  • Burrows’ Zeta in Authorship Attribution
  • Zeta for Authorship Attribution: Shakespeare (Craig and Kinney 2009)
  • Further uses and discussion of Keyness
  • Keyness for gender (Weidman and O’Sullivan 2018)
  • Keyness for Genre (Schöch 2018)
  • Zeta and Company / Beyond words
  • Quantitative Evaluation:Classification Task
  • Data: Corpus of French contemporary fiction
  • Evaluation Task: Genre Classification
  • Results #1
  • Results #2
  • Qualitative Evaluation:subgenre profiles
  • Why qualitative evaluation?
  • Our approach: match keywords with sugenre profiles
  • Example: ‘setting’ in science fiction
  • Results: Interpretability of measures
  • Conclusion: Findings and Outloook
  • What have we found out so far?
  • What are the next steps?
  • Thank you for your kind attention!
  • References
  • Bonus slides
  • Measures with references (Du, Dudar, and Schöch 2022)
  • All corpora (Du, Dudar, and Schöch 2022)
  • Correlation between measures (Du, Dudar, and Schöch 2022)
  • Keyness in stylo: genre (A.C. Doyle)
  • f Fullscreen
  • s Speaker View
  • o Slide Overview
  • e PDF Export Mode
  • ? Keyboard Help