Measuring Manuscripts · ESU DH 2026
How do NER models with different linguistic and historical training backgrounds process highly abbreviated Latin regesta?
This project explores the use of named entity recognition (NER) on the Latin regesta of Repertorium Germanicum I. These texts differ substantially from the modern, standardized language on which many NER systems are trained: they are highly compressed, contain frequent abbreviations, and include inflected or Latinized forms of personal and geographical names.
This project applies three pretrained NER models to text segments extracted from the XML and compares their outputs. Each model represents a different training context: general Latin, medieval documentary Latin, and modern multilingual NER. The aim is not to identify a definitively “best” model, but to investigate how different NER systems respond to the linguistic and editorial challenges of historical regesta.
The Repertorium Germanicum, produced by the German Historical Institute in Rome, records people, churches, and places connected with the German-speaking territories of the Holy Roman Empire as they appear in Vatican registers. This project uses its first volume,RG I, available through Repertorium Germanicum Online.
RG contains regesta rather than complete transcriptions. These concise scholarly summaries preserve essential information about people, places, institutions, dates, and legal actions without reproducing the complete wording of the original documents.
“Conrado dicto Schilher ecclesiam par. in Tronstat Herbip. dioc.”
This short extract contains a personal name, a place name, and several abbreviations. It demonstrates the compressed and entity-dense character of the regesta. Such features are common across the corpus: abbreviations occur in 87.9% of all extracted segments. Historical and Latinized names, abbreviated contexts, and compressed syntax all present potential challenges for NER.
The XML was parsed into 9,434 individual text segments: 3,845 heads and 5,589 sublemmata belonging to 3,845 lemmas. XML markup for abbreviations, dates, and source references was extracted into separate tables.
For the NER analysis, structured source references were removed, while original spelling, punctuation, dates, and abbreviations were preserved. Each head and sublemma was then stored as an individual row, producing a consistent tabular dataset that could be processed by all three models.
Three pretrained models were selected to represent three different training contexts: a general Latin NLP pipeline, a model trained on medieval documentary languages, and a modern multilingual NER model. None of the models was further trained or fine-tuned on RG I.
General Latin baseline
LatinCy is a general-purpose Latin NLP pipeline based on spaCy. It was selected as the Latin-specific baseline because it is designed to process Latin texts and includes a named entity recognition component.
Labels:
PERSON,
LOC,
NORP
Medieval documentary model
This XLM-RoBERTa model was fine-tuned on approximately 8,000 annotated medieval charter texts written in Medieval Latin, Old French, and Old Spanish, dating from the eleventh to the fifteenth centuries. Its chronological and documentary training context is therefore the closest to RG I.
Labels:
PERS,
LOC
Modern multilingual baseline
This XLM-RoBERTa model was fine-tuned for NER in ten modern high-resource languages, including German but not Latin. It was selected to test whether multilingual training, particularly exposure to German, helps recognize Germanic personal and place names occurring in Latinized historical texts.
Labels:
PER,
LOC,
ORG
The three models use slightly different names for their entity categories.
LatinCy uses PERSON, Medieval RoBERTa uses
PERS, and Multilingual XLM-R uses PER.
For the comparison, PERS and PER were converted
to PERSON.
The quantitative comparison focuses on PERSON and
LOC, since these are the only two categories shared by all
three models. The model-specific categories NORP and
ORG were kept in the complete results but were not included
in the exact-overlap comparison.
All three models were applied to the same 9,434 text segments. Since RG I
has no gold-standard NER annotations, the following results describe
differences and similarities in model output rather than model accuracy.
Only PERSON and LOC, the two labels shared by all
three models, are included in the direct comparison.
The first comparison counts how many different PERSON and LOC strings were identified by each model.
| Model | Unique forms | Relative number |
|---|---|---|
| LatinCy | 4,281 | |
| Medieval RoBERTa | 7,602 | |
| Multilingual XLM-R | 7,301 |
Medieval RoBERTa produced the largest number of unique entity strings, followed closely by Multilingual XLM-R. LatinCy produced substantially fewer unique forms. This may reflect differences in entity boundaries and name segmentation, but it does not indicate which model is more accurate.
These figures represent unique surface forms, not uniquely identified historical people or places. For example, Johannes, Iohannes, and Johannes de Leone are counted as different strings.
Two predictions were counted as an exact match only when they occurred in the same segment and had identical character boundaries and entity labels. The Jaccard percentage compares the exact matches with the combined set of predictions produced by each pair of models.
| Model pair | Exact matches | Jaccard | Relative overlap |
|---|---|---|---|
| LatinCy × Medieval RoBERTa | 2,670 | 8.8% | |
| LatinCy × Multilingual XLM-R | 2,064 | 7.5% | |
| Medieval RoBERTa × Multilingual XLM-R | 5,232 | 22.3% |
Medieval RoBERTa and Multilingual XLM-R showed the highest exact agreement, with 5,232 shared predictions and a Jaccard similarity of 22.3%. LatinCy showed considerably lower exact agreement with both Transformer models.
Exact matching is deliberately strict. For example, Johannes and Johannes de Leone are counted as different predictions because their entity boundaries do not match. The percentages therefore measure output similarity rather than accuracy.
Since RG I has no gold-standard NER annotations, the following analysis does not provide a formal evaluation of model errors. Instead, it compares two cases in which the models produce different entity boundaries, labels, and surface forms. The XML structure and the surrounding Latin context are used to interpret these differences.
Case 1 · Abbreviated and Variant Name
Lemma head: Johannes de Kingstein rector par. eccl. de Brengarten Constant. dioc.
Sublemma processed by the models: cf. Joh. de Kunigstein.
| Model | Predicted entities | Observation |
|---|---|---|
| LatinCy |
Ioh
PERSON
|
Identifies only the abbreviated given name and changes Joh. to the normalized form Ioh. |
| Medieval RoBERTa |
Joh. de
PERSON;
Kunigstein.
PERSON
|
Detects the complete expression but separates it into two person entities. |
| Multilingual XLM-R |
Joh. de Kunigstein.
PERSON
|
Treats the complete multiword name as a single person entity. |
The lemma head provides the fuller form Johannes de Kingstein, while the sublemma contains the abbreviated and orthographically different form Joh. de Kunigstein. All three models react to the personal name, but only XLM-R produces a single entity boundary covering the complete expression. The case also demonstrates a limitation of segment-level processing: none of the models can access the fuller name contained in the lemma head.
Case 2 · Person and Place Boundaries
Jacobus filius quondam Petri de Leyden presb. Traiect. dioc. perp. vicarius eccl. s. Petri Middelburgen. Traiect. dioc.
| Model | Predicted entities | Observation |
|---|---|---|
| LatinCy |
Iacobus
PERSON;
Petri
PERSON;
Leyden
PERSON;
Traiect
PERSON;
Petri
PERSON;
Middelburgen
LOC;
Traiect
PERSON
|
Finds the largest number of entity components, but fragments personal names and labels both occurrences of Traiect. as persons. |
| Medieval RoBERTa |
Jacobus
PERSON;
Petri de
PERSON;
Leyden
PERSON;
Petri Middelburgen.
PERSON
|
Separates Petri de Leyden and merges the saint’s name with the following geographical expression. |
| Multilingual XLM-R |
Jacobus
PERSON;
Petri de Leyden
PERSON;
Petri Middelburgen.
PERSON
|
Produces the most coherent boundary for Petri de Leyden, but merges Petri and Middelburgen. and misses both occurrences of Traiect.. |
Context suggests that Petri de Leyden is the name of Jacobus’s father, while Leyden is also a place-derived element within that name. Later in the text, Petri refers to St Peter in the name of a church, while Middelburgen. and Traiect. provide geographical information. No model captures all of these distinctions.
PERSON entity. A nested annotation scheme could
instead treat the full expression as PERSON while also
marking Leyden as a LOC. Neither choice is
automatically correct without an explicit annotation guideline.
These cases show that the main challenge is not simply whether a model detects an entity. The models also differ in orthographic normalization, entity boundaries, and the interpretation of place names embedded in personal and ecclesiastical expressions. Consequently, the model with the highest number of predictions is not necessarily the model that produces the most useful analysis for historical regesta.
This project extracted 9,434 text segments from the RG I XML, applied three pretrained NER models, and compared their entity counts, output agreement, and predictions in selected cases. In the current setup, LatinCy appears to be the least suitable model for this corpus: it frequently fragments multiword names, assigns questionable labels to abbreviated place names, and changes original forms through orthographic normalization.
The choice between the two Transformer models is less straightforward. Medieval RoBERTa has the closest historical training context, while Multilingual XLM-R often produces more coherent name boundaries despite not being trained on Latin NER data. A more reliable evaluation would require greater familiarity with the editorial conventions of RG I, clearer decisions about person–place boundaries, and a small manually annotated reference sample.