Photo: Claudio Schwarz · Unsplash

A multilingual, sentence-level news corpus.

MEDIATEXT is a research corpus compiled at poltextLAB. News articles are segmented to the sentence and coded for CAP (Comparative Agendas Project) policy topic, sentiment, and named entities. Two releases are documented in companion data papers currently in editorial review — MediaText Central-Eastern Europe (1.4M+ V4 articles) and MediaText NYT (the New York Times, 1851–present). Both share a single coding stack and codebook.

What you're looking at

MEDIATEXT is a sentence-level news corpus released in two sub-collections. MediaText CEE covers Czechia, Hungary, Poland, and Slovakia at sentence granularity — over 1.4 million articles coded for CAP policy topic, per-sentence sentiment, and named entities. MediaText NYT covers the New York Times from 1851 to the present, with the closed historical period (1851–2024) deposited on Harvard Dataverse and OSF. Each release is documented in a companion data paper currently in editorial review.

Coverage

  • Czechia · 1990–2024
  • Hungary · 1990–2024
  • Poland · 1990–2024
  • Slovakia · 1990–2024
  • New York Times · 1851–present (closed period 1851–2024)

How the corpus was built

Source articles were collected from publisher feeds and archives, stripped to clean main-text with trafilatura, segmented into sentences, and labelled by the coding stack — the Babel Machine multilingual transformer paired with the perfAPI gateway running gpt-oss:120b. Each sentence receives a CAP major and minor topic, a sentiment score, named-entity spans, and (for MediaText NYT) emotion labels (anger, fear, sadness, joy, surprise). Document-level aggregates are derived underneath.

Topic coding

Sentences are coded under the Comparative Agendas Project (CAP) Major + Minor taxonomy — the same policy taxonomy used across the parlTEXT and ParlSpeech datasets. The codebook is documented in the companion data papers; an in-corpus copy ships alongside the data deposit.

Where the data lives

A public reading portal at mediatext.poltextlab.com exposes overview tables, sentiment distributions, NER word clouds, and a per-row data explorer for the V4 release. The full programmatic release — vector embeddings on every text variable plus structured metadata — is deposited on Harvard Dataverse and OSF. See Releases for the per-corpus distribution table.

Where it came from

MEDIATEXT is compiled at poltextLAB — the AI laboratory of the ELTE Centre for Social Sciences, Budapest — under the V-SHIFT Momentum project led by Miklós Sebők. It extends poltextLAB's earlier corpus work: the Hungarian Comparative Agendas Project, the parlTEXT parliamentary-speech corpus, and the Babel Machine multilingual classifier.

License

Data is released under Creative Commons BY-NC 4.0. Please cite the relevant data paper and credit the originating publisher as the source of the underlying article text.

01
Sentence-level granularity
Every article is segmented into sentences. Each sentence carries its own coded labels — CAP topic, sentiment, NER spans — with document-level aggregates rolled up underneath.
02
Multilingual
Czech, Hungarian, Polish, and Slovak for the V4 release; English for the NYT release, with archival language detection covering Spanish (/es/), Arabic, French, Chinese, and German via fastText + heuristics.
03
Two access tiers
Public reading portal + Dataverse deposit for title-level metadata and binary text-availability indicators; gated programmatic deposit for vector embeddings on every text variable.
04
Versioned releases
Each sub-corpus has explicit coverage windows (1851–2024 for the NYT closed period; 1990–2024 for V4). Future vintages are appended as numbered releases on Dataverse rather than overwritten in place.