Which NLP corpus?

What is an NLP text corpus?

An NLP corpus (plural: “corpora”) is a collection of text organised into datasets. Corpora can be comprised of newspapers, novels, social media, transcripts of interviews or television shows, or other authentic text. An NLP corpus is useful for corpus linguistics, computational linguistics, and natural language processing research. A corpus could be raw text as originally spoken or written, or it could include annotations (annotated for parts of speech, such as nouns, verbs, and adjectives, or annotated by topic).

List of multilingual text corpora for natural language processing

If you want to train your own large language model, try developing a stylometry model, or simply hone your NLP skills, you will need an NLP corpus to work with.

Some of the best known NLP corpora include:

• The Open American National Corpus (OANC)
• The British National Corpus (BNC)
• Project Gutenberg
• Text corpora on authorship attribution (University of Neuchatel, the CLC group)
• The CLEF PAN corpora and tasks
• The Federalist Papers - a series of 85 essays written by Alexander Hamilton, John Jay, and James Madison. Some are of disputed authorship.

One easy way to load an NLP corpus in your code is with the HuggingFace Datasets package.

Below you can search a comprehensive table of NLP corpora in different languages. If you see one that is missing, please contact Thomas Wood at Fast Data Science, the maintainer of this list.