List of multilingual text corpora for natural language processing

If you want to train your own large language model, try developing a stylometry model, or simply hone your NLP skills, you will need a corpus to work with.

Some of the best known corpora include:

• The Open American National Corpus (OANC)
• The British National Corpus (BNC)
• Project Gutenberg
• Text corpora on authorship attribution (University of Neuchatel, the CLC group)
• The CLEF PAN corpora and tasks
• The Federalist Papers - a series of 85 essays written by Alexander Hamilton, John Jay, and James Madison. Some are of disputed authorship.

One easy way to load datasets in your code is with the HuggingFace Datasets package.

Below you can search a comprehensive table of corpora in different languages. If you see one that is missing, please contact Fast Data Science, the maintainer of this list.