Text datasets · GitHub stars

1alibaba / DataXDataX is the open source version of Alibaba Cloud DataWorks data integration.JavaLicense not declared17,383

2brightmart / nlp_chinese_corpusA collection of Chinese-language corpora for natural-language processing.MIT · Permissive9,937

3openlm-research / open_llamaOpenLLaMA, a permissively licensed open source reproduction of Meta AI’s LLaMA 7B trained on the RedPajama datasetApache-2.0 · Permissive7,518

4SophonPlus / ChineseNlpCorpusCollect, organize, and publish Chinese natural language processing corpus/data sets, and work with people with lofty ideals to promote the development of Chinese natural language processing.Jupyter NotebookLicense not declared6,613

5cocodataset / cocoapiCOCO API - Dataset @ http://cocodataset.org/Jupyter NotebookLicense not declared6,390

6CSAILVision / semantic-segmentation-pytorchPytorch implementation for Semantic Segmentation/Scene Parsing on MIT ADE20K datasetPythonBSD-3-Clause · Permissive5,088

7larashero3-dotcom / lieflat-less-ai-toneAn AI-tone removal skill grounded in a 5.2-million-character corpus studyPythonLicense not declared4,946

8minimaxir / textgenrnnEasily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code.PythonLicense not declared4,917

9fchollet / ARC-AGIThe Abstraction and Reasoning CorpusJavaScriptApache-2.0 · Permissive4,855

10InsaneLife / ChineseNLPCorpusChinese natural language processing data set, used for daily experiments. Additional submissions and merges are welcome.PythonLicense not declared4,627

11first20hours / google-10000-englishThis repo contains a list of the 10,000 most common English words in order of frequency, as determined by n-gram frequency analysis of the Google's Trillion Word Corpus.License not declared4,500

12CLUEbenchmark / CLUEDatasetSearchSearch all Chinese NLP data sets, with commonly used English NLP data sets attachedPythonLicense not declared4,459

13wainshine / Chinese-Names-CorpusChinese personal names corpus. Name generator. Chinese name, surname, first name, title, Japanese name, translated name, English name. It can be used for Chinese word segmentation and name entity recognition.Apache-2.0 · Permissive4,344

14esbatmop / MNBVCMNBVC (Massive Never-ending BT Vast Chinese corpus) is a very large-scale Chinese corpus. 40T data trained against chatGPT. The MNBVC data set includes not only mainstream culture, but also data from various niche cultures and even Martian literature. The MNBVC data set includes all forms of plain text Chinese data such as news, compositions, novels, books, magazines, papers, lines, posts, wikis, ancient poems, lyrics, product introductions, jokes, embarrassing stories, chat records, etc.MIT · Permissive4,297

15NVlabs / ffhq-datasetFlickr-Faces-HQ Dataset (FFHQ)PythonLicense not declared4,203

16commaai / researchdataset and code for 2016 paper "Learning a Driving Simulator"PythonBSD-3-Clause · Permissive4,128

17TommyZihao / Train_Custom_DatasetLabel your own data sets, train, evaluate, test, and deploy your own artificial intelligence algorithmsJupyter NotebookLicense not declared4,125

18verazuo / jailbreak_llms[CCS'24] A dataset consists of 15,140 ChatGPT prompts from Reddit, Discord, websites, and open-source datasets (including 1,405 jailbreak prompts).Jupyter NotebookMIT · Permissive3,879

19linhandev / datasetList of medical imaging data sets 『An Index for Medical Imaging Datasets』License not declared3,640

20ben1234560 / AiLearning-Theory-ApplyingQuick Start to AI Theory and Practical Applications: Basic knowledge, Transformer, NLP, ML, DL, and competitions. Contains numerous annotations and datasets, aiming to make it easy for everyone to understand and replicate.Jupyter NotebookMIT · Permissive3,585