pipette
ENEnglish

Comparison of Common Crawl News & GDELT

Ameir El Ouadi, David Beskow

Preprint con versión publicadaFácil de leer

En palabras de los autores

The corpus of worldwide news is important for natural language processing, knowledge graphs, large language models, and other technical efforts. Additionally, this corpus is important for understanding the people, places, organizations, and events that interact in real-time every day. This paper compares two news datasets used for these tasks today, namely the Global Database of Events, Language, and Tone (GDELT) and Common Crawl News. Our research highlights the strengths and limitations of each dataset, analyzing their content and coverage. Notably, while GDELT relies on broadcasts, prints, and web news from across the globe, Common Crawl focuses on news sites from around the world gathered through web crawling. Our analysis revealed considerable differences in where the two datasets gather their news sources.

Resultado principalEl resumen no menciona limitaciones.

Apareció: viernes, 2 de octubre. arXiv. Preprint con versión publicada.

Versión publicada: El Ouadi, A., & Beskow, D. (2024, April). Comparison of common crawl news & GDELT. In 2024, IEEE International Systems Conference (SysCon) (pp. 1-3). IEEE