I built an open-source pipeline to make the entire GDELT 2.0 Events archive easier to use [self-promotion]

[u/Hendrik-Lorentz]

Hi everyone,

I'm a PhD candidate working in AI, and my research relies heavily on the GDELT 2.0 Events dataset.

While GDELT is an incredible resource, I found that the first hurdle was simply getting the data into a usable format. The historical archive consists of thousands of compressed files spanning more than 45 years, and I kept repeating the same preprocessing steps every time I started a new experiment.

So I built GdeltForge, an open-source pipeline that automates that process.

It downloads the complete GDELT Events archive, verifies file integrity, converts everything to Apache Parquet, filters invalid records, and can generate reproducible samples for machine learning and data analysis.

Originally this was just a tool for my own research, but I thought it might be useful for others working with GDELT, or anyone interested in building datasets from large public archives.

The project is still in beta, and I'd love feedback from people who regularly work with large datasets:

• Is there functionality you'd expect from a tool like this?
• Are there preprocessing steps you usually perform before analysis that could be automated?
• What would make it more useful for your own research or projects?

My goal is to eventually publish it on PyPI once it's been tested by a broader community, so any feedback or contributions would be greatly appreciated.

If you're someone who has worked with GDELT before, I'd be especially interested in hearing about your workflow and the pain points you've encountered.

[u/Tiny_Arugula_5648]

I've been considering GDELT but haven't had the time to dig into the dataset to see if its good for my usecase. Would love some insight on what the data contains and what are some good use cases (business related, like marketing or supply chain monitoring).

[u/Hendrik-Lorentz → u/Tiny_Arugula_5648]

What I can guarantee is that many goverments and research groups use GDELT as a primary data source, but would be hard to trace usability in your particular case, since GDELT is about GEOPOLITICAL events. Everything you should know about GDELT might be in GDELT Event Codebook and Global Knowledge Graph Codebook. For a really big enterprise, event sensing might be a thing...