Dataset - Le Monde Guerre en Ukraine
Always the same boring english datasets.
Out of curiosity and as an avid reader of Le Monde, here a Dataset collected from my fav newspaper : 1 year coverage of the Ukraine Invasion (Feb 24 2022 -> 2023) as well as the tools used to build it.
You might want to check the subsequent analysis I made out of this data on the sibling repo or access this rendered version
Important : the data is collected and shared by me for educational & research purposes only ; premium articles (suscriber only) have been truncated to first 2500 characters.
Dataset
Download /dataset (Compressed Parquet, 40mb)
236 k comments and associated articles (2 k unique), title, content (truncated if premium), desc & date
Remarks / Limitations :
- Articles truly about Ukraine War, not a simple mention, using a prior filter on articles tags.
- Lives and Blog type articles not collected; all other types are (Edito etc.)
- Articles authors (journalists) not collected, purposely.
- No distinction between comments and replies-to-comment.
- No timestamp, only associated article (last) publication date.
Workflow, things you might re-use
1. Data Collection
Custom API (lmd_ukr/api.py)
- To be seen as a good, but not top (i.e. scalable etc.) one-shot-project" API, shared "as is"
- Le Monde does not offer a public API --as the New York Times ;)
- Personal credentials (suscriber) are required, because comments are suscribers-only
- Built using httpx for requests & selectolax for parsing
- API & use examples with caching are available in lmd_ukr/examples; added some documentation in-code (rate limits etc.)
2. Dataset prep
- Checkout lmd_ukr/build_sqlite_dataset.py and build_parquet_dataset.ipynb
- Parsed data populated into an sqlite db with two tables articles and comments with shared key article_id
- This was optional, but wanted to refresh my skills and it allows to remove duplicates when building our db
- Formating / cleaning using Polars, wanted to benchmark v. Pandas (cf. notebook)
- Final file is a joined articles-comments (tidy) parquet file.
Data Usage
I created and shared this dataset for educational purposes only. Just wanted to have a French dataset, --if possible from my favorite newspaper on a topic I'm following daily; instead of exploring the same boring english datasets we're used to. It could be used for various natural language processing tasks.
- Topic modeling
- Troll detection (not enough fields though in my opinion)
- Generate summaries or headlines for articles (and compare to "desc" for instance)
- trending & various generative tasks of your choice
- (...)