mini-kep / parser-rosstat-kep

Provide clean macroeconomic time series as CSV files at stable URL

Parser	KEP
Data source	Short-term Economic Indicators (KEP) by Rosstat as MS Word files
Parsing result	Annual, quarterly and monthly time series in CSV files
Releases	See 2018 schedule

Concept

In this repo I publish a dataset of Russian macroeconomic time series as machine-readable CSV files. I keep track of monthly macroeconomic data releases (vintages) since April 2009. Original files by Rosstat are in MS Word format.

Interface

manage.py does the following job:

• download and unpack MS Word files from Rosstat
• convert MS Word to interim csv files
• parse interim csv files to get pandas dataframes with time series (at annual, quarterly and monthly frequency)
• save dataframes as CSV files at stable URL
• create Excel file, if needed

Access to parsing result

Stable URL: https://github.com/mini-kep/parser-rosstat-kep/tree/master/data/processed/latest

import pandas as pd

def get_dataframe_from_web(freq):
    filename = f'df{freq}.csv'
    url = ('https://raw.githubusercontent.com/mini-kep/parser-rosstat-kep/'
          f'master/data/processed/latest/{filename}')
    return pd.read_csv(url, converters={0: pd.to_datetime}, index_col=0)

dfa, dfq, dfm = (get_dataframe_from_web(freq) for freq in 'aqm')

Repo management

Around this schedule on a Windows machine I run:

invoke add <year> <month>

and commit changes to this repo.

This command:

• downloads a rar file from Rosstat,
• unpacks MS Word files,
• dumps all tables from MS Word files to an interim CSV file,
• parses interim CSV file to three dataframes by frequency
• validates parsing result
• transforms some variables (eg. deaccumulates government expenditures)
• saves dataframes as processed CSV files
• saves csv for latest date
• saves an Excel file for latest date.

Same job can be done by manage.py

Parcer summary

Parcer	mini-kep
Job	Parse sections of Short-term Economic Indicators (KEP) monthly Rosstat publication
Source URL	Rosstat KEP page
Source type	MS Word
Frequency	Monthly
When released	Start of month as in schedule
Code	https://github.com/epogrebnyak/mini-kep/tree/master/src/
Test health
Test coverage
Documentation
CSV endpoint	https://github.com/epogrebnyak/mini-kep/tree/master/data/processed/latest
Transformation	Government revenue/expenses deaccumaulated to monthly values
Validation	Hardcoded checkpoints and consistency checks

All historic raw data available on internet?

• Yes
• No (data prior to 2016-12 is in this repo only)

Notes

• We follow cookiecutter-data-science template for directory structure.

• Windows and MS Word are required to create interim text dumps from MS Word files. Оnce these text files are created, they can be parsed on a linux machine.

• This repo replaces a predecessor, data-rosstat-kep, which could not handle vintages of macroeconomic data.