sdam-au / GI_ETL

Repository for extraction, transformation and cleaning of PHI dataset

Scripts extraction, transformation, preprocessing and metadata enrichement of ancient Greek inscriptions

• ETL

---

Purpose

The purpose of this repository is to extract the PHI dataset from numerous .csv files into one tabular object in Python (pandas dataframe), clean it, enrich it by additionally metadata from Trismegistos etc. and to generate publishable datasets.

---

Authors

• Vojtěch Kaše , SDAM project, [e-mail]
• Petra Heřmánková , SDAM project, [e-mail]
• Adéla Sobotková , SDAM project, [e-mail]

License

CC-BY-SA 4.0, see attached License

DOI

[Here will be DOI or some other identifier once we have it]

References

[Here will go related articles or other sources we will publish/create]

---

How to use this repository

Sources and prerequisites

[Describe the provenance of data used in the scripts contained and clarify how it is harvested and what other prerequisites are required to get the scripts working. In case of pure tool attribute any reused scripts to source, etc., license and specify any prerequisites or technical requirements.]

Data

Anything else on data metadata and data used. Link to data repository or explanatory article.

Data storage:

SDAM_root/SDAM_data/PHI folder on sciencedata.dk

• 1_0_py_PHI_EXTRACTION_FROM_ZIP_RAW.ipynb)

  • input: PHI-raw-csv-2020-06-19.zip
  • output: PHI_merged_[timestamp].json

• 1_1_r_PHI_SCRAPING_MISSING_DATA.Rmd

  • input: PHI_merged_[timestamp].json
  • output: PHI_enriched_[timestamp].json

• 1_2_r_PHI_CLEANING_TEXT.Rmd.

  • input1: PHI_enriched_[timestamp].json
  • output: PHI_cleaned_[timestamp].json (latest verified version: 2020-06-29)

• 1_3_EXTRACTING_DATES.ipynb

  • input: PHI_cleaned_[timestamp].json
  • output1: PHI_cleaned_dated_[timestamp.json
  • output2: overview data samples in gsheet PHI_overview

• 1_4_MODELLING_DATES.ipynb

  • input: PHI_cleaned_dated_[timestamp.json
  • output1: PHI_cleaned_dated_[timestamp.json with new column: random_dates
  • output2: PHI_sim_time_blocks_[timestamp].json

Software

1. Google Colab or Jupyter Notebooks
2. R, version 4.0.1

Registered account

1. Google Colab

Hardware

1. Multiple-screen
2. Mouse
3. A lot of Coffee

---

Installation

[Describe the steps necessary to install the tool/package; example: https://gist.github.com/PurpleBooth/109311bb0361f32d87a2]

---

Instructions

[Describe first steps, how to use the current repository by a typical user - the digital historian with limited technical skills]

1. First, do ...
2. Second, do ...
3. Third, go to ...

Screenshots