XCS Translator

XCS Translator is a web app to convert EXCEL file to JSON and SQL so that you can quickly and easily import data from your sheets to your database.

First you need to define your excel template in sheetconfig.json file. All uploaded data will be validated against the template definition. You can also add format constraints for each column using validate.js syntax.

The uploaded files will be available in storage directory with following structure:

storage
├── data1
│   ├── archive
│   │   ├── excel
│   │   └── json
│   ├── excel
│   ├── failed
│   │   ├── excel
│   │   └── json
│   └── json
└── data2
    ├── archive
    │   ├── excel
    │   └── json
    ├── excel
    ├── failed
    │   ├── excel
    │   └── json
    └── json

While being processed, your raw excel file will be stored in data1/excel or data2/excel directory. The json conversion result will be stored in json directory. After successfully inserted into database, your raw excel file will be moved to archive directory. If some exceptions occurred due to validation failure, the excel file will be moved to failed directory. Failed files should be fixed manually and re-uploaded to pass the validation.

Requirements

You need Node.js v10.22 or greater to be available in your system.

Configuration

Environment variables

The file of .env.example becomes the example how the environment variable should be set in the server.

Sheet configuration file

The columns in excel files being uploaded to XCS Translator will be mapped to database fields. You can configure what column mapped to what field in sheetconfig.json file. Please take a look on the sample provided in this repository and edit it to match your excel template specifications.

The structure of sheet configuration file:

[
  {
    "type": "data1",
    "source": {
      "headerRow": 1,
      "startingDataRow": 2,
      "columns": [
        {
          "col": "A",
          "title": "NAME",
          "constraints": {}
        },
        {
          "col": "B",
          "title": "DIAGNOSE"
        }
        // ...
      ]
    },
    "database": {
      "host": "db_host",
      "port": 3306,
      "user": "db_user",
      "password": "db_password",
      "dbName": "db_name",
      "connectionLimit": 5
    },
    "destinations": [
      {
        "kind": "patient",
        "table": { "name": "db_patient" },
        "columns": [
          { "col": "A", "name": "name" },
          { "col": "B", "name": "diagnose" }
          // ...
        ]
      },
      {
        "kind": "specimen",
        "table": {
          "name": "db_specimen",
          "foreignKey": {
            "sourceIndex": 0,
            "field": "patient_id"
          }
        }
        // ...
      }
    ]
  },
  {
    "type": "data2"
    // ...
  }
]

This app can handle multiple templates which then mapped to different databases. One entry in root array corresponds to one template.

Defining source excel file template

headerRow: (required) determines in what row is the header in excel template
startingDataRow: (required) determines in what row the data starts from
columns: (required) defines the columns of the excel template
- col: (required) column identifier (A, B, C, ...)
- title: (required) title of the column
- constraints: (optional) constraints of the value for the said column

The constraints is using validate.js syntax and you can use all standard validators available.

Common example of adding constraints:

The column can not be empty

{
  "col": "A",
  "title": "NAME",
  "constraints": {
    "presence": true
  }
}

The value should match a regex pattern

{
  "col": "C",
  "title": "GENDER",
  "constraints": {
    "format": {
      "pattern": "[MF]"
    }
  }
}

The value should exists and not shorter than a required limit

{
  "col": "A",
  "title": "NAME",
  "constraints": {
    "presence": true,
    "length": { "minimum": 3 }
  }
}

Database configuration

Provide these value in database configuration section for each data type:

host
port
user
password
dbName
connectionLimit

Mapping from excel to target database table fields

Data from one excel file can be inserted to multiple tables in one database, and you can pick what columns to be mapped to each table.

kind: (required) is the identifier for the table (the table name will be fetched from environment variable)
table: (required)
- name: (required) is database table name
- foreignKey: (optional only if this table contains foreign key from previous table in the list)
  - sourceIndex: (required) is index of source table in the list
  - field: (required) is the FK field name in this table
columns: (required)
- col: (required) the source column in excel template (A, B, C, ...)
- name: (required) the target field in database table

Run in development mode

Add write permission to storage directory (in MacOS or Linux run chmod a+w storage)
Run cp .env.example .env
Edit .env placeholder value
Edit sheetconfig.json file
Run npm run ci
Run npm run dev
Server is running in configured port (default to 8080)

Run in production mode

Add write permission to storage directory (in MacOS or Linux run chmod a+w storage)
Run cp .env.example .env
Edit .env placeholder value
Edit sheetconfig.json file
Run npm run ci
Run npm start
Server is running in configured port (default to 8080)

Clear archive directory

Run npm run clear-archive

hyperjumptech / xcs-translator