czarcrab / DataManager

Data Manager

A tool to view normalization state of a database and help modify tables / columns to attain a desired Normal Form.

Developers can:

• Run this on your DB to find any inconsistencies (detect mode)
• Parse flattened data through it's API (REST) and get normalized arrays returned (act mode) Administrators can:
• Run this to find compliance with 3rd normalization requirements
• Use the tool to clone a DB into a new normalized or intentional de-normalized version of itself
• Store configuration from such a conversion job
• Use the tool as ETL to routinely transfer data (Incremental is not handled in current release.)

Technical Objectives

• Full JavaScript Stack
• TDD with agile approach
• Rules based approach
• Convention over configuration with over-ride
• Restful API
• Command line interface

Detailed Scope (Product Backlog)

Use Cases

1. Read DB and get all structure (meta-data) information including
   1. Table and column names and types
   2. Foreign Key relationships
   3. Primary Key relationships
   4. Unique constraints (indexes)
2. Show tables with sample data in grid
3. Show relationships with in grid
4. Show relationships in an ER diagram
5. Allow user to modify relationships
6. Allow user to modify table names
7. Allow user to modify column names
8. Tell user the current Normal Form of database from the following options:
   1. 1 NF
   2. 2 NF
   3. 3 NF
   4. BCNF (3.5 NF)
   5. 4 NF, 5 NF, DKNF and 6 NF are left out for a future release
9. Allow user to clone the DB to a new form, the user can also goto a lower Normal Form
10. Allow user to save transformation configuration
11. Allow user to save connection configuration
12. Allow user to pass in configuration on CLI call

Assumptions (Convention)

1. id (int 11), primary_id, unid (char 32 or char 36) named columns are considered PK for the table
2. For non transactional databases, FK columns are structured with double underscore like subjects__id
3. Underscore is the naming convention followed in database for eg. first_name
4. All columns with double underscore are considered dependent on a foreign table
5. Columns starting with underscore are considered for caching purposes and are not processed by the system. For eg. a subjects__id might be followed by a _subjects__name
6. Foreign key prefix is exactly the same spelling as it's associated table name without any singularization or pluralization

Out of scope

1. does not handle composite primary keys
2. incremental data loading in ETL mode

Releases (Sprint Backlog)

1. Express.js with CLI in detect mode
2. Express.js with CLI in act mode
3. Rest API
4. Basic responsive UI using Backbone.js
5. Drag-drop on grids and advanced UI features
6. ER diagrams (view mode)
7. Drag-drop on ER diagrams