massiltag / CSV-Indexer

CSV indexing engine, made in Java, JAX-RS - RESTEASY.

Indexer

Indexer project as part of the LU3IN107 - WEB Technology teaching unit.
Supervised by Olivier PITTON
Sorbonne University

Contributors

• Francis MURRAY
• Massil TAGUEMOUT
• Baptiste PIRES

Project description:

• The goal of this project is to conceive and develop an Indexing Engine facilitating fast and accurate information retrieval from a database.

Getting Started

• To run the project, execute mvn jetty:run-war

Documentation

• Swagger API Documentation: http://localhost:8080/
• OpenAPI Specification: http://localhost:8080/api/openapi.json
• Project Information and Reports: Execute mvn site:run in a separate terminal. The reports will be generated and visible at http://localhost:9000/

Operations:

• Create Table

  • API Call (POST): http://localhost:8080/api/indexer/createTable
  • Request Body example:

    {
      "name": "TableName",
      "columns": [
        {
          "name": "VendorID",
          "type": "Integer"
        },
        {
          "name": "tpep_pickup_datetime",
          "type": "String"
        },
        {
          "name": "tpep_dropoff_datetime",
          "type": "String"
        },
        {
          "name": "passenger_count",
          "type": "Integer"
        },
        {
          "name": "trip_distance",
          "type": "Double"
        },
        {
          "name": "RatecodeID",
          "type": "Integer"
        },
        {
          "name": "store_and_fwd_flag",
          "type": "String"
        },
        {
          "name": "PULocationID",
          "type": "Integer"
        },
        {
          "name": "DOLocationID",
          "type": "Integer"
        },
        {
          "name": "payment_type",
          "type": "Integer"
        },
        {
          "name": "fare_amount",
          "type": "Double"
        },
        {
          "name": "extra",
          "type": "Double"
        },
        {
          "name": "mta_tax",
          "type": "Double"
        },
        {
          "name": "tip_amount",
          "type": "Double"
        },
        {
          "name": "tolls_amount",
          "type": "Double"
        },
        {
          "name": "improvement_surcharge",
          "type": "Double"
        },
        {
          "name": "total_amount",
          "type": "Double"
        },
        {
          "name": "congestion_surcharge",
          "type": "Integer"
        }
      ]
    }

• Add Indexes

  • API Call (POST): http://localhost:8080/api/indexer/addIndexes
  • Request Body example:

    {
      "tableName": "TableName",
      "indexes": [
          "VendorID", 
          "passenger_count"
    
        ]
    }

• Upload data

  • API Call (POST): http://localhost:8080/api/indexer/uploadData
  • Request Body example:
    • form-data
    • file = yellow_tripdata_2019-01.csv (csv files can be downloaded here: https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page)

• Start Indexing

  • API Call (POST): http://localhost:8080/api/indexer/startIndexing
  • Request Body: none

• Query

  • API Call (POST): http://localhost:8080/api/indexer/query
  • Request Body example:

    {
      "type": "SELECT",
      "cols": ["VendorID", "passenger_count"],
      "from": "TableName",
      "operator": "AND",
      "where": {
        "VendorID": {
          "operator": "=",
          "value": 2
        },
        "fare_amount": {
          "operator": "=",
          "value": 5
        }
      },
      "limit": 5000
    }