OhadRubin / Awesome-System-Design

This is a project submission for the course Advanced System Design in Tel Aviv University.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Build Status codecov

Awesome System Design

Installation

  1. Clone the repository and enter it:

    $ git clone git@github.com:OhadRubin/awesome-system-design.git
    ...
    $ cd awesome-system-design/
  2. Run the installation script and activate the virtual environment:

    $ ./scripts/install.sh
    ...
    $ source .env/bin/activate
    [awesome-system-design] $ # you're good to go!
  3. To check that everything is working as expected, run the tests:

    $ pytest tests/
    ...

Usage

  • In order to run this project, make sure you have docker-compose installed and run:

    $ ./scripts/run-pipeline.sh
    

Client module

Usage

  • To use the client via the command line run
        $ python -m asd.client upload-sample -h 127.0.0.1 --port 8000 ../sample.mind.gz
  • To use it via python:
        >>> from asd.client import upload_sample
        >>> upload_sample(host='127.0.0.1', port=8000, path='../sample.mind.gz')
        

Notes

  • Our protocol is implemented in the client as a get request to let the server know which fields the client has, where the server returns the available parsers,
  • This is followed by a post request by the client to send the Snapshot.
  • We added to the protobuf file another class called Packet, which we use the combine the user and the snapshot and send it using our protocol.

Server module

Usage

  • To use via python

        >>> from asd.server import run_server
        >>> def print_message(message):
        ...     print(message)
        >>> run_server(host='127.0.0.1', port=8000, publish=print_message)
        # listen on host:port and pass received messages to publish
  • To use via cli

    $ python -m asd.server run-server \
        -h/--host '127.0.0.1'          \
        -p/--port 8000                 \
        'rabbitmq://127.0.0.1:5672/'

Notes

  • We used flask_restful to implement the RESTful interface.

Parsers module

Usage

  • To use with python:

    >>> from asd.parsers import run_parser
    >>> data = ...
    >>> result = run_parser('pose', data)
  • To use with the cli:

    $ python -m cortex.parsers parse 'pose' 'snapshot.raw' > 'pose.result'
    • This interace accepts a parser name and a path to some raw data, as consumed from the message queue, and prints the result, as published to the message queue (optionally redirecting it to a file.
  • Note that we only accept .raw files,

How to add a new parser

The parser file must contain either of the following:

  1. A class with a method parse with the signature (self, context, snapshot)
  2. A method that begins with parse_, for example parse_feelings and has the signature (context, snapshot).
    • For example: feelings.py Either way, it should return a dict, the keys for the dict will be the values that will be saved via the saver. Also, add in upload_sample in client the in line 36 the field for the parser

Saver module

Usage

  • To use with python:
    >>> from asd.saver import Saver
    >>> saver = Saver(database_url)
    >>> data =>>> saver.save('pose', data)
  • To save a result with the cli:
    python -m asd.saver save                     \
        -d/--database 'sqlite:///./data/asd.sqlite' \
        'pose'                                       \
        'pose.result' 
    
  • To run the saver using the cli:
    python -m asd.saver run-saver  "sqlite:///./data/asd.sqlite"  'rabbitmq://127.0.0.1:5672'

Notes

  • We choose to use sqlite as the backend because of the nice integration with sqlalchamy.

API module

Usage

  • To use with python:
    >>> from asd.api import run_api_server
    >>> run_api_server(
    ...     host = '127.0.0.1',
    ...     port = 5000,
    ...     database_url = 'sqlite:///./data/asd.sqlite',
    ... )
    … # listen on host:port and serve data from database_url
  • To use with the cli:
    $ python -m asd.api run-server \
        -h/--host '127.0.0.1'       \
        -p/--port 5000              \
        -d/--database 'sqlite:///./data/asd.sqlite'

Notes

  • We used flask_restful to implement the RESTful interface.

CLI module

Usage

$ python -m asd.cli get-users
…
$ python -m asd.cli get-user 1
…
$ python -m asd.cli get-snapshots 1
…
$ python -m asd.cli get-snapshot 1 2
…
$ python -m asd.cli get-result 1 2 'pose'

GUI module

Usage

  • Using python:
    >>> from asd.gui import run_server
    >>> run_server(
    ...     host = '127.0.0.1',
    ...     port = 8080,
    ...     api_host = '127.0.0.1',
    ...     api_port = 5000,
    ... )
  • Using the cli:
    $ python -m asd.gui run-server \
        -h/--host '127.0.0.1'       \
        -p/--port 8080              \
        -H/--api-host '127.0.0.1'   \
        -P/--api-port 5000

Notes

  • We used flask
  • For the gui, we implemented a infinite scrolling view.

About

This is a project submission for the course Advanced System Design in Tel Aviv University.


Languages

Language:Jupyter Notebook 92.5%Language:Python 5.6%Language:Shell 1.2%Language:HTML 0.7%Language:Dockerfile 0.0%