Giters

hoangnguyen177 / IOLibrary

IOInterface

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Add database option

hoangnguyen177 opened this issue · comments

commented

Add an option to store messages in database. Check performance.

commented

All the server needs to store is messages being transferred between clients (sources and sinks). Considering between RDMS Postgres | MongoDB | Redis.

http://slashdot.org/topic/bi/sql-vs-nosql-which-is-better/
http://redis.io/

From preliminary research, looks like non-SQL is better since since they store data as native JS objects or JSON.
Options:
MongoDB
Redis
Couch
Riak

commented

MongoDB seems to be a better option to deal with large datasets. MongoDB it is then.

commented

to enable database recording:

node app.js 9090 / true

commented

A better way needed to organise between runs. i.e so users can view results from previous runs as well. Not just the current one.

commented

From the portal, only one instance of a workflow can be executed at one time. But the portal does not keep track of the runs. I need to store information about runs if want to access the history.

From the portal, it needs to record:

  • run id
  • workflow id
  • start run
  • end run
  • maybe provenance database

In the server, it needs to record:

  • actor definition - so that the history portlet can get the appropriate objects to display data
  • run id, and time
  • workflow id
  • messages exchanged between IOActors, IOPortlets. The history porlet will just display messages from IOActors.

In the portlet, it needs to record:

  • history of the messages

Now the question is whether the database should be centralised ?

commented

Note that enabling viewing history of the current run is easy. But viewing history of previous runs are harder. Keep in mind for the future extension.

commented

Possible Solution:

  • in each workflow, three parameters are required: "username", "password" and "runid". username and password are string parameters
  • when the workflow is executed, these two parameters are passed to workflow
  • IOActor will look for these two parametrs in the top level container, if not --> throw exception
  • add these two parameters in the definition

Changes needed:

  • WorkflowsList portlet
  • Slavisa kepler client
  • IOActor
  • app.js to include run id, etc.