Add database option
hoangnguyen177 opened this issue · comments
Add an option to store messages in database. Check performance.
All the server needs to store is messages being transferred between clients (sources and sinks). Considering between RDMS Postgres | MongoDB | Redis.
http://slashdot.org/topic/bi/sql-vs-nosql-which-is-better/
http://redis.io/
From preliminary research, looks like non-SQL is better since since they store data as native JS objects or JSON.
Options:
MongoDB
Redis
Couch
Riak
MongoDB seems to be a better option to deal with large datasets. MongoDB it is then.
to enable database recording:
node app.js 9090 / true
A better way needed to organise between runs. i.e so users can view results from previous runs as well. Not just the current one.
From the portal, only one instance of a workflow can be executed at one time. But the portal does not keep track of the runs. I need to store information about runs if want to access the history.
From the portal, it needs to record:
- run id
- workflow id
- start run
- end run
- maybe provenance database
In the server, it needs to record:
- actor definition - so that the history portlet can get the appropriate objects to display data
- run id, and time
- workflow id
- messages exchanged between IOActors, IOPortlets. The history porlet will just display messages from IOActors.
In the portlet, it needs to record:
- history of the messages
Now the question is whether the database should be centralised ?
Note that enabling viewing history of the current run is easy. But viewing history of previous runs are harder. Keep in mind for the future extension.
Possible Solution:
- in each workflow, three parameters are required: "username", "password" and "runid". username and password are string parameters
- when the workflow is executed, these two parameters are passed to workflow
- IOActor will look for these two parametrs in the top level container, if not --> throw exception
- add these two parameters in the definition
Changes needed:
- WorkflowsList portlet
- Slavisa kepler client
- IOActor
- app.js to include run id, etc.