Emoncms Development Overview (September 2014)

[Trystan Lea]

(Rough draft 9th September 2014 - to be improved)

Summary of recent development

Emoncms development in the first half of 2014 focused on improving the data core of emoncms, how timeseries data is stored. Documentation on how the timeseries data engines work and the more recent development of buffered writing available in the bufferedwrite branch can be found here: https://github.com/openenergymonitor/documentation/tree/master/BuildingBlocks/TimeSeries

The buffered writing reduced write load has led to the testing of writing to SD cards instead of harddrives again which reduces the complexity and cost of a raspberrypi emoncms basestations considerably, the current ready-to-go SD card image uses this bufferedwriting branch: https://github.com/emoncms/emoncms/tree/bufferedwrite

Emonhub: With the excellent work by Paul (https://github.com/pb66) on bringing the new version of EmonHub together (previously called oem_gateway by Jerome https://github.com/Jerome-github) emonhub has now become the standard way of connecting the rfm12pi adapter board to emoncms whether local or remote, dropping the need for the old raspberrypi emoncms module while providing new features such as the ability to send data to multiple remote and local servers, in-built buffering when the internet connection is down and flexible decoding of rfm12pi binary data into different datatypes.

In addition to core data development and emonhub there have been improvements to the emoncms input interface and changes to the visualisations such as the bargraph so that it can generate daily data from accumulating watt-hour data or daily averages (power, temperature, etc ) from phpfiwa engine feeds.

The last overview development update (March 2014) can be read here: http://openenergymonitor.org/emon/node/4008 & http://openenergymonitor.org/emon/node/3995

Going forward there are many ideas for further developments, I've listed the ones I been thinking about and working on below. While further work can be done to improve the data core of emoncms Im quite happy with the current implementation for now, it has reached a point of relative stability and looks like it will continue to allow emoncms.org to scale at current rates within the next 6-12 months. I intent to focus more of my time on the application side of emoncms, primarily the OpenBEM building energy modelling module which Im working on in collaboration with Carbon Coop. I will also be working with John Cantor on improving the support for heatpump monitoring and Richard Hatfield on the emoncms + scheduler + packetgen heating controller.

Data core:

Buffered writing: At present the implementation of buffered writing used in the bufferedwrite branch of emoncms that is used on SD cards is a relatively simple implementation and needs more work to be used in the full version (master branch) of emoncms used on emoncms.org. But it would be worth doing this work in order to get the improved performance both in terms of reduced disk wear and potentially faster access speeds if item 1 below was implemented. There are two things that would be good to improve on:

1. Ability to buffer in RAM over longer time spans while being able to access the data seamlessly across both RAM + Persistent disk. At the moment the data can only be accessed after it has been written to disk and so if the commit time is slow say 10-30 mins you will only be able to see an update to that data every 10-30mins.
2. Ability to write updates in addition to append only writes to disk in order to make it possible to edit historical data and produce daily data such as the kwh per day data generated by the power to kwhd input processor.

Other Features:

• Ability for any user to become an administrator of a set of sub-accounts that the user has authentication for. Useful for organisations providing monitoring services across several installations/sites.
• MyElectric style pre-built dashboard interface for heatpump monitoring, along with documentation on how to interpret the data coming from a heatpump monitor. Could automatically calculate theoretical COP from temperature data.
• Building thermal performance monitoring & modelling, OpenBEM + integration with monitored data.
• Control: continue work on scheduler, packetgen. Investigate potential for integration of event module into what is currently input processing and then ability to carry out control actions on a given condition.
• Publish last feed values to an MQTT topic, see: http://openenergymonitor.org/emon/node/5826
• Input data into emoncms via MQTT

For notes on modifying EmonHub to be used with emoncms+packetgen for control see: https://github.com/emoncms/emoncms/blob/bufferedwrite/docs/emonhubmod.md

Longer term:

• Open data / research: search through public feeds, curate high quality datasets with relevant meta data for different sustainable energy equipment, solar pv, heatpumps, building thermal performance data etc.

Bugs and issues

A brief summary of bugs and issues to fix listed in full in the full issue list. Please create a new github issue rather than posting a comment below with a bug/issue.

• feed/create does not check for enabled/available engines - needs better checking here.
• Better notification of if data is actually writting to disk or not. Last time updated updates even if there is a storage error at the moment #175
• Fix and update debian install packages #242
• Get datapoint editor tools working again - and on emoncms.org #206
• CSV import tool

[chaveiro]

First issue fix already on extended branch.