Ideas: visualize HPI data in a dashboard

[karlicoss]

For quantified self stuff some heavy manipulation with pandas is usually necessary (e.g. I'm doing this in my dashboard), so I doubt a general purpose web dashboard would cover it (unless it support JS snippets, which would be very helpful!). But it would be cool to have a quick way of overviewing/visualizing/aggregating data in the browser even if it's not perfect.

Quick googling results in:

• metabase: https://github.com/metabase/metabase#metabase
• redash: https://github.com/getredash/redash#readme

Both of them rely on some database (e.g. sqlite). While it's a bit inconvenient, probably good enough as the first order approximation. Since I'm extensively using NamedTuples/dataclasses, it's possible to adapt the data automatically without any boilerplate. In addition, cachew already dumps sqlite databases, which can be used as input data.

It would be also cool to have a native app (less hassle + better performance), but I'm not sure how to even start googling for that.

Related: KrauseFx/FxLifeSheet#34 (I think we have similar goals!)

[karlicoss]

https://gridstudio.io/ looks promising

[karlicoss]

Started integrating with influxdb & Grafana, works pretty well for time series-like stuff

HPI/my/bluemaestro.py

Lines 194 to 220 in 6d9bc29

 def fill_influxdb() -> None: 
     from itertools import islice 
     from .core.common import asdict 
  
     from influxdb import InfluxDBClient # type: ignore 
     client = InfluxDBClient() 
     db = 'db' 
     mname = __name__.replace('.', '_') 
     client.delete_series(database=db,  measurement=mname) 
     def dissoc(d, k): 
         del d[k] 
         return d # meh 
     jsons = ({ 
         'measurement': mname, 
         # todo maybe good idea to tags with database file/name? to inspect inconsistencies etc.. 
         # 'tags': {'activity': e.activity}, 
         'time': e.dt.isoformat(), 
         'fields': dissoc(asdict(e), 'dt'), 
     } for e in measurements()) 
     from more_itertools import chunked 
     # "The optimal batch size is 5000 lines of line protocol." 
     # some chunking is def necessary, otherwise it fails 
     for chunk in chunked(jsons, n=5000): 
         cl = list(chunk) 
         logger.debug('writing next chunk %s', cl[-1]) 
         client.write_points(cl, database=db) 
     # todo "Specify timestamp precision when writing to InfluxDB."? 

https://twitter.com/karlicoss/status/1361100437332590593

HPI/my/rescuetime.py

Lines 85 to 109 in 6239879

 def fill_influxdb() -> None: 
     from .core.common import asdict 
  
     from influxdb import InfluxDBClient # type: ignore 
     client = InfluxDBClient() 
     db = 'db' 
     measurement = __name__.replace('.', '_') 
     client.delete_series(database=db,  measurement=measurement) 
     # client.drop_database(db) 
     # todo create if not exists? 
     # client.create_database(db) 
     # todo handle errors.. not sure how? maybe add tag for 'error' and fill with emtpy data? 
     vit = (e for e in entries() if isinstance(e, Entry)) 
     jsons = ({ 
         'measurement': measurement, # hmm, influx doesn't like dots? 
         # hmm, so tags are autoindexed and might be faster? 
         # not sure what's the big difference though 
         # "fields are data and tags are metadata" 
         'tags': {'activity': e.activity}, 
         'time': e.dt.isoformat(), 
         'fields': {'duration_s': e.duration_s}, 
         # todo asdict(e), 
     } for e in vit) 
     # todo do we need to batch? 
     client.write_points(jsons, database=db) 

https://twitter.com/karlicoss/status/1360369025122000898