DSS: Deep Sensor Systems
From training and deployment of ViTs to development of real-time cross-platform mobile apps
DSS was developed by SINTEF Medical Image Analysis with aim to integrate AIs into sensor systems.
This project serves as a demonstration on how to do it, and does not claim to be a generic framework.
Below there are described some of the key features of this project, but to see what else is possible, please, see the wiki.
Continuous integration
| Build Type | Status |
|---|---|
| Test Training |  |
| Test Flutter |  |
| Build APK |  |
How to your train own model?
Setup
When using this framework, it is a good idea to setup a virtual environment:
virtualenv -ppython3 venv --clear
source venv/bin/activate
pip install -r requirements.txt
The following dependencies will be installed:
pandas<=1.5.3tensorflow<=2.12.0tensorflow-addons<=0.19.0tensorflow-datasets<=4.8.3
Tested with Python 3.7.9 on Win10, macOS, and Ubuntu Linux operating systems. Also tested with Python 3.10.4 on GitHub Codespaces.
Note that to activate the virtual environment on Windows instead run ./venv/Scripts/activate.
Usage
To train a model, simply run:
python main.py
The script supports multiple arguments. To see supported arguments, run python main.py -h.
Training history
To visualize training history, use TensorBoard (with example):
tensorboard --logdir ./output/logs/gesture_classifier_arch_rnn/
Example of training history for a Recurrent Neural Network (RNN) can be seen underneath:
The figure shows macro-averaged F1-score for each step during training, with black curve for training and blue curve for validation sets. Best model reached a macro-averaged F1 score of 99.66 % on the validation set, across all 20 classes.
Disclaimer: This model was only trained for testing purposes. The input features were stratified on sample-level and not patient-level, and thus validation performance will likely not represent true performance on new data. However, having a trained model enables us to test it in a Mobile app.
Available datasets
SmartWatch Gestures
The current data used to train the AI model is the SmartWatch Gestures dataset, which is available in tensorflow-datasets. The dataset has the following structure:
FeaturesDict({
'attempt': tf.uint8,
'features': Sequence({
'accel_x': tf.float64,
'accel_y': tf.float64,
'accel_z': tf.float64,
'time_event': tf.uint64,
'time_millis': tf.uint64,
'time_nanos': tf.uint64,
}),
'gesture': ClassLabel(shape=(), dtype=tf.int64, num_classes=20),
'participant': tf.uint8,
})
How to test the model in a mobile app?
Converting model to TF-Lite
In order to be able to use the trained model in a mobile app, it is necessary to convert the model to a compatible format. TensorFlow Lite is an inference engine tailored for mobile devices. To convert the model to TF-Lite, simply run this command:
python dss/keras2tflite.py -m /path/to/pretrained/saved_model/ -o /path/to/save/converted/model.tflite
Model integration and testing in app
A simple Mobile app was developed in Flutter, which demonstrates the AI in action using the accelerometer data from the mobile phone in real time. The data can also be stored and deleted locally.
To use the app, you need an Android phone and have developer mode enabled (see here for how to enable it). Then simply download the APK from here, double-click to install, and use the app as you normally would.
Info on how the mobile app was developed (and how to make your own app), can be found in the wiki.
Acknowledgements
The training framework was mainly developed using Keras with TensorFlow backend.
The mobile app was developed using Flutter, which is a framework developed by Google. For the app, the following open packages were used (either MIT, BSD-2, or BSD-3 licensed):
How to cite?
If you found this project useful, please, consider citing it in your research article:
@software{andre_pedersen_2023_7701510,
author = {André Pedersen and Ute Spiske and Javier Pérez de Frutos},
title = {andreped/DSS: v0.2.0},
month = mar,
year = 2023,
publisher = {Zenodo},
version = {v0.2.0},
doi = {10.5281/zenodo.7701510},
url = {https://doi.org/10.5281/zenodo.7701510}
}