API for Generating Mouse Movements with Neural Networks
Send POST requests to automatically move your mouse with a neural network!
(Note: The api now only generates coordinates and no longer moves the mouse for you.)
Table of Contents
pymousegan
This is the Python library containing the code for creating neural networks.
The training is done in the Colaboratory notebook. pymousegan
contains the models and training pipeline for the GAN.
Example notebooks are located at python/notebooks
Getting Started
git clone https://github.com/jchen42703/ai_mouse_movements.git
cd python
pip install .
Dependencies
numpy
tensorflow
pandas
matplotlib
Training Pipeline
Preprocessing
- Translated so that the starting coordinate is
(0, 0)
. - Scaled so that the destination coordinates is
(1, 1)
. - Reflection across all axes done during training.
GAN
The model used in the current version is a BidirectionalLSTMDecoderGenerator
from an AdditiveBasicGAN
with a BidirectionalLSTMDiscriminator
(with minibatch discrimination) and BidirectionalLSTMDecoderGenerator
. The full example is located at https://github.com/jchen42703/ai_mouse_movements/python/README.md.
Here are the model summaries:
[JS API]
Getting Started [Client]
cd js
npm install .
nodemon index.js
- Install dependencies with
npm install
nodemon index.js
ornode index.js
to run the server onPORT=3000
.- Send a
POST
request (json
) tohttp://localhost:3000/
, such as:
{
"start": [1, 1],
"destination": [82 ,55]
}
Model Format Conversion
tf.keras
to .json
From pip install tensorflowjs
tensorflowjs_converter --input_format=keras model/weights.h5 model/tfjs_model
Dependencies
@tensorflow/tfjs
@tensorflow/tfjs-node
express
nodemon
for convenience
How does it work?
POST
request tohttps://localhost:3000/
express
handles thePOST
request and calls the prediction functionloadAndPredict
.- The function returns a promise and when it resolves, the output is a list of coords and lags:
[x, y, lag]
- The
lag
is the time inms
that the mouse stays at that coordinate
- The
{
"coords": [
[
1,
1,
24.451885223388672
],
[
1.789207100868225,
1.6034066677093506,
23.39274024963379
],
[
2.462282180786133,
2.276571035385132,
24.84036636352539
],
[
2.7074904441833496,
2.716768264770508,
26.283510208129883
],
[
2.862687110900879,
3.18359637260437,
27.842201232910156
],
...
]
}
How fast is the api?
On average, it runs from 390ms to 430ms
- For cold starts: 500ms - 600ms