This is a fixed-point training simulation framework based on tensorflow.
There are two phases that we care about in deploying NN to hardware with only fixed-point calculation units:
- TSDH: Training on Software, Deploy a fixed-point model on Hardware.
- THDH: fixed-point Training on Hardware directly, Deployed on Hardware.
In this documentation, unless otherwise specified, "fixed-point training" are refering to both of these phases in this documentation.
We have four things to be quantitized potentially:
- weights
- gradients of weights
- activations
- gradients of activations
For transparent conversion to fixed point simulation, we supply a context manager for managing fixed point configuration: nics_scope.fixed_scope, and wrapper of nn operations. Eg. Dense for tf.layers.dense, Conv2d for tf.layers.conv2d.
Using these wrappers inside the context manager, the above four things in the models created will be handled transparently. You can also manually insert .apply(quantitize, fix_cfg, name=name) operation into necessary places.
TODO: Describe the format of the fixed configuration file.
In the above section, using the context manager with suitable quantization configs, you can simulate the fixed-point computation on the hardware. This is a simulation of hardware constraints.
However, to do better training, or to explore what training techniques can be applied to enable better fixed-point training, we supply a strategy interface, and some pre-defined strategies, see Strategies for the pre-defined strategies.
You can implement your own strategy by creating a class inheriting nics_fix.Strategy, and implement its methods:
pre_weightpost_weightpre_weight_gradpost_weight_gradpre_activationpost_activationpre_activation_gradpost_activation_grad
These methods all receive a tensor as input, and should return a tensor. By default, they just return their input tensor. The pre_* functions's output tensor will be fed into its corresponding quantitize operation, and the output tensor of the quantitize operation will be fed into the post_* functions.
When using strategy, call nics_scope.fixed_scope with a strategy config. You should specify where these strategies should be used in the strategy config.
You can nested multiple strategies, the first strategy's pre_* methods will be first executed among all the strategies, while its post_* methods will be last executed.
NoiseStrategy: Insert noise into points. See the class' documentation string for details.
TODO: Describe the format of the strategy configuration file.
- See
examples/config_fix_wag.yaml.samplefor a example of fixed-point configuration. - See
examples/config_strategy_noise.yaml.samplefor a example of strategy configuration, using the pre-definedNoiseStrategy.
- See
examples/mnist/mlp.pyfor a simple example of a shallow fixed-point MLP on Mnist. - See
examples/cifar10/cifar10_train.pyfor an example of training fixed-point CNN (VGG11) on Cifar10.
Try cd examples/mnist && python mlp.py --cfg ../config_fix_wag.yaml.sample.
Use the environment variable NICS_FIX_LOGLEVEL to control the log level in nics_fix package. Avaiable log levels are {"debug", "info", "warning", "error", "fatal"}. By default, the log level is set to "warning".
There are some helper utils that helps to write cleaner test scripts.
nics_fix.kwargs_scopeandnics_fix.kwargs_scope_by_type: A context manager that will try to supply common default keyword arguments to the registered methods called in the context. Seeexamples/cifar10/cifar10_train.pyfor an example of use.
Use nf.fixed_model_saver(fixed_mapping) to get a patched tf.train.Saver to save fixed model. Usually, the argument fixed_mapping should be the one yield by the nf.fixed_scope context manager.
If you want the saved model's weights to be already fixed, pass keyword argument fixed_weight=True to nf.fixed_model_saver.
See examples/mnist/mlp.py for an example of saving fixed model, and examples/mnist/mlp_eval.py for an example of loading a saved fixed model and run evaluation only.