Net2Net
Net2Net implementation on PyTorch for any possible vision layers (nn.Linear, nn.Conv2d, nn.Conv3D, even wider operator btw nn.ConvX to nn.Linear). Checkout the paper for more detail
Observations:
-
Using BatchNorm between layers, improves the competence of Net2Net. Otherwise, Net2Net approach is not able to get comparable results to a network trained from scratch.
-
Inducing noise to new units and connections prelude to better networks. The effect is more evident without BathNorm layer.
-
Normalizing layer weights before any Net2Net operation increases the speed of learning and gives better convergence. Even so, it worths to investgate better normalization methods than L2 norm.