ConvNetSharp compare to other Deep Learning frameworks

[DavidGriscti]

Hi,
It's great work you made.
How does ConvNetSharp compare to other Deep Learning frameworks like Keras in the accuracy of the prediction of DNN regression?
Have you done any tests?
Thanks.

[Cédric Bovar]

Hi,

I haven't done any comparison with other deep learning frameworks.
That would be interesting to do (especially to detect possible errors in ConvNetSharp) but I currently haven't got much time to do that.
Of course I expect mainstream frameworks to be more correct/accurate/performant.

[DavidGriscti]

Hi.
When I use a small BatchSize ,For example, 100,200... the ConvNetSharp neural network will not converge.
But when I use the frameworks like Keras, and use the same small BatchSize(even 1) and network parameters, the Keras neural network will converge well.
I can't find the reason...

[DavidGriscti]

        double[][] traindata = ReadFromCsvFile(@"C:\LD\train111.csv", true);
        double[][] trainout = ReadFromCsvFile(@"C:\LD\out111.csv", true);

        var featurenum = traindata[0].Length;
        var outputnum = trainout[0].Length;

        Net<double> net = new ConvNetSharp.Core.Net<double>();

        net.AddLayer(new InputLayer(1, 1, featurenum));

        net.AddLayer(new FullyConnLayer(128));
        net.AddLayer(new ReluLayer());
        //net.AddLayer(new DropoutLayer(0.1));

        net.AddLayer(new FullyConnLayer(128));
        net.AddLayer(new ReluLayer());
        // net.AddLayer(new DropoutLayer(0.1));

        net.AddLayer(new FullyConnLayer(outputnum));
        net.AddLayer(new RegressionLayer());


        // Training
        int maxEpochs = 150;
        int nBatchSize = 100;//traindata.Length;

        //train input
        var featuresPerRow = traindata[0].Length;

        var flattenedInput = new double[traindata.Length * featuresPerRow];
        for (int i = 0; i < traindata.Length; i++)
        {
            var row = traindata[i];
            for (int j = 0; j < row.Length; j++)
            {
                flattenedInput[(i * featuresPerRow) + j] = row[j];
            }
        }
        var volumetrain = BuilderInstance.Volume.From(flattenedInput, new Shape(1, 1, featuresPerRow, traindata.Length));


        //train output
        var featuresPerRowOut = trainout[0].Length;

        var flattenedOut = new double[trainout.Length * featuresPerRowOut];
        for (int i = 0; i < trainout.Length; i++)
        {
            var row = trainout[i];
            for (int j = 0; j < row.Length; j++)
            {
                flattenedOut[(i * featuresPerRowOut) + j] = row[j];
            }
        }
        var volumeout = BuilderInstance.Volume.From(flattenedOut, new Shape(1, 1, featuresPerRowOut, trainout.Length));


        var trainer = new SgdTrainer(net) { LearningRate = 0.01, Momentum = 0.9, BatchSize = nBatchSize , L2Decay = 0.001 };

        for (int i = 0; i < maxEpochs; i++)
        {
            trainer.Train(volumetrain, volumeout);
            Console.WriteLine("Train Loss:" + trainer.Loss);

        }

        Console.ReadKey();

[Cédric Bovar]

Hi,

That's interesting. Can you show your keras code as well?
Could you try to set L2Decay to 0 in Keras and in ConvNetSharp? It's not currently used in ConvNetSharp.

[DavidGriscti]

Hi,
There is keras code ,it is using R:
set.seed(1234)
library(keras)

root_mean_squared_error= function(y_true, y_pred) {
k_sqrt(k_mean(k_square(y_pred - y_true),keepdims = T))
}

metric_rmse <- custom_metric("rmse", function(y_true, y_pred) {
1/(1+k_sqrt(k_mean(k_square(y_pred - y_true), keepdims = )))*100
})

trainf <- read.csv('c:/LD/train111.csv', header = T)
yf <- read.csv('c:/LD/out111.csv', header = T)

train.y=yf$LS
train.x=data.matrix((trainf))

gc()
filepath="c:/LD/best.hdf5"
checkpoint = callback_model_checkpoint(filepath, monitor='val_loss', verbose=1, save_best_only=T, mode='min')

earlyStopping=callback_early_stopping(monitor="val_loss", patience=30, verbose=1, mode='min')
callbacks_list = list(checkpoint,earlyStopping)

model <- keras_model_sequential()

r<-ncol(train.x)

model %>%
layer_dense(units =128, activation = 'relu', input_shape = c(r)) %>%
layer_dropout(rate = 0.0) %>%
layer_dense(units = 128,activation = 'relu') %>%
layer_dropout(rate = 0.0) %>%
layer_dense(units = 1)

model %>% compile(
loss = root_mean_squared_error,
optimizer = optimizer_sgd(lr = 0.01, momentum = 0.9, decay = 0.01),
metrics = metric_rmse
)

#summary(model)

history <- model %>% fit(
train.x, train.y, callbacks=callbacks_list,
epochs = 150, batch_size = 8
)

[DavidGriscti]

[Cédric Bovar]

I noticed something in your C# code: the 4th dimension of your input/output data should be the batch size. But I see it's the whole dataset size. You need to adapt the code to call the train method on each batch (if batch size = 1, once for each entry in the dataset).

I should emove the batch size parameter of the trainer because it's confusing: it should deduce it from the shape of the input.

[DavidGriscti]

Thank you for pointing out the mistake.
Could you write an example code or pseudocode for using the batch and train?

[Cédric Bovar]

You can get some inspiration from this line in the mnist example.

I'm travelling and I don't have access to a computer, typing code with a mobile is not ideal