Data size and decoding time

[gaetanboulard]

Should decoding time really be proportional to the size of the training data?

Context:
I have trained 2 models using the same parameters.
The 2nd model trained on twice as much data, which resulted in bigger char and work mappings.

While decoding my test dataset, I observed that the 2nd model took twice as long.

It is my understanding that once the weights are set in training, the size of our vocabulary should have almost no impact on decoding.