Compiling many languages is slow

[Ievgen Pyrogov]

Barring the time to download the language JSON files, the following code takes around 60 seconds to compile on my computer (2018 MBP Pro, 2,2 GHz 6-Core Intel Core i7, 16Gb RAM):

locales = ~w(
  ar hy sq az be bn bg my bs ca zh hr cs
  da nl en-GB et fo fi fr de el he hi hu
  is id it ja kk ko ku lv lt mk ms mt mr
  sr-Latn-ME nb pl pt pa ro ru sr sk sl so es
  sw sv fil ta th tr uk ur vi cy
)

Mix.install([
  {:jason, "~> 1.0"},
  {:ex_cldr, "== 2.37.2"},
])

defmodule Sample.Cldr do
  use Cldr,
    locales: locales,
    default_locale: "en-GB",
    providers: []
end

This was extracted from an actual application that's running in production. Compiling this currently the single slowest part of compiling the entire application (over 1000 modules).

Is there a way in principle the compilation speed could be improved here, or is compiling Sample.Cldr will always be inherently slow? I noticed the compilation time is smaller if the amount of languages is small. But I'm curious whether there's anything else left to do, independently of the number of languages.

[Kip Cole]

@gmile, unfortunately with the current design the compile time of the ex_cldr backend is inherently slow. There is a significant amount of compile time code generation from the CLDR locale data. The time is largely proportional to the number of locales configured as your have also seen. In fact the slowest part is actually code generation in the erlang compiler which I think is related to the number of function clauses on a number of functions. There isn't, to my knowledge, a way to make this faster with the current design.

On the positive side, runtime performance is subjectively reasonably fast. That's the tradeoff.

Perhaps there's a way to cache the ex_cldr modules in CI?

[Ievgen Pyrogov]

I think is related to the number of function clauses on a number of functions

Out of curiocity, could you elaborate on this part a little bit? What's happening during the compilation?

[Benjamin Milde]

During compilation all the downloaded data is turned into functions on various modules, so at runtime nothing is looked up on blobs of data, but only by patterns on functions.

[Kip Cole]

Closing as no planned since there is no obvious opportunity to improve compile performance with the current architecture. I may, some time in the future, look at ex_cldr 3.0. At that time I will experiment with other data encapsulation strategies which might improve this.