2.34: Compile error `map key "person_names" was not found.`

Question

2.34: Compile error `map key "person_names" was not found.`

rubas opened this issue 2 years ago · comments

We can't compile ex_cldr anymore after update to 2.34. The locales don't see to be updated automatically.

== Compilation error in file lib/marko_cldr.ex ==
** (RuntimeError) Locale file :de is invalid - map key "person_names" was not found.
    (ex_cldr 2.34.0) lib/cldr/locale/loader.ex:201: anonymous fn/4 in Cldr.Locale.Loader.assert_valid_keys!/2

Changelog

Formatting Person Names: Added data and structure for formatting people's names. For more information on why this feature is being added and what it does, see the background document. A new library, ex_cldr_person_names will be developed to expose and exploit this data by year end 2022.

Dominic Rubas · Answer 1 · Mon Oct 24 2022 17:31:18 GMT+0800 (China Standard Time)

Deleting rm -rf priv/cdlr triggers a re-download of the updates locales.

Dominic Rubas · Answer 2 · Mon Oct 24 2022 17:32:28 GMT+0800 (China Standard Time)

But we also have force_locale_download only enabled for prod.

use Cldr,
    ....
    force_locale_download: Mix.env() == :prod
end

Kip Cole · Answer 3 · Mon Oct 24 2022 19:17:04 GMT+0800 (China Standard Time)

Until I can work out why this is happening (and I've tried everything I can think of to reproduce it without success) I recommend:

mix deps.clean ex_cldr

before running any mix deps.upgrade.

In addition, I recommend not including {:ex_cldr, "~> .....} directly in your mix.exs. Only configure the domain libraries (like ex_cldr_numbers etc) and have them pull the correct :ex_cldr transitively.

ardhitama · Answer 4 · Sun Oct 30 2022 12:44:36 GMT+0800 (China Standard Time)

happened to me as well after upgrading from 2.33.2 to 2.34.0

my mix:

{:ex_money, "~> 5.12"},
{:ex_cldr_dates_times, "~> 2.12"},

mix deps.clean ex_cldr doesn's solve the problem, deleting priv/clrd/locales/* solved it.

stack:

Generating AppName.Cldr for 3 locales named [:en, :id, :und] with a default locale named :id

== Compilation error in file lib/app_name/cldr.ex ==
** (RuntimeError) Locale file :id is invalid - map key "person_names" was not found.
    (ex_cldr 2.34.0) lib/cldr/locale/loader.ex:201: anonymous fn/4 in Cldr.Locale.Loader.assert_valid_keys!/2
    (elixir 1.14.1) lib/enum.ex:2468: Enum."-reduce/3-lists^foldl/2-0-"/3
    (ex_cldr 2.34.0) lib/cldr/locale/loader.ex:199: Cldr.Locale.Loader.assert_valid_keys!/2
    (ex_cldr 2.34.0) lib/cldr/locale/loader.ex:128: Cldr.Locale.Loader.do_get_locale/3
    (ex_cldr 2.34.0) lib/cldr/locale/cache.ex:117: Cldr.Locale.Cache.do_get_locale/2
    (ex_cldr 2.34.0) lib/cldr/config/config.ex:2212: Cldr.Config.decimal_formats_for/2
    (elixir 1.14.1) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2
    (elixir 1.14.1) lib/enum.ex:1658: Enum."-map/2-lists^map/1-0-"/2

Andreas Lappe · Answer 5 · Mon Nov 07 2022 17:17:24 GMT+0800 (China Standard Time)

Yes, same here, also without a direct dependency on ex_cldr.
Should the .json files be excluded from vcs?

Kip Cole · Answer 6 · Mon Nov 07 2022 17:47:22 GMT+0800 (China Standard Time)

I apologies that you’re having this issue. And especially that I can’t figure out why it is happening - I am unable to reproduce it.

Upgrading (directly or transitively) on every system I’ve tried completely replaces the deps/ex_cldr directory and therefore removes the locales associated with the old version. Then, when compiling, the missing locale results in a download of the version that is appropriate to that release.

The only exception to this is when you have a :data_dir configured for your ex_cldr backend that is not in the deps/ex_cldr directory, In those cases you can set the :force_locale_download key in the backend which will always download the locales at compile time.

What is being reported is being very confusing. It looks like the deps/ex_cldr directory is being merged not replaced. And therefore while the code is updated, the locales in priv/cldr/locales remains. I can’t figure out how that can happen.

Now there are three cases being reported (thank you!) but over 6k downloads of this release. From this I have to assume most (many?) upgrades are working fine. So I’m trying to work out what you have in common.

May I ask each of you to let me know:

Which operating system?
Which Elixir release?
Which OTP release?
Do you set either :data_dir or :force_locale_download in your configuration?

Please be assured I want to fix his issue with urgency, I’m just not able to find a reproducible case at this point.

Andreas Lappe · Answer 7 · Mon Nov 07 2022 17:56:50 GMT+0800 (China Standard Time)

Absolutely no need to apologize :-)

I did not set the data_dir but I noticed that the downloaded files are in apps/my_app/priv/cldr/locales, not in deps/ex_cldr/…. This at least might explain while it's not being replaced automatically. Also, the .json files have been committed to vcs in my case. I just removed them now. I can't explain why the files are located there though…

Which operating system?
macOS 12.6.1

Which Elixir release?
1.13.4

Which OTP release?
v24.3.4

Do you set either :data_dir or :force_locale_download in your configuration?

No. I did try :force_locale_download in the cldr module configuration last week in another project and this worked. I removed the key afterwards again and everything compiled fine.

Please be assured I want to fix his issue with urgency, I’m just not able to find a reproducible case at this point.

Let me know if you need more input.

Peter Brown · Answer 8 · Wed Nov 16 2022 23:58:21 GMT+0800 (China Standard Time)

I just ran into this with another person on my team after upgrading to 2.34. Forcing the locale download updated the "es" locale file and allowed us to compile. I'm on macOs 12.6, Elixir 1.13.2, and OTP 24.3.4.

Our configuration looks like this:

defmodule MyApp.Translations.Cldr do
  @moduledoc false

  use Cldr,
    locales: ["en", "es"]
end

Changing to this temporarily fixed it:

defmodule MyApp.Translations.Cldr do
  @moduledoc false

  use Cldr,
    locales: ["en", "es"],
    force_locale_download: true
end

ardhitama · Answer 9 · Fri Nov 18 2022 12:24:49 GMT+0800 (China Standard Time)

@kipcole9 no need to apologize, this problem is just a mere annoyance behind much bigger effort you've put into.

Which operating system?
Ubuntu 22.04.1 LTS on WSL2
Which Elixir release?
IEx 1.14.1 (compiled with Erlang/OTP 25)
Which OTP release?
Erlang/OTP 25 [erts-13.1.2] [source] [64-bit] [smp:24:24] [ds:24:24:10] [async-threads:1] [jit:ns]
Do you set either :data_dir or :force_locale_download in your configuration?
no and no

Malian · Answer 10 · Tue Nov 22 2022 19:14:40 GMT+0800 (China Standard Time)

Same issue here locally. When I deploy to production, I do not have an issue. Removing the fr.json file fixed the issue locally.
My configuration is the same as @ardhitama.

Kip Cole · Answer 11 · Wed Mar 15 2023 10:53:11 GMT+0800 (China Standard Time)

I now suspect this issue arises if the :otp_app config parameter is used in a Cldr backend configuration. For example:

defmodule MyAppWeb.Cldr do
  use Cldr,
    locales: ["ja", "en"],
    gettext: MyAppWeb.Gettext,
    # This parameter causes the locale data to be stored in the priv_dir
    # for `my_app_web` and therefore is not erased when `ex_cldr` is
    # upgraded.
    otp_app: :my_app_web,
    providers: [Cldr.Calendar, Cldr.DateTime, Cldr.List, Cldr.Number, Cldr.Unit]
end

If you're patient enough to stick with me on this, can you please confirm or deny if you are using that config parameter and you are having the issue?

In April 2023 I'll ship ex_cldr version 2.37.0 which will version the locale data files and auto-upgrade them when required so this issue can finally be closed. For now though I'd certainly like to confirm (or deny) my suspicions.

Malian · Answer 12 · Wed Mar 15 2023 20:30:53 GMT+0800 (China Standard Time)

@kipcole9 I have two Cldr module in the project where the issue occurred (monolith repository). One of the Cldr module use the opt_app parameter, the other does not. How could I help you to validate that the cause of the issue?

Peter Brown · Answer 13 · Wed Mar 15 2023 22:42:32 GMT+0800 (China Standard Time)

If you're patient enough to stick with me on this, can you please confirm or deny if you are using that config parameter and you are having the issue?

I am not using the otp_app config parameter.

ardhitama · Answer 14 · Sat Mar 18 2023 13:03:32 GMT+0800 (China Standard Time)

If you're patient enough to stick with me on this, can you please confirm or deny if you are using that config parameter and you are having the issue?

Yes, I'm using :otp_app config param

Kip Cole · Answer 15 · Wed Mar 22 2023 20:54:23 GMT+0800 (China Standard Time)

The cldr43 branch now includes locale file version checking and will download the right version if its not found. That will finally close this issue I believe. Target date is early April for release in alignment with CLDR 43.

Kip Cole · Answer 16 · Fri Apr 28 2023 13:16:07 GMT+0800 (China Standard Time)

With the release today of ex_cldr version 2.37.0 I believe this issue is now resolved.

When configuring locales, the locale installer will check if the version of any current locale is up-to-date and if not it will download the version for the current release of ex_cldr.

For most uses, the default location of the locale data means that the locales are deleted when a new version of ex_cldr is installed. For installations that use the :otp_app or :data_dir configuration options, deleting the old locale data is not guaranteed.

Thank to everyone for your patience. Please do give it a try. And, as always, please reopen the issue if required.