Unassigned/non-standard (compound) language and dialect codes

Question

Unassigned/non-standard (compound) language and dialect codes

agutkin opened this issue 3 years ago · comments

Wiktionary has entries for several languages and dialects with unofficial codes we can't scrape. Some examples of these include

Central Franconian: gmw-cfr
Old Galician/Portuguese: roa-opt
Westrobothnian: gmq-bot

possibly among others. The first part of the code denotes a valid ISO 639-3 language group, while the second part looks like a temporary assignment.

This issue is not a bug. It is simply intended for the book-keeping purposes. I suppose this is not related to #329.

Lucas Ashby · Answer 1 · Fri Jun 25 2021 07:06:06 GMT+0800 (China Standard Time)

codes.py produces a JSON of these when it's run. (I haven't looked at that functionality in a while but I assume it still appropriately collects all the codes we can't match.)

We've never done anything with these unmatched languages but the ones you've linked to definitely have enough entries to warrant supporting them.

Alexander Gutkin · Answer 2 · Fri Jun 25 2021 07:23:23 GMT+0800 (China Standard Time)

Aye, precisely. I should've opened this a long time ago - I noticed Westrobothnian has had a decent-sized lexicon for a long while.

Alexander Gutkin · Answer 3 · Sat Jun 26 2021 03:42:44 GMT+0800 (China Standard Time)

Looking at unmatched_languages.json it turns out that the Wiktionary language codes are rather systematically constructed.

The ones which are probably most problematic (in terms of work involved to support them) are the *-proto languages, but the remaining few five or six are probably reasonably easy to support. I guess what we have here is an edge case where the the wiktionary code maps to a non-existent compound ISO where the first part has to be a valid ISO language group name and should be verifiable, while the second can come from the configuration file.

Kyle Gorman · Answer 4 · Mon Jun 28 2021 22:03:37 GMT+0800 (China Standard Time)

@agutkin can you clarify: what's the action item you imagine here? We add config information for these language so that you can do wikipron gmw-cfr etc.?

Alexander Gutkin · Answer 5 · Mon Jun 28 2021 22:52:12 GMT+0800 (China Standard Time)

Yes, precisely.

Aryaman Arora · Answer 6 · Wed Jul 28 2021 21:38:02 GMT+0800 (China Standard Time)

BTW all of Wiktionary's invented non-ISO-compliant codes are listed here: https://en.wiktionary.org/wiki/Module:languages/datax

Gabriel Thompson · Answer 7 · Mon Aug 14 2023 05:49:11 GMT+0800 (China Standard Time)

codes.py produces a JSON of these when it's run.

I tested codes.py and it actually isn't outputting anything for the unmatched_languages.json file – the file just reads {} after the program is ran.

I think this is the result of a commit I made a few weeks ago. codes.py used to error when it encountered something like gmw-cfr. This was changed so that it throws a warning instead. However, I think this change unintentionally made it so that if a language isn't matched, it doesn't get added to unmatched_languages but instead just continues past it.

If we change the try statement starting at line 177 to:

try:
    iso639_lang = iso639.Language.match(wiktionary_code)
except iso639.language.LanguageNotFoundError:
    unmatched_languages[wiktionary_code] = {
        "wiktionary_name": wiktionary_name
    }
    logging.warning(
        "Could not find language with code %s", wiktionary_code
    )
    continue

then that should be back to normal.

I guess what we have here is an edge case where the the wiktionary code maps to a non-existent compound ISO where the first part has to be a valid ISO language group name and should be verifiable, while the second can come from the configuration file.

I'm confused about the format this file would have. Would it look something like this?

{
    "poz": {
        "mly-pro": "Proto-Malayic",
        "pro": "Proto-Malayo-Polynesian"
    },
    "gmq": {
        "scy": "Scanian",
        "bot": "Westrobothnian"
    },
    "gmw": {
        "cfr": "Central Franconian"
    }
}

That way, given a language code like gmw-cfr, Wikipron could identify the gmw as valid (using the iso639 module), and then use the config file to verify that cfr is a valid suffix and get the name of the name Wikipron uses for gmw-cfr. Sorry if I'm misunderstanding.

Kyle Gorman · Answer 8 · Mon Aug 14 2023 05:58:53 GMT+0800 (China Standard Time)

I tested codes.py and it actually isn't outputting anything for the unmatched_languages.json file – the file just reads {} after the program is ran.

I think this is the result of a commit I made a few weeks ago. codes.py used to error when it encountered something like gmw-cfr. This was changed so that it throws a warning instead. However, I think this change unintentionally made it so that if a language isn't matched, it doesn't get added to unmatched_languages but instead just continues past it.

If we change the try statement starting at line 177 to:
try:
    iso639_lang = iso639.Language.match(wiktionary_code)
except iso639.language.LanguageNotFoundError:
    unmatched_languages[wiktionary_code] = {
        "wiktionary_name": wiktionary_name
    }
    logging.warning(
        "Could not find language with code %s", wiktionary_code
    )
    continue
then that should be back to normal.

I see what you did there. Go for it.

I'm confused about the format this file would have. Would it look something like this?

I'll defer to @agutkin; Sasha, what'd you have in mind? What @sonofthomp proposes makes sense to me though.

Alexander Gutkin · Answer 9 · Mon Aug 14 2023 16:10:03 GMT+0800 (China Standard Time)

Yes, Kyle, @sonofthomp suggestion looks good to me as well.