Nested lists for subdialect info
jhdeov opened this issue · comments
I noticed this problem for Armenian and a colleague told me it's also found in Portuguese. For some languages, the pronunciation entry can use a nested list, such that
- The first level contains the main dialect name
- The second level contains a subdialect name
For example, Portuguese <afetar> has a level-1 entry for (standard) Brazilian Portuguese. But this entry has 3 level-2 entries for different regions of Brazil. As of now, WikiPron scraps all 4 pronunciations as part of "Brazilian Portuguese". But that obfuscates the fact that the 4 entries correspond to separate subdialects.
It would be nice if the script could 'fix' this somehow. Maybe you can add an extra column to the scraped content, such that the new column would keep the name of its line-entry's name. For example, for afetar, maybe you could return something like
afetar | a f e t a ɹ | Brazil
afetar | a f e t a ɻ | Paulista
afetar | a f e t a ʁ | South Brazil
afetar | a f e t a χ | Carioca
@jacksonllee would love your thoughts on this. Keeps coming up...
I'd rather just have them in separate files and if people want to combine
them they're welcome to. I can't think how I'd use the three-column file
for actual processing without just filtering it into separate dialects, so
we should do that from the jump.
I share the same intuition. In general, augmenting the scraped data beyond the current two-column format would be what we should avoid within WikiPron. In this particular, I agree that just having separate files for individual subdialectal varieties is the way to go.
As for implementation -- if the Portugues afetar example is representative (the Armenian example shows that it's not -- I'll come back to this below), it might be possible to tighten the IPA extraction so that it more narrowly targets the individual <li> code somehow, while ignoring the distinction between the top-level <ul> (for "Portugal" and "Brazil") versus the embedded <ul> (for "Paulista", "South Brazil", and "Carioca"). If we do this, a scraping run will result in as many files as there are dialects and subdialects having been specified. The IPA extraction update could be done as a custom extractor (as has been done for other languages, for other reasons), if we're concerned about changing the default extraction behavior and its potential unknown effects for all other languages.
The Armenian երկրորդ example shows the embedded <ul> with just the word "colloquial" as the dialect/variety indicator, which is a challenge. For instance, if someone specifies "Eastern Armenian, colloquial" for the --dialect flag in a scraping run, there will be no result currently, because there's no way WikiPron would know that it should target the "colloquial" part under "Eastern Armenian, standard" in the given Wiktionary HTML. For cases like this, it would be ideal if the upstream Wiktionary data could be updated, so that the dialect/indicator is more verbose (i.e., showing "Eastern Armenian, standard", "Eastern Armenian, colloquial", "Western Armenian, standard", and "Western Armenian, colloquial" all in full) and that we then could use a similar strategy within WikiPron like what the previous paragraph describes for Portuguese.
I managed to get the Armenian Wiktionary editors to upgrade their script so that the colloquial entries now have a more informative text, like Portuguese.
Did the upstream editors fix this as far as our code is concerned?
Well for Armenian, most of the entries now look like Portuguese. There are still probably some stragglers because of some older entries. But I can find+fix those once I find out how the WikiPron code is handling the canonical cases like Portuguese.