Character Combinations for Segment Grouper
LinguList opened this issue · comments
I just experimented with a very simple method to allow for character combinations with SEgmentGrouper. The idea is that we use brackets to combine segments, so we can express what we would express in several lines in one. Later, we use a rule-parser, to make several lines out of these statements. This can be handy for orthography profile creation, and I have tested it in the profiler for JavaScript.
The code in Python is very simple:
def parse_rule(rule, delimiter=" "):
# first a grouping after brackets
groups = [[""]]
in_bracket = False
for char in rule:
if char == "[":
if not groups[-1]:
groups += [""]
in_bracket = True
elif char == "]":
in_bracket = False
else:
if in_bracket:
if char == delimiter:
groups[-1] += [""]
else:
groups[-1][-1] += char
else:
if char == delimiter:
groups += [[""]]
else:
groups[-1][-1] += char
return groups
It can be extended by adding a character for diacritics, that is later replaced.
Could you add a couple of examples?
Yes.
Suppose, you want to say "I know there are 5 vowels aeiou in my data, and I want to convert all vowels with -n to a nasal vowel. So you would then traditionally write several rule:
an ã
un ũ
... ...
With the rule-parser (better name can be developed later) for the orthoprofiles, you can write:
[a e i o u] n
And this will yield:
an
en
in
on
un
But you can also write for the conversion part:
[a e i o u] \u0303
and this yields:
ã
ẽ
ĩ
õ
ũ
We do not want to have these rules in an uncontrolled form in an orthography profile, or in a segmentgrouper, when doing conversions, but we may want to be able to have them as a preliminary step to guide the creation of a profile.
How many of these rules can one have on one line? What does [a b] c [d e] expand to?
And can they be nested?
list(product(*groupeditems))So it is always the cartesian product.
WAIT, I just figured out we use groups in the JavaScript tool, so you can also write something like:
[1 a b] c [2 d e]
This then translates to:
a c d
a c e
b c d
b c e
But
[1 a b c] c [1 a b c]
is
a c a
b c b
c c c
Experience shows that this functionality is important in many cases.
But it also shows I should think about this a bit more, as it is not clear how to include this in the workflow...
So it may be useful to NOT start working on anything in Python yet, but rather make sure that this JavaScript code is somehow explained somewhere, so more people than only me could use it ;-)
Yes, in particular this [1 ...] ... [2 ...] syntax isn't obvious at all. I guess the canonical example to lean on here would be regular expressions. So maybe aca|bcb|ccc rather than [1 a b c] c [1 a b c]. Not much of a shortcut in that case, but a lot more transparent.
Yes. I guess the problem is that we must somehow restrict the power of regular expressions to the typical use cases in linguistics. I'll keep reflecting about this.
I wouldn't want to allow full regular expressions, but re-using a subset of regex syntax to describe the rules would seem useful.
But aca|bcb|ccc would actually resolve to a c a|b c b|c c c, right? The problem for the formulation of regular expressions is always that they workon only on the level of chars, and if you work with characters, you need lots of workarounds when wanting to work with segments.
But one should probably first formulate the major principles of what one wants to generate here before discussing the representation. Being able to represent that a group of sounds like [a e i o u] should be combined with [n m ŋ] by Cartesian product is already one important aspect.
Experience shows that we also want to be able to model the second cases, where [a e i o u] is combined with [n m ŋ] AND [a e i o u], but the cartesian product should only concern the combination of the first group with the second one, and the third group should be combined in by just matching segments.
So we have two basic operations (that can be combined),
Linear matching:
[1 a e i][1 a e i]
aa
ee
ii
Cartesian product:
[1 a e i][2 a e i]
aa
ae
ai
ea
ee
ei
ia
ie
ii
And combination:
[1 a e i][2 m n][1 a e i]
ama
ana
eme
ene
imi
ini
And here, I wonder if one can use regular expression syntax to model this behaviour that is marked in the ad-hoc syntax with labeled groups.