rstr is bugged (^$ enclosed match)

[Kyryl]

Hello Ali,
Hope all is well.
Yesterday I've taken a look at some other optimization option like rstr.
I've successfully implemented rstr as an optional patch for nextvi.
I even made a multi threaded version too.
[1] https://github.com/kyx0r/nextvi/blob/patches/rstr.patch

As I was testing, I found a bug. Searches like ^int$ do not work correctly.
I have fixed the problem in the patch above, never mind the macro craziness though
(Yes I am that hardcore and determined to optimize constant if statements out the
hot loops)

Basically my implementation fixes the bug by running 2 separate loops one for ^ and last one
for $, if ^ loop matched and $ is set the execution is handed to the $ loop for another round
of verification.
This solution may not be in realm of neatvi's standards but I haven't thought of any other ways.
I just made it work and efficiently.

Offtopic:
I have made another cool patch for my needs, this one is like grep implementation in nextvi
[2] https://github.com/kyx0r/nextvi/blob/patches/grep.patch

From our last discussion I have came up with some even better solutions and optimizations
to led_render() function, basically this is a good as it will ever get. I guess the truly genius
part was how I reassign the hl attributes after the led_bounds() call.
Now I can actually give a proper answer as to why this problem is so damn hard,
and it's hard because any character can be at any arbitrary screen position out
of sequence, this is the definition of pure chaos that no algorithm can really
solve efficiently and with the same results as naive approach.
I even tried some other genius idea I had, that was to try min - max the pointers
of chrs array that will get placed on the screen, then try to run a bound from
min to max pointer. This does work for the most part except when characters
get reordered in such a way that the computed character at chrs[0] and chrs[n]
both can meet on the same screen. When something like this happens obviously
the bound will not be a bound anymore, it will be exact copy of entire string. It will
be impossible to highlight a line that gets reordered like that without processing
entire string, which sadly is utterly slow. That is why current solution in led.c
relies on a guaranteed string, but it's not going to be exactly the same when
text is in reverse. Still it's way better than anything I ever had before and very
well defined behavior.

Kind Regards,
Kyryl.

[Ali Gholami Rudi]

Hello Kyryl, Kyryl ***@***.***> wrote:

Hope all is well.

Thanks and you I hope you are as well.

Yesterday I've taken a look at some other optimization option like rstr. I've successfully implemented rstr as an optional patch for nextvi. I even made a multi threaded version too. [1] https://github.com/kyx0r/nextvi/blob/patches/rstr.patch As I was testing, I found a bug. Searches like ^int$ do not work correctly. I have fixed the problem in the patch above, never mind the macro craziness though (Yes I am that hardcore and determined to optimize constant if statements out the hot loops) Basically my implementation fixes the bug by running 2 separate loops one for ^ and last one for $, if ^ loop matched and $ is set the execution is handed to the $ loop for another round of verification. This solution may not be in realm of neatvi's standards but I haven't thought of any other ways. I just made it work and efficiently.

If I understand the problem correctly, this smaller patch may solve the problem too. Does it work? diff --git a/rstr.c b/rstr.c index 0a77132..8c3ca55 100644 --- a/rstr.c +++ b/rstr.c @@ -89,10 +89,10 @@ int rstr_find(struct rstr *rs, char *s, int n, int *grps, int flg) end = s + strlen(s) - len - 1; if (end < beg) return -1; - if (rs->lbeg) - end = beg; if (rs->lend) beg = end; + if (rs->lbeg) + end = s; for (r = beg; r <= end; r++) { if (rs->wbeg && r > s && (isword(r - 1) || !isword(r))) continue;

Offtopic: I have made another cool patch for my needs, this one is like grep implementation in nextvi [2] https://github.com/kyx0r/nextvi/blob/patches/grep.patch

Interesting.

From our last discussion I have came up with some even better solutions and optimizations to led_render() function, basically this is a good as it will ever get. I guess the truly genius part was how I reassign the hl attributes after the led_bounds() call. Now I can actually give a proper answer as to why this problem is so damn hard, and it's hard because any character can be at any arbitrary screen position out of sequence, this is the definition of pure chaos that no algorithm can really solve efficiently and with the same results as naive approach. I even tried some other genius idea I had, that was to try min - max the pointers of chrs array that will get placed on the screen, then try to run a bound from min to max pointer. This does work for the most part except when characters get reordered in such a way that the computed character at chrs[0] and chrs[n] both can meet on the same screen. When something like this happens obviously the bound will not be a bound anymore, it will be exact copy of entire string. It will be impossible to highlight a line that gets reordered like that without processing entire string, which sadly is utterly slow. That is why current solution in led.c relies on a guaranteed string, but it's not going to be exactly the same when text is in reverse. Still it's way better than anything I ever had before and very well defined behavior.

It may also be a good idea to use a simple rendering algorithm when no CR2L characters appear on the line. If I remember correctly, I tried it last time and it was to some extent effective. The following function is probably left from that experiment. diff --git a/led.c b/led.c index 68014e5..7eec4d1 100644 --- a/led.c +++ b/led.c @@ -52,6 +52,43 @@ static void led_markrev(int n, char **chrs, int *pos, int *att) } } +/* render and highlight a line; fast version */ +static char *led_render1(char *s0, int cbeg, int cend, char *syn) +{ + int *att; /* att[i]: the attributes of i-th character */ + int att_old = 0; + struct sbuf *out; + int i; + int att_blank = 0; /* the attribute of blank space */ + char *s; + int cpos = 0; + int soff = 0; + att = syn_highlight(xhl ? syn : "", s0); + /* generate term output */ + out = sbuf_make(); + sbuf_str(out, conf_lnpref()); + for (s = s0; cpos < cend && s[0]; soff++, s += uc_len(s)) { + int att_new = soff >= 0 ? att[soff] : att_blank; + int cwid = ren_cwid(s, cpos); + sbuf_str(out, term_att(att_new, att_old)); + att_old = att_new; + if (cpos < cend && cpos + cwid > cbeg) { + if (ren_translate(s, s0)) + sbuf_str(out, ren_translate(s, s0)); + else if (uc_isprint(s)) + sbuf_mem(out, s, uc_len(s)); + else + for (i = 0; i < cwid; i++) + if (cpos + i >= cbeg && cpos + i < cend) + sbuf_chr(out, ' '); + } + cpos += cwid; + } + free(att); + sbuf_str(out, term_att(0, att_old)); + return sbuf_done(out); +} + /* render and highlight a line */ static char *led_render(char *s0, int cbeg, int cend, char *syn) { Best wishes, Ali

[Kyryl]

Ali Gholami Rudi ***@***.***> wrote:

If I understand the problem correctly, this smaller patch may solve the problem too. Does it work? diff --git a/rstr.c b/rstr.c index 0a77132..8c3ca55 100644 --- a/rstr.c +++ b/rstr.c @@ -89,10 +89,10 @@ int rstr_find(struct rstr *rs, char *s, int n, int *grps, int flg) end = s + strlen(s) - len - 1; if (end < beg) return -1; - if (rs->lbeg) - end = beg; if (rs->lend) beg = end; + if (rs->lbeg) + end = s; for (r = beg; r <= end; r++) { if (rs->wbeg && r > s && (isword(r - 1) || !isword(r))) continue;

It works. Nice solution too. I may use this for single threaded rstr patch cause I feel like having 2 loops + that macro menace is posibly causing Icache thrashing, cutting the number of loops from 16 to 8 should make help. Originally, I thought that replacing the strlen(s) in rstr would significantly improve performance, however after rounds of benchmarks it made it worse because the linux kernel (my use subject) has lots of small strings and that could be excluded by only comparing lengths. Also some portion of processing is subtracted and *r is less efficient than r <= end Sadly though for multi core code I need those loops in this way so that I can terminate threads on demand in (O1) time with no special syncronization. Here are the rough numbers that I recall from yesterday: Search all occurances of the word "favorite" on linux kernel - default nextvi (regex using pikevm) 47.00s rstr.patch (default impl) 26.50s rstr.patch (insane macros + no strlen) 29.00s rstr.patch (insane macros) 26.00s threaded_rstr.patch (insane macros + no strlen) 26.11s ( * this is actually IO bound, 4 cpus at 40% ) threaded_search.patch (full regex) 31.00s ( * this is actually IO bound, 4 cpus at 50% ) Standard gcc -O2 used. So as you can see in practice rstr does not benefit much from extra cores in use, but things may be different were it not be my slow emmc storage. threaded_search.patch is very close to single core rstr a few more cores and it might get beaten. Insane macros reduce number of instructions executed but gives +500 more Imisses.

It may also be a good idea to use a simple rendering algorithm when no CR2L characters appear on the line. If I remember correctly, I tried it last time and it was to some extent effective. The following function is probably left from that experiment.

Thing is though, even non CR2L characters can be rearanged since the regex rules leave it kind of open ended. Of course these regex can be crafted specifically to only work on CR2L, but what good is it if it is to trash or not work on anything else. I write text rtl sometimes with a regex that does not even care about CR2L, just because I like weird things :) Kind Regards, Kyryl.