Bt.cover (and bt.cut) render line segment incorrectly when endpoint is close to border

[Elijah Bodden]

When bt.cover or bt.cut is operating on two sets of polylines polylinesA and polylinesB, if any segment of polylinesA has one endpoint inside of polylinesB and one within 0.0001 of an edge of polylinesB (on the inside), the segment will render the opposite of how it should (it will render even though it shouldn't with cover, and won't render even though it should with cut). Example.

Looks like the bug is caused by the following two lines in pointInPolyline

// if on edge
const distanceToSegment = pDistance(point, [xj, yj], [xi, yi]);
if (distanceToSegment < 1e-4) return edge;

I don't know enough about the codebase to know why being on the lines needs to count as outside instead of inside (or thus how to fix the bug without breaking other stuff) but @leomcelroy probably knows. If this is necessary for other functions, removing it seems to at least not break bt.cover and bt.cut and fixes the bug, so we could maybe add an option that's only true when we call from the boolean function and change the lines to something like

// if on edge
const distanceToSegment = pDistance(point, [xj, yj], [xi, yi]);
if (distanceToSegment < 1e-4 && !ops.reffererIsBooleanFunction) return edge;

to disable this code in the context where it causes the bug

[Elijah Bodden]

Okay, patch actually does have bad results - when two edges are perfectly overlapping they both disappear in cover. I've tried some alternative solutions but haven't found any working solutions without this problem

[Elijah Bodden]

UPDATE: I was wrong, the "bad results" I was seeing were actually my test code being flawed in a way that made the bug unbreak it. After fixing the test code, the the bad results with the patch went away. That means the patch (which does resolve the bug) may actually work. Playing around with it, I don't see any bad effects, but there may be edge cases where the tolerance is needed even in cut and cover.