Wrong results for ray/box intersection on systems without isnanf

[Sebastian Keller]

Experienced behavior

On systems without isnanf, such as those using musl instead of glibc, sometimes a ray is wrongly detected to intersect a box. This is breaking picking in gnome-shell on Alpine Linux or Void Linux (when using musl): https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/3976

This is due to a typo in the non-HAVE_ISNANF code path in graphene_ray_intersect_box():

graphene/src/graphene-ray.c

Line 566 in a28aacb

 if (ty_max > tx_max || fpclassify (tx_max) == FP_NAN) 

This should be ty_max < tx_max as is already correctly done in the HAVE_ISNANF code path.

Expected behavior

Only detect an intersection when there is one.

Steps to reproduce

Try running the following program with glibc and with musl:

#include <graphene.h>

int main (int argc, char *argv[]) {
    graphene_point3d_t min, max;
    graphene_box_t box;

    graphene_point3d_init (&min, -14.001682, 50.620972, -50.368336);
    graphene_point3d_init (&max, 12.924316, 72.161774, -50.368336);
    graphene_box_init (&box, &min, &max);
   
    graphene_point3d_t o;
    graphene_vec3_t d;
    graphene_ray_t r;

    graphene_point3d_init (&o, 0.0, 0.0, 0.0);
    graphene_vec3_init (&d, -0.187414, 0.804487, -0.563628);

    graphene_ray_init (&r, &o, &d);

    if (graphene_ray_intersects_box (&r, &box)) {
        return 1;
    }

    return 0;
}

Operating system in use

Fedora 34, but graphene compiled against musl instead of glibc to test this behavior.

SIMD implementation in use

SSE2