bboxHexEstimate calls greatCircleDistanceKm with lat,lng instead of radians

[Jesse Montrose]

My first thought was that this was a build issue on my side. I'm new to the library, and still not sure I've got this right. Given a polygon approximately 1 square km, the diagonal call to greatCircleDistanceKm returns 82km instead of 1.4.

1.4 is the number I get calculating manually, but more importantly, is the number I get calling h3 python.

I noticed this while investigating oddly high estimates in maxPolygonToCellsSize, and my theory is that because this bug is returning higher numbers in most cases, the effects are hidden by the bounds filtering in polygonToCells.

I tried this naive fix, which gives me the correct results for my case, but unsurprisingly causes some other tests to fail, which I haven't investigated.

diff --git a/src/h3lib/lib/bbox.c b/src/h3lib/lib/bbox.c
index 033b02ed..b7680cbe 100644
--- a/src/h3lib/lib/bbox.c
+++ b/src/h3lib/lib/bbox.c
@@ -116,10 +116,10 @@ H3Error bboxHexEstimate(const BBox *bbox, int res, int64_t *out) {
 
     // Then get the area of the bounding box of the geoloop in question
     LatLng p1, p2;
-    p1.lat = bbox->north;
-    p1.lng = bbox->east;
-    p2.lat = bbox->south;
-    p2.lng = bbox->west;
+    p1.lat = degsToRads(bbox->north);
+    p1.lng = degsToRads(bbox->east);
+    p2.lat = degsToRads(bbox->south);
+    p2.lng = degsToRads(bbox->west);
     double d = H3_EXPORT(greatCircleDistanceKm)(&p1, &p2);
     double lngDiff = fabs(p1.lng - p2.lng);
     double latDiff = fabs(p1.lat - p2.lat);

Here is the test I've been working with:

    TEST(circleDistanceIncorrect) {
        printf("\n\nTEST START\n\n");
        // Python
        // h3.point_dist((52.524754, 13.455842), (52.515767, 13.441110),
        //   'km') 1.4114413820554332
        // h3.point_dist((52.524754, 13.455842), (52.515767, 13.441110), 'rads')
        //   0.00022154132650843334

        LatLng radp1, radp2;
        radp1.lat = degsToRads(52.524754);
        radp1.lng = degsToRads(13.455842);
        radp2.lat = degsToRads(52.515767);
        radp2.lng = degsToRads(13.441110);

        printf("\nrad dist_km %f\n", greatCircleDistanceKm(&radp1, &radp2));
        printf("\nrad dist_rad %f\n", greatCircleDistanceRads(&radp1, &radp2));

        LatLng p1, p2;
        p1.lat = 52.524754;
        p1.lng = 13.455842;
        p2.lat = 52.515767;
        p2.lng = 13.441110;

        double dist_km = greatCircleDistanceKm(&p1, &p2);
        double dist_rad = greatCircleDistanceRads(&p1, &p2);
        printf("\ndist_km %f\n", dist_km);
        printf("\ndist_rad %f\n", dist_rad);
        // incorrect dist_km == 82.434516
        // incorrect dist_rad = 0.012939
        t_assert(dist_km == 1.4114413820554332, "got expected km distance");
    }

[Nick Rabinowitz]

I think the issue here is just that all H3 coordinate input, in the C library, is expected to be in radians. We expose a degsToRads function to help with this, and in general the bindings (JS, Python, etc) all assume input in degrees, but in the C library itself everything is in radians. The BBox coordinates should be in radians, because the GeoPolygon coordinates should be in radians.

[Jesse Montrose]

Ah, you are so right. I was caught off guard by "polygonToCells takes a given GeoJSON-like data structure" in the docs and sticking to GeoJSON's WGS84 requirement.

And of course, that means I was also ignoring the "latitude in radians" in the LatLng struct...

Thanks!