Compatibility with SODA

[Raphael Dumas]

Similar to Explore Ohana integration #45 .

The SODA API seems to be a great and simple way to access open city/government data, and one can request data in JSON format with locations (but not geoJSON). It would be nice to have finda be able to interface directly with these, so that maps could be updated when the data updates.

I found this project in python to pull SODA data and convert it to geoJSON however I had difficulty getting it to work with SoQL queries.

[Sarah Laplante]

I would be cool if we could do the conversion in js so it would be all client-side and wouldn't require any hosting.

[Raphael Dumas]

Yayaya! I meant to imply that. Not a js/github expert so wasn't sure what would be possible.

[Raphael Dumas]

So I found GeoJSON.js, which seems like it should easily do the conversion. I added geojson.js & geojson.min.js to the /lib, and then tried modifying the require.js stuff in script.js, plus adding an import in /src/data/geojson.js as well as some code to handle querying the SODA api. But it doesn't appear to be playing well with require.js (it's written for node?).

This is my attempt

There's something here about making the two play well together but I will take a look later.

[Raphael Dumas]

I.... sort of got this to work in a somewhat inelegant way.

1. I installed GeoJSON.js referenced above using the npm node packet manager.
2. I referenced this using a <script> tag on index.html. I could not get the node module to work with requirejs, despite many attempts.
3. Changed the geojson_source to an http soql query
4. Changed src/data/geojson.js to query using the soql query, then process it using GeoJSON.js
5. It works!

Outstanding issues

1. radumas/sodafinda#1

Improvements to be made

1. The config.json could be modified to reflect the option of a static geojson or soql query (and src/data/geojson.js could have some conditional statements added to work with both options.
2. Making the GeoJSON.js node module work with requirejs.

[Harlan Weber]

Also make sure to update the README and wiki with instructions for how to connect with SODA!

[Paul Swartz]

My thought for how this could get brought back into the fold: right now, geojson_src is a top-level attribute. Instead, it could look something like this:

"data": {
  "href": "link to some JSON",
  "process": "function body"
}

Then, if "process" is defined, we'd create a function out of that body, and pass the data through it before it gets sent to the other parts of Finda as GeoJSON.

It's not the easiest thing in the world, but it would work with the existing Finda setup and allow the SODA-based versions of Finda to coexist with the others.

[Raphael Dumas]

So Socrata has brought GeoJSON support online, though I guess the JSON functionality of Finda would still be useful since the migration to the new API hasn't completed yet.

If I understand correctly you're suggesting to have the href field be the address to the JSON source, either local or HTTP request. If the process is defined, this would be code that Finda would execute to process the JSON to what is appropriate for leaflet to map?

I already wrote the beginnings of a conditional expression here which handles the selection of either GeoJSON or Soda as a source of data. If SODA, then it performs the processing @paulswartz wrote to convert from JSON to GeoJSON. Though the location data type isn't standardized across Socrata data sources, so the name of this parameter would have to be user-specified. So something like:

"data":  {
    "href": "link to some JSON",
    "type": "JSON",
    "location_field": "location"
}

If one wanted to extend support to, for example, Well-Known Text in a CSV, the processing part could be done according to the suggestion above. As I think of the variety of different formats geographic data can take.... being friendly to supporting them all seems a little futile. I guess as the public need develops.

Thoughts?

[Paul Swartz]

The fact that there are a variety of different formats is why I proposed the process field. As you mentioned, it would be Javascript code that Finda would execute to convert whatever data was fetched from the href field into the GeoJSON format that the rest of Finda expects. Whether that's parsing CSV, converting the SODA into GeoJSON, or parsing some binary format, it'll all be handled in that one place. Finda won't need to ship support for all those different formats.