Mechanism to create datasets over the API
warmfusion opened this issue · comments
I've found it difficult to create new datasets with the column and datatypes I need - Whilst its possible through the user interface, I'd prefer an API interaction so that new datasets can be quickly made when new data sources are found to load into the system.
Would it be possible to create a dataset using something like
d = client.dataset.new({ name=>'warmfusions-data', columns = [ 'id' => 'numeric', 'name' => 'text' ], 'key'=> 'id' } )
client.dataset.delete(d.key)
Perhaps even a little #9 trick:
client.dataset.metadata.put( { update => Date.now, summary => 'Some little text' } )
Monitoring the network traffic through the web interface, I found https://opendata.socrata.com/api/imports2.json to be the correct url to create a new dataset. This, of course, requires authentication. I can't seem to find this info on the dev site, but a quick search of its github repo reveals that this is, in fact, documented.
My my @xmunoz, you sure are crafty. :)
More correctly, they were documented in old content on our old dev site. That's a deprecated SODA 1.0 API that we'll be removing over the coming months. I can't stop you from using them, but when 2.0 replacements are rolled out in the coming months they'll eventually be be removed. 👍
@chrismetcalf - So will this 2.0 replacement include this feature?
Yep. I wish I had design docs and a timeline to share with you but we're not quite to that point yet. If you'd like, I'll keep this bug open and we can share details as we get to that point. 😎
Is 2.0 out yet? Or is there a way to programmatically create datasets? We're importing a bunch of datasets (~400) from an existing data catalog and would love to automate it with python. Thanks!
I'm assuming this is possible because you can create datasets via Safe FME. I also would like to programmatically create them via an API
@mappingvermont @timwis New dataset creation APIs aren't available yet, and Safe FME uses the very old import API instead of an explicit creation API.
There are internal APIs that allow new datasets to be created - otherwise we wouldn't be able to create them from Socrata's tools either. But they're very old and not something I'd be proud to release to the world. If you poke around in my personal Github long enough, you may or may not find some example code that uses them.
@nitsjuly and I are working on plans for replacements for them, and we'd love feedback and ideas.
Any hints @chrismetcalf? That's a lot of repos :P I'll share if you share
@timwis Only because you're such a nice guy ;)
https://github.com/chrismetcalf/zillow-data/blob/master/bin/uploader
Standard warnings about private/deprecatable APIs apply. 
Thanks @chrismetcalf, but I think I got it working with the new API this morning:
https://github.com/timwis/socrata-creator
Working on incorporating it into sodapy rather than a separate library: https://github.com/xmunoz/sodapy/pull/8/files
...so that pull request has been merged but I just came across socrata-python-deprecated which uses the same API calls. Did I use an even older API @chrismetcalf? It certainly feels newer...
@timwis Actually they're the same API, its a very old private API. 😄