civic.json for metadata
chriswhong opened this issue · comments
Develop a standard schema for civic.json to add non-github metadata to civic app projects.
The projects page can look for civic.json in the root of every repo to add things like:
-Tag for in-progress, alpha, beta, complete
-Tag for needs (I need a UX dev, I need a rails guru, etc)
-Tag for sector (transportation, housing, government, council)
Please respond to this issue with ideas for what should be included in civic.json
Related conversation in Chicago
Here's a first draft.
status should be from a set of allowed values: beta, on hold, production, etc. (Not sure what these should be)
thumbnail is just a url to an image for use in the catalog.
bornAt links this project to particular event.
geography defines a city, county, or other geography that this project is meant to serve.
needs is an array of need objects, text is arbitrary.
types is an array of type objects, text is arbitrary.
categories is an array of category objects, text is arbitrary.
Should we limit types, needs and categories to predefined set of values? Or, perhaps we should have a set list of authorized types, but allow for arbitrary types under some different key?
[
{
"status": "production - ongoing",
"thumbnailUrl": "http://civic.com/hack.png",
"bornAt": "National Day of Civic Hacking 2014",
"geography": "NYC",
"needs": [
{"need": "frontend dev"},
{"need": "UX Help"},
{"need": "rails developer"}
],
"types": [
{"type": "dataset"},
{"type": "mobile app"},
{"type": "web app"},
{"type": "data visualization"},
{"type": "policy document"}
],
"categories": [
{"category": "transportaion"},
{"category": "public safety"},
{"category": "politics"},
{"category": "news"},
{"category": "housing"},
{"category": "education"},
{"category": "health"}
]
}
]
Perhaps add
"socialMedia":{
"twitter":"@myCoolProject",
"facebook":"http://www.facebook.com/myCoolProject"
}
Incorporated a test version of civic.json into a few repos and got all the backend stuff working to look for this file and add it to the rest of a repo's data. Check it out, it's live now at projects.betanyc.us
Saw the civic.json project and the demo implementation on projects.betanyc.us. Awesome to see you guys run with this.
I've been thinking about it a bit more and I'm not sure a civic.json file added to each project is the answer. Here's my thinking:
The main idea behind tracking a list of repos via Github is to remove the need for curation and maintenance of these civic projects. Project directory sites in the past have quickly gotten stale and less useful shortly after the initial round of curation is done. I think a JSON file that needs to be manually updated is better and de-centralized, but with it the work of curation still remains and and actually becomes less consistent.
So, the question I pose is: what information is missing from the Github API that will help us better organize and collaborate on our civic projects?
One theme I've seen with the spec lined out by @chriswhong is a place to ask for specific skilled help (designer, python developer, etc). From my experience, these needs change all the time and are often more nuanced than a simple "we need X." To this end, wouldn't it be better to leverage the issue tracker and perhaps come up with a framework for opening and closing needs? Something like "Need: home page design." That way, when the need is met, it can be closed more naturally via conversation & commits, rather than someone remembering to update a JSON file.
We could also automatically detect the license type by looking for LICENSE.txt. Things like location and categories may still need a civic.json or some other solution, but my gist here is to look at ways to hook in to existing functionality like the issue tracker and establish a set of Github standards that accomplish our goals and help spur on collaboration if followed.
Curious to get some thoughts on this from @fgregg @evz @jpvelez @fkh @benbalter @kfogel @GovInTrenches
+1 for Github practices that help get work done and produce documentation
programmatically as a byproduct.
Scratching my head on how to do elegantly that for location, categories,
and data sources though.
Data sources = url of data or identified you used to build the app or
analysis. Will make it easier to connect datasets to projects.
On Mon, Jan 20, 2014 at 3:33 PM, Derek Eder notifications@github.comwrote:
Saw the civic.json project https://github.com/BetaNYC/civic.json and
the demo implementation on projects.betanyc.us. Awesome to see you guys
run with this.I've been thinking about it a bit more and I'm not sure a civic.json file
added to each project is the answer. Here's my thinking:The main idea behind tracking a list of repos via Github is to remove the
need for curation and maintenance of these civic projects. Project
directory sites http://commons.codeforamerica.org/ in the past have
quickly gotten stale and less useful shortly after the initial round of
curation is done. I think a JSON file that needs to be manually updated is
better and de-centralized, but with it the work of curation still remains
and and actually becomes less consistent.So, the question I pose is: what information is missing from the Github
API that will help us better organize and collaborate on our civic
projects?One theme I've seen with the spec lined out by @chriswhonghttps://github.com/chriswhongis a place to ask for specific skilled help (designer, python developer,
etc). From my experience, these needs change all the time and are often
more nuanced than a simple "we need X." To this end, wouldn't it be better
to leverage the issue tracker and perhaps come up with a framework for
opening and closing needs? Something like "Need: home page design." That
way, when the need is met, it can be closed more naturally via conversation
& commits, rather than someone remembering to update a JSON file.We could also automatically detect the license type by looking for
LICENSE.txt. Things like location and categories may still need a
civic.json or some other solution, but my gist here is to look at ways to
hook in to existing functionality like the issue tracker and establish a set
of Github standards that accomplish our goals and help spur on
collaboration if followed.Curious to get some thoughts on this from @fgregghttps://github.com/fgregg
@evz https://github.com/evz @jpvelez https://github.com/jpvelez @fkhhttps://github.com/fkh
@benbalter https://github.com/benbalter @kfogelhttps://github.com/kfogel
@GovInTrenches https://github.com/GovInTrenches—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-32799153
.
In the spirit of having as few fields as possible, I think "bornAt" and "status" are probably not critical. Status can be inferred from recent commit history.
Derek Eder notifications@github.com writes:
So, the question I pose is: what information is missing from the
Github API that will help us better organize and collaborate on our
civic projects?
Amen -- that's the right question, I think.
One theme I've seen with the spec lined out by @chriswhong is a place
to ask for specific skilled help (designer, python developer, etc).
From my experience, these needs change all the time and are often more
nuanced than a simple "we need X." To this end, wouldn't it be better
to leverage the issue tracker and perhaps come up with a framework for
opening and closing needs? Something like "Need: home page design."
That way, when the need is met, it can be closed more naturally via
conversation & commits, rather than someone remembering to update a
JSON file.
Using the GitHub issue tracker API for stuff like this seems like the
right way to go.
We could also automatically detect the license type by looking for
LICENSE.txt.
There are some projects out there already to automate license detection
(since it's not always in LICENSE.txt, and even when it is it's not
always formatted in exactly the same way from project to project).
http://www.fossology.org is one such, but I think there are others too.
Things like location and categories may still need a civic.json or some
other solution, but my gist here is to look at ways to hook in to
existing functionality like the issue tracker and establish a set of
Github standards that accomplish our goals and help spur on
collaboration if followed.
+1. The GitHub API is a pretty rich place, and getting richer all the
time as far as I can tell. Much better to invest time in using it to
solve problems (even in cases where it currently does so imperfectly)
than to create new systems for solving those problems (because the new
systems would be imperfect too, and they'd be new).
-K
-
One of the things I'd like to do is to enable searching by subject. Once we're able to plug in civic.json into a UI for people who have never heard of github, it would be useful to be able to sort by subject matter. There's a big typography project that would need to happen to successfully categorize the wide world of projects (which @smartchicago has been looking at) - but at minimum being able to help tag projects by things like "311", "bikes", "legislation" would be useful.
-
The other thing in terms of process that would be interesting would be to integrate this into different hosting services. Rather than host all these apps on our own servers, to have Civic Commons / CfA Commons be a hosting provider. (Like Smart Chicago is locally.) That would ensure that we would grab the right information when it gets built (since they would need to provide that as part of the deal) and it would mean somebody is always looking at the list. (We'd have to figure out exact logistics on that but it's one way to solve the problem of sites going stale)
-
Contact information - At minimum, I should be able to ping the technical person's email so that even if a non-technical person doesn't get anything else about what's going on - they can still reach out.
- Yes, that's what we mean by "categories." The question is how to do this seamlessly, so people don't have to do extra work to enter them.
- Not sure what you mean by "integrate this into different hosting services." CfA commons doesn't host apps, it just links to them.
- We might be able to grab this from the "Contact Us" section of the README, if it's there. This is usually pretty standard.
Sorry, was responding to @derekeder via email and hadn't seen the previous post. Categories would almost need a whole separate discussion if we're doing pre-defined values anything beyond arbitrary. Perhaps we could start with 'tags' as a start-point and then add categories as pre-defined values.
The hosting idea hasn't happened yet, but I'm saying it would be a possible solution to making sure somebody is always updating the entries and at least checking to make sure the projects still exist.
Folksonomy -> taxonomy is a good idea, except it would be difficult to get
people to update a bunch of disparate repos once we landed on a taxonomy.
If we were to define a taxonomy, the way to do it would be to amass several
more civic tech project lists, and distill public policy areas (health,
311, transportation) and project types (analysis, app) from that.
What uses does a taxonomy serve? What would be possible to do that is not
possible to do now?
On Mon, Jan 20, 2014 at 5:58 PM, Juan-Pablo Velez
notifications@github.comwrote:
Folksonomy -> taxonomy is a good idea, except it would be difficult to get
people to update a bunch of disparate repos once we landed on a taxonomy.If we were to define a taxonomy, the way to do it would be to amass several
more civic tech project lists, and distill public policy areas (health,
311, transportation) and project types (analysis, app) from that.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-32809111
.
773.888.2718
2231 N. Monticello Ave
Chicago, IL 60647
Finding projects that pertain to a policy area like 311/transit or type
statistical analysis/infrastructure/civic app, and analyzing the breakdown
projects according to these criteria (probably less important.)
Making things easier to find might lead to more project awareness / use /
reuse. To be worth doing, it would have to be substantially more useful
than search.
Agree, bornAt isn't really necessary, I just thought it might be neat to
tie projects to events. This could just as easily be in the description
and would still show up in searches based on the description.
My thoughts on status are that I would like some way to distinguish what is
a finished product versus what is a work in progress. I think there are
lots of projects that start and never get finished, which is fine, but it
would be great to filter the finished projects to make a nicer "showcase"
list of actual, working, show-able projects.
On Mon, Jan 20, 2014 at 4:45 PM, Juan-Pablo Velez
notifications@github.comwrote:
In the spirit of having as few fields as possible, I think "bornAt" and
"status" are probably not critical. Status can be inferred from recent
commit history.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-32800036
.
My thoughts on categories are that they should be more tag-like for now,
while the number of projects as small. Just like when tagging a blog post,
add as many as come to mind and hope it ends up in someone's search/filter.
When we have hundred or thousands of data points that are all tagged, we
can have a better discussion about what the "official" category values
ought to be. Just my two cents.
-C
On Mon, Jan 20, 2014 at 7:03 PM, Juan-Pablo Velez
notifications@github.comwrote:
Finding projects that pertain to a policy area like 311/transit or type
statistical analysis/infrastructure/civic app, and analyzing the breakdown
projects according to these criteria (probably less important.)Making things easier to find might lead to more project awareness / use /
reuse. To be worth doing, it would have to be substantially more useful
than search.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-32809398
.
Good stuff. It would be useful for feature selection to come up with some user stories for how someone would use this tool and for what purpose. I'd rather work towards actually engaging people rather than making a bunch of 'it would be cool if' features.
@tothebeat blogged about how he wasn't sure what project to help out on. I could see this list serving the need of "I'm a developer with X skill. What projects are out there that I could plug in to?"
Are there others?
Hi everyone.
To second the @tothebeat request, at Code for America we get an immense amount of offers for volunteer help. With a civic.json standard and good Github Issue labels, we could easily build a tool that would filter projects on programming language, category, time commitment, and geography. With that we could offer up interesting projects for volunteers to work on. I found a rough prototype someone else built called Issuehub.io. The extra metadata adds a lot of value to that type of tool.
I could be convinced either way if it should be a file in each repo or one master list.
Andrew: @derekeder is working on that. Ping him.
@evz and I made some changes to the hack night projects list to support a custom project-needs label.
Check it out: http://opengovhacknight.org/projects.html#/?search=project needs
The idea is if you need something for your project and you're looking for outside help, create a new issue describing it and add the project-needs label to it and it will show up on our list.
One thing that I'm not keen on about issuehub.io is it hooks in to all of Github and looks at some very general labels. I don't know about you guys, but just because I file an issue on Github doesn't mean I'm asking for someone else to fix it for me. More often than not, it's just a reminder for me to do it later.
We'll be showing this off at tomorrow's hack night. We'll see how people use it.
Awesome. Cheap feedback: "projects need" bit would look better under the
project stats, in the right-column.
- Juan-Pablo Velez
312-218-5448
jpvelez@gmail.com
On Mon, Jan 27, 2014 at 11:56 AM, Derek Eder notifications@github.comwrote:
@evz https://github.com/evz and I made some changeshttps://github.com/open-city/civic-json-worker/commit/545c24e82821152635b2571d9e75e21cafa7c8deto the hack night projects list to support a custom
project-needs label.Check it out: http://opengovhacknight.org/projects.html#/?search=projectneeds
The idea is if you need something for your project and you're looking for
outside help, create a new issue describing it and add the _project-needs_label to it and it will show up on our list.One thing that I'm not keen on about issuehub.io is it hooks in to all of
Github and looks at some very general labels. I don't know about you guys,
but just because I file an issue on Github doesn't mean I'm asking for
someone else to fix it for me. More often than not, it's just a reminder
for me to do it later.We'll be showing this off at tomorrow's hack night. We'll see how people
use it.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-33401523
.
This is rad and that it's clickable and has an explanation of the need is
extra rad.
-C
On Mon, Jan 27, 2014 at 1:25 PM, Juan-Pablo Velez
notifications@github.comwrote:
Awesome. Cheap feedback: "projects need" bit would look better under the
project stats, in the right-column.
- Juan-Pablo Velez
312-218-5448
jpvelez@gmail.comOn Mon, Jan 27, 2014 at 11:56 AM, Derek Eder notifications@github.comwrote:
@evz https://github.com/evz and I made some changes<
chihacknight/civic-json-worker@545c24e82821152635b2571d9e75e21cafa7c8de>to
the hack night projects list to support a custom
project-needs label.Check it out:
http://opengovhacknight.org/projects.html#/?search=projectneedsThe idea is if you need something for your project and you're looking
for
outside help, create a new issue describing it and add the
_project-needs_label to it and it will show up on our list.One thing that I'm not keen on about issuehub.io is it hooks in to all
of
Github and looks at some very general labels. I don't know about you
guys,
but just because I file an issue on Github doesn't mean I'm asking for
someone else to fix it for me. More often than not, it's just a reminder
for me to do it later.We'll be showing this off at tomorrow's hack night. We'll see how people
use it.—
Reply to this email directly or view it on GitHub<
https://github.com/chriswhong/betanyc-projects-list/issues/5#issuecomment-33401523>.
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-33404450
.
Thats great. I agree that a specifically asking for help can be really
helpful and it fits into the UI of your project page nicely. NYC has it
over on the right column pulling from the civic.json file. I'm with you
that Github is a better place for that.
Another method I've used well is having beginner labels on an issues. Then
I link to the list of beginner
issueshttps://github.com/codeforamerica/bizfriendly-web/issues?labels=beginner+friendly&page=1&state=open,
so people have a good place to get started as they are learning the
project. Could that fit in as well, or is being linked to from the README
enough?
Andrew Hyder ::
Developer Relations Engineer
2013 Code For America Fellow
@hackyourcity http://twitter.com/hackyourcity
On Mon, Jan 27, 2014 at 9:56 AM, Derek Eder notifications@github.comwrote:
@evz https://github.com/evz and I made some changeshttps://github.com/open-city/civic-json-worker/commit/545c24e82821152635b2571d9e75e21cafa7c8deto the hack night projects list to support a custom
project-needs label.Check it out: http://opengovhacknight.org/projects.html#/?search=projectneeds
The idea is if you need something for your project and you're looking for
outside help, create a new issue describing it and add the _project-needs_label to it and it will show up on our list.One thing that I'm not keen on about issuehub.io is it hooks in to all of
Github and looks at some very general labels. I don't know about you guys,
but just because I file an issue on Github doesn't mean I'm asking for
someone else to fix it for me. More often than not, it's just a reminder
for me to do it later.We'll be showing this off at tomorrow's hack night. We'll see how people
use it.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-33401523
.
Just realized you can move the projects over to the right hand size because
those are all columns, so moved them below 'contributors' so things look
less cramped: http://opengovhacknight.org/projects.html
#details
I would suggest basing civic.json on Schema.org's existing SoftwareApplication schema definition:
Thanks! This is very interesting... probably overkill, but a great
reference for describing a project.
On Tue, Jan 28, 2014 at 10:28 AM, sheldonrampton
notifications@github.comwrote:
I would suggest basing civic.json on Schema.org's existing
SoftwareApplication schema definition:http://schema.org/SoftwareApplication
Reply to this email directly or view it on GitHubhttps://github.com//issues/5#issuecomment-33488536
.
Chriswhong, Sheldon makes a great point. We should be using an open stander.
Hi everyone, just joining this discussion. You guys have put in a lot of great work over the last several months to make this a reality. I'm happy to contribute where I can.
It's possible to implement an open standard such as the one @sheldonrampton mentioned by using a JSON-LD context for our civic.json files. Here's an example:
{
"@context": {
"schema": "http://schema.org",
"thumbnail": "schema:thumbnailUrl",
"types": "schema:applicationCategory",
"categories": "schema:keywords"
},
"thumbnail": "*url*",
"types": [
"Dataset",
"Mobile App"
],
"categories": [
"Transportation",
"Public Safety"
],
"status": "alpha",
"schema:Place": "nyc",
"needs": [
"frontend",
"ux",
"rails"
]
}I couldn't find schema.org terms definitions for "status" and "needs", but it isn't out of the question to develop our own definitions for these - or any other civic.json terminology - to go alongside this standard. It would take some time, obviously, and may be a pre-optimization.
We also probably don't want to require each developer to have to add this context to each and every one of their civic.json files, especially when nearly all of it is going to be redundant (unless we want them all to learn about Linked Data, which wouldn't be horrible). One solution JSON-LD allows would be to use context as a url, like this:
{
"@context": "*contextUrl*",
"more fields": "..."
}The contextUrl would then map to a webpage where our context lives. This still requires an additional field for the developer, so its worth exploring to see if we can do all the open standards bit on the backend and then just require a specific vocabulary: "needs", "categories", "types", etc... This would allow 0 change to the current civic.json format, and we'd still be using open standards. I think it would work as long as our json endpoint had the context object.
Getting into this late because there's interest in Oakland around this (holler.)
Putting on my run-of-the-mill Brigade member fedora, I'm thinking through what hooking up to this involves on the user end because it seems like minimizing friction is a big part of it.
It seems like the original thought was to place a civic.json file, but -- per @derekeder's comments here -- the direction is moving towards a hybrid of using GitHub Issues for needs, and a civic.json file for the rest?
(Apologies if my understanding's incorrect -- this discussion is spanning a bunch of repos, threads, and email chains.)
IMHO, I'd vote for doing less, and just sticking with civic.json. My rationale is:
- Each additional thing you're asking end-users to do will reduce adoption + increase likelihood of data staleness/non-conformity.
- The
needsattribute outlined in the spec seem more like general roles for participation, rather than concrete tasks/features. Concomitantly, these needs probably won't change very often. That said, I know the Chicago mafia has a lot more experience with this, so tell me if I'm crazy. - If we're asking people to add a
civic.jsonfile already, it seems like it minimizes friction to just keepneedsin there.
I totally get the desire to make use of GitHub's infrastructure for this, but because using Issues doesn't seem to obviate the need for most of the civic.json file, I'm worried about the additional layer of complexity this adds for not that much benefit.
Fundamentally it seems like it's more a substantive discussion of how we're defining "need": a concrete task to be done vs. a skill set. I worry about the former suffering from overspecification.
Anyway, just thinking through this as someone who'd love to implement it in my city/brigade, and trying to sort out what exactly it will entail.
cc @migurski @ondrae for your thoughts, given the work you're doing on civic-json-worker
@daguar the best entry point is to read this overview of civic-json-worker and civic.json.
the overall project roadmap is here.
tl;dr: the goal is to describe projects with as little data entry burden as possible. that means:
- get most project fields from Github API
- reach a consensus on a very short list of additional - and likely optional fields - project topic (311, transit), type (web app, data analysis)
- figure out how to make these fields as easy to gather as humanly possible. that means either tease them out from useful github issues (i.e. getting project need from repo issue that also helps describe the need) or make find clever ways to make the data entry easy (autocomplete tags on the project page, drop in fields into a github issue so non-repo-owners can document projects, etc.)
And how's this being carried out?
- Brigade project pages simply submit github repo urls to civic-json-worker, a project tracking API that goes out and collects all the fields from github. the project page then pull the complete project data to display on the page. @chriswhong is writing a blog post to help existing brigade website read/write to the API from their existing pages, and is probably going to turn betanyc's project page into a generic project page for new brigades to use.
- civic-json-worker needs to be adapted to accept and serve projects from multiple cities. right now it only does it for Chicago (we decided to build on the initial flask version of the API, not the NYC node version. NYC's project page will swap out their backend once that happens.) @ondrae and @evz are leading this, the work is happening on CfA's new fork of civic-json-worker.
- civic.json started off as a file you dropped into your repo to document everything. but this concept is limited because you can get many of the fields from the Github API, and it puts the burden on the repo owner to fill in / maintain the additional fields. we already experimented with this in Chicago, and it didn't work. better to have the fields be a byproduct of issue tracker activity and/or to make it easy for non-repo-owners to maintain the fields. so now civic.json is going to be programmatic output of civic-json-worker. we make robots build them, instead of humans, and we pass them to the projects pages for display. @chriswhong and myself are organizing the discussion around what fields should be in the spec, and what are some efficient ways to make robots build them; work is going on in this repo.
Thanks JP & Dave, this is a super useful discussion and I think we’re honing in on an incredibly useful community resource here.
FWIW, I am somewhat skeptical of the idea that meaningful needs and project description will simply fall out of existing Github features. CfA’s desire here is to have a high-quality, primary source of project-related data. Github will help with that, but it can’t be the sole source of information. I view Github metadata as one set of data that we collect to build up a database, but I want to be sure that we’re open to alternative sources: sites other than Github, projects other than code, and so forth.
I’ve posted some related comments on codeforamerica/civic-json-worker issue 14 and 13.
@migurski No argument there. "Get everything from Github" was just a
first-stab proxy for "make data gathering as easy as humanly possible
because otherwise it will fail. Get everything you can through robots, make
the rest stupid simple."
@chriswhong Can you close this issue so we can official move the discussion
to the civic-json-worker and civic.json repos?
Donezo