Use Marky or other tool to get less noisy diffs
weitzman opened this issue · comments
Initial implementation done but I would prefer top send HTML there instead of sending an URL. Had lots of trouble with POSTING to marky - missing form fields.
Another potential strategy: What if we changed the selectors from "body" to whatever that particular webpage uses to contain its actual policy text? For example, in the Snapchat TOS the policy is contained in a <div class="column">.
Pros:
-Could select much more specific areas of the page, leaving off changes to head code or footer URLS
-Some websites might even have an id attribute on the div containing their policies! 😮
Cons:
-More work required per added page
-Some pages will probably still have messy HTML resulting in noisy diffs
-Possibly the JSON recipes don't even support specifying classes? I don't have much experience with this kind of setup.
EDIT: Poking around more I see there are already some sites that use an ID (good ol' Google) and that you sure can specify an ID or class selector, so now I've found stuff I can contribute 😃
@weitzman I found another tool that converts inputted HTML to markdown, but I'm not sure what diffibot's capabilities are -- this requires pasting HTML into one text field and then copying the output from another (there's no file export)
But this one may be exactly what we need -- It's a python script that seems able to do the conversion by both importing and exporting relevant files. Looks like you'll be able to feed it a privacy.selected.html and export markdown into privacy.md
@weitzman I think I found a winner: html.md is a node.js library that allows importing a .html file and exporting as .md. I think the command would run like this: htmlmd *.selected.html. Alas, I don't think there's an option to recursively run it through all subfolders, so we may need to run it in each recipe's respective dir.
Other options that work with node.js are domchristie's To-Markdown and netgusto's upndown, though these seem to only take strings as input.
Finally figured out how to send the selected HTML to the same web service. I prefer this to introducing new converters for now.
I reran difficode and all the new markdown is in the diffi repo. At first glance, it looks like a big improvement.
Closing this issue. We can open more as needed.
Thanks again for your help here, @lynnyliz.