Improve component file mergability by supporting another serialization format

[Devin Abbott]

The problem

Lona's .component files are hard to merge. This is an important problem to solve, since Lona components are specifically designed to be the "source of truth" for large teams, and edits made by 2 people will be fairly common.

Component files should be at least as merge-friendly as code files:

• changes to unrelated lines are generally automatic
• changes to the same lines are a nuisance and may require some manual touch-up, but usually aren't a big deal

Lona's .component files are mainly hard to merge because they're stored as JSON:

1. JSON is hard to edit by hand, since it has lots of punctuation that's easy to get wrong: e.g. commas, curly braces, square brackets, quote marks. Often tweaks need to be made manually when merging, which can easily lead to syntax errors.

2. Adding or deleting a line (which is what git merges usually do) can easily result in an invalid file:

   1. since trailing commas aren't allowed, modifying the contents of an object or array can lead to a syntax error
   2. since any key/value pair can exist in any object (not much contextual info), git's merging algorithm has a hard time determining whether lines have been moved vs added/removed

3. Editors and tools often don't provide very good diagnostics about where the syntax errors are, which makes it hard to recover from bad merges. For example, parsing a JSON file in node gives the character position of the error:

   > JSON.parse(fs.readFileSync('package.json').toString() + ",")
   SyntaxError: Unexpected token , in JSON at position 3173
   

4. The Lona layer hierarchy is fairly hard to understand from looking at the JSON file, and this is the part of the file that generally needs the most manual touch-up. When editing a deeply nested JSON value, it's hard to tell tell which component you're editing (the name field may be far away, both for the current component and its parent).

Solution idea

I think we should consider introducing another option for .component file serialization format. For any workspace, the user can choose whether to save components as JSON or X, and X will be the default for better mergability. Our JSON and X files will be 1-to-1. The compiler and LS should be able to read X and convert it to JSON, and vice versa. External tools can leverage the compiler (or perhaps another shared dep) to convert all component files to JSON, since that'll be easier for most tools to work with. For LS, we can decide whether to read X natively with a Swift lib, or invoke some JS through JSC to convert X to JSON.

I happened to find this diagram from 2 years ago when I was just starting Lona that illustrates the main idea:

Lona components were never supposed to be JSON on disk, but that happened to be most convenient to get off the ground, and it ended up sticking.

Some options for X:

• JSON5, https://github.com/json5/json5 - a superset of JSON with support for trailing commas (good for automatic edits), unquoted keys (good for manual edits), and some other convenience things, while still keeping the familiarity JSON. This improves (1) and (2), doesn't change (4), and may change (3) depending on the parser implementation. E.g. Card.component as json5: https://gist.github.com/dabbott/31373139f869d038625e4f20274805ee
• XML - This improves (1) and (2), and should greatly improve (4). (3) may change depending on the parser implementation. This also has the benefit of being somewhat familiar to designers since it's HTML-like. It has the added complexity of needing to decide whether each key/value should be an element or attribute, based on how each will affect readability and mergability. Note that since XML elements can't be used as attributes (like in JSX), any nestable value would need to be an element -- although we could decide this dynamically based on the actual value. E.g. Card.component as XML: https://gist.github.com/dabbott/383d6d464ed2055a5b813989188ba8e2
• JSX - For the most part, this has the benefits of JSON5 + XML. The main advantages over XML are that elements can be used as attributes, seamless embedding of JSON within attributes, and the flexibility of potentially deciding to use any other JS syntax later. JSX doesn't support namespaced attributes like XML, but that can be avoided with something like a reserved _ prefix or a wrapper for Lona properties, e.g. _meta: {}. The downside of JSX is that the parsing libraries are much more complicated and exclusively for JS. E.g. Card.component as JSX: https://gist.github.com/dabbott/4a7b869b93b68aab3040b318aa7993f0
• Other?

Interesting, but probably not:

• JSX and store the component as an actual React Native class component with a render function. I used to like this idea, since the Lona components could potentially be runnable as React Native code. However, most of the component file doesn't map directly to React Native anymore since that turned out to be too limiting, so I don't think this should be a goal.
• TSX/Flow - It'd be potentially pretty powerful to have a typed version of Lona files that could be compile-time type-checked in some way. However, I'm not sure exactly what that would help with.
• YAML, TOML - These have less punctuation and better line-wise merging, but handle deep object/array nesting very poorly

Longer-term:

• It'd be nice to have a built-in visual diff/merge tool in LS. Merges will always still happen outside of LS too, but for complex merges we could recommend doing them in LS.

My current thinking

I think the XML version of components looks pretty nice and should help with mergability quite a bit. I don't think the added complexity of JSX is worth the advantages.

I would probably start by writing a JS lib that converts Lona components between JSON and XML, and then use that in both the compiler/LS. Then if we decide to stick with XML, at some point I would swap the JS version for a native Swift XML lib for perf in LS.

What do you think?

[Mathieu Dutour]

That's an interesting problem for sure.

Are you confident that xml would improve mergeability? Because git is line based, I don't think xml would actually be much better. While it might not create an invalid xml file as often, it could totally create a valid xml file but invalid component. And I'd say that's worse?

[Devin Abbott]

I'm pretty confident it'll be better than JSON in a lot of the common cases, although I'd definitely want to experiment with it for a while to get a better understanding before we commit to it. One thing that would help is having some concrete example merges as test cases. Then we could get a more scientific comparison JSON vs. XML. That'd also inform how we decide what should be an attribute vs. element.

Your point about valid xml but invalid component is very possible... we'll probably have to try it to find out. It's probably easier to end up with a valid XML file, but I don't know if that means it's more likely to be an invalid component. It's pretty easy to screw up the JSON hierarchy too when trying to fix syntax errors.