Add support for "relative" BIP32 derivation paths

[Richard Brady]

• I'm submitting a ... question about the decisions made in the repository / feature request.

• Summary

When using deriveHdPath to derive a node using a "relative" path, the prefix m or M must be included even when the source node is not a master(depth 0) node.

For example if I have a source node with absolute path m/1/2 and I want to reach a destination node with absolute path m/1/2/3/4, I must specify the relative path between them as m/3/4. I think it would be more intuitive to accept the relative path as /3/4 or 3/4 or ./3/4 as m should be reserved for nodes with depth 0 and m/3/4 should result in a node with depth 2.

It could also accept m/1/2/3/4 and then, observing that the starting node has depth 2 and index 2, drop the appropriate prefix from the derivation path and proceed from there.

  // Derive a master node:
  const masterNode = deriveHdPrivateNodeFromSeed(crypto, hexToBin("000102030405060708090a0b0c0d0e0f"));
  if (!masterNode.valid) return console.error(`Error with initial derivation.`);
  console.log('Master node depth:', masterNode.depth);  // 0
     
  // Derive a depth 2 node:
  const depth2Path = "m/1/2";
  const depth2Node = deriveHdPath(crypto, masterNode, depth2Path);
  if(typeof depth2Node === 'string') return console.error(depth2Node);
  console.log('Node depth:', depth2Node.depth); // 2
  
  // Derive a depth 4 node (absolute path):
  const depth4Path = "m/1/2/3/4";
  const depth4Node = deriveHdPath(crypto, depth2Node, depth4Path);
  if(typeof depth4Node === 'string') return console.error(depth4Node);
  console.log('Node depth:', depth4Node.depth); // 6 (expected 4)

  // Dervice a depth 4 node (relative path, absolute notation):
  const depth4PathRelativeAbs = "m/3/4";
  const depth4NodeRelativeAbs = deriveHdPath(crypto, depth2Node, depth4PathRelativeAbs);
  if(typeof depth4NodeRelativeAbs === 'string') return console.error(depth4NodeRelativeAbs); 
  console.log('Node depth:', depth4NodeRelativeAbs.depth); // 4

  // Derive a depth 4 node (relative path, currently generates an error):
  const depth4PathRelative = "/3/4";
  const depth4NodeRelative = deriveHdPath(crypto, depth2Node, depth4PathRelative);
  if(typeof depth4NodeRelative === 'string') return console.error(depth4NodeRelative); // Error
  console.log('Node depth:', depth4NodeRelative.depth); 

• Other information

See here for an executable demo of the code above.

[Jason Dreyzehner]

Thanks for opening an issue @rnbrady! You're definitely right, this was an oversight in the original API. I like your idea for denoting the "relative" derivation too.

Since it's not part of the BIP32 spec, I think using a unique character to denote relative derivation might be clearer, so I think I'd prefer the ./ prefix rather than just /.

One minor issue: we can't check that the provided node is actually the correct one for the designated part of the path:

  const masterNode = deriveHdPrivateNodeFromSeed(crypto, hexToBin("000102030405060708090a0b0c0d0e0f"));
  if (!masterNode.valid) return console.error(`Error with initial derivation.`);
  console.log('Master node depth:', masterNode.depth);  // 0
     
  const depth2Path = "m/1'/0";
  const depth2Node = deriveHdPath(crypto, masterNode, depth2Path);
  if(typeof depth2Node === 'string') return console.error(depth2Node);
  // depth2Node is "m/1'/0"
  
  const depth4Path = "m/2'/0/0/0";
  const depth4Node = deriveHdPath(crypto, depth2Node, depth4Path);
  if(typeof depth4Node === 'string') return console.error(depth4Node);
  // depth4Node is "m/1'/0/0/0" rather than "m/2'/0/0/0"

Though I think we can just note that prominently in the tsdocs.

It also looks like we're not validating depth to confirm it is less than or equal to the maximum of 255, so we should add a HdNodeDerivationError.depthExceedsMaximum too.

I'll try to get this in the next major release. 👍 (Happy to take a PR as well!)

[Richard Brady]

That is a good point re not being able to detect whether the path matches the node. The alternative would be to return an error if an absolute path is requested with a non-master starting node.

[Richard Brady]

Today I learned that by convention wallet apps which provide a xpub for export (e.g. Electron Cash, Bitcoin.com Wallet, Crescent) will provide the xpub for the account level node (depth 3). This makes sense as the first 3 levels are usually hardened (m/44'/145'/0') per BIP44 which makes the master xpub useless (these apps seems to support only one such "account", there is not way to add more other than by creating whole new HDWs at mnemonic / seed / master node level, which I found counterintuitive).

So it looks like the depth 3 import followed by relative path derivation would be a common use case. I'll ask around for thoughts on how devs would expect to handle.

[Jason Dreyzehner]

To summarize some out-of-band discussion: no matter how we design this API, some unintuitive-ness is going to remain due to choices in BIP32. (You can see the same excessive complexity in how HD keys are currently defined for the compiler.)

Most notably, developers often don't understand that a path like M/0'/0 (or M/0H/0) is fundamentally impossible according to the syntax defined by BIP32 – you cannot perform hardened derivation with a master public key, and the M indicates that the "master" node is "neutered" (the terminology used by BIP32 to indicate conversion from an extended private key to an extended public key).

In the BIP32 spec, this is clarified by wrapping the "neutered" node in N( ... ) to indicate at which step the conversion occurs, and derivation continues from that point using the public derivation algorithm. Though this is itself an odd choice: the purpose of the path syntax is to simplify the description of nested function calls with an easy-to-parse "pipe" syntax: derivePrivate(derivePrivate(derivePrivate(master, 0), 1), 2) becomes m/0/1/2. Instead of building in a syntax for abstracting the N(...) function too, BIP32 just places it directly in the path: derivePublic(derivePublic(neuter(derivePrivate(master, 0H)), 1), 2) becomes N(m/0H)/1/2. Note also the logical inconsistency here: the / character normally abbreviates derivePrivate([prefix], [suffix]), but if the N( ... ) is present in the path, that changes to derivePublic([prefix], [suffix]) when outside of the "neutered" segment of the path.

This all to say: I think these baked-in "gotchas" and inconsistencies might warrant defining a more consistent dialect of the BIP32 path syntax.

Some goals:

• all paths should be "supported" – no clever "gotchas" like M/0' not being possible (even though the user clearly wants neuter(derivePrivate(master, 0))).
• paths should indicate their own validation strategy – I really like @rnbrady's intuition that given the path m/1/2/3/4 and a node of depth 2, the method should know to simply finish the derivation with derivePrivate(derivePrivate(node, 3), 4). Likewise, if given a node which already has a depth of 4, deriveHdPath should know that something unexpected is happening and return an error (to avoid someone eventually losing money/keys). It would be great if our path syntax maximizes our ability to validate and return errors with unexpected inputs.

It might also be useful to choose a different prefix character (other than m or M) to make it extremely easy to differentiate our syntax from those trying to adhere more closely to BIP32.

---

One proposal, example:

• Algorithm: derivePublic(derivePublic(neuter(derivePrivate(m, 0H)), 1), 2))
• BIP32: N(m/0')/1/2
• Our absolute path: r/0'/p/1/2
  • uppercase/alphanumeric format: R/0H/P/1/2
• Our relative path from r/0': ./p/1/2
  • uppercase/alphanumeric format: ./P/1/2
• Our relative path from r/0'/p: ./1/2
• Our relative path from r/0'/p/1: ./2

"Absolute" paths – paths which define the path to a public or private key from a "level 0" node – should be prefixed with r/ instead of m/ or M/ (for "root" – we can switch our internal terminology to replace master with root in this codebase). There's no need to distinguish between "neutered" and non-"neutered" roots.

"Relative" paths would be prefixed with ./ (some other libraries already use this syntax, but since it's not standardized by BIP32, I don't think we need to worry about distinguishing our paths from those libraries).

Instead of codifying the "neutered" terminology with the character N, we use p to represent where the node is converted to "public". (This is also good to help differentiate ours from the BIP32 spec syntax.) We could also pick another verb like "convert", "public", "expose", "reveal", etc., but I haven't thought of anything that seems more intuitive than p for "public". (It also works logically as an abbreviation for "private" or "private-to-public". Either way, it's clearly the level at which the conversion happens and further derivations switch algorithms.)

By convention, both r and p should be lowercase, which improves legibility (over capitalized characters). Hardened derivation is represented with ' (as is already common, though this is not standardized in BIP32). This is also helpful for legibility vs H as used by the BIP32 syntax.

Because there are cases where capital characters are useful (e.g. QR code alphanumeric mode), we should also standardize an uppercase syntax, where we also replace ' with H (since ' can't be used), e.g. R/0H/P/1/2. Each path must use either the lowercase or uppercase formats, mixed case paths would return an error.

---

Thoughts?

With this path syntax, we'd be able to simplify hdPublicKeyDerivationPath, privateDerivationPath, and publicDerivationPath in bitauth template HD keys to just path, since each component can be easily derived.

[Jason Dreyzehner]

Thinking about it more: using the p character as a separator makes it hard to visually parse the "depth" of a path, so it's easy to confuse two paths as being the same depth:

• r/0/1/2
• r/0/p/2

Maybe a better option is to use a colon (:) to indicate the separation (and paths may have no more than one colon). So our example becomes:

• Algorithm: derivePublic(derivePublic(neuter(derivePrivate(m, 0H)), 1), 2))
• BIP32: N(m/0')/1/2
• Our absolute path: r/0':1/2
  • uppercase/alphanumeric format: R/0H:1/2
• Our relative path from r/0': ./:1/2
  • uppercase/alphanumeric format: (same)
• Our relative path from r/0':: ./1/2
• Our relative path from r/0':1: ./2

This makes depth much clearer (and reduces the visual noise of extra / characters):

• r/0/1/2
• r/0:2

Note, it's still possible to indicate in the path whether the final result should be the public or private node:

• private: r/0'/1'/2
• public: r/0'/1'/2:

(Also considered separating with /: rather than just :, but that seems to add unnecessary visual noise in most cases.)

[Jason Dreyzehner]

One remaining ambiguity with using : – should relative paths be prefixed with ./ even if that segment wouldn't otherwise have an initial /?

E.g. for absolute path r/0'/1':2/3, given the node at r/0'/1' (which is a private node), what is the relative path? Some options:

(This sort of became a stream of consciousness – we may have to settle on the exact syntax after trying some implementations.)

---

Option 1: always prefix with ./

This starts with ./, so many developers will intuitively recognize it as a relative path. But it also means that getting the relative path from the absolute path is more complicated than splitting the absolute path at the right character (we only add ./ if the suffix doesn't already have a /):

• Our absolute path: r/0'/1':2/3
• Our relative path from r/0': ./1':2/3
• Our relative path from r/0'/1': ./:2/3 (4 /s total)
• Our relative path from r/0'/1':: ./2/3 (4 /s total)
• Our relative path from r/0'/1':2: ./3

Option 2: only add the .

This doesn't require any logic to move from absolute to relative paths – the . stands in for the exact path of the source node. But it's much less clear to an unfamiliar developer that they're looking at a relative path:

• Our absolute path: r/0'/1':2/3
• Our relative path from r/0': ./1':2/3
• Our relative path from r/0'/1': .:2/3 (same 3 /s)
• Our relative path from r/0'/1':: .2/3 (same 3 /s)
• Our relative path from r/0'/1':2: ./3

Option 3: switch to /: as the separator

This makes relative paths start with ./ so they're clearly "relative", and it also makes splitting the absolute path a little clearer. Though it may be less clear which node has the "public-to-private" transformation applied:

• Our absolute path: r/0'/1'/:2/3
• Our relative path from r/0': ./1':2/3
• Our relative path from r/0'/1': ./:2/3
• Our relative path from r/0'/1'/:: ./2/3 (note: 5 /s here)
• Our relative path from r/0'/1'/:2: ./3

Option 4a: switch to :/ as the separator

Like option 2, relative paths wouldn't start with ./, so it's a little less clear. As a positive, the : is within the "segment" (between / ... /) of the node where the conversion happens, so you can think of it as another modifier like hardening (') for the node where it appears. E.g. r/0': takes a root private node, and derives hardened node 0, then derives its public node.

Applied:

• Algorithm: derivePublic(derivePublic(neuter(derivePrivate(m, 0H)), 1), 2))
• Our absolute path: r/0'/1':/2/3
• Our relative path from r/0': ./1':/2/3
• Our relative path from r/0'/1': .:/2/3 (strange .: prefix)
• Our relative path from r/0'/1':: ./2/3
• Our relative path from r/0'/1':/2: ./3

Option 4b: switch to :/ as the separator, require ./

This could also use the approach from option 1, where we just always require relative paths to start with ./ (so "absolute-to-relative" is a little more complex):

• Our absolute path: r/0'/1':/2/3
• Our relative path from r/0': ./1':/2/3
• Our relative path from r/0'/1': ./:/2/3 (adds ./, so 5 /s total)
• Our relative path from r/0'/1':: ./2/3
• Our relative path from r/0'/1':/2: ./3

This makes the implementation messier than option 4a though.

Option 5: abandon ./, just split the path

We could abandon trying to prefix relative paths with ./, and simply split the absolute path at the precise token where the path is divided:

• Our absolute path: r/0'/1':2/3
• Our relative path from r/0': /1':2/3
• Our relative path from r/0'/1': :2/3
• Our relative path from r/0'/1':: 2/3
• Our relative path from r/0'/1':2: /3

After fighting with the other options, this is looking very good 😅

[Jason Dreyzehner]

Maybe the best option: don't support relative paths. YAGNI

There are probably exceptionally few applications where it would be a good idea to even use a relative path. For the most part, explicitly-defined absolute paths are the best choice: they're easy to parse, extremely clear, and can offer built-in safeguards (like sanity-checking the depth of the provided node against the depth of the requested absolute path).

With an absolute path, you can also easily determine whether a public or private node is needed to complete the derivation. E.g. for the path r/0'/1':2/3, if we only have a root public node, derivation isn't possible. But if we have a public node at depth 2, deriveHdPath can trust that we've given it r/0'/1': and complete the final 2/3.

In the unusual case that the user wants a "relative path", it's probably safer for them to compute an absolute path from the relative components they expect to use, then provide that absolute path to our safer, absolute-paths-only derivation method (which will sanity-check their result and error if something unexpected happens).

[Richard Brady]

Wow, thanks for the great write-up. A few observations in response:

On the necessity of relative paths

Maybe the best option: don't support relative paths. YAGNI

I think there are some significant use cases where it is needed. For example imagine a business owner operates a BIP44 style HDW with the following accounts:

• m/44'/145'/0': Current account
• m/44'/145'/1': Retail store 1
• m/44'/145'/2': Retail store 2
• m/44'/145'/3': E-commerce site

The business owner would provide his webmaster and each store manager with a unique extended public key with depth 3. They would load those xpubs into their point-of-sale (POS) system or payment backend, which would then do a relative derivation ./0/i for each payment they take.

The combination of derivation path m/44'/145'/0' combined with exporting a depth 3 extended public key N(m/44'/145'/0') is common. I have personally used such a scheme (without knowing it at the time) at my meetups to take payments for beer and food. The Bitcoin Cash Register app runs on tablets operated by bar staff and generates addresses based on an imported xpub, which has been exported from the Bitcoin.com Wallet where I hold the private keys for the meetup's wallets. Having gone back to check the xpubs, they are depth 3. I've also confirmed that Electron Cash behaves the same way.

In the BIP32 spec this use case is referred to as Unsecure money receiver. There is also a Recurrent business-to-business transactions use case which would have similar requirements for relative paths.

The developer of such a POS system needs to do a relative derivation to generate the payment addresses. It seems strange to expect them to add an (arbitrary) prefix to form an absolute path, only so that the library can strip it back off again to perform the relative derivation.

Some opinions on the options presented

I don't see the benefit of introducing a new prefix, especially now that I understand the meaning of M properly. I found r to be ambiguous as it could stand for either root or relative.

I don't see the benefit of adding a positional neuter symbol such as : or p, because:

• Its presence anywhere to the left of any ' will generate an error.
• Its presence anywhere else indicates a neutered node as the final result.
• Its position makes no difference to that result.

Having considered all of this, the scheme that now makes most sense to me is:

• keep everything as is (private node implies private derivation),
• add support for relative paths (I prefer Option 5, i.e. no introduction of new characters),
• a developer wanting a neutered result should explicitly call deriveHdPublicNode rather than relying on a symbol in the path.

Unfortunately my understanding of BitAuth templates and compilers is minimal so I wasn't able to take their requirements into consideration.

[Jason Dreyzehner]

Ya, after this discussion, I think you're right – we should probably leave the API mostly as-is, and just allow support for "relative" paths without any new character scheme, e.g. 2/3. Public nodes produce public nodes, private nodes produce private nodes, and I'll have to make sure that the final documentation introduces the concepts clearly enough to make the API understandable. 👍

For future completeness of this discussion, this BIP proposal now exists too: BIP32 Path Templates. Though I don't think Libauth needs to support it at all – it's easy to for a downstream library to create an implementation with the primitives provided by Libauth's BIP32 support.