Standard URI for ipfs and ipns protocols (Discussion)
larsks opened this issue · comments
I would like to add ipfs support to a tool that expects a URL-format specification. Hypothetically, let's say I wanted to add ipfs suport to curl. I would need a scheme:data format specification that follows the standard url format.
I asked about this on irc and immediately folks started trying to direct me away from URLs to the multiaddr spec. Setting aside for the moment then I'm not clear what problem multiaddr is trying to solve or why URLs aren't appropriate, some tools will simply require a URL format to operate.
In the absence of any other suggestions, I would like to suggest that we document the following standard forms:
-
ipfs:<hash>[/<path>]for IPFS objects, as in:ipfs:QmPXME1oRtoT627YKaDPDQ3PwA8tdP9rWuAAweLzqSwAWT/readme -
ipns:<hash>[/path]for IPNS names:ipns:QmXfrS3pHerg44zzK6QKQj6JDk8H6cMtQS7pdXbohwNQfK/pages/gpg.md
I strongly agree that IPFS objects should be identifiable by URI, mostly because of uniformity as described by RFC 3986:
Uniformity provides several benefits. It allows different types
of resource identifiers to be used in the same context, even when
the mechanisms used to access those resources may differ. It
allows uniform semantic interpretation of common syntactic
conventions across different types of resource identifiers. It
allows introduction of new types of resource identifiers without
interfering with the way that existing identifiers are used. It
allows the identifiers to be reused in many different contexts,
thus permitting new applications or protocols to leverage a pre-
existing, large, and widely used set of resource identifiers.
I don't care if go-ipfs uses URIs internally, or if browsers will support it, or anything like that – there should be a canonical, standard way to refer to IPFS objects using URIs.
The above suggestion seems entirely reasonable to me:
ipfs:QmPXME1oRtoT627YKaDPDQ3PwA8tdP9rWuAAweLzqSwAWT/readme
I could also see an argument for IPFS identifiers being URNs per RFC 2141:
urn:ipfs:QmPXME1oRtoT627YKaDPDQ3PwA8tdP9rWuAAweLzqSwAWT/readme
IPNS is separate. I could see an argument for it being a separate scheme, as proposed above:
ipns:QmXfrS3pHerg44zzK6QKQj6JDk8H6cMtQS7pdXbohwNQfK/pages/gpg.md
On the other hand, I could see the IPNS public key hash being a naming authority per RFC 3986.
Many URI schemes include a hierarchical element for a naming
authority so that governance of the name space defined by the
remainder of the URI is delegated to that authority (which may, in
turn, delegate it further). The generic syntax provides a common
means for distinguishing an authority based on a registered name or
server address, along with optional port and user information.
Authorities are preceded by a double slash, so:
ipns://QmXfrS3pHerg44zzK6QKQj6JDk8H6cMtQS7pdXbohwNQfK/pages/gpg.md
…and if IPNS public key hashes are interpreted as an authority, distinct from the global (no-authority) IPFS paths, IPFS and IPNS could be viewed as two halves of one scheme:
ipfs:QmPXME1oRtoT627YKaDPDQ3PwA8tdP9rWuAAweLzqSwAWT/readme
ipfs://QmXfrS3pHerg44zzK6QKQj6JDk8H6cMtQS7pdXbohwNQfK/pages/gpg.md
@willglynn, thanks for your comments.
While working with this in practice, I realized that one may want to provide IPFS gateway information as part of the URL. Again, using the hypothetical example of adding IPFS support to something like curl, I need a way to tell the utility which IPFS endpoint to use. If I'm not running one locally, the utility needs to know where to find an API to fetch ipfs:QmPXME1oRtoT627YKaDPDQ3PwA8tdP9rWuAAweLzqSwAWT/readme.
Should this information always be specified external to the URL (e.g., configuration options to the tool)? Or does permitting this in the URL make sense? That would give us something like:
ipfs://<host>:<port>/QmPXME1oRtoT627YKaDPDQ3PwA8tdP9rWuAAweLzqSwAWT/readme
Where <host> and <port> specify an endpoint, and can be omitted, making the typical URL look like:
ipfs:///QmPXME1oRtoT627YKaDPDQ3PwA8tdP9rWuAAweLzqSwAWT/readme
In my view, IPFS gateway information is external to the URI. I think it's analogous to HTTP/FTP/SOCKS proxy information for which tools like curl are configured using separate parameters or environment variables. An IPFS object should always have the same identifier regardless of how it is accessed, and especially in the case of IPFS, I think identity is more fundamental than location.
Yeah, that was mostly my inclination as well. Externally configured it is, then.
The more I think about it, the more I favor treating IPNS names as a URI authority. Consider:
ip[nf]s://multihash/object
Retrieval would be processed as "resolve IPNS multihash, retrieve object".
http://domain.name/object
Retrieval would be processed as "DNS look up domain.name IN A, connect to resulting IP, HTTP GET /object".
ip[nf]s://<domain.name>/object
Retrieval would be processed as "DNS look up domain.name IN TXT, resolve resulting IPFS/IPNS identifier, retrieve object".
I feel this quote belongs here :-)
I want to remind everyone here that we're not actually limited by any rules. It is of course convenient and nice to work productively with everybody else, but there are certain mistakes we should not continue to make.
I'll give you an example of a break from tradition (or rather... a return to even older tradition). it is a strong goal to mend the rift between UNIX and the Web. That is, "ipfs links" should be exactly the same in both the Web, and UNIX. meaning:
/ipfs/<hash>/<path>, NOTipfs://<hash>/<path>(explicitly disobeying the scheme that the W3C insists on). Luckily for us, this is technically feasible, though it does have its bumps to work around. (and easiest done with a TLD :) ). That's fine for us as the upside of mending part of this awful rift is worth a lot.
So the canonical way of representing IPFS address is /ipfs/<hash>/<path> (at least that was the consensus in April, maybe it changed since then? :).
That being said, IMHO it would not hurt to additionally provide a silent support of URLs like ipfs://<hash>/<path> for interoperability with legacy software that can't use /ipfs/<hash>/<path> natively.
@lidel nothing has changed since that quote :)
Indeed; IPFS can choose whatever syntax it wants for canonical use. I'm suggesting that there be an official, standardized way of encoding IPFS identifiers into URIs, since it is better to interoperate with URI-centric tools than not, and since it's better to have one way of encoding IPFS identifiers into URIs than multiple incompatible methods.
I would favor ipfs:/ipfs/<hash>/<path> over ipfs://. RFC 3986's URI grammar would permit ipfs:object, ipfs:/object, and ipfs://object, but the first two constructs are interpreted as paths, while the third would be interpreted as an authority with no path. Per RFC 2718:
2.1.2 Improper use of "//" following "<scheme>:"
Contrary to some examples set in past years, the use of double
slashes as the first component of the <scheme-specific-part> of a URL
is not simply an artistic indicator that what follows is a URL:
Double slashes are used ONLY when the syntax of the URL's <scheme-
specific-part> contains a hierarchical structure as described in RFC
2396. In URLs from such schemes, the use of double slashes indicates
that what follows is the top hierarchical element for a naming
authority. (See section 3 of RFC 2396 for more details.) URL
schemes which do not contain a conformant hierarchical structure in
their <scheme-specific-part> should not use double slashes following
the "<scheme>:" string.
If the IPFS namespace is one big path hierarchy, then mapping IPFS / to URI ipfs:/ seems appropriate, and conversion to/from URIs is just* a matter of a five-character prefix.
Ok, so as long as this discussion is about interoperability layer,
I like the idea of going with a single prefix for all IPFS resources (dropping ipns://).
Starting with <hash> IPFS is hierarchical, so perhaps we should go with
ipfs://ipfs/<hash>/<path>(explicit/ipfs/)ipfs://ipns/<hash>/<path>
Additionally we could agree to default to ipfs resource if the first segment does not match ipfs nor ipns:
ipfs://<hash>/<path>(implicit/ipfs/)
What are your thoughts on this?
A goal from @jbenet's post:
please don't do this. please please please have identifiers exactly how we have them, everywhere. Simply
/ipfs/...and/ipns/....
If this is the objective, then the corresponding URIs should be ipfs:/ipfs/… and ipfs:/ipns/…. URI-centric tools would behave properly with both absolute and relative paths of the above canonical (scheme-less) IPFS form. A resource retrieved from the URI ipfs:/ipfs/hash/A referring to an absolute /ipfs/hash/B would be understood as ipfs:/ipfs/hash/B, which is consistent. Such a resource referring to a relative B would be understood to be ipfs:/ipfs/hash/B, which is also consistent.
Notably, these same references – /ipfs/hash/B and B – would work unchanged whether the underlying resource appears at the filesystem's /ipfs/hash/A, at http://gateway.ipfs.io/ipfs/hash/A, at http://localhost:8080/ipfs/hash/A, or in URI-space at ipfs:/ipfs/hash/A.
Using double-slashes (//) would sacrifice this property, since a resource at ipfs://hash/A referring to /ipfs/hash/B would resolve to ipfs://hash/ipfs/hash/B.
Makes sense. This way a rule for legacy layer would be super simple: always add ipfs: as a prefix to the canonical name. That is all.
Great! Sounds like a workable direction.
There are a couple other possible pain points I can think of when mapping IPFS paths <-> URIs. Mostly, URIs are specified to be a particular subset of US-ASCII and certain characters have specific meanings, so unless IPFS conventions happen to be compatible, we'll need to use percent-encoding to bridge the gap. The specifics depend on some gory details:
- What characters are permitted in IPFS path segments? UNIX is arguably way too permissive, Windows reserves far more characters plus certain entire filenames, and various other systems are somewhere in the middle. (Edit: am I reading this right? Any byte sequence in a
stringis a legal path, provided it starts with e.g./ipfs/<base58 hash>/?) - Do IPFS paths use a specific character encoding, and if so, which? UNIX doesn't, while Windows and OSX do; potentially ill-formed UTF-16 (prompting creation of WTF-8) and a particular normalization of UTF-8 respectively. (Edit: Links seem to be UTF-8. Is this enforced? Where and how?)
- If IPFS paths use a specific encoding, which component(s) are aware of this? Do any components perform any Unicode normalization?
- Is the concept of a query string (
?foo) meaningful to a resource addressable at an IPFS URI? (I think probably no.) - Is the concept of a fragment (
#foo) meaningful to a resource addressable at an IPFS URI? (I think probably yes.)
I have a proposal for a content addressing URI scheme here: https://bentrask.com/notes/content-addressing.html. I traded emails with Juan Benet about it a year ago when I first wrote it. Obviously I couldn't convince him.
RFC 6920 proposes a different but similar scheme.
It would be nice if the URI scheme was common between different projects (IPFS, Camlistore, my own). A single content address could be resolved over various different systems depending on context.
Edit: I compared four different proposals here.
Sorry, I'm late to the party.
Thanks very much @lidel for representing my viewpoint :)
I'm going to try responding only to things i think are unresolved. ask again if i missed something
ipfs://<hash>/<path>
I think @lidel elucidated my viewpoint excellently, and i do not need to express again that this is not desired becuase it complicates things, and forces us to add a 2+ protocol identifiers.
In my view, IPFS gateway information is external to the URI. I think it's analogous to HTTP/FTP/SOCKS proxy information for which tools like curl are configured using separate parameters or environment variables. An IPFS object should always have the same identifier regardless of how it is accessed, and especially in the case of IPFS, I think identity is more fundamental than location.
Exactly right. URLs on the HTTP gateways ARE NOT IPFS Paths/URIs (they contain one, in a larget HTTP URL).
would favor ipfs:/ipfs// over ipfs://. RFC 3986's URI grammar would permit ipfs:object, ipfs:/object, and ipfs://object, but the first two constructs are interpreted as paths, while the third would be interpreted as an authority with no path. Per RFC 2718:
Hmmmm, i'm not sure. I understand the spec... it may be ok to consider /ipfs and /ipns as naming authorities for the purposes of UX. Most people who ever see/use URLs always see them in http://. I think we should support all :, :/, ://, but actually redirect all to :// as the canonical one.
Makes sense. This way a rule for legacy layer would be super simple: always add ipfs: as a prefix to the canonical name. That is all.
I like this, maybe we make :/ canonical? it just looks so odd. and people will be weirded out by it... (we must support :// at least)
What characters are permitted in IPFS path segments? UNIX is arguably way too permissive, Windows reserves far more characters plus certain entire filenames, and various other systems are somewhere in the middle. (Edit: am I reading this right? Any byte sequence in a string is a legal path, provided it starts with e.g. /ipfs//?)
Do IPFS paths use a specific character encoding, and if so, which? UNIX doesn't, while Windows and OSX do; potentially ill-formed UTF-16 (prompting creation of WTF-8) and a particular normalization of UTF-8 respectively. (Edit: Links seem to be UTF-8. Is this enforced? Where and how?)
Yes, the paths are supposed to be UTF-8 strings. We should be enforcing it, though i dont think it's being enforced atm.
Is the concept of a query string (?foo) meaningful to a resource addressable at an IPFS URI? (I think probably no.)
Not at this time, though it may become relevant.
Is the concept of a fragment (#foo) meaningful to a resource addressable at an IPFS URI? (I think probably yes.)
yes.
I have a proposal for a content addressing URI scheme here: https://bentrask.com/notes/content-addressing.html. I traded emails with Juan Benet about it a year ago when I first wrote it. Obviously I couldn't convince him.
Sorry @btrask :/ -- i just disagree :)
Edit: I compared four different proposals here.
you didn't compare the proper IPFS URIs, which are paths:
/ipfs/QmeeQhGoyMQc7eQWERE88kFFq4WbdVRrjHctZhH1hPHNds/006/mdag.waist.png
/ipns/QmfVrBzjaXjWWfC7UhFrZnnFFMYA1ENPCjzxAtREaQz8MS/006/mdag.waist.png
/ipns/ipfs.io/docs/install
These are all valid in IPFS. Soon we should also have:
/dns/ipfs.io/docs/install
The first component is the protocol "scheme".
One remaining thing to address: the protocol scheme
I've known for some time now that we're going to need to have + support a protocol scheme identifier, for all the use cases that absolutely require one. _Instead of using only ipfs: for everything, I'm planning to use something that's valid for the entire "Unix Web", that is, a suite of protocols that want to work both on the web and on unix and want the "same identifier" niceness.
I think we should use one of:
unixweb:
nixweb:
nix:
x:
As in:
x:/ipfs/QmeeQhGoyMQc7eQWERE88kFFq4WbdVRrjHctZhH1hPHNds/006/mdag.waist.png
x:/ipns/QmfVrBzjaXjWWfC7UhFrZnnFFMYA1ENPCjzxAtREaQz8MS/006/mdag.waist.png
x:/ipns/ipfs.io/docs/install
x:/dns/ipfs.io/docs/install
x:/bitcoin/<bitcoin-txn>
x:/bittorrent/<magnet-hash>
but happy to hear more suggestions. I know it's rude to use a one-letter schme identifier... but hey... nobody else is using it.
I understand Juan. Sorry for the misleading comparison.
If IPFS addresses are paths, what would you think about simply using file:// URLs?
file:///ipfs/QmeeQhGoyMQc7eQWERE88kFFq4WbdVRrjHctZhH1hPHNds/006/mdag.waist.png
file:///ipns/QmfVrBzjaXjWWfC7UhFrZnnFFMYA1ENPCjzxAtREaQz8MS/006/mdag.waist.png
file:///ipns/ipfs.io/docs/install
From my experience file:// proved to be problematic:
👍 Yes, it makes IPFS work out-of-the box with legacy software (as long as you have IPFS filesystem mounted via FUSE driver provided by go-ipfs).
👎 ..but if you don't have root access and/or can't set up FUSE -- bad luck
👎 if you use MS Windows or other non-unix system -- bad luck
👎 a lot of confusion due to "File not found" errors.
IMO file:// should be left as a workaround for people who can set up FUSE on local system and we should come up with a new protocol scheme for canonical use.
As a minimalist I really like the x: scheme described by @jbenet :-)
The x:/ipfs/... scheme might be similar enough to a Windows file path (e.g. "drive X", which many people have) that it would be misinterpreted by browsers and auto-converted to file:///X:/ipfs/.... Can someone on Windows check the behavior of browsers when you type that into the URL bar or use it as a link href?
Windows is supposed to use backslashes but forward slashes are often accepted and silently corrected to backslashes.
@gatesvp or someone else using windows, could you please check what happens with x:// above?
Just tested on Windows, x:/ becomes file:///x:/ (I tried it in the URL bar and as a link href in Chrome). However, strings longer than one character (xx:/) are kept as a protocol.
@mappum is that the case even if you install a protocol handler for x:// ?
is that the case even if you install a protocol handler for x:// ?
Yes, just tried adding a protocol handler to the registry for x:, the URL was still transformed.
I'm using both ?foo#bar for the ia book reader, so think they should both be supported :)
also 👍 for minimalism
Hm.. xx: is not bad, but how about xn: ? (uniXNamespace)
may be worth doing:
nix://
nixweb://
wish unix:// wasn't taken by unix sockets.
I'd go with
nix://
I like this, maybe we make
:/canonical? it just looks so odd. and people will be weirded out by it... (we must support://at least)
nix:/ipfs/base58/resource parses as { scheme: "nix", authority: null, path: "/ipfs/base58/resource" }. A single slash denotes an absent authority followed by an absolute path. I think this matches the intent of IPFS (a single global namespace using absolute paths assigned by no central authority), which is why I suggest it as the canonical form.
nix://ipfs/base58/resource parses { scheme: "nix", authority: "ipfs", path: "/base58/resource" }. This breaks IPFS absolute paths since the first path component has moved into the authority part of the URI. I think that makes this a non-starter.
nix:///ipfs/base58/resource parses as { scheme: "nix", authority: "", path: "/ipfs/base58/resource" }. Triple slashes denote an empty authority followed by an absolute path, which is equivalent enough to the no-authority URI that it's not wrong. :/// should be supported, either in addition to or in lieu of :/. Note also that at least one library conflates these two address forms.
Yesterday we had thought of using get://, but that's not good for non-read functionality. writes, etc.
some more:
nix:// nixweb:// unixweb:// uweb:// dweb:// path:// endpoint:// ep:// unixpath:// up:// fp:// web3://
nix://ipfs/base58/resourceparses{ scheme: "nix", authority: "ipfs", path: "/base58/resource" }. This breaks IPFS absolute paths since the first path component has moved into the authority part of the URI. I think that makes this a non-starter.
I dont think it does, the browser tools could undo that change for the user.
my problem with :, :/, and :/// is that it's not what 90% of users will expect to see. regular users have no idea what the hell all of these are for, but hey do know http:// and that's what they're going to type. so we have to support it regardless.
a worry with nix:// is that it means
- noun 1. nothing.
- "apart from that, nix"
- exclamation 1. expressing denial or refusal.
- "“I owe you some money.” “Nix, nix.”"
- verb (NORTH AMERICAN) 1. put an end to; cancel.
- "he nixed the deal just before it was to be signed"
which is not ideal. this is the sort of thing 99% of internet users will be confused by, so it should be as clear as we can make it
Attaching IPFS / to :// does break references.
Picture a /ipfs/base58/document referring to related-resource, /ipfs/otherbase58/linked-document, and /ipns/domain.name/. If IPFS were mounted to a UNIX filesystem, these resolve as:
related-resource=>/ipfs/base58/document/ipfs/otherbase58/linked-document=>/ipfs/otherbase58/linked-document/ipns/domain.name/=>/ipns/domain.name/
These work as expected – i.e. as paths.
If that same document were retrieved from http://gateway.ipfs.io/ipfs/base58/document, those references become:
related-resource=>http://gateway.ipfs.io/ipfs/base58/related-resource/ipfs/otherbase58/linked-document=>http://gateway.ipfs.io/ipfs/otherbase58/linked-document/ipns/domain.name/=>http://gateway.ipfs.io/ipns/domain.name/
These work as expected because gateway.ipfs.io is treated as an authority and the IPFS paths are treated as URI paths.
If that same document were retrieved from foo:/ipfs/base58/document or foo:///ipfs/base58/document, those references become:
related-resource=>foo:/orfoo:///ipfs/base58/related-resource/ipfs/otherbase58/linked-document=>foo:/orfoo:///ipfs/otherbase58/linked-document/ipns/domain.name/=>foo:/orfoo:///ipns/domain.name/
Again these work because the authority is constant, and the IPFS paths are mapped to URI paths.
If that same document were retrieved from foo://ipfs/base58/document those references become:
related-resource=>foo://ipfs/base58/related-resource/ipfs/otherbase58/linked-document=>foo://ipfs/ipfs/otherbase58/linked-document(notefoo://ipfs/ipfs)/ipns/domain.name/=>foo://ipfs/ipns/domain.name/(notefoo://ipfs/ipns)
Please do not make foo://ipfs/base58/document the canonical IPFS URI format.
Can this be worked around client-side? Yes, but there are many more clients that assume paths are paths than there are IPFS implementations that would assume something different. I don't want to patch wget and curl and Heritrix and Scrapy and every other tool I use that follows links to have special awareness of IPFS paths just because users are used to typing foo://bar into a browser window.
the browser tools could undo that change for the user
If users typing in foo://bar is an important enough use case to suggest browser tools dedicated to fixing it, then those browser tools should redirect foo://bar to foo:/bar or foo:///bar instead, rather than trying to support base URIs of foo://bar.
Please do not make foo://ipfs/base58/document the canonical IPFS URI format.
we have to make this work. it's not an option. 99% of people on the internet will try it. I believe that we can teach the browser's nix protocol resolver how to make it work. it may be hacky, but it will prevent massive confusion. (try explaining to your grandmother why foo: and foo:/ and foo:/// work but not foo://, which is, coincidentally, everything she may be used to).
I will add that i understand your post well. im saying we have to work around the limitations.
don't want to patch wget and curl and Heritrix and Scrapy and every other tool I use that follows links to have special awareness of IPFS paths
None of these know what to do with an ipfs path at all. they will have to be modified, or delegate to the OS protocol handler, which is precisely where we can put a workaround.
I'm in favor of modifying those tools to be aware of IPFS as a URI scheme. I'm against making tools aware that an apparent reference to foo://ipfs/ipns/domain.name needs to be understood as foo://ipns/domain.name – along with every other special case that needs to be introduced if IPFS / is foo:// – and I'm against defining a URI scheme that requires second-guessing.
Are users really going to enter IPFS URIs by hand, base58 hashes and all? If not, what are we actually trying to support? ipfs://ipns/domain.name/?
people may copy paste "/ipfs/<hash>/..." and add nix:// beforehand.
and yes, nix://ipns/domain.name
Suggestion 1: scheme separation. Make a separate URL scheme for IPNS: ipns://domain.name/ (with optional slashes, so that even ipns:domain.name is also supported). It simplifies the form of the following suggestions.
Suggestion 2: the shortest default. Drop the (now) reduntant part from ipfs://ipfs/… and thus make ipfs:hash or ipfs:hash/somePath the default URL scheme for IPFS files and directories. (Here “the default” means that this scheme must be used in “Copy this document's URL”, “Copy Image Location” and other similar places.) Also support a double-slashed version ipfs://hash or ipfs://hash/path.
Suggestion 3: the lame browsers' crutch. While not making it default, also support the ipfs://ipfs/ipfs/hash / ipfs://ipfs/ipfs/hash/path form. (Here ipfs appears thrice: as a protocol, as a fake hostname and as the first part of the path.) Pronounce that if an IPFS browser does not use its own (“true IPFS”) converter of relative URLs (in IPFS documents) to absolute IPFS URLs, it must use this (highly redundant) form as the document's address (base URL for its relative URLs). For the same reason, ipfs://ipfs/ipns/domain.name URLs should open ipns:domain.name resources.
Suggestion 4: the IPFS→Web gates. Web sites that display IPFS resources should use http://example.org/ipfs/hash or http://example.org/ipfs/hash/path form.
Expected results. From ipfs:hash/dir/doc the possible relative URLs will lead to:
| URL's medium | base URL | relative URL | resulting absolute URL |
|---|---|---|---|
| FUSE | /ipfs/hash/dir/doc | aFile | /ipfs/hash/dir/aFile |
| FUSE | /ipfs/hash/dir/doc | /ipfs/otherHash/img | /ipfs/otherHash/img |
| FUSE | /ipfs/hash/dir/doc | /ipns/domain.name | /ipns/domain.name |
| gate | http://ipfs.io/ipfs/hash/dir/doc | aFile | http://ipfs.io/ipfs/hash/dir/aFile |
| gate | http://ipfs.io/ipfs/hash/dir/doc | /ipfs/otherHash/img | http://ipfs.io/ipfs/otherHash/img |
| gate | http://ipfs.io/ipfs/hash/dir/doc | /ipns/domain.name | http://ipfs.io/ipns/domain.name |
| lame browser | ipfs://ipfs/ipfs/hash/dir/doc | aFile | ipfs://ipfs/ipfs/hash/dir/aFile |
| lame browser | ipfs://ipfs/ipfs/hash/dir/doc | /ipfs/otherHash/img | ipfs://ipfs/ipfs/otherHash/img |
| lame browser | ipfs://ipfs/ipfs/hash/dir/doc | /ipns/domain.name | ipfs://ipfs/ipns/domain.name |
| true IPFS browser | ipfs:hash/dir/doc | aFile | ipfs:hash/dir/aFile |
| true IPFS browser | ipfs:hash/dir/doc | /ipfs/otherHash/img | ipfs:otherHash/img |
| true IPFS browser | ipfs:hash/dir/doc | /ipns/domain.name | ipns:domain.name |
You may introduce nix: later to get the following:
| URL's medium | base URL | relative URL | resulting absolute URL |
|---|---|---|---|
| lame browser | nix://nix/ipfs/hash/dir/doc | aFile | nix://nix/ipfs/hash/dir/aFile |
| lame browser | nix://nix/ipfs/hash/dir/doc | /ipfs/otherHash/img | nix://nix/ipfs/otherHash/img |
| lame browser | nix://nix/ipfs/hash/dir/doc | /ipns/domain.name | nix://nix/ipns/domain.name |
| true IPFS browser | nix:ipfs/hash/dir/doc | aFile | nix:ipfs/hash/dir/aFile |
| true IPFS browser | nix:ipfs/hash/dir/doc | /ipfs/otherHash/img | nix:ipfs/otherHash/img |
| true IPFS browser | nix:ipfs/hash/dir/doc | /ipns/domain.name | nix:ipns/domain.name |
Cons:
-
URLs are longer.
-
A catch-all URL handler has to be an umbrella-type handler, i.e. the new URL subtype handlers has to be registered within it (using some new and unknown API) instead of being registered in the browser/OS as top-level URL schemes (using some old and well-known API).
Be aware of the OS and package manager already named "Nix" which has been growing in popularity lately. Might not matter. http://nixos.org/
Yesterday we had thought of using get://, but that's not good for non-read functionality. writes, etc.
@jbenet I didn't realise the (legacy) URI format needed to support writes? How would that work? I thought the main purpose was to allow people to view files in current browsers, and writes would require a proper library like ipfs.js, which would presumably use paths rather than URIs?
@davidar imagine an HTTP POST to an IPNS name, or something. it may be able to work. and this muxing handler is meant to be more general than IPFS, so it may include other protocols with write, etc.
@Mithgol the one thing you're missing in your sugesstions is that the whole point is to always include the full canonical ipfs path. Always include all of
/ipfs/<hash>/<path>
/ipns/<name>/<path>
including starting / and protocol identifier (ipfs, ipns). the point is to preserve unix compativility with the exact same identifiers so that humans can trivially easily copy and paste without edit. (prefix additions being the only allowable crutch)
@jbenet Then, with prefix additions being the only allowable crutch (and unix compatibility being more preferred than shorter URLs), I agree with @davidar that the two-letter URL scheme names are the best. (Because, as @mappum has discovered, schemes can't be any shorter than that.)
However, instead of at:// or to:// (which are good, but too general), I suggest fs: to indicate that a new filesystem type is accessed. (Double-slashed fs:// should also be supported, but not as the default. It should be usual for only a single slash, starting a Unix path, to follow fs:.)
The above table of URLs in browsers would then take the following form:
| URL's medium | base URL | relative URL | resulting absolute URL |
|---|---|---|---|
| lame browser | fs://fs/ipfs/hash/dir/doc | aFile | fs://fs/ipfs/hash/dir/aFile |
| lame browser | fs://fs/ipfs/hash/dir/doc | /ipfs/otherHash/img | fs://fs/ipfs/otherHash/img |
| lame browser | fs://fs/ipfs/hash/dir/doc | /ipns/domain.name | fs://fs/ipns/domain.name |
| true IPFS browser | fs:/ipfs/hash/dir/doc | aFile | fs:/ipfs/hash/dir/aFile |
| true IPFS browser | fs:/ipfs/hash/dir/doc | /ipfs/otherHash/img | fs:/ipfs/otherHash/img |
| true IPFS browser | fs:/ipfs/hash/dir/doc | /ipns/domain.name | fs:/ipns/domain.name |
The latter URLs become 4 character longer than in my first suggestion (which was designed for URL shortness):
fs:/ipfs/hashsuch asfs:/ipfs/Qme3pKkedCfcebDmkZz5tU2suMjHP4EPi29NMNNxQrQzwyare 55 characters long,- and, as always, in the
fs:/ipfs/hash/pathform the path's length is added.
Well, four characters are not much.
Great work! i really like all of these:
at:
to:
ref:
fs:
i think they're all great. form a cs standpoint, i like fs:// most, but from watching humans dictate urls, i like at: ref: to:. (at and to, have pronunciation confusion with @ and 2, but it may be ok).
From a cs standpoint, i like fs:// most, but from watching humans dictate urls, i like at: ref: to:
That was my thought too. HTTP is an awful scheme name, something that can actually be pronounced easily (and conjures roughly the right idea in nontechnical people) would be much better.
BTW, to me nix:// looks like it refers to https://nixos.org/
Which uses a special /nix filesystem to save packages, and would combine
with ipfs very nicely.
On Tue, Sep 15, 2015 at 2:58 AM, David A Roberts notifications@github.com
wrote:
From a cs standpoint, i like fs:// most, but from watching humans dictate
urls, i like at: ref: to:That was my thought too. HTTP is an awful scheme name, something that can
actually be pronounced easily (and conjures roughly the right idea in
nontechnical people) would be much better.—
Reply to this email directly or view it on GitHub
#1678 (comment).
@wscott agreed, nix:// would likely be confusing
I would suggest that any URL scheme other than ipfs: would be extremely confusing and more likely to lead to conflicts down the road. Why are we resisting using the protocol name as the url prefix, which is the bog standard behavior for just about everything else (like http: and ftp: and rtsp: and imap:...)?
If we are going to defy the standard, lets do it elegantly:
I propose
://
Lars, we have multiple protocols, and we want to support more. Read the full thread for more.
—
Sent from Mailbox
On Tue, Sep 15, 2015 at 11:08 AM, BrendanBenshoof
notifications@github.com wrote:
If we are going to defy the standard, lets do it elegantly:
I propose://
Reply to this email directly or view it on GitHub:
#1678 (comment)
@jbenet I saw your post above on that topic, but I don't see any examples of what you mean. What do you mean by "multiple protocols"? The suggestions I've seen so far (nix:, x:, etc) seem destined to cause confusion.
There are examples out there of existing URL schemes for protocol combinations (like qemu+tcp:// vs qemu+ssh:// for libvirt connections, or git+http:// or git+ssh:// for providing pip with information about a git repository...), but I don't know if any of these would be appropriate since it's not clear what your goal is.
i dont have time to keep repeating myself. my goal is clear enough: #1678 (comment) i don't want to be rude and not answer you, so if i ask to read up, please try once more, or look elsewhere for the same information. if i keep answering the same questions over and over, i can't push forward in other pressing things.
The goal: build procotols which bridge unix and the web, by using only path identifiers.
the w3c-recommended protocol scheme discussed here (<something>:) is a workaround for all protocols which share this goal. (/ipfs, /ipns, /dns, /ip4, /ip6, /tcp, /udp for starters, but i have many more in mind: /bittorrent, /bitcoin, /9p ...).
qemu+tcp:// vs qemu+ssh:// for libvirt connections, or git+http:// or git+ssh://
this is absurd and needs to stop. it doesn't layer well at all. it's a hack around the inadequacies of the scheme identifier, and we're not going to be limited by it. anyway, i don't want to discuss again these inadequacies, i want to finalize the workaround. i will give a talk soon enough about all this and the reasoning behind this break. if you'd like me to bring it sooner, open an issue on https://github.com/ipfs/notes/
see also jbenet/random-ideas#28
I agree with @larsks; it would be surprising to call the scheme anything besides ipfs if the only meaningful paths are defined in the IPFS spec.
RFC 7595 offers some guidance, including:
3.8. Scheme Name Considerations
…
Schemes SHOULD NOT use names that are either very general purpose or
associated in the community with some other application or protocol.
Schemes also SHOULD NOT use names that are overly general or
grandiose in scope (e.g., that allude to their "universal" or
"standard" nature).
A scheme name is not a "protocol." However, like a service name as
defined in Section 5 of [RFC6335], it often identifies a particular
protocol or application. If a scheme name has a one-to-one
correspondence with a service name, then the names SHOULD be the
same.
4. Guidelines for Provisional URI Scheme Registration
…
o If no permanent, citable specification for the scheme definition
is included, credible reasons for not providing it SHOULD be
given.
It sounds like there is a master plan to unify many things into unixweb, and I look forward to reading such a proposal. Until then, I think we should narrow the scope of this issue and hammer out a common way to refer to IPFS objects using URIs.
@willglynn sorry to disappoint. We're doing it. I'm don't mean to be a jerk, this has been the plan for a year or more (we just hadn't settled on a meta-protocol prefix). This has gotten lots of discussion. you're welcome to read up on it on the various fora (irc logs, mails, github, etc), but we're following this path. If you want to help us with defining the "unixweb" spec, you're very welcome to. But we're not going to "not do it" just because a few new people haven't caught up. hope that makes sense?
@Mithgol @davidar I think fs:// makes the most sense. it's actually very clear in pronunciation (at/to are ambiguous), and it is close enough to point out this is meant to be a meta-scheme that just accesses filesystem-path-based protocols. (ref is nice too, but it's not clear to the protocol)
@jbenet I'm not meaning to be a jerk either, but neither am I sufficiently interested in the unixweb concept that I'll dig through a year of transcripts to learn more, and you haven't pointed me to anything more concrete, so I'm left asking questions.
Why does the potential future development of unixweb mean we can't have a URI scheme specifically for IPFS? The examples you've given upthread are all other protocols mapped into the unixweb space. What's different about the relationship between IPFS and unixweb that couples them so tightly? Why can't IPFS be just another protocol supported by a unixweb mux – another protocol that can exist independent of unixweb?
@jbenet I'm not meaning to be a jerk either, but neither am I sufficiently interested in the unixweb concept that I'll dig through a year of transcripts to learn more, and you haven't pointed me to anything more concrete, so I'm left asking questions.
I understand but also know that if i just answer to every person that doesn't want to search, the one stuck spending extra time repeating himself is me. I don't need to explain anything to move forward, so in practice I won't.
Why does the potential future development of unixweb mean we can't have a URI scheme specifically for IPFS? The examples you've given upthread are all other protocols mapped into the unixweb space. What's different about the relationship between IPFS and unixweb that couples them so tightly? Why can't IPFS be just another protocol supported by a unixweb mux – another protocol that can exist independent of unixweb?
Here's one important reason: because we would otherwise be asking users to install N protocol scheme handlers. we're asking apps to support N protocol scheme handlers. it's already going to be 3 for ipfs (/ipfs, /ipns, /dns) maybe more. (note that IPNS can be used entirely separately from IPFS!). We want it to be really, really easy to add support for more protocols incrementally without having to bug other developers.
if i just answer to every person that doesn't want to search
I'm happy to search documents, not discussions. Link me to a document and I'll read it, even if it's just a napkin sketch on a wiki filled with spelling errors, so long as it captures the current thinking on a topic. Point me to "over a year" of "irc logs, mails, github, etc" and I'll ask questions.
I don't need to explain anything to move forward, so in practice I won't.
…questions which you are free to disregard. My intent isn't to bother anyone, it's to understand. If you find yourself repeating information for new people, perhaps that information needs to be organized in a more accessible format.
it's already going to be 3 for ipfs (
/ipfs, /ipns, /dns) maybe more.
At present, all three are handled by a process called ipfs, talking the IPFS protocol, with other machines running ipfs, exchanging data as described in a paper called "IPFS - Content Addressed, Versioned, P2P File System". This sounds like an ipfs: scheme.
unixweb: is a concept that I've seen alluded to in a few places, and given that you've pointed me to "various fora" instead of specs or code, I assume this concept only exists as discussion. My understanding is that it will be a single protocol scheme – unixweb: or fs: or whatever falls out. This implies having a single handler for the entire namespace, since platforms with pluggable handlers tend to work at the scheme level, rather than the scheme-and-path level. I assume that this handler will delegate e.g. /bitcoin to an existing Bitcoin implementation, rather than itself being a kitchen sink, which is why I called it a mux above. If that's the case, it seems like unixweb: could map /ipfs and /ipns to a local ipfs daemon, and IPFS would be just another protocol supported by this scheme.
We could have one scheme to refer specifically to IPFS today, and another to this grand unified hierarchy once it exists. Content in IPFS wouldn't care if it's being addressed using the local filesystem, or from http://gateway.ipfs.io/, or from ipfs:, or from unixweb:, because the paths relevant to IPFS would map to the same places in each context.
I am interested in IPFS in part because it exists today. I want a way to refer to IPFS objects in URIs. If IPFS and unixweb:/fs: are inseparable, well… okay. I don't understand why they would be inseparable, but fine. All I want is a way to refer to IPFS objects in URIs. Let's pick one.
It seems like people are missing the point that ipfs URIs are legacy
support only, they aren't supposed to be first class citizens, the paths
are.
On Wed, 16 Sep 2015 07:24 Will Glynn notifications@github.com wrote:
if i just answer to every person that doesn't want to search
I'm happy to search documents, not discussions. Link me to a document
and I'll read it, even if it's just a napkin sketch on a wiki filled with
spelling errors, so long as it captures the current thinking on a topic.
Point me to "over a year" of "irc logs, mails, github, etc" and I'll ask
questions.I don't need to explain anything to move forward, so in practice I won't.
…questions which you are free to disregard. My intent isn't to bother
anyone, it's to understand. If you find yourself repeating information for
new people, perhaps that information needs to be organized in a more
accessible format.it's already going to be 3 for ipfs (/ipfs, /ipns, /dns) maybe more.
At present, all of three are handled by a process called ipfs, talking
the IPFS protocol, with other machines running ipfs, exchanging data as
described in a paper called "IPFS - Content Addressed, Versioned, P2P File
System". This sounds an ipfs: scheme.unixweb: is a concept that I've seen alluded to in a few places, and
given that you've pointed me to "various fora" instead of specs or code, I
assume this concept only exists as discussion. My understanding is that it
will be a single protocol scheme – unixweb: or fs: or whatever falls out.
This implies having a single handler for the entire namespace, since
platforms with pluggable handlers tend to work at the scheme level, rather
than the scheme-and-path level. I assume that this handler will delegate
e.g. /bitcoin to an existing Bitcoin implementation, rather than itself
being a kitchen sink, which is why I called it a mux above. If that's the
case, it seems like unixweb: could map /ipfs and /ipns to a local ipfs
daemon, and IPFS would be just another protocol supported by this scheme.We could have one scheme to refer specifically to IPFS today, and another
to this grand unified hierarchy once it exists. Content in IPFS wouldn't
care if it's being addressed using the local filesystem, or from
http://gateway.ipfs.io/, or from ipfs:, or from unixweb:, because the
paths relevant to IPFS would map to the same places in each context.I am interested in IPFS in part because it exists today. I want a way to
refer to IPFS objects in URIs. If IPFS and unixweb:/fs: are inseparable,
well… okay. I don't understand why they would be inseparable, but fine.
All I want is a way to refer to IPFS objects in URIs. Let's pick one.—
Reply to this email directly or view it on GitHub
#1678 (comment).David A Roberts
https://davidar.io
Other day I had a small insight about a name.
How about just "ion:" (ION) ?
If you like the name, I can come with an analogy relating to physics. You cant think of a plasma ball representing the internet today, and a plasma NEON light (with a bunch of IONs inside) representing IPFS network.
And we can play with some marketing terms, like "Internet of Nodes", "Internet of Networks"...
A couple of counterarguments:
- As it was said above, the IPFS URLs are designed for the comfort of users of Unix filesystems. If
fs:is used, such users have to delete that 3-letter prefix to get a Unix path. Ifion:is used, they have to delete 4 characters, that's 33⅓% less comfortable for them. - We should use
fs:instead ofion:because such basic and ubiquitous thing (an URL scheme) should convey the initial semantic meaning (that IPFS is a filesystem) and not a metaphor (that IPFS is a neon plasma ball), especially for the people that would see such URLs for the first time in their lives. Even though the former is dull and the latter is cool.
We should use fs: instead of ion: because such basic and ubiquitous thing (an URL scheme) should convey the initial semantic meaning (that IPFS is a filesystem) and not a metaphor (that IPFS is a neon plasma ball), especially for the people that would see such URLs for the first time in their lives. Even though the former is dull and the latter is cool.
agreed
Ok, so is my understanding correct that for legacy purposes a consensus is to support both notations:
fs:/ipfs/hash/dir/doc
fs://ipfs/hash/dir/doc
Y/n:question:
I believe so, yes. (maybe fs:///... too?)
Yeah, better safe than sorry :)
It makes sense to accept (but not endorse) even fs:ipfs/hash/dir/doc. Just to be safe.
I agree with fs:/ and its safe variations.
Not a good idea. I have Apache Tomcat on port 8080 and it is by default.
@batcer that is because 8080 is the default for many HTTP-based apps, including proxies and servers.
I think it makes a good default for HTTP gateway too.
If you have a conflict, just change the port in your config file:
(..)
"Addresses": {
"API": "/ip4/127.0.0.1/tcp/5001",
"Gateway": "/ip4/127.0.0.1/tcp/8080",
(..)
@lidel Doesn't IPFS deserve dedicated port by default? Why use so common one?
8080 is only used by HTTP gateway, and as such it follows unwritten convention championed by other HTTP proxies (a port in 8xxx range).
I think that real 'IPFS port' is the swarm one, and it is 'dedicated' -- 4001
@lidel Well, guys, you now know of end user's problem, it's up to you to decide - to solve it or to let end user solve it. Best regards.
@jbenet I feel this discussion (last 5 comments) should be moved to a new ticket ("Default port for HTTP Gateway" or something similar), is it possible to move it?
@batcer I am running 3 apps that default to 8080. I understand your point.
Moving to something like 8001 would unify the 'look and feel' of default ports, and solve us some setup time.
@lidel sure go ahead, have to move it manually.
I've been reading lot of stuff related with ipfs and I hope I can help on this somehow.
Note that I did not read this entire issue, but wanted to express my opinion since I'm thinking to start working on top of IPFS
I see IPFS as a huge candidate for the future web, but the only way to make it possible, IPFS has to be a nice structured URI so it can be adopted by modern browsers.
I don't think they should be differentiated between their mediums. If that implementation comes true, there will be a period of implementation which probably will raise plugins by thirds on browsers to support it. If you start dividing the schemas such I was reading upward ( ipfs:// ipfs: and fuse moint points /ipfs ) this could be a bad beginning, having to maintain more than one notation and support on all of them (IE is a real guidance example of what shouldn't be repeated).
What intrigued me was, how would major websites migrate things to IPFS? They cannot rely on HTTP transfers over gateway.ipfs.io/(ipfs|ipns) or over their own gateways ipfs.domain.com/(ipfs|ipns), they must be able to have HTTP servers redirecting to ipfs://domain.com so they can instruct people that they should see ipfs:// instead of http(s)?://.
I would propose a single implementation that will allow future implementations and maintenance:
ipfs://[ipns-domain-endpoint]
ipfs://[ipns-node-endpoint]
ipfs://[ipfs-folder-hash]/path/to/file.html
ipfs://[ipfs-file-hash]
@cusspvz +1
Note that I did not read this entire issue, but wanted to express my opinion
@cusspvz thanks for voicing your input, but commenting on issues this long (and clearly this involved and controversial) without reading them is probably one of the rudest things you could do here... please do not do this!! Everything you said has already been discussed, and everything has already stabilized into solutions that work through all the problems. Note also that what you suggest is strictly wrong in multiple ways. You can figure out why by reading the issue.
To summarize (though really, read it!):
- canonical NURI is the path.
- where NURI means Nestable URI. more universal than URIs. if you do not understand read the issue.
/ipfs/Qm.../foo/bar,/ipns/Qm.../foo/bar, and/dns/foo.com/bar.- use this everywhere you can!
- compatibility schema prefix for use in today's applications:
- both ipfs-specific AND non-ipfs specific flavors (to be better citizens)
fs:/andipfs:/used as:ipfs:/ipfs/Qm..../foo/barandfs:/ipfs/Qm.../foo/baripnsanddnsprotocols work the same way:fs:/dns/foo.com/barandfs:/ipns/Qm.../foo/bar- note that our tools will also redirect two slashes
ipfs://andfs://though this is strictly wrong for non-authority based protocols, according to W3C rules.
- https://ipfs.io gateway urls http prefix for a canonical NURIs
- strictly for convenience to help tools that speak HTTP but do not speak IPFS.
- transition plan: start with links like
https://ipfs.io/<path>until browser implementations exist. - tools and browser extensions that speak IPFS SHOULD resolve these directly, and NOT use gateways.
- once implementations exist in browsers, switch the defaults. this ensures a safe, no-friction transition.
- canonical NURI is the path.
- where NURI means Nestable URI. more universal than URIs. if you do not understand read the issue.
/ipfs/Qm.../foo/bar,/ipns/Qm.../foo/bar, and/dns/foo.com/bar.- use this everywhere you can!
I'd just like to emphasize this since it took me a long time to understand the logic of it.
An IPFS path works as a relative URL. That means it is a resource name without a canonical location. Browsers can interpret it without conversion as relative to the current server, which presumably supports the IPFS interface.
I can only call this a brilliant hack. It gets IPFS up and running very quickly for what it wants to do, but I think it has some long term tradeoffs. Using a new URI protocol makes more sense and is probably a better choice in the long run, but it breaks browser support right now.
I would go so far as to say this is the distinguishing feature of IPFS. Love it or leave it.
Sorry Juan for not giving you enough credit!
@jbenet thanks for replying. 👍
Note that I did not read this entire issue, but wanted to express my opinion
@cusspvz thanks for voicing your input, but commenting on issues this long (and clearly this involved and controversial) without reading them is probably one of the rudest things you could do here... please do not do this!! Everything you said has already been discussed, and everything has already stabilized into solutions that work through all the problems. Note also that what you suggest is strictly wrong in multiple ways. You can figure out why by reading the issue.
I did read this issue, but I've jumped things that were out of my knowledge scope, thats why I've said I did not read the entire issue.
Everything you said has already been discussed, and everything has already stabilized into solutions that work through all the problems.
I've commented because almost all proposals have ipfs and ipns on NURI, which doesn't seems right for a common web user that doesn't knows what dns or ipns is. Is there a possibility to remove those prefixes?
| most of what I've seen | what I'm proposing |
| /ipfs/file-hash | /file-hash |
| ipfs://ipfs/QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D | ipfs://QmTkzDwWqPbnAh5YiV5VwcTLnGdwSNsNTn2aDxdXBFca7D |
| /ipns/peer-id | /peer-id |
| /ipns/domain | /domain |
| ipfs://ipns/facebook.com | ipfs://facebook.com |
https://ipfs.io gateway urls http prefix for a canonical NURIs
That will increase ipfs gateway server's traffic, isn't a solution for the temporary implementation to have that prefixed gathered from a text record of the domain browser is located and make that part of the specs for browser first stage implementation?
That will increase ipfs gateway server's traffic, isn't a solution for the temporary implementation to have that prefixed gathered from a text record of the domain browser is located and make that part of the specs for browser first stage implementation?
My understanding of current consensus:
Until native support is possible, we want to support DNS TXT records via browser addons (sadly browsers don't have API for reading TXT records, so we use one from local go-ipfs: #1054).
Having that, if a site announces IPFS hash there, addon will replace HTTP request with IPFS one.
To avoid unnecessary load of public gateway each site can set up its own HTTP gateway for clients that do not support IPFS (with an old school CDN in front of it)
Just skimming through this thread, a few things occur to me as a bit of a RFC geek:
ISTM ipfs paths are clearly hierarchical, and AFAIK follow the rules for hierarchical paths which is what the slashes indicate, i.e. <scheme>://. And since there's really no "authority" component, you'd want to skip that entirely, just as the file protocol does, and append your slash-prefixed absolute path to your <scheme>://, e.g. <scheme>:///some/path. FWIW it's only the authority component where you have to worry about ascii/punycode bullshit -- in practice, you can use just about any damn character you want in the path, unicode and all (even reserved chars, sans ? and #, and perhaps < and > for html/MIME reasons). Though you'd probably want to define a canonicalization to reign this in a bit.
I see @jbenet commented that paths are supposed to represent something like web+unix -- as it turns out, defining a scheme as web+ is kind of a thing (not yet in IE or Safari, but still available widely enough to be immediately useful). ISTM the web+ prefix was carved out just for this purpose, so what the hell -- why not web+unix///ipfs/<hash>/...?
Juan also suggested that the first component of a typical unixy NURI path pretty much specifies a scheme (or perhaps something like a subscheme). You could define schemes for each of these subschemes, e.g. web+unix.ipfs///<hash>/... (or something like that) -- each w/ the obvious desugaring to NURI paths. This would allow custom protocol handlers to be registered per unix subscheme. But this is overkill -- a single web+unix scheme ought to do just fine. Either way, there are lots of lovely things js libs could do w/ custom protocol handlers -- mobile apps too. Definitely worth considering -- and if a canonicalization were defined which is isomorphic to the NURI representation, then the problem of having two distinct addressing schemes goes away. In this light, web+unix:/// would logically just represent some base uri, e.g. https://ipfs.io/ by default.
And when content is consistently served up through the same gateway, e.g. https://ipfs.io, Juan's vision of just using these absolute paths, e.g. /ipfs/... /dns/... ought to hang together pretty well. With the gateway as the base uri, absolute paths would resolve exactly as you'd want. This ought to work all the same if you're running a private ipfs w/ your own gateway.
Though when using the DNS TXT record stuff you'd still want everything resolved through the gateway, or you'll blow up the otherwise lovely cache properties. Ideally I'd want to install a something like a service worker on the gateway to route all ipfs requests. Although for ipfs's use cases ISTM you wouldn't even need to wait for service workers -- you could make due with appcache, caching a little app that js ipfs could append as an iframe and interface w/ through postMessage. Assets would naturally leverage browser cache semantics, and some assets (e.g. pins) could even be dumped into indexeddb to get a stronger browser-wide cache of the content that really matters. Together, these two layers of caching would allow assets to be reused across any process -- a huge win, especially in a future where js modules and related static assets can be referenced by hash and imported from any number of apps. Even though interfacing with this cached gateway service would require postMessage, cross-origin communication would be essentially free thanks to Transferable objects. (For consistency, this same interface can be implemented for node-platform specific (non-browserified) code as a little service that passes back fds.)
In situations where an ipfs server is running locally, it'd be nice to be able to tap the system's local cache, and its discovery capabilities to get blobs with better locality properties. There are a bunch of ways a local server could be made to intercede on ipfs.io requests to allow all web apps using ipfs in any browser to share the same blob pool (unburdened by any idb utilization limits). But not over https with a valid cert...
One possible solution would be to allow something like <name>.ipns.io, where the owner of the namespace could run a daemon which could manage the whole letsencrypt acme dance -- verify itself by serving up a /.well-known/acme-challenge endpoint to fetch DV certs for <name>.ipns.io, and start serving up your namespace -- and by extension, the whole damn ipfs keyspace, with the appropriate TLS config (and perhaps HPKP-pinned to letsencrypt's CA) -- zero-conf TLS. Of course there'd also have to be some means of configuring the locally cached ipfs.io webapp/service thingie to read through some specific ipns.io subdomain (or any domain for that matter)... But meh, typing all that out, it's still pretty complex -- there's probably a much easier way. Still, IMHO you really couldn't (rather, shouldn't) create the kind of shared pooling service/CDN infrastructure thing described above w/o certs signed by a major-browser-trusted CA. At least not until something like DANE gets widespread browser support (many years from now at best, more likely never).
Though what I'd really love to see is a way to go in the other direction -- a way to allow a domain (or a subzone) to be delegated to something like an ipns nameserver gateway, configured by records within some designated ipns path (perhaps the /dns/ stuff already covers this ground?). ISTM the entire letsencrypt/acme lifecycle could be delegated to the ipfs infrastructure, and the https gateway could be counted on to serve up some specific blob -- e.g. a small top level html page -- with the appropriate cert via SNI. That top level page can pull in any number of assets via https://ipfs.io, but the https gateway would only have to use a custom domain cert when serving that one root blob. It might serve this blob only for a set of explicitly specified paths, or for any path for the host (allowing for pushState). Updating the app just requires changing the blob reference, and users could use a tool like hyperboot to make this even nicer. This would give the world a way to do reliable https-hosted apps in a completely disconnected fashion -- and would make the ssl renewal process a lot more resistant to censorship and eavesdropping.
Regardless of whether or not any sensitive underlying assets are encrypted, https is still non-optional for ipfs-hosted web-apps. The https://ipfs.io/dns/<domain>/ path is a good start for dev experiments but serving multiple ipfs web apps from the same origin demolishes the only reliable security boundaries in the clusterfuck that is the web platform's security model. At least for now, custom domains over https are the only viable option for a "production" app -- and ISTM they really are the last remaining barrier to entry for p2p web apps (ipfs-based or otherwise) to truly function as Real apps. (Well, this and some db/index magic iprs happens to set up nicely -- I'll try to write more on that in a more relevant issue).
Apologies for the huge wall of words -- I was just poking through the FAQs and wanted to make a few points about URI schemes -- and then I forgot to stop typing...
note that our tools will also redirect two slashes ipfs:// and fs:// though this is strictly wrong for non-authority based protocols, according to W3C rules.
Hmm, just read over the W3C material on URIs and this looks right -- apparently I missed the note in RFC3986 banning authority-less URIs w/ hierarchical namespaces (all these years I've just assumed the file:// scheme was valid -- whoops)...
So strike my suggestion re: web+unix:///ipfs/<hash> -- though web+unix:ipfs/<hash> looks pretty nice to me.
The most important counterargument here is that web+unix is unnecessarily four times longer than fs (8 characters instead of 2 characters) without bringing any significant benefit.
One significant benefit for ipfs-based web apps (which is what I was largely focused on) is navigator.registerProtocolHandler
I may not have been terribly clear, but I was actually trying to suggest this not as an alternative to the "stopgap" scheme (fs is where this conversation landed, I think?) -- but as a separate thing entirely that could be incredibly handy for web apps. It's not quite the same as the fs stopgap scheme -- instead it's an isomorphic mapping of NURLs into HTTP GET requests to some designated gateway base uri of the user's choosing...
This is what started me down the path of that painfully long, rambling sidenote about possible strategies for a shared browser cache and serving apps off custom domains -- in that kind of world NURL paths fall down, and the fs scheme wouldn't fare much better. You could hook fs: prefixed link clicks w/ some js -- on any pages you control, at least (just as some evil page could hook them to point some MITM'd gateway) -- but there's something pretty appealing being able to add a single registerProtocolHandler to tell the browser how it should handle all ipfs gateway links. IIRC this should also work in script tags, link tags, etc. That'd be a pretty significant benefit right there.
But perhaps more importantly, web+unix is more descriptive of what's actually going down -- a mapping of unix onto the web -- rather, HTTP GET requests into a specified ipfs gateway. Alternatively, web+ipfs looks about right to me -- or even web+fs (though I suspect that'd be a lot more prone to possible collisions -- but this is land grab territory, and it may be early enough to stake that claim).
ISTM there's little worry in fretting over few extra readable characters when you step back and think about the large strings of gibberish that will often follow 😄
@deanlandolt thank you, that was very illuminating for me. Might you have an opinion about how https://github.com/okTurtles/dnschain could integrate with ipns hashnames?
Respect.
@jbshirk beautiful! I hadn't seen DNSChain yet -- definitely a lot in there for me to chew on. I'm not exactly sure what you had in mind WRT to integrating w/ ipns though.
But WRT to DNS in general -- yeah, I agree it's pretty damn broken, but it's really the only game in town for the entry point of a web app targeting the washed, shaven masses. I keep hoping someone's going eventually figure out how to unbreak it, but yeah, DNSSEC/DANE won't do (even if there were a legitimate chance it might ever be available in all major browsers).
While I have no idea how we could actually fix DNS, and, I'd also assert that we can't completely avoid it in reasonable near-term future, there are definitely things we an do to minimize our exposure. In fact, with a simple bootstrapping lib (and the requisite js ipfs libs), we could reduce any given (unsophisticated) user's reliance on DNS queries all the way down to a single query the very first time the app is loaded (zero for sophisticated users, perhaps by way of something like DNSChain+ipfs).
And when served over HTTPS with a good config and well-pinned certs (e.g. like some kind of hosted SSL entry point service I described above), we could shrink the attack surface down to a point where the MITM threat model is pretty much confined to 3-letter agency territory.
Assuming the existence of the kind of app-cached gateway or service worker I'd described, we could go even further and abuse the browser's terribly leaky privacy to periodically probe for domains that are known to be served by the SSL entry point service but have not verified themselves. We could implement a protocol within the bootstrap code that allows all bootstrapped apps to be added as an iframe and could be interacted with by the parent frame through an available cross-origin communication channel. This would allow us to issue challenges to any bootstrapped app and perhaps warn the user accordingly when misbehavior is detected. (Conveniently, this would also allow web apps to function as data services within a given browsing context, exposing some subset of their functionality and origin-guarded local data, e.g. whatever they keep in their private IndexedDB, WebSQL, etc.)
For an added layer of paranoia we could draw up a little protocol for installing and verifying the little bootstrap manager service thing. The provided bootstrap html would include a payload with something like a pointer to an ipfs hash referencing the current DNS records of the host (with DNS changes kept merkle-y, as well as a timestamp and nonce.
The app could initialize ipfs and start downloading deps right away, but in parallel it would attempt to "register" itself with the host by POSTing back the payload it received (DNS hash, nonce and timestamp). It'd retry with backoff up until some timeout value is exceeded, or it gets a 201 (if successful) or 4xx response (if failed). A successful POST would include a location header pointing to the ipfs hash of the record that got created -- the POSTed payload, signed by the host.
(NB: I just realized I completely gutted important bits of this when cleaning this up -- the idea was to have an isolated ipfs iframe act as the initial "preboot" manager connected to any one of the IPs in the DNS A/AAAA records to do initial interfacing of ipfs to verify the bootstrapped server.)
The content of this hash represents proof that the bootstrap html was delivered directly the origin it claims to be from, and wasn't tampered with by an intermediary. It could still be tampered with by the user, or a malicious process in userspace. So we should probably do the same dance when installing additional apps. Apps would probably need to verify everything that's signed, but interestingly, they wouldn't necessarily need all the ipfs deps, and could reuse a single connection pool exposed by the manager thing (definitely if run as a service worker, but also if this all went down within a SharedWorker).
Damn -- again, that's a lot more words than necessary (and likely belongs in an entirely different issue, if anyone finds the idea appealing). But ISTM this ought to pretty much button up the DNS hole without new protocols and trying to drag the browser venders along.
I meant something like turning
gateway.ipfs.io/ipns/Qme67efGcE9iX6r8Wk6CrHUsmWckLbnMHQ19g5YK15doUY/tw5.htm
into
gateway.ipfs.io/ipns/joesdomain/tw5.htm
On Nov 25, 2015 5:57 PM, "Dean Landolt" notifications@github.com wrote:
@jbshirk beautiful! I hadn't seen DNSChain yet -- definitely a lot in
there for me to chew on. I'm not exactly sure what you had in mind WRT to
integrating w/ ipns though.
Hey could you keep it on-topic? This issue is already very long and we need to be able to tell what the results of the discussion are -- we already came to a consensus. Thank you!
@deanlandolt Oh, I get it now: your intention was to work around JSFoundation/standards#12. Thank you for explaining.
comment thread is now locked, For those trying to get an idea of what consensus was reached here, read above @jbenet's comment here: #1678 (comment)
thanks for keeping things on topic-- @deanlandolt maybe can continue that thread here: ipfs/notes#74 -- ill respond there eventually
Update: we will also add (not replace) a web+ prefix to the schemes above (for chrome):
web+fs:
web+ipfs:
Chrome extension handlers force web+ prefix: https://developers.google.com/web/updates/2011/06/Registering-a-custom-protocol-handler?hl=en
This issue is very old and should be closed. A lot changed since 2015:
- The
fs:/was droped as an idea. We switched toipfs://,ipns://anddweb:/- Discussion from 2015-2017 was summarized in ipfs/specs#152
- We are working with browser vendors and standard bodies to whitelist mentioned protocols in HTML spec and other places
- The latest IPFS addressing standards can be found at ipfs/in-web-browsers/ADDRESSING.md
ip[nf]s://<domain.name>/objectRetrieval would be processed as "DNS look up
domain.name IN TXT, resolve resulting IPFS/IPNS identifier, retrieveobject".
I was reading this, hoping to find just that. IPFS wants to get away from hosting locations for reasons of persistency, but IPNS is different. I love the idea of using DNS (and DNSSEC) and getting very powerful bookmarks:
- Stored domain name, for human reference where it once originated from
- Stored IPNS key, so we can lookup the names from anyone who pins them, along with new versions from the same origin
- Stored IPFS hash, so we can retrieve the version that we bookmarked for as long as it is pinned
I will add the spec to my reading list, it's useful stuff!