Multi-source providers
Tahul opened this issue · comments
Multi-source is now possible with @nuxt/content, but is still an experimental feature.
Local sources (same filesystem) are already supported, but remote ones are still a pending task.
Here is a list of providers we could have in mind to support:
- Other GitHub repositories
MDC Syntax would most likely only be available from providers that can return raw text data though.
Here is what package is used to manage content sources, and where should these drivers be implemented:
https://github.com/unjs/unstorage
Most provider sources could be supported by using a custom unstorage driver and fetch
. We just need to map each into a Content module compatible format (List, parsed (alternative to markdown AST?) and Meta)
BTW, while this is a nice idea to have CMS support with content style queries, I feel this is kinda out of the content module's scope. Almost each of the providers mentioned above has their unique abilities that could be limited when going behind abstraction and we already have a Nuxt module for a couple of them.
I would separate this issue into:
- Multi-source support (fetching from other repos from the same content format. continuation of #265)
- A discussion to investigate the possibility and benefits of integration with external CMS providers.
Agree to split this issue in two, also I do believe that it is out of the scope of content regarding MDC and other providers personally.
Creating one following #265 shall be enough
Adding content sources from other GitHub Repositories is enough because in most cases we need separate docs repository from docs website source code. This feature helps prevents the website from copying.
These are more ideas for future possible enhancements than real objectives.
I think supporting other GitHub repositories is an evidence, and supporting Notion would be a really nice-to-have feature.
I completely agree for some of these providers, a standalone module would fit the use case better than being in the content module itself.
Still, I think at some point the content module has something to offer to these standalone modules, which is MDC parsing and rendering!
MDC Syntax allows the usage of project components by parsing a syntax that can be stored by any content provider, and I think providing the support for this syntax to these externals providers would be amazing adoption-wise for @nuxt/content.
Local sources (same filesystem) are already supported
Kindly add some easy-to-understand documentation on how to fetch from different folders on same PC. The current documentation on using drivers and all is confusing.
- What are drivers? What are the driver options? Where to find list of such options? What is
resolve
? - What's the difference between the
name
,prefix
, andsource dir
in thebase
? And why to have different names for them? - How will fetching from private GitHub repo work? How to authenticate it? Or will it even work?
The reason I started using Content module (and Nuxt) is because it is very easy to learn and understand, even for non-coders. The whole Content website is very simple. So, whenever this multi-source feature becomes fully functional, kindly simplify this part of the documentation as well, especially as if oriented towards a novice level, and not for people who already know this stuff (because you all already do 😁).
Thanks.
Mutli source is next in our roadmap, it was not a blocker for the initial release.
We do need it since we plan to have all of our docs in one website 😊
@Tahul, @atinux Kindly help by filling these blanks 😄
My Nuxt Content app is in this folder C:\Users\manas\OneDrive\Projects\nuxt3-content2
on my PC.
What base
config should be added in sources
object for the following 3 types of folders (apart from the existing content
folder)?
1. ✅ SOLVED: If another folder content-2
present in the same folder as the app at the root level?
Issue
- Simply adding another item in the `sources` as string `content-2` works, and even fetching works properly. - But to add `prefix` to it, we need an Object syntax inside the `sources` item, for which adding `driver: fs` and `base` properties is compulsory. - So, what should be `base` for such folder?{
name: 'content-2',
prefix: '/f2',
driver: 'fs',
base: path.resolve('.', 'content-2') // ~~What should come here?~~ Never mind, got it.
}
👍🏻 SOLVED
So, the dot '.' represents the root.Also, the name
automatically becomes the value of the _source
key inside the file's frontmatter. This need not be the name of the folder where the md files are stored. This is simply just another frontmatter key. This should be documented on the website. 😁 Can we Query the files using this _source
key inside .where
options? Most probably, but haven't tried yet.
Note to maintainers: Kindly add in the documentation that name
becomes _source
key in frontmatter.
2. ✅ SOLVED: If a folder content-2
is present in the same app's folder, but nested inside some other folder?
Issue
C:\Users\manas\OneDrive\Projects\nuxt3-content2\docs\content-2
- I don't want the whole
docs
folder to be considered for the Content Module, but just thecontent-2
child folder present in it.
{
name: 'content-2 inside docs',
prefix: '/docsc2',
driver: 'fs',
base: path.resolve('.','docs/content-2') // ~~What should come here?~~ Never mind, got it.
}
👍🏻 SOLVED
From the root .
, now you can add any path easily.
3. ✅ SOLVED: If another folder Daily
is present elsewhere on the computer?
Issue
- This new folder is present in
C:\Users\manas\OneDrive\Documents\Obsidian Vaults\Master
. - This folder contains other file types as well, apart from markdown files.
{
name: 'Daily', // ~~Is space allowed in the folder name here?~~ Yes, it just gets copied as frontmatter key.
prefix: '/daily', // This is main. It is what is used to query the posts.
driver: 'fs',
base: path.resolve('C:/Users/manas/OneDrive/Documents/Obsidian Vaults/Master','Daily')
~~// What should come here?~~
// Never mind, got it.
}
👍🏻 SOLVED
Even on Windows, use /
in path, instead of default \
. https://stackoverflow.com/questions/31847712/node-js-fs-module-and-windows-paths.
😮Another issue of Folder with images
While base
issue is solved, but because the Daily
folder contains images as well, this is what I get in the Terminal. And even the page takes so much time to reload that fetching doesn't work takes half a minute.
Why doesn't the Nuxt Content module simply exclude these file types? So, creating a new issue for it.
Thanks in advance. With these three questions solved, it will be very easy for future users (non-coders) as well. ❤️
Any update on this idea?
I'm also looking for a way to fetch remote data.
Most provider sources could be supported by using a custom unstorage driver and
fetch
.
This would definitely fit my need. Any chance to get a working example on this? Even if this is done by hand first.
Happy to help
I'm also looking to fetch data from different directories, and have a fallback if the content directory is empty.
For example, if the <content>
directory is empty, we could have a fallback <another-content-directory>
that will always have a default content in it.
That could be useful when working with multiple repos coming from a single upstream.
Glad to help.
The fallback strategy is pretty tricky to do actually @ricardoaguiar but is interesting.
Do you have a reproduction to test?
Could we get an example of just how to use the existing http
driver to fetch documents from an API? Would be super-helpful.