[Packager] doesn't resolve modules that are symlinks
JaapRood opened this issue · comments
As described in the browserify handbook you can use symlinks in the node_modules folder to reference modules. This is mainly useful for preventing ../../../../ requires.
With a symlink setup from the project root setup like:
ln -s src/app node_modules/app
in any file in the project you can:
var app = require('app');
var somethingElse = require('app/lib/something-else');However, the packager can't resolve them: Requiring unkonwn module "app".
Edit:
Check out this repo to easily reproduce the error above:
https://github.com/JaapRood/react-native-packager-symlink-bug
is it a bug or is it by design?
symlinks would be really handy actually :(
Just ran into this trying to develop a npm library locally using npm link
I thought we supported this use case. Thanks
Perhaps related, relative requiring a module above the iOS/React Native directory fails the same way:
// in index.ios.js
var Auth = require('../common/auth');
Error: Requiring unknown module "../common/auth". If you are sure the module is there, try restarting the packager.
I tried symlinking that higher-level directory within the React Native project to fix, but it failed as well, leading me to this issue.
I finally had some time to make an example to prove it's not working. Check out this repo which references the example app component through a symlink, which doesn't get resolved.
https://github.com/JaapRood/react-native-packager-symlink-bug
Now, if only I had any idea on why.
I have an internal fix for this. Aiming to get it out next sync :)
Awesome @amasad - looking forward to this, will make lib development easier
Great @amasad, that would be immensely helpful!
@amasad no pressure, but when to expect next sync? :) will it be in 0.6.0 release?
still an issues as to 0.6.0 and 0.7.0-rc ...
That's strange. I'll give @JaapRood's test project a spin see what the problem is.
👍 Same problem here. Symlinks are very useful, especially when I don't want to make a node package just to share a component between two related React apps.
it generally usefull for many global acessible parts of your application: constants, configs, reusable components replacing react-native base components (styled Text for example)
and none of other methods of requiring modules from applications root seen to work too :( this is why i'm hoping for this one so much.
@amasad - any updates here?
👍
Figured out a little more about the cause -- the issue goes away when switching to the Node-based watcher instead of the Watchman-based one. The watchman homepage says, "Watchman does not follow symlinks. It knows they exist, but they show up the same as any other file in its reporting." so this could be the root case.
Support for symlinks seemed to get pretty strong pushback in facebook/watchman#105 so perhaps the packager could use a combination of Watchman for most files and the Node watcher for symlinked packages. npm linked packages reside in the first node_modules directory (you'd have to go out of your way to end up with node_modules/pkg1/node_modules/<linkedpkg>), so that might simplify things a little. I tried adding a --noWatchman option to the packager but ran into fd limit issues unfortunately.
The npm link workflow is really important for anyone developing a React Native module so I do think this is a priority for the external RN community.
I'm not sure if this is related or not. I have a symlinked folder for some of my js files in the project root. These are some internal libraries that are part of another repo. I am importing them using require('./lib/file') where lib is a symlink to a folder. This is working in the packager in 0.7.1 and not working in 0.8.0-rc. I am using io.js.
I figured I'd put this here in case some code is changing in 0.8.0-rc related to this bug...
For 0.8.0-rc, the problem is Fastfs._getRoot returns null, when the local module located outside of the app dir, because the packager can only resolve modules within the app dir by default.
To solve the problem, I must launch the packager with an additional argument --root=<path_to_local_module>.
I'm not sure what's the best solution for this issue?
I use 0.8.0, after close all terminal, xcode, restart. then everything is OK.
For those who have missed the release of 0.9.0-rc it's listed there as a known issue.
Is there any good workaround here?
I've found that this can be worked around by uninstalling watchman and therefore forcing react native to use the default node file watcher, however this does have speed drawbacks, and you can come up against issues with the number of files being accessed depending on the project.
-------- Original message --------
From: Braden Simpson notifications@github.com
Date: 14/08/2015 20:25 (GMT+00:00)
To: facebook/react-native react-native@noreply.github.com
Cc: Gethin Webster gethin.webster@agilityworks.co.uk
Subject: Re: [react-native] [Packager] doesn't resolve modules that are symlinks (#637)
Is there any good workaround here?
Reply to this email directly or view it on GitHubhttps://github.com//issues/637#issuecomment-131214239.
Sorry for the delay, was out for a while.
@amasad - seems like this was working for some people in 0.7.1 and not in 0.8.0-rc, as reported in #2076
What happened is that we started using watchman for the filesystem crawling. My understanding is that watchman doesn't follow symlinks but should treat them like any other file. Which in theory should still work. I can try to debug this soon. Alternatively we can add an option to force not using watchman for filesystem crawling.
Hi guys, any update on this ?
My only way to develop a react native library is to git clone my repo inside my node_modules, isn't it ?
I'd also love to see this fixed! In the meantime it would great if there was something written up on the best workflow to develop third-party components, specifically when it comes to testing them in an example app (which is where this problem comes into play).
git clone your repo inside node_modules sounds like a way to do this.
When working e.g. on the CLI I usually make changes in node_modules to test them quickly and once I'm ready to commit I copy the changed files to the right location.
This problem is frustrating. Please be more friendly to module developers, RN!
My workaround is to resolve the symbolic link and pass a --root parameter to the packager, so that I can use npm link again
solveLocalModule() {
DIR="$1"
if [[ -L "$DIR" ]];
then
MODULE_PATH=`readlink $DIR`
if [[ -L "$MODULE_PATH" ]];
then
solveLocalModule "$MODULE_PATH"
fi
fi
}
solveLocalModule "node_modules/<local_module>"
cd node_modules/react-native/packager
if [ $MODULE_PATH ];
then
echo "solved LocalModule -> $MODULE_PATH"
./packager.sh --root="$MODULE_PATH"
else
./packager.sh
fi
👍
I recommend collaborating on Watchman if this issue is very important to you. Personally I've been developing under node_modules instead since that's easier.
Thanks, I'll do that :)
Agree that developing straight into node_modules seems to be the best way right now.
Bit of a warning though, that with 0.13.0-rc the packager will fail with an error if it finds the same module installed in both node_modules and node_modules/(package-being-developed)/node_modules. I'm trying to figure out whether it's just an odd setup that I've got that results in this duplication, but it may end up being an additional hurdle to plugin development :/
This is not a good practice. We need to get support for symlinks easily! Come on!
How will we get better development, modules, tools, if developing modules that use other modules is a complete pain?
We have a few components that we share throughout local projects with npm link that are affected by this
@amasad I was wondering whether you got anywhere with what you mentioned in your last comment a while ago, or perhaps could share some pointers on where to dig around for those less familiar with the codebase?
What happened is that we started using watchman for the filesystem crawling. My understanding is that watchman doesn't follow symlinks but should treat them like any other file. Which in theory should still work. I can try to debug this soon. Alternatively we can add an option to force not using watchman for filesystem crawling.
For those affected by this bug: please let your voice be heard on Product Pains where React Native is prioritising what issues need their attention. https://productpains.com/post/react-native/packager-doesnt-resolve-modules-that-are-symlinks/ https://productpains.com/post/react-native/symlink-support-for-packager/
There is already a Product Pains item for this: https://productpains.com/post/react-native/symlink-support-for-packager/. Best to pick one to consolidate votes.
@ide totally, thanks for letting me know. I guess I've got to up my search abilities!
For people who are currently having issues with this, I just wanted to share what I'm doing for now.
In my main project, which has a handful of local dependencies that I'd normally pull in with npm link, I have a Makefile step that will recursively copy all my local libraries into my project's node_modules directory. It's not a perfect solution, but it is as simple as running make link for now. I suggest others do the same until proper symlink detection is supported
+1
+1 Please fix this issue!
While there's progress being made here, we should improve the error message and throw an exception if there's a symlink pointing to this issue/product pains. Otherwise people will keep getting confused until they figure out that react native doesn't support symlinks
With the progress being made on node, the stock file watcher might now be
good enough for most people using react native. So we should re-evaluate
having watchman be the default, then huge projects like Facebook can opt
into using watchman. While the majority don't have to install watchman and
can also use symlinks.
If someone is up for it please research if the NodeWatcher in the sane
library is good for all platforms that we support. And can scale up nicely
to relatively large projects.
On Friday, January 8, 2016, Christopher Chedeau notifications@github.com
wrote:
While there's progress being made here, we should improve the error
message and throw an exception if there's a symlink pointing to this
issue/product pains. Otherwise people will keep getting confused until they
figure out that react native doesn't support symlinks—
Reply to this email directly or view it on GitHub
#637 (comment)
.
Simple solution: Uninstall watchman and run the packager as usual. npm linked modules work and symlinks are followed.
Simple solution: Uninstall watchman and run the packager as usual. npm linked modules work and symlinks are followed.
I'm getting ERROR Watcher took too long to load. Try running watchman version from your terminal.
Could anybody suggest how to use node file watcher instead of watchman ?
I think uninstalling watchman is indeed the simplest thing to achieve that
On Wed, Jan 20, 2016 at 12:44 AM, Vitaliy Potapov notifications@github.com
wrote:
Simple solution: Uninstall watchman and run the packager as usual. npm
linked modules work and symlinks are followed.I'm getting ERROR Watcher took too long to load. Try running watchman
version from your terminal.Could anybody suggest how to use node file watcher instead of watchman ?
—
Reply to this email directly or view it on GitHub
#637 (comment)
.
Oh no kidding. I had no idea that there was fallback support in the packager for running it without watchman. I believe I only have watchman installed to support the packager, perhaps legacy versions required it? Great news, will 🔥
@amasad
Exactly after uninstalling watchman I'm getting that error.
Is watcher to use placed somewhere in packager config?
It worked after uninstalling watchman! Hooray for fallbacks :)
However when I update my symlinked package, the packager doesn't pick up on it and re-build the app. Any way to fix this?
@tuckerconnelly if you can come up with an isolated reproduction steps I'll happily fix it (or accept patches). The underlying watching library is Sane.
Also cc @kittens who was looking into making node the default watcher
From a cursory investigation it looks like fs.watch supporting symlinks depends on the OS API that's being used. Although there are libraries like chokidar that appear to have some code for symlinks that we (the packager or sane) could potentially take?
I naively tried using chokidar and it timed out on a real project, but I didn't investigate or optimize at all so maybe you could get it working.
OK, so here's the repro steps:
-
react-native init AwesomeProject
-
mkdir test
-
cd test
-
npm init (all defaults)
-
Make an index.js, set it to:
module.exports = function () { console.log('hey') }
-
npm link
-
cd ../AwesomeProject
-
npm link test
-
Add this to your index.android.js :
import myFunction from 'test' myFunction()
-
brew uninstall watchman
-
react-native run-android
-
Open chrome debugger to check the console.log hey
-
Change console.log('hey') to console.log('ho')
-
Reload the javascript in the android emulator
-
Still says 'hey'
Environment:
Mac OS X 10.11.3
react-native 0.20.0
react-native-cli 0.1.10
node 5.5.0
npm 3.7.2
Genymotion Galaxy S6 API 23
Thank you :D
Here's a crude workaround:
Create file watchAndCopy.js in your react-native project root:
var fs = require('fs-extra');
var watch = require('node-watch');
var rimraf = require('rimraf');
var exec = require('child_process').exec;
var packageName = 'my-package';
var packagePath = '../'+packageName;
console.log('Cleaning node_modules/'+packageName)
rimraf('node_modules/'+packageName, function () {
console.log('Copying '+packageName)
fs.copy(packagePath, 'node_modules/'+packageName, function (err) {
if (err) return console.error(err)
})
// Couldn't get rimraf to delete .git via function call.
// This seems to work, though.
exec('rimraf node_modules/'+packageName+'/.git', function () {
console.log('Watching '+packageName)
watch(packagePath, function(filename) {
var localPath = filename.split(packageName).pop()
var destination = 'node_modules/'+packageName+localPath
console.log('Copying '+filename+' to '+destination)
fs.copy(filename, destination, function (err) {
if (err) return console.error(err)
})
})
})
})Replace my-package with your package and make sure the path is right. Make sure rimraf is installed globally (npm install rimraf -g). Run node watchAndCopy.js and you're good to go.
Plus side is it works with watchman so it's super fast :)
Declaring my local dependency as follow :
dependencies: { my_local_module: "file:../../path/to/local/module" }
then npm install my_local_module
seems to work for me.
Npm doesn't use symlinks for 'file:'. It just copies the folder.
On Saturday, 5 March 2016, Mehdi Chamouma notifications@github.com wrote:
Declaring my local dependency as follow :
dependencies: {
my_local_module: "file:../../path/to/local/module"
}
then npm install my_local_module
seems to work for me.—
Reply to this email directly or view it on GitHub
#637 (comment)
.
Sebastian McKenzie
As a workaround, is it possible to simply configure multiple roots in the RN packager so that it picks up code from multiple distinct folders?
According to https://github.com/facebook/react-native/blob/master/local-cli/util/Config.js, you can put custom configuration for the packager. It should be named rn-cli.config.js.
Here is an example configuration:
https://gist.github.com/andon/bdf061e607f68def26fa#file-rn-cli-config-js
@andon That's right, thank you for adding the links. I'm curious if using the CLI config actually solves the issue? Might be missing something :)
@mkonicek Well, it doesn't :)
I was just pointing out that the RN packager can pick up code from multiple roots.
The initial problem still exists, in a sense that you can't use symlinks as you would normally do with npm-link.
I've used RN packager custom configuration as an alternative way to do this kind of project setup.
I met the same issue too. Any one can solve it now?
I tried a tricky solution and success, it is just copy the module to YOUR_PROJECT/node_modules/ whenever the module has changes.
Here is the example:
(The file structure for example copied from v0.8.0-rc.2 cannot resolve symbolic linked modules #2076, looks as following: )
my_app
|-- node_modules
|-- react-native
|-- my-module -> ~/.npm/my-module (another symbolic link pointing to my local module)
If i use npm link ~/.npm/my-module under my_app, it will cause error: "Unable to resolve module...".
So, I use gulp.watch, whenever my-module has changes, it will copy my-module folder to my_app/node_modules/my-module.
☝️ I'm doing the same thing at the moment FWIW. I know there's a lot of people that are encountering this problem. Unfortunate workaround, but it will make things easier after you set it up for the first time.
@nikki93 investigated using bindfs, which lets you mount a directory elsewhere in your filesystem, and found that it partly works -- the packager/watchman can see the files within the mounted directory, but it doesn't pick up changes so it's not 100% useful for development & iteration. It seems that bindfs doesn't propagate whatever mechanism watchman uses to detect changes.
As a workaround you can launch the packager manually with multiple roots
node_modules/react-native/packager/packager.sh --projectRoots <ROOT-1>,<ROOT2>,<ROOT-3>
I don't know the advantages of Watchman but like others have pointed out, a fix in the meantime is just uninstalling Watchman:
brew uninstall watchman
I'm having issues with this as well. I can pass in the project roots and the packer will pick it up but it will only watch for file changes in the root. Here's what I've seen so far with
- react-native 0.22.2
- Uninstall watchman
- let's say i have
/mysymlink - Use --projectRoots or make a rn-cli.config.js to inject
/mysymlinkinto projectRoots
Changes to the files in that folder root are picked up, but it does not recurse at all, so /mysymlink/index.js works but /mysymlink/foo/bar.js does not.
Debugging this a little bit it seems like:
The packager server uses a filewatcher
https://github.com/facebook/react-native/blob/master/packager/react-packager/src/Server/index.js#L13
The filewatcher comes from node-haste
https://github.com/facebook/node-haste/blob/master/src/FileWatcher/index.js#L80
Which uses sane
Sane uses walker to walk the tree
https://github.com/amasad/sane/blob/master/src/node_watcher.js#L329
(note the dir callback)
and finally the walker
https://github.com/daaku/nodejs-walker/blob/master/lib/walker.js#L76
doesn't recurse down symbolic links (whereas it does if isDirectory() is true)
So somewhere in this whole chain it needs to recurse for this solution to actually work
+1
Thanks for the help everyone - I got this working. I found that using --root ../my-common-module flag works, but it won't pick up transitive dependencies inside ../my-common-module. I get the usual: 'Unable to resolve module from ...' error.
Has anyone got a workaround that works with transitive dependencies?
@richardgill are the dependencies outside your root? In that case you can specify multiple roots
--projectRoots ../my-common-module ../my-other-dependencies
I went on to try again on Ubuntu. Weirdly this gives the same behavior after running npm un watchman as before. The behaviour is as follows:
When I run react-native start, the packager follows my symlinks. Files are bundled and my app works. Now for any change I make to symlinked files, the watcher does not work. So it follows symlinks but it does not watch them.
I couldn’t use a second project root, because the RN packager would find duplicate files in my package’s node_modules dir. I ended up syncing the package’s changed files into the consuming app’s node_modules dir: artsy/emission@1049aa7#diff-b9cfc7f2cdf78a7f4b91a753d10865a2R12
This is hindering me from making really (or relatively) awesome RN modules :(
If this is because of Watchman (used by the packager), I'm thinking we could simply add an option to disable Watchman.
@mkonicek I think there are 2 issues. Consider a repo layout like the following and running the RN packager from ./example
my_package
├── example
│ ├── node_modules/my_package (-> ../../)
│ ├── package.json
├── lib
├── node_modules
├── package.json
- Watchman doesn’t follow symlinks, so changes made in
./libwon’t be noticed by the RN packager. - Without watchman, the RN packager will follow the symlink and through it find
./node_moduleswhich will typically (for e.g. testing purposes) contain all of the same dependencies as exist in./example/node_modules, which leads to the RN packager complaining about duplicate source files.
This is really frustrating :/
Someone should really find a proper solution for this thing. developing without npm link is kinda foolish.
copying the dependency into node_modules, is also strange, what do you do on deployment?
Hi all, being tired of how screwed up this situation is for us I wrote a simple tool that listens to changes in one folder and copies them to another folder. It's called WML (Watchman-Links, since it's based on watchman). Check it out here.
It's very simple to use:
# add the link to wml using `wml add <src> <dest>`
wml add ~/my-package ~/main-project/node_modules/my-package
# start watching for changes
wml start@dutzi I heard about wml earlier today. Thanks for writing that! I also like that it doesn't put your npm package under your global node_modules the way that npm link does.
being tired of how screwed up this situation is for us I wrote a simple tool
Haha yeah. When things truly are an issue then someone will fix them 😁
@dutzi This is great 👍
WIll try it for sure :)
I was able to get a setup like @alloy's work for android by running the packager from the parent directory (my_package).
Here is the setup.
my_package
├── example
│ ├── node_modules
│ ├── package.json
├── lib
├── node_modules
├── package.json
Here are the steps
-
Inside
my_packagerunnpm install --save-dev react-native react. So we can run react-native commands frommy_package -
Edit
example/android/src/main/java/[..]/MainApplication.javato add following insidenew ReactNativeHost(this) { //here }. This tells react-native to check "example/index.android.js" for entry file. Instead of "index.android.js"@Override protected String getJSMainModuleName() { return "example/index.android"; } -
Add a
.watchmanconfigfile tomy_packagewith content{ "ignore_dirs": ["example/node_modules"] }Other packages needed by the example app need to be added to devDependancies of my_package.
-
run
watchman watch-del-all -
run
react-native run android --root examplefrommy_package
This works for me. And when committed to git should work for anyone who clones.
Looks like the pain will be over soon :) #9009
I am using gulp to copy from my modules ,Just pasting snippet may be it will help someone
gulp.task('watch-folder', function () {
var packages = [
{
src: ["../components/appmaker-components/**/*.{js,png}"],
dest: "./node_modules/appmaker-components"
}
];
var tasks = packages.map(function (item) {
console.log(item);
var watcher = $.watch(item.src);
watcher.on('change', function (event) {
console.log(event);
// console.log('File ' + event + ' was ' + event + ', running tasks...');
});
return gulp.src(item.src)
.pipe(watcher)
.pipe(gulp.dest(item.dest));
});
return merge(tasks);
});
I have posted the solution in
http://stackoverflow.com/a/38976236/3360028
I've just upgraded to React Native 0.35 which includes #9792 and thus should fix this issue.
I've created a node_modules/src symlink to src but when I include a component (import Button from 'src/components/Button/') it says that I'm requiring an unknown module.
With plain React (created with create-react-app) under Firefox it works.
I've tried to rerun both react-native run-android and react-native start but it doesn't change anything
Yeah, I think symlinks are still busted. @ericvicenti is looking into the general problem of "how does one develop a React Native library with best practices" so 1/ people should throw their random ideas at him and 2/ I am hopeful this will get fixed en route.
@lacker I think we're close.
I just tried now on using react-native@0.42.0-rc2 and lerna@2.0.0-beta.37 in https://github.com/infinitered/reactotron (specifically https://github.com/infinitered/reactotron/tree/master/packages/demo-react-native)
I tried a few different things.
Thing 1
If we just try to do a symlink, and the target library has it's own symlink it, then the packager can't resolve it.
Thing 2
If i symlink a react-native-based dependency, the packager detects it just fine 🎉 , but if that dependency has it's own react-native in its node_modules, then all hell breaks loose. You get duplicate @providesModules if they're the same RN, and conflicted @providesModules if they're different.
Lerna Almost Saves The Day
Lerna can now deal with this by hoisting that dependency up to the root of the repo. 🎉
But if you do that for both the host app, then the cli doesn't work (because the host app doesn't have node_modules/react-native/*.
So, if we manually symlink that to '../../node_modules/react-native', the package goes off on a spiritual journey rounding up everything in the mono repo, eventually dying because of conflicts. I understand that. The packager was designed to be at the root level of an app, not a monorepo.
The idea experience for me
Up until 40, I've been just developing my 3rd party stuff right in the node_modules. I'd then manually copy & paste the stuff out. It felt bad (like editing files in production) and I cried myself to sleep, but you know what? It worked. Live reloading worked too. But now, the packager has to be restarted.
It'd be great if we could use npm link to drive this. It'd feel like normal node module development.
I haven't explored rn-cli.config.js blacklisting & package roots yet.
I also volunteer as tribute if you need people to test stuff.
Any ideas when this would be resolved?
Any ideas when this would be resolved?
Hey guys,
I got tired with dealing with the symlink handling so I forked react-native and patched the packager to fix the issues I was having. This might not fix everyones issues but it has certainly fixed mine. The issue that I was having (and have had since the dawn of time in using react-native) was caused by the packagers inability of discovering peer dependencies on symlinked modules (npm link or yarn link). I've updated the resolver for the packager to be aware of the root paths of the project and automatically add these paths to the search query used to resolve modules. The prior implementation would try to be smart about resolving the paths based on the modules real path which breaks on symlinked modules.
Take it or leave it, but you are all welcome to use it. I've patched both the 0.42-stable branch and the 0.43-stable branch but haven't really tested 0.43 since it relies on an alpha version of react.
To use 0.42-stable update package.json react-native dependency to:
"react-native": "https://github.com/braunsquared/react-native/tarball/b576d07039dfaa0045fcff8b463f7490a86a7465"
For 0.43-stable use:
"react-native": "https://github.com/braunsquared/react-native/tarball/843be3173f617de0e13fc219f1aa2a809b94d1a3"
Not sure but probably node option --preserve-symlinks can help you somehow. https://nodejs.org/dist/latest-v6.x/docs/api/cli.html#cli_preserve_symlinks
I had similar problem with symlinked files/modules (but on regular node project) and this option resolves problems for me.
We definitely need to find way how to run React Native Packager with --preserve-symlinks option.
Any update on this? I was surprised to see npm and just normal *nix symlinks dont work at all
+1
+1
+1
+1
+1