Add suport for Bun runtime - Issue with `net.Socket.write`
JuroOravec opened this issue · comments
Hi, I have a web scraper built on top of the Crawlee framework. I wanted to run it with Bun instead of Node. However, it failed, and the stack trace (at the bottom) led to proxy-chain
.
I don't think the issue is with Bun. I tried running the following script with Bun, and both new Socket()
and socket.write
were defined:
const { Socket } = require('net');
console.log({
socket: new Socket(),
write: (new Socket()).write,
});
Error stack trace:
2 | Object.defineProperty(exports, "__esModule", { value: true });
3 | exports.customConnect = void 0;
4 | const tslib_1 = require("tslib");
5 | const net_1 = tslib_1.__importDefault(require("net"));
6 | const util_1 = require("util");
7 | const asyncWrite = (0, util_1.promisify)(net_1.default.Socket.prototype.write);
^
TypeError: undefined is not an object (evaluating 'net_1.default.Socket.prototype.write')
at /Users/presenter/repos/proj-name/node_modules/proxy-chain/dist/custom_connect.js:7:41
at globalThis (/Users/presenter/repos/proj-name/node_modules/proxy-chain/dist/custom_connect.js:26:24)
at require (:1:20)
at /Users/presenter/repos/proj-name/node_modules/proxy-chain/dist/server.js:21:6
at globalThis (/Users/presenter/repos/proj-name/node_modules/proxy-chain/dist/server.js:546:17)
at require (:1:20)
at /Users/presenter/repos/proj-name/node_modules/proxy-chain/dist/index.js:5:8
at globalThis (/Users/presenter/repos/proj-name/node_modules/proxy-chain/dist/index.js:8:59)
at require (:1:20)
at /Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/anonymize-proxy.js:4:6
at globalThis (/Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/anonymize-proxy.js:33:30)
at require (:1:20)
at /Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/puppeteer/puppeteer-controller.js:7:6
at globalThis (/Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/puppeteer/puppeteer-controller.js:111:30)
at require (:1:20)
at /Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/puppeteer/puppeteer-plugin.js:7:6
at globalThis (/Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/puppeteer/puppeteer-plugin.js:156:26)
at require (:1:20)
at /Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/fingerprinting/hooks.js:4:6
at globalThis (/Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/fingerprinting/hooks.js:87:35)
at require (:1:20)
at /Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/browser-pool.js:14:6
at globalThis (/Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/browser-pool.js:591:22)
at require (:1:20)
at /Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/index.js:29:8
at globalThis (/Users/presenter/repos/proj-name/node_modules/@crawlee/browser-pool/index.js:48:129)
at require (:1:20)
at /Users/presenter/repos/proj-name/node_modules/@crawlee/browser/internals/browser-crawler.js:7:6
at globalThis (/Users/presenter/repos/proj-name/node_modules/@crawlee/browser/internals/browser-crawler.js:416:37)
at require (:1:20)
at /Users/presenter/repos/proj-name/node_modules/@crawlee/browser/index.js:5:8
at globalThis (/Users/presenter/repos/proj-name/node_modules/@crawlee/browser/index.js:6:69)
ENV:
macOS 13.2.1
node: v16.13.0
Dependencies:
"proxy-chain@2.3.0"
"apify": "^3.1.4",
"apify-client": "^2.7.1",
"cheerio": "^1.0.0-rc.12",
"crawlee": "^3.3.1",
Bun is not fully compatible with Node. See https://bun.sh/docs/runtime/nodejs-apis#node-net, where they write:
If you run into any bugs with a particular package, please open an issue. Opening issues for compatibility bugs helps us prioritize what to work on next.
So I'd recommend doing that, we can't fix it here...
@jancurn Please don't judge so fast and have a look at the error I posted.
The error said that undefined is not an object (evaluating 'net_1.default.Socket.prototype.write')
.
But in my test, the (new Socket()).write
function was defined in Bun. So it didn't seem to be an issue on the Bun side, implying that the issue is in proxy-chain
.
What's more, I think I just found the issue, and it's here this line 5:
const asyncWrite = promisify(net.Socket.prototype.write);
Which is then called here on line 14
await asyncWrite.call(socket, 'HTTP/1.1 200 Connection Established\r\n\r\n');
For some reason, net.Socket.prototype
is undefined
in Bun, so net.Socket.prototype.write
throws the error.
However, (new net.Socket()).write
is defined, and following:
new Socket().write('HTTP/1.1 200 Connection Established\r\n\r\n')
returns true
.
So that's what I think the issue is. However, I haven't worked with Sockets before, and I'm not 100% sure what's the prurpose of that file, so I don't know if the behaviour of new net.Socket().write
in Bun is the same as net.Socket.prototype.write
in Node. But common sense suggests that it should be.
Further updates:
-
Locally, I've replaced
net.Socket.prototype.write
withnew net.Socket().write
, andproxy-chain
wasn't causing errors anymore. -
Next up, there was an error with
node_modules/@crawlee/browser-pool/proxy-server.js
with lineserver.server.unref();
I looked into it. The
unref
should refers tohttp.Server.unref
. For some reason, this isn't define in Bun, and this seems to be genuine error on their side (it's not even reported in their docs). -
Out of curiosity, I just commented out that line, to see if I get the crawler to work. It printed the initial log with system info
INFO System info {"apifyVersion":"3.1.4","apifyClientVersion":"2.7.1","crawleeVersion":"3.3.1","osType":"Darwin","nodeVersion":"v18.15.0"}
However, the run still ended in an error. Here, the
promises_1.opendir
refer tofs.promises.opendir
(node:fs). Unfortunately, none of theopendir
functions are currently defined Bun (fs.opendirSync
, fs.opendir,
fs.promises.opendir`).ERROR (0, promises_1.opendir) is not a function. (In '(0, promises_1.opendir)(keyValueStoreDir)', '(0, promises_1.opendir)' is undefined) TypeError: (0, promises_1.opendir) is not a function. (In '(0, promises_1.opendir)(keyValueStoreDir)', '(0, promises_1.opendir)' is undefined) at <anonymous> (/Users/presenter/repos/apify-actor-facebook/node_modules/@crawlee/memory-storage/cache-helpers.js:110:25)
So to sum up:
- Yes, the ticket can remain closed, it's currently not possible to run Apify crawlers with Bun, because of (at least) 2 unsupported features.
- For future refernce, once / if this becomes relevant, then the first issue can be resolved by replacing
net.Socket.prototype.write
withnew net.Socket().write
.
Sorry, you're right. I think we just need to get rid of the problematic line and change the code of the customConnect
function in https://github.com/apify/proxy-chain/blob/master/src/custom_connect.ts to something like this:
const asyncWrite = util.promisify(socket.write).bind(socket);
await asyncWrite.call(socket, 'HTTP/1.1 200 Connection Established\r\n\r\n');
Would you care to create a pull request?
I made a PR for the socket one (#522), since I'm already in the flow. Couldn't verify the tests. I leave it up to you to decide whether it should go in or not. Have a nice evening!
I couldn't resist testing further, so just summarizing what I learnt:
-
I managed to get start a Playwright crawler in Bun with following changes to the Apify packages:
- I commented out the
server.server.unref();
in@crawlee/browser-pool/proxy-server.js
- I replaced
fs.promises.opendir(dirName)
withfs.promises.readdir(dirName, { withFileTypes: true })
in@crawlee/memory-storage/cache-helpers.js
- NOTE: Good thing is that with the
withFileTypes: true
option, bothopendir
andreaddir
resolve to an iterable of Dirent. Bad thing, from my understandingopendir
yields the entries one-by-one as they are found, whereasreaddir
resolves only once all items have been found. So replacingopendir
withreaddir
might add extra waiting time.
- NOTE: Good thing is that with the
- I commented out the
-
With changes in step 1., I managed to start a Playwright crawler, to the point where Playwright command was executed. Afterwards, there is an issue on Playwright side with
child_process.spawn
. You can find more about that issue here:
Many thanks for the analysis! Please can you post this to https://github.com/apify/crawlee/issues instead? Otherwise the Crawlee team will not look into it...
Closing this issue here for now