Tips for building on mac? No luck with binary or docker thus far
S4lt5 opened this issue · comments
Currently using:
macos montery
node v18.11.0
rancher desktop 1.6.1 for container build
When I run yarn run release with mac or linux, I fail with varying errors.
> yarn run release --platform linux
✔ nx run models:build [existing outputs match the cache, left as is]
✔ nx run graph:build [existing outputs match the cache, left as is]
✔ nx run parser:build [existing outputs match the cache, left as is]
✔ nx run client:build:production [existing outputs match the cache, left as is]
✔ nx run server:build [existing outputs match the cache, left as is]
✖ nx run server:release
> pkg@5.8.0
/Users/[REDACTED]/.pkg-cache/v3.4/fetched-v16.16.0-macos-x64-signed: No such file or directory
> Error! Cannot generate bytecode
pkg fails to run "codesign" utility. Due to the mandatory signing
requirement of macOS, executables must be signed. Please ensure the
utility is installed and properly configured.
Same issue when building --platform mac
It's worth noting that I do have codesign in my PATH
Docker gets farther, and appears to build my version, but fails when running it.
>docker compose build
...
...
=> ERROR [redeye-core:latest redeye-linux-builder 5/5] RUN yarn run release --platform=linux 39.6s
------
> [redeye-core:latest redeye-linux-builder 5/5] RUN yarn run release --platform=linux:
#0 4.290
#0 4.292 > NX Running target release for 2 project(s) and 5 task(s) they depend on:
#0 4.292
#0 4.292 - parser
#0 4.292 - server
#0 4.292
#0 4.292 With additional flags:
#0 4.292 --platform=linux
#0 4.292
#0 4.293
#0 4.983
#0 4.984 > nx run models:build [remote cache]
#0 4.984
#0 4.984 Compiling with SWC for models...
#0 4.984 Successfully compiled: 27 files with swc (67.66ms)
#0 5.008
#0 5.008 > nx run graph:build [remote cache]
#0 5.008
#0 5.008 vite v3.1.3 building for production...
#0 5.008 transforming...
#0 5.008 ✓ 28 modules transformed.
#0 5.008 rendering chunks...
#0 5.008 ../../dist/packages/graph/graph.es.js 69.13 KiB / gzip: 17.36 KiB
#0 5.008 ../../dist/packages/graph/assets/index.7f7ce887.css 973.22 KiB / gzip: 736.92 KiB
#0 5.008 ../../dist/packages/graph/graph.umd.js 57.00 KiB / gzip: 16.07 KiB
#0 5.008 Copying asset files...
#0 5.008 Done copying asset files.
#0 5.008 Bundle complete.
#0 5.409
#0 5.409 > nx run parser:build [remote cache]
#0 5.409
#0 5.409 Compiling with SWC for parser...
#0 5.409 Successfully compiled: 41 files with swc (47.54ms)
#0 5.609
#0 5.609 > nx run client:build:production [remote cache]
#0 5.609
#0 5.609 vite v3.1.3 building for production...
#0 5.609 transforming...
#0 5.609 transforming...
#0 5.609 ✓ 9 modules transformed.
#0 5.609 rendering chunks...
#0 5.609 ✓ 3790 modules transformed.
#0 5.609 rendering chunks...
#0 5.609 ../../dist/applications/client/assets/Logo-Dark.cca6b38e.svg 7.72 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-sans-all-400.205b5e5a.woff2 51.82 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-sans-all-400-italic.f8bbd0e3.woff2 56.15 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-sans-all-500.1212e7ab.woff2 55.14 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-sans-all-500-italic.c62c7ee9.woff2 59.40 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-sans-all-600.d8b4efc9.woff2 55.66 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-sans-all-600-italic.3778adf3.woff2 58.68 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-sans-all-700.36fc9410.woff2 51.78 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-sans-all-700-italic.ac0eed09.woff2 55.39 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-mono-all-400.0581085d.woff2 32.63 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-mono-all-400-italic.738db8c6.woff2 36.53 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-mono-all-500.10aef5a3.woff2 33.51 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-mono-all-500-italic.91e97a3c.woff2 37.52 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-mono-all-600.01f3197a.woff2 34.47 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-mono-all-600-italic.36e707a0.woff2 38.11 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-mono-all-700.25a2510f.woff2 33.80 KiB
#0 5.609 ../../dist/applications/client/assets/ibm-plex-mono-all-700-italic.88c19110.woff2 36.98 KiB
#0 5.609 ../../dist/applications/client/assets/file-worker.348138a2.js 1.90 KiB
#0 5.609 ../../dist/applications/client/index.html 0.87 KiB
#0 5.609 ../../dist/applications/client/assets/index.cbda0dd0.css 303.86 KiB / gzip: 35.74 KiB
#0 5.609 ../../dist/applications/client/assets/vendor.00901628.js 306.09 KiB / gzip: 96.90 KiB
#0 5.609 ../../dist/applications/client/assets/index.05ec905a.js 3691.92 KiB / gzip: 797.72 KiB
#0 5.609 Bundle complete.
#0 6.194
#0 6.195 > nx run server:build [remote cache]
#0 6.196
#0 6.196 Compiling with SWC for server...
#0 6.196 Successfully compiled: 56 files with swc (66.29ms)
#0 14.02
#0 14.02 > nx run server:release --platform=linux
#0 14.02
#0 14.02 > pkg@5.8.0
#0 14.02 > Fetching base Node.js binaries to PKG_CACHE_PATH
#0 14.02
#0 14.02
#0 14.02 node:internal/fs/utils:347
#0 14.02 throw err;
#0 14.02 ^
#0 14.02
#0 14.02 Error: ENOENT: no such file or directory, stat '/root/.pkg-cache/v3.4/fetched-v16.16.0-linux-x64.downloading'
#0 14.02 at Object.statSync (node:fs:1583:3)
#0 14.02 at Object.statSync (/app/node_modules/graceful-fs/polyfills.js:318:34)
#0 14.02 at statSync (/app/node_modules/fs-extra/lib/util/stat.js:10:52)
#0 14.02 at getStatsSync (/app/node_modules/fs-extra/lib/util/stat.js:24:19)
#0 14.02 at Object.checkPathsSync (/app/node_modules/fs-extra/lib/util/stat.js:49:33)
#0 14.02 at Object.moveSync (/app/node_modules/fs-extra/lib/move-sync/move-sync.js:14:28)
#0 14.02 at WriteStream.<anonymous> (/app/node_modules/pkg-fetch/lib-es5/utils.js:90:56)
#0 14.02 at WriteStream.<anonymous> (node:internal/util:453:5)
#0 14.02 at WriteStream.onclose (node:internal/streams/end-of-stream:147:14)
#0 14.02 at WriteStream.emit (node:events:513:28) {
#0 14.02 errno: -2,
#0 14.02 syscall: 'stat',
#0 14.02 code: 'ENOENT',
#0 14.02 path: '/root/.pkg-cache/v3.4/fetched-v16.16.0-linux-x64.downloading'
#0 14.02 }
#0 14.02
#0 14.02 > NX ERROR: Something went wrong in run-commands - Command failed: pkg dist/applications/server/package.json -t node16-linux -o release/linux/RedEye
#0 14.02
#0 14.02 Pass --verbose to see the stacktrace.
#0 14.02
#0 39.23
#0 39.23 > nx run parser:release --platform=linux
#0 39.23
#0 39.24 > pkg@5.8.0
#0 39.24 > Fetching base Node.js binaries to PKG_CACHE_PATH
#0 39.24
#0 39.24
#0 39.24 > Warning Failed to make bytecode node16-x64 for file /snapshot/app/node_modules/supports-color/index.js
#0 39.24
#0 39.24
#0 39.24
#0 39.24 > NX Running target "release" failed
#0 39.25
#0 39.25 Failed tasks:
#0 39.25
#0 39.25 - server:release
#0 39.25
#0 39.41 Nx Cloud made it possible to reuse 5 tasks: https://nx.app/runs/CjpHlyxcIr
#0 39.41
------
failed to solve: executor failed running [/bin/sh -c yarn run release --platform=linux]: exit code: 1I think my current plan is try to build the mac binaries in the a container build step and copy them out.
I would like to add that I can run in dev just fine, but building still fails in all cases for me.
Are you building on an M1 Mac? We've seen this problem once before with the codesign failing on M1 Macs when building the x64 version but it was isolated at the time.
We're also investigating a potential issue on Ubuntu (even in Docker) when initially downloading the pkg binaries. We haven't been able to reproduce the problem on our end yet, but we're zeroing in on a few candidates for where the process might be failing. I'll update this ticket once we've found a solution!
I'm building on an intel-based mac pro, if you have some hints I'd be happy to dig in and try to help figure it out.
I was able to reproduce on an intel mac also AFTER building the mac release.. to fix temporarily:
rm -rf redeyegit clone https://github.com/cisagov/RedEye && cd RedEyegit switch mainnpm install -g yarnyarn installyarn release:all --verbose
did the trick for me to build on all platforms including linux.
Still running into the docker build issue, when I did it manually in the build image It looked like the download completed before the check occured, thus it looks for '/root/.pkg-cache/v3.4/fetched-v16.16.0-linux-x64.downloading when it is already downloaded.
I think this might be a general package manager bug, will run it down further.
can confirm i'm still experiencing this issue using docker-compose on current develop branch
I have been running this down today. It's something going wrong with the pkg fetch (or lack of waiting).
I can confirm that debugging the containers manually the pkg-fetch commands work flawlessly when run via CLI in terminal.
I can also confirm that my failures happen when the release command runs, and tries to interact with the temporary .downloading file. Why? I have not grok'd enough of the install process to understand.
However, the following Dockerfile naively fixes the current problem by explicitly running pkg fetch first.
FROM node:16 as redeye-builder
WORKDIR /app
COPY ./ ./
RUN yarn install --immutable --inline-builds
RUN npx pkg-fetch --platform mac --node-range node16
RUN npx pkg-fetch --platform linux --node-range node16
RUN npx pkg-fetch --platform win --node-range node16
RUN yarn run release:all
FROM node:16 as redeye-linux-builder
WORKDIR /app
COPY ./ ./
RUN yarn install --immutable --inline-builds
RUN npx pkg-fetch --platform linux --node-range node16
RUN yarn run release --platform=linux
### CORE IMAGE ###
FROM ubuntu as redeye-core
WORKDIR /app
COPY --from=redeye-linux-builder /app/release/linux .
ENTRYPOINT [ "/bin/bash", "-l", "-c" ]
CMD ["./RedEye"]I'll try to get a better fix by adjusting the actual project configs if I can figure it out.
I tested running the docker-compose script in develop on a clean M1 Mac and Intel Mac today with no errors or issues.
@Yablargo, I'm glad to see that fixed it for you! The PR I merged that closed this issue added the pkg-fetch calls in our release commands. Do you have the latest code from develop pulled down and still run into the same problem?
Yes.
I think the key is the following: Error: ENOENT: no such file or directory, stat '/root/.pkg-cache/v3.4/fetched-v16.16.0-linux-x64.downloading'
That "downloading" file only exists when the package is actively fetching, and then disappears afterwards. When I manually went into the container and did the commands in order, I got the same error 100% of the time using both the docker desktop and rancher desktop implementations.
When I explicitly did a npx pkg-fetch the problem went away.
I can also confirm that AFTER the job fails, the fetched file is there, and no longer in a .downloading state.
Hmm, want to add that nope, the fetch command itself fails with that error, the plot thickens.
> NX ERROR: Something went wrong in run-commands - Command failed: pkg-fetch --platform mac --node-range node16
#0 21.53@Yablargo Because this problem appears to pop up under specific environments. Would you be willing to create a PR with your Dockerfile changes that solved the issue for you?
Sure. I'll keep trying to figure out the root cause, but I'd be happy to submit a PR in the meanwhile.
Trying to get to the root of this:
const tempFile = `${file}.downloading`;
fs.mkdirpSync(path.dirname(tempFile));
const ws = fs.createWriteStream(tempFile);
const totalSize = Number(res.headers.get('content-length'));
let currentSize = 0;
res.body.on('data', (chunk: Buffer) => {
if (totalSize != null && totalSize !== 0) {
currentSize += chunk.length;
log.showProgress((currentSize / totalSize) * 100);
}
});
res.body.pipe(ws);
return new Promise<void>((resolve, reject) => {
stream.finished(ws, (err) => {
if (err) {
log.disableProgress();
fs.rmSync(tempFile);
reject(wasReported(`${err.name}: ${err.message}`));
} else {
log.showProgress(100);
log.disableProgress();
fs.moveSync(tempFile, file);
resolve();
}
});
});
}The promise finishes, no error is thrown, but there is no file and nothing written there. Scratching my head on this one so far.
fs.moveSync(tempfile,file) fails because tempFile does not exist.
If I had to commit to a source right now, I would guess that fs.createWriteStream() is not syncronous and is not awaited, but I have to read the docs more.
I looked into this at the pkg-fetch level and I can't wrap my head around what is going on. The only thing that seems to make a little sense is the fetch somehow running twice, at the same time, and one picks up the file and moves it and the 2nd instance fails.
fs.createWriteStream(tempFile) resolves instantly and is not async, and then at fs.moveSync() the file is already gone.
I think when I checked after running the file was already moved, so that might be actually what is happening.
After the latest merge, I can confirm that I can build without any modification of the source develop branch via docker(/compose)
@GoldingAustin I think I finally found the root cause of this...
They're running at the exact same time, both fetching, and one wins the race.
I messed with a debugging pkg-fetch and I could NOT reproduce the effect any other way than running two at the exact same time.
I can confirm that the paralell build is ultimately what is causing the problems.
"release": "yarn nx run-many --parallel=1 --target=release --all",
"release:all": "yarn nx run-many --parallel=1 --target=release-all --all",fixes the build for me. However, this is probably worse since builds will in general run much slower. I am gonna make a public POC and submit to pkg-fetch
I put a repro up at https://github.com/Yablargo/pkgfetch-262-repro
https://github.com/Yablargo/pkgfetch-262-repro/actions/runs/3498583178/jobs/5859132361
And got it to fail publically in their runner.
I opened vercel/pkg-fetch#262 and linked the repro there.
@GoldingAustin I don't think the original authors of pkg-fetch really care about the error based on activity in their repo, so I would go with the original fix!