Figure out why some deploys aren't working
kentcdodds opened this issue · comments
I think it's because they're just so big, but they are exiting with exit code 1 during startup in the deployment so I really have no idea why... Will probably have to try and run docker locally to see what's up.
- https://github.com/epicweb-dev/data-modeling - this one seemed to start working when I made the seed data smaller. So that's why I'm thinking it's because they're too big.
- https://github.com/epicweb-dev/web-auth - this one's huge and no amount of making the dbs smaller will help that. It's just big. It should definitely still run though.
since deploys app can not run any playground or exercises app, maybe you can modify workshops deploy.yml to skip all
exercises npm install skip all exercises scripts
The deploy.yml isn't a problem here. It creates the docker images just fine and even starts the app without issue. It's just that it doesn't stay started...
Actually, I just noticed that we're maybe hitting a size limit or something: https://community.fly.io/t/deploy-timing-out-but-actually-succeeding/14384/7?u=kentcdodds
what i was meant was that on deployed you can modify setup.js script to only install the workshop dependencies, you don't need to install any exercise dependencies. you can also skip most of setup-custom.js
Ok, let's see if this works: epicweb-dev/web-auth@57a02f0
I'm going to sleep. I'll find out in the morning. 😅
this command will do nothing if globstar is off
RUN rm -rf ./exercises/**/{data.db,package.json}
#9 [5/6] RUN rm -rf ./exercises/**/{data.db,package.json}
#9 DONE 0.3s
#10 [6/6] RUN npm install --omit=dev
#10 61.01
#10 61.01 added 1101 packages, and audited 1343 packages in 1m
maybe you need to add RUN shopt -s globstar before you run it?
Interesting. It should be removed in the github action anyway.
In any case, I had an idea last night that maybe I could make it more like this:
Instead of adding all the content to the docker image, we just make the app download the repo on startup at runtime. I think GitHub has an API we could use to download a zip of the repo and then we unzip it and set the KCDSHOP_CONTEXT_CWD to the unzipped location. I'm pretty sure that should work and the image should be much smaller.
Interesting. It should be removed in the github action anyway.
No, added 1101 packages indicate that it does not remove package.json files
when i test it locally only 472 packages installed
$ rm -rf ./exercises/**/{data.db,package.json}
$ npm install --omit=dev
added 472 packages, and audited 714 packages in 10s
Alright, I'll give it a shot 👍
Yeah, that made it much smaller. But I think it's still too big because it still failed to start up.
I think we need to have a runtime solution. Just a script that downloads and extracts a zip of the project somewhere so all the projects don't have to be in the image.
I'm really happy with this solution I developed (with ChatGPT). I think I want to adopt this with the other repos because I think it would be faster. The image is now closer to 600MB which is much better than 7.9GB 😅
Unfortunately turns out the size of the image was probably never the issue. Instead it's probably something wrong with having that number of projects which causes things to go wrong. If you check the ChatGPT conversation, you'll find logs like this:
2023-08-05T01:03:42.653 app[9080554b003287] sjc [info] 2023/08/05 01:03:42 listening on [fdaa:0:23df:a7b:b2e2:1f93:e7ac:2]:22 (DNS: [fdaa::3]:53)
2023-08-05T01:03:42.927 app[9080554b003287] sjc [info] setting up swapfile...
2023-08-05T01:03:42.927 app[9080554b003287] sjc [info] swapfile already exists
2023-08-05T01:03:43.390 app[9080554b003287] sjc [info] Cloning into '/myapp/workshop-content'...
2023-08-05T01:03:55.459 app[9080554b003287] sjc [info] Updating files: 10% (1883/17917) Updating files: 11% (1971/17917) Updating files: 12% (2151/17917) Updating files: 13% (2330/17917) Updating files: 14% (2509/17917) Updating files: 15% (2688/17917) Updating files: 16% (2867/17917) Updating files: 17% (3046/17917) Updating files: 18% (3226/17917) Updating files: 19% (3405/17917) Updating files: 20% (3584/17917) Updating files: 21% (3763/17917) Updating files: 22% (3942/17917) Updating files: 22% (4048/17917) Updating files: 23% (4121/17917) Updating files: 24% (4301/17917) Updating files: 25% (4480/17917) Updating files: 26% (4659/17917) Updating files: 27% (4838/17917) Updating files: 28% (5017/17917) Updating files: 29% (5196/17917) Updating files: 30% (5376/17917) Updating files: 31% (5555/17917) Updating files: 32% (5734/17917) Updating files: 33% (5913/17917) Updating files: 34% (6092/17917) Updating files: 35% (6271/17917) Updating files: 36% (6451/17917) Updating files: 37% (6630/17917) Updating files: 38% (6809/17917) Updating files: 38% (6981/17917) Updating files: 39% (6988/17917) Updating files: 40% (7167/17917) Updating files: 41% (7346/17917) Updating files: 42% (7526/17917) Updating files: 43% (7705/17917) Updating files: 44% (7884/17917) Updating files: 45% (8063/17917) Updating files: 46% (8242/17917) Updating files: 47% (8421/17917) Updating files: 48% (8601/17917) Updating files: 49% (8780/17917) Updating files: 50% (8959/17917) Updating files: 51% (9138/17917) Updating files: 52% (9317/17917) Updating files: 53% (9497/17917) Updating files: 54% (9676/17917) Updating files: 55% (9855/17917) Updating files: 55% (10007/17917) Updating files: 56% (10034/17917) Updating files: 57% (10213/17917) Updating files: 58% (10392/17917) Updating files: 59% (10572/17917) Updating files: 60% (10751/17917) Updating files: 61% (10930/17917) Updating files: 62% (11109/17917) Updating files: 63% (11288/17917) Updating files: 64% (11467/17917) Updating files: 65% (11647/17917) Updating files: 66% (11826/17917) Updating files: 67% (12005/17917) Updating files: 68% (12184/17917) Updating files: 69% (12363/17917) Updating files: 70% (12542/17917) Updating files: 71% (12722/17917) Updating files: 71% (12786/17917) Updating files: 72% (12901/17917) Updating files: 73% (13080/17917) Updating files: 74% (13259/17917) Updating files: 75% (13438/17917) Updating files: 76% (13617/17917) Updating files: 77% (13797/17917) Updating files: 78% (13976/17917) Updating files: 79% (14155/17917) Updating files: 80% (14334/17917) Updating files: 81% (14513/17917) Updating files: 82% (14692/17917) Updating files: 83% (14872/17917) Updating files: 84% (15051/17917) Updating files: 85% (15230/17917) Updating files: 86% (15409/17917) Updating files: 87% (15588/17917) Updating files: 87% (15665/17917) Updating files: 88% (15767/17917) Updating files: 89% (15947/17917) Updating files: 90% (16126/17917) Updating files: 91% (16305/17917) Updating files: 92% (16484/17917) Updating files: 93% (16663/17917) Updating files: 94% (16842/17917) Updating files: 95% (17022/17917) Updating files: 96% (17201/17917) Updating files: 97% (17380/17917) Updating files: 98% (17559/17917) Updating files: 99% (17738/17917) Updating files: 100% (17917/17917) Updating files: 100% (17917/17917), done.
2023-08-05T01:03:56.198 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 6, heapUsed: 6, rss: 46 }
2023-08-05T01:03:56.494 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 21, heapUsed: 13, rss: 64 }
2023-08-05T01:03:56.699 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 25, heapUsed: 17, rss: 74 }
2023-08-05T01:03:56.931 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 27, heapUsed: 20, rss: 77 }
2023-08-05T01:03:57.893 app[9080554b003287] sjc [info] 🐨 Let's get learning!
2023-08-05T01:03:57.894 app[9080554b003287] sjc [info] Local: http://localhost:8080
2023-08-05T01:03:57.894 app[9080554b003287] sjc [info] On Your Network: http://172.19.133.114:8080
2023-08-05T01:03:57.894 app[9080554b003287] sjc [info] Press Ctrl+C to stop
2023-08-05T01:03:57.897 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 47, heapUsed: 37, rss: 100 }
2023-08-05T01:03:58.142 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 52, heapUsed: 45, rss: 107 }
2023-08-05T01:03:58.366 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 58, heapUsed: 49, rss: 112 }
2023-08-05T01:03:58.586 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 65, heapUsed: 58, rss: 119 }
2023-08-05T01:03:58.805 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 76, heapUsed: 67, rss: 131 }
2023-08-05T01:03:59.028 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 71, heapUsed: 58, rss: 126 }
2023-08-05T01:03:59.245 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 74, heapUsed: 65, rss: 130 }
2023-08-05T01:03:59.404 health[9080554b003287] sjc [info] Health check on port 8080 is now passing.
2023-08-05T01:03:59.540 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 75, heapUsed: 66, rss: 131 }
2023-08-05T01:03:59.876 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 86, heapUsed: 67, rss: 144 }
2023-08-05T01:04:00.170 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 93, heapUsed: 84, rss: 151 }
2023-08-05T01:04:00.398 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 97, heapUsed: 66, rss: 155 }
2023-08-05T01:04:00.637 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 97, heapUsed: 73, rss: 156 }
2023-08-05T01:04:00.898 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 100, heapUsed: 90, rss: 161 }
2023-08-05T01:04:01.649 app[9080554b003287] sjc [info] Memory (MB): { heapTotal: 124, heapUsed: 113, rss: 190 }
2023-08-05T01:04:02.351 app[9080554b003287] sjc [info] fatal error: all goroutines are asleep - deadlock!
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] goroutine 1 [chan receive]:
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/helpers.(*ThreadSafeWaitGroup).Wait(...)
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/helpers/waitgroup.go:36
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] main.runService.func2()
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:114 +0x59
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] main.runService(0x1)
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:160 +0x4ed
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] main.main()
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/main.go:232 +0x9e9
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] goroutine 6 [chan receive]:
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] main.runService.func1()
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:98 +0x4a
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] created by main.runService
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:97 +0x1e5
2023-08-05T01:04:02.354 app[9080554b003287] sjc [info] goroutine 7 [chan receive]:
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] main.(*serviceType).sendRequest(0xc000023bc0, {0x8ddb00, 0xc0001c62a0})
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:192 +0xfa
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] main.runService.func3()
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:125 +0x39
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] created by main.runService
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:122 +0x31c
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] goroutine 8 [chan receive]:
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler.(*scanner).scanAllDependencies(0xc00004d400)
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler/bundler.go:1832 +0x232
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler.ScanBundle({_, _, _, _, _, _}, {_, _}, _, {0xc0000f1ef0, ...}, ...)
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler/bundler.go:1220 +0xb3c
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api.rebuildImpl({0xc000064420, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, 0x1, 0x0, ...}, ...}, ...)
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api/api_impl.go:1418 +0x226
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api.(*internalContext).rebuild(_)
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api/api_impl.go:961 +0x2a5
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api.(*internalContext).Rebuild(...)
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api/api_impl.go:1022
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api.Build({0x0, 0x4, 0x0, 0xc000023fb0, 0x0, {0x0, 0x0}, 0x0, 0x0, {0x0, ...}, ...})
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api/api.go:378 +0xbb
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] main.(*serviceType).handleBuildRequest(0xc000023bc0, 0x0?, 0xc000023c50?)
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:760 +0xed8
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] main.(*serviceType).handleIncomingPacket.func2()
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:235 +0x68
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] created by main.(*serviceType).handleIncomingPacket
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:233 +0x1c5
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] goroutine 12 [chan receive]:
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] main.(*serviceType).sendRequest(0xc000023bc0, {0x8ddb00, 0xc0001c6330})
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:192 +0xfa
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] main.(*serviceType).convertPlugins.func2.4({{0xc000028320, 0x49}, {0x94a286, 0x4}, {0x0, 0x0}, {0x0, 0x0}})
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/cmd/esbuild/service.go:1051 +0x626
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api.(*pluginImpl).onLoad.func1({{0x0, 0x0}, {{0xc000028320, 0x49}, {0x94a286, 0x4}, {0x0, 0x0}, 0x0}})
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/pkg/api/api_impl.go:1932 +0x19d
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler.runOnLoadPlugins({0xc000064480?, 0x1, 0x40d2c8?}, 0xc000064480?, {0xa296b8, 0xc000062420?}, 0x91ad00?, {0xc0000fc690, 0xc0000104e0, 0xc0000104f8, ...}, ...)
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler/bundler.go:910 +0xe33
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler.parseFile({{0xa296b8, 0xc000062420}, {0xc0000fc690, 0xc0000104e0, 0xc0000104f8, 0xc000062400, 0x4, 0xc0000f0450}, 0xc00004d800, 0xc000064420, ...})
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler/bundler.go:135 +0x227
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] created by github.com/evanw/esbuild/internal/bundler.(*scanner).maybeParseFile
2023-08-05T01:04:02.355 app[9080554b003287] sjc [info] github.com/evanw/esbuild/internal/bundler/bundler.go:1360 +0xaaa
2023-08-05T01:04:02.670 app[9080554b003287] sjc [info] INFO Main child exited normally with code: 1
2023-08-05T01:04:02.670 app[9080554b003287] sjc [info] INFO Starting clean up.
2023-08-05T01:04:02.671 app[9080554b003287] sjc [info] WARN hallpass exited, pid: 257, status: signal: 15 (SIGTERM)
2023-08-05T01:04:02.675 app[9080554b003287] sjc [info] 2023/08/05 01:04:02 listening on [fdaa:0:23df:a7b:b2e2:1f93:e7ac:2]:22 (DNS: [fdaa::3]:53)
2023-08-05T01:04:03.670 app[9080554b003287] sjc [info] [ 21.467217] reboot: Restarting system
esbuild is just really unhappy. I remember having a similar issue on my website (evanw/esbuild#1608), but I added the same queuing solution to avoid esbuild being run more than once at a time and it was ineffective.
I'm at a total loss. I really want this to work because these deploys are very important for people wanting to quickly reference my material.
Try to generate all the mdx cache files from docker before runing start script
That thought occurred to me as well. It would require the huge docker image again, but if it gets this to work then that's better than not working at all!
Ok, I've posted an issue on esbuild: evanw/esbuild#3287
I'm just at a total loss
There are a couple ssh actions, you can shell into the actions runner and run the commands.
Try to cmpile mdx directly with @mdx-js/mdx
On deployed app instead of compiling all mdx files at statrtup. Do it only when each exercise start
Do it only when each exercise start
Unfortunately, we have to compile everything up front because it's how we know the titles of each exercise and step.
I just tried applying this clone-at-runtime solution to another workshop with large files (the database files mostly) and the improvement is very noticeable. The workshop's not quite half finished and it was already taking almost 7 minutes to deploy (with a 2.3 GB image) and startup. With the new approach, it now deploys and starts up in 2.5 minutes (with a 664 MB image).
So I definitely want to stick with this approach.
Just need to figure out evanw/esbuild#3287 and we'll be golden.
I figured it out! We were adding file watchers to all the files in the deployed version! I guess it was just too many files to watch and it was definitely unnecessary anyway, so I added a little code for skipping the watching in the deployed version and it works great now. I've applied this new deployment strategy to all the workshops now.