mlc-ai / web-llm

High-performance In-browser LLM Inference Engine

Home Page:https://webllm.mlc.ai

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Engine not instantiating for WebWorker

kitzj opened this issue · comments

commented

I got web-llm successfully installed in Rails no problem, and porting the "get-started" example into my Rails app works perfectly. However, when I try to emulate the "get-started-web-worker" example, the engine fails to instantiate. (both I just removed types to make it javascript)

The thing is the "get-started-web-worker" example works correctly out of the box using the official example, so it's strange that the exact same code in Rails only works for the non-web-worker.

As you can see in the "all_logs" screenshot, the "Before G" with the WebWorkerEngine gets logged. The other log is "End of promise" indicating that getPromise successfully (?) returns B, but for some reason the log after yield this.getPromise(g) ("End of promise") never logs. Which seems weird if getPromise seems to run fine.

It does seem strange that there isn't any error that was returned. The only thing I could see as a potential issue is within the WebWorkerEngine (see "pending_promise.png" screenshot) where the value of the caller gets logged as a TypeError (below). This is also strange to see as Rails compiled the web-llm package as a .js file so I think there shouldn't be any type errors.

TypeError: 'caller', 'callee', and 'arguments' properties may not be accessed on strict mode functions or the arguments objects for calls to them
at Function.invokeGetter (:3:28)

Any ideas what the issue might be here?

Here is the respective code:
`// Worker.js
import { EngineWorkerHandler, Engine } from "@mlc-ai/web-llm";

// Hookup an engine to a worker handler
const engine = new Engine();
const handler = new EngineWorkerHandler(engine);
self.onmessage = (msg) => {
handler.onmessage(msg);
};

// Main.js
const engine = await webllm.CreateWebWorkerEngine(
/worker=/ new Worker(new URL("./worker.js", import.meta.url), {
type: "module",
}),
/modelId=/ selectedModel,
/engineConfig=/ { initProgressCallback: initProgressCallback }
);

// @mlc-ai--web-llm.js
/ function CreateWebWorkerEngine(A, Q, B) {
return __awaiter(this, void 0, void 0, function
() {
const g = new WebWorkerEngine(A);

g.setInitProgressCallback(
  B === null || B === void 0 ? void 0 : B.initProgressCallback
);

console.log("BEFORE G", g);

yield g.reload(
  Q,
  B === null || B === void 0 ? void 0 : B.chatOpts,
  B === null || B === void 0 ? void 0 : B.appConfig
);

return g;

});
}

getPromise(A) {
const Q = A.uuid;
const executor = (A, B) => {
const cb = (Q) => {
Q.kind == "return"
? A(Q.content)
: Q.kind != "throw"
? B("Uknown msg kind " + Q.kind)
: B(Q.content);
};
this.pendingPromise.set(Q, cb);
};
const B = new Promise(executor);
this.worker.postMessage(A);
console.log("End of promise");
return B;
}
reload(A, Q, B) {
return __awaiter(this, void 0, void 0, function* () {
const g = {
kind: "reload",
uuid: crypto.randomUUID(),
content: { modelId: A, chatOpts: Q, appConfig: B },
};
yield this.getPromise(g);
console.log("End of reload");
});
}`

pending_promise
all_logs