Giters

ngxson / wllama

WebAssembly binding for llama.cpp - Enabling in-browser LLM inference

Home Page:https://huggingface.co/spaces/ngxson/wllama

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Post on Reddit/r/LocalLlama?

flatsiedatsie opened this issue · comments

commented

Wllama is currently the only project that can run larger models like Mistral 7B in browsers that do not support WebGPU out of the box yet ( Safari, Firefox).

That's quite a feat, and newsworthy? Might be fun to create a post about it on Reddit?

commented

Yes it would be nice. I heard about this reddit but I don't use social network very often, I almost forgot about this.

Btw, do you think we should make a proper chat UI example before making a post of social platforms? (I also have LinkedIn)

I already have some reactjs code laying around from alpaca.cpp-webui so I just adapt it real quick for wllama (may take me 1 or 2 days to finish)

commented

That might be a good idea. But why people insist on using React I will never understand. IMHO it makes it so needlessly difficult to look at the actual code, and copy it. But that's up to you :-)

commented

I'd agree that from software engineer perspective, a more "structural" framework like angular would be a better choice. But the other half of my brain (the UX designer) wants a language that is easier to express the UX/UI. React turns out to be easy enough for non-tech designers to understand.

Tooling for reactjs is also easier which cuts down a lot of time (this is mostly personal preference - I still appreciate other frameworks like angular, vue, etc.)

commented

I've created a quick minimal demo for you:

https://flatsiedatsie.github.io/wllama_mistral_7b_demo/

Since your time and skills are valuable and rare, it makes more sense to spend it on developing Wllama itself, rather than creating web demos?

At the same time, building web demos is within my reach, and I would be happy to do so.

// added some small improvements

commented

@flatsiedatsie Thanks a lot! This is already a very good start.

You can already share your version on reddit if you want. I'll include a link on README to point to your site (or you can make a PR if you want; it can be put under "Demo" section)

I'm planning to do a full typescript + react (or something else but still have typescript) in the future. The reason is that personally I need a good UI with a manageable code structure to really test out new features. Maybe I'll focus on that in next week or the week after that.

For now as you said, I should invest my time into fixing memory problems.

commented

Feel free to add it to the readme. I'll can do a post on Reddit, sure.

The reason is that personally I need a good UI with a manageable code structure to really test out new features. Maybe I'll focus on that in next week or the week after that.

Ah, that makes sense.

commented

Perhaps this is of interest:
mlc-ai/web-llm#391

It's effort to integrate WebLLM into an existing chat UI that is pretty advanced. Who knows, perhaps WebLLM and Wllama could both be integrated, similar to my project. Win-win.