Neat idea. Regarding the performance, I think you could get a lot better performance by training a small classifier model, essentially an embedding model, and using the LLM as the distillation source. This would both be much smaller, addressing your desire for it to run in browser, while also being much more performant, addressing your quantization need. Using the full LLM is a bit overkill and you can extract the core of what you're looking for out of it with something a little custom.

This is great. But the vast majority of non-gaming non-terminal-loving-developers human beings want to avoid the keyboard at all costs.

They are doing it via trackpads, mousepads, touch screens, etc. Which are all inputs that transcend language or the ability to find meaningful words.

This is cool. Thanks for posting. I could see this adaptive UI / reactive intent UX become valuable for a couple use cases: 1. Tutorial modes for complicated UX that gradually introduces the user to new features as they "progress" through the website. 2. Warning users that their inputs could lead to unforeseen consequences. Think "you forgot to attach the file you talked about in the email" warning that Gmail already does.

This feels similar to the experience you get when asking ChatGPT a programming question. The UI shifts to better display code and provides buttons to copy content.

This is a very cool idea, I think this could potentially be more powerful by (at least initially) limiting it to small sections of the UI (Like you've done in your example)

You could take it a step further and, instead of adding/removing buttons, simply updating the copy of (a) button(s) based on the question, which I imagine you could manage with a very small model.

Cool demo and a neat write-up! I've been experimenting with similar patterns to differentiate between search queries/question answering and presenting specific outputs to the user.

Thanks for taking your time to write this up

Cool idea. FYI the GitHub link at the bottom leads to "page not found". Maybe the repo is not public?

Interesting, but also true that intent can be much more accurately passed with words, than inferred with guessing? Visit apple.com, reorganize this list in alphabet order, find me a ...

You get it

This looks like a neat idea, but I'm not too positive about it.

This makes using computers even harder to explain to people who do not spend their entire day keeping up with the latest developments. They cannot form a mental image or reuse any memory of what will happen next, because it is all context dependent.

On the other end of the spectrum, for power users, dynamically adapting user interfaces can also be quite annoying. One can't type ahead, or use shortcut keys, because one doesn't know what the context will be. Having to wait any positive amount of time for feedback is limiting.

Then again, there are probably tons of places where this is useful. I'm just a bit disappointed that we (as a society) haven't gotten the basics covered: programming still requires text files that can be sent to a matrix printer, and the latency of most applications is increasing instead of decreasing as computers become faster.

We're always coming back full circle: https://en.m.wikipedia.org/wiki/Ubiquity_(Firefox)

LLMs generating just-in-time UI has a lot of interest and effort going into it. It's usually called "generative UI" or "dynamic UI generation". It was a pretty hot about a year ago. Here's a HF blog on it https://huggingface.co/blog/airabbitX/llm-chatbots-30. Also check out Microsoft's Adaptive Cards. Nielsen Group wrote about it too. https://www.nngroup.com/articles/generative-ui/

The problem is it's hard to come up with better examples than your toy examples of weather and maps. Goodness there are so many travel planning demos. Who actually wants the context switch of a UI popping up mid-typed-sentence? Is a date picker really more convenient than typing "next school break"? Visualizations are interesting -- but that changes the framing from soliciting input to densifying information presentation. Datagrids and charts'll be valuable.

Anyway, it's a space that's still starving for great ideas. Good luck!

What the Windows search box wanted to be.

This seems like a very expensive way to do basic NLP