The Community Space
olivierdehaene's Hugging Face Space, chat-llm-streaming, has become a gathering point for anyone who wants to test-drive multiple large language models without switching tabs or accounts. The page greets visitors with the tagline "Discover amazing ML apps made by the community" and showcases 336 likes alongside 15 active agents. A runtime error surface during metadata fetching — exit code 128, caused by a containerd task timeout — is visible on the space, but it hasn't dimmed the room's appeal. The error reads: "failed to create containerd task: failed to create shim task: context deadline exceeded: unknown." Still, the space draws interest, and the community metric (likes + agents) suggests a steady flow of curious experimenters.
Multi-Model Support
The app.py source reveals the engine under the hood. It ships with support for six model families plus bloomz variants, giving users a dropdown choice that includes OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5, OpenAssistant/oasst-sft-1-pythia-12b, google/flan-t5-xxl, google/flan-ul2, bigscience/bloom, and EleutherAI/gpt-neox-20b. Each selection triggers a parameter makeover: the chat format, slider visibility, and default values all pivot depending on the model family. The description states plainly, "In this app, you can explore the outputs of multiple LLMs when prompted in this way." That's the promise — one interface, many capacities.
The code defines a get_usernames function that maps model names to conversation prefixes. OpenAssistant models use <|prompter|> and <|assistant|> tokens; GPT-NeoXT opts for <human>: and <bot>: ; the T5/Flan pair and Bloom variants fall into a generic User: / Assistant: scheme. This mapping isn't cosmetic — it shapes how the streaming pipeline prefixes and suffixes every turn, and how history gets reconstructed on the client.
Streaming Parameters
Once a model is chosen, the UI surfaces knobs that affect generation. Typical P mass, top-p, temperature, top-k, repetition penalty, and a text watermark checkbox each have a default and a visible/hidden toggle depending on the model. OpenAssistant picks hide typical_p, top_p, and top_k, leaving temperature and repetition penalty visible. Together's GPT-NeoXT shows typical_p, top_p, and top_k while consigning temperature to a secondary row. Flan and Bloom models default to the full set: top-p, top-k, temperature, repetition penalty, and watermark. The code updates these sliders in real time via a radio_on_change callback that fires whenever the model radio selection shifts.
The predict function is where tokens actually flow. It builds a conversation history from prior chatbot turns, prepends a model‑specific preprompt, and streams new tokens either via client.generate_stream with watermark control (for OpenAssistant) or via a top‑p/top‑k/temperature/truncation combo (for everything else). Each yielded chat entry gets stripped of leading/trailing whitespace and re‑paired into user/assistant tuples. The streaming loop also nudges partial words past the user and assistant name boundaries so the display stays clean.
Conversation Format
The app enforces a conversational prompt pattern that every model must respect. The canonical form looks like this:
User: <utterance>
Assistant: <utterance>
User: <utterance>
Assistant: <utterance>
...
The source code reinforces this by stripping any input that doesn't start with the model's user name prefix, and by ensuring every model response ends with the assistant name token. This keeps the history tractable for the streaming accumulator and makes the chatbot widget render predictably regardless of which model is active. The openchat_preprompt, a multi‑sentence blurb about being "helpful, polite, honest, and friendly," gets injected only for the NeoXT model, while the other families skip the prelude entirely.
Technical Details and Launch
The demo launches with gr.queue(concurrency_count=16).launch(debug=True), meaning up to 16 generation workers can run simultaneously — a deliberate choice for a public space where many may be probing different models at once. The description block, wrapped in triple backticks, reminds users that language models can be conditioned to act like dialogue agents through a conversational prompt. A markdown attribution notes "Powered by: Text Generation Inference," pointing to the GitHub repo that handles the actual inference serving. An optional disclaimer links to the official OpenChatKit feedback app for those who want the full experience beyond this demo.
The source also exports a reset_textbox helper that clears the input field after a submit or button click, and a b1.click path that fires the same predict logic as the enter key. Both routes converge on the same [chatbot, state] output, so the user can type, press Enter, or click the go button and get identical behavior.
Discovering the Community
What makes this space stand out is the community‑first tagline. "Discover amazing ML apps made by the community" isn't empty marketing — the space lists 15 agents, shows 336 likes, and openly displays a runtime error rather than hiding it. That transparency is rare. Most Hugging Face spaces polish every detail before going public; this one invites people in while things are still being ironed out. The error message about containerd task creation is a genuine hiccup, not a staged screenshot. Yet the space remains usable, and the model selector lets anyone jump between open‑source LLMs without leaving the page.
Visitors who stick around discover that the parameter sliders aren't just decorative. Changing the temperature on the fly, tweaking top‑p, or turning on watermarking produces observable differences in the model's replies. That interactivity, combined with the breadth of model families, turns a simple chat widget into a quick‑compare sandbox. You can float a prompt through flan‑t5-xxl, then immediately run the same prompt through bloom and see where the answers diverge. The streaming pipeline preserves token‑by‑token text, so the divergence becomes apparent in real time.
The space also models good failure behavior. When the metadata fetch fails, the error is surface‑level and impossible to miss. Newcomers see that even popular, maintained spaces hit snags — Docker timeouts, containerd shim failures, network delays — and the public record of that glitch humanizes the platform. It's a small detail, but it matters when the goal is community discovery rather than a polished product demo.
Wrap‑up
olivierdehaene's chat‑llm‑streaming space lives up to its tagline. It lets anyone with a browser probe half a dozen major LLM families in a single conversation, tweak generation knobs, and watch tokens stream onto the screen in the standard user/assistant format. The runtime error is there for all to see, the model list is long enough to be useful, and the parameter controls give enough knobs to matter without overwhelming. For a community‑driven showcase, it hits the sweet spot: functional, transparent, and genuinely exploratory. If you've ever wondered how a prompt looks across different models, this is the fastest way to find out — no local GPU, no API key, just a Hugging Face account and a sense of curiosity.
<p>~ Written with the texture of a person who's tried too many model‑switching UIs and finally found one that doesn't make them want to quit.</p>