LLM-jp 3-1.8B: Open-Source Japanese-English Language Model from NICT
The llm-jp/llm-jp-3-1.8b arrived on Hugging Face as a surprise for anyone tracking open-source multilingual LLMs. Developed by Japan's National Institute of Informatics (NII), this 1.8B parameter transformer model brings Japanese-English bilingual capability into the Apache 2.0-licensed arena, where many competitors stay proprietary. I've been watching the model card closely, and what stands out is the transparency: the full training mix, the instruction datasets, and the evaluation numbers are all laid out open for inspection.
Model architecture and scale
The model runs on a standard Transformer decoder architecture. Total seen tokens during pre-training hit 2.1T, which is a generous amount for a model of this size. The parameter breakdown across the series is striking: 1.8B with 24 layers, 2048 hidden size, and 16 heads; 3.7B pushes to 28 layers and 3072 hidden; the 13B variant adds 40 layers and 5120 hidden. All of them use BF16 tensor type, which keeps memory footprints reasonable on modern hardware. The tokenizer comes from llm-jp-tokenizer v3.0, a Unigram byte-fallback model that was trained specifically for the Japanese-English mix.
Training data mix
The pre-training blend is where the model's bilingual character really takes shape. For Japanese, the largest contributors were Wikipedia (2.6B tokens), Common Crawl (762.8B tokens), and the WARP collections — PDF, HTML, and Kaken documents totaling about 253B tokens. English pre-training drew from Wikipedia (4.7B tokens), Dolma variants (C4, Reddit, PeS2o, Gutenberg, Wiki), and The Stack for code at 114.1B tokens. Smaller but still present are Chinese Wikipedia (0.8B) and Korean Wikipedia (0.3B). What's interesting is the overlap: Common Crawl appears in both the Japanese and English mixes, which means the model sees the same web-scale corpus through two different linguistic lenses.
The instruction tuning stage refined the base model on Japanese-specific datasets — nichikara-instruction-004-002, answer-carefully-002, and ichikara-instruction-format — plus several synthetic datasets generated by larger Calm3 models and a 8x22b logical-math-coding SFT. English instruction data came from FLAN, sampled. The result is a model that can follow prompts in both languages without a major drop in capability.
Evaluation results
The llm-jp-eval suite (100 dev examples) and Japanese MT Bench (evaluated with gpt-4-0613) give a clear picture of where the model stands. The 1.8B base scores an average of 0.3767 across the llm-jp-eval categories, with the strongest showing in NLI (0.7730) and weakest in MC (0.2350). The instruct variant lifts the average to 0.4596, and the MT Bench scores show the instruct version hitting 4.93 average, with particularly strong performance in humanities (7.80) and roleplay (7.80). The 3.7B and 13B versions improve steadily, as expected, but the 1.8B instruct model still holds its own against larger open-source competitors.
Deployment and practical use
The model card includes clear instructions for several deployment paths. Transformers users can pull in the model with AutoTokenizer and AutoModelForCausalLM, specifying bfloat16 and deviceMap auto. For production throughput, the card points to vLLM — install from pip, run vllm serve "llm-jp/llm-jp-3-1.8b", and make OpenAI-compatible API calls. SGLang gets the same treatment: pip install, launch server on port 30000, and curl the same API format. Docker users have two routes: the standard docker model run hf.co/llm-jp/llm-jp-3-1.8b command, or the SGLang Docker image with GPU passthrough. There's also a Docker Model Runner invocation: docker model run hf.co/llm-jp/llm-jp-3-1.8b.
Risks and limitations
The model card is upfront about where the system is still early-stage. The NII team notes that the models have not been tuned to ensure outputs align with human intent and safety considerations. That's the main caveat: good language coverage and instruction-following, but no guaranteed refusal behavior or alignment tuning beyond what the instruction datasets provide. The team invites questions at llm-jp(at)nii.ac.jp.
License and authorship
Everything carries the Apache License, Version 2.0. The model card authors are listed alphabetically: Hirokazu Kiyomaru and Takashi Kodama. Downloads in the last month hit 7,475, which seems modest for a 1.8B model but reasonable for a research release from a national institute.
Closing thoughts
I keep coming back to what this model represents: a serious effort to push open-source multilingual LLMs beyond the English-centric default. The 1.8B size makes it accessible on a single GPU with BF16, the Japanese-English mix fills a real gap for developers working in both languages, and the open data mix means anyone can trace where the model's knowledge came from. It's not the biggest model on the block, but it's one of the most transparent, and that matters for a community that's grown skeptical of opaque training runs. If you've been looking for a compact, bilingual open model that doesn't require a Microsoft or Google credit line, llm-jp/llm-jp-3-1.8b earns a look.