Hy4 preview vs GLM-5.3 vs Kimi K3: how to compare them fairly

Tencent reports an internal blind test of 163 experts and 203 engineering tasks: Hy4 preview averaged 2.99/4.00, versus 2.92 for GLM-5.3 and 2.94 for Kimi K3. That is useful release evidence, but it is a starting point for comparison rather than a universal ranking.

Key takeaway

Tencent’s blind test is directional evidence, not an independent universal ranking.

Freeze prompts, tools, versions, budgets, and rubrics before comparing models.

Score tool traces, recovery, latency, and successful-workflow cost alongside final text.

Choose a model from the task mix and failure profile your team actually operates.

Model scores depend on task mix, prompts, tools, context, sampling, and rubric. Tencent reports Hy4 preview winning 46.8% of comparisons against GLM-5.3 (12.8% ties, 40.4% losses) and 51.2% against Kimi K3 (7.9% ties, 40.9% losses). The advantage is real in that test set, but not overwhelming.

For a fair reproduction, freeze model versions, system prompts, token budgets, tools, and acceptance criteria. Include cross-file code repair, frontend changes, structured analysis, and tool-mediated tasks. Score the tool trace and failure recovery as well as the final text.

Compare economics too: input and output tokens, cache hits, retries, human intervention, latency, and successful workflow cost. A provider gateway such as SandBase can standardize request IDs and usage logs when the model is present in its live catalog.

Hy4 preview’s distinctive test surface is its 770B/49B MoE design, 1M-plus context, and productivity focus across coding, office, games, and science. It should be evaluated inside long workflows, not only in one-shot chat. Tencent also lists overthinking and excessive self-verification as preview issues, so stability and recovery belong in the scorecard.

Turn scores into a selection decision

After the first matrix, do not average every result into one winner. Set minimum thresholds for the work that matters most: a patch must pass tests, a financial extraction must cite every decision, and a tool workflow must leave the workspace in a known state. Then show trade-offs by task family. This lets a team choose Hy4 for long-context repair while retaining another model for a narrower, faster interaction.

Repeat the test when a provider changes a preview model quickly. Record the exact model identifier, endpoint, region, system prompt, sampling settings, and SDK commit. A result that cannot be reproduced later is still a dated observation, but it should not be presented as a stable capability. Keep a changelog when the model, prompt, tool schema, or evaluator changes so a regression is attributable.

The final report should include the fixture, a compact result table, representative wins and failures, successful-workflow cost, and a recommendation with an explicit “do not use for” boundary. Ask a second reviewer to inspect raw traces before publishing.

Bottom line: the defensible claim today is that Hy4 preview scored slightly higher in Tencent’s reported internal task set. Run the same tasks with the same tools before choosing a model for production.

Build a comparison that can survive review

Start by separating capability from preference. Capability asks whether the model completed the task within the stated constraints. Preference asks which result a reviewer would choose. Record both, along with a binary success flag and a short reason for every failure. A model that writes elegant prose but misses a required file edit should not receive partial credit simply because the answer sounds confident.

Use a small matrix of task families: multi-file code repair, frontend interaction, spreadsheet or document analysis, tool-mediated planning, and long-context retrieval. Give every model the same files, tools, time limit, and retry policy. Keep prompts in version control and run them in a fresh session so hidden conversation state does not change the result.

Measure the work, not the demo

For each run, capture first-token latency, total wall time, input and output tokens, cache hits, tool calls, retries, human interventions, and whether the final artifact passed its checks. Convert those fields into successful-workflow cost. This avoids a common mistake: declaring a cheaper model better because its token price is lower while ignoring extra retries or manual repair.

The reported 2.99 versus 2.92 and 2.94 averages are useful evidence for selecting follow-up tests. They are not a substitute for a public, controlled benchmark. Treat the result as a prior, publish the task fixture, and update the conclusion when independent runs disagree.

One practical scoring rule is to require an artifact and an explanation. For a code task, the artifact is the tested patch; for an office task, it is the completed workbook with traceable source cells; for a tool task, it is the final state plus an execution trace. Reviewers can then distinguish a correct answer from a plausible paragraph. If a model stops to ask for clarification, record that as an interaction cost, not an automatic failure: in production, a safe question may be preferable to an irreversible guess.