Methodology
Each setup (model + quant + engine) runs the same 10 tasks inside the real Hermes Agent, in a sandbox, 3 times each. Each run passes only if an automatic checker finds the correct result on disk.
The 10 tasks
| # | Task | What it tests | What the agent must do |
|---|---|---|---|
| t01 | Exact reply | Instruction following | Reply with one exact token and nothing else. |
| t02 | Single terminal command | One tool call | Run one shell command and write its exact output to a file. |
| t03 | Write a file | File tool, exact content | Create a file with exact multi-line content. |
| t04 | CSV total | Read + compute | Read a CSV, sum one column for one category, write the number. |
| t05 | Rename across files | Multi-file edit | Rename a function across a small Python package so the tests pass. |
| t06 | Fix a bug | Debugging loop | Make a failing unit test pass without editing the test. |
| t07 | JSON to CSV | Data transform | Convert JSON records to CSV with given columns and sort order. |
| t08 | Multi-step report | Tool chaining | Find matching files, count their lines, write a formatted report. |
| t09 | Needle in a long log | Long context (~86K–130K tokens, depending on the tokenizer) | Find one specific error id in a large log and write it to a file. |
| t10 | Respect a protected file | Constraint following | Finish a small task without modifying a file it was told not to touch. |
All tasks run offline (no web access), so results don't depend on websites or search APIs. Task files and checkers will be published alongside the first results.
What we measure
- Task completion %: passed runs ÷ all runs (10 tasks × 3 repeats). The checker inspects the files the agent left behind, not what the agent says it did.
- Tool-call success %: tool calls that ran without an error ÷ all tool calls the model attempted, read from the Hermes Agent session log.
- Decode / prefill tok/s: single-stream speed on the same server and settings, from llama-server's own timings (about 2,400 prompt tokens, 128 generated tokens, median of 3). Speed is only measured when nothing else uses the machine (night runs, or a separate run when the machine is free). Day runs check correctness only, so their speed is shown as pending until it has been measured.
- Server memory: memory used by the llama-server process right after loading the model with 64K context, as reported by
nvidia-smi(the DGX Spark shares one 128 GB pool between CPU and GPU). We don't show whole-machine memory, because the test machine also runs other things. - Avg turns: model calls per task. Fewer turns for the same result means a more efficient agent.
Fixed settings
- Hermes Agent in one-shot mode, pinned version, terminal and file tools only, max 15 turns per task, 5-minute timeout per task (15 minutes for the long-context task t09).
- 64K context unless the model supports less. The long-context task t09 pastes a log of about 86K tokens (gpt-oss tokenizer; about 127K–130K with the Qwen and Gemma tokenizers), so for t09 the server runs with the smallest of 128K, 192K or 256K context that leaves room for Hermes Agent's own prompt below its automatic compression point (75% of the window). Sampling settings from the model card.
- Engines: llama.cpp (GGUF), vLLM (FP8/NVFP4/MXFP4), Ollama (context raised to at least 64K). Every result records the engine version.
- Hardware: NVIDIA DGX Spark (GB10, 128 GB unified memory). The agent runs in an isolated sandbox with no access to personal files.
Fixed: t09 harness problem (2026-10-23)
Our first runs started every model with a 64K context. The t09 prompt turned out to be about 86K tokens, not the ~45K we had estimated, so Hermes Agent rejected it as too long in about one second and the model was never called. Every setup lost the same 3 of 30 runs. We fixed the harness (bigger context for t09, see above) without changing the task, reran t09 three times for every tested setup and replaced the invalid t09 runs in the published scores. The invalid runs are archived, not deleted. Runs that Hermes Agent rejects this way are now flagged as harness errors instead of being counted as model failures.
Retests
We retest when a new Hermes Agent release, engine version or notable model comes out. Old results are kept with their date and versions.
Limits
- 10 tasks is a small sample: treat differences of a few points as a tie.
- Results are for this hardware and these settings. Your speeds will differ on other GPUs.
- Sponsors and affiliate partners never see results early and have no influence on them.