# MoACubed: teaching Hermes which profiles actually work

*2026-09-06*

> MoACubed adds feedback, targeted retesting, and fixed baselines to learn which Hermes profiles are actually useful for different kinds of work.


I have been running Hermes with multiple profiles for a while. That gives me useful specialization, but it also creates an uncomfortable question:

**Which profile should handle a given kind of task?**

It is easy to answer that question with vibes. One profile feels sharper. Another writes cleaner code. A third catches edge cases the others miss. Then one impressive run becomes a permanent routing rule.

That is not evaluation. That is anecdote with a config file.

**MoACubed** is an attempt to make the comparison evidence-based.

Repository: [github.com/rahlquist/moacubed](https://github.com/rahlquist/moacubed)

## What it does

MoACubed is a set of three Hermes skills:

- `moaquery` (`moaq`) asks the human what worked, what failed, which reference models helped, and where the aggregator went wrong.
- `moatesting` (`moat`) takes a concrete complaint and retests the disputed point across a small set of relevant profiles.
- `moalevelset` (`moal`) runs a fixed task against multiple profiles under the same conditions and compares the results against an explicit ideal.

The point is not to produce a universal leaderboard. The point is to build a capability map:

```text
profile A → strong at Python debugging
profile B → strong at UI implementation
profile C → strong at research synthesis
profile D → cheap and adequate for simple tasks
```

That is more useful than declaring one profile “best.”

## The feedback loop

A completed MoA task produces a trace. `moaq` asks the human for a small amount of grounded feedback:

- Was the result accepted?
- Which references were useful?
- Which were misleading or irrelevant?
- Did the aggregator synthesize the advice well?
- Was the result actually verified?
- What specifically should have been different?

That review becomes evidence for `moat`.

If the complaint is narrow, the retest stays narrow. Suppose the aggregator correctly identifies a timeout bug but changes the wrong layer. There is no reason to replay the entire task with every profile. `moat` can test the remediation itself:

```text
same issue
same starting state
same acceptance checks
different candidate profiles
```

Then it records which candidate produced the better verified result, how many iterations it needed, and what it cost.

`moal` handles the other side of the system: a small, stable baseline task. The initial fixture combines a code defect, a regression test, a UI change, and verification. A user can replace it with a task that better represents their own workload.

## What gets measured

The system records more than a final score. Each result includes:

- profile identity;
- provider and model;
- reasoning effort;
- skills, tools, MCP access, and system-instruction fingerprint;
- MoA aggregator and reference models;
- fanout mode and reference token cap;
- correctness and completeness;
- verification quality and safety;
- code and UI-specific results;
- model calls, tool calls, wall time, and token usage;
- total agent-loop iterations;
- productive, unproductive, and disproportionate iterations;
- rework and failed attempts.

That last group matters. Ten iterations to isolate a race condition may be fine. Ten iterations to change a CSS color is a routing signal.

MoACubed does not punish necessary debugging. It flags repeated inspections, unchanged retries, ignored tool output, circular reasoning, needless refactoring, and effort that is out of proportion to the task.

## It also tries not to waste tokens

Before testing profiles, MoACubed performs a distinctiveness preflight. If two profiles are materially different, it lets the test proceed. If they are nearly identical, it warns. If they are substantively identical, it pauses before spending tokens.

That is not just an optimization. Duplicate configurations create fake confidence. Two identical profiles succeeding is repeatability evidence, not evidence that two different profile designs succeeded.

The records are profile-local and append-only. Each tested profile gets a human-readable `moacubed.md` ledger backed by JSONL records. Historical results remain auditable instead of being overwritten by the latest opinion.

## What it does not do

MoACubed is deliberately narrower than a general-purpose agent evaluator.

It does not:

- claim that one model is universally best;
- reduce human feedback, objective verification, and evaluator judgement to the same signal;
- treat model agreement as proof of correctness;
- rerun every profile for every complaint;
- automatically change production routing in the first version;
- mark an unverified result as successful because it sounds confident;
- treat a profile name as meaningful when its model, skills, and instructions are identical to another profile;
- expose credentials from traces or configuration;
- replace task-specific acceptance tests with a judge model's taste.

The evaluator can recommend routing. A human still decides whether to apply it.

## Current state

This is an early implementation, not a finished benchmark service. The public repository currently contains:

- the project plan and specification;
- a standard UI/code baseline fixture;
- profile discovery and configuration fingerprints;
- exact and near-duplicate detection;
- isolated workspace materialization;
- a Hermes subprocess runner;
- browser-check support;
- scoring gates and comparison logic;
- the three initial skill definitions;
- a passing local test suite.

The next useful step is real data: run the baseline against genuinely different profiles, collect `moaq` reviews, and see whether the targeted comparisons produce routing advice that survives verification.

That is the test for the project. Not whether the score tables look sophisticated. Whether they help choose a better profile for the next task without burning tokens on pointless comparisons.

## Links

- [MoACubed repository](https://github.com/rahlquist/moacubed)
- [MoAQuery skill](https://github.com/rahlquist/moacubed/tree/master/skills/moaquery)
- [MoATesting skill](https://github.com/rahlquist/moacubed/tree/master/skills/moatesting)
- [MoALevelSet skill](https://github.com/rahlquist/moacubed/tree/master/skills/moalevelset)