Did your agent really improve?

Progress in machine learning has often been driven by defining benchmarks and hill climbing on them. ImageNet was a foundational effort that helped propel computer vision for a decade. Agent development follows this same pattern, relying on benchmarks to signal capability. However, an agent can pass these benchmarks through exploits or weak tests rather than real capability, meaning a score alone does not reveal whether progress is genuine.

This question has a track record in machine learning: as Recht et al. showed in Do ImageNet Classifiers Generalize to ImageNet? (ICML, 2019), rebuilding a benchmark test set can lead to significant drops (11–14% top-1 accuracy on ImageNet). This forced the research field to confront a critical question: were models overfitting to specific test cases, or failing to generalize to new task variants? A similar observation resurfaces in code generation tasks: EvalPlus expanded HumanEval’s test coverage roughly 80× and observed pass@k fall by as much as 28.9% across 26 models. Their analysis showed that “passing” solutions had only survived because the original tests were too narrow to catch their bugs.

We apply that same scrutiny to terminal agents, evaluating them on Terminal Bench 2 (TB2), a suite of real terminal tasks such as editing files, transforming data, wrangling git, setting up systems, and debugging broken environments. Many teams building internal terminal agents prioritize reliability on messy, varied real-world tasks over leaderboard rank, making TB2 a strong testbed for studying generalization on realistic tasks.

Consider a TB2 task like filter-js-from-html, where an agent writes a Python script that strips JavaScript from HTML. A single benchmark score on this task cannot answer three critical questions: did the agent actually succeed or just slip through an exploitable verifier; did it truly fail or did a broken verifier reject a correct solution; and if it did succeed, does that capability generalize to variants of the task? The first two questions dictate whether the benchmark’s scores are valid at all, while the third determines an agent’s robustness under task variation. To answer them, we rebuilt TB2 into task variants (TB2-Fn), allowing us to read the agents’ behavior across these variants instead of just looking at their pass rates.

TB2-Fn

Motivated by Recht et al. (2019), we built TB2-Fn (TB2 Fidian Edition), a new benchmark of rebuilt TB2 tasks that exercise the same underlying skills. For instance, the filter-js-from-html task in TB2 has at least two variants:

Task

Original

Variant

Variant

Goal

◎

Sanitize an HTML file: strip the dangerous content, leave the document’s structure intact.

Instruction what the task requires

Strip out

All JavaScript: <script>, on* handlers, javascript: URLs.

Resource-loading & phishing elements: iframe, object/embed, <base>, <form>, external stylesheet <link>, meta-refresh.

JavaScript hidden in 10 obfuscation tricks: SVG/MathML scripts, CSS expression(), data: URIs, entity-encoded javascript:, nested <scr<script>ipt>, on* handlers.

Preserve

The formatting byte-for-byte.

Images, internal links, and document structure.

Safe <svg>/<math> elements and benign inline styles.

Every variant preserves the original underlying goal, which is to strip dangerous content from HTML while keeping the document’s structure intact. We only rewrite the instruction and the verification component. We ran multiple refinement rounds across the environment setup, task instructions, and test scripts. In each round, we used human expert review and Meerkat, an AI audit tool, to audit the tasks and example trajectories generated from a sanity agent run. These audits flagged 51.7% of the task variants for issues (e.g. potential exploits an agent could use to cheat), which we subsequently refined further.

In the last round of refinement, we filtered out any remaining invalid tasks, namely unsolvable ones, duplicates of the original tasks, and tasks Meerkat flagged for potential exploits. Among the remaining valid tasks, we sampled one task variant for each original TB2 task, resulting in a total of 89 new tasks for TB2-Fn. Every task variant ships with its instruction, verifier, environment, and reference solution. The task set is not yet public: releasing it now would risk training contamination. It will be released once that risk is managed.

We evaluated three frontier LLMs: Gemini 3.1 Pro, Opus 4.7, and GPT 5.5 on both TB2 and our new benchmark TB2-Fn, using two terminal agent harnesses (Terminus 2 and Kira) and each model’s native CLI (Gemini CLI for Gemini, Claude Code for Opus, and Codex CLI for GPT). For every model and harness combination, we ran 5 attempts per task. Throughout this post, the error bars on our charts denote the margin of error (MOE), measured as the half-width of the 95% confidence interval around the reported passing rate: $\mathrm{MOE}=1.96\times \sqrt{\frac{\widehat{p}(1-\widehat{p})}{n}}$, where $\widehat{p}$ = reported passing rate and $n$ = total number of attempts.

Swipe to explore the chart →

View full-size chart →

We observe that all agent × harness combinations fail to maintain their performance from TB2 to TB2-Fn. On non-native CLI harnesses, Gemini falls the most, 16 points with Terminus 2 and 12 with Kira; GPT drops about 4–6 points and Opus loses the least, about 3–5 points. On native CLI harnesses, the drops range from 5 to 11 points.

What is the real task-variant shift?

Agent benchmarks like TB2 are known to be imperfect, as studied in Establishing Best Practices for Building Rigorous Agentic Benchmarks. In another study, AgentLens found that many trajectories in the coding agent benchmark SWEBench were counted as successes even though they are just “lucky passes” (incomplete solutions that passed against insufficient test coverage).

Hence, to understand the real performance impacts, we reviewed every original TB2 task and categorized them into the following three groups:

1. Score-inflating tasks. These tasks allowed incorrect agents to pass in TB2 via possible exploits, leaked solution secrets, or any task loopholes. These effectively inflated the original benchmark scores.
2. Score-deflating tasks. These tasks penalized correct agents in TB2 due to broken environment setup, over-strict test cases, or hidden task requirements. These effectively deflated the original scores.
3. Neutral tasks. These were considered correctly constructed tasks. Any performance shift in this group isolates the impact of task variation, stripped of any inflation or deflation effects.

We define the categories of issues for score-inflating tasks and score-deflating tasks below:

To classify TB2 tasks into these categories, we sampled trajectories and employed expert human reviewers and the AI audit tool Meerkat to review the tasks (verifier, instruction, environment), the sampled traces, and the generated solutions. We also compared our findings against any community-reported issues on TB2. As a result, we identified 7 score-inflating tasks and 27 score-deflating tasks in total (see Appendix A for the lists of tasks in these groups). These 34 flagged tasks, about 38% of the benchmark, represent prevalent verification and task-construction bugs.

Using this data split of 7/27/55 samples, we measured the performance across all agents and models (Terminus 2, Kira, and native CLI harnesses across 3 frontier models) in each split:

Swipe to explore the chart →

View full-size chart →

First, score-inflating tasks collapse by 40 points on average from TB2 to TB2-Fn across all model × harness runs. TB2 overestimated agent success on these tasks at a 77.5% passing rate, against a true rate of only 37.5%.

Secondly, score-deflating tasks recovered from 47.3% in TB2 to 56.2% in TB2-Fn. While modest (+8.9 points), this recovery is significant because it occurred despite the tasks being rewritten into new variants, which would otherwise pull scores down.

Finally, in the neutral group, we observed a task-variant shift of -12 points, a deeper decline than the -7.9 points when measured on all 89 tasks. This deeper decline reflects the true “robustness gap”, the extent to which agents fail when faced with task variation.

Which agent is sensitive to the real task-variant shift?

To understand why our initial findings on the full 89-task benchmark may be misleading, we analyzed how each model’s performance delta from TB2 to TB2-Fn distributes across the above three data categories.

We also designed a paired test with bootstrap: every TB2-Fn task has a parent TB2 task, and each task is measured with multiple attempts. So far the MOE error bars we have reported describe each passing rate on its own, but the quantity we actually care about is the delta, which carries two nested sources of noise: (1) which tasks happen to be in the set, and (2) run-to-run nondeterminism within each task.

We therefore tested the per-task performance change with a task-level clustered bootstrap (Efron, 1979; Cameron, Gelbach & Miller, 2008) by resampling both tasks and attempts-within-task (see Appendix B for the implementation details of our test). We report the 95% confidence interval of the delta per model next to the average number:

Model	Score-inflating Δ [95% CI]	Score-deflating Δ [95% CI]	Neutral Δ [95% CI]
Gemini 3.1 Pro	−36.2 [−63.8, −10.5]	+1.0 [−19.2, +20.4]	−14.4 [−24.4, −5.5]
Opus 4.7	−35.2 [−64.8, −9.5]	+13.3 [−8.2, +34.1]	−10.5 [−20.1, −0.9]
GPT 5.5	−48.6 [−77.1, −21.0]	+12.0 [−5.8, +33.2]	−11.2 [−20.7, −2.1]

Under this test, we found that the shifts are statistically significant for the score-inflating and neutral groups (intervals exclude zero) but not for the score-deflating group. The deflating point estimates are positive for GPT (+12.0) and Opus (+13.3) and near zero for Gemini (+1.0), but the 95% confidence intervals include zero, so we cannot conclude that TB2’s broken verifiers were holding GPT and Opus down.

Secondly, in the neutral tasks, the statistically significant performance drops indicate current models are still sensitive to task-variant shifts, even when stripping noisy tasks. We found that this per-model divergence under task-variant shift in TB2-Fn echoes the broader observation in D’Amour et al. (2022) that models with near-identical benchmark scores can behave very differently under data distribution shifts. Let us strip out the noisy splits and look only at the results on the 55 neutral tasks:

Swipe to explore the chart →

View full-size chart →

Comparing all models, we still could not find a consistent pattern like what Recht et al. (2019) found: their model performance drops in ImageNet task variants are very consistent, following linear approximation from the original ImageNet results. In our case, while we only have limited data points (3 models × 3 harnesses = 9 data points), one possible explanation is that foundation models are trained on far more data than traditional ImageNet models. As an untested hypothesis, some models may have been exposed to these tasks during training (as noted by Dominguez et al. (2025)); if a model saw these tasks during training, its score on the original TB2 tasks is inflated, so its measured drop to the rebuilt variants appears larger.

Difficulty-controlled task variants

Before concluding that models like Gemini are significantly more sensitive to task-variant shift, we reviewed how well our TB2-Fn task variants were constructed. Engstrom et al. (2020) and Sturgeon (2026) found that replicated benchmarks such as ImageNet-v2 and GSM-Symbolic (a variant of GSM8K benchmark) can inflate the performance drops from the original benchmarks if the task construction mechanisms were not properly designed and audited.

We therefore recreated task variants with more controlled difficulty. We developed four criteria to accept any new task variant:

• C1 - solvability: a reference solution exists and passes all tests
• C2 - skill equivalence: the variant exercises the same skillset/capabilities as its parent TB2 task.
• C3 - cost equivalence: the reference solution’s cost stays within 1.5× of the parent’s, in both trajectory length (tokens) and execution time.
• C4 - construction discipline: the variant is produced only by rule-based, difficulty-preserving surface transforms. Inspired by Sarkar et al. (2023), we defined 3 categories of surface transforms, namely input shift, output shift, and environment shift (see Appendix C for the description and example of the transforms).

We denote a task variant that passes all criteria 1 to 3 as a difficulty-controlled task while one that passes all 4 criteria as difficulty-invariant. For criteria 2 and 3 above, note that a difficulty-invariant task’s reference solution is the original task’s oracle solution with only surface transforms (e.g. renaming output artifacts). A difficulty-controlled task’s reference solution was authored by Opus 4.8, reviewed by humans, and must pass all tests.

For experiments in this section, we sampled 11 TB2 tasks out of the above 55 neutral tasks (see Appendix D for the list of 11 tasks). For each original TB2 task, instead of measuring the results on a single task variant, we created 10 different difficulty-controlled tasks (denoted TB2-Fn-C) and 10 difficulty-invariant tasks (denoted TB2-Fn-I).

We can compare all three variants against each criterion in the following table:

Criterion	TB2-Fn	TB2-Fn-C	TB2-Fn-I
C1 - solvability	✓	✓	✓
C2 - skill equivalence		✓	✓
C3 - cost equivalence		✓	✓
C4 - construction discipline			✓

Similar to previous experiments, we ran 5 attempts per task variant and reported the passing rate per model in each variant set, first broken down by harness:

Swipe to explore the chart →

View full-size chart →

and then pooled across all harnesses:

Swipe to explore the chart →

View full-size chart →

We also performed pair tests like before and reported the averages of performance deltas and their 95% confidence intervals per model:

Model	TB2 → TB2-Fn-I Mean Δ [95% CI]	TB2 → TB2-Fn-C Mean Δ [95% CI]	TB2 → TB2-Fn Mean Δ [95% CI]
Gemini 3.1 Pro	−8.9 [−21.0, +2.6]	−21.7 [−42.5, −0.6]	−54.6 [−71.2, −38.2]
Opus 4.7	−5.6 [−11.4, +0.7]	−14.8 [−26.7, −3.5]	−39.1 [−58.3, −19.1]
GPT 5.5	−1.6 [−7.9, +4.4]	−16.0 [−31.2, −3.5]	−41.2 [−58.8, −24.2]

The uncontrolled TB2-Fn variants for these 11 tasks were deliberately more difficult than their parent tasks, which is why their drops here (for example Gemini -54.6) far exceed the -14.4 neutral-task average reported earlier (the cost analysis below shows TB2-Fn runs at 1.69× the parent’s tokens). Similar to Engstrom et al. (2020), we observed that the original performance drops in TB2-Fn were inflated when we strictly controlled the level of difficulty in task variants in TB2-Fn-I and TB2-Fn-C. Secondly, we found that the more a variant set departs from its parent tasks, the larger the measured performance drop. Thirdly, in the difficulty-invariant set TB2-Fn-I, we observe that the 95% confidence intervals of deltas span over zero, and hence, performance drops are not statistically significant for these variants. However, across all three dataset variants, Gemini’s point estimate is consistently the most negative, including on the difficulty-controlled set TB2-Fn-C (Gemini -21.7 [-42.5, -0.6], Opus -14.8 [-26.7, -3.5], GPT -16.0 [-31.2, -3.5]). Because these confidence intervals overlap heavily, this observation represents a trend rather than a statistically established difference in sensitivity.

Are the tasks really difficulty-controlled? To quantify difficulty divergence across the three variant datasets, we used realized cost ratios (agents spend more compute on harder tasks; Snell et al., 2024), meaning the cost an agent actually incurred to solve a task relative to the cost on the original TB2 tasks. Specifically, we calculated these by analyzing generated trajectory logs, extracting token counts, episodes (agent turns), and agent runtime. Note that we reported these metrics here to reflect the genuine difficulty levels and validate our task designs. Hence, we filtered the data to include only successful traces to exclude any cost overhead from failed attempts.

Because a cost ratio is the agent cost on a task variant divided by the cost on its parent TB2 task, we summarized the statistics below with the geometric means of the ratios and performed a paired test like before. Each value is the geometric mean ratio against TB2 with its 95% confidence interval:

Cost metric	TB2-Fn	TB2-Fn-C	TB2-Fn-I
Tokens	1.69× [1.08, 2.68]	1.02× [0.93, 1.12]	1.00× [0.91, 1.10]
Turns	1.29× [1.04, 1.61]	0.99× [0.95, 1.07]	0.99× [0.94, 1.05]
Runtime	1.38× [1.06, 1.83]	1.09× [0.99, 1.20]	1.07× [0.97, 1.18]

From the statistics, we observe that as we tighten the difficulty level, the realized cost ratios against the original tasks (by trajectory tokens, episodes, and agent runtime) drop sharply from TB2-Fn to the controlled sets and sit near 1 for both TB2-Fn-C and TB2-Fn-I, which are close to each other and to the parent task’s cost. This shows that our criterion of cost equivalence can be used as a good proxy to design difficulty-controlled task variation.

Did the model memorize the original tasks? In our initial analysis, we performed simple text pattern matching in Gemini’s generated trajectories and did not find evidence of the model reciting the same solution or artifacts (e.g. file names, data paths, output names) as the trajectories in the original task. We suggest more sophisticated methods such as Li et al. (2025) could be used to diagnose the output artifacts for any evidence of model memorization. In this post, we only highlight that the significant performance drops of Gemini are consistent with both a capacity gap, where the model lacks the underlying capability even on clean tasks (as shown in the results on our TB2-Fn neutral tasks), and an adaptivity limit, where the model solves the original task but fails under surface-preserving transforms (as shown in the results on our difficulty-controlled variants TB2-Fn-I/C).

Case study: Eval-driven patching of the agent harness

From our results, we have found that Gemini is more vulnerable when evaluated under rigorous tests. So how do we act on this? Improving a foundation model is not easy, requiring massive amounts of data and incurring significant compute cost.

Here it helps to remember that an agent is never just its underlying model. It is the model + the harness that wraps it: the system prompts, tool definitions, context management, and control flow that turn a language model into an agent. Recent research such as Code as Agent Harness and Meta Harness considered the harness an important substrate to improve agent capabilities, boosting the performance of the agents by optimizing the different components of the harness. Hence, the harness should be treated as a critical part of the agent (as also positioned here by Lee, 2026), complementing foundational large language models the agent is based on.

In this section, we conducted a simple case study to improve Gemini at the agent harness level. Specifically, we ran a harness improvement loop where task variants are created and the agent must improve across all variants:

• Expand a task: We first expand an input task into a family of task variants. This ensures that when we measure the agent, we are measuring a capability, not a single test instance.
• Run and evaluate: We run the agent across the entire task family for multiple attempts and look for patterns in behavior across the variants.
• Diagnose failure modes: Once evaluation is complete, we move from scoring to trajectory-level diagnosis. We audit agent trajectories to isolate the root cause of failure and look for the “why” behind every regression.
• Engineer patches: Finally, we engineer surgical patches to the harness, such as refining system prompts, adjusting tool constraints, or updating the execution flow.

To see this in action, let’s look at how we applied this loop to patch the Kira harness for Gemini on an example task in TB2. Here, we look at the large-scale-text-editing task whose goal is “to transform a large CSV file (1 million rows) using efficient Vim macros. The agent must create a Vim script that defines three distinct macros to transform a CSV to match another CSV byte-for-byte.”

First, we can think of several ways to expand the tasks into 5 task variants:

Seed

large-scale-text-editing

Swap two columns of a 1,000,000-row CSV, with Vim macros

col1,col2 → col2,col1

002

Reverse & re-delimit

trim spaces · tab→pipe · reverse all 4 fields

"sun\tmon\ttue\twed  " → "wed|tue|mon|sun"

003

Recase & bracket

lowercase · comma→semicolon · wrap each in [ ]

APPLE,BANANA,CHERRY → [apple];[banana];[cherry]

004

Duplicate & shout

field → field-FIELD · pipe→semicolon

fox|bear|elk → fox-FOX;bear-BEAR;elk-ELK

005

Pairwise swap

swap 1↔2 & 3↔4 · lowercase · comma→colon

AA,BB,CC,DD,EE → bb:aa:dd:cc:ee

006

Strip, swap & tag

trim · swap · UPPERCASE · →pipe · append DONE

"  hello  ;  world  " → "WORLD|HELLO|DONE"

Each task variant keeps the seed task’s core challenge intact, which is to rewrite every one of one million rows with three Vim macros, in under 200 keystrokes, matching the expected output byte-for-byte. We only change the surface of the task: the delimiter, the casing, the column order, the field count, or the whitespace padding.

Even though the transformations look different, every variant stresses the agent capabilities to handle large-scale data efficiently in limited command budgets. So a harness that overfits the original seed task (e.g. one that has effectively memorized its actions for a two-column swap) won’t hold up across this task family.

Now let’s look at how we can improve the Gemini agent harness iteratively (hover on a patch or analysis box to see the full content):

Swipe to explore the chart →

View full-size chart →

We tracked our progress through four distinct iterations:

Iteration 1: The agent struggled with high-volume data manipulation, resulting in a 60% pass rate. Diagnostic analysis revealed the agent frequently relied on heredoc syntax, which caused the shell to hang and triggered timeout errors.

• Patch 1: We implemented a system prompt rule discouraging heredoc usage and modified the harness to inject keystrokes if the shell stalls.

Iteration 2: This improved performance on complex task variants, lifting the overall pass rate to 77%. However, we found a new failure mode: the agent was wasting turns attempting to debug Vim macros across the entire one-million-row dataset, again leading to timeout errors.

• Patch 2: We added a constraint encouraging the agent to verify Vim macros on a small data subset before scaling to the full dataset.

Iteration 3: The aggregate pass rate remained stagnant at 77%, revealing a classic “whack-a-mole” dynamic. While the new patch solved our data-volume issues (jumping success rates on specific difficult tasks from 40% to 100%), it inadvertently regressed other tasks. Our trajectory-level analysis showed that the heredoc hang re-emerged in these regression trajectories.

Crucially, these regression cases were only detected because we evaluated the agent across the entire task family. Had we relied on a standard aggregate score, the 77% metric would have looked flat, leaving us blind to the regression introduced by the last patch.

This reflects a similar failure mode documented by AgentLens, which found that “Lucky Passes” (e.g. successes driven by brute-forced retries or partial fixes slipping through test gaps) can hide systemic brittleness. A standard pass rate treats those lucky passes identically to robust successes; our trajectory-level analysis on task variants ensures we don’t mistake lucky passes for genuine progress.

• Patch 3: To solve this, we moved from prompt-based guidance to a hard guardrail: we modified the harness to detect heredoc patterns and automatically inject escape sequences.

Iteration 4: The refined agent reached a new high, achieving a 90% average pass rate. With most tasks now succeeding consistently (80–100%), we accepted the patches to the Kira harness.

Evaluation. To make sure that we are not overfitting the patches to the current task family, we evaluated the patched Kira at each iteration on a held-out set of 5 new task variants. These task variants were independently created from the original task and we never used them in our improvement loop.

Swipe to explore the chart →

View full-size chart →

We observed that the performance gains of our patched Kira held steady on five unseen variants of the same task. This suggests that our Kira patches were not just “hacks” overfit to a single failure. We did not just patch a bug; we engineered a more robust harness for this task family.

Conclusion

Evaluating AI agents requires moving beyond static, narrow benchmarks. “Success” on a leaderboard often masks a lack of true generalization and is distorted by systemic verification bugs or lucky passes. Our findings echo a recent, growing call for a more rigorous science of AI evaluation (Reuel et al., 2024, Wallach et al., 2025; Salaudeen et al., 2025; Weidinger et al., 2025).

As we showed in our case study to improve a Gemini harness, treating the harness as a core, tunable part of the agent allows us to systematically address vulnerabilities. This process relies on eval-driven diagnosis because a single aggregate pass rate can look flat while hiding a regression. For example, at iteration 3, the score stayed at 77% while a patch had fixed some tasks and silently regressed others, a shift that only trajectory-level analysis could catch. Building resilient harnesses shifts the evaluation focus from “can it pass this test?” to “can it reliably solve this problem?”

Appendix A

Score-inflating tasks

Bug category	Tasks
Solution leak	fix-git (PR #53), path-tracing (#61), model-extraction-relu-logits (#60), fix-code-vulnerability, cobol-modernization
Test loophole	gpt2-codegolf (#62), git-leak-recovery

Score-deflating tasks

Bug category	Tasks
Instruction issue	adaptive-rejection-sampler (PR #53), build-pmars (#59), configure-git-webserver (#39), query-optimize (PR #53), sam-cell-seg (#45), torch-tensor-parallelism (PR #53), raman-fitting
Test/verifier issue	filter-js-from-html (#46), hf-model-inference (PR #53), install-windows-3.11 (PR #53), polyglot-c-py (PR #53), polyglot-rust-c (#37), break-filter-js-from-html (#26), winning-avg-corewars (#17), extract-elf
Environment issue	build-pov-ray (PR #53), caffe-cifar-10 (#44), compile-compcert (PR #53), extract-moves-from-video (PR #53), financial-document-processor (PR #53), make-doom-for-mips (#44), mcmc-sampling-stan (PR #53), mteb-leaderboard (#27), mteb-retrieve (PR #53), overfull-hbox (PR #53), torch-pipeline-parallelism (PR #53), train-fasttext (PR #53)

Appendix B

We designed a paired test with bootstrap as follows:

# Inputs (per model):
#   tasks       : list of tasks
#   tb2[i]      : list of 0/1 outcomes, TB2 attempts for task i
#   tbfn[i]     : list of 0/1 outcomes, TB2-Fn attempts for task i
observed_delta = mean( mean(tbfn[i]) - mean(tb2[i]) for i in tasks )

B = 10000
boot = []
for b in range(B):
    sampled = random_choice(tasks, size=len(tasks), replace=True)   # (1) resample TASKS
    diffs = []
    for i in sampled:
        # (2) resample ATTEMPTS
        p = mean(random_choice(tb2[i],  len(tb2[i]),  replace=True))
        q = mean(random_choice(tbfn[i], len(tbfn[i]), replace=True))
        diffs.append(q - p)
    boot.append(mean(diffs))

CI_95 = (percentile(boot, 2.5), percentile(boot, 97.5))

Appendix C

We created the following rules to construct a difficulty-controlled task variant from a TB2 task. Borrowing the input/output shift decomposition of Sarkar et al. (2023), we group our rules into three data shift types:

• Input shift: we perturb the instruction the agent reads while leaving the underlying goal and the verifier invariant (paraphrase instruction, rename identifiers, reorder independent rules).
• Output shift: we move the target artifact the agent must produce, so the verifier checks the same content at a new location (rename output artifact).
• Environment shift: we relocate where the task lives without changing its files or goal (relocate working directory).

Transform rule	What changes	Why difficulty is unchanged	Example changes (original → variant)
Input shift
Paraphrase instruction	Instruction wording only	same constraints, only reworded	”Write a regex expression that matches dates in the format YYYY-MM-DD…” → “Construct a regular expression that captures calendar dates written as YYYY-MM-DD…”
Rename identifiers	names of provided files/symbols cited in the task prompt	a pure label swap in task instructions	eval.py → sanity_check.py; sequences.fasta → assembly_inputs.fasta
Reorder independent rules	order of constraints in the task instruction	Collective constraints are the same	In regex-log, swap the “February ≤ 29 days” rule and the “no leading zeros in octets” rule
Output shift
Rename output artifact	output filename/path	verifier reads the new file/path with the same expected content	save to /app/date_pattern.txt → save to /app/regex.txt
Environment shift
Relocate working directory	the task environment’s workdir path	same files, different location	/app → /workspace/task

Appendix D

11 tasks we used for difficulty-controlled task variant experiments:

• dna-assembly
• feal-differential-cryptanalysis
• feal-linear-cryptanalysis
• mailman
• nginx-request-logging
• path-tracing-reverse
• qemu-alpine-ssh
• regex-log
• sparql-university
• sqlite-db-truncate
• tune-mjcf

For each of the 11 tasks, we produced the variant tasks through the following pipeline:

1. Construction: We pick a transform from the Appendix C families and apply it consistently across every file of the task, including the instruction.md, tests folder, the reference solutions, and the environment files. We use Opus 4.8 to perform this editing. The two variant types (TB2-Fn-I and TB2-Fn-C) differ in how the reference solution is obtained: for difficulty-invariant tasks (TB2-Fn-I), the transform is purely surface (rename files/identifiers/paths, reorder independent rules, relocate the working directory), so its reference solution is simply the original TB2 reference with the same names/paths rewritten, meaning no new reasoning is introduced. For difficulty-controlled tasks (TB2-Fn-C), the change goes beyond the surface, so Opus 4.8 authors a fresh reference solution that is then reviewed by humans.
2. Validation (criterion C1): Every candidate is run through an oracle solution: we execute its reference solution in the task’s Docker container and then run the verifier. The variant is kept only if it scores reward as 1.
3. Cost check (C3): We compare the reference solution’s size / trajectory tokens / execution time against the reference solution of the corresponding original TB2 task and drop anything exceeding 1.5×.
4. Review (C2, C4): Skill equivalence and construction discipline are enforced by the fixed transform rules plus random check by human reviewers of the variant tasks; any borderline cases are dropped or revised further (by repeating from step 1).

Variants that fail any step above are discarded or revised further by additional rounds of LLM and human reviewers. In the final round, only those tasks that survive all checks enter the evaluation.

Citation

If you find this work useful, please cite:

@misc{fidian2026tb2fn,
  title  = {Did your agent really improve?},
  author = {Fidian},
  year   = {2026},
  month  = {June},
  url    = {https://fidian.ai/blog/did-your-agent-really-improve},
}