Autonomous research

Karpathy's autoresearch is a sharp idea: give an agent a real experiment, let it run overnight. It edits the code, runs a fixed-budget trial, checks whether the metric improved, keeps the change or reverts it, and repeats. You wake up to a log and, with luck, a better result. The move that makes it work: you never touch the code — you write the agent's instructions and let it drive. You program the research org, not the implementation.

That's a precise fit for Workbooks. The experimenter is an agent — a server-side brain whose bash runs and measures code inside a /work sandbox — and "keep what improves, loop" is a tiny server around it. This lesson builds the whole loop and is honest about what the sandbox can and can't run.

The experimenter

The brain describes how to run one honest experiment. Pick a task whose result the sandbox can actually measure — here, lowering the size of a compressor whose candidate is compiled and run in-sandbox (a real, attributable metric), in the spirit of autoresearch:

agent :researcher do
  prompt """
  You run ONE optimization experiment in /work. `codec.c` is yours to edit. `bench.sh` is
  fixed — it compiles codec.c and prints `score=<bytes>` for the test corpus. LOWER is
  better. Form ONE hypothesis, make the smallest edit that tests it, run `bash bench.sh`,
  READ the real score, and report it on the LAST line as `score=<number>`. One variable at
  a time so every delta is attributable; report the number you measured, not hoped for.
  """

  tools coreutils, cc                 # capabilities: edit files + the C→wasm compiler
  grant fs                            # permissions: files only — no network for a trial
  limit turns: 30, timeout: 300_000   # guardrails per trial
end

The agent is fully defined by the block — instructions, the capabilities it may use, the permission boundary, and its stop conditions. Note grant fs with no net: a trial can read and write /work but can't reach the network, so an unattended run can't phone home. The compile-and-run goes through its kits inside the sandbox.

The loop

Around the experimenter sits the org: run N trials, and keep an edit only when the metric actually improves — otherwise discard it and carry the previous best forward. This loop is an ordinary Elixir reduce — for a fixed pipeline that's all you need (a flow block is the declarative option when you want named, reusable steps). Nexus.Agent.run/1 returns the final answer (where the score was reported) and vfs_files (the code left behind), which is all the loop needs:

server :lab do
  # the experimenter, defined inline with the same fields as the agent block
  @researcher [
    prompt: "You run ONE optimization experiment in /work. Edit codec.c, run `bash bench.sh`, " <>
            "READ the score, report `score=<bytes>` on the last line. One change at a time.",
    tools: [:coreutils, :cc],
    grant: [:fs],
    limit: [turns: 30, timeout: 300_000]
  ]

  # Run `rounds` trials; the best code + a log fall out the bottom.
  def run(files, rounds \\ 100) do
    init = %{files: files, best: 1.0e12, kept: 0, log: []}

    Enum.reduce(1..rounds, init, fn n, st ->
      {:ok, %{answer: a, vfs_files: edited}} =
        Nexus.Agent.run(
          [task: "Trial ##{n}. Current best score is #{st.best} — try to beat it.",
           seed: st.files] ++ @researcher    # hand the current best code to this trial
        )

      case parse_score(a) do
        s when s < st.best ->
          %{st | files: edited, best: s, kept: st.kept + 1,
                 log: [%{n: n, kept: true, score: s} | st.log]}

        s ->
          # no improvement → revert: keep the previous best `files`, drop `edited`
          %{st | log: [%{n: n, kept: false, score: s} | st.log]}
      end
    end)
  end

  defp parse_score(text) do
    case Regex.run(~r/score=([\d.]+)/, text || "") do
      [_, v] -> String.to_float(v)
      _ -> 1.0e12                          # a trial that didn't report a score never wins
    end
  end
end

That's the entire research org. seed: st.files hands the current best code to each trial; on success the loop adopts edited, on failure it simply doesn't — the "revert" is just not updating the accumulator. After rounds runs you have the best files, the best score, and a log of what was tried and what stuck. Run it on a schedule and it researches while you sleep.

What runs where (be honest about the wall)

The wasm sandbox runs what compiles to wasm — C/Rust/Zig benchmarks, data crunching, algorithms. It does not run a GPU training job; that's a host workload behind the bedrock wall. The pattern is identical either way — the difference is the experimenter reaches heavy compute through a host exec capability (a trusted broker) rather than running it in-guest. Pick the metric your substrate can actually measure, and the loop is the same.

Tuning the org, not the code

The payoff of programming the agent instead of the implementation: to change the research strategy, you edit prose. Want bolder hypotheses? Loosen the "smallest edit" rule. Want a specific axis chased? Say so in :researcher. Want breadth instead of a hill-climb? Run many experimenters from different seeds at once — the agents-at-scale fan-out — and keep the best across all of them.

What to take away