Field mapping

XReduce's default scoring path expects each line of your benchmark JSONL to have an input field and an expected_output field. If your data uses different names, or packs multiple values into a structured input, benchmarks/config.yaml tells XReduce how to map your fields to what the SDK expects.

You don't write this file by hand. benchmarks/config.yaml is auto-generated by xreduce init (and regenerated by xreduce remap when you change your benchmark data). XReduce reads the first line of your JSONL, figures out which shape it is, and writes the right mapping. This page documents the schema so you can read it, edit it, and override the inferred values when you need to.

When you need a field mapping

Three common cases:

• Your JSONL already uses input and expected_output. No mapping needed. The SDK passes values straight through.
• Your JSONL uses different field names. (question/answer, prompt/completion, query/sql.) You need a field_map.
• Your input field is structured - a dict, a list, or a nested object rather than a plain string. You need a field_map plus an input_serializer.

Usually you don't have to figure out which case applies - xreduce init --force inspects your JSONL and writes the right benchmarks/config.yaml automatically.

Shape 1 - SDK defaults

Your JSONL uses input and expected_output with plain-string values:

{"input": "The capital of France is", "expected_output": "Paris"}
{"input": "The largest ocean is the", "expected_output": "Pacific"}

No benchmarks/config.yaml needed. The SDK defaults handle this.

Shape 2 - alternate field names

Your JSONL uses different field names. The input is still a plain string:

{"question": "What is 2+2?", "answer": "4"}
{"question": "Capital of Spain?", "answer": "Madrid"}

benchmarks/config.yaml tells XReduce which field is input and which is expected output:

field_map:
  input: "question"
  expected_output: "answer"

Shape 3 - structured input

Your input field isn't a string - it's a dict or a list that needs to be serialized before the model sees it:

{"input": {"schema": "CREATE TABLE users (id INT, name TEXT)", "question": "Count users"}, "sql": "SELECT COUNT(*) FROM users"}
{"input": {"schema": "CREATE TABLE orders (id INT, total FLOAT)", "question": "Sum totals"}, "sql": "SELECT SUM(total) FROM orders"}

The benchmarks/config.yaml adds an input_serializer directive so XReduce knows to flatten the structured input into a string before passing it to the model:

field_map:
  input: "input"
  expected_output: "sql"
input_serializer: "json_to_string"

The same pattern works for list-valued inputs - multi-turn conversations, for example:

{"input": [{"role": "user", "content": "Hi"}, {"role": "user", "content": "Translate 'hello' to Spanish"}], "expected_output": "hola"}

Letting init figure it out

If you've dropped a JSONL into benchmarks/ and want XReduce to generate the mapping for you, re-run init with --force:

xreduce init model-examples/my-model --force

XReduce reads the first line of your JSONL, figures out which shape it is, and writes the appropriate benchmarks/config.yaml. If it doesn't recognize your field names, it stops and tells you to write the mapping manually - rather than guessing.

Writing field mappings manually

You only need to write benchmarks/config.yaml by hand if init can't infer the shape. In practice this happens when:

• Your JSONL uses field names that don't match any of the common patterns init recognizes (input/expected_output, question/answer, prompt/completion, and so on).
• You want to override init's choice - for example, forcing a specific serializer on input data it would otherwise pass through as a string.

The full benchmarks/config.yaml schema:

field_map:
  input: "<your-input-field-name>"
  expected_output: "<your-output-field-name>"

input_serializer: "json_to_string"   # optional - only if input is structured

system_prompt: |                     # auto-generated by task_type, editable
  <your prompt here>

field_map and input_serializer are owned by xreduce init and xreduce remap - they're regenerated from the benchmark JSONL each time. system_prompt is auto-generated the first time (from the task-defaults registry, keyed by the model's task_type) and then preserved on subsequent remap calls so your edits survive.

See Benchmark data for the system_prompt section - how it works, why it matters, and how to edit it effectively.