Test reports

The contract every language implements — TESTING.md

Test results as a platform standard

Every suite the platform builds reports its results the same way, whatever language it is written in: a JUnit XML file at one known path, and one OTLP log record per test case carrying the same fields. A new language joins by producing both, and the console renders it with no change.

The contract
JUnit report /tmp/junit.xml inside the test container, the same path in every language module (junitPath in each main.go)
Per-case record an OTLP log record to CI_OTLP_LOGS_ENDPOINT, keyed by ci.build_run

The record's attributes, which are JUnit's own fields under OpenTelemetry names:

attribute JUnit equivalent notes
test.case.name testcase/@name required; everything else is optional
test.suite.name testsuite/@name the file, usually
test.case.classname testcase/@classname diverges from the suite for parameterised and class-based tests, which is when grouping matters
test.outcome — passed, failed, error, skipped
test.case.result.status — the OTel spelling: pass, fail, skipped
test.duration testcase/@time seconds, as a string
test.failure.type failure/@type
test.failure.message failure/@message first line only; the traceback goes in the body
ci.stage, ci.origin — test / project, so the record joins the build's other logs

The record's body is what the case printed — stdout, stderr, and the traceback on failure — one line per record, sharing the case's trace and span ids.

Why both, and why the console reads the records

The XML is the interchange format. An IDE reads it, Argo's UI reads it, and so would any future artifact store, none of them knowing this platform exists. That is what makes this a standard rather than a private schema.

The console does not read it, because the file never leaves the build container: build-artifacts is a ClusterIP service on the workflow plane and the console runs on the data plane, with no route between them. The log records already reach Loki keyed by build_run, so the same fields travel a path that works. If the artifact store ever gains a published port, the file is already being written.

Why the platform writes the report, not a reporter gem

pytest has --junitxml built in, so Python costs nothing. RSpec does not, and rspec_junit_formatter would have to be added to every application's Gemfile and lockfile — against this instrumentation's own rule, which is that the application changes nothing: no spec_helper edit, no spec/support file, no .rspec line. The plugin arrives by --require, and it writes the XML itself (CiJUnit in ci_span_profiles.rb, about fifty lines).

The same reasoning will apply to jest-junit when the frontend module joins.

Escaping, which is where hand-written XML goes wrong

CiJUnit escapes &, <, >, " and ', and strips the control characters XML 1.0 forbids outright — they cannot be escaped, only removed, and Ruby test output contains them (ANSI colour, \r from progress bars). A failure message containing ]]> or a stray control byte is exactly the input that turns a test report into an unparseable file, and it arrives on the day the suite is already broken. Verified against an independent parser rather than by inspection.

A report that cannot be written must never fail the suite: the tests ran, and their result is the thing that matters.

What the console does with it

/builds/{run}/tests — totals, failures first with their message and captured output, the ten slowest cases, every case filterable by outcome and suite, and a per-suite breakdown. Both filters live in the URL, so a filtered report is a link.

A missing duration is left missing. Counting it as zero would make an untimed test look instant, sort it below every real one, and report a 200-case suite as taking 0.00s — which reads as a fast suite rather than as absent timings. The page says how many cases were timed whenever that is not all of them.

Builds that ran before test.duration existed still get timings, because every test span carries a real wall-clock duration and the view joins them by span id, falling back to the case name. The record wins where both exist: it is the suite's own measurement of the test, where the span also contains the fixture teardown the tracer wrapped around it.

Adding a language

  1. Run the suite with the framework's JUnit reporter writing to junitPath.
  2. Emit one log record per case with at least test.case.name and test.outcome.
  3. Nothing in the console changes.

Step 2 is what the console reads today. A suite that only manages step 1 still passes and still reports nothing per case, which the tests page says in those words rather than rendering an empty table.