← Journal / articles

Write the claim before you count the tests

A test result becomes useful when the reader can tell which claim it supports. A large total can obscure that connection if several kinds of evidence are presented as interchangeable proof.

Three separate evidence forms sit within a bounded illuminated area, illustrating why different tests support different claims. AI-generated artwork, directed and reviewed by Black Ice Labs.

A test result becomes useful when the reader can tell which claim it supports. A large total can obscure that connection if several kinds of evidence are presented as interchangeable proof.

The distinction is practical. A model can explore combinations that would be expensive or inappropriate to exercise against a real system. A small native check can show what a process actually observed under specified conditions. An application trial can examine whether a particular job still works. Choosing among them begins with the question the team needs to answer.

A public example with an explicit boundary

Proxy's published Black Ice CryoGate documentation describes a contextual containment prototype. Its evaluation separates synthetic cases, focused benign native observations and review evidence. It leaves application compatibility and production protection unproven. This article draws on that published account and reports no new execution.

Those limits are useful information for anyone deciding what could be evaluated next. They prevent a reader from silently extending a result about a policy predicate into a promise about an entire application.

Mechanism documentation has its own place. Linux describes seccomp as a way to reduce exposed system calls within a broader security design. Docker documents a default seccomp profile and ways to override it. Neither document attests the configuration or behavior of a particular installation. A reference can explain how a mechanism works while the installation still needs its own evidence.

Write the proposed claim in ordinary language

Consider a hypothetical program that reads a permitted input and produces a report. “Works with the policy” leaves several interpretations open. A more useful proposed claim would explain that the program can finish a defined report from a known input under a named configuration, while a selected disallowed operation remains denied.

That sentence helps expose missing decisions. What counts as finishing? Which output should be compared? What happens when an input is unavailable? Does the claim concern startup, normal completion, or a failure path?

A test plan can answer those questions before the team has invested in an impressive-looking total. It can also reveal that the proposed claim is too broad for the environment available.

Define the observation that would change the answer

For the hypothetical report program, a startup message would leave completion unanswered. A completed report would still leave other workloads outside the evidence. A denial would need an observation that distinguishes the intended policy decision from some unrelated failure.

These are design questions for an authorized evaluation. They are not instructions to probe a live service or remove an existing control. A useful plan identifies an appropriate environment and a way to stop if its assumptions fail.

It should also preserve the baseline. If a comparison changes both the policy and the application configuration, the reader has to disentangle which change explains the result. The narrower the intended conclusion, the easier it is to design a comparison that can challenge it.

Keep review and reproduction distinct

Reading another person's receipt can be valuable review. Repeating an execution adds a different observation. A summary should say which occurred, because the next reader may rely on that distinction when deciding what remains to be checked.

The same discipline applies to tools and access. Black Ice Labs announced a Defense Access milestone in October. Access to a research tool does not establish a later finding. The result still needs a method, a defined observation and limits appropriate to the evidence. No new capability or outcome is claimed here.

Make a narrower result usable

A conclusion can be modest and still move a project forward. If a result establishes one behavior under one set of conditions, explain how that changes the next decision. A team may now know which application behavior needs evaluation, which assumption needs confirmation, or why a proposed trial is premature.

That is more useful than expanding the wording until the test appears to answer every question. The broader statement may be easier to market, but it leaves the next engineer with a less accurate starting point.

Write the claim first. Then choose evidence capable of changing your mind about it, and preserve the unanswered questions when you publish the result.

Sources

Proxy — CryoGate technical evaluation and containment-boundary article: public project evidence and stated limits.

Linux — Seccomp BPF and Docker — Seccomp profiles: mechanism documentation, not installation attestations.

Levon Azevedo — Defense Access milestone: a historical access announcement, not evidence of subsequent research outcomes.

Research home: blackicelabs.ca/research

More articles ↗

Eva

Site guide · published material
  1. Eva

    Hello. I’m Eva, your guide to Black Ice Labs. What would you like to explore?

Explore answers from published pages. AI conversation is being connected.