Collection Profile

Companion to the Personal Data Portability Protocol (PDPP) core spec.

A connector conforms to PDPP by producing a source declaration valid under Core §5 and serving its data through a resource server conforming to Core §8. No particular collection method is required.

Overview

The Collection Profile defines how connectors collect data from source platforms and write it to a PDPP resource server. It is one fulfillment mechanism for the PDPP core protocol; pre-collected data, manual imports, and other ingestion mechanisms are equally valid.

The Collection Profile is architecturally separate from the core protocol. A resource server serving pre-collected data needs no awareness of this profile. A connector runtime implementing this profile needs no awareness of grant semantics beyond what is explicitly passed to it in the START message.

Collection method abstraction

Connectors abstract over the source platform’s data access interface. The runtime does not standardize the connector’s source-specific collection logic; it standardizes only the runtime contract around bindings, scope, state, and emitted messages. A connector that collects data via browser automation and one that calls a platform’s export API both use the same START/RECORD/STATE/DONE protocol, the same binding matching, and the same state management.

This abstraction is intentional. Many platforms do not currently offer structured data portability APIs. The browser_automation binding enables connectors that drive a browser to collect data from a platform's web UI. As platforms adopt data portability standards or offer their own APIs, connector implementations can change without changing the consent surface, grant enforcement, or query API.

Status of this document

This profile is informative. It describes what the reference connector runtime does and the artifacts it exchanges with connectors; it defines no conformance requirement. Core Section 1 states how a connector conforms to PDPP.

1. Connector Manifest Extensions

The core manifest (Section 7 of the core spec) defines the consent surface. The Collection Profile adds execution-specific fields.

{
  "protocol_version": "0.1.0",
  "connector_id": "https://registry.pdpp.dev/connectors/spotify",
  "version": "2.0.0",
  "display_name": "Spotify",
  "runtime_requirements": {
    "bindings": {
      "network": { "required": true },
      "interactive": { "required": true }
    }
  },
  "capabilities": {
    "human_interaction": ["credentials", "otp"]
  },
  "streams": [
    {
      "name": "top_artists",
      "incremental": true
    }
  ]
}

Collection-specific manifest fields

A stream declaration must not set both state_stream and parent_streams. A stream with neither field is its own checkpoint parent (self-mapped): its own STATE messages govern its own commit eligibility, exactly as in a manifest that predates this section.

Checkpoint dependency

Some streams do not carry their own cursor. A stream that rides another stream's checkpoint (for example, a reactions or attachment-metadata stream co-emitted alongside the messages that reference it) declares that relationship in the manifest so the runtime can compute which staged STATE commits are safe to persist when a run ends in a certified stream-scoped failure (see DONE) or reports incomplete detail coverage (see DETAIL_COVERAGE).

There are two declaration shapes, and a stream must use at most one. Each shape requires a specific coverage_strategy value on the same stream; a runtime must reject a manifest declaring state_stream or parent_streams with any other (or absent) coverage_strategy value on that stream (see Validation rule 7 below):

• state_stream (single parent). streams[].state_stream is a string naming exactly one other declared stream, and requires coverage_strategy: "checkpoint_window" on the declaring stream. It declares a static one-to-one mapping: this stream is always covered by that one parent's checkpoint, and the connector never emits DETAIL_COVERAGE for it — the runtime projects its checkpoint status from the parent's commit outcome directly, with no run-time override. A runtime must reject any DETAIL_COVERAGE message naming a state_stream-declared stream as its stream field, as a protocol violation (fail closed), regardless of what state_stream value that message reports. Use this shape for a stream that has no independent hydration lane of its own.
• parent_streams (one or many parents). streams[].parent_streams is a non-empty array naming one or more other declared streams, and requires coverage_strategy: "parent_detail_accounting" on the declaring stream. It declares that this stream runs its own list+detail hydration lane and emits one DETAIL_COVERAGE message per parent boundary per run (see DETAIL_COVERAGE); each declared parent's checkpoint is gated independently by its own coverage report and gap accounting, and a runtime must reject any DETAIL_COVERAGE message whose state_stream value is not a member of this declared set, as a protocol violation (see Precedence between manifest and run-time evidence). Use this shape for a detail stream that can be fed by more than one independently checkpointed list stream in the same run, or that proves its own coverage rather than inheriting a parent's outcome unconditionally.

Validation

A conformant runtime must validate every stream’s checkpoint-dependency declaration before spawning the connector, and must reject the manifest (fail closed, run not started) if any of the following holds:

1. Self-reference. state_stream equals the declaring stream's own name, or parent_streams contains the declaring stream's own name.
2. Unknown stream. state_stream, or any entry of parent_streams, names a stream not present in manifest.streams.
3. Duplicate parent. parent_streams contains the same stream name more than once.
4. Both fields present. A stream declares both state_stream and parent_streams. A runtime must enforce this directly (as its own explicit check) rather than relying solely on state_stream and parent_streams being gated to mutually exclusive coverage_strategy values as an incidental side effect — the two fields being individually valid only under different coverage_strategy values does not excuse a runtime from also rejecting a manifest that sets both.
5. Empty parent_streams. parent_streams is present but has zero entries. (Omit the field entirely for a self-mapped stream.)
6. Cycle. Following state_stream/parent_streams edges from any stream, by any path, returns to that same stream. A cycle makes the dependency graph unresolvable to a topological commit order and must be rejected at manifest validation, not discovered at run time. This holds even for a runtime that resolves only direct, one-level parent declarations: two or more direct edges can still form a cycle (for example A.state_stream = B and B.state_stream = A, or a longer chain through direct edges only), so a runtime must implement genuine cycle detection over the declared dependency graph — see non-normative notes below.
7. coverage_strategy mismatch. A stream declares state_stream without coverage_strategy: "checkpoint_window" on that stream, or declares parent_streams without coverage_strategy: "parent_detail_accounting" on that stream.

A manifest that passes this validation must have every stream’s checkpoint-dependency edges forming a directed acyclic graph, terminating in one or more streams that are self-mapped (own checkpoint parent).

Non-normative notes:

• These are direct, single-level declarations — a stream names its parent(s) directly, not through a transitive chain it expects the runtime to resolve. A runtime may additionally reject a manifest whose declared edges are more than one level deep (e.g., stream C names parent B, and B itself names parent A) if its implementation does not resolve transitive chains; the reference implementation validates exactly one level and does not resolve chains beyond it.
• Rule 6 (cycle rejection) is a genuine, implemented check, not a consequence that falls out of rules 1–5 for free. A prior draft of this section claimed a one-level-only resolver made cycle rejection "vacuously true" through rules 1–5 alone; that claim was false — two or more direct edges can still form a cycle (A.state_stream = B and B.state_stream = A, or a longer chain through direct edges only) that no single-stream rule can see, since rules 1–5 each inspect one stream's own declared edges in isolation. The reference implementation performs real cycle detection (a depth-first search with a visiting/visited coloring) over the complete declared dependency graph — every state_stream and parent_streams edge across every stream — before spawning the connector, and has conformance tests proving rejection of a 2-cycle, a 3-cycle, and a mixed state_stream/parent_streams cycle, alongside a non-regression case proving two streams legitimately sharing one parent is not a false-positive cycle. This detection is provider-neutral: it operates purely on the manifest's declared graph, with no connector-specific logic.

Standard bindings

2. Connector Run Protocol

Connectors communicate with the runtime via newline-delimited JSON (JSONL) over stdin/stdout. Each message is a single JSON object followed by a newline.

Runtime binding matching

Before spawning a connector, the runtime checks the manifest's runtime_requirements.bindings against its own capabilities. If the runtime cannot satisfy a required binding, the run must fail with a clear error before the connector process is spawned. This follows the Kubernetes scheduler pattern: connectors declare requirements, runtimes advertise capabilities.

Connector process state machine

The connector process transitions through the following states:

State transition table:

Protocol violations:

• A connector must not emit INTERACTION while already in waiting_for_interaction. A runtime that receives a second INTERACTION in this state must terminate the connector process and mark the run as failed. Note (non-normative): Runtimes that process connector messages sequentially via a single-threaded message queue may make this violation unrepresentable in practice, because the queue serializes INTERACTION processing. The protocol rule remains valid for correct connector behavior and for runtime architectures that dispatch messages concurrently.
• A connector that receives INTERACTION_RESPONSE while in collecting (no pending INTERACTION) should treat it as a fatal protocol error, write a diagnostic to stderr, and exit with non-zero status.
• START is exactly-once. It must be the first message sent by the runtime. A connector that receives START while in any state other than initializing must treat it as a fatal protocol error.

Runtime behavior on failure: The runtime must not persist STATE checkpoints from a run that terminates in the failed state, except for the certified stream-scoped failure described under DONE or the restart-abandonment exception described under Restart abandonment. State is otherwise persisted only after a successful DONE.

SKIP_RESULT is a message emitted while in the collecting state. It does not cause a state transition.

3. Messages

Runtime to Connector

START

Initializes a collection run.

{
  "type": "START",
  "run_id": "run_abc123",
  "collection_mode": "incremental",
  "scope": {
    "streams": [
      {
        "name": "top_artists",
        "time_range": {
          "since": "2025-10-11T00:00:00Z"
        },
        "fields": [
          "id",
          "name",
          "genres",
          "popularity",
          "source_updated_at"
        ]
      }
    ]
  },
  "state": {
    "top_artists": { "last_updated": "2026-03-01T00:00:00Z" }
  },
  "bindings": {
    "browser_automation": {
      "interface": "cdp",
      "ws_url": "ws://127.0.0.1:39011/devtools/browser/abc"
    },
    "network": {}
  }
}

The START message does not include the raw grant or access token. It carries a normalized scope object instead. scope is not itself a grant and has no authorization force; it is the collection target for this run. For grant-driven runs, the runtime must derive scope from the grant, must not construct a scope broader than the grant permits, and may narrow it further according to local fulfillment policy (for example, collecting only the stale streams needed to satisfy the current request). For proactive runs, the runtime derives scope from user preferences or local policy.

scope fields

CollectionStream fields

START.scope carries normalized collection targets only. It does not include issuance-time concepts such as necessity or unresolved view names; the runtime resolves those before spawning the connector.

Connector obligations for scope:

• A connector must not emit RECORD messages for streams absent from scope.streams.
• If resources or time_range is present for a stream, the connector must apply those constraints before emitting RECORD messages for that stream.
• If fields is present for a stream, the connector must not emit additional top-level fields in RECORD data for that stream, except that it may include schema-required or ingest-required top-level fields if the runtime omitted them accidentally.
• A connector that cannot honor a declared resources, time_range, or fields constraint for a stream must either emit SKIP_RESULT with reason: "scope_not_supported" and omit records for the skipped target, or fail the run. It must not silently broaden or ignore the constraint.
• A connector may retrieve broader source-side data transiently when the source platform cannot filter precisely, but it must still emit RECORD messages consistent with scope.

Connector compliance is not the only enforcement backstop. The runtime and downstream write path must reject or discard emissions that fall outside the declared scope.

State management: State is maintained at two levels:

• Global state: Used and advanced only by proactive runs (no grant). Represents archival completeness for the user's data store.
• Grant-scoped state: Used and advanced by continuous grant runs, keyed by grant_id. The runtime reads and writes this namespace through GET/PUT /v1/state/{connector_id}?grant_id={grant_id}. It ensures recurring app syncs are incremental without interfering with global archival cursors.
• Single-use runs: Receive state: null. STATE messages emitted during single-use runs are not persisted.

bindings contains a descriptor for every binding declared required: true in the manifest. For every required binding, the runtime must include a valid descriptor. Connectors must treat a missing required binding as a fatal protocol error. Connectors must ignore unknown binding keys.

INTERACTION_RESPONSE

Reply to an INTERACTION request.

{
  "type": "INTERACTION_RESPONSE",
  "request_id": "req_001",
  "status": "success",
  "data": { "email": "user@example.com", "password": "..." }
}

On timeout, the runtime must send a response with status: "timeout" rather than leaving the connector blocked indefinitely.

Connector to Runtime

RECORD

A single data record. Same envelope as the core spec (Section 4).

{
  "type": "RECORD",
  "stream": "top_artists",
  "key": "4Z8W4fKeB5",
  "data": {
    "id": "4Z8W4fKeB5",
    "name": "Radiohead",
    "genres": ["alternative rock"],
    "popularity": 82,
    "source_updated_at": "2026-03-28T00:00:00Z"
  },
  "emitted_at": "2026-04-06T15:01:00Z"
}

The op field (upsert or delete) is a directive to the resource server and is not stored as part of the record data.

STATE

Checkpoint for incremental sync.

{
  "type": "STATE",
  "stream": "top_artists",
  "cursor": { "last_updated": "2026-03-28T00:00:00Z" }
}

The runtime persists STATE only after preceding records are durably written to the resource server. Connectors should emit STATE periodically (e.g., every 1000 records) rather than only at the end of a stream.

stream names the checkpoint (state_stream), which can differ from the data stream(s) it covers. A stream is its own checkpoint unless the manifest declares state_stream or parent_streams for it (see Checkpoint dependency). Every STATE message's stream must be a checkpoint stream: either a data stream with no declared parent (self-mapped), or a stream named as another stream's state_stream/parent_streams target. A checkpoint stream's commit eligibility depends on the coverage and failure evidence of every data stream mapped to it — see Eligible-checkpoint algorithm.

The cursor object is opaque to the runtime and the resource server: its structure is defined by the connector and interpreted only by the connector on the next run.

INTERACTION

Request input from a user or agent. The connector blocks (does not emit further messages) until INTERACTION_RESPONSE arrives on stdin.

{
  "type": "INTERACTION",
  "request_id": "req_001",
  "kind": "credentials",
  "message": "Log in to Spotify",
  "schema": {
    "type": "object",
    "properties": {
      "email": { "type": "string" },
      "password": { "type": "string", "format": "password" }
    },
    "required": ["email", "password"]
  },
  "timeout_seconds": 300
}

SKIP_RESULT

Signals that a stream or resource was intentionally skipped. Does not cause a state transition.

{
  "type": "SKIP_RESULT",
  "stream": "playlists",
  "reason": "rate_limited",
  "message": "Skipped playlists: rate limit reached"
}

SKIP_RESULT may also be used when a connector cannot honor a declared scope element for a stream or resource. In that case the reason must be scope_not_supported.

SKIP_RESULT may carry an optional recovery_hint. See Recovery hints below for its shape and validation rules — the same rules apply here as for DONE.error.recovery_hint.

SKIP_RESULT may carry an optional typed continuation fact when a bounded page completed and the runtime owns the next page. It must contain boundary, slice_start, slice_end, considered, covered, remaining: true, and owner: "runtime". The counts bind the continuation to the exact proven page; a runtime must not treat an ordinary retryable skip as a healthy continuation merely because its separate coverage denominator is full. The fact proves only that slice. It must not imply complete history.

DETAIL_COVERAGE

Reports how completely a connector hydrated per-record detail for one checkpoint-parent boundary in a list+detail collection lane (a lane that fetches a list of records and then fetches per-record detail for at least a subset of them). Does not cause a state transition.

{
  "type": "DETAIL_COVERAGE",
  "reference_only": true,
  "stream": "message_attachments",
  "state_stream": "messages",
  "required_keys": ["msg_1", "msg_2", "msg_3"],
  "hydrated_keys": ["msg_1", "msg_2"],
  "gap_keys": ["msg_3"]
}

A connector with a list+detail lane must emit one DETAIL_COVERAGE message per distinct state_stream boundary after that boundary's detail work for the run settles, and must place it after the last RECORD or DETAIL_GAP it emits for that detail stream and boundary in the run. A connector exempt from a per-record detail fetch (flat streams only) is not required to emit DETAIL_COVERAGE.

Non-normative note: this ordering rule is a conformance obligation on the connector; the reference implementation does not currently enforce message sequence/ordering for DETAIL_COVERAGE at the runtime level (it accepts the message whenever it arrives and evaluates coverage as of terminal DONE). A runtime may choose to validate ordering explicitly; the eligible-checkpoint algorithm's correctness in the reference implementation does not depend on runtime-side ordering enforcement, only on the connector honestly reporting complete state by the time DONE is evaluated.

Key-set validation. Within one portable v0.1 DETAIL_COVERAGE message:

• required_keys, hydrated_keys, and gap_keys must each contain no duplicate key.
• Every key in hydrated_keys or gap_keys must also appear in required_keys.
• A key in required_keys that appears in neither outcome set is an unaccounted key: its parent boundary's coverage is incomplete (see Eligible-checkpoint algorithm).

Multi-parent streams. A detail stream declared with manifest parent_streams (see Checkpoint dependency) may emit more than one DETAIL_COVERAGE message in the same run — one per parent boundary that settled. The runtime must evaluate and gate each declared parent's checkpoint independently from the others' coverage; it must not reject two DETAIL_COVERAGE messages solely because they share the same stream while their state_stream values differ. Every state_stream value reported must be a member of the stream's manifest-declared parent_streams set — a runtime must reject a DETAIL_COVERAGE naming a state_stream outside that set as a protocol violation (see Precedence between manifest and run-time evidence).

Reference-implementation extension: optional_skip_keys. The reference implementation accepts an additional optional_skip_keys outcome set as a non-portable extension. It must not be used to claim portable v0.1 conformance: an independent v0.1 runtime may reject the extension or treat those keys as unaccounted, and the v0.1 checkpoint algorithm below does not credit it. A runtime and connector that explicitly opt into the reference extension may credit a key only when the connector has affirmative, provider-authored evidence of a terminal, record-specific absence. Operator or deployment configuration alone is never sufficient. An HTTP status code alone, response age alone, transport failure, retry exhaustion, or generic access denial must leave the key unaccounted and therefore retryable. A future profile revision must standardize the manifest declaration and value vocabulary before accepted absence becomes a portable checkpoint outcome.

DETAIL_GAP

Reports a durable, retryable per-record detail failure within a list+detail lane. Does not cause a state transition.

{
  "type": "DETAIL_GAP",
  "stream": "message_attachments",
  "parent_stream": "messages",
  "record_key": "msg_3",
  "reason": "temporary_unavailable",
  "retryable": true,
  "detail_locator": { "message_id": "msg_3" }
}

Parent scoping and key collision. A DETAIL_GAP names the checkpoint boundary it accounts for via parent_stream. For a stream with exactly one declared parent (state_stream, or a single-entry parent_streams), parent_stream may be omitted; the runtime must treat the omission as naming that one parent. For a stream with more than one declared parent, the runtime must treat a DETAIL_GAP with no parent_stream, or with a parent_stream not matching the coverage report being evaluated, as not accounting for that report's required key — even when the stream and record_key match exactly. The same detail key can be legitimately gapped under one parent while hydrated or covered under a different parent in the same run; a gap recorded against one parent must not satisfy another parent's coverage. A DETAIL_GAP emitted before this section's parent-scoping rule existed (no parent_stream) satisfies coverage only for a stream that has exactly one declared parent; it must not be treated as satisfying any one parent of a stream with more than one declared parent.

A gap_keys entry in DETAIL_COVERAGE is not by itself proof of a durable retry obligation. The runtime must additionally confirm a matching DETAIL_GAP exists for the same stream and parent boundary before crediting that key as accounted; an unmatched gap_keys entry leaves the key unaccounted for coverage purposes, with the same fail-closed effect as a key omitted from every outcome set.

STREAM_EVIDENCE

Reports independently measured, final enumeration-outcome counts for a stream declared with manifest state_stream (a checkpoint-dependent child with no cursor of its own, ineligible to emit DETAIL_COVERAGE). Does not cause a state transition and never gates any checkpoint commit.

{
  "type": "STREAM_EVIDENCE",
  "reference_only": true,
  "stream": "message_bodies",
  "considered": 214,
  "outcomes": {
    "emitted": 200,
    "unchanged": 11,
    "gapped": 2,
    "unaccounted": 1
  }
}

emitted + unchanged + gapped + unaccounted must equal considered exactly; every field is a disjoint partition of the same considered keys, not an independent measurement that happens to sum correctly by construction.

Every count field above (considered and all four outcomes.* fields) is normatively capped at 9007199254740991 (Number.MAX_SAFE_INTEGER), not an arbitrary-precision non-negative JSON integer. This is a v0.1 implementation bound, not a claim that a real stream cannot exceed this cardinality: it exists so every conformant runtime can represent, compare, and arithmetic- check these fields using the host language's native double-precision number type without a bignum dependency, since JSON itself imposes no integer size limit and a value above 2^53 - 1 cannot round-trip exactly through a standard JSON parse into a JS/most-language native number. A value of 9007199254740992 or greater for any of these five fields fails the "is a non-negative integer" check in the rejection rule below exactly as a negative or non-integer value does.

A connector may emit at most one STREAM_EVIDENCE per stream per run. It is optional: a state_stream child with no independent hydration lane, or that chooses not to measure itself, emits nothing and keeps the existing checkpoint-inheritance/unknown projection behavior for that stream unchanged.

An enumeration boundary is established for a stream in a run when the connector's hydration lane for that stream actually ran its enumeration step (walked a window, list, or snapshot boundary and attempted to account for every key it found) at least once during the run — as opposed to a run that skipped that lane entirely (for example, a scheduled run whose window for this stream was empty by schedule, or a run mode that never invokes this stream's hydration lane at all). A connector must not emit STREAM_EVIDENCE for a run that did not establish an enumeration boundary for that stream; it must not substitute considered: 0, outcomes: {emitted: 0, unchanged: 0, gapped: 0, unaccounted: 0} for a withheld message, since that shape is indistinguishable on the wire from a boundary that was genuinely walked and found empty.

Sender-side obligations, not runtime-enforceable. The considered derivation constraint above and the enumeration-boundary requirement in this paragraph both bind the connector's internal measurement process, which the wire message cannot carry and a receiving runtime cannot observe or verify: considered computed honestly at the hydration site and considered computed as emitted + gapped (an identity over the run's own successful-emission and gap counts, with no independent term) are bit-identical integers on the wire, and a boundary genuinely walked and found empty is bit-identical to an all-zero outcomes object substituted without walking anything. A runtime must not reject a STREAM_EVIDENCE message for violating either constraint — there is no wire-observable signal to reject on — and neither constraint is a portable v0.1 conformance property a third party can test for compliance or non-compliance, the same posture this profile already takes with the optional_skip_keys extension below. These MUSTs exist to state the honesty contract a connector author commits to when choosing to emit this message at all, not a check any runtime performs.

outcomes.unchanged carries the same non-runtime-enforceable character: a suppressed key never crosses the wire at all, so no receiving runtime can independently verify the count. outcomes.emitted and outcomes.gapped are different — see the closure note below — because both correspond to messages (RECORD, DETAIL_GAP) the runtime does independently observe and count.

STREAM_EVIDENCE is deliberately distinct from DETAIL_COVERAGE: it carries no state_stream field, no key sets, and a runtime must not let accepting, rejecting, or omitting a STREAM_EVIDENCE message affect any checkpoint stream's commit eligibility, including under the Eligible-checkpoint algorithm — see the DETAIL_COVERAGE prohibition below, which this message provides the alternative for.

A runtime must reject (fail closed, as a protocol violation) a STREAM_EVIDENCE message when any of the following holds:

1. reference_only is not true.
2. stream is not present in the run's scope.streams.
3. stream is not declared with manifest state_stream (i.e. is parent_streams-declared or self-mapped).
4. Any of considered, outcomes.emitted, outcomes.unchanged, outcomes.gapped, or outcomes.unaccounted is not a non-negative integer no greater than 9007199254740991 (Number.MAX_SAFE_INTEGER; see the field table above), or outcomes.emitted + outcomes.unchanged + outcomes.gapped + outcomes.unaccounted does not equal considered exactly.
5. A STREAM_EVIDENCE was already accepted for the same stream in the same run. "Same run" means the same run_id (see START): a resumed or retried collection that the runtime assigns a new run_id is a different run for this rule, and a STREAM_EVIDENCE accepted under a prior run_id does not count as "already accepted" against a subsequent run_id for the same stream. A runtime may implement this scoping by any mechanism that achieves the same-run_id guarantee (for example, keying on the wire run_id field explicitly, or on an internal object whose lifetime is provably one-to-one with a single run_id); this profile does not prescribe the mechanism, only the guarantee.

A runtime must not silently drop a rejected STREAM_EVIDENCE in a way that lets the stream fall through to checkpoint inheritance as if nothing had been reported.

Runtime-verifiable closure over emitted and gapped (implementation- specific, not a root-protocol must). Unlike unchanged, both emitted and gapped correspond to wire messages (RECORD, DETAIL_GAP) a runtime already independently observes and counts, so a runtime may additionally reject a STREAM_EVIDENCE whose outcomes.emitted or outcomes.gapped value is not reconcilable against its own count of those messages for the stream — for example, requiring outcomes.emitted to equal the number of distinct keys the runtime's own ingest path durably accepted for the stream this run (closing intra-run duplicate-key inflation and claimed-but-rejected records in one check, since a rejected or repeated key never adds a second distinct entry), and requiring outcomes.gapped to not exceed the runtime's own durable DETAIL_GAP count for the stream this run. This profile does not standardize the exact reconciliation a runtime performs — only that emitted/gapped, unlike unchanged, are candidates for it because the runtime has an independent, wire-observed count to check against. The reference implementation's specific mechanism is normative in openspec/specs/reference-implementation-runtime/, not in this root protocol document.

Compatibility. This profile does not require a runtime to reject an unrecognized message type; a runtime's dispatch behavior for unknown types is an implementation choice. The reference implementation's runtime treats an unrecognized msg.type as fatal (it fails the entire run rather than ignoring the message), so STREAM_EVIDENCE is a breaking wire addition for that runtime's older builds, and any runtime with the same fail-closed dispatch posture. A runtime with this posture that will receive STREAM_EVIDENCE must be deployed, with STREAM_EVIDENCE recognized, before any connector build that emits STREAM_EVIDENCE is deployed against it — a coordinated, runtime-first rollout, not independent connector/runtime versioning. A new runtime paired with an old connector that never emits STREAM_EVIDENCE is unaffected regardless of dispatch posture (that direction always degrades gracefully, because the message is simply never sent).

Read-model note. An accepted STREAM_EVIDENCE fact is folded into that stream's terminal collection data on every run-termination path (success, failure, timeout, cancellation), the same as any other stream's accepted runtime fact. Whether that run's fact is the one a runtime surfaces to the owner as a stream's current coverage is governed entirely by whatever run-selection policy the runtime already applies to every other stream's coverage fact, unmodified by this message: a STREAM_EVIDENCE-derived fact must not be surfaced in place of the run a runtime's existing selection policy would otherwise choose, and accepting a STREAM_EVIDENCE message must not itself become, or override, that selection policy.

Coverage projection is out of scope for this section. This profile defines the wire message, its validation, and its non-interaction with checkpoint commit. It does not define a portable v0.1 coverage-condition vocabulary (values such as "fully covered," "partially covered," or "unknown") for any stream shape, STREAM_EVIDENCE-derived or otherwise — no such vocabulary exists elsewhere in this profile either. A runtime may derive its own owner-facing coverage projection from an accepted STREAM_EVIDENCE fact's considered/outcomes partition using whatever read-model policy it already applies to every other stream's considered/covered pair, by first deriving a covered value from the outcomes that genuinely account for a considered key without leaving it a shortfall (emitted + unchanged) and deliberately excluding gapped and unaccounted from that sum (for example, treating the derived covered < considered with no pending retryable gap as a proven shortfall, distinct from a proven-complete derived covered === considered, and treating a pending retryable gap as taking precedence over that shortfall/complete distinction). This profile does not standardize that vocabulary or its precedence, must not be read as requiring any specific one, and a runtime may choose not to project a coverage condition from STREAM_EVIDENCE at all. A runtime must not, however, treat an accepted STREAM_EVIDENCE fact with a positive enumeration boundary and nonzero outcomes.gapped or outcomes.unaccounted as evidence the stream is fully covered — whatever vocabulary a runtime's own read model uses, the message's field-level guarantee (outcomes.emitted + outcomes.unchanged + outcomes.gapped + outcomes.unaccounted === considered) must not be laundered into a stronger claim than the numbers themselves support. A future profile revision may standardize a portable coverage-condition vocabulary; until then this is runtime-specific, non-portable behavior, the same posture this profile already takes with optional_skip_keys (see DETAIL_COVERAGE). The reference implementation's own coverage-projection behavior for STREAM_EVIDENCE is specified as a capability requirement in openspec/specs/reference-implementation-runtime/ (pending archive of openspec/changes/prove-state-stream-child-coverage/), not in this root protocol document.

PROGRESS

Optional progress update for display in runtime UIs.

{
  "type": "PROGRESS",
  "stream": "messages",
  "message": "Downloaded 500 of 2196 messages",
  "count": 500,
  "total": 2196
}

DONE

Signals completion. Must be the final message emitted by the connector.

{
  "type": "DONE",
  "status": "succeeded",
  "records_emitted": 2196
}

On failure:

{
  "type": "DONE",
  "status": "failed",
  "records_emitted": 0,
  "error": { "message": "Authentication failed", "retryable": true }
}

A failed run certifies a stream-scoped failure only when both of these conditions hold:

1. DONE.error.code is stream_collection_failed.
2. The run previously emitted at least one in-scope SKIP_RESULT with reason: "stream_collection_failed" and a non-empty stream naming each failed data stream.

A runtime must verify condition 2 structurally (a named, in-scope SKIP_RESULT was actually observed this run) and must not treat DONE.error.code alone as certification. If condition 1 holds but no in-scope SKIP_RESULT{reason: "stream_collection_failed"} was observed, the run is an ordinary uncertified failure and the default fail-closed rule applies: no staged STATE is persisted.

Cancellation precedence. If the runtime has recorded an owner-initiated cancellation for the run (for example, a mid-run revocation), the runtime must resolve the run as cancelled and must not evaluate or apply the stream-scoped-failure exception, even if the connector emitted a structurally certified DONE{status: "failed", error.code: "stream_collection_failed"} before the runtime observed the cancellation. Cancellation is checked, and takes precedence, before a terminal DONE is evaluated for certification.

Eligible-checkpoint algorithm

For a certified stream-scoped failure, the runtime may persist staged STATE for checkpoint streams that do not cover any named failed data stream. A runtime that implements this exception must compute eligibility as follows, run against the run's complete staged STATE map (every checkpoint stream for which a STATE message was received this run):

1. Resolve each failed data stream to its checkpoint stream(s). For each data stream named by an in-scope SKIP_RESULT{reason: "stream_collection_failed"}, compute its set of checkpoint parents using the manifest's static declaration, per Precedence between manifest and run-time evidence: the manifest's state_stream (single parent), or the manifest's full declared parent_streams set (every declared parent, whether or not it received a live DETAIL_COVERAGE report this run — a failed stream's live evidence is inherently incomplete, so every declared parent is a candidate to withhold); if neither is declared, the stream is self-mapped (its own name is its one checkpoint parent).
2. Union every failed data stream's checkpoint parents into one set of ineligible checkpoint streams.
3. Compute detail-coverage shortfalls independently of the failure. For every staged checkpoint stream, evaluate every DETAIL_COVERAGE report gating it: a report is incomplete if any required_keys entry is unaccounted (present in neither hydrated_keys nor a gap_keys entry backed by a matching DETAIL_GAP for the same stream and parent boundary — see DETAIL_GAP), or if a manifest-declared parent relationship for an in-scope detail stream has no DETAIL_COVERAGE report at all this run. Add every checkpoint stream with an incomplete report to the ineligible set. A reference-implementation extension may additionally credit its explicitly opted-in optional_skip_keys, but that is not portable v0.1 behavior.
4. Commit every staged checkpoint stream not in the ineligible set. The runtime must not persist STATE for any checkpoint stream in the ineligible set (from step 2 or step 3).
5. Partial checkpoint-store failure. If persisting an individual eligible checkpoint stream's STATE fails after the eligibility set is computed (for example, a resource-server write error), the runtime must fail the run as a runtime error. A partial commit failure must not be reported as succeeded, and any checkpoint stream not yet committed at the point of failure remains eligible for retry on the next run. The runtime must make the following observable, though not necessarily from a single field or object: the total count of checkpoint streams staged and the total count durably committed before the failure (a bounded numeric summary; this may be all a single terminal-result field exposes), the identity of the specific checkpoint stream whose persistence attempt failed (for example, in a diagnostic message or a dedicated failure event), and the identity of each checkpoint stream that was durably committed before the failure (for example, via a per-stream commit event emitted at the time of that commit, or by reading back the durably persisted state after the run). A runtime is not required to expose a single response field naming every staged-and-committed stream together; it must not expose only bounded counts with no path at all to recovering which specific streams committed.

The run's own status remains failed regardless of how many checkpoint streams commit under this exception; every named failed data stream, and every checkpoint stream withheld under steps 2–3, remains unproven and eligible for retry on the next run.

A missing or mismatched terminal code, a missing or untargeted skip, an out-of-scope stream, a protocol violation, an invalid terminal count or exit code, a process exit without valid DONE, or cancellation must preserve the default fail-closed rule and persist no staged STATE.

Restart abandonment. A process exit without valid DONE caused by the CONTROLLER being replaced or restarted -- not by the connector failing -- may persist a staged STATE checkpoint for a checkpoint stream that satisfies ALL of the following, and must persist none otherwise:

1. The run’s terminal reason is a controller-lifecycle reason (the controller died; the connector did not report failure).
2. The checkpoint stream is not a declared detail-coverage parent in the connector’s manifest, so it can never face a DONE-time coverage verdict. Eligibility is derived from the MANIFEST alone; a connector must not be able to declare its own eligibility.
3. The stream has no pending detail gap and no unproven coverage for the completed prefix.

This exception exists because a walk longer than the interval between controller restarts can otherwise never converge -- it is a completeness failure, not a slowness one. It does not weaken the invariant that a cursor must not advance past records whose coverage was not proven: a stream that could face such a verdict is excluded by condition 2, and unproven coverage is excluded by condition 3. The run’s status remains failed/abandoned and every withheld stream stays eligible for retry.

error may carry code and/or recovery_hint, in addition to the required message and retryable:

• code is a stable, connector-defined cause identity (e.g. distinguishing one failure mode from another). It is a bounded snake_case identifier (a lowercase letter followed by up to 63 lowercase letters, digits, or underscores), an identity rather than an instruction — the runtime must not treat code as, or derive, an owner-facing recovery action from it.
• recovery_hint is the connector's declaration of the owner-facing recovery action. It uses the exact same bounded shape and vocabulary as SKIP_RESULT.recovery_hint — see Recovery hints.

code and recovery_hint answer different questions (what went wrong vs. what to do about it) and must be validated and consumed independently; a runtime must not infer one from the other.

Precedence between manifest and run-time evidence

A data stream's checkpoint parent(s) are declared statically in the manifest (state_stream or parent_streams). The manifest is authoritative: it declares the permitted parent shape, and the connector's own DETAIL_COVERAGE messages (state_stream field, per message) may only select or report evidence within that declared shape for a given run — they must not introduce a parent the manifest did not declare, and must not override a static single-parent declaration. A runtime must resolve a data stream's checkpoint parent(s), and validate live evidence against that resolution, as follows:

1. state_stream (static single parent). The stream's checkpoint parent is always the manifest's declared state_stream, for every run, with no run-time override. The connector must not emit DETAIL_COVERAGE naming this stream as stream at all; a runtime must reject any such message as a protocol violation (fail closed), regardless of what state_stream value it reports. A state_stream-declared stream may instead emit STREAM_EVIDENCE to report an independently measured coverage fact about itself; that message cannot gate any checkpoint commit and does not relax this prohibition.
2. parent_streams (declared parent set). The stream's checkpoint parents are the manifest's declared set. A runtime must reject any DETAIL_COVERAGE message naming this stream as stream whose state_stream value is not a member of the declared set, as a protocol violation (fail closed). For a declared parent that HAS a live DETAIL_COVERAGE report this run, that report's own coverage/gap accounting gates its checkpoint (see Eligible-checkpoint algorithm). For a declared parent with NO live report this run, where the data stream is in-scope and has staged state, the runtime must treat that parent boundary as unproven and withhold it — the same fail-closed treatment as an incomplete coverage report — rather than silently dropping it from the dependency set or silently treating it as satisfied.
3. Self-mapping is the default. If the stream declares neither state_stream nor parent_streams, its own name is its one checkpoint parent, and it is unaffected by any of the above.

This is a manifest-is-authoritative model: live evidence for a run can select or report within the manifest's declared shape, but can never contradict, widen, or override it. There is a conflict-rejection case for manifest vs. run-time evidence, unlike prior drafts of this section: a state_stream-declared stream that emits any DETAIL_COVERAGE is rejected outright, and a parent_streams-declared stream's DETAIL_COVERAGE naming a parent outside its declared set is rejected outright. A manifest declaring parent_streams: ["a", "b"] may see the connector's live evidence for a given run name only "a" (for example, if "b"'s boundary produced no detail this run); the runtime withholds "b"'s checkpoint as unproven for this run rather than treating the narrower live report as redefining the declared dependency.

Recovery hints

SKIP_RESULT.recovery_hint and DONE.error.recovery_hint share one bounded, provider-neutral shape and vocabulary:

• recovery_hint is either a bare string from the closed action vocabulary below, or an object { action: string, retryable?: boolean } where action must be present and from that vocabulary, and retryable, if present, must be a boolean.
• An empty object {} or an object with only retryable field is a protocol violation: if a connector supplies a recovery hint as an object, the action field is mandatory.
• Action vocabulary: retry_by_runtime, retry_on_connector_upgrade, refresh_credentials, manual_action_required, update_selector, upstream_unblock, not_retriable, unknown.
• A connector requests a specific owner-facing recovery action only through recovery_hint. A present, valid recovery_hint is authoritative: a runtime must not override it, and must not treat code, message, or any other connector-authored free-form text as the connector's requested action.
• A runtime must treat an absent recovery_hint as "no hint declared," and may fall through to its own generic, connector-neutral policy for choosing a default action — for example from the retryable flag, or from bounded, provider-neutral classification of the error text (such as recognizing generic authentication or browser-infrastructure failures). That fallback must not infer provider-specific intent, and must not be, or become, a connector-specific text/identity heuristic.
• A recovery_hint that is present but does not match the shape or vocabulary above is a protocol violation: the runtime must reject the enclosing message (fail closed), not silently drop the field or substitute a guessed action.

4. Connector and runtime behavior summary

A conformant connector:

1. Reads START from stdin before emitting any messages.
2. Emits only valid JSONL messages as defined in this profile.
3. Emits DONE as the final message in all cases (including failures where possible).
4. Emits STATE periodically for streams that support incremental sync.
5. Does not store secrets (credentials, OTP codes) in STATE.
6. Does not emit INTERACTION while in waiting_for_interaction.
7. Treats missing required bindings as fatal errors.
8. Exits with status 0 on succeeded, non-zero on failed or cancelled.
9. Emits RECORD messages only within the scope provided in START: no undeclared streams, no records outside declared resources or time_range, and no extra top-level fields when fields is present.
10. If it cannot honor a declared resources, time_range, or fields constraint, emits an explicit SKIP_RESULT or fails the run; it never silently broadens scope.
11. If it runs a list+detail lane, emits DETAIL_COVERAGE per checkpoint-parent boundary per run, with every required key accounted in hydrated_keys or gap_keys backed by a matching DETAIL_GAP (see DETAIL_COVERAGE). The reference implementation's optional_skip_keys extension is not required for portable v0.1 conformance.
12. Scopes every DETAIL_GAP to the checkpoint-parent boundary it accounts for via parent_stream whenever the affected detail stream has more than one declared parent.
13. If it emits STREAM_EVIDENCE for a state_stream-declared stream, emits at most one per stream per run, measures considered and each outcomes.* field at that stream's own hydration site as a disjoint partition summing to considered (never a parent-count alias, never an identity derived from its own successful-emission/gap counts), and withholds the message entirely for a run that established no genuine enumeration boundary for that stream (see STREAM_EVIDENCE).

A conformant connector runtime:

1. Performs binding matching before spawning the connector process.
2. Sends START as the first and only START message.
3. Handles INTERACTION messages by prompting the user or agent and sending INTERACTION_RESPONSE.
4. Sends INTERACTION_RESPONSE with status: "timeout" if no response arrives within timeout_seconds.
5. Persists STATE only after preceding records are durably written.
6. Does NOT persist STATE on cancelled runs or uncertified failed runs; for a certified stream-scoped failure, persists only staged checkpoint streams computed eligible under the Eligible-checkpoint algorithm. Checks for and honors an owner-initiated cancellation before evaluating a terminal DONE for stream-scoped-failure certification.
7. Uses the connector's global state namespace for proactive runs, the grant_id-scoped namespace for continuous grant runs, and state: null for single_use runs.
8. Terminates the connector process on protocol violations.
9. Does not log or persist credential data from INTERACTION_RESPONSE.
10. Sends an explicit non-empty scope in START. For grant-driven runs, this scope is a normalized, possibly narrowed projection of the grant and must not include wildcard stream names.
11. For grant-driven runs, never constructs a scope broader than the grant permits.
12. Rejects or discards connector emissions that fall outside the declared scope before durable write.
13. Terminates an active grant-driven run as soon as practical after learning that the grant was revoked.
14. Validates every stream's checkpoint-dependency declaration (state_stream/parent_streams) before spawning the connector, and fails closed (does not start the run) on self-reference, an unknown parent, a duplicate parent, both fields present on one stream, an empty parent_streams, or a cycle of any length in the declared dependency graph (see Checkpoint dependency: Validation). Cycle detection (rule 6) is implemented and tested as a genuine check over the complete declared graph — not satisfied vacuously by the other checks — because two or more direct edges can form a cycle that a single-stream check cannot see (see the non-normative notes under Validation).
15. Resolves a data stream's checkpoint parent(s) for a run from the manifest's static declaration only (state_stream or the full declared parent_streams set), and validates any live DETAIL_COVERAGE evidence against that declared shape: rejects a state_stream-declared stream's DETAIL_COVERAGE outright, rejects a parent_streams-declared stream's DETAIL_COVERAGE naming an undeclared parent, and withholds a declared parent that received no live report this run rather than dropping it or treating it as satisfied (see Precedence between manifest and run-time evidence).
16. Fails the run as a runtime error, without reporting succeeded, if persisting an eligible checkpoint's STATE fails partway through committing multiple staged checkpoints; makes the staged/committed counts, the failing checkpoint stream's identity, and each committed checkpoint stream's identity observable (not necessarily from one field — see Eligible-checkpoint algorithm step 5).
17. Rejects (fail closed) a STREAM_EVIDENCE message that fails any of the five validation rules in STREAM_EVIDENCE, and folds an accepted one into that stream's own terminal collection fact without letting it affect any checkpoint stream's commit eligibility or any DETAIL_COVERAGE shortfall determination, including under the Eligible-checkpoint algorithm.

5. TypeScript Types

type InteractionKind = 'credentials' | 'otp' | 'manual_action';
type StreamState = Record<string, Record<string, unknown>>;
type TimeRange = { since?: string; until?: string };
type CollectionStream = {
  name: string;
  resources?: string[];
  time_range?: TimeRange;
  fields?: string[];
};
type CollectionScope = {
  streams: CollectionStream[];
};

type ManifestStream = {
  name: string;
  incremental?: boolean;
  // A stream must declare at most one of state_stream / parent_streams.
  // Neither present means the stream is its own checkpoint (self-mapped).
  state_stream?: string;
  parent_streams?: string[]; // non-empty when present
  [key: string]: unknown;
};

type RuntimeMessage =
  | {
      type: 'START';
      run_id: string;
      collection_mode: 'full_refresh' | 'incremental';
      scope: CollectionScope;
      state: StreamState | null;
      bindings: Record<string, Record<string, unknown>>;
    }
  | {
      type: 'INTERACTION_RESPONSE';
      request_id: string;
      status: 'success' | 'cancelled' | 'timeout';
      data?: Record<string, unknown>;
    };

type ConnectorMessage =
  | {
      type: 'RECORD';
      stream: string;
      key: string | string[];
      data: Record<string, unknown>;
      emitted_at: string;
      op?: 'upsert' | 'delete';
    }
  | {
      type: 'STATE';
      stream: string;
      cursor: Record<string, unknown>;
    }
  | {
      type: 'INTERACTION';
      request_id: string;
      kind: InteractionKind;
      message: string;
      schema?: Record<string, unknown>;
      timeout_seconds?: number;
    }
  | {
      type: 'SKIP_RESULT';
      stream?: string;
      reason?: string;
      message?: string;
      recovery_hint?: RecoveryHint;
    }
  | {
      type: 'DETAIL_COVERAGE';
      reference_only: true;
      stream: string;
      state_stream: string;
      required_keys: (string | number)[];
      hydrated_keys: (string | number)[];
      gap_keys?: (string | number)[];
    }
  | {
      type: 'DETAIL_GAP';
      stream: string;
      parent_stream?: string;
      record_key?: string | number;
      reason?: string;
      retryable?: boolean;
      detail_locator?: Record<string, unknown>;
      list_cursor?: Record<string, unknown>;
      last_error?: Record<string, unknown>;
      gap_id?: string;
      lease_id?: string;
    }
  | {
      type: 'STREAM_EVIDENCE';
      reference_only: true;
      stream: string;
      considered: number;
      outcomes: {
        emitted: number;
        unchanged: number;
        gapped: number;
        unaccounted: number;
      };
    }
  | {
      type: 'PROGRESS';
      stream?: string;
      message: string;
      count?: number;
      total?: number;
    }
  | {
      type: 'DONE';
      status: 'succeeded' | 'failed' | 'cancelled';
      records_emitted: number;
      error?: { code?: string; message: string; recovery_hint?: RecoveryHint; retryable: boolean };
    };

type RecoveryAction =
  | 'retry_by_runtime'
  | 'retry_on_connector_upgrade'
  | 'refresh_credentials'
  | 'manual_action_required'
  | 'update_selector'
  | 'upstream_unblock'
  | 'not_retriable'
  | 'unknown';

type RecoveryHint = RecoveryAction | { action: RecoveryAction; retryable?: boolean };