# Scrapwire Protocol Specification

Status: Draft v1

This document defines the Scrapwire protocol as a standalone interoperability target. Implementations SHOULD be able to use this document without studying a reference implementation.

Normative terms such as MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as described in RFC 2119 / RFC 8174.

## 1. Goals

The protocol is designed to provide:

- private, secure, and anonymous communication over Veilid-backed transport and storage;
- communication that remains entirely within the Veilid network and does not require or reference IP networking;
- interoperable support for direct messaging, group messaging, and one-publisher/many-followers publications;
- support for one-time and persistent invitations across all supported scopes;
- robust behavior under Veilid latency and partial propagation;
- clean separation between service/protocol logic and interface logic;
- forward-compatible extension points for future features such as call signaling.

## 2. Non-Goals

Protocol v1 does not attempt to define:

- any direct, fallback, or negotiated IP-based transport;
- plaintext-accessible message storage for non-participants;
- voice or media transport;
- UI behavior, rendering, input handling, or interface-specific state;
- large-scale public discovery infrastructure beyond invitation-based reachability.

Voice and media transport are out of scope for v1. The protocol MUST, however, reserve safe extension points for future call signaling payloads. Any future call signaling defined for Scrapwire MUST remain Veilid-native and MUST NOT introduce IP-networking references into protocol flows.

## 3. Security and Privacy Requirements

An implementation is non-conformant if it allows any of the following without deliberate user choice:

- exposure of a peer's identity, location, IP address, or comparable network-identifying information;
- plaintext access to messages, attachments, control payloads, or publication content by non-participants;
- mandatory use of stable, globally linkable identifiers when scoped pseudonyms suffice.

All protocol communication MUST occur over Veilid-backed transport/storage primitives. Implementations MUST NOT require, negotiate, advertise, discover, exchange, or otherwise reference IP addresses, hostnames, ports, sockets, ICE/STUN/TURN data, or other direct-network rendezvous material as part of Scrapwire protocol operation.

Implementations MUST minimize metadata leakage. Persistent public invitations MAY intentionally reveal that a user, group, or publication is reachable, but MUST NOT reveal more than the user-visible sharing model requires.

## 4. Terminology

- **Scope**: one of `dm`, `group`, or `publication`.
- **Identity Root**: long-lived local secret from which scoped protocol identities and invite material are derived.
- **Conversation**: a DM, group, or publication instance.
- **Manifest**: metadata that points to feeds, epochs, and capabilities for a conversation.
- **Feed**: logical append-only writer-owned stream of protocol frames, realized as an ordered sequence of Veilid-backed segments.
- **Segment**: a single Veilid-backed storage object that carries one contiguous portion of a Feed's history.
- **Control Feed**: feed containing membership, policy, or capability state rather than end-user content.
- **Envelope**: the decrypted logical message object.
- **Outer Frame**: transport-facing frame carrying ciphertext and minimal sync metadata.
- **Epoch**: versioned cryptographic era for a conversation.
- **Blob Reference**: encrypted capability/reference to attachment or image content.
- **Blob Transport Profile**: named interoperability profile that defines how logical blob references map to concrete Veilid-backed blob publication, retrieval, and capability encoding.
- **Invite**: redeemable capability used to initiate or join a scope.
- **Capability Advertisement**: payload describing supported protocol versions and optional features.

## 5. Protocol Layers

The protocol is divided into three layers:

1. **Transport and persistence layer**: Veilid-backed routing, feed storage, and blob storage.
2. **Scrapwire service/protocol layer**: identities, invites, crypto sessions, ordering, sync, blobs, and extension handling.
3. **Interface layer**: TUI, GUI, web, mobile, or other presentation logic.

The transport and persistence layer for Scrapwire protocol traffic is Veilid-only. Conformant implementations MUST NOT define alternate direct-network transports for protocol compatibility.

Logical feeds MUST use segmented Veilid-backed records rather than a single forever-growing record. This segmented model is the canonical Scrapwire history/storage model because conformant implementations MUST account for Veilid record/subkey limits and MUST avoid unbounded hot-record growth.

The service/protocol layer MUST remain independent of any specific interface. A conformant implementation MUST be able to expose protocol operations without embedding UI state or rendering concerns in core messaging logic.

## 6. Conformance Targets

### 6.1 Mandatory v1 Support

An interoperable v1 implementation MUST support:

- `dm` scope;
- `group` scope;
- `publication` scope;
- one-time and persistent contact invites;
- one-time and persistent group invites;
- one-time and persistent publication invites;
- encrypted text messages;
- encrypted attachment and image references;
- segmented feed storage and traversal;
- versioned envelopes;
- capability advertisement and negotiation;
- deterministic ordering, deduplication, and replay handling;
- safe handling of unknown payload kinds.

### 6.2 Optional v1 Features

An implementation MAY support the following as optional user-visible features while remaining conformant:

- reactions;
- message edits and deletes;
- read receipts;
- image preview metadata;
- publication follow-request approval workflows;
- local blob caching policies beyond the minimum needed for message access.

Optional features MUST be surfaced via capability advertisement. A sender MUST NOT require a receiver to understand an optional feature unless support has been advertised.

## 7. Object Model

### 7.1 Identity Root

Each local user has an Identity Root. The Identity Root MUST remain local and MUST NOT be published directly as a protocol identifier. Implementations SHOULD derive scope-local identifiers and invite material from the Identity Root to reduce linkability.

### 7.2 Scoped Member Identifier

Each conversation member has a scope-local member identifier. Implementations SHOULD avoid using a single stable identifier across unrelated conversations unless the user explicitly chooses that behavior.

### 7.3 Conversation Manifest

Each conversation has a manifest containing at minimum:

- `conversation_id`;
- `scope`;
- `protocol_version`;
- `active_epoch`;
- referenced feeds;
- current feed tips / active segment references for writable feeds;
- advertised capabilities;
- invite policy metadata, where applicable.

### 7.4 Feed Reference

Each Feed Reference MUST identify:

- the feed identifier;
- the owning writer/member;
- the transport/storage reference needed to read it;
- the current tip / active segment reference when the feed is writable;
- the epoch range in which the feed is valid, if applicable.

### 7.5 Segment Object

Feed history/storage is canonically segmented. A logical Feed MUST be represented as an ordered chain of Segment objects rather than as one forever-growing Veilid record.

Each Segment object MUST define at least:

- `feed_id`;
- `segment_index`;
- `epoch`;
- `prev_segment_ref`;
- `next_segment_ref`;
- `state`;
- `message_count`;
- `byte_count`;
- `checkpoint_summary`.

Segment field semantics:

- `feed_id` MUST identify the logical Feed to which the Segment belongs.
- `segment_index` MUST increase monotonically within a Feed and SHOULD begin at `0` for the first Segment.
- `epoch` MUST identify the conversation epoch covered by the Segment. A Segment MUST NOT span multiple epochs.
- `prev_segment_ref` MUST reference the immediately preceding Segment when one exists and MUST be absent for the first Segment in a Feed.
- `next_segment_ref` MUST be absent while `state = active`. A sealed Segment SHOULD include `next_segment_ref` when a successor exists and MAY omit it only when no successor exists.
- `state` MUST be either `active` or `sealed`.
- `message_count` MUST report how many protocol frames/messages are stored in the Segment.
- `byte_count` MUST report the encoded size of the Segment payload stored for sync/history purposes.
- `checkpoint_summary` MUST provide an integrity/checkpoint summary sufficient to detect truncation, corruption, or unexpected content replacement for that Segment.

### 7.6 Feed Segmentation Rules

- Each writable Feed MUST have exactly one `active` Segment at a time.
- `sealed` Segments are immutable and MUST NOT accept additional frames after sealing.
- Segment rotation MUST occur before a Segment exceeds implementation-defined byte or message-count thresholds.
- Segment rotation MAY also occur at epoch, policy, membership, or other scope-defined boundaries when doing so preserves the semantics in this document.
- Conversation manifests MUST point to the current writable Feed tips / `active` Segments needed for forward sync.
- History traversal MUST follow Segment links backward via `prev_segment_ref` from the current tip toward older sealed Segments.

### 7.7 Blob Reference

Each Blob Reference MUST contain enough information to fetch and decrypt the blob without revealing plaintext to non-participants, including:

- blob identifier;
- transport/storage reference;
- encrypted content key or wrapped capability;
- content type;
- size;
- integrity digest;
- optional filename and preview metadata.

## 8. Conversation Scopes

### 8.1 Direct Message (`dm`)

A DM is a two-party scope.

- Each participant MUST write only to their own content feed.
- Each participant content feed MUST use the canonical segmented Feed model, with one active Segment per writer.
- A DM MUST use asynchronous setup followed by Double Ratchet-protected content exchange.
- Implementations MUST merge the two participant feeds locally.

### 8.2 Group (`group`)

A Group is a multi-party scope.

- Group membership and epoch changes MUST use MLS.
- A Group MUST have exactly one control feed and one content feed per writing member.
- The control feed and each member content feed MUST use the canonical segmented Feed model.
- Group implementations SHOULD rotate affected Segments at MLS epoch and membership boundaries so sealed history aligns with membership-visible history.
- Content payloads in a Group MUST be bound to an MLS epoch.

### 8.3 Publication (`publication`)

A Publication is a one-publisher, many-followers scope.

- A Publication MUST have one publisher content feed.
- A Publication SHOULD have a publisher control feed for policy, capability, and epoch state.
- Publisher content and control feeds MUST use the canonical segmented Feed model.
- Publication implementations SHOULD rotate affected Segments at publication epoch/key or policy boundaries.
- Followers MUST NOT be required to reveal themselves to other followers merely to read the publication.
- Publication crypto MUST be simpler than Group crypto when single-writer semantics make MLS unnecessary.

## 9. Outer Frame and Inner Envelope

### 9.1 Outer Frame

Each stored or transported unit is an Outer Frame with minimal plaintext metadata for sync and dedup:

```text
OuterFrame {
  version
  conversation_id
  feed_id
  segment_index
  seq
  ciphertext
}
```

The Outer Frame MUST NOT expose payload type or plaintext content. Implementations MAY include additional transport metadata only if it does not weaken the privacy requirements of this document.

Outer Frame requirements:

- `segment_index` MUST identify the containing Segment within the logical Feed.
- `seq` MUST remain monotonic within the logical Feed even when the Feed rotates to a new Segment.

### 9.2 Inner Envelope

The decrypted content is an Envelope:

```text
Envelope {
  version
  epoch
  author_member_id
  seq
  parents[]
  created_at
  kind
  body
}
```

Requirements:

- `version` MUST identify the envelope schema version.
- `epoch` MUST identify the current crypto era for Group and Publication scopes and MAY be `0` for an initial DM bootstrap phase.
- `seq` MUST be monotonic within a feed.
- `parents[]` SHOULD be used for causal reconstruction, replies, edits, and deletes.
- `kind` MUST identify the payload type.

## 10. Payload Catalog

### 10.1 Core v1 Payload Kinds

The following payload kinds are defined for v1:

- `text.message`
- `attachment.ref`
- `image.ref`
- `capability.advert`
- `invite.contact`
- `invite.group`
- `invite.publication`

### 10.2 Optional v1 Payload Kinds

The following payload kinds are defined but optional:

- `message.reaction`
- `message.edit`
- `message.delete`
- `receipt.read`
- `group.notice`
- `publication.notice`
- `group.join_request`
- `group.join_accept`
- `publication.follow_request`
- `publication.follow_accept`

### 10.3 Reserved Future Payload Kinds

The following names are reserved for future use and MUST NOT be repurposed:

- `call.offer`
- `call.accept`
- `call.reject`
- `call.update`
- `call.hangup`

### 10.4 Unknown Payload Handling

When a client encounters an unknown payload kind:

- it MUST preserve conversation integrity;
- it MUST NOT treat the frame as malformed solely because the kind is unsupported;
- it SHOULD render the item as unsupported or ignore it safely;
- it MUST NOT emit replacement payloads that alter transcript semantics.

## 11. Capability Advertisement

Every conversation MUST support capability advertisement via `capability.advert`.

An advertisement MUST identify:

- supported protocol version range;
- mandatory baseline features;
- optional supported features;
- named interoperability profiles required by those features when applicable.

Examples of feature tokens:

- `dm.v1`
- `group.mls.v1`
- `publication.v1`
- `blob.v1`
- `image.v1`
- `blob.transport.stigmerge.v1`
- `invite.persistent.v1`
- `message.edit.v1`
- `call.signal.v1`

A sender MUST NOT rely on an optional feature until the remote side has advertised support for it.

A sender MUST NOT transmit `attachment.ref` or `image.ref` to a peer unless a compatible blob transport profile has been advertised or otherwise fixed by the selected interoperability profile.

## 12. Direct Message Protocol

### 12.1 Contact Invite Semantics

Contact invites come in two forms:

- **one-time**: redeemable once;
- **persistent**: redeemable multiple times until expired or revoked.

Contact invites MUST be capability-based. They MUST NOT require the recipient to learn the inviter's IP address or network location.

### 12.2 DM Bootstrap Flow

The DM bootstrap flow is:

1. Bob issues a contact invite.
2. Alice obtains and redeems the invite.
3. Alice sends DM initiation material containing:
   - supported version range;
   - supported features;
   - async session setup material sufficient for Bob to start a secure DM.
4. Bob accepts and returns corresponding setup material.
5. Both parties derive a DM session.
6. Subsequent content messages use Double Ratchet.

### 12.3 DM Feed Layout

A DM MUST contain exactly two content feeds, one per participant. Each participant MUST write only to their own feed. Implementations MUST derive transcript order from per-feed sequence plus causal references rather than from a single shared chain.

Each DM content feed MUST be segmented. The two participant feeds MAY rotate independently on byte/count thresholds and MAY rotate on session-policy boundaries, but each writable DM feed MUST still expose exactly one active Segment.

### 12.4 DM Crypto

DM content messages MUST use Double Ratchet after bootstrap. The concrete crypto profile identifier MUST be advertised and matched by both peers before content exchange proceeds.

## 13. Group Protocol

### 13.1 Group Invite Semantics

Group invites come in two forms:

- **one-time**: intended for a single join;
- **persistent**: reusable entry point for multiple joins.

A Group invite MUST permit joining without first establishing a personal DM relationship with an existing member.

### 13.2 Group Layout

A Group MUST contain:

- one control feed;
- one content feed per writing member;
- an MLS-backed epoch state.

The control feed MUST carry membership changes, MLS commits, policy changes, and capability state.

Each Group feed MUST be segmented. The control feed SHOULD seal and rotate when MLS epoch or membership changes occur, and member content feeds SHOULD rotate at the same boundaries when implementations need sealed history to align with epoch visibility.

### 13.3 Group Join Flow

The Group join flow is:

1. Invitee obtains a Group invite.
2. Invitee redeems the invite or submits a join request, depending on invite policy.
3. Existing group state emits admission material.
4. MLS welcome or equivalent epoch material is delivered to the joining member.
5. The joining member learns the active manifest and visible feeds.
6. Content messages are exchanged under the current epoch.

### 13.4 Group Crypto

Groups MUST use MLS for membership and epoch management. A conformant implementation MUST support at least one shared MLS profile. Implementations MUST reject incompatible MLS profiles rather than silently downgrading.

### 13.5 Group History Rule

By default, a new member MUST NOT gain plaintext access to content from before their admitted epoch unless prior history is explicitly re-shared.

## 14. Publication Protocol

### 14.1 Publication Invite Semantics

Publication invites come in two forms:

- **one-time**: private publication access for a specific follower or small controlled audience;
- **persistent**: reusable public or semi-public publication access.

Publications MAY support follower approval workflows. If approval is required, that requirement MUST be explicit in the invite policy.

### 14.2 Publication Layout

A Publication MUST contain:

- one publisher content feed;
- optionally one publisher control feed;
- one publication manifest;
- publication epoch/key state.

Followers MUST NOT be required to publish writable feeds solely to consume publication content.

Publication feeds MUST be segmented. Followers SHOULD sync the publisher's active Segment first, then backfill older sealed Segments lazily or according to local retention policy.

### 14.3 Publication Flow

The Publication follow flow is:

1. Follower obtains a Publication invite.
2. Follower redeems the invite.
3. If approval is required, a follow request is submitted and either accepted or rejected.
4. Follower receives manifest and active epoch information.
5. Follower reads publisher content feed and associated blobs.

### 14.4 Publication Privacy

Implementations SHOULD preserve follower privacy by default. A publication MUST NOT reveal follower identity or follower list visibility more broadly than the publication model requires.

## 15. Blob and Image Model

### 15.1 Logical Blob Semantics

Attachments and images MUST use the Blob Reference model.

The payload kinds `attachment.ref` and `image.ref` define logical message semantics only. They MUST carry the blob capability/reference, integrity binding, and user-visible metadata needed to fetch and decrypt the content, but they MUST NOT by themselves imply any single concrete Veilid blob backend.

Blob send flow:

1. sender encrypts the blob with fresh content key material;
2. sender publishes the encrypted blob through a Veilid-backed blob transport profile;
3. sender transmits a Blob Reference or Image Reference payload;
4. receiver fetches and decrypts the blob via the service layer.

Image sharing MUST be representable as a specialized blob reference rather than as a protocol special case that bypasses the attachment model.

### 15.2 Blob Transport Profiles

A concrete attachment/image backend MUST be defined by a named Blob Transport Profile.

A Blob Transport Profile MUST specify at least:

- how the logical blob identifier and transport/storage reference map to Veilid-backed storage objects;
- how the encrypted capability or wrapped key material is encoded for interoperable retrieval;
- any profile-specific publish, fetch, retry, or segmenting rules required for interoperability;
- profile/version identifiers and compatibility requirements.

Interoperable implementations MUST agree on a concrete Blob Transport Profile before exchanging attachments or images. If no compatible Blob Transport Profile is shared, `attachment.ref` and `image.ref` support MUST be treated as unavailable even when both sides support logical blob references.

`blob.transport.stigmerge.v1` is an appropriate v1 Blob Transport Profile candidate. It defines one concrete Veilid-backed blob backend for Scrapwire interoperability, but it is not part of the Scrapwire message semantics themselves.

#### `blob.transport.stigmerge.v1`

`blob.transport.stigmerge.v1` defines a Veilid-native Blob Transport Profile that delegates publication and retrieval to a Stigmerge-compatible backend. It governs only how a logical Blob Reference maps to concrete blob publication and fetch; it does not alter the message semantics of `attachment.ref` or `image.ref`.

Implementations claiming support for `blob.transport.stigmerge.v1` MUST agree on the following:

- the profile identifier is exactly `blob.transport.stigmerge.v1`;
- the Blob Reference `transport/storage reference` field MUST carry the canonical Stigmerge share reference sufficient for a peer to resolve the blob via Veilid alone; for v1 interoperability, this reference MUST be the canonical textual encoding of the Stigmerge share key / Veilid record key returned by publication;
- the Blob Reference `blob identifier` and `integrity digest` fields MUST bind to the exact encrypted blob content represented by that Stigmerge share; a receiver MUST reject the fetch as an integrity failure if the resolved share metadata or fetched ciphertext does not match the advertised identifier/digest;
- publication and fetch for this profile MUST be performed through a Stigmerge-compatible Veilid backend, and implementations MUST NOT require or reference any direct-network, IP-based, or out-of-band rendezvous to make the blob available;
- a sender MUST NOT transmit `attachment.ref` or `image.ref` using this profile until publication has yielded a stable Stigmerge share reference that another conformant implementation can resolve and fetch from Veilid;
- a receiver presented with this profile and a valid Blob Reference MUST resolve the advertised Stigmerge share, fetch the encrypted blob content it exposes, verify the required integrity binding, and only then pass the verified ciphertext into the selected blob decryption layer;
- transient resolve/fetch failures SHOULD be treated as retriable transport failures rather than transcript corruption; retries MUST NOT change the advertised blob identifier, transport/storage reference, or integrity digest, and if republication changes any of those values the sender MUST emit a new Blob Reference;
- cancellation is local execution state only: cancelling an in-progress fetch MUST NOT invalidate an already-emitted Blob Reference, and cancelling an in-progress publish before a stable Stigmerge share reference exists MUST be treated as no publication having occurred.

## 16. Ordering, Deduplication, and Sync

### 16.1 Message Identity

Each logical event MUST have a deterministic identifier derived from at least:

- `conversation_id`
- `epoch`
- `feed_id`
- `segment_index`
- `seq`

### 16.2 Ordering

Implementations MUST reconstruct transcript order from:

- per-feed sequence;
- causal parents;
- scope-specific merge rules.

Implementations MUST NOT assume global real-time delivery order.

### 16.3 Deduplication

Retrying transmission of the same logical event SHOULD preserve event identity where feasible so deduplication is safe.

### 16.4 Replay and Resync

Implementations MUST be able to:

- replay feeds after restart;
- ignore duplicate frames;
- recover from delayed propagation;
- rebuild derived local state from manifests plus feed contents.

When syncing a Feed, clients SHOULD fetch the `active` Segment referenced by the manifest first and MAY backfill older `sealed` Segments lazily or according to policy. Clients reconstruct older history by walking `prev_segment_ref` links backward from the current tip.

## 17. Versioning and Compatibility

### 17.1 Baseline Rule

Every protocol object that crosses implementation boundaries MUST be versioned.

### 17.2 Additive Evolution

New fields SHOULD be added in an additive, optional form whenever possible. Existing field semantics MUST NOT be silently changed across versions.

### 17.3 Compatibility Rule

If a sender requires a feature not supported by the receiver, the sender MUST either:

- negotiate a mutually supported alternative; or
- fail the operation explicitly.

Silent downgrade that weakens privacy or security is forbidden.

## 18. Serialization and Crypto Profiles

This document defines logical protocol objects and interoperability rules. Wire-level serialization, concrete crypto profiles, and concrete Blob Transport Profiles MUST be fixed by named profile identifiers before cross-implementation interoperability testing begins.

An implementation MUST advertise and verify:

- envelope/profile version;
- DM crypto profile;
- Group MLS profile;
- blob encryption profile;
- blob transport profile when attachments/images are exchanged.

Implementations MUST refuse to interoperate when the relevant profiles do not match or do not have a defined compatibility rule.

## 19. Extensibility Rules

Future features MUST be introduced by:

- defining a new payload kind;
- advertising support via capability advertisement;
- preserving existing payload semantics.

Future call signaling MUST follow this model. Adding `call.*` payloads in a later version MUST NOT require changing the base DM, Group, Publication, or Blob models.

## 20. Implementation Roadmap

### Phase 0: Foundation

- freeze scope definitions;
- freeze manifest structure;
- freeze envelope fields;
- freeze capability advertisement format.

### Phase 1: Direct Messages

- one-time and persistent contact invites;
- DM bootstrap;
- two-feed DM model;
- segmented DM feed storage;
- Double Ratchet content exchange;
- mandatory text payloads.

### Phase 2: Blobs

- blob encryption and integrity rules;
- `attachment.ref` and `image.ref`;
- fetch, cache, and retry behavior.

### Phase 3: Groups

- one-time and persistent Group invites;
- MLS-backed membership;
- control feed plus per-member feeds;
- segmented feed rotation and traversal;
- group history rules.

### Phase 4: Publications

- one-time and persistent Publication invites;
- single-writer publication feed;
- segmented publication feed storage;
- follower approval and privacy rules.

### Phase 5: Hardening

- capability-negotiated optional features;
- replay/resync validation;
- metadata leakage review;
- interoperability testing against an independent implementation.

### Phase 6: Future Extensions

- reserve `call.*` signaling without altering existing payload semantics;
- extend optional payload catalog via capability advertisement.

## 21. Summary of Mandatory Interoperability Behavior

To claim conformance with Scrapwire Protocol v1, an implementation MUST:

- implement the three scopes: DM, Group, and Publication;
- support one-time and persistent invites for each relevant scope;
- use segmented Feed storage rather than a single forever-growing record;
- use Double Ratchet for DM content after bootstrap;
- use MLS for Group membership and epoch state;
- use single-writer publication semantics for Publication scope;
- use encrypted Blob References for attachments and images;
- support versioned envelopes and capability advertisements;
- handle unknown payloads safely;
- preserve privacy, security, and anonymity requirements defined in this document.

An implementation MAY add optional features, but MUST do so through capability advertisement and without breaking these mandatory behaviors.