STAR: settling the details
On the path to stabilizing STreaming ARchives
These are the aspects of STAR which I feel haven't been fully settled yet, and a summary of all other feedback to date. Thoughts and pushback welcome! I'll keep an eye out for follow-ups on any of the linked places below, and on responses to this post.
The current draft for the STAR spec in DASL is in PR: https://tangled.org/dasl.ing/dasl.ing/pulls/16/round/2 -- it's a rewrite since the last microcosm STAR version, any inconsistency should defer to the DASL PR.
Main unsettled questions
Q1: Format determinism
- varint: SHOULD vs MUST for readers -- stdlib varint decoders are typically not strict, so it might be nice to make readers checking be a SHOULD.
Update: my position on this is now: we keep the MUST. CAR's DASL spec technically requires it, it's not new strictness. Yeah, lots of implementations will not be compliant if they use stdlib/common varint libraries -- hopefully the interop test vectors nudge them toward being strict.
My compromise position: we do bijou64 instead of varint.
Q2: Metadata $type authority
- authority: should we recommend using domain authority (like NSID)
- with optional
#frag(bnewbold.net -- can you say more about this? do we have a syntax in atproto for bare nsid with a fragment suffix already?)
- with optional
- my current view is that there is no official authority of STAR
$types. It's useful for content negotiation. producers and consumers usually know who eachother are and why they are talking to eachother, so when i'm processing a$type: atproto-space-repo, it's not an random STAR from the universe. If someone makes an unrelated application profile and uses the same$type, it's only a problem if they start getting collected together. ...- is this naive? we already almost could have been at risk of a collision like that, since I was going to make an independent STAR format for atproto spaces before spaces got on board with adopting STAR. if the adoption had happened later, the "official" application profile might have needed to pick a worse
$typestring to avoid a collision with my independent one, or reuse it "in the same universe" and have more of a risk of actual $type confusion. - i hope that some basic guidance in creating application profiles (prefix it with a name you feel authoritative over, recommend nsid, whatever) can mostly avoid this?
- is this naive? we already almost could have been at risk of a collision like that, since I was going to make an independent STAR format for atproto spaces before spaces got on board with adopting STAR. if the adoption had happened later, the "official" application profile might have needed to pick a worse
Update! require $type to be a utf-8 string, make a SHOULD for constraining it to chars that can be used in a mime type parameter for content negotiation, mention nsid as a possible approach to avoid collisions.
Q3: Data control bytes
Update! There's now a whole post just for Q3:
Update update! Probably going with a key prefix shifted up by 1, which looks like this: https://microcosm.tngl.io/star/spec-edits/shifted-key-len.html
No control bytes, but we detect early truncation.
Q4: Format-level maximum sizes
- I dropped them in the DASL draft, since CAR's specs don't list any. The numbers I picked (64MiB meta and keys, 64 GiB values) felt very arbitrary. Application profiles should require smaller limits that suit their needs, and reader implementations should provide interfaces to enforce lower limits before attempting to read any content anyway... so what does a spec-level limit add? I think values should be allowed to be pretty big (readers can even offer chunked value-streaming interfaces), which makes it hard to pick a number that feels like a meaningful backstop for memory exhaustion. But, having no limits specified seems like playing loose with it. Maybe it's fine?
Update: yeah this came to consensus easy enough: no format-level limits, just (strongly) recommend application profiles include limits for meta, key sizes, and value sizes.
Initial dasl.ing issue comments
- Actually there was not really any pushback in this thread, mostly very nice words of support :)))
STAR in atproto spaces PR comments
- ngerakines.me:
$typeshould be be an NSID like all other lexicons so it can be resolved to a lexicon schema to be described, documented, etc. (See Q2) - dholms.at: option example: DASL group maintains a list of STAR profiles (See Q2)
- bnewbold.net: we might recommend folks defining new STAR profiles use NSID in
$type. It might not be ideal to blur or create ambiguity over the namespace of a "protocol-level string" (See Q2) - matthieu.bsky.team: NSID makes makes publication of a new version of the schema easier (though
com.atprotolexicons are atproto spec). alternatively could use a different string than$typeto avoid confusion. (See Q2)
Review/Feedback (2026-10-16) from bnewbold.net
- Rename this just STAR, from STAR-lite
- Security section TODOs
$typepreference (see Q2)- Termination indicator (see Q3)
- Per-block integrity checks, more integrity checks generally
- Bijou64 discouraged
- Encountering "EOF" (earlier proposal's poor wording choice)
Initial dasl.ing PR comments
- mosh.bsky.social: "parse errors should be considered fatal" -> "must"
- mosh.bsky.social: dasl spec should have some links out while in HNS status
- vmx.cx: clarification of canonical varint prefix wording
- vmx.cx: ordered pairs (sorted vs ordered). suggestion to call it "sequenced" instead of "ordered", other suggestions. Do we need this to be spelled out as a requirement? (or, is it even required?)
Implementation feedback (2026-10-24) from bnewbold.net
- Note for clarity that the overall meta object can not be empty (
$typerequired) $type: note empty string allowed (or require it be non-empty)- must canonical varint be a MUST for readers or can it be SHOULD? stdlib impls tend to accept non-canonical/encodings (see Q1)
- Semantics of a pair with empty key and empty value are kind of weird, should it be disallowed?
Intended changes
TODO: meta strictness
be explicit that readers should reject non-canonical meta, or else note that whether an archive is deterministic depends on that.
TODO: other $type rules
explicitly note that metadata must be present, which is implied since $type is always required. explicitly note that empty-string $type is allowed.
Appendix: earlier Q3 content:
I removed this from the "unsettled questions" section since it's now covered in the new post, but for the record, here's what it originally said:
- Right now there is no explicit end-of-archive signal -- an archive truncated at the boundary of any data pair is a valid archive, at least at the format level. So, unless applications add their own check that covers truncation, they can silently miss data when reading truncated archives. Currently, that will just be an Application Profile guidance: put a "number of records" or "MST root hash" or lthash over the set or checksum of keys or whatever, in the
meta.- I don't think there's a really nice way to do an end-of-archive signal with the current concatenated (len | key | len | val) format. anything in-band would be unfortunate... i suppose we could reserve <empty key | empty val> or something but i don't like it. The only other thing I can see is using an invalid
len... which means a non-canonical len. This seems like a very bad idea, since it ties an internal varint detail to the format, and might lead to unsafe failures with decoders that permit non-minimalvarints on read. I don't like it.
- I don't think there's a really nice way to do an end-of-archive signal with the current concatenated (len | key | len | val) format. anything in-band would be unfortunate... i suppose we could reserve <empty key | empty val> or something but i don't like it. The only other thing I can see is using an invalid
- A control byte between each data pair would provide an opportunity for signalling like end-of-archive. Like,
0xAAmeans "another keypair" (endless scream);0xEEmeans "end of archive" (eeee stop!), etc.- I don't think we'd want any other control bytes, but if we ever did, there'd be a place for that.
- It's not much overhead except in applications with very small keys and values, and/but would be near-free in size when archives are compressed.
- My biggest gripe is aesthetic. It feels like it adds noisiness to the currently-very-simple description. I can get over it if we need it. Do we need it? CAR gets by without, but we're aiming for stricter-than-CAR.