j3d1/toolshed
2
1
Fork 0
Code Issues 9 Pull requests 1 Projects Releases Packages Wiki Activity

stash

Browse source
This commit is contained in:
j3d1 2026-08-27 01:39:03 +02:00
parent d56784eb8d
commit 6ee5ae38b1
8 changed files with 246 additions and 153 deletions
Download patch file Download diff file Expand all files Collapse all files

60
docs/glossary.md
View file

@ -47,7 +47,8 @@ and talks to it directly, regardless of where the frontend itself was loaded fro
A name that's unique within its own scope and carries, as part of itself, enough information to say
where it's authoritative, without needing a central registry to look it up. The general term
covering [user handles](#user-handle), [group handles](#group-handle) (proposed),
[classification handles](#classification-handle), and [item handles](#item-handle) (proposed).
[classification handles](#classification-handle), [User-Qualified IDs](#user-qualified-id), and
[Domain-Qualified Short IDs](#domain-qualified-short-id).
*See: [federation.md](federation.md#unique-handles)*
**Strict resolution** (Implemented)
@ -109,7 +110,8 @@ The asymmetric keypair backing exactly one [user handle](#user-handle); together
keypair backing it are what's called an [identity](#identity). The private key signs requests made
as that user and never leaves their control; the public key is handed out to establish trust in the
handle (at registration, or via a [friend](#friend-friendship) exchange) and is used to verify
signatures. Only user handles carry a keypair, not groups, classification handles, or item handles.
signatures. Only user handles carry a keypair, not groups, classification handles, or
User-Qualified IDs.
*See: [federation.md](federation.md#cryptography)*
**Membership list** (Proposed)
@ -157,8 +159,8 @@ a property alias specifically, also states a unit conversion (even if it's "iden
**Classification handle** (Implemented)
Umbrella term for a [tag](#tag-property-category), [property](#tag-property-category), or
[category](#tag-property-category) handle, of the form `origin#type:name`, e.g.
`git:base#property:length`. Distinct from a [user handle](#user-handle) or [item
handle](#item-handle): it names a reusable classification concept, not an actor or an owned thing.
`git:base#property:length`. Distinct from a [user handle](#user-handle) or [User-Qualified
ID](#user-qualified-id): it names a reusable classification concept, not an actor or an owned thing.
*See: [federation.md](federation.md#tags-properties-and-categories)*
**Definition fingerprint** (Partially implemented)
@ -201,29 +203,47 @@ strings today.
## Items & Physical Labels
**Item Handle** (Proposed)
The compact identifier for a specific item that's meaningful outside its owner's own account:
`user@domain.tld:id`, the owner's [user handle](#user-handle) plus a [local id](#local-id). Used
where it's already clear from context that it's a Toolshed item, e.g. inside the app, in exports,
in logs, so it doesn't need to spell that out or be openable on its own. Contrast with [Item
URL](#item-url), the self-contained form for when no such context can be assumed.
*See: [items-labels.md](design-in-progress/items-labels.md#open-design-questions)*
**Domain-Qualified Short ID** (Implemented)
A [domain](#domain) paired with a [short id](handles-and-shortids.md#short-ids) token, e.g.
`toolsheddomain.tld:~DyU`. A bare short id token is opaque and scoped to whichever backend minted
it, nothing in the token itself says which backend's id numbering to read it against, so it only
means something inside the app instance that's currently talking to that backend. Prefixing it with
its domain supplies the missing piece, the same way any other handle's domain half does, so the
pair keeps resolving correctly even after being copied out of the instance it was minted on. Unlike
a [User-Qualified ID](#user-qualified-id) or [Item URL](#item-url), it isn't limited to owned kinds
(any kind in the short id registry can be domain-qualified) and isn't meant to be openable with zero
context, no scheme, no path, opaque bit-packed payload, it belongs inside the app or between things
that already speak its short-id format, not on a physical label.
*See: [handles-and-shortids.md](handles-and-shortids.md#domain-qualified-short-id)*
**Item Label** (Proposed)
**Item Label** (Implemented)
A physical, scannable encoding (QR code, barcode, or similar) of an item's [Item URL](#item-url),
meant to be printed and stuck on the physical object it refers to.
*See: [items-labels.md](design-in-progress/items-labels.md#goals)*
**Item URL** (Proposed)
The self-contained URL form of an [Item Handle](#item-handle), for use with no context at all, e.g.
an [Item Label](#item-label): `https://<any frontend>/i/user@domain.tld/id`. Has to open directly
to the right frontend and land on the right item on its own, since the reader can't be assumed to
already know what it is or which server it belongs to. Any frontend can serve this URL, the host
named in it isn't part of the item's identity, only the handle in its path is.
**Item URL** (Implemented)
The self-contained URL form of an item's [User-Qualified ID](#user-qualified-id), for use with no
context at all, e.g. an [Item Label](#item-label): `https://<any frontend>/i/user@domain.tld/id`.
Has to open directly to the right frontend and land on the right item on its own, since the reader
can't be assumed to already know what it is or which server it belongs to. Any frontend can serve
this URL, the host named in it isn't part of the item's identity, only the handle in its path is.
*See: [items-labels.md](design-in-progress/items-labels.md#open-design-questions)*
**Local id** (Implemented)
An item's identifier as it exists today: unique only within its owner's own inventory, not
meaningful outside that owner's account. The starting point both the [Item
Handle](#item-handle) and [Item URL](#item-url) build on.
meaningful outside that owner's account. The starting point both a [User-Qualified
ID](#user-qualified-id) and [Item URL](#item-url) build on.
*See: [federation.md](federation.md#items), [items-labels.md](design-in-progress/items-labels.md#problem)*
**User-Qualified ID** (Implemented)
The compact identifier for a specific owned thing (an item, a storage location, ...) that's
meaningful outside its owner's own account: `<owner-handle>:<kind><local-id>`, e.g.
`alice@example.com:i42`, the owner's [user handle](#user-handle) (or [group
handle](#group-handle) for a group-owned thing) plus a one-letter kind tag (`i` for item, `s` for
storage location) and a [local id](#local-id). Used where it's already clear from context that it's
Toolshed data, e.g. inside the app, in exports, in logs, so it doesn't need to spell that out or be
openable on its own. Generalizes what used to be a bespoke, item-only "Item Handle" concept: the
kind letter is what an item-specific `user@domain.tld:id` shape was missing, nothing in that string
said it was specifically an item. Contrast with [Item URL](#item-url), the self-contained form for
when no such context can be assumed.
*See: [handles-and-shortids.md](handles-and-shortids.md#user-qualified-id)*

154
docs/handles-and-shortids.md
View file

@ -4,8 +4,8 @@ This is the syntax-level reference for two related but separate naming schemes.
"Unique Handles" section covers *why* Toolshed hands out handles at all and what each kind (user,
group, tag/property/category) means conceptually; this document covers the parsing rules those
handles have to follow once they're written down or embedded somewhere - legal characters and
escaping. It also covers short ids end to end: a separate, newer scheme for packing small integer
id chains into a compact token, implemented in `frontend/src/short-id.js`.
escaping. It also covers short ids end to end: a separate scheme for packing small integer id
chains into a compact token.
## Handle syntax
@ -15,14 +15,14 @@ A username ends up embedded, unescaped, in several composite formats beyond its
can't contain any character that already means something else in one of those: `@` (the
user/domain separator in a user handle), `#` (the group-handle prefix, and the origin/type
separator in a classification handle, see federation.md's Tags, Properties, and Categories
section), `:` (the id delimiter in a proposed Item Handle, the type/name delimiter in a
classification handle, and the delimiter in a signed request's `Authorization` header), `+`
(reserved as the URL-embedding escape for `#`, see below), `~` (the short-id token prefix, see
Short IDs below, and a proposed collision-disambiguation suffix delimiter on a tag's origin), and
`/` (the path-segment delimiter every handle and id ultimately sits next to once embedded in a
URL). This has to be enforced by an explicit validator rather than left to Django's default
`UnicodeUsernameValidator` (`^[\w.@+-]+\Z`), which currently permits both `@` and `+` (its own
regex doesn't happen to allow `#`, `:`, or `/`, but that's incidental, not a designed restriction).
section), `:` (the kind/id delimiter in a User-Qualified ID (see below), the type/name delimiter in
a classification handle, the domain/token delimiter in a Domain-Qualified Short ID (see below), and
the delimiter in a signed request's `Authorization` header), `+` (reserved as the URL-embedding
escape for `#`, see below), `~` (the short-id token prefix, see Short IDs below, and a
collision-disambiguation suffix delimiter on a tag's origin), and `/` (the path-segment delimiter
every handle and id ultimately sits next to once embedded in a URL). This has to be enforced by an
explicit validator rather than left to a framework default, which doesn't draw the line in the same
place.
### Embedding a `#`-bearing handle in a URL
@ -44,17 +44,45 @@ otherwise forbidden in every field a handle is built from (see Reserved characte
group name or tag name could itself contain a literal `+`, it would be indistinguishable from an
escaped `#` once decoded.
Implemented in `frontend/src/handle-url.js` (`encodeHandleForUrl`/`decodeHandleFromUrl`).
### User-Qualified ID
Owned things (an item, a storage location, ...) can be referred to outside their owner's own
account: anything whose id is only unique within one owner's own numbering needs an owner-qualified
form to resolve globally.
A **User-Qualified ID** is an owner handle (a user handle, or a group handle for a group-owned
thing) with a kind letter and a local id appended, separated by a single `:`:
`<owner-handle>:<kind><local-id>`, e.g. `alice@example.com:i42`. This is the same `origin#type:name`
pattern classification handles already use, built on `:` instead of `#` so it avoids the URL
fragment-escaping problem (see Embedding a `#`-bearing handle in a URL above) — a User-Qualified ID
isn't meant to sit directly in a URL path segment.
Its payload is a plain decimal local id with nothing self-describing baked in, unlike a Short ID's
bit-packed payload, so the owner is spelled out as a full handle rather than just a domain:
`example.com:i42` would be ambiguous between every user on that domain with local item id `42`;
`alice@example.com:i42` isn't. An owner handle already carries its domain, so a User-Qualified ID is
cross-domain-resolvable as written, with no separate domain-qualified wrapper needed.
The kind letter is a small, closed registry. A group handle already looks visibly different from a
user handle (`#` prefix), so unlike the short id kind registry below, this one doesn't need separate
letters for a user-owned vs. group-owned kind — the owner half already says which it is.
| letter | kind | notes |
|---|---|---|
| `i` | item | covers both a user-owned item and a group-owned item |
| `s` | storage location | covers both a user-owned and a group-owned storage location |
`workflow` has no letter assigned; the registry can grow without breaking anything already printed.
Kinds with no owner (`category`, `group`, `file`) don't fit this scheme: `category` and `group` each
already have their own dedicated handle form. `file` has neither an owner to qualify by nor a
dedicated handle of its own; a Domain-Qualified Short ID (below) is the fallback for it.
## Short IDs
A general encoding for turning a small, fixed-shape list of integers into a compact, URL-safe
token, with no server-side lookup table involved: the code *is* the data, nothing is stored
server-side to make it resolvable. Originally proposed to answer items-labels.md's open "what does
the handle/URL actually look like" question, but the encoding itself isn't item-specific; anything
currently addressed by a short chain of small integers is a candidate. Implemented and tested in
`frontend/src/short-id.js`; try it live at `/~<token>` (`frontend/src/views/ShortId.vue`), which
decodes whatever token is in the URL and also lists worked examples for every registered kind.
server-side to make it resolvable. Not item-specific; anything currently addressed by a short chain
of small integers is a candidate.
### Shape: a kind tag, then a fixed list of integers
@ -67,29 +95,21 @@ range" - so kind 3 is encoded as escape + chunked-int `0`, kind 4 as escape + `1
costs nothing for a kind that already fits in the direct range, and keeps the tag itself extensible
forever without ever having to widen it out from under codes that were already printed. A narrow
tag only pays off if kind usage is actually skewed the way id values are (a few kinds dominate),
which is why the registry below is ordered by expected frequency, cheapest (most-used) kind first:
which is why the registry below is ordered by expected frequency, cheapest (most-used) kind first.
Kind ids are scoped **per arity**, not globally: the same id can (and does) name a different kind
depending on how many fields follow it, since the arity is always known before the kind id needs
disambiguating:
| kind | name | fields | notes |
|---|---|---|---|
| 0 | `item` | `owner_identity_id`, `item_local_id` | dominant case - the primary physical-label use case |
| 1 | `storage_location` | `owner_identity_id`, `storage_location_id` | also label-printed |
| 2 | `category` | `category_id` | label-adjacent (tagging); global, no owner |
| 3 | `workflow` | `owner_identity_id`, `workflow_id` | shared in-app, not printed - first to pay the escape's cost |
| 4 | `group` | `group_id` | shared even less often; global, no owner |
| 5 | `file` | `file_id` | least often shared standalone; global, deduplicated by content hash |
`owner_identity_id` is `KnownIdentity.pk` (`backend/authentication/models.py`), not
`ToolshedUser.pk`. Every local account already has exactly one stable `KnownIdentity` row
(`ToolshedUser.public_identity`, created once at registration and never recreated), and every
friend this backend knows about - local or remote - is represented by that same table, unique on
`(username, domain)`. So one small integer already stands in for "this owner, as known by this
backend" for both cases, with no separate local-vs-remote branching needed, and it's the same row
federation.md's Cryptography section already treats as the trust anchor for a handle's public key.
It appears on `item`, `storage_location`, and `workflow` because their backing models
(`InventoryItem`, `StorageLocation`, `WorkflowInstance`) all FK `ToolshedUser` directly; `category`,
`group`, and `file` skip it because their models are global/unscoped (`Group` has an unowned
`members` M2M, `File` is deduplicated globally by content hash), so a bare row id is already
everything needed to look them up.
| kind id | arity | name | fields | notes |
|---|---|---|---|---|
| 0 | 2 | `item` | `owner_identity_id`, `item_local_id` | dominant case - a personally-owned item, the primary physical-label use case |
| 0 | 1 | `category` | `category_id` | label-adjacent (tagging); global, no owner - shares id 0 with `item` since the two never need the same arity |
| 1 | 2 | `group_item` | `owner_group_id`, `item_local_id` | same use case as `item`, but for a group-owned item |
| 1 | 1 | `group` | `group_id` | shared even less often; global, no owner |
| 2 | 2 | `storage_location` | `owner_identity_id`, `storage_location_id` | also label-printed |
| 2 | 1 | `file` | `file_id` | least often shared standalone; global, deduplicated by content hash |
| 3 | 2 | `workflow` | `owner_identity_id`, `workflow_id` | shared in-app, not printed - needs the escape range, since every other slot in the direct range (0-2) is already double-booked across the two arities in use |
| 4 | 2 | `group_storage_location` | `owner_group_id`, `storage_location_id` | same use case as `storage_location`, but for a group-owned location (see `group_item` above for the same owner/owner_group split); the direct range is fully double-booked, so this is the second kind that needs the escape range |
A short id is inherently scoped to the backend that minted it (an "owner" field is a row that only
exists in, and only means anything to, that one backend's database), not a portable replacement for
@ -119,13 +139,12 @@ Standard base64 assumes byte-aligned (8-bit) input, grouping 3 bytes into 4 outp
padding to a byte boundary before encoding. Since there's no byte layer here to begin with, the
bit-packed stream is instead packed directly into 6-bit groups and mapped straight onto the
URL-safe base64 alphabet (RFC 4648 §5: `-` and `_` in place of `+` and `/`), with the final
character's unused low bits padded with zeros. That padding is safe by construction: the decoder
always knows exactly how many integers a given kind calls for, and a chunk's continuation bit is
`1 = more follows`, so a run of zero-padding at the very end can never be misread as "one more
chunk" - it decodes as a terminated chunk, at which point every field the schema called for has
already been produced and decoding simply stops. No `=` padding characters are needed either; those
exist in classic base64 purely to communicate trailing-byte padding, and there is no byte layer
here to need that.
character's unused low bits padded with zeros. That padding is safe by construction: it's always
0-5 zero bits (just enough to reach the next multiple of 6), and the decoder stops reading fields
the moment a chunk decodes to 0 with nothing left after it - by the last-field-nonzero rule above,
that can only be the padding, never a genuine field, so it's discarded rather than counted. No `=`
padding characters are needed either; those exist in classic base64 purely to communicate
trailing-byte padding, and there is no byte layer here to need that.
### The leading `~`
@ -140,6 +159,35 @@ short id can be handed to any part of the stack without first checking which enc
there.
### Domain-Qualified Short ID
A short id token is deliberately opaque and scoped to whichever backend minted it: handed a bare
`~DyU`, nothing in the token itself says which backend's numbering it should be read against.
That's fine as long as the token stays inside a context that already knows the answer (the app the
user is currently looking at), but not once a token needs to travel outside that context, e.g.
pasted into a message to a friend on a different domain, or logged somewhere not tied to one
backend.
The fix is the same one every other cross-domain handle in this project already uses: pair the
opaque part with the domain that's authoritative for it. A **Domain-Qualified Short ID** is a short
id token prefixed with a domain and a literal `:`, e.g. `toolsheddomain.tld:~DyU`. The domain half
is exactly a handle's domain half (see federation.md's Unique Handles section): not a location,
just a statement of which backend to resolve the token against. Decoding still means handing the
`~token` half to that backend's decoder, same as ever; it's just no longer ambiguous which
backend's decoder to hand it to.
This is deliberately not the same thing as a User-Qualified ID (`owner-handle:i42`) or an Item URL.
It isn't a URL and isn't meant to be openable by something that doesn't already know what a
Toolshed short id is: there's no scheme, no path, and the payload after `:~` is bit-packed base64,
opaque to a human. It belongs to the same "context already makes clear it's Toolshed data" class as
a compact User-Qualified ID, meant for use inside the app (or between things that already speak the
short-id format), not for a physical label or a link shared outside it. What it adds over a bare
`~token` is that it no longer depends on "whichever backend I currently happen to be talking to" —
the domain travels with it, so it keeps resolving to the same entity once copied elsewhere. Unlike
a User-Qualified ID, it isn't limited to owned kinds: every kind in the short id registry can be
domain-qualified the same way, since the domain only ever names which backend's token namespace
applies, not which kind the token decodes to or whether that kind has an owner.
### Worked examples
Encoding `kind = item` (0), `owner_identity_id = 7`, `item_local_id = 42`:
@ -152,17 +200,19 @@ Total: 17 meaningful bits, padded to the next multiple of 6 (18) with one zero b
base64 characters: **`~DyU`**.
The same worked-out form for one example of every registered kind - `Bits` is the same
space-separated segmentation (kind tag, escape offset if present, each field, then padding) the
Examples table on `/~<token>` (`frontend/src/views/ShortId.vue`) shows for every registered kind:
space-separated segmentation: kind tag, escape offset if present, each field, then padding. Kinds
are grouped by shared id below to make the per-arity reuse visible:
| Kind | Fields | Serialized | Bits | Token |
|---|---|---|---|---|
| `item` | `owner_identity_id: 7`, `item_local_id: 42` | `[0, 7, 42]` | `00 00111 1001001010 0` | `~DyU` |
| `storage_location` | `owner_identity_id: 3`, `storage_location_id: 1000` | `[1, 3, 1000]` | `01 00011 100111111001000 00` | `~Rz8g` |
| `category` | `category_id: 5` | `[2, 5]` | `10 00101 00000` | `~ig` |
| `workflow` | `owner_identity_id: 2`, `workflow_id: 9` | `[3, 2, 9]` | `11 00000 00010 01001 0` | `~wCS` |
| `group` | `group_id: 11` | `[4, 11]` | `11 00001 01011` | `~wr` |
| `file` | `file_id: 123` | `[5, 123]` | `11 00010 1011101011 0` | `~xXW` |
| `category` | `category_id: 5` | `[0, 5]` | `00 00101 00000` | `~Cg` |
| `group_item` | `owner_group_id: 5`, `item_local_id: 42` | `[1, 5, 42]` | `01 00101 1001001010 0` | `~SyU` |
| `group` | `group_id: 11` | `[1, 11]` | `01 01011 00000` | `~Vg` |
| `storage_location` | `owner_identity_id: 3`, `storage_location_id: 1000` | `[2, 3, 1000]` | `10 00011 100111111001000 00` | `~hz8g` |
| `file` | `file_id: 123` | `[2, 123]` | `10 1011101011` | `~rr` |
| `workflow` | `owner_identity_id: 2`, `workflow_id: 9` | `[3, 2, 9]` | `11 00000 00010 01001 0` | `~wiS` |
| `group_storage_location` | `owner_group_id: 5`, `storage_location_id: 1000` | `[4, 5, 1000]` | `11 00001 00101 100111111001000 000` | `~wln5A` |
`workflow`, `group`, and `file` are kinds 3-5, so their `Bits` column shows the escape tag (`11`)
followed by its own offset segment payload fields.