168 lines
No EOL
12 KiB
Markdown
168 lines
No EOL
12 KiB
Markdown
# Handles and Short IDs
|
||
|
||
This is the syntax-level reference for two related but separate naming schemes. [federation.md](federation.md)'s
|
||
"Unique Handles" section covers *why* Toolshed hands out handles at all and what each kind (user,
|
||
group, tag/property/category) means conceptually; this document covers the parsing rules those
|
||
handles have to follow once they're written down or embedded somewhere - legal characters and
|
||
escaping. It also covers short ids end to end: a separate, newer scheme for packing small integer
|
||
id chains into a compact token, implemented in `frontend/src/short-id.js`.
|
||
|
||
## Handle syntax
|
||
|
||
### Reserved characters
|
||
|
||
A username ends up embedded, unescaped, in several composite formats beyond its own handle, so it
|
||
can't contain any character that already means something else in one of those: `@` (the
|
||
user/domain separator in a user handle), `#` (the group-handle prefix, and the origin/type
|
||
separator in a classification handle, see federation.md's Tags, Properties, and Categories
|
||
section), `:` (the id delimiter in a proposed Item Handle, the type/name delimiter in a
|
||
classification handle, and the delimiter in a signed request's `Authorization` header), `+`
|
||
(reserved as the URL-embedding escape for `#`, see below), `~` (the short-id token prefix, see
|
||
Short IDs below, and a proposed collision-disambiguation suffix delimiter on a tag's origin), and
|
||
`/` (the path-segment delimiter every handle and id ultimately sits next to once embedded in a
|
||
URL). This has to be enforced by an explicit validator rather than left to Django's default
|
||
`UnicodeUsernameValidator` (`^[\w.@+-]+\Z`), which currently permits both `@` and `+` (its own
|
||
regex doesn't happen to allow `#`, `:`, or `/`, but that's incidental, not a designed restriction).
|
||
|
||
### Embedding a `#`-bearing handle in a URL
|
||
|
||
A literal `#` can't appear unescaped in a URL path segment: per RFC 3986, `#` starts the URI's
|
||
fragment component, so any URL-parsing client (a browser, a QR scanner, a link preview) treats
|
||
everything from the first unescaped `#` onward as a fragment and never sends it to the server at
|
||
all, before a request is even made, not merely a server-side quirk to work around. The usual fix is
|
||
to percent-encode it (`%23`), but that's exactly the encode/decode step a self-contained item URL
|
||
(a physical label, a link shared outside the app) is designed to avoid for anything that sits
|
||
directly in a path segment (`@` needs no such treatment). Instead, whenever a handle containing a
|
||
`#` (a group handle, or a tag/property/category
|
||
handle) has to appear as a raw URL path segment, substitute `+` for `#` in that rendering only:
|
||
`#groupname@domain` becomes `+groupname@domain` in a URL, and `origin#type:name` becomes
|
||
`origin+type:name`. This is a URL-embedding convention, not a second handle format: the canonical
|
||
handle, the one used in the API, in signed requests, in the database, and everywhere else a handle
|
||
is written or displayed, is unchanged and is still written `#groupname@domain`. Reversing the
|
||
substitution when parsing a path segment back into a handle is unambiguous only because `+` is
|
||
otherwise forbidden in every field a handle is built from (see Reserved characters above); if a
|
||
group name or tag name could itself contain a literal `+`, it would be indistinguishable from an
|
||
escaped `#` once decoded.
|
||
|
||
Implemented in `frontend/src/handle-url.js` (`encodeHandleForUrl`/`decodeHandleFromUrl`).
|
||
|
||
## Short IDs
|
||
|
||
A general encoding for turning a small, fixed-shape list of integers into a compact, URL-safe
|
||
token, with no server-side lookup table involved: the code *is* the data, nothing is stored
|
||
server-side to make it resolvable. Originally proposed to answer items-labels.md's open "what does
|
||
the handle/URL actually look like" question, but the encoding itself isn't item-specific; anything
|
||
currently addressed by a short chain of small integers is a candidate. Implemented and tested in
|
||
`frontend/src/short-id.js`; try it live at `/~<token>` (`frontend/src/views/ShortId.vue`), which
|
||
decodes whatever token is in the URL and also lists worked examples for every registered kind.
|
||
|
||
### Shape: a kind tag, then a fixed list of integers
|
||
|
||
Every short id starts with a small, fixed-width **kind** tag saying which schema the rest of the
|
||
bits should be read against, followed by exactly the integer fields that kind's schema calls for,
|
||
in a fixed order. `kind` is a small, closed, slow-growing set, so it doesn't need to be
|
||
self-delimiting the way the integer fields do: 2 bits directly name kinds 0-2, and the all-ones
|
||
value (3) is an escape meaning "the real kind follows as the next field, offset by this direct
|
||
range" - so kind 3 is encoded as escape + chunked-int `0`, kind 4 as escape + `1`, and so on. This
|
||
costs nothing for a kind that already fits in the direct range, and keeps the tag itself extensible
|
||
forever without ever having to widen it out from under codes that were already printed. A narrow
|
||
tag only pays off if kind usage is actually skewed the way id values are (a few kinds dominate),
|
||
which is why the registry below is ordered by expected frequency, cheapest (most-used) kind first:
|
||
|
||
| kind | name | fields | notes |
|
||
|---|---|---|---|
|
||
| 0 | `item` | `owner_identity_id`, `item_local_id` | dominant case - the primary physical-label use case |
|
||
| 1 | `storage_location` | `owner_identity_id`, `storage_location_id` | also label-printed |
|
||
| 2 | `category` | `category_id` | label-adjacent (tagging); global, no owner |
|
||
| 3 | `workflow` | `owner_identity_id`, `workflow_id` | shared in-app, not printed - first to pay the escape's cost |
|
||
| 4 | `group` | `group_id` | shared even less often; global, no owner |
|
||
| 5 | `file` | `file_id` | least often shared standalone; global, deduplicated by content hash |
|
||
|
||
`owner_identity_id` is `KnownIdentity.pk` (`backend/authentication/models.py`), not
|
||
`ToolshedUser.pk`. Every local account already has exactly one stable `KnownIdentity` row
|
||
(`ToolshedUser.public_identity`, created once at registration and never recreated), and every
|
||
friend this backend knows about - local or remote - is represented by that same table, unique on
|
||
`(username, domain)`. So one small integer already stands in for "this owner, as known by this
|
||
backend" for both cases, with no separate local-vs-remote branching needed, and it's the same row
|
||
federation.md's Cryptography section already treats as the trust anchor for a handle's public key.
|
||
It appears on `item`, `storage_location`, and `workflow` because their backing models
|
||
(`InventoryItem`, `StorageLocation`, `WorkflowInstance`) all FK `ToolshedUser` directly; `category`,
|
||
`group`, and `file` skip it because their models are global/unscoped (`Group` has an unowned
|
||
`members` M2M, `File` is deduplicated globally by content hash), so a bare row id is already
|
||
everything needed to look them up.
|
||
|
||
A short id is inherently scoped to the backend that minted it (an "owner" field is a row that only
|
||
exists in, and only means anything to, that one backend's database), not a portable replacement for
|
||
a `user@domain.tld` handle, which stays the form to use anywhere cross-domain resolution actually
|
||
matters. Resolving a short id still goes through the same friend/signature checks as everything
|
||
else, unchanged; nothing about how the code looks grants any authority of its own (see Guessability
|
||
below).
|
||
|
||
### Packing one integer: dynamic bit depth
|
||
|
||
Each integer field is made self-delimiting with **continuation chunking** (UTF-8/LEB128-style):
|
||
split the value into fixed-size chunks (4 data bits each, most-significant chunk first), each
|
||
preceded by one continuation bit meaning "another chunk follows" (`1`) or "this is the last chunk"
|
||
(`0`). A value like `42` (`0b101010`) needs two 4-bit chunks, costing 10 bits total (2 × (1
|
||
continuation + 4 data)); `7` fits in one chunk, costing 5 bits. This was chosen over an
|
||
Elias-gamma-style unary/delimiter scheme (encode the value's bit-length in unary, then that many
|
||
literal bits): unary is cheaper for single-digit values but its prefix grows every time the value's
|
||
bit-length grows, so it never wins once ids pass single digits, which is the common case here (auto
|
||
increment database ids realistically sitting in the tens through low-hundred-thousands over an
|
||
installation's life). A 4-bit chunk width is a reasonable fixed default across that whole range;
|
||
per-field tuning was checked against both a uniform and a skewed (geometric) distribution and never
|
||
won by more than a fraction of a character, not enough to justify a tuning knob.
|
||
|
||
### From bits to text: base64 without the byte layover
|
||
|
||
Standard base64 assumes byte-aligned (8-bit) input, grouping 3 bytes into 4 output characters and
|
||
padding to a byte boundary before encoding. Since there's no byte layer here to begin with, the
|
||
bit-packed stream is instead packed directly into 6-bit groups and mapped straight onto the
|
||
URL-safe base64 alphabet (RFC 4648 §5: `-` and `_` in place of `+` and `/`), with the final
|
||
character's unused low bits padded with zeros. That padding is safe by construction: the decoder
|
||
always knows exactly how many integers a given kind calls for, and a chunk's continuation bit is
|
||
`1 = more follows`, so a run of zero-padding at the very end can never be misread as "one more
|
||
chunk" - it decodes as a terminated chunk, at which point every field the schema called for has
|
||
already been produced and decoding simply stops. No `=` padding characters are needed either; those
|
||
exist in classic base64 purely to communicate trailing-byte padding, and there is no byte layer
|
||
here to need that.
|
||
|
||
### The leading `~`
|
||
|
||
Every token is prefixed with a literal `~`, so a short id in a URL looks like `~DyU`. Its only job
|
||
is to mark "everything after me decodes as one of these": URL-safe base64 never produces a `~`
|
||
itself, so the prefix can never be confused with the payload, and none of Toolshed's other
|
||
path-segment formats (bare usernames, `user@domain` handles, slugs, plain numeric ids) start with
|
||
`~` either. `~` is one of RFC 3986's `unreserved` characters (§2.3, the same class as letters,
|
||
digits, `-`, `.`, and `_`), a stronger guarantee than merely being legal in a path segment: it's
|
||
never a target for percent-encoding and never carries special meaning in any URI component, so a
|
||
short id can be handed to any part of the stack without first checking which encoding rules apply
|
||
there.
|
||
|
||
|
||
### Worked examples
|
||
|
||
Encoding `kind = item` (0), `owner_identity_id = 7`, `item_local_id = 42`:
|
||
|
||
- `kind`: 2 fixed bits → `00`
|
||
- `owner_identity_id = 7`: fits in one 4-bit chunk → 5 bits (`00111`)
|
||
- `item_local_id = 42`: needs two 4-bit chunks → 10 bits (`1001001010`)
|
||
|
||
Total: 17 meaningful bits, padded to the next multiple of 6 (18) with one zero bit, yielding 3
|
||
base64 characters: **`~DyU`**.
|
||
|
||
The same worked-out form for one example of every registered kind - `Bits` is the same
|
||
space-separated segmentation (kind tag, escape offset if present, each field, then padding) the
|
||
Examples table on `/~<token>` (`frontend/src/views/ShortId.vue`) shows for every registered kind:
|
||
|
||
| Kind | Fields | Serialized | Bits | Token |
|
||
|---|---|---|---|---|
|
||
| `item` | `owner_identity_id: 7`, `item_local_id: 42` | `[0, 7, 42]` | `00 00111 1001001010 0` | `~DyU` |
|
||
| `storage_location` | `owner_identity_id: 3`, `storage_location_id: 1000` | `[1, 3, 1000]` | `01 00011 100111111001000 00` | `~Rz8g` |
|
||
| `category` | `category_id: 5` | `[2, 5]` | `10 00101 00000` | `~ig` |
|
||
| `workflow` | `owner_identity_id: 2`, `workflow_id: 9` | `[3, 2, 9]` | `11 00000 00010 01001 0` | `~wCS` |
|
||
| `group` | `group_id: 11` | `[4, 11]` | `11 00001 01011` | `~wr` |
|
||
| `file` | `file_id: 123` | `[5, 123]` | `11 00010 1011101011 0` | `~xXW` |
|
||
|
||
`workflow`, `group`, and `file` are kinds 3-5, so their `Bits` column shows the escape tag (`11`)
|
||
followed by its own offset segment payload fields. |