From 395a9b156a1c84f712fb0f4c9b15d1ee2c803003 Mon Sep 17 00:00:00 2001 From: jedi Date: Thu, 20 Aug 2026 05:04:30 +0200 Subject: [PATCH] stash --- docs/design-in-progress/groups-mvp.md | 113 ++++++++++ docs/design-in-progress/groups.md | 233 +++++++++++++++++++++ docs/design-in-progress/image-caching.md | 166 +++++++++++++++ docs/design-in-progress/items-labels.md | 131 ++++++++++++ docs/design-in-progress/tags.md | 252 +++++++++++++++++++++++ docs/glossary-todo.md | 141 +++++++++++++ docs/glossary.md | 229 ++++++++++++++++++++ docs/handles-and-shortids.md | 168 +++++++++++++++ docs/index.md | 1 + 9 files changed, 1434 insertions(+) create mode 100644 docs/design-in-progress/groups-mvp.md create mode 100644 docs/design-in-progress/groups.md create mode 100644 docs/design-in-progress/image-caching.md create mode 100644 docs/design-in-progress/items-labels.md create mode 100644 docs/design-in-progress/tags.md create mode 100644 docs/glossary-todo.md create mode 100644 docs/glossary.md create mode 100644 docs/handles-and-shortids.md diff --git a/docs/design-in-progress/groups-mvp.md b/docs/design-in-progress/groups-mvp.md new file mode 100644 index 0000000..5fcaefe --- /dev/null +++ b/docs/design-in-progress/groups-mvp.md @@ -0,0 +1,113 @@ +# Groups MVP (Design in Progress) + +Status: not implemented. [groups.md](groups.md) works out the underlying model (what a group is, +how "acting as a group" is authenticated across federation). This document doesn't re-derive any +of that; it takes that design as given and asks a narrower question: what is the smallest set of +screens and actions that makes groups usable, reusing the UI patterns Toolshed already has for +friends and inventory rather than inventing new ones. + +## Goals + +- Ship something a user can actually click through: create a group, add/remove members, and have + the group own items, end to end. +- Reuse existing screens and interaction patterns wherever the shape already matches, instead of + designing new ones (see below). +- Leave every deferred piece as an explicit non-goal rather than a silent gap, so it's clear what + MVP does and doesn't cover. + +## Non-goals (for now) + +- **Group-to-group friending, and letting outside users friend a group.** groups.md's model + supports both, but the only audience for a group's items in MVP is the group's own members; + "private" already covers that (see "Availability policy for group items" below). Extending + visibility to non-members via the group's own friends list is real, valuable, and deferred as a + fast-follow, not designed here. +- **Transferring an existing personally-owned item's ownership to a group.** MVP only supports + creating a *new* item directly owned by a group. Moving an already-existing item across owners is + a separate piece of work (ownership transfer isn't something Toolshed supports for user-to-user + either today). +- **Deleting a group.** MVP has no explicit "delete this group" action. A group's lifecycle is just + "created, membership changes over time"; the orphaning guard (a group can't be left with zero + members, see groups.md's Known Gaps) means a group that's no longer wanted just sits unused rather + than needing a teardown flow. +- **Any governance, roles, or per-item permissions.** Same non-goal as groups.md: every member has + equal, full privileges over the group and everything it owns. + +## User-facing features + +### Creating a group + +A new "Groups" section, entry point styled like Inventory's "Add" button. The form is just a name; +the handle (`#name@yourdomain`) is derived from it the same way a username becomes part of a user's +handle. The creator becomes the group's first member automatically — there's no empty-group state +to design for. + +### My Groups list + +A "Groups" nav entry/page, modeled directly on `Friends.vue`: a table of groups the current user is +a member of (name/handle, member count), each row linking into that group's detail page. No +separate "discover groups you're not in" browsing for MVP — you land in a group by being added to +it, the same way you become friends with someone by request/accept, not by browsing a directory of +all users. + +Known limitation: this list only ever queries the member's own home backend, so it only shows +groups actually hosted there (groups you created, or joined on your own domain). Membership itself +works regardless of which backend hosts the group — a remote member can still be invited, accept, +and fully edit/delete the group's items (see "Owning items as a group" below) — but a group hosted +on someone else's backend won't show up in your own "My Groups" list, because unlike friendship +(which both sides record), group membership is only ever recorded on the group's own home backend, +and there's no index anywhere of "which other backends has this identity been added to." Making a +remote membership discoverable would need a small personal pointer index (written by the client at +join time) plus a handle-based group lookup on the group's own backend; deferred as a fast-follow +alongside group-friending. + +### Group detail page + +One page per group, with two sections, each reusing an existing pattern wholesale: + +- **Members** — the add/remove-row pattern from `Friends.vue`'s friend list: an inline "add member" + field (enter a handle, `user@domain`), and a remove (trash icon) action per row. Any member can + add or remove any other member — flat privilege, no confirmation step beyond the orphaning guard + (removing the group's last member is blocked, with an error explaining why, rather than silently + emptying the group). +- **Group inventory** — the exact table/grid pattern from `Inventory.vue` (Name, Availability + Policy, Amount, Edit/Delete), scoped to items owned by this group, with the same "Add" button + leading into the existing item-creation form (see below). + +A "Leave group" action removes the current user from Members; if they're the last member, it's +blocked by the same orphaning guard. + +### Owning items as a group + +The existing item create/edit form (`InventoryNew.vue` / `InventoryEdit.vue`) gets one new field: +an "Owner" selector, defaulting to "Myself," with the groups you belong to as the other options. +Every other field on that form (tags, properties, availability policy, storage location, quantity) +is unchanged, and a group-owned item behaves exactly like a personally-owned one everywhere else in +the app (edit, delete, detail view) — any member can edit or delete it, the same way the owner can +today. + +### Where group-owned items show up + +The main "Inventory" page stays scoped to items you personally own, unchanged from today. A group's +items are visible and managed in exactly one place: that group's detail page. This keeps "my +inventory" meaning one thing (what I personally own) and avoids merging two different item lists +with different edit semantics into one view for MVP. + +### Availability policy for group items + +The item form's existing Availability Policy field (private/share/lend/rent/sell) is unchanged and +applies to group-owned items the same way it does today. Since group-friending is out of scope for +MVP (see Non-goals), "share/lend/rent/sell" have no wider audience to expand to yet, only "private" +is fully meaningful right now — a private group item is visible to and editable by every group +member, which is already the core value the Problem statement in groups.md is after (a shared +workshop's members all having a say over shared equipment). The field stays as-is rather than being +trimmed down to just "private," so nothing needs to change on it once group-friending ships. + +## Open scoping call + +Deferring group-friending is the one judgment call in this document worth flagging explicitly: +it means an MVP group can't yet share an item with anyone outside its own membership, which is a +real limitation, not just a simplification. It was scoped out because it pulls in a second piece of +UI (a group's own Friends tab, and "who can accept a friend request on the group's behalf") that +isn't needed for the core "shared ownership among members" use case to work end to end. Worth +confirming this is the right line before building against it. diff --git a/docs/design-in-progress/groups.md b/docs/design-in-progress/groups.md new file mode 100644 index 0000000..966b1f4 --- /dev/null +++ b/docs/design-in-progress/groups.md @@ -0,0 +1,233 @@ +# Groups (Design in Progress) + +Status: not implemented. This document collects the problem, goals, and open design questions for +adding groups to Toolshed. Nothing here is settled; it's a starting point for discussion. + +## What a group is + +A group exists to model shared ownership of items, the way a club, workshop, or company owns +equipment collectively rather than any one person owning it. A group has members, and all members +have equal privileges to edit the items the group owns; there's no owner-vs-member distinction +within a group, membership itself is the privilege. + +This is a sharper (and more consequential) definition than "a label you can share things with": +it means a group needs to be able to *own* things, not just receive shared access to them the way +a friend does. That has implications worked out below. + +## Problem + +The only relationship Toolshed currently models between users is pairwise friendship (see +[federation.md](../federation.md)), and every item has exactly one owning user. That's enough for +"I trust this one specific person" and "I personally own this thing," but it breaks down for +anything collectively owned: + +- A shared workshop, tool library, or team can't own equipment as a unit. Today it has to belong + to one specific person's account, which is a poor fit and doesn't reflect who actually has a say + over it. +- There's no way for several equally-privileged people to edit the same item; edit rights today + are entirely tied to the single `owner` field. +- Adding or removing a member of an informal group currently means renegotiating friendships and + re-sharing individually; there's no shared object whose membership can just be edited once. + +## Goals + +- Let a set of users collectively own items, with every member holding equal edit rights over + those items. +- Let membership be managed in one place instead of via N pairwise arrangements. +- A request from a client to any backend may not depend on any other backend being online at the + same time — not to construct the request, and not to verify it. Concretely: the receiving server + must be able to verify the request using only the request itself plus keys it has already cached + from prior trust (friend-accept), with the group's own authoritative backend and the requesting + member's home server both unneeded and unreachable-safe at verification time; and the client must + be able to send the request using only what it already has cached, with the group's authoritative + backend unneeded and unreachable-safe at send time too. This is the same property plain user + requests already have (see federation.md's Cryptography section); group requests must not regress + it on either side. +- Fit into the existing handle system (see federation.md's "Unique Handles" section): a group + should be nameable and referenceable using the same handle shape a user is. +- Stay optional and additive. Pairwise friendship and single-user ownership should keep working + exactly as they do now for people who never touch groups. + +## Non-goals (for now) + +- Group governance beyond flat, equal membership (voting, roles, hierarchies). Equal privileges + for all members is the whole model for now; anything more layers on top later if needed. +- Fine-grained per-item permissions within a group (e.g. "this member can edit but not delete"). + Membership is the only privilege level. + +## Open design questions + +### Does a group need its own keypair? + +No. A group's day-to-day existence is a membership roster maintained by whichever backend is +authoritative for the group's handle (the same "authoritative backend" idea a user's domain already +implies, see federation.md's Servers subsection). A request made "as the group" is an ordinary +request, signed with an actual member's own personal private key, together with a claim of which +group it's acting on behalf of (`acting_as`), signed as part of the same payload as the rest of the +request. + +A receiving server must be able to authenticate such a request using only what arrives in the +request plus keys it already holds; no other server, including the group's own authoritative +backend, needs to be reachable at verification time. This is accomplished with **membership +certificates**: on request, a group's authoritative backend issues a current member a small +signed statement of the shape "handle X, public key P, is a member of group #G, valid from T1 until +T2," signed with the group's own private key. Issuance is on-demand (the member asks, rather than +the backend pushing renewals on a schedule), but it is its own separate action, decoupled from +sending any particular group request: a member fetches and refreshes this certificate from the +group's backend whenever they happen to be online, caches it locally, and later attaches whichever +certificate they currently hold to a request they make "as the group." Sending that request never +itself triggers a live fetch from the group's backend — if the cached certificate has expired and +the group's backend isn't reachable right then, the request simply can't be sent as the group yet; +the client doesn't fall back to contacting the group's backend synchronously to get one. + +A receiving server checks two signatures, using keys it already has cached, with no outgoing call +to anyone: the member's signature over the request, using the public key embedded in the +certificate itself, and the certificate's own signature, using the group's public key, learned and +cached exactly the way any friend's key is, at the point the group was friended. If both check out +and the certificate hasn't expired, the request is authorized. + +The certificate is what lets a receiving server trust a specific member's public key at all, for +members it has never individually friended: that trust is vouched for by the group's already-cached +key, rather than requiring a separate key-exchange with every member of every group a user happens +to be friends with. Equal privileges for all members falls out of this directly, since any member's +own key plus a valid certificate is sufficient proof. + +Removing a member takes effect once their existing certificate expires, not the moment the +backend's roster is edited; certificate lifetime is the parameter that governs how quickly a +removal actually takes hold (see "How long should a membership certificate be valid for?" below, +and Security below). + +### Are group handles different from user handles, or is a group just a special kind of user? + +The handle should look almost exactly like a user handle, just prefixed with `#`: `#groupname@domain` +instead of `groupname@domain`. It's resolved the same way and referenceable in the same places (e.g. +as an item's owner, or as a friend-list entry), so the reuse of the existing federation model is +unaffected. The prefix exists only to keep group and user handles from occupying the same namespace on +a domain: without it, "is `groupname@domain` a user or a group" would depend on which one happened to +register the name first, and the two could never be told apart just by looking at the handle. With the +prefix, a domain can have both a `climbing@domain` user and a `#climbing@domain` group with no +collision and no ambiguity about which is which, and any code path that resolves a handle can dispatch +on the actor kind (user vs. group) from the handle's own shape, before it even needs to ask a server. + +But underneath, a group isn't really "a special kind of user," it's backed by a fundamentally +different kind of identity: a membership list instead of a single keypair (see above). A user +handle answers "is this request authentically from this one identity," a group handle answers "is +this request from someone currently entitled to act for this collective identity." Those are +different questions even though the answer to both ends up being "yes, forward the request." Groups +and users are probably best thought of as two kinds of actor that share a handle format and most of +the surrounding plumbing (ownership, friendship, availability policy), rather than one being a +special case of the other. + +### Should a group be able to grant read access to non-members ("group friends")? + +Yes, this should reuse the same mechanism a user's own sharing already uses. Since a group is an +actor with a handle, it can have its own friends the same way a user does, and a group's owned +items can go through the same availability-policy check (private vs. visible to the group's +friends) that an individual's items already do. Nothing new needs to be invented here, it's the +existing friendship and availability-policy machinery applied to a second kind of actor. + +### Should groups be able to befriend other groups? + +Yes, for the same reason: if a group is an actor with a handle and a friends list, there's no +reason the other side of that friendship has to be a user specifically. Two clubs befriending each +other so each can see the other's shared equipment is the same mechanism as two users befriending +each other, just with both sides being groups instead of one or zero. + +### Should anyone be able to share directly with a specific group, instead of with "my friends" generally? + +This is the one piece that isn't just reuse of what exists today. Right now, an item's +availability policy is all-or-nothing across *all* of the owner's friends, there's no way to share +with a subset of friends, individually or as a named group. Letting an item be shared with one +specific group (a user's own item, shared with a club they belong to, say) instead of with every +friend equally would be a genuine generalization of the current sharing model, not something that +falls out of adding groups as an actor. + +Worth noting: this capability would be just as useful for individual users wanting to share with a +subset of their friends, without a group being involved at all. It might make more sense to design +"share with a specific target (user, group, or named subset)" as its own piece of work, rather than +building it as a groups-only feature. + +### How long should a membership certificate be valid for? + +This is the tuning knob the certificate design introduces, and it's a real trade-off rather than a +detail to defer. A short validity window (say, hours) keeps the staleness window after a removal +small, but means a member who's offline for longer than that can't act as the group at all until +they reconnect and refresh. A long window (days or weeks) is more forgiving of intermittent +connectivity but leaves a removed member's old certificate usable for longer. Whatever default is +picked, a member should be able to fetch a fresh certificate well before the old one expires while +still online, so the common case isn't "offline for exactly the wrong amount of time." + +## Interaction with availability policy + +Items currently have an availability policy (private / share / lend / rent / sell) that's a +property of the item, not a list of who it applies to, "friends" is implicit and applies equally to +all of them. A group-owned item works the same way, just with the group's own friends as the +implicit audience instead of an individual's. Targeted sharing (the question above) would extend +this, not replace it. + +## Known gaps in the design + +None of these block the design, but they're real gaps that need an explicit answer before +implementation. + +### Ambiguity + +- **What exactly is signed.** The `acting_as` claim and the certificate's identifying fields must be + signed as part of the same payload the member's key signs, not as free-standing, unsigned data + alongside it. If they aren't inside the signed bytes, `acting_as` can be swapped after signing, + turning a personal request into a group one or vice versa, or one group's request into another's. +- **Who can change membership, and how.** Equal privileges to edit items doesn't by itself say + whether that equality extends to *requesting or renewing certificates for others, or editing the + authoritative backend's issuance list itself*. Flat and unilateral (any member can add or remove + any member) is the simplest reading of "no owner-vs-member distinction," but it's a materially + different trust model from "equal edit rights over items" and deserves its own explicit decision. + +### Security + +- **Confused deputy on `acting_as`.** Trust in an `acting_as` claim reduces to trust in the group's + authoritative backend's issuance decisions: a malicious or compromised backend can sign a + certificate for a handle that was never really a member, and every receiving server that trusts + the group's key will accept it. The backend's issuance discipline, and the security of its own + private key, is a single point of failure for the group as a whole. +- **Membership staleness window.** There is a window after a member is removed during which their + existing certificate keeps working: exactly the certificate's remaining validity period. This is a + strictly worse revocation story than individual friendship, where trust is keyed to a public key + learned once with no expiry, but the window is a bounded, chosen parameter (see "How long should a + membership certificate be valid for?" above) rather than open-ended. +- **Blast radius of a single compromised member key.** Because membership is flat and unilaterally + editable by any member, a compromised personal key doesn't just expose that person's own items, + as with an ordinary account compromise, it exposes edit rights over everything the group owns for + as long as that member's certificate remains valid, and can be used to obtain a certificate for an + attacker-controlled handle as a permanent member before anyone notices. This risk is inherent to + "equal privileges, no roles" as a model, worth flagging even though richer governance is a + non-goal for now. +- **Self-lockout / orphaning.** Nothing in the design stops a group's last member from leaving (or + removing everyone else) from the backend's issuance list, which would leave group-owned items with + no one able to obtain a valid certificate for that owner at all once existing certificates expire. + The backend should guard against removing the last member, but that guard doesn't address a + member unilaterally removing every *other* member, which the flat model otherwise permits. +- **Audit trail depends on discipline.** Since any member's signature plus a valid certificate + satisfies authorization, "the group edited this item" is never sufficient for an audit trail; the + actual signer's handle (from the certificate's embedded public key) must always be logged + alongside the group claim, or member-level accountability is lost entirely. + +### Maintainability + +- **Two actor kinds sharing one code path.** `Group` should have the same shape as `User` for the + things that matter (a `.friends` set, a `.handle`), so ownership/friending/availability-policy + code can stay actor-agnostic. That reuse only holds if future code is disciplined about not + special-casing `User` in ways that assume a single, non-expiring keypair (e.g. "cache the owner's + public key forever, no expiry check needed") — a shortcut that would silently break the moment + the owner turns out to be a group, where the *acting member's* key is only good until its + certificate expires. +- **Certificate issuance and refresh is a client responsibility.** A member's client needs to + refresh its certificate before it expires, handle in-flight group actions failing closed if it + didn't (the same as any expired-credential error), and surface refresh failures to the user + rather than as a confusing rejected request. +- **Expanded federation test surface.** Every existing federation test implicitly assumes the + request's signer and its authorized actor are the same handle. `acting_as` plus an embedded + membership certificate means the whole request-verification path needs testing for the + signer-vs-actor split and the certificate's own signature and expiry checks, including + cross-domain cases (group hosted on one domain, member's key registered on another, item owned by + the group sitting on a third) and expiry-boundary cases (certificate expires mid-flight, is + refreshed concurrently with a request, etc). diff --git a/docs/design-in-progress/image-caching.md b/docs/design-in-progress/image-caching.md new file mode 100644 index 0000000..d2469c7 --- /dev/null +++ b/docs/design-in-progress/image-caching.md @@ -0,0 +1,166 @@ +# Authenticated image caching (Design in Progress) + +Status: not implemented. This document proposes a fix for a real performance gap: every +authenticated image in the app is refetched, re-verified, and re-decoded from scratch on every page +load, even though the backend already sends headers built for exactly the opposite. + +## Problem + +Images are served from `GET /media/` and `GET /media///` +(`backend/files/media_urls.py`), both gated behind `SignatureAuthentication` +(`backend/authentication/signature_auth.py`): the client signs the full request URL with an Ed25519 +key and sends `Authorization: Signature @:`. There's no cookie and no +URL-embedded token — auth lives entirely in a request header that a browser has no way to attach to +a plain ``. So `AuthenticatedImage.vue` does it by hand: `fetch()` with the header, +`.blob()`, `URL.createObjectURL()`, assign that to `src` (`federation.js`'s `getRaw`, +`fileCache.js`). `fileCache.js` is a module-level `Map` — it dedupes concurrent requests and holds +decoded blobs for the life of the page, but it's memory-only. Reload the page (or just navigate +between the SPA's route-based chunks in a way that re-mounts things) and it's gone; every image the +user has already looked at gets fetched, signature-verified, and blob-decoded all over again. + +Meanwhile the backend response already carries `ETag`, `Cache-Control: max-age=31536000, private, +immutable`, and a 365-day `Expires` (`_cache_headers`, `media_urls.py`) — because `src` is a +SHA-256 hash-addressed path, the same URL can only ever mean the same bytes, forever. Those headers +are correct and unused: nothing durable in the client ever consults them. This design closes that +gap using the browser's own Cache Storage API, without touching the backend. + +## Goals + +- Make a previously-viewed image load instantly on the next page load / browser restart, not just + within the current tab's JS session. +- Do it without weakening the authorization model: a signature is still required and verified + server-side for the *first* fetch of a given file by a given identity. Caching must not let one + identity's cached bytes leak to a different identity sharing the same browser. +- Reuse the backend's existing headers rather than inventing a parallel freshness scheme — content + is immutable, so a cache hit needs zero revalidation, ever. +- No backend changes. This is purely a client-side storage question. + +## Non-goals (for now) + +- **Revoking already-cached bytes when access changes** (e.g. an unfriend). The backend's own + 1-year `Cache-Control` already accepts that risk today for anything an HTTP-compliant cache might + hold; a persistent client cache extends the shelf life of that same accepted risk, it doesn't + introduce a new one. Not solving revocation here. +- **Prefetching / warming the cache ahead of navigation.** Real optimization, separate piece of + work; this document is about not throwing away work already done. +- **A Service Worker that reinstates plain ``.** Sketched below as a follow-up because it's + the "real" fix for the root cause (no way to attach a header to an `` request), but it's a + bigger lift (SW lifecycle, an extra message-passing bridge for signing) than the storage win alone + needs. Scoped out of this pass. + +## Design: persist `fileCache` with the Cache Storage API + +`window.caches` (the `CacheStorage` interface) is available to any page context, not just inside a +Service Worker — `caches.open(name)` gives a store of real `Request`/`Response` pairs that survives +reloads and browser restarts, backed by the browser's own disk quota. That's the missing tier; +nothing else about `fileCache.js`'s existing shape needs to change. + +**Two tiers, not one:** + +- **L1 — in-memory `Map`** (what exists today). Kept as-is: within a single page + session, components just want the already-created object URL back without re-touching storage at + all. Same LRU/budget logic (`MAX_BYTES`), unchanged. +- **L2 — `CacheStorage`**, consulted on an L1 miss, before falling back to the network. Holds raw + `Response` objects (not blobs), keyed by the same request used for the authenticated fetch. + +Revised `get(key, fetcher)` flow: + +1. L1 hit → return the object URL, as today. +2. L1 miss → check `cache.match(request)`. Hit → `.blob()` the cached response, create the object + URL, populate L1, done. **No conditional GET, no revalidation** — the response is `immutable`, + so if it's in the cache it's still correct by construction. +3. L2 miss → run the existing authenticated `getRaw()` fetch. On success, `cache.put(request, + response.clone())` before consuming the body, then proceed as today (`.blob()`, object URL, + populate L1). + +**Namespacing by identity, not one global cache.** `Cache-Control: private` on the response is the +backend telling shared caches to stay out — correct, since access is per-requester +(`_accessible_files`'s friends-or-self check). A single browser-wide `CacheStorage` bucket keyed +only by URL would quietly turn into exactly the shared cache that header is warning off, *if* this +browser ever holds more than one local identity (switching accounts, a shared machine). Concretely: +open the cache as `images-${username}@${domain}` (derived from the active `state.keypair`, the same +identity that produces the signature) rather than a single `"images"` name. Same-identity re-fetches +get the full cache benefit; a different identity in the same browser starts with an empty bucket and +goes through the normal authenticated-fetch-then-verify path, same as it does today. `invalidate()` +and `clear()` already exist on `FileCache` but nothing calls them — wire `clear()` to also +`caches.delete(currentNamespace)` and call it on logout/identity-switch, which is the natural, +already-there hook for this. + +**Storage budget.** L2 doesn't need its own hard byte cap the way L1 does — `CacheStorage` is +subject to the browser's own storage-pressure eviction, which is the right backstop for "durable but +not sacred" data like this. Optionally call `navigator.storage.persist()` once at startup to ask the +browser to exempt the origin from casual eviction under pressure; harmless to skip if declined. + +**Net effect:** a returning user's already-seen images (inventory thumbnails, profile pictures, +friends' shared items) render from disk with zero network round-trips and zero re-verification, +using exactly the durability guarantee (`immutable`, hash-addressed) the backend already asserts. +First-time images are unaffected — same authenticated fetch as today, just now also written to L2 on +the way through. + +## Follow-up worth flagging: a Service Worker to restore plain `` + +The deeper cost isn't just the network round-trip — it's that every image, cached or not, is forced +through manual `fetch → blob → createObjectURL`, so the browser's native image pipeline (off-main +thread decode, `loading="lazy"`, `fetchpriority`, responsive `srcset`) is unavailable, and object +URLs have to be manually revoked (`fileCache.js` already does this correctly, but every new call +site is a chance to leak one). The reason the app can't use plain `` at all is that nothing +can attach the `Authorization: Signature` header to a browser-initiated image request. + +A Service Worker can, because its `fetch` handler intercepts requests — including image loads — +before they leave the page, and can substitute its own request in place of the original: + +- On a `fetch` event where `event.request.destination === 'image'` and the URL matches `/media/`, + check the (identity-namespaced) `CacheStorage` first; hit → respond straight from cache, no + network at all. +- Miss → the SW doesn't have the signing key (it lives in page memory / `localStorage`, neither + reachable from a SW), so it asks the one controlled client (`self.clients.get(event.clientId)` — + the specific tab that issued the request, not "any open tab") for a signature over this exact URL + via `postMessage`/`MessageChannel` — an in-process round trip, not a network call — attaches the + returned header, performs the real fetch, stores the result in `CacheStorage`, and responds with + it. +- Once this exists, `AuthenticatedImage.vue` can go back to `` directly; the SW is what makes that legal despite the custom auth scheme. + +### Scoping the signing bridge: a compromised SW must not become a "sign anything" oracle + +The message bridge above is the one new capability this design adds that doesn't exist today: a +channel through which something can ask the page to sign a URL on its behalf. A Service Worker is a +long-lived, network-interposing piece of code — exactly the kind of thing a supply-chain compromise +or an XSS-planted `registration.update()` would target. If the page's message handler blindly signs +whatever URL the request names, a compromised SW stops being "something that can read images this +identity can already see" and becomes "something that can get a validly-signed request for *any* +endpoint" — e.g. `POST /api/inventory/items/5/delete` or `POST /api/friends/accept` — and then just +replay it directly against the real backend. That's a full account-takeover primitive smuggled in +through what was supposed to be an image-caching optimization, and it's strictly worse than not +having the bridge at all. + +The fix has to live on the page side of the channel, since the SW is the presumed-compromised +component in this threat model and can't be trusted to police itself. Treat the message handler as a +dedicated, narrow function — not a thin wrapper around the app's general-purpose signer +(`createSignAuth` in `federation.js`, which is used for arbitrary API calls elsewhere in the app) — +that: + +- **Ignores any method the request claims and always signs as `GET`.** The bridge never accepts a + body/`data` field from the SW at all, which closes off the entire class of mutating requests + (`POST`/`PUT`/`PATCH`) regardless of what path is named. +- **Validates the path against a strict allowlist grammar before signing anything**, rather than a + loose "starts with `/media/`" check. `src` values are hash-addressed — + `/media///<64-hex-char-sha256>.` for originals, with an optional `/<32|64|256>/` + size prefix for thumbnails. Because the variable part is constrained to `[0-9a-f]`, a regex over + that exact shape is effectively a closed grammar: `.` and `/` (the characters path traversal or + extra-segment tricks would need) simply aren't in the hex alphabet, so there's no meaningfully + malformed input that still matches. Anything that doesn't match — a different endpoint, an + encoded traversal attempt, an extra query string — is refused, silently or with a logged warning, + never signed. +- Optionally also checks the URL's host against the identity's home domain or its current friend + servers (belt-and-suspenders — a signature is bound to the exact signed URL string, so it can't be + replayed against a different host than the one named in it, but this catches a compromised SW + fishing for signatures against a host that happens to also trust this key for unrelated reasons). + +With this in place, the worst a fully compromised SW can do is obtain signed `GET`s for images the +current identity is already authorized to fetch — the same blast radius as "can read the +already-authorized image cache" — not an oracle for arbitrary authenticated mutation. + +Deferred because it adds real surface area (SW registration/update lifecycle, this scoped +message-passing bridge, first-load-before-SW-is-active edge cases) beyond what the storage change +alone needs. Worth doing as a second pass once the simpler win above is in and paying off. diff --git a/docs/design-in-progress/items-labels.md b/docs/design-in-progress/items-labels.md new file mode 100644 index 0000000..6a636fb --- /dev/null +++ b/docs/design-in-progress/items-labels.md @@ -0,0 +1,131 @@ +# Item Handles & Physical Labels (Design in Progress) + +Status: not implemented. This document collects the problem, goals, and open design questions for +giving inventory items stable identifiers and physical (scannable) labels. Nothing here is +settled. + +## Problem + +As described in federation.md's "Unique Handles" section, an item today is identified only by a +local id scoped to its owner, it isn't given an explicit, portable handle the way a tag, property, +or category is. That's fine as long as the only thing ever addressing an item is the owning user's +own signed API traffic. It stops being fine the moment something *outside* that loop needs to +refer to the item: + +- A friend who borrowed a physical tool has no way to look it up other than finding it in the + owner's shared inventory list by eye. + There's nothing you could put on a sticker. +- If items ever need to be referenced from outside their owner's own requests (a group's shared + view, a lending record, a printed label), there's currently no stable identifier to reference + that's meaningful outside the owner's own account. +- A local database id isn't something we'd want to expose or rely on externally: it's an + implementation detail of one backend's storage, not a handle with the same guarantees + (uniqueness, meaning, longevity) the rest of the federation model gives every other kind of + entity. + +Put simply: every other kind of thing in Toolshed (users, tags, properties, categories) has a +handle that means something outside of one database. Items don't, and physical labeling is the +clearest case where that gap actually matters. + +## Goals + +- Give an item a handle that's meaningful and resolvable outside its owner's own account, without + requiring items to become shared/reusable entities the way tags are (an item is still owned by + exactly one person; see the "Items" subsection of federation.md for why that keeps things + simple). +- Support a physical label (QR code, barcode, or similar) that can be printed and stuck on a real + object, such that scanning it gets you to the right item on the right backend. +- Make the label survive the normal life of a physical object: it gets lent out, comes back, + maybe changes which storage location it lives in, all without needing a new label printed. + +## Non-goals (for now) + +- Turning items into shareable/reusable entities across owners (that's what tags/categories are + for; see [tags.md](tags.md)). An item handle identifies *this specific person's specific thing*, + not a class of thing. +- Solving inventory tracking/auditing (check-in/check-out logs) as a whole system; that can build + on top of a stable item handle once one exists, but isn't the same problem. + +## Open design questions + +**What does the handle look like?** +The natural extension of the existing scheme is owner handle + local id, that's enough to be +globally unique (no two users share a handle, and ids are already unique within one user's +inventory) without inventing a new namespace. Worth deciding whether the id should be the existing +internal database id (simple, but leaks a little implementation detail and a rough count of +someone's inventory) or a separate opaque id generated for exactly this purpose (see the +unguessability question below). + +Two distinct formats are needed, because "an item handle" is used in two different situations: + +- *A compact handle, for use where context already makes clear it's a Toolshed item.* Inside the + app, in exports, in logs, anywhere the reader already knows they're looking at Toolshed data, + the handle doesn't need to spell that out or be openable on its own. This can be as short as + `user@domain.tld:id`, the item's owner handle with `:id` appended, mirroring how a tag/category + handle already appends `:name` after its origin (see federation.md's Unique Handles section). + No new delimiter concept, just the same pattern applied to items. + +- *A self-contained URL, for use with no context at all.* A physical label, a link shared outside + the app, has to work without the reader already knowing what it is or which server it belongs + to, so it needs to open directly to the right frontend, resolve the right backend, and land on + the right item. That means it has to encode the same information (owner handle + item id) as a + full URL, e.g. `https://toolshed.webdomain.tld/i/alice@example.com/42`, note the owner's handle + can be embedded in a path segment as-is (`@` doesn't need escaping in a URL path), which keeps it + one segment shorter than splitting the handle back into `domain/user`, and means the handle is + visible unmodified inside the link rather than reassembled from separate parts. If the owner is + ever a group rather than a user (see groups.md), its handle carries a leading `#`, which does need + the `+`-for-`#` substitution described in [handles-and-shortids.md](../handles-and-shortids.md)'s + Handle syntax section before it can sit in a path segment, e.g. + `https://toolshed.webdomain.tld/i/+climbing@example.com/42`. The frontend host + in this URL (`toolshed.webdomain.tld`) doesn't have to be, and generally won't be, the backend + authoritative for `example.com`, any frontend can resolve any handle (see federation.md's Servers + subsection), so this is just whichever frontend happens to be handling the link, not part of the + item's identity. + +Both formats should stay as short as the encoded information allows, this matters most for the URL +form, since it's the one that ends up in a QR code or printed label where physical size is a real +constraint (see the labels goal above). + +**How does this fit with the frontend's existing routes?** +There's already a `/inventory/shared/:user/:id` route (`InventoryDetailForeign`), but today +`:user` is just a bare username with no domain, i.e. it only works for a friend on the viewer's own +domain, and the view itself doesn't yet do anything domain-aware with that param. A resolvable +global handle needs the full `user@domain.tld` and a lookup step this route doesn't have yet. Two +ways to reconcile that: extend the existing route to take a full handle in the `:user` segment +(`/inventory/shared/alice@example.com/42`, no new route shape needed, just a richer meaning for the +param it already has), or treat the short `/i/...` URL as a dedicated, minimal entry point whose +only job is to resolve a handle and then hand off into whatever the richer in-app view ends up +being. The two aren't mutually exclusive: the short form is what needs to be small enough to print, +the in-app route doesn't have the same constraint and can stay more descriptive. + +**What does scanning a label actually do?** +Probably: the label encodes a URL or handle-like string; scanning it opens the frontend, which +resolves the owner's domain the same way it resolves any other handle (see federation.md), and +lands on that item. This reuses the discovery mechanism that already exists for logging in as a +handle, rather than inventing a second one. + +**Does resolving a label require authorization?** +An item's availability policy already controls who can see it (owner-only if private, friends if +shared, etc.). A label should presumably respect the same policy rather than being a backdoor that +makes a private item visible to literally anyone who finds the physical object and scans its code. +That means resolving a label isn't a free public lookup, it goes through the same friend/signature +checks as everything else, which has UX implications (an anonymous finder of a lost tool can't +necessarily see who it belongs to). + +**Does the label need to be opaque/unguessable?** +If item ids are small sequential integers, a label built from a guessable id lets anyone enumerate +a user's items by scanning or guessing nearby numbers, even if each individual lookup is +authorization-checked. Probably wants some amount of unguessability even before authorization is +considered, as a defense-in-depth measure. + +**What survives item changes?** +Storage location, availability policy, name, and description can all change over the life of an +object without it becoming a "different" item. The label should point at the handle, not at any of +that mutable data, so none of those changes require a new label. The one thing that probably *does* +need a decision is deletion: does a handle ever get reused, or is it retired for good once an item +is deleted (retiring seems safer, avoids an old label resolving to an unrelated new item later)? + +**Relationship to lending/borrowing.** +A scannable label is the obvious hook for a future "mark as borrowed / returned" flow. Not solving +that now, but the handle scheme chosen here should be able to carry that later without a redesign, +i.e. it should be able to identify the item independent of who currently physically has it. diff --git a/docs/design-in-progress/tags.md b/docs/design-in-progress/tags.md new file mode 100644 index 0000000..71b48a2 --- /dev/null +++ b/docs/design-in-progress/tags.md @@ -0,0 +1,252 @@ +# Tags, Properties & Categories (Design in Progress) + +Status: partially implemented (the handle scheme and strict resolution described in federation.md +exist today); this document is about the rough edges and open questions in that design, not a +proposal to build it from scratch. + +## Problem + +Tags, properties, and categories are handles of the form `origin#type:name` (see federation.md's +"Unique Handles" section), and a reference to a handle a server doesn't recognize is left +unresolved rather than guessed at or merged. That strictness is deliberate and avoids silent data +corruption, but it pushes a real cost onto users and creates some open problems: + +- **Discoverability.** There's no way today to search or browse "what origins/taxonomies already + exist" before creating a new tag. In practice this likely means people invent their own local + tags for things a shared, canonical origin already covers, simply because they didn't know it + existed. +- **Fragmentation.** Because uniqueness is scoped per origin, nothing stops semantically identical + concepts from existing under different names or different origins at once (`drill` vs. + `power-drill`, one server's `origin:mytools` vs. another's `origin:community-tools`). Nothing + reconciles these; they just coexist. +- **No aliasing/synonyms.** If a user starts out with a local tag and later wants to adopt a shared + origin's equivalent, there's no supported way to merge or alias the two; existing items keep + pointing at the old handle. +- **Trust and quality of shared origins.** An origin is "whatever the classification is considered + to have come from," which could be an imported reference dataset. Nothing in the current design + addresses who maintains such a dataset, how it's kept correct, or what happens when it changes + (a category gets renamed or split upstream, e.g.). +- **Localization.** A tag/category name is a single string. There's no notion of the same concept + having a different display name per language while still resolving to the same handle. +- **Property values aren't typed.** A property's value on an item is a plain string. A property + definition can carry a unit (`unit_symbol`/`unit_name`), but nothing declares that its values are + actually numbers meant to be read in that unit, versus, say, free text that happens to mention a + unit. This is already a problem for one server in isolation (see the worked example below), and + federation doesn't create it, it just means it now has to be solved consistently across origins + instead of once. +- **Handle collisions.** `origin` is a free-text string with nothing enforcing that it's actually + unique to one definition. Fragmentation (above) is two different strings for the same thing; + this is the more dangerous mirror image, the same handle string ending up attached to two + different definitions, which is exactly the scenario the strict-resolution design was supposed + to make impossible (see the second worked example below). + +### Worked example: filtering by a numeric range across origins + +Say two independently-run servers both end up with a property called "voltage," each under its own +origin: `git:base#property:voltage` and `some-other-origin#property:voltage`. A user who's friends +with people on both wants to filter their combined view of shared items by, say, `0 < voltage < +4`. Two separate problems stack up: + +1. **Are these the same property?** Per the fragmentation problem above, the two handles are, and + stay, unrelated as far as the system is concerned, same name, different origin, no connection. + A range filter built against one handle simply won't match items tagged with the other, even + though a person looking at both would probably call them "the same thing." +2. **Even if they were recognized as the same thing, are the values comparable?** A range filter + needs actual numbers in a known unit. If one server's items store `"3.7"` and the other's store + `"3700"` (volts vs. millivolts), a numeric comparison across the two silently produces nonsense + unless the unit is known and converted. If either side stores the value as loosely-formatted + text (`"3.7V"`, `"~3.7"`) rather than a bare number, it may not be reliably parseable as a + number at all. + +So a cross-origin range filter needs both an aliasing/equivalence answer (are `voltage` and +`voltage` the same concept) and a units/typing answer (are their values actually numbers, in units +that convert cleanly into each other), and the first being solved doesn't imply the second is. + +### Worked example: the same handle meaning two different things + +Say `git:ee2` names a specific reference dataset that started life in one shared git history, and +two servers each imported it, at different times, from what has since become two diverging +branches (or forks) of that history. Both servers now have a property whose handle is the exact +same string, `git:ee2#property:charging_voltage`, but whose actual definition, say, unit, or +dimensions has since diverged between the two branches. Neither server did anything wrong; each +one faithfully imported "`git:ee2`" as it existed at the time. + +This is a materially worse problem than the fragmentation/voltage example above. Fragmentation is +a missed opportunity, two things that should be linked aren't, and the failure is visible (the +filter just doesn't match as much as a person would expect). A handle collision is silent: nothing +about the two servers exchanging data suggests anything is wrong, both sides say +`git:ee2#property:charging_voltage`, so anything that trusts equal-handle-means-equal-definition +(exactly what the strict-resolution design promises, and exactly what a filter, an alias, or a +plain item-detail display would rely on) can silently combine or display incompatible values as if +they were the same thing. This is the specific failure mode the whole handle design exists to +prevent, so a scheme where it can still happen is a real gap, not just an inconvenience. + +The root cause is that `origin` is a free-text label describing where something came from, not an +identifier that's actually bound to a specific, fixed piece of content. A name like `git:ee2` reads +as if it points to something immutable, but nothing about the origin field enforces that, `ee2` +could easily be a branch or tag name rather than a specific commit, i.e. a pointer that can keep +moving, and two importers pinned to it at different times without ever taking on a different +handle to show for it. + +This isn't hypothetical, it's exactly how the reference data included in this repository already +works. The files in `backend/shared_data/` (`base.json`, `ee.json`, etc.) are origin datasets: +`configure.py`'s import step sets `origin = "git:" + filename` for everything a file defines +(`configure.py:119`), and a file can declare a fixed dependency on another one by name, e.g. +`ee.json`'s `"depends": ["git:base"]`. The intent is clearly that these files are immutable once +committed, and that a `depends` entry is a pin to a specific, settled parent, not a moving target, +but nothing in the code enforces that today, it's a convention people are expected to follow. + +There's already a piece of the machinery needed to enforce it, though: import already computes a +sha256 of each file's raw content and stores it (`configure.py:125`, saved onto +`ImportedIdentifierSets.hash`, which is `unique=True` alongside `name`, `hostadmin/models.py:15`). +It's just not used for the thing it would be useful for, the "already imported, skipping" check +(`configure.py:129-130`) matches on `name` alone, it never recomputes the hash of the file being +imported and compares it against the hash already on record for that name. So even on a single +server, an edited `ee.json` re-imported under its old filename wouldn't be noticed as a change, +let alone flagged as a conflict, and the hash never leaves that server's own bookkeeping to be +compared against what a friend server has on record for the same name. + +## Goals + +- Make it easy to find and reuse an existing origin/handle before inventing a new local one, to + reduce fragmentation without weakening the strict-resolution guarantee that already exists. +- Give users a path to move a locally-invented tag onto a shared origin later, without losing or + having to manually re-tag their existing items. +- Keep the core guarantee intact: a handle always means one specific, traceable thing, nothing + should be implicitly merged or reinterpreted across origins. +- Make that guarantee actually hold, not just assumed: two servers that both use a given handle + should either really mean the same definition, or have some way to find out they don't, rather + than the collision staying silent. + +## Non-goals (for now) + +- Building a moderation/governance system for shared origins. Worth thinking about, but a bigger + problem than this document is trying to scope. +- Free-text/fuzzy tag matching in search. Search UX can layer on top of resolved handles without + changing what a handle means. + +## Open design ideas + +**A small set of well-known, shipped origins.** +Toolshed could ship with one or a few canonical origins covering common tool/inventory categories +out of the box, so that a fresh server already has a sensible baseline vocabulary to reuse instead +of every server reinventing "drill," "screwdriver," "power tools," etc. independently. Doesn't +solve fragmentation for everything, but raises the floor. + +**Explicit aliasing rather than merging.** +Rather than trying to detect and merge "equivalent" tags automatically (risky, exactly the kind of +implicit behavior the strict-resolution design intentionally avoids), a tag could carry an explicit, +user-initiated "supersedes"/"alias of" pointer to another handle. Items already tagged with the old +handle could then be offered a one-time, explicit re-tag rather than a silent change of meaning. + +For properties specifically, an alias needs to claim more than "these mean the same thing," it +needs to claim the values are comparable, which means recording a unit conversion (possibly just +"identical unit, factor 1") alongside the alias, not just a bare pointer. An alias with no stated +conversion should probably be treated as "same concept, values not (yet) comparable," a range +filter has no business guessing a conversion on its own. + +**Filters default to per-handle, and only widen on an explicit alias.** +Following directly from the strict-resolution philosophy in federation.md: a range filter should +only ever combine two distinct property handles into one filterable facet because of an explicit +alias (see above) that also states the values are comparable, never because their names or units +happen to match. Absent that, two same-named properties from different origins should just show up +as two separate filters, visibly distinct, rather than the UI silently guessing they're the same +and producing a filter result that mixes incomparable values. + +**Typed property values.** +Giving a property definition a declared value type (number, text, boolean, ...) in addition to its +existing unit metadata would let both the frontend and the alias/conversion mechanism above know +whether "range filter" even applies to a given property, and would close the gap where a value +happens to look numeric but isn't guaranteed to parse as one. This is useful even without +federation in the picture, cross-origin comparison just makes the gap load-bearing instead of +cosmetic. + +**Origin metadata/versioning.** +If an origin represents an imported dataset, giving it its own version or changelog would let a +server know when the upstream taxonomy it imported has moved on, and decide explicitly whether to +re-import, rather than silently drifting from what other servers using the "same" origin now have. +This only helps if everyone's still on one shared timeline, though, it doesn't by itself address +diverging forks/branches ending up with the same name (see below). + +**Pin origins to immutable content, not movable names.** +The `git:ee2` collision happens because the origin string names something mutable (a branch/tag) +rather than something fixed. If an origin string were derived from the content itself, e.g. a hash +of the definition, or a specific immutable commit rather than a branch, two independent imports +could never end up with the same string unless the content was actually identical at that point, +collisions would become structurally impossible rather than just unlikely. This is a bigger change +than the versioning idea above: it's not tracking change over time, it's making the identifier +itself incapable of silently referring to different things. + +A concrete version of this: use the git blob hash of the file a property/tag/category was defined +in as (part of) its origin. If the reference dataset already lives in a git repo, this is free, +git's already computed it, and it's independently checkable, any server holding or able to fetch +the same repo can recompute the hash from the content and confirm for itself, rather than trusting +a label. This is a real improvement over free-text `origin` strings, but it isn't a free lunch: + +- *Granularity.* A blob hash identifies a whole file, not a single property. If a file defines + several properties together, editing any one of them changes every other property's "identity" + in the same file too, even though nothing about them changed. Either definitions need to be + one-per-file for the hash to mean what's intended, or the hash needs to cover just the relevant + entry rather than the literal git blob. +- *It converts every edit into a fork.* Since any change, including a typo fix, changes the hash, + a routine upstream correction mechanically fragments what's still the same property into two + handles. That's consistent with "never silently reinterpret a handle," but it means the aliasing + mechanism above stops being a nice-to-have and becomes the primary upgrade path, every legitimate + edit needs an explicit "supersedes" link, or old items are stranded on a stale, now-orphaned hash. +- *A hash alone is an identity, not a location or a label.* It proves two things are the same (or + aren't), but doesn't say where to fetch the content from if you don't already have it, and isn't + human-readable. Pairing it with a location (which repo) and a mnemonic (which release/name it + corresponds to) alongside the hash keeps the discoverability goal intact instead of trading it + away for collision-proofing. + +**Detect collisions on contact, as a backstop.** +Even with better-behaved identifiers going forward, existing data and human-typed origin strings +mean collisions can't be ruled out entirely. Whenever two servers interact over a handle they both +claim to know (e.g. as part of resolving an alias, or federated search), comparing a fingerprint of +the full definition, not just the handle string, would let a mismatch surface as an explicit +conflict to resolve, rather than being silently trusted. This is the same instinct as the "explicit +alias must state whether things are comparable" idea above, applied in the opposite direction, here +the handles already match and the system needs to actively check whether that trust is warranted. + +The needed ingredient already exists locally and just isn't being used this way: the sha256 hash +already computed and stored per import (see above) is exactly the kind of definition fingerprint +this needs. Two changes would make it actually do the job: first, comparing it on every import +(including a "re-import" of a name already on record), not just recording it once, so a locally +edited file gets caught before it's ever presented to anyone else, and second, exchanging it as +part of whatever federated interaction references a shared-origin handle, so two servers can +compare hashes for the same name and find out they've diverged instead of assuming they haven't. + +**What this means for the handle actually on the wire.** +Putting the above together, the everyday handle shouldn't change shape at all. It stays +`origin#type:name`, e.g. `git:ee2#property:charging_voltage`, exactly as it is today. The reason is +redundancy: this string is what appears on every single reference (every item's tag list, every +property assignment), potentially many times per item across many items, while a hash only ever +needs to be known once per origin. Carrying a full hash on every occurrence would repeat the same +value over and over for no benefit beyond what knowing it once already provides. + +Instead, the hash stays where it already lives, attached to the origin as a whole (extending +`ImportedIdentifierSets`, see above), and gets exchanged at the points where two servers actually +need to agree on one, e.g. the first time a friend's item references an origin a server doesn't +already have a hash on record for. First contact just records it, same as resolving any unfamiliar +handle today; a later mismatch against what's on record is the collision, and that's the point +where it needs to become visible rather than silently trusted. + +Only once a collision has actually been found does the wire format need to say more than +`origin#type:name`, because at that point there genuinely are two different things sharing a name +and something has to distinguish them for a person sorting it out. A short, abbreviated hash +appended to the origin, the same idea git itself relies on for short commit hashes, keeps that +escape hatch usable: `git:ee2~0f3a9c1e#property:charging_voltage` versus +`git:ee2~7bc82a04#property:charging_voltage`. This longer form is exception-path plumbing for +resolving an already-detected conflict, not something that changes the size or shape of handles in +the common case. + +**Search across known origins.** +Before creating a new tag/category, a creation flow could search across origins the local server +already knows about (its own, plus any it's imported) and surface likely existing matches. This is +a UX/workflow fix rather than a change to the handle model itself, it doesn't need to touch +resolution semantics at all. + +**Display name vs. handle.** +Separating "the name that appears in the handle" (stable, part of the identity) from "the label +shown to a user" (translatable, cosmetic) would allow localization without affecting resolution or +uniqueness, since resolution would stay keyed on the handle, not the display string. diff --git a/docs/glossary-todo.md b/docs/glossary-todo.md new file mode 100644 index 0000000..1b91583 --- /dev/null +++ b/docs/glossary-todo.md @@ -0,0 +1,141 @@ +# Glossary terminology TODO + +Working list from a repo-wide audit of where code/docs use a different word for a concept that +[glossary.md](glossary.md) already gives a canonical name. Nothing here has been changed yet, this +is a collection point before any renaming/edit work starts. Grouped by glossary term; only real +inconsistencies are listed, not every correct usage that was checked and cleared. + +## Backend + +- `frontend/src/federation.js` — the whole module (`class ServerSet`, `add(server)`, every + request method) talks about "server" throughout where the concept is a **Backend**. +- `frontend/src/store.js` — `getHomeServers`, `getFriendServers`, `getAllKnownServers`, + `setAllFriendsServers`/`all_friends_servers`, `home_servers`, `lookupServer` (~line 313, 325, + 332, 359, 76-77, 107-108, 334). +- `frontend/src/views/Friends.vue:16,43` — user-visible table column labeled "Server". +- `docs/design-in-progress/tags.md` (lines 10, 20, 45, 76-78, 117, 204, 228-229) — "server" used + throughout for what the glossary calls Backend (same looseness federation.md already has, but + worth normalizing here too since tags.md is in active editing). +- `issues.md:58` — "federated home servers" / "user identity" conflation. +- `deploy/dev/docker-compose.yml` (`instance_a`/`instance_b`) and `docs/development.md:106-108` — + "instance"/"backend instance" as a near-synonym for Backend. +- `cli-client/toolshed-client.py:11-70`, `README.md:92` — `--host`/`self.host` for "which backend + to talk to". + +## Discovery + +- `frontend/src/store.js:313` — `lookupServer` action *is* the discovery operation, never named + "discovery". +- `frontend/src/store.js:347` — `could not resolve server for friend` — "resolve" used instead. +- `docs/design-in-progress/items-labels.md:99`, `docs/development.md:108` — describe the + discovery operation via "resolves"/"direct the frontend to the correct backend" without naming + it (minor, but candidates for a one-word tightening). + +## Handle / User handle + +- ~~`backend/authentication/models.py` (`class KnownIdentity`...), `signature_auth.py` + (`author_identity`...), `frontend/src/identity.js` (`serializeIdentityRecord`...)~~ — no longer a + finding: the glossary now has an explicit **Identity** entry (handle + keypair, held together as + the unit a backend trusts), and this is exactly what these already name. No renaming needed here; + if anything, these are the parts of the codebase the new Identity entry should point to as + reference implementations. +- `backend/toolshed/serializers.py:49-57` (`FriendSerializer`) — API field is literally named + `"username"` but its value is a full handle (`username + '@' + domain`). Already + self-acknowledged in a comment at `frontend/src/store.js:404`. **Highest-value single fix** — + it's a live API contract, not just an internal name. +- `frontend/src/store.js` — several action params destructured as `{username}` that actually carry + a full handle: `lookupServer` (313), `getFriendServers` (359), `fetchFriendProfile` (401-405), + `login` (276-282). +- `frontend/src/views/Login.vue` (lines 24, 27-28, 82-83, 102-105, 115-117) — form label/variable + "Username" for a field that must be a full user handle (`user@domain`, per its own validation + message at line 103). +- `frontend/src/router.js:51` — route param `/inventory/shared/:user/:id` uses `:user` for what's + meant to eventually be a full handle; contrast with the sibling route at line 61 which already + correctly uses `:handle`. (Already called out by items-labels.md itself, so low-risk to leave + as-is, but listed for completeness.) + +## Availability policy, Friend/Friendship, Signature/Signing, Strict resolution, Actor, Targeted sharing + +No real inconsistencies found — implemented code already uses the glossary's own terms +consistently (`availability_policy` field name throughout backend+frontend; `friend`/`befriend` +consistently; `Signature`/`sign`/`verify` consistently; `_HandleNotFound`/`_resolve_handle` in +`backend/toolshed/offlinedata.py` implement strict resolution faithfully without needing to name +it; Actor and Targeted sharing are unimplemented with no competing name anywhere). + +- Checked and cleared, not a real conflict: `frontend/src/neigbors.js`'s `NeighborsCache`/ + "neighbor" vocabulary — refers to unreachable backend *domains* during discovery, not to + friendship, despite reading like a synonym at a glance. + +## Group / Group handle / Membership list + +- `backend/backend/settings.py:36` — `django.contrib.auth` ships a built-in `Group` model, shown + as "Groups" in the Django admin. Not unregistered anywhere. Will collide by name with the + proposed actor-type Group once that's implemented — worth a decision now (unregister the + built-in admin Group, or otherwise disambiguate) before the real feature lands. +- `issues.md` (issue #3, "Group Concept", ~lines 62-156) — a standalone proposal that conflicts + with the already-settled `docs/design-in-progress/groups.md` design on three points at once: + - `Group.public_key`/`private_key` fields (contradicts "does a group need its own keypair? No"). + - `GroupMembership` backed by a signed `membership_certificate` rather than a plain membership + list (contradicts the glossary's Membership list entry). + - Bare `Group.handle` strings with no `#` prefix, e.g. `"makerspace-nord"`, and a + `GroupInvitationIncoming.group_handle` field/API surface (`POST /api/groups/` etc., ~lines + 65-66, 117-121, 150-156) that never uses the `#groupname@domain` shape. + This is a design-conflict issue, not a wording tweak — `issues.md` should be reconciled with (or + explicitly marked superseded by) `groups.md` before anyone implements from it. + +## Keypair / Private key / Public key + +- Wire-format drift on the one field that actually crosses the network: `befriender_key` is used + for a public key at `frontend/src/store.js:435,449` and `backend/toolshed/api/friend.py:107`, + while the model field, serializer field, and UI all call the same value + `befriender_public_key`/`public_key` (`backend/authentication/models.py:144`, + `backend/toolshed/serializers.py:65`, `backend/toolshed/api/friend.py:118`, + `frontend/src/views/Friends.vue:81`). +- `cli-client/toolshed-client.py` (`--key`, `TOOLSHED_KEY`, `self.signing_key`, ~lines 12, 52, 57) + and `README.md:92` — never say "private key," just "key"/"Toolshed key", even though it's + specifically the private half. + +## Origin + +- `backend/configure.py:130` ("Identifier set {} already imported, skipping") and the model + `ImportedIdentifierSets` (`backend/hostadmin/models.py:13-19`) — call an imported origin dataset + an "identifier set". +- `issues.md:206-210` — Instance Admin TODO list: "identifier-sets" for **Origin** and bare + "identifiers" for **Classification handle**, both alternate terms not matching glossary names. + +## Alias + +- `backend/shared_data/ee_packages.json:40,64` — two tags already carry an `"alias"` field in real + data (e.g. `SOT54` → `alias: "TO-92"`), but shaped as a bare name string, not an + `origin#type:name` handle pointer, and with no unit-conversion concept. It's silently dropped on + import today (`Tag`/`TagSerializer` have no `alias` field). Not a different-word issue, but the + design doc (which calls Alias "Proposed") doesn't acknowledge this pre-existing, inert + precedent — worth reconciling either the data or the doc. + +## Tag / Property / Category + +- `frontend/src/components/workflow/workflows/BulkItemImportWorkflow.vue:495` — CSV + column-auto-mapping heuristic treats `"type"` as a synonym for Category: + `lowerColumn.includes('category') || lowerColumn.includes('type')`. + +## Item Handle + +- `frontend/src/views/Search.vue:30,52,110` — a field literally named `handle` is computed here + (`e.owner==this.user ? e.id : "shared/"+e.owner+"/"+e.id`), but it's a router-path fragment, not + an Item Handle: no domain-qualified `user@domain.tld:id` shape, and `e.owner` is a bare username. + Whoever implements the real Item Handle later is likely to collide with this existing variable. + +## Item Label + +- `frontend/src/components/workflow/workflows/FotoFirstBulkImportWorkflow.vue:567-572,822` and + `FotoFirstBulkImportWorkflow2.vue:593-598,847` — checkbox "Generate QR codes for items" / + `importOptions.generate_qr_codes` names exactly the Item Label concept but never uses that term + (and the option is currently unwired — declared and defaulted `true` but never read elsewhere). + +## Item URL / Local id / Domain / Frontend / Definition fingerprint / Fragmentation / Handle collision / Classification handle + +No real inconsistencies found — each already uses consistent, glossary-matching vocabulary +(`id`/`item_id` for Local id; `origin` kept cleanly separate from `domain` everywhere it's used; +`get_handle()` consistently for Classification handle; no competing names found anywhere for +Fragmentation, Handle collision, or Definition fingerprint, which also doesn't collide with the +unrelated `File.hash` content-hash field despite both being called "hash"). diff --git a/docs/glossary.md b/docs/glossary.md new file mode 100644 index 0000000..2a3cc62 --- /dev/null +++ b/docs/glossary.md @@ -0,0 +1,229 @@ +# Glossary + +This page collects one unambiguous name for each concept discussed in [federation.md](federation.md) +and the [design-in-progress](design-in-progress/) documents, so later writing can refer to them +consistently instead of reinventing or subtly renaming them. Entries are grouped by topic, and +alphabetical within each group. Each one notes whether the concept exists in Toolshed today or is +still a design proposal, and links back to where it's discussed in full. + +Two terms are easy to conflate and worth telling apart up front: a **domain** is who a handle +belongs to; an **origin** is where a classification entry came from. They look similar in prose but +are unrelated concepts, see their entries below. + +--- + +## Federation Basics + +**Backend** (Implemented) +The server software that stores an [actor](#actor)'s data and is currently authoritative for a +[domain](#domain). Deliberately decoupled from both the domain (a backend isn't tied to one domain, +and can be authoritative for many at once) and the [frontend](#frontend) (the app a user loads isn't +necessarily served by the backend that ends up handling their requests). Prefer "backend" over the +looser word "server" when precision matters, "server" is used informally in places to mean either +backend or domain interchangeably. +*See: [federation.md](federation.md#servers)* + +**Discovery** (Implemented) +The lookup a [frontend](#frontend) performs to find which [backend](#backend) is currently +authoritative for a [domain](#domain), given a [handle](#handle). Operational/DNS-level detail +about how this lookup works belongs in the deployment docs, not here, this glossary only fixes the +name for the concept. +*See: [federation.md](federation.md#servers)* + +**Domain** (Implemented) +The half of a [handle](#handle) after the `@`, e.g. `toolsheddomain.tld`. An authority record, not +a location or a piece of software: it says which [backend](#backend) currently vouches for handles +under it, nothing more. Not the same thing as an [origin](#origin), see the note at the top of this +page. +*See: [federation.md](federation.md#servers)* + +**Frontend** (Implemented) +The client application a user interacts with. Independent of any one [domain](#domain) or +[backend](#backend): given a handle, it performs [discovery](#discovery) to find the right backend +and talks to it directly, regardless of where the frontend itself was loaded from. +*See: [federation.md](federation.md#servers)* + +**Handle** (Implemented) +A name that's unique within its own scope and carries, as part of itself, enough information to say +where it's authoritative, without needing a central registry to look it up. The general term +covering [user handles](#user-handle), [group handles](#group-handle) (proposed), +[classification handles](#classification-handle), and [item handles](#item-handle) (proposed). +*See: [federation.md](federation.md#unique-handles)* + +**Strict resolution** (Implemented) +The rule that a reference to a [handle](#handle) a backend doesn't recognize is left unresolved +rather than guessed at or silently merged into something that looks similar. The foundational +guarantee the rest of the handle system, and most of the open problems in +[tags.md](design-in-progress/tags.md), are trying to preserve or actually make hold. +*See: [federation.md](federation.md#tags-properties-and-categories)* + +## Actors, Identity & Sharing + +**Actor** (Proposed) +Umbrella term for anything that can hold a handle, have friends, and own items. Today this only +means [user](#user). [Groups](#group) are a proposed second kind of actor, so that "friendship," +"ownership," and "handle" all mean the same thing regardless of which kind of actor is involved. +*See: [groups.md](design-in-progress/groups.md#are-group-handles-different-from-user-handles-or-is-a-group-just-a-special-kind-of-user)* + +**Availability policy** (Implemented) +A setting on an item (`private` / `share` / `lend` / `rent` / `sell`) controlling who besides the +owner can see it. Today the audience for any non-`private` policy is implicitly "all of the owner's +[friends](#friend-friendship)," equally, there's no way to name a narrower audience. See [targeted +sharing](#targeted-sharing) for the proposed alternative. +*See: [groups.md](design-in-progress/groups.md#interaction-with-availability-policy)* + +**Friend / Friendship** (Implemented for users; proposed for groups) +A mutual, explicitly-established trust relationship between two [actors](#actor). Established by a +friend-request/accept exchange, which is also the point a [public key](#keypair-private-key-public-key) +is first trusted for that handle. Currently only exists between users; groups having friends, and +groups befriending groups, are proposed extensions of the same mechanism, not a new one. +*See: [federation.md](federation.md#cryptography), [groups.md](design-in-progress/groups.md#should-a-group-be-able-to-grant-read-access-to-non-members-group-friends)* + +**Group** (Proposed) +A second kind of [actor](#actor), modeling collective ownership (a club, workshop, or company) +rather than any one person owning something. All members hold equal edit rights over what the group +owns; membership itself is the privilege, there's no separate owner/member distinction within a +group. Backed by a [membership list](#membership-list) rather than a [keypair](#keypair-private-key-public-key), +and identified by a [group handle](#group-handle). +*See: [groups.md](design-in-progress/groups.md#what-a-group-is)* + +**Group handle** (Proposed) +A [group](#group)'s handle: a name and [domain](#domain) written like a [user handle](#user-handle) +but prefixed with `#`, e.g. `#groupname@toolsheddomain.tld`. The prefix keeps groups and users in +disjoint namespaces on the same domain (no squatting collision between a user and a group wanting +the same name) and lets an actor's kind be read directly off its handle, without a lookup. +*See: [federation.md](federation.md#users-and-groups), [groups.md](design-in-progress/groups.md#are-group-handles-different-from-user-handles-or-is-a-group-just-a-special-kind-of-user)* + +**Identity** (Implemented) +A [user handle](#user-handle) paired with the [keypair](#keypair-private-key-public-key) that backs +it, held together as the one unit a [backend](#backend) actually trusts: not just a name, and not +just key material, but both at once. This is what gets established at registration for one's own +handle, and what gets recorded on [friend](#friend-friendship)-accept for someone else's handle. +Only [users](#user) have an identity in this sense, since a [group](#group) is deliberately backed +by a [membership list](#membership-list) instead of a keypair, there's no key half for a group +handle to pair with. +*See: [federation.md](federation.md#cryptography)* + +**Keypair / Private key / Public key** (Implemented, users only) +The asymmetric keypair backing exactly one [user handle](#user-handle); together, a handle and the +keypair backing it are what's called an [identity](#identity). The private key signs requests made +as that user and never leaves their control; the public key is handed out to establish trust in the +handle (at registration, or via a [friend](#friend-friendship) exchange) and is used to verify +signatures. Only user handles carry a keypair, not groups, classification handles, or item handles. +*See: [federation.md](federation.md#cryptography)* + +**Membership list** (Proposed) +The record of which [user handles](#user-handle) currently belong to a [group](#group), maintained +by whichever backend is authoritative for the group's handle. What backs a group's identity in +place of a keypair: a request "as the group" is a normal signed request from a current member, plus +a check against this list, not a request signed by some shared group key. +*See: [groups.md](design-in-progress/groups.md#does-a-group-need-its-own-keypair)* + +**Signature / Signing** (Implemented) +The act of authenticating a request as genuinely coming from a specific [user +handle](#user-handle), unaltered, using that handle's private key. Establishes authenticity and +integrity only, not confidentiality (that's HTTPS's job) and not, today, protection against replay. +*See: [federation.md](federation.md#cryptography)* + +**Targeted sharing** (Proposed) +Sharing an item with one specific [actor](#actor) (a particular friend, or a particular group) +instead of the current all-or-nothing [availability policy](#availability-policy) audience of every +friend equally. Flagged as a generalization useful beyond groups specifically, not a groups-only +feature. +*See: [groups.md](design-in-progress/groups.md#should-anyone-be-able-to-share-directly-with-a-specific-group-instead-of-with-my-friends-generally)* + +**User** (Implemented) +The original, and currently only implemented, kind of [actor](#actor): backed by exactly one +[keypair](#keypair-private-key-public-key) and identified by a [user handle](#user-handle). +*See: [federation.md](federation.md#users-and-groups)* + +**User handle** (Implemented) +A user's username paired with its [domain](#domain), written like an email address, e.g. +`user@toolsheddomain.tld`. Unique only within its domain, not across all of Toolshed. Contrast with +a [group handle](#group-handle), which is the same shape but prefixed with `#`. Paired with its +[keypair](#keypair-private-key-public-key), the two together are called an [identity](#identity). +*See: [federation.md](federation.md#users-and-groups)* + +## Classification: Tags, Properties & Categories + +**Alias** (Proposed) +An explicit, one-directional "supersedes"/"alias of" pointer from one [classification +handle](#classification-handle) to another, asserting they mean the same thing. Never inferred +automatically, always a deliberate act, in keeping with [strict resolution](#strict-resolution). For +a property alias specifically, also states a unit conversion (even if it's "identical unit, factor +1"); without one, the values behind the two handles aren't assumed to be comparable. +*See: [tags.md](design-in-progress/tags.md#open-design-ideas)* + +**Classification handle** (Implemented) +Umbrella term for a [tag](#tag-property-category), [property](#tag-property-category), or +[category](#tag-property-category) handle, of the form `origin#type:name`, e.g. +`git:base#property:length`. Distinct from a [user handle](#user-handle) or [item +handle](#item-handle): it names a reusable classification concept, not an actor or an owned thing. +*See: [federation.md](federation.md#tags-properties-and-categories)* + +**Definition fingerprint** (Partially implemented) +A hash of a classification entry's full definition, used to tell whether two [actors](#actor) +that both use the same handle actually mean the same thing. A sha256 of each `shared_data/*.json` +file is already computed and stored per import (`ImportedIdentifierSets.hash`), but it's only ever +recorded, not compared, so it doesn't yet catch a [handle collision](#handle-collision) in +practice. +*See: [tags.md](design-in-progress/tags.md#open-design-ideas)* + +**Fragmentation** (Known problem) +Two different [classification handles](#classification-handle) that mean, or were intended to +mean, the same real-world concept (different origins, or a locally-invented tag versus a shared +one). The opposite failure from a [handle collision](#handle-collision): visible and merely +wasteful, rather than silent and dangerous. +*See: [tags.md](design-in-progress/tags.md#problem)* + +**Handle collision** (Known problem) +Two [actors](#actor) independently ending up with the exact same [classification +handle](#classification-handle) string backing two different definitions, e.g. two servers that +each imported `git:ee2` from what has since become diverging branches. The dangerous mirror image +of [fragmentation](#fragmentation): silent, because nothing about the interaction signals that +anything's wrong, both sides just say the same string. +*See: [tags.md](design-in-progress/tags.md#worked-example-the-same-handle-meaning-two-different-things)* + +**Origin** (Implemented) +The first component of a [classification handle](#classification-handle), naming where that tag, +property, or category came from, e.g. `git:base` in `git:base#property:length`. Not necessarily a +server or a domain, it can equally be a shared reference dataset (like the files in +`backend/shared_data/`) or a server's own locally-invented taxonomy. Not the same thing as a +[domain](#domain), see the note at the top of this page. +*See: [federation.md](federation.md#tags-properties-and-categories)* + +**Tag / Property / Category** (Implemented) +The three kinds of classification entity an item can reference, each identified by a +[classification handle](#classification-handle). A property additionally carries unit metadata +(`unit_symbol`/`unit_name`), though property *values* on an item are plain, undeclared-type +strings today. +*See: [federation.md](federation.md#tags-properties-and-categories), [tags.md](design-in-progress/tags.md)* + +## Items & Physical Labels + +**Item Handle** (Proposed) +The compact identifier for a specific item that's meaningful outside its owner's own account: +`user@domain.tld:id`, the owner's [user handle](#user-handle) plus a [local id](#local-id). Used +where it's already clear from context that it's a Toolshed item, e.g. inside the app, in exports, +in logs, so it doesn't need to spell that out or be openable on its own. Contrast with [Item +URL](#item-url), the self-contained form for when no such context can be assumed. +*See: [items-labels.md](design-in-progress/items-labels.md#open-design-questions)* + +**Item Label** (Proposed) +A physical, scannable encoding (QR code, barcode, or similar) of an item's [Item URL](#item-url), +meant to be printed and stuck on the physical object it refers to. +*See: [items-labels.md](design-in-progress/items-labels.md#goals)* + +**Item URL** (Proposed) +The self-contained URL form of an [Item Handle](#item-handle), for use with no context at all, e.g. +an [Item Label](#item-label): `https:///i/user@domain.tld/id`. Has to open directly +to the right frontend and land on the right item on its own, since the reader can't be assumed to +already know what it is or which server it belongs to. Any frontend can serve this URL, the host +named in it isn't part of the item's identity, only the handle in its path is. +*See: [items-labels.md](design-in-progress/items-labels.md#open-design-questions)* + +**Local id** (Implemented) +An item's identifier as it exists today: unique only within its owner's own inventory, not +meaningful outside that owner's account. The starting point both the [Item +Handle](#item-handle) and [Item URL](#item-url) build on. +*See: [federation.md](federation.md#items), [items-labels.md](design-in-progress/items-labels.md#problem)* diff --git a/docs/handles-and-shortids.md b/docs/handles-and-shortids.md new file mode 100644 index 0000000..324f8c0 --- /dev/null +++ b/docs/handles-and-shortids.md @@ -0,0 +1,168 @@ +# Handles and Short IDs + +This is the syntax-level reference for two related but separate naming schemes. [federation.md](federation.md)'s +"Unique Handles" section covers *why* Toolshed hands out handles at all and what each kind (user, +group, tag/property/category) means conceptually; this document covers the parsing rules those +handles have to follow once they're written down or embedded somewhere - legal characters and +escaping. It also covers short ids end to end: a separate, newer scheme for packing small integer +id chains into a compact token, implemented in `frontend/src/short-id.js`. + +## Handle syntax + +### Reserved characters + +A username ends up embedded, unescaped, in several composite formats beyond its own handle, so it +can't contain any character that already means something else in one of those: `@` (the +user/domain separator in a user handle), `#` (the group-handle prefix, and the origin/type +separator in a classification handle, see federation.md's Tags, Properties, and Categories +section), `:` (the id delimiter in a proposed Item Handle, the type/name delimiter in a +classification handle, and the delimiter in a signed request's `Authorization` header), `+` +(reserved as the URL-embedding escape for `#`, see below), `~` (the short-id token prefix, see +Short IDs below, and a proposed collision-disambiguation suffix delimiter on a tag's origin), and +`/` (the path-segment delimiter every handle and id ultimately sits next to once embedded in a +URL). This has to be enforced by an explicit validator rather than left to Django's default +`UnicodeUsernameValidator` (`^[\w.@+-]+\Z`), which currently permits both `@` and `+` (its own +regex doesn't happen to allow `#`, `:`, or `/`, but that's incidental, not a designed restriction). + +### Embedding a `#`-bearing handle in a URL + +A literal `#` can't appear unescaped in a URL path segment: per RFC 3986, `#` starts the URI's +fragment component, so any URL-parsing client (a browser, a QR scanner, a link preview) treats +everything from the first unescaped `#` onward as a fragment and never sends it to the server at +all, before a request is even made, not merely a server-side quirk to work around. The usual fix is +to percent-encode it (`%23`), but that's exactly the encode/decode step a self-contained item URL +(a physical label, a link shared outside the app) is designed to avoid for anything that sits +directly in a path segment (`@` needs no such treatment). Instead, whenever a handle containing a +`#` (a group handle, or a tag/property/category +handle) has to appear as a raw URL path segment, substitute `+` for `#` in that rendering only: +`#groupname@domain` becomes `+groupname@domain` in a URL, and `origin#type:name` becomes +`origin+type:name`. This is a URL-embedding convention, not a second handle format: the canonical +handle, the one used in the API, in signed requests, in the database, and everywhere else a handle +is written or displayed, is unchanged and is still written `#groupname@domain`. Reversing the +substitution when parsing a path segment back into a handle is unambiguous only because `+` is +otherwise forbidden in every field a handle is built from (see Reserved characters above); if a +group name or tag name could itself contain a literal `+`, it would be indistinguishable from an +escaped `#` once decoded. + +Implemented in `frontend/src/handle-url.js` (`encodeHandleForUrl`/`decodeHandleFromUrl`). + +## Short IDs + +A general encoding for turning a small, fixed-shape list of integers into a compact, URL-safe +token, with no server-side lookup table involved: the code *is* the data, nothing is stored +server-side to make it resolvable. Originally proposed to answer items-labels.md's open "what does +the handle/URL actually look like" question, but the encoding itself isn't item-specific; anything +currently addressed by a short chain of small integers is a candidate. Implemented and tested in +`frontend/src/short-id.js`; try it live at `/~` (`frontend/src/views/ShortId.vue`), which +decodes whatever token is in the URL and also lists worked examples for every registered kind. + +### Shape: a kind tag, then a fixed list of integers + +Every short id starts with a small, fixed-width **kind** tag saying which schema the rest of the +bits should be read against, followed by exactly the integer fields that kind's schema calls for, +in a fixed order. `kind` is a small, closed, slow-growing set, so it doesn't need to be +self-delimiting the way the integer fields do: 2 bits directly name kinds 0-2, and the all-ones +value (3) is an escape meaning "the real kind follows as the next field, offset by this direct +range" - so kind 3 is encoded as escape + chunked-int `0`, kind 4 as escape + `1`, and so on. This +costs nothing for a kind that already fits in the direct range, and keeps the tag itself extensible +forever without ever having to widen it out from under codes that were already printed. A narrow +tag only pays off if kind usage is actually skewed the way id values are (a few kinds dominate), +which is why the registry below is ordered by expected frequency, cheapest (most-used) kind first: + +| kind | name | fields | notes | +|---|---|---|---| +| 0 | `item` | `owner_identity_id`, `item_local_id` | dominant case - the primary physical-label use case | +| 1 | `storage_location` | `owner_identity_id`, `storage_location_id` | also label-printed | +| 2 | `category` | `category_id` | label-adjacent (tagging); global, no owner | +| 3 | `workflow` | `owner_identity_id`, `workflow_id` | shared in-app, not printed - first to pay the escape's cost | +| 4 | `group` | `group_id` | shared even less often; global, no owner | +| 5 | `file` | `file_id` | least often shared standalone; global, deduplicated by content hash | + +`owner_identity_id` is `KnownIdentity.pk` (`backend/authentication/models.py`), not +`ToolshedUser.pk`. Every local account already has exactly one stable `KnownIdentity` row +(`ToolshedUser.public_identity`, created once at registration and never recreated), and every +friend this backend knows about - local or remote - is represented by that same table, unique on +`(username, domain)`. So one small integer already stands in for "this owner, as known by this +backend" for both cases, with no separate local-vs-remote branching needed, and it's the same row +federation.md's Cryptography section already treats as the trust anchor for a handle's public key. +It appears on `item`, `storage_location`, and `workflow` because their backing models +(`InventoryItem`, `StorageLocation`, `WorkflowInstance`) all FK `ToolshedUser` directly; `category`, +`group`, and `file` skip it because their models are global/unscoped (`Group` has an unowned +`members` M2M, `File` is deduplicated globally by content hash), so a bare row id is already +everything needed to look them up. + +A short id is inherently scoped to the backend that minted it (an "owner" field is a row that only +exists in, and only means anything to, that one backend's database), not a portable replacement for +a `user@domain.tld` handle, which stays the form to use anywhere cross-domain resolution actually +matters. Resolving a short id still goes through the same friend/signature checks as everything +else, unchanged; nothing about how the code looks grants any authority of its own (see Guessability +below). + +### Packing one integer: dynamic bit depth + +Each integer field is made self-delimiting with **continuation chunking** (UTF-8/LEB128-style): +split the value into fixed-size chunks (4 data bits each, most-significant chunk first), each +preceded by one continuation bit meaning "another chunk follows" (`1`) or "this is the last chunk" +(`0`). A value like `42` (`0b101010`) needs two 4-bit chunks, costing 10 bits total (2 × (1 +continuation + 4 data)); `7` fits in one chunk, costing 5 bits. This was chosen over an +Elias-gamma-style unary/delimiter scheme (encode the value's bit-length in unary, then that many +literal bits): unary is cheaper for single-digit values but its prefix grows every time the value's +bit-length grows, so it never wins once ids pass single digits, which is the common case here (auto +increment database ids realistically sitting in the tens through low-hundred-thousands over an +installation's life). A 4-bit chunk width is a reasonable fixed default across that whole range; +per-field tuning was checked against both a uniform and a skewed (geometric) distribution and never +won by more than a fraction of a character, not enough to justify a tuning knob. + +### From bits to text: base64 without the byte layover + +Standard base64 assumes byte-aligned (8-bit) input, grouping 3 bytes into 4 output characters and +padding to a byte boundary before encoding. Since there's no byte layer here to begin with, the +bit-packed stream is instead packed directly into 6-bit groups and mapped straight onto the +URL-safe base64 alphabet (RFC 4648 §5: `-` and `_` in place of `+` and `/`), with the final +character's unused low bits padded with zeros. That padding is safe by construction: the decoder +always knows exactly how many integers a given kind calls for, and a chunk's continuation bit is +`1 = more follows`, so a run of zero-padding at the very end can never be misread as "one more +chunk" - it decodes as a terminated chunk, at which point every field the schema called for has +already been produced and decoding simply stops. No `=` padding characters are needed either; those +exist in classic base64 purely to communicate trailing-byte padding, and there is no byte layer +here to need that. + +### The leading `~` + +Every token is prefixed with a literal `~`, so a short id in a URL looks like `~DyU`. Its only job +is to mark "everything after me decodes as one of these": URL-safe base64 never produces a `~` +itself, so the prefix can never be confused with the payload, and none of Toolshed's other +path-segment formats (bare usernames, `user@domain` handles, slugs, plain numeric ids) start with +`~` either. `~` is one of RFC 3986's `unreserved` characters (§2.3, the same class as letters, +digits, `-`, `.`, and `_`), a stronger guarantee than merely being legal in a path segment: it's +never a target for percent-encoding and never carries special meaning in any URI component, so a +short id can be handed to any part of the stack without first checking which encoding rules apply +there. + + +### Worked examples + +Encoding `kind = item` (0), `owner_identity_id = 7`, `item_local_id = 42`: + +- `kind`: 2 fixed bits → `00` +- `owner_identity_id = 7`: fits in one 4-bit chunk → 5 bits (`00111`) +- `item_local_id = 42`: needs two 4-bit chunks → 10 bits (`1001001010`) + +Total: 17 meaningful bits, padded to the next multiple of 6 (18) with one zero bit, yielding 3 +base64 characters: **`~DyU`**. + +The same worked-out form for one example of every registered kind - `Bits` is the same +space-separated segmentation (kind tag, escape offset if present, each field, then padding) the +Examples table on `/~` (`frontend/src/views/ShortId.vue`) shows for every registered kind: + +| Kind | Fields | Serialized | Bits | Token | +|---|---|---|---|---| +| `item` | `owner_identity_id: 7`, `item_local_id: 42` | `[0, 7, 42]` | `00 00111 1001001010 0` | `~DyU` | +| `storage_location` | `owner_identity_id: 3`, `storage_location_id: 1000` | `[1, 3, 1000]` | `01 00011 100111111001000 00` | `~Rz8g` | +| `category` | `category_id: 5` | `[2, 5]` | `10 00101 00000` | `~ig` | +| `workflow` | `owner_identity_id: 2`, `workflow_id: 9` | `[3, 2, 9]` | `11 00000 00010 01001 0` | `~wCS` | +| `group` | `group_id: 11` | `[4, 11]` | `11 00001 01011` | `~wr` | +| `file` | `file_id: 123` | `[5, 123]` | `11 00010 1011101011 0` | `~xXW` | + +`workflow`, `group`, and `file` are kinds 3-5, so their `Bits` column shows the escape tag (`11`) +followed by its own offset segment payload fields. \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index 95588dd..756dc7e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -9,5 +9,6 @@ This is the documentation for the Toolshed project. It is a work in progress. - [Deploying Toolshed](deployment.md) - [Development Setup](development.md) - [About Federation](federation.md) + - [Handles and Short IDs](handles-and-shortids.md)