# Tags, Properties & Categories (Design in Progress)

Status: partially implemented (the handle scheme and strict resolution described in federation.md
exist today); this document is about the rough edges and open questions in that design, not a
proposal to build it from scratch.

## Problem

Tags, properties, and categories are handles of the form `origin#type:name` (see federation.md's
"Unique Handles" section), and a reference to a handle a server doesn't recognize is left
unresolved rather than guessed at or merged. That strictness is deliberate and avoids silent data
corruption, but it pushes a real cost onto users and creates some open problems:

- **Discoverability.** There's no way today to search or browse "what origins/taxonomies already
  exist" before creating a new tag. In practice this likely means people invent their own local
  tags for things a shared, canonical origin already covers, simply because they didn't know it
  existed.
- **Fragmentation.** Because uniqueness is scoped per origin, nothing stops semantically identical
  concepts from existing under different names or different origins at once (`drill` vs.
  `power-drill`, one server's `origin:mytools` vs. another's `origin:community-tools`). Nothing
  reconciles these; they just coexist.
- **No aliasing/synonyms.** If a user starts out with a local tag and later wants to adopt a shared
  origin's equivalent, there's no supported way to merge or alias the two; existing items keep
  pointing at the old handle.
- **Trust and quality of shared origins.** An origin is "whatever the classification is considered
  to have come from," which could be an imported reference dataset. Nothing in the current design
  addresses who maintains such a dataset, how it's kept correct, or what happens when it changes
  (a category gets renamed or split upstream, e.g.).
- **Localization.** A tag/category name is a single string. There's no notion of the same concept
  having a different display name per language while still resolving to the same handle.
- **Property values aren't typed.** A property's value on an item is a plain string. A property
  definition can carry a unit (`unit_symbol`/`unit_name`), but nothing declares that its values are
  actually numbers meant to be read in that unit, versus, say, free text that happens to mention a
  unit. This is already a problem for one server in isolation (see the worked example below), and
  federation doesn't create it, it just means it now has to be solved consistently across origins
  instead of once.
- **Handle collisions.** `origin` is a free-text string with nothing enforcing that it's actually
  unique to one definition. Fragmentation (above) is two different strings for the same thing;
  this is the more dangerous mirror image, the same handle string ending up attached to two
  different definitions, which is exactly the scenario the strict-resolution design was supposed
  to make impossible (see the second worked example below).

### Worked example: filtering by a numeric range across origins

Say two independently-run servers both end up with a property called "voltage," each under its own
origin: `git:base#property:voltage` and `some-other-origin#property:voltage`. A user who's friends
with people on both wants to filter their combined view of shared items by, say, `0 < voltage <
4`. Two separate problems stack up:

1. **Are these the same property?** Per the fragmentation problem above, the two handles are, and
   stay, unrelated as far as the system is concerned, same name, different origin, no connection.
   A range filter built against one handle simply won't match items tagged with the other, even
   though a person looking at both would probably call them "the same thing."
2. **Even if they were recognized as the same thing, are the values comparable?** A range filter
   needs actual numbers in a known unit. If one server's items store `"3.7"` and the other's store
   `"3700"` (volts vs. millivolts), a numeric comparison across the two silently produces nonsense
   unless the unit is known and converted. If either side stores the value as loosely-formatted
   text (`"3.7V"`, `"~3.7"`) rather than a bare number, it may not be reliably parseable as a
   number at all.

So a cross-origin range filter needs both an aliasing/equivalence answer (are `voltage` and
`voltage` the same concept) and a units/typing answer (are their values actually numbers, in units
that convert cleanly into each other), and the first being solved doesn't imply the second is.

### Worked example: the same handle meaning two different things

Say `git:ee2` names a specific reference dataset that started life in one shared git history, and
two servers each imported it, at different times, from what has since become two diverging
branches (or forks) of that history. Both servers now have a property whose handle is the exact
same string, `git:ee2#property:charging_voltage`, but whose actual definition, say, unit, or
dimensions has since diverged between the two branches. Neither server did anything wrong; each
one faithfully imported "`git:ee2`" as it existed at the time.

This is a materially worse problem than the fragmentation/voltage example above. Fragmentation is
a missed opportunity, two things that should be linked aren't, and the failure is visible (the
filter just doesn't match as much as a person would expect). A handle collision is silent: nothing
about the two servers exchanging data suggests anything is wrong, both sides say
`git:ee2#property:charging_voltage`, so anything that trusts equal-handle-means-equal-definition
(exactly what the strict-resolution design promises, and exactly what a filter, an alias, or a
plain item-detail display would rely on) can silently combine or display incompatible values as if
they were the same thing. This is the specific failure mode the whole handle design exists to
prevent, so a scheme where it can still happen is a real gap, not just an inconvenience.

The root cause is that `origin` is a free-text label describing where something came from, not an
identifier that's actually bound to a specific, fixed piece of content. A name like `git:ee2` reads
as if it points to something immutable, but nothing about the origin field enforces that, `ee2`
could easily be a branch or tag name rather than a specific commit, i.e. a pointer that can keep
moving, and two importers pinned to it at different times without ever taking on a different
handle to show for it.

This isn't hypothetical, it's exactly how the reference data included in this repository already
works. The files in `backend/shared_data/` (`base.json`, `ee.json`, etc.) are origin datasets:
`configure.py`'s import step sets `origin = "git:" + filename` for everything a file defines
(`configure.py:119`), and a file can declare a fixed dependency on another one by name, e.g.
`ee.json`'s `"depends": ["git:base"]`. The intent is clearly that these files are immutable once
committed, and that a `depends` entry is a pin to a specific, settled parent, not a moving target,
but nothing in the code enforces that today, it's a convention people are expected to follow.

There's already a piece of the machinery needed to enforce it, though: import already computes a
sha256 of each file's raw content and stores it (`configure.py:125`, saved onto
`ImportedIdentifierSets.hash`, which is `unique=True` alongside `name`, `hostadmin/models.py:15`).
It's just not used for the thing it would be useful for, the "already imported, skipping" check
(`configure.py:129-130`) matches on `name` alone, it never recomputes the hash of the file being
imported and compares it against the hash already on record for that name. So even on a single
server, an edited `ee.json` re-imported under its old filename wouldn't be noticed as a change,
let alone flagged as a conflict, and the hash never leaves that server's own bookkeeping to be
compared against what a friend server has on record for the same name.

## Goals

- Make it easy to find and reuse an existing origin/handle before inventing a new local one, to
  reduce fragmentation without weakening the strict-resolution guarantee that already exists.
- Give users a path to move a locally-invented tag onto a shared origin later, without losing or
  having to manually re-tag their existing items.
- Keep the core guarantee intact: a handle always means one specific, traceable thing, nothing
  should be implicitly merged or reinterpreted across origins.
- Make that guarantee actually hold, not just assumed: two servers that both use a given handle
  should either really mean the same definition, or have some way to find out they don't, rather
  than the collision staying silent.

## Non-goals (for now)

- Building a moderation/governance system for shared origins. Worth thinking about, but a bigger
  problem than this document is trying to scope.
- Free-text/fuzzy tag matching in search. Search UX can layer on top of resolved handles without
  changing what a handle means.

## Open design ideas

**A small set of well-known, shipped origins.**
Toolshed could ship with one or a few canonical origins covering common tool/inventory categories
out of the box, so that a fresh server already has a sensible baseline vocabulary to reuse instead
of every server reinventing "drill," "screwdriver," "power tools," etc. independently. Doesn't
solve fragmentation for everything, but raises the floor.

**Explicit aliasing rather than merging.**
Rather than trying to detect and merge "equivalent" tags automatically (risky, exactly the kind of
implicit behavior the strict-resolution design intentionally avoids), a tag could carry an explicit,
user-initiated "supersedes"/"alias of" pointer to another handle. Items already tagged with the old
handle could then be offered a one-time, explicit re-tag rather than a silent change of meaning.

For properties specifically, an alias needs to claim more than "these mean the same thing," it
needs to claim the values are comparable, which means recording a unit conversion (possibly just
"identical unit, factor 1") alongside the alias, not just a bare pointer. An alias with no stated
conversion should probably be treated as "same concept, values not (yet) comparable," a range
filter has no business guessing a conversion on its own.

**Filters default to per-handle, and only widen on an explicit alias.**
Following directly from the strict-resolution philosophy in federation.md: a range filter should
only ever combine two distinct property handles into one filterable facet because of an explicit
alias (see above) that also states the values are comparable, never because their names or units
happen to match. Absent that, two same-named properties from different origins should just show up
as two separate filters, visibly distinct, rather than the UI silently guessing they're the same
and producing a filter result that mixes incomparable values.

**Typed property values.**
Giving a property definition a declared value type (number, text, boolean, ...) in addition to its
existing unit metadata would let both the frontend and the alias/conversion mechanism above know
whether "range filter" even applies to a given property, and would close the gap where a value
happens to look numeric but isn't guaranteed to parse as one. This is useful even without
federation in the picture, cross-origin comparison just makes the gap load-bearing instead of
cosmetic.

**Origin metadata/versioning.**
If an origin represents an imported dataset, giving it its own version or changelog would let a
server know when the upstream taxonomy it imported has moved on, and decide explicitly whether to
re-import, rather than silently drifting from what other servers using the "same" origin now have.
This only helps if everyone's still on one shared timeline, though, it doesn't by itself address
diverging forks/branches ending up with the same name (see below).

**Pin origins to immutable content, not movable names.**
The `git:ee2` collision happens because the origin string names something mutable (a branch/tag)
rather than something fixed. If an origin string were derived from the content itself, e.g. a hash
of the definition, or a specific immutable commit rather than a branch, two independent imports
could never end up with the same string unless the content was actually identical at that point,
collisions would become structurally impossible rather than just unlikely. This is a bigger change
than the versioning idea above: it's not tracking change over time, it's making the identifier
itself incapable of silently referring to different things.

A concrete version of this: use the git blob hash of the file a property/tag/category was defined
in as (part of) its origin. If the reference dataset already lives in a git repo, this is free,
git's already computed it, and it's independently checkable, any server holding or able to fetch
the same repo can recompute the hash from the content and confirm for itself, rather than trusting
a label. This is a real improvement over free-text `origin` strings, but it isn't a free lunch:

- *Granularity.* A blob hash identifies a whole file, not a single property. If a file defines
  several properties together, editing any one of them changes every other property's "identity"
  in the same file too, even though nothing about them changed. Either definitions need to be
  one-per-file for the hash to mean what's intended, or the hash needs to cover just the relevant
  entry rather than the literal git blob.
- *It converts every edit into a fork.* Since any change, including a typo fix, changes the hash,
  a routine upstream correction mechanically fragments what's still the same property into two
  handles. That's consistent with "never silently reinterpret a handle," but it means the aliasing
  mechanism above stops being a nice-to-have and becomes the primary upgrade path, every legitimate
  edit needs an explicit "supersedes" link, or old items are stranded on a stale, now-orphaned hash.
- *A hash alone is an identity, not a location or a label.* It proves two things are the same (or
  aren't), but doesn't say where to fetch the content from if you don't already have it, and isn't
  human-readable. Pairing it with a location (which repo) and a mnemonic (which release/name it
  corresponds to) alongside the hash keeps the discoverability goal intact instead of trading it
  away for collision-proofing.

**Detect collisions on contact, as a backstop.**
Even with better-behaved identifiers going forward, existing data and human-typed origin strings
mean collisions can't be ruled out entirely. Whenever two servers interact over a handle they both
claim to know (e.g. as part of resolving an alias, or federated search), comparing a fingerprint of
the full definition, not just the handle string, would let a mismatch surface as an explicit
conflict to resolve, rather than being silently trusted. This is the same instinct as the "explicit
alias must state whether things are comparable" idea above, applied in the opposite direction, here
the handles already match and the system needs to actively check whether that trust is warranted.

The needed ingredient already exists locally and just isn't being used this way: the sha256 hash
already computed and stored per import (see above) is exactly the kind of definition fingerprint
this needs. Two changes would make it actually do the job: first, comparing it on every import
(including a "re-import" of a name already on record), not just recording it once, so a locally
edited file gets caught before it's ever presented to anyone else, and second, exchanging it as
part of whatever federated interaction references a shared-origin handle, so two servers can
compare hashes for the same name and find out they've diverged instead of assuming they haven't.

**What this means for the handle actually on the wire.**
Putting the above together, the everyday handle shouldn't change shape at all. It stays
`origin#type:name`, e.g. `git:ee2#property:charging_voltage`, exactly as it is today. The reason is
redundancy: this string is what appears on every single reference (every item's tag list, every
property assignment), potentially many times per item across many items, while a hash only ever
needs to be known once per origin. Carrying a full hash on every occurrence would repeat the same
value over and over for no benefit beyond what knowing it once already provides.

Instead, the hash stays where it already lives, attached to the origin as a whole (extending
`ImportedIdentifierSets`, see above), and gets exchanged at the points where two servers actually
need to agree on one, e.g. the first time a friend's item references an origin a server doesn't
already have a hash on record for. First contact just records it, same as resolving any unfamiliar
handle today; a later mismatch against what's on record is the collision, and that's the point
where it needs to become visible rather than silently trusted.

Only once a collision has actually been found does the wire format need to say more than
`origin#type:name`, because at that point there genuinely are two different things sharing a name
and something has to distinguish them for a person sorting it out. A short, abbreviated hash
appended to the origin, the same idea git itself relies on for short commit hashes, keeps that
escape hatch usable: `git:ee2~0f3a9c1e#property:charging_voltage` versus
`git:ee2~7bc82a04#property:charging_voltage`. This longer form is exception-path plumbing for
resolving an already-detected conflict, not something that changes the size or shape of handles in
the common case.

**Search across known origins.**
Before creating a new tag/category, a creation flow could search across origins the local server
already knows about (its own, plus any it's imported) and surface likely existing matches. This is
a UX/workflow fix rather than a change to the handle model itself, it doesn't need to touch
resolution semantics at all.

**Display name vs. handle.**
Separating "the name that appears in the handle" (stable, part of the identity) from "the label
shown to a user" (translatable, cosmetic) would allow localization without affecting resolution or
uniqueness, since resolution would stay keyed on the handle, not the display string.
