j3d1/toolshed
2
1
Fork 0
Code Issues 9 Pull requests 1 Projects Releases Packages Wiki Activity

Compare commits

base: j3d1:snapshot/alpha-2026-9
Branches Tags
j3d1:stable
j3d1:jedi/proto/frontend
j3d1:snapshot/alpha-2026-9
j3d1:jedi/dev/frontend
j3d1:jedi/dev/shared_data
j3d1:jedi/cli-client
j3d1:jedi/better-docs
j3d1:jedi/dev/docker
j3d1:jedi/proto/backend
j3d1:jedi/dev/backend
j3d1:busti/proto/frontend
j3d1:jedi/fast-ci
..
compare: j3d1:stable
Branches Tags
j3d1:jedi/proto/frontend
j3d1:snapshot/alpha-2026-9
j3d1:stable
j3d1:jedi/dev/frontend
j3d1:jedi/dev/shared_data
j3d1:jedi/cli-client
j3d1:jedi/better-docs
j3d1:jedi/dev/docker
j3d1:jedi/proto/backend
j3d1:jedi/dev/backend
j3d1:busti/proto/frontend
j3d1:jedi/fast-ci

No commits in common. "snapshot/alpha-2026-9" and "stable" have entirely different histories.
snapshot/a ... stable

241 changed files with 2430 additions and 85567 deletions
Download patch file Download diff file Expand all files Collapse all files

5
README.md
View file

@ -1,8 +1,5 @@
# toolshed # toolshed
## foo
## Development ## Development
``` bash ``` bash
@ -90,4 +87,4 @@ for detailed instructions see [docs](/docs/deployment.md).
``` bash ``` bash
cli-client/toolshed-client.py --key <hex private key> --user name@example.com --host 1.2.3.4:8000 getinventory cli-client/toolshed-client.py --key <hex private key> --user name@example.com --host 1.2.3.4:8000 getinventory
``` ```

4
backend/.env.dist
View file

@ -1,4 +1,2 @@
ALLOWED_HOSTS="localhost,127.0.0.1" ALLOWED_HOSTS="localhost,127.0.0.1"
SERVE_X_ACCEL_REDIRECT=True

13
backend/Dockerfile Normal file
View file

@ -0,0 +1,13 @@
FROM python:alpine
WORKDIR /app
RUN apk add --no-cache gcc musl-dev python3-dev
COPY requirements.txt /app
RUN pip install --upgrade pip && pip install -r requirements.txt
COPY . /app
RUN python configure.py
RUN python manage.py collectstatic --noinput
CMD python manage.py runserver 0.0.0.0:8000 --insecure
# TODO serve static files with nginx and remove --insecure
EXPOSE 8000

44
backend/authentication/admin.py
View file

@ -1,14 +1,6 @@
from django.contrib import admin from django.contrib import admin
from django.contrib.auth.models import Group as DjangoPermissionGroup
from authentication.models import ToolshedUser, KnownIdentity, FriendRequestOutgoing, FriendRequestIncoming, \ from authentication.models import ToolshedUser, KnownIdentity, FriendRequestOutgoing, FriendRequestIncoming
AccountPreference, Group, GroupInvite, GroupInviteIncoming, GroupMembership
# django.contrib.auth's own permission-groups model is unused (this app authenticates via
# signature_auth, not Django's built-in permission system) and otherwise collides in the admin
# sidebar with our own Group (the actor-type group), which is registered below under the same
# "Groups" label.
admin.site.unregister(DjangoPermissionGroup)
class ToolshedUserAdmin(admin.ModelAdmin): class ToolshedUserAdmin(admin.ModelAdmin):
@ -16,11 +8,6 @@ class ToolshedUserAdmin(admin.ModelAdmin):
search_fields = ('username', 'email', 'first_name', 'last_name', 'is_staff', 'is_active', 'date_joined', 'domain') search_fields = ('username', 'email', 'first_name', 'last_name', 'is_staff', 'is_active', 'date_joined', 'domain')
class AccountPreferenceAdmin(admin.ModelAdmin):
list_display = ('user', 'key', 'value')
search_fields = ('user__username', 'key')
class KnownIdentityAdmin(admin.ModelAdmin): class KnownIdentityAdmin(admin.ModelAdmin):
list_display = ('username', 'domain', 'public_key') list_display = ('username', 'domain', 'public_key')
search_fields = ('username', 'domain', 'public_key') search_fields = ('username', 'domain', 'public_key')
@ -36,36 +23,7 @@ class FriendRequestIncomingAdmin(admin.ModelAdmin):
search_fields = ('secret', 'befriender_username', 'befriender_domain', 'befriendee_user', 'befriender_public_key') search_fields = ('secret', 'befriender_username', 'befriender_domain', 'befriendee_user', 'befriender_public_key')
class GroupAdmin(admin.ModelAdmin):
list_display = ('name', 'domain', 'get_members')
search_fields = ('name', 'domain', 'members__username')
@admin.display(description='Members')
def get_members(self, obj):
return ', '.join(str(member) for member in obj.members.all())
class GroupInviteAdmin(admin.ModelAdmin):
list_display = ('secret', 'group', 'invitee_username', 'invitee_domain')
search_fields = ('secret', 'group__name', 'invitee_username', 'invitee_domain')
class GroupInviteIncomingAdmin(admin.ModelAdmin):
list_display = ('secret', 'group_name', 'group_domain', 'inviter_username', 'inviter_domain', 'invitee_user')
search_fields = ('secret', 'group_name', 'group_domain', 'inviter_username', 'inviter_domain', 'invitee_user')
class GroupMembershipAdmin(admin.ModelAdmin):
list_display = ('user', 'group_name', 'group_domain', 'created_at')
search_fields = ('user__username', 'group_name', 'group_domain')
admin.site.register(ToolshedUser, ToolshedUserAdmin) admin.site.register(ToolshedUser, ToolshedUserAdmin)
admin.site.register(KnownIdentity, KnownIdentityAdmin) admin.site.register(KnownIdentity, KnownIdentityAdmin)
admin.site.register(FriendRequestOutgoing, FriendRequestOutgoingAdmin) admin.site.register(FriendRequestOutgoing, FriendRequestOutgoingAdmin)
admin.site.register(FriendRequestIncoming, FriendRequestIncomingAdmin) admin.site.register(FriendRequestIncoming, FriendRequestIncomingAdmin)
admin.site.register(AccountPreference, AccountPreferenceAdmin)
admin.site.register(Group, GroupAdmin)
admin.site.register(GroupInvite, GroupInviteAdmin)
admin.site.register(GroupInviteIncoming, GroupInviteIncomingAdmin)
admin.site.register(GroupMembership, GroupMembershipAdmin)

96
backend/authentication/api.py
View file

@ -9,11 +9,8 @@ from rest_framework.authtoken.models import Token
from rest_framework.authtoken.views import ObtainAuthToken from rest_framework.authtoken.views import ObtainAuthToken
from rest_framework.response import Response from rest_framework.response import Response
from authentication.models import ToolshedUser, AccountPreference from authentication.models import ToolshedUser
from authentication.signature_auth import SignatureAuthenticationLocal, SignatureAuthentication, \ from authentication.signature_auth import SignatureAuthenticationLocal
split_userhandle_or_throw
from files.models import File
from files.serializers import FileSerializer
from hostadmin.models import Domain from hostadmin.models import Domain
router = routers.SimpleRouter() router = routers.SimpleRouter()
@ -56,71 +53,15 @@ class UserViewSet(viewsets.ModelViewSet):
permission_classes = [IsAuthenticated, IsAdminUser] permission_classes = [IsAuthenticated, IsAdminUser]
@api_view(['GET', 'PATCH']) @api_view(['GET'])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthenticationLocal]) @authentication_classes([SignatureAuthenticationLocal])
def getUserInfo(request): def getUserInfo(request):
"""Get or update the authenticated local user's own account info; only the account owner may
call this on their own home server (see getUserProfile for viewing a friend's public profile)."""
user = request.user user = request.user
if request.method == 'PATCH':
old_file = user.profile_picture
if 'profile_picture' in request.data:
profile_picture = request.data.get('profile_picture')
if profile_picture is None:
user.profile_picture = None
elif type(profile_picture) == dict:
serializer = FileSerializer(data=profile_picture)
if not serializer.is_valid():
return Response(serializer.errors, status=400)
user.profile_picture = serializer.save()
else:
return Response({'profile_picture': 'Must be null or an object with data and mime_type.'}, status=400)
elif 'profile_picture_id' in request.data:
profile_picture_id = request.data.get('profile_picture_id')
if profile_picture_id is None:
user.profile_picture = None
else:
try:
user.profile_picture = File.objects.get(id=profile_picture_id)
except File.DoesNotExist:
return Response({'profile_picture_id': 'File does not exist.'}, status=400)
user.save()
if old_file and old_file != user.profile_picture and old_file.connected_items.count() == 0 \
and old_file.profile_picture_users.count() == 0 and old_file.staged_by_workflows.count() == 0:
old_file.file.delete(save=False)
old_file.delete()
return Response({ return Response({
'username': user.username, 'username': user.username,
'domain': user.domain, 'domain': user.domain,
'email': user.email, 'email': user.email
'profile_picture': FileSerializer(user.profile_picture).data if user.profile_picture else None,
})
@api_view(['GET'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication])
def getUserProfile(request, handle):
"""Get another local user's public profile by handle, e.g. so a friend can look up an avatar;
caller must be a friend of that user (or the user itself, signing with their own known
identity rather than local credentials)."""
try:
username, domain = split_userhandle_or_throw(handle)
except ValueError:
return Response(status=400)
try:
target = ToolshedUser.objects.get(username=username, domain=domain)
except ToolshedUser.DoesNotExist:
return Response(status=404)
if target not in request.user.friends_or_self():
return Response(status=403)
return Response({
'username': target.username,
'domain': target.domain,
'profile_picture': FileSerializer(target.profile_picture).data if target.profile_picture else None,
}) })
@ -158,40 +99,11 @@ def registerUser(request):
return Response({'errors': {'domain': 'Domain does not exist or is not open for registration'}}, status=400) return Response({'errors': {'domain': 'Domain does not exist or is not open for registration'}}, status=400)
@api_view(['GET', 'PUT'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthenticationLocal])
def account_preferences(request):
"""Get or bulk-upsert the authenticated user's account-level preferences: GET returns the
current preferences as {key: value}; PUT sets/overwrites one or more, leaving unspecified
keys untouched."""
if request.method == 'PUT':
if not isinstance(request.data, dict):
return Response({'detail': 'Expected an object of key/value pairs.'}, status=400)
for key, value in request.data.items():
AccountPreference.objects.update_or_create(user=request.user, key=key, defaults={'value': value})
preferences = {pref.key: pref.value for pref in request.user.preferences.all()}
return Response(preferences)
@api_view(['DELETE'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthenticationLocal])
def account_preference_detail(request, key):
"""Reset a single account-level preference back to its default by deleting it."""
AccountPreference.objects.filter(user=request.user, key=key).delete()
return Response(status=204)
router.register(r'users', UserViewSet) router.register(r'users', UserViewSet)
urlpatterns = [ urlpatterns = [
path('', include(router.urls)), path('', include(router.urls)),
path('user/', getUserInfo), path('user/', getUserInfo),
path('user/<str:handle>/', getUserProfile),
path('register/', registerUser), path('register/', registerUser),
path('token/', UserAuthToken.as_view()), path('token/', UserAuthToken.as_view()),
path('self/preferences/', account_preferences),
path('self/preferences/<str:key>/', account_preference_detail),
] ]

20
backend/authentication/migrations/0002_toolsheduser_profile_picture.py
View file

@ -1,20 +0,0 @@
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('files', '0001_initial'),
('authentication', '0001_initial'),
]
operations = [
migrations.AddField(
model_name='toolsheduser',
name='profile_picture',
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL,
related_name='profile_picture_users', to='files.file'),
),
]

26
backend/authentication/migrations/0003_accountpreference.py
View file

@ -1,26 +0,0 @@
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('authentication', '0002_toolsheduser_profile_picture'),
]
operations = [
migrations.CreateModel(
name='AccountPreference',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('key', models.CharField(max_length=255)),
('value', models.JSONField()),
('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='preferences',
to='authentication.toolsheduser')),
],
options={
'unique_together': {('user', 'key')},
},
),
]

18
backend/authentication/migrations/0004_alter_accountpreference_id.py
View file

@ -1,18 +0,0 @@
# Generated by Django 4.2.2 on 2026-08-09 13:08
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('authentication', '0003_accountpreference'),
]
operations = [
migrations.AlterField(
model_name='accountpreference',
name='id',
field=models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
),
]

56
backend/authentication/migrations/0005_group_groupinviteincoming_groupinvite_and_more.py
View file

@ -1,56 +0,0 @@
# Generated by Django 4.2.2 on 2026-08-19 13:44
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('authentication', '0004_alter_accountpreference_id'),
]
operations = [
migrations.CreateModel(
name='Group',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=255)),
('domain', models.CharField(max_length=255)),
('members', models.ManyToManyField(related_name='member_of_groups', to='authentication.knownidentity')),
],
),
migrations.CreateModel(
name='GroupInviteIncoming',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('secret', models.CharField(max_length=255)),
('group_name', models.CharField(max_length=255)),
('group_domain', models.CharField(max_length=255)),
('inviter_username', models.CharField(max_length=255)),
('inviter_domain', models.CharField(max_length=255)),
('created_at', models.DateTimeField(auto_now_add=True)),
('invitee_user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='group_invites_incoming', to=settings.AUTH_USER_MODEL)),
],
),
migrations.CreateModel(
name='GroupInvite',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('secret', models.CharField(max_length=255)),
('invitee_username', models.CharField(max_length=255)),
('invitee_domain', models.CharField(max_length=255)),
('created_at', models.DateTimeField(auto_now_add=True)),
('group', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='invites', to='authentication.group')),
],
),
migrations.AddIndex(
model_name='group',
index=models.Index(fields=['name', 'domain'], name='group_idx'),
),
migrations.AlterUniqueTogether(
name='group',
unique_together={('name', 'domain')},
),
]

28
backend/authentication/migrations/0006_groupmembership.py
View file

@ -1,28 +0,0 @@
# Generated by Django 4.2.2 on 2026-08-26 16:25
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('authentication', '0005_group_groupinviteincoming_groupinvite_and_more'),
]
operations = [
migrations.CreateModel(
name='GroupMembership',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('group_name', models.CharField(max_length=255)),
('group_domain', models.CharField(max_length=255)),
('created_at', models.DateTimeField(auto_now_add=True)),
('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='group_memberships', to=settings.AUTH_USER_MODEL)),
],
options={
'unique_together': {('user', 'group_name', 'group_domain')},
},
),
]

75
backend/authentication/models.py
View file

@ -86,8 +86,6 @@ class ToolshedUser(AbstractUser):
domain = models.CharField(max_length=255, default='localhost') domain = models.CharField(max_length=255, default='localhost')
private_key = models.CharField(max_length=255) private_key = models.CharField(max_length=255)
public_identity = models.ForeignKey(KnownIdentity, on_delete=models.CASCADE, related_name='user') public_identity = models.ForeignKey(KnownIdentity, on_delete=models.CASCADE, related_name='user')
profile_picture = models.ForeignKey('files.File', on_delete=models.SET_NULL, null=True, blank=True,
related_name='profile_picture_users')
objects = ToolshedUserManager() objects = ToolshedUserManager()
class Meta: class Meta:
@ -113,20 +111,6 @@ class ToolshedUser(AbstractUser):
return self.public_identity.public_key return self.public_identity.public_key
class AccountPreference(models.Model):
"""A single account-level (server-synced, cross-device) preference as a key/value pair;
device-level preferences are intentionally *not* stored here."""
user = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, related_name='preferences')
key = models.CharField(max_length=255)
value = models.JSONField()
class Meta:
unique_together = ('user', 'key')
def __str__(self):
return f"{self.user}: {self.key}"
class FriendRequestOutgoing(models.Model): class FriendRequestOutgoing(models.Model):
secret = models.CharField(max_length=255) secret = models.CharField(max_length=255)
befriender_user = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, related_name='friend_requests_outgoing') befriender_user = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, related_name='friend_requests_outgoing')
@ -141,62 +125,3 @@ class FriendRequestIncoming(models.Model):
befriender_public_key = models.CharField(max_length=255) befriender_public_key = models.CharField(max_length=255)
befriendee_user = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, related_name='friend_requests_incoming') befriendee_user = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, related_name='friend_requests_incoming')
created_at = models.DateTimeField(auto_now_add=True) created_at = models.DateTimeField(auto_now_add=True)
class Group(models.Model):
name = models.CharField(max_length=255)
domain = models.CharField(max_length=255)
members = models.ManyToManyField(KnownIdentity, related_name='member_of_groups')
class Meta:
unique_together = ('name', 'domain')
indexes = [
models.Index(fields=['name', 'domain'], name='group_idx'),
]
def __str__(self):
return f"#{self.name}@{self.domain}"
def is_member(self, identity):
return self.members.filter(pk=identity.pk).exists()
class GroupInvite(models.Model):
"""A pending invite tracked on the group's own home backend, checked when the invitee's accept
request arrives (mirror: GroupInviteIncoming on the invitee's backend; see
docs/design-in-progress/groups-mvp.md)."""
secret = models.CharField(max_length=255)
group = models.ForeignKey(Group, on_delete=models.CASCADE, related_name='invites')
invitee_username = models.CharField(max_length=255)
invitee_domain = models.CharField(max_length=255)
created_at = models.DateTimeField(auto_now_add=True)
class GroupInviteIncoming(models.Model):
secret = models.CharField(max_length=255)
group_name = models.CharField(max_length=255)
group_domain = models.CharField(max_length=255)
inviter_username = models.CharField(max_length=255)
inviter_domain = models.CharField(max_length=255)
invitee_user = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, related_name='group_invites_incoming')
created_at = models.DateTimeField(auto_now_add=True)
class GroupMembership(models.Model):
"""A pointer kept on a member's own home backend, recording that the local user is a member
of a group that may be hosted here or on a remote domain (mirrors FriendRequestIncoming's
role: independently recorded on the member's own side, not just the group's authoritative
backend). Written once the invitee's accept request against the group's home backend has
succeeded (see GroupInvitesIncomingAccept), analogous to how a friendship is independently
recorded on both sides via KnownIdentity.friends rather than only on one. See
docs/design-in-progress/groups-mvp.md's 'Known limitation'."""
user = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, related_name='group_memberships')
group_name = models.CharField(max_length=255)
group_domain = models.CharField(max_length=255)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
unique_together = ('user', 'group_name', 'group_domain')
def __str__(self):
return f"{self.user} MEMBER_OF #{self.group_name}@{self.group_domain}"

11
backend/authentication/serializers.py
View file

@ -1,6 +1,6 @@
from rest_framework import serializers from rest_framework import serializers
from authentication.models import ToolshedUser, Group from authentication.models import ToolshedUser
class OwnerSerializer(serializers.ReadOnlyField): class OwnerSerializer(serializers.ReadOnlyField):
@ -10,12 +10,3 @@ class OwnerSerializer(serializers.ReadOnlyField):
def to_representation(self, value): def to_representation(self, value):
return value.username + '@' + value.domain return value.username + '@' + value.domain
class GroupOwnerSerializer(serializers.ReadOnlyField):
class Meta:
model = Group
fields = ['name', 'domain']
def to_representation(self, value):
return f"#{value.name}@{value.domain}"

63
backend/authentication/signature_auth.py
View file

@ -2,7 +2,7 @@ from nacl.exceptions import BadSignatureError
from nacl.signing import VerifyKey from nacl.signing import VerifyKey
from rest_framework import authentication from rest_framework import authentication
from authentication.models import Group, KnownIdentity, ToolshedUser from authentication.models import KnownIdentity, ToolshedUser
def split_userhandle_or_throw(userhandle): def split_userhandle_or_throw(userhandle):
@ -16,27 +16,6 @@ def split_userhandle_or_throw(userhandle):
return username, domain return username, domain
def split_grouphandle_or_throw(grouphandle):
if not grouphandle.startswith('#'):
raise ValueError('Group handle must be in the format #name@domain')
return split_userhandle_or_throw(grouphandle[1:])
def resolve_owner_handle(handle):
"""Resolves a "user@domain" or "+name@domain" handle to (owner_user, owner_group), exactly one set. Raises ValueError if the handle doesn't parse at all (no '@'); returns (None, None) if it parses but names nothing that exists."""
is_group = handle.startswith('+')
name, domain = split_userhandle_or_throw(handle[1:] if is_group else handle)
if is_group:
try:
return None, Group.objects.get(name=name, domain=domain)
except Group.DoesNotExist:
return None, None
try:
return ToolshedUser.objects.get(username=name, domain=domain), None
except ToolshedUser.DoesNotExist:
return None, None
def verify_request(request, raw_request_body): def verify_request(request, raw_request_body):
authentication_header = request.META.get('HTTP_AUTHORIZATION') authentication_header = request.META.get('HTTP_AUTHORIZATION')
@ -95,32 +74,6 @@ def verify_incoming_friend_request(request, raw_request_body):
return False return False
def verify_incoming_group_invite(request, raw_request_body, handle_field, key_field):
"""Self-certifying verifier for the group invite/accept dance. See
docs/implementation.md#group-invite-and-accept-self-certifying-verification."""
try:
username, domain, signed_data, signature_bytes_hex = verify_request(request, raw_request_body)
except ValueError:
return False
try:
claimed_handle = request.data[handle_field]
claimed_key = request.data[key_field]
except KeyError:
return False
if not claimed_handle or not claimed_key:
return False
if username + "@" + domain != claimed_handle:
return False
if len(claimed_key) != 64:
return False
verify_key = VerifyKey(bytes.fromhex(claimed_key))
try:
verify_key.verify(signed_data.encode('utf-8'), bytes.fromhex(signature_bytes_hex))
return True
except BadSignatureError:
return False
def authenticate_request_against_known_identities(request, raw_request_body): def authenticate_request_against_known_identities(request, raw_request_body):
try: try:
username, domain, signed_data, signature_bytes_hex = verify_request(request, raw_request_body) username, domain, signed_data, signature_bytes_hex = verify_request(request, raw_request_body)
@ -153,17 +106,11 @@ def authenticate_request_against_local_users(request, raw_request_body):
class SignatureAuthentication(authentication.BaseAuthentication): class SignatureAuthentication(authentication.BaseAuthentication):
def authenticate(self, request): def authenticate(self, request):
identity = authenticate_request_against_known_identities(request, request.body.decode('utf-8')) return authenticate_request_against_known_identities(
# Bare None (not a (None, None) tuple) tells DRF to try the next authenticator, instead request, request.body.decode('utf-8')), None
# of treating the request as authenticated with an empty user.
if identity is None:
return None
return identity, None
class SignatureAuthenticationLocal(authentication.BaseAuthentication): class SignatureAuthenticationLocal(authentication.BaseAuthentication):
def authenticate(self, request): def authenticate(self, request):
user = authenticate_request_against_local_users(request, request.body.decode('utf-8')) return authenticate_request_against_local_users(
if user is None: request, request.body.decode('utf-8')), None
return None
return user, None

8
backend/authentication/tests/helpers.py
View file

@ -3,7 +3,7 @@ import json
from django.test import TestCase, Client from django.test import TestCase, Client
from nacl.encoding import HexEncoder from nacl.encoding import HexEncoder
from authentication.models import ToolshedUser, KnownIdentity, Group from authentication.models import ToolshedUser, KnownIdentity
from hostadmin.models import Domain from hostadmin.models import Domain
from nacl.signing import SigningKey from nacl.signing import SigningKey
@ -86,9 +86,3 @@ class UserTestMixin:
domain=self.f['example_com'].name) domain=self.f['example_com'].name)
self.f['ext_user1'] = DummyExternalUser('extuser1', 'external.org') self.f['ext_user1'] = DummyExternalUser('extuser1', 'external.org')
self.f['ext_user2'] = DummyExternalUser('extuser2', 'external.org') self.f['ext_user2'] = DummyExternalUser('extuser2', 'external.org')
class GroupTestMixin:
def prepare_groups(self):
self.f['group1'] = Group.objects.create(name='group1', domain=self.f['example_com'].name)
self.f['group1'].members.add(self.f['local_user1'].public_identity)

105
backend/authentication/tests/test_auth.py
View file

@ -1,5 +1,4 @@
import json import json
import base64
from django.test import Client, RequestFactory from django.test import Client, RequestFactory
from nacl.encoding import HexEncoder from nacl.encoding import HexEncoder
@ -7,7 +6,6 @@ from nacl.signing import SigningKey
from authentication.models import ToolshedUser, KnownIdentity from authentication.models import ToolshedUser, KnownIdentity
from authentication.tests import UserTestMixin, SignatureAuthClient, DummyExternalUser, ToolshedTestCase from authentication.tests import UserTestMixin, SignatureAuthClient, DummyExternalUser, ToolshedTestCase
from files.models import File
class AuthorizationTestCase(ToolshedTestCase): class AuthorizationTestCase(ToolshedTestCase):
@ -242,7 +240,6 @@ class UserApiTestCase(UserTestMixin, ToolshedTestCase):
self.assertEqual(reply.json()['username'], 'testuser1') self.assertEqual(reply.json()['username'], 'testuser1')
self.assertEqual(reply.json()['domain'], 'example.com') self.assertEqual(reply.json()['domain'], 'example.com')
self.assertEqual(reply.json()['email'], 'test1@abc.de') self.assertEqual(reply.json()['email'], 'test1@abc.de')
self.assertIsNone(reply.json()['profile_picture'])
def test_user_info2(self): def test_user_info2(self):
target = "/auth/user/" target = "/auth/user/"
@ -252,50 +249,6 @@ class UserApiTestCase(UserTestMixin, ToolshedTestCase):
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json()['username'], 'testuser1') self.assertEqual(reply.json()['username'], 'testuser1')
self.assertEqual(reply.json()['domain'], 'example.com') self.assertEqual(reply.json()['domain'], 'example.com')
self.assertIsNone(reply.json()['profile_picture'])
def test_user_info_patch_profile_picture(self):
content = base64.b64encode(b'user-profile-image').decode('utf-8')
reply = self.client.patch('/auth/user/', self.f['local_user1'], {
'profile_picture': {
'data': content,
'mime_type': 'image/png'
}
})
self.assertEqual(reply.status_code, 200)
self.assertTrue(reply.json()['profile_picture'])
self.assertEqual(reply.json()['profile_picture']['mime_type'], 'image/png')
self.assertEqual(File.objects.count(), 1)
self.f['local_user1'].refresh_from_db()
self.assertIsNotNone(self.f['local_user1'].profile_picture)
def test_user_info_patch_profile_picture_clear(self):
encoded_content = base64.b64encode(b'user-profile-image').decode('utf-8')
test_file = File.objects.create(mime_type='image/png', data=encoded_content)
self.f['local_user1'].profile_picture = test_file
self.f['local_user1'].save()
reply = self.client.patch('/auth/user/', self.f['local_user1'], {'profile_picture': None})
self.assertEqual(reply.status_code, 200)
self.assertIsNone(reply.json()['profile_picture'])
self.f['local_user1'].refresh_from_db()
self.assertIsNone(self.f['local_user1'].profile_picture)
self.assertFalse(File.objects.filter(id=test_file.id).exists())
def test_user_info_patch_profile_picture_invalid(self):
reply = self.client.patch('/auth/user/', self.f['local_user1'], {'profile_picture': 'invalid'})
self.assertEqual(reply.status_code, 400)
def test_user_info_patch_profile_picture_id(self):
encoded_content = base64.b64encode(b'user-profile-image-by-id').decode('utf-8')
test_file = File.objects.create(mime_type='image/jpeg', data=encoded_content)
reply = self.client.patch('/auth/user/', self.f['local_user1'], {'profile_picture_id': test_file.id})
self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json()['profile_picture']['id'], test_file.id)
def test_user_info_patch_profile_picture_id_not_found(self):
reply = self.client.patch('/auth/user/', self.f['local_user1'], {'profile_picture_id': 999999})
self.assertEqual(reply.status_code, 400)
def test_user_info_fail(self): def test_user_info_fail(self):
reply = self.anonymous_client.get('/auth/user/') reply = self.anonymous_client.get('/auth/user/')
@ -355,52 +308,6 @@ class UserApiTestCase(UserTestMixin, ToolshedTestCase):
self.assertEqual(reply.status_code, 403) self.assertEqual(reply.status_code, 403)
class UserProfileByHandleApiTestCase(UserTestMixin, ToolshedTestCase):
"""Tests for GET /auth/user/<handle>/ - viewing another (friend) user's public profile."""
def setUp(self):
super().setUp()
self.prepare_users()
self.f['local_user1'].friends.add(self.f['ext_user1'].public_identity)
self.anonymous_client = Client(SERVER_NAME='testserver')
self.client = SignatureAuthClient()
def test_view_friend_profile(self):
target = '/auth/user/' + str(self.f['local_user1']) + '/'
reply = self.client.get(target, self.f['ext_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json()['username'], 'testuser1')
self.assertEqual(reply.json()['domain'], 'example.com')
self.assertIsNone(reply.json()['profile_picture'])
self.assertNotIn('email', reply.json())
def test_view_own_profile_via_handle(self):
target = '/auth/user/' + str(self.f['local_user1']) + '/'
reply = self.client.get(target, self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json()['username'], 'testuser1')
def test_view_profile_not_friend(self):
target = '/auth/user/' + str(self.f['local_user1']) + '/'
reply = self.client.get(target, self.f['ext_user2'])
self.assertEqual(reply.status_code, 403)
def test_view_profile_unknown_user(self):
target = '/auth/user/nosuchuser@example.com/'
reply = self.client.get(target, self.f['ext_user1'])
self.assertEqual(reply.status_code, 404)
def test_view_profile_bad_handle(self):
target = '/auth/user/notahandle/'
reply = self.client.get(target, self.f['ext_user1'])
self.assertEqual(reply.status_code, 400)
def test_view_profile_unauthenticated(self):
target = '/auth/user/' + str(self.f['local_user1']) + '/'
reply = self.anonymous_client.get(target)
self.assertEqual(reply.status_code, 403)
class FriendApiTestCase(UserTestMixin, ToolshedTestCase): class FriendApiTestCase(UserTestMixin, ToolshedTestCase):
def setUp(self): def setUp(self):
super().setUp() super().setUp()
@ -412,26 +319,26 @@ class FriendApiTestCase(UserTestMixin, ToolshedTestCase):
self.client = SignatureAuthClient() self.client = SignatureAuthClient()
def test_friend_local(self): def test_friend_local(self):
reply = self.client.get('/api/v1/friends/', self.f['local_user1']) reply = self.client.get('/api/friends/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
def test_friend_external(self): def test_friend_external(self):
reply = self.client.get('/api/v1/friends/', self.f['ext_user1']) reply = self.client.get('/api/friends/', self.f['ext_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
def test_friend_fail(self): def test_friend_fail(self):
reply = self.anonymous_client.get('/api/v1/friends/') reply = self.anonymous_client.get('/api/friends/')
self.assertEqual(reply.status_code, 403) self.assertEqual(reply.status_code, 403)
def test_friend_fail2(self): def test_friend_fail2(self):
target = "/api/v1/friends/" target = "/api/friends/"
signature = self.f['local_user1'].sign("http://testserver2" + target) signature = self.f['local_user1'].sign("http://testserver2" + target)
header = {'HTTP_AUTHORIZATION': 'Signature ' + str(self.f['local_user1']) + ':' + signature} header = {'HTTP_AUTHORIZATION': 'Signature ' + str(self.f['local_user1']) + ':' + signature}
reply = self.anonymous_client.get(target, **header) reply = self.anonymous_client.get(target, **header)
self.assertEqual(reply.status_code, 403) self.assertEqual(reply.status_code, 403)
def test_friend_fail3(self): def test_friend_fail3(self):
target = "/api/v1/friends/" target = "/api/friends/"
unknown_user = DummyExternalUser('extuser3', 'external.org', False) unknown_user = DummyExternalUser('extuser3', 'external.org', False)
signature = unknown_user.sign("http://testserver" + target) signature = unknown_user.sign("http://testserver" + target)
header = {'HTTP_AUTHORIZATION': 'Signature ' + str(unknown_user) + ':' + signature} header = {'HTTP_AUTHORIZATION': 'Signature ' + str(unknown_user) + ':' + signature}
@ -439,7 +346,7 @@ class FriendApiTestCase(UserTestMixin, ToolshedTestCase):
self.assertEqual(reply.status_code, 403) self.assertEqual(reply.status_code, 403)
def test_friend_fail4(self): def test_friend_fail4(self):
target = "/api/v1/friends/" target = "/api/friends/"
signature = self.f['local_user1'].sign("http://testserver" + target) signature = self.f['local_user1'].sign("http://testserver" + target)
header = {'HTTP_AUTHORIZATION': 'Auth ' + str(self.f['local_user1']) + ':' + signature} header = {'HTTP_AUTHORIZATION': 'Auth ' + str(self.f['local_user1']) + ':' + signature}
reply = self.anonymous_client.get(target, **header) reply = self.anonymous_client.get(target, **header)

9
backend/backend/asgi.py
View file

@ -1,3 +1,12 @@
"""
ASGI config for backend project.
It exposes the ASGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/4.1/howto/deployment/asgi/
"""
import os import os
from django.core.asgi import get_asgi_application from django.core.asgi import get_asgi_application

54
backend/backend/settings.py
View file

@ -1,26 +1,26 @@
"""
Django settings for backend project.
Generated by 'django-admin startproject' using Django 4.2.2.
For more information on this file, see
https://docs.djangoproject.com/en/4.1/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/4.1/ref/settings/
"""
import os import os
import subprocess
import dotenv import dotenv
from pathlib import Path from pathlib import Path
# Build paths inside the project like this: BASE_DIR / 'subdir'. # Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent BASE_DIR = Path(__file__).resolve().parent.parent
def _git_commit():
# Docker dev bind-mounts the real .git dir at /git (see docker-compose.yml); bare-metal dev
# finds it by walking up from BASE_DIR instead. Prod has no .git at all, so both fail and we
# fall back to the GIT_COMMIT build-arg/env var (see Dockerfile.backend/playbook.yml).
cmd = ['git', '--git-dir=/git'] if os.path.isdir('/git') else ['git']
try:
return subprocess.check_output(
[*cmd, 'rev-parse', '--short', 'HEAD'], cwd=BASE_DIR, stderr=subprocess.DEVNULL
).decode().strip()
except (subprocess.CalledProcessError, FileNotFoundError, OSError):
return os.environ.get('GIT_COMMIT', 'unknown')
dotenv.load_dotenv(BASE_DIR / '.env') dotenv.load_dotenv(BASE_DIR / '.env')
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/4.1/howto/deployment/checklist/
SECRET_KEY = os.environ.get('SECRET_KEY', None) SECRET_KEY = os.environ.get('SECRET_KEY', None)
if SECRET_KEY is None: if SECRET_KEY is None:
raise Exception('environment variable SECRET_KEY not set. try running `configure.py` or setting it manually') raise Exception('environment variable SECRET_KEY not set. try running `configure.py` or setting it manually')
@ -30,7 +30,6 @@ DEBUG = os.environ.get('DEBUG', 'False').lower() == 'true'
# Application definition # Application definition
TOOLSHED_VERSION = "0.0.0-dev.0" TOOLSHED_VERSION = "0.0.0-dev.0"
GIT_COMMIT = _git_commit()
INSTALLED_APPS = [ INSTALLED_APPS = [
'django.contrib.admin', 'django.contrib.admin',
@ -87,8 +86,6 @@ CORS_ALLOW_ALL_ORIGINS = True
USE_X_FORWARDED_HOST = True USE_X_FORWARDED_HOST = True
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https') SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
SERVE_X_ACCEL_REDIRECT = os.environ.get('SERVE_X_ACCEL_REDIRECT', 'False').lower() == 'true'
ROOT_URLCONF = 'backend.urls' ROOT_URLCONF = 'backend.urls'
TEMPLATES = [ TEMPLATES = [
@ -109,15 +106,21 @@ TEMPLATES = [
WSGI_APPLICATION = 'backend.wsgi.application' WSGI_APPLICATION = 'backend.wsgi.application'
# Database
# https://docs.djangoproject.com/en/4.1/ref/settings/#databases
DATABASES = { DATABASES = {
'default': { 'default': {
'ENGINE': 'django.db.backends.sqlite3', 'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.environ.get('TOOLSHED_DB_PATH', BASE_DIR / 'db.sqlite3'), 'NAME': BASE_DIR / 'db.sqlite3',
} }
} }
AUTH_USER_MODEL = 'authentication.ToolshedUser' AUTH_USER_MODEL = 'authentication.ToolshedUser'
# Password validation
# https://docs.djangoproject.com/en/4.1/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [ AUTH_PASSWORD_VALIDATORS = [
{ {
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator', 'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
@ -133,6 +136,9 @@ AUTH_PASSWORD_VALIDATORS = [
}, },
] ]
# Internationalization
# https://docs.djangoproject.com/en/4.1/topics/i18n/
LANGUAGE_CODE = 'en-us' LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC' TIME_ZONE = 'UTC'
@ -141,17 +147,17 @@ USE_I18N = True
USE_TZ = True USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.1/howto/static-files/
STATIC_ROOT = 'staticfiles' STATIC_ROOT = 'staticfiles'
STATIC_URL = '/static/' STATIC_URL = '/static/'
MEDIA_ROOT = os.environ.get('TOOLSHED_USERFILES_PATH', 'userfiles') MEDIA_ROOT = 'userfiles'
MEDIA_URL = '/media/' MEDIA_URL = '/media/'
# Pinned explicitly (rather than left to the backend process's ambient umask) so group-read # Default primary key field type
# is guaranteed for nginx/www-data regardless of how the container is started - see # https://docs.djangoproject.com/en/4.1/ref/settings/#default-auto-field
# SERVE_X_ACCEL_REDIRECT and playbook.yml's `location /redirect_media/`.
FILE_UPLOAD_PERMISSIONS = 0o640
FILE_UPLOAD_DIRECTORY_PERMISSIONS = 0o750
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField' DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'

27
backend/backend/urls.py
View file

@ -1,3 +1,18 @@
"""backend URL Configuration
The `urlpatterns` list routes URLs to views. For more information please see:
https://docs.djangoproject.com/en/4.1/topics/http/urls/
Examples:
Function views
1. Add an import: from my_app import views
2. Add a URL to urlpatterns: path('', views.home, name='home')
Class-based views
1. Add an import: from other_app.views import Home
2. Add a URL to urlpatterns: path('', Home.as_view(), name='home')
Including another URLconf
1. Import the include() function: from django.urls import include, path
2. Add a URL to urlpatterns: path('blog/', include('blog.urls'))
"""
from django.contrib import admin from django.contrib import admin
from django.urls import path, include from django.urls import path, include
from drf_yasg import openapi from drf_yasg import openapi
@ -19,14 +34,10 @@ urlpatterns = [
path('djangoadmin/', admin.site.urls), path('djangoadmin/', admin.site.urls),
path('auth/', include('authentication.api')), path('auth/', include('authentication.api')),
path('admin/', include('hostadmin.api')), path('admin/', include('hostadmin.api')),
path('api/version/', include('toolshed.api.version')), path('api/', include('toolshed.api.friend')),
path('api/v1/', include('toolshed.api.info')), path('api/', include('toolshed.api.inventory')),
path('api/v1/', include('toolshed.api.friend')), path('api/', include('toolshed.api.info')),
path('api/v1/', include('toolshed.api.group')), path('api/', include('toolshed.api.files')),
path('api/v1/', include('toolshed.api.idmap')),
path('api/v1/', include('toolshed.api.inventory')),
path('api/v1/', include('toolshed.api.files')),
path('api/v1/', include('toolshed.api.offlinedata')),
path('media/', include('files.media_urls')), path('media/', include('files.media_urls')),
path('docs/', schema_view.with_ui('swagger', cache_timeout=0), name='api-docs'), path('docs/', schema_view.with_ui('swagger', cache_timeout=0), name='api-docs'),
] ]

9
backend/backend/wsgi.py
View file

@ -1,3 +1,12 @@
"""
WSGI config for backend project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/4.1/howto/deployment/wsgi/
"""
import os import os
from django.core.wsgi import get_wsgi_application from django.core.wsgi import get_wsgi_application

117
backend/configure.py
View file

@ -8,80 +8,67 @@ import dotenv
from django.db import transaction, IntegrityError from django.db import transaction, IntegrityError
def yesno(prompt, default=False): class CmdCtx:
if not sys.stdin.isatty():
return default
yes = {'yes', 'y', 'ye'}
no = {'no', 'n'}
if default: def __init__(self, args):
yes.add('') self.args = args
else:
no.add('')
hint = ' [Y/n] ' if default else ' [y/N] ' def yesno(self, prompt, default=False):
if not sys.stdin.isatty() or self.args.noninteractive:
while True: return default
choice = input(prompt + hint).lower() elif self.args.yes:
if choice in yes:
return True return True
elif choice in no: elif self.args.no:
return False return False
yes = {'yes', 'y', 'ye'}
no = {'no', 'n'}
if default:
yes.add('')
else: else:
print('Please respond with "yes" or "no"') no.add('')
hint = ' [Y/n] ' if default else ' [y/N] '
def configure(): while True:
# Keys this function may generate/update; tracked so an unwritable .env (e.g. a prod choice = input(prompt + hint).lower()
# container configured via --env-file) can still print them for the operator to apply manually. if choice in yes:
tracked_keys = ['SECRET_KEY', 'ALLOWED_HOSTS'] return True
unwritable = False elif choice in no:
return False
if not os.path.exists('.env'):
if yesno("the .env file does not exist, do you want to create it?", default=True):
if not os.path.exists('.env.dist'):
print('No .env.dist file found')
else: else:
for key in dotenv.dotenv_values('.env.dist'): print('Please respond with "yes" or "no"')
if key not in tracked_keys:
tracked_keys.append(key)
from shutil import copyfile
try:
copyfile('.env.dist', '.env')
except PermissionError:
unwritable = True
dotenv.load_dotenv('.env')
if not os.getenv('SECRET_KEY'): def configure(ctx):
if not os.path.exists('.env'):
if not ctx.yesno("the .env file does not exist, do you want to create it?", default=True):
print('Aborting')
exit(0)
if not os.path.exists('.env.dist'):
print('No .env.dist file found')
exit(1)
else:
from shutil import copyfile
copyfile('.env.dist', '.env')
env = dotenv.load_dotenv('.env')
if not env or not os.getenv('SECRET_KEY'):
from django.core.management.utils import get_random_secret_key from django.core.management.utils import get_random_secret_key
print('No SECRET_KEY found in .env file, generating one...') print('No SECRET_KEY found in .env file, generating one...')
secret_key = get_random_secret_key() with open('.env', 'a') as f:
os.environ['SECRET_KEY'] = secret_key f.write('\nSECRET_KEY=')
try: f.write(get_random_secret_key())
with open('.env', 'a') as f: f.write('\n')
f.write('\nSECRET_KEY=')
f.write(secret_key)
f.write('\n')
except PermissionError:
unwritable = True
# TODO rename ALLOWED_HOSTS to something more self-explanatory # TODO rename ALLOWED_HOSTS to something more self-explanatory
current_hosts = os.getenv('ALLOWED_HOSTS') current_hosts = os.getenv('ALLOWED_HOSTS')
print('Current ALLOWED_HOSTS: {}'.format(current_hosts)) print('Current ALLOWED_HOSTS: {}'.format(current_hosts))
if yesno("Do you want to add ALLOWED_HOSTS?"): if ctx.yesno("Do you want to add ALLOWED_HOSTS?"):
hosts = input("Enter a comma-separated list of allowed hosts: ") hosts = input("Enter a comma-separated list of allowed hosts: ")
joined_hosts = current_hosts + ',' + hosts if current_hosts else hosts joined_hosts = current_hosts + ',' + hosts if current_hosts else hosts
os.environ['ALLOWED_HOSTS'] = joined_hosts dotenv.set_key('.env', 'ALLOWED_HOSTS', joined_hosts)
try:
dotenv.set_key('.env', 'ALLOWED_HOSTS', joined_hosts)
except PermissionError:
unwritable = True
if unwritable:
print('Could not write .env (read-only working directory) - resulting configuration:')
for key in tracked_keys:
print('{}={}'.format(key, os.getenv(key, '')))
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "backend.settings") os.environ.setdefault("DJANGO_SETTINGS_MODULE", "backend.settings")
import django import django
@ -89,20 +76,21 @@ def configure():
django.setup() django.setup()
if not os.path.exists('db.sqlite3'): if not os.path.exists('db.sqlite3'):
if not yesno("No database found, do you want to create one?", default=True): if not ctx.yesno("No database found, do you want to create one?", default=True):
print('Aborting') print('Aborting')
exit(0) exit(0)
from django.core.management import call_command from django.core.management import call_command
call_command('migrate') call_command('migrate')
if yesno("Do you want to create a superuser?"): if ctx.yesno("Do you want to create a superuser?"):
from django.core.management import call_command from django.core.management import call_command
call_command('createsuperuser') call_command('createsuperuser')
call_command('collectstatic', '--no-input') call_command('collectstatic', '--no-input')
if yesno("Do you want to import all categories, properties and tags contained in this repository?", default=True): if ctx.yesno("Do you want to import all categories, properties and tags contained in this repository?",
default=True):
from hostadmin.serializers import CategorySerializer, PropertySerializer, TagSerializer from hostadmin.serializers import CategorySerializer, PropertySerializer, TagSerializer
from hostadmin.models import ImportedIdentifierSets from hostadmin.models import ImportedIdentifierSets
from hashlib import sha256 from hashlib import sha256
@ -205,11 +193,7 @@ def testdata():
import django import django
django.setup() django.setup()
testdata_path = os.environ.get('TOOLSHED_SETUP_PATH', 'testdata.py') if os.path.exists('testdata.py'):
if os.path.exists(testdata_path):
if testdata_path != 'testdata.py':
import sys
sys.path.append(os.path.dirname(testdata_path))
from testdata import create_test_data from testdata import create_test_data
create_test_data() create_test_data()
else: else:
@ -222,6 +206,7 @@ def main():
parser = ArgumentParser(description='Toolshed Server Configuration') parser = ArgumentParser(description='Toolshed Server Configuration')
parser.add_argument('--yes', '-y', help='Answer yes to all questions', action='store_true') parser.add_argument('--yes', '-y', help='Answer yes to all questions', action='store_true')
parser.add_argument('--no', '-n', help='Answer no to all questions', action='store_true') parser.add_argument('--no', '-n', help='Answer no to all questions', action='store_true')
parser.add_argument('--noninteractive', '-x', help="Run in noninteractive mode", action='store_true')
parser.add_argument('cmd', help='Command', default='configure', nargs='?') parser.add_argument('cmd', help='Command', default='configure', nargs='?')
args = parser.parse_args() args = parser.parse_args()
@ -229,8 +214,10 @@ def main():
print('Error: --yes and --no are mutually exclusive') print('Error: --yes and --no are mutually exclusive')
exit(1) exit(1)
ctx = CmdCtx(args)
if args.cmd == 'configure': if args.cmd == 'configure':
configure() configure(ctx)
elif args.cmd == 'reset': elif args.cmd == 'reset':
reset() reset()
elif args.cmd == 'testdata': elif args.cmd == 'testdata':

129
backend/files/media_urls.py
View file

@ -1,17 +1,6 @@
import io
import os
from datetime import timedelta
from django.core.files.base import ContentFile
from django.core.files.storage import default_storage
from django.http import HttpResponse from django.http import HttpResponse
from django.urls import path from django.urls import path
from django.db.models import Q
from django.conf import settings
from django.utils.http import http_date
from django.utils.timezone import now
from drf_yasg.utils import swagger_auto_schema from drf_yasg.utils import swagger_auto_schema
from PIL import Image
from rest_framework import status from rest_framework import status
from rest_framework.decorators import api_view, permission_classes, authentication_classes from rest_framework.decorators import api_view, permission_classes, authentication_classes
from rest_framework.permissions import IsAuthenticated from rest_framework.permissions import IsAuthenticated
@ -20,129 +9,27 @@ from rest_framework.response import Response
from authentication.signature_auth import SignatureAuthentication from authentication.signature_auth import SignatureAuthentication
from files.models import File from files.models import File
THUMBNAIL_SIZES = (32, 64, 256)
def _accessible_files(request):
# Shared by media_urls and thumbnail_urls: a file is visible if the requester is
# friends-or-self with whatever references it (item, profile picture), a member of the group
# that owns the item it's attached to, or it's their own staged photo.
return File.objects.filter(
Q(connected_items__owner__in=request.user.friends_or_self(), connected_items__is_deleted=False) |
Q(connected_items__owner_group__in=request.user.member_of_groups.all(), connected_items__is_deleted=False) |
Q(profile_picture_users__in=request.user.friends_or_self()) |
Q(staged_by_workflows__owner__in=request.user.user.all())
).distinct()
def _cache_headers(etag):
# Content is hash-addressed and can never change under a given URL, so it's cacheable forever.
return {
'ETag': etag,
'Cache-Control': 'max-age=31536000, private, immutable',
'Expires': http_date((now() + timedelta(days=365)).timestamp()),
}
@swagger_auto_schema(method='GET', auto_schema=None) @swagger_auto_schema(method='GET', auto_schema=None)
@api_view(['GET']) @api_view(['GET'])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication]) @authentication_classes([SignatureAuthentication])
def media_urls(request, hash_path): def media_urls(request, hash_path):
# CORS is added automatically by middleware, except via X-Accel-Redirect, where nginx's
# /redirect_media/ block must set it instead.
#
# Looked up by the derived storage path, not the raw hash, to match FileSerializer.name
# (used for AuthenticatedImage's `src`) and the existing test suite (MediaUrlTestCase).
try: try:
file = _accessible_files(request).get(file=hash_path) file = File.objects.filter(connected_items__owner__in=request.user.friends_or_self()).distinct().get(
file=hash_path)
# Must run before this check, else a bare hash + If-None-Match would let anyone probe return HttpResponse(status=status.HTTP_200_OK,
# file existence for files they can't see. content_type=file.mime_type,
if request.META.get('HTTP_IF_NONE_MATCH') == file.hash: headers={
return HttpResponse(status=status.HTTP_304_NOT_MODIFIED) 'X-Accel-Redirect': f'/redirect_media/{hash_path}',
'Access-Control-Allow-Origin': '*',
cache_headers = _cache_headers(file.hash) }) # TODO Expires and Cache-Control
if settings.SERVE_X_ACCEL_REDIRECT:
return HttpResponse(status=status.HTTP_200_OK,
content_type=file.mime_type,
headers={
'X-Accel-Redirect': f'/redirect_media/{hash_path}',
**cache_headers,
})
else:
# Reads via FieldFile.open() (not file.file.path) since tests swap in an in-memory storage backend.
with file.file.open('rb') as fh:
content = fh.read()
return HttpResponse(status=status.HTTP_200_OK,
content_type=file.mime_type,
headers=cache_headers,
content=content)
except File.DoesNotExist:
return Response(status=status.HTTP_404_NOT_FOUND)
def _thumbnail_rel_path(file_hash, size):
# Mirrors hash_upload()'s sharding under thumbnails/<size>/, reachable via the same nginx alias as originals.
return os.path.join('thumbnails', str(size), file_hash[:2], file_hash[2:4], file_hash[4:6],
file_hash[6:] + '.jpg')
@swagger_auto_schema(method='GET', auto_schema=None)
@api_view(['GET'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication])
def thumbnail_urls(request, size, hash_path):
if size not in THUMBNAIL_SIZES:
return Response(status=status.HTTP_404_NOT_FOUND)
try:
file = _accessible_files(request).get(file=hash_path)
etag = f'{file.hash}_{size}'
if request.META.get('HTTP_IF_NONE_MATCH') == etag:
return HttpResponse(status=status.HTTP_304_NOT_MODIFIED)
# Read/write via default_storage, not a hand-rolled path, to work with both real-disk
# and in-memory test storage.
rel_path = _thumbnail_rel_path(file.hash, size)
if not default_storage.exists(rel_path):
# Always re-encoded as JPEG regardless of original format - simpler than preserving transparency at this scale.
with file.file.open('rb') as fh:
image = Image.open(fh)
image.thumbnail((size, size))
# Flatten through RGBA before dropping to RGB. See docs/implementation.md#rgba-flattening-avoids-revealing-black-under-transparent-pixels.
rgba = image.convert('RGBA')
flattened = Image.new('RGB', rgba.size, (255, 255, 255))
flattened.paste(rgba, mask=rgba.getchannel('A'))
buffer = io.BytesIO()
flattened.save(buffer, 'JPEG', quality=90)
default_storage.save(rel_path, ContentFile(buffer.getvalue()))
cache_headers = _cache_headers(etag)
if settings.SERVE_X_ACCEL_REDIRECT:
return HttpResponse(status=status.HTTP_200_OK,
content_type='image/jpeg',
headers={
'X-Accel-Redirect': f'/redirect_media/{rel_path}',
**cache_headers,
})
else:
with default_storage.open(rel_path, 'rb') as fh:
content = fh.read()
return HttpResponse(status=status.HTTP_200_OK,
content_type='image/jpeg',
headers=cache_headers,
content=content)
except File.DoesNotExist: except File.DoesNotExist:
return Response(status=status.HTTP_404_NOT_FOUND) return Response(status=status.HTTP_404_NOT_FOUND)
urlpatterns = [ urlpatterns = [
path('<int:size>/<path:hash_path>/', thumbnail_urls),
path('<path:hash_path>', media_urls), path('<path:hash_path>', media_urls),
] ]

7
backend/files/models.py
View file

@ -1,7 +1,4 @@
from types import SimpleNamespace
from django.core.files.base import ContentFile from django.core.files.base import ContentFile
from django.core.files.storage import default_storage
from django.db import models, IntegrityError from django.db import models, IntegrityError
from django.db.models import Model from django.db.models import Model
@ -43,10 +40,6 @@ class FileManager(models.Manager):
else: else:
raise ValueError('data must be a base64 encoded string or file and hash must be provided') raise ValueError('data must be a base64 encoded string or file and hash must be provided')
if not self.filter(hash=kwargs['hash']).exists(): if not self.filter(hash=kwargs['hash']).exists():
# Clears a stale orphan already at this hash's canonical path before saving. See docs/implementation.md#stale-orphan-cleanup-at-the-canonical-hash-path.
expected_path = hash_upload(SimpleNamespace(hash=kwargs['hash']), '')
if default_storage.exists(expected_path):
default_storage.delete(expected_path)
return super().create(**kwargs) return super().create(**kwargs)
else: else:
raise IntegrityError('File with this hash already exists') raise IntegrityError('File with this hash already exists')

2
backend/files/serializers.py
View file

@ -14,7 +14,7 @@ class FileSerializer(serializers.Serializer):
def to_representation(self, instance): def to_representation(self, instance):
return {'id': instance.id, 'name': instance.file.url, 'size': instance.file.size, return {'id': instance.id, 'name': instance.file.url, 'size': instance.file.size,
'mime_type': instance.mime_type, 'hash': instance.hash} 'mime_type': instance.mime_type}
def create(self, validated_data): def create(self, validated_data):
return File.objects.get_or_create(**validated_data)[0] return File.objects.get_or_create(**validated_data)[0]

222
backend/files/tests.py
View file

@ -1,21 +1,13 @@
import io
import os
import zlib
from django.conf import settings
from django.core.files.base import ContentFile from django.core.files.base import ContentFile
from django.core.files.storage import DefaultStorage, default_storage from django.core.files.storage import DefaultStorage
from django.db import IntegrityError, transaction from django.db import IntegrityError, transaction
from django.test import Client, override_settings from django.test import Client
from authentication.tests import SignatureAuthClient, ToolshedTestCase, UserTestMixin, GroupTestMixin from authentication.tests import SignatureAuthClient, ToolshedTestCase, UserTestMixin
from toolshed.models import InventoryItem
from toolshed.tests import InventoryTestMixin from toolshed.tests import InventoryTestMixin
from nacl.hash import sha256 from nacl.hash import sha256
from nacl.encoding import HexEncoder from nacl.encoding import HexEncoder
from PIL import Image
import base64 import base64
from files.media_urls import THUMBNAIL_SIZES
from files.models import File from files.models import File
anonymous_client = Client() anonymous_client = Client()
@ -113,18 +105,6 @@ class FilesTestCase(FilesTestMixin, ToolshedTestCase):
self.assertEqual(File.objects.count(), 3) self.assertEqual(File.objects.count(), 3)
self.assertEqual(countdir(DefaultStorage(), ''), 3) self.assertEqual(countdir(DefaultStorage(), ''), 3)
def test_file_upload_reclaims_stale_orphan_at_canonical_path(self):
# Regression test for a stale orphan at the canonical hash path. See docs/implementation.md#stale-orphan-cleanup-at-the-canonical-hash-path.
expected_path = f"{self.f['hash4'][:2]}/{self.f['hash4'][2:4]}/{self.f['hash4'][4:6]}/{self.f['hash4'][6:]}"
default_storage.save(expected_path, ContentFile(self.f['test_content4']))
self.assertTrue(default_storage.exists(expected_path))
self.assertFalse(File.objects.filter(hash=self.f['hash4']).exists())
file = File.objects.create(mime_type='text/plain', data=self.f['encoded_content4'])
self.assertEqual(file.file.name, expected_path)
self.assertEqual(file.file.read(), self.f['test_content4'])
class MediaUrlTestCase(FilesTestMixin, UserTestMixin, InventoryTestMixin, ToolshedTestCase): class MediaUrlTestCase(FilesTestMixin, UserTestMixin, InventoryTestMixin, ToolshedTestCase):
def setUp(self): def setUp(self):
@ -140,7 +120,6 @@ class MediaUrlTestCase(FilesTestMixin, UserTestMixin, InventoryTestMixin, Toolsh
self.f['item2'].files.add(self.f['test_file1']) self.f['item2'].files.add(self.f['test_file1'])
@override_settings(SERVE_X_ACCEL_REDIRECT=True)
def test_file_url(self): def test_file_url(self):
reply = client.get( reply = client.get(
f"/media/{self.f['hash1'][:2]}/{self.f['hash1'][2:4]}/{self.f['hash1'][4:6]}/{self.f['hash1'][6:]}", f"/media/{self.f['hash1'][:2]}/{self.f['hash1'][2:4]}/{self.f['hash1'][4:6]}/{self.f['hash1'][6:]}",
@ -186,198 +165,3 @@ class MediaUrlTestCase(FilesTestMixin, UserTestMixin, InventoryTestMixin, Toolsh
self.f['ext_user1']) self.f['ext_user1'])
self.assertEqual(reply.status_code, 404) self.assertEqual(reply.status_code, 404)
self.assertTrue('X-Accel-Redirect' not in reply.headers) self.assertTrue('X-Accel-Redirect' not in reply.headers)
def test_file_url_only_connected_via_deleted_item(self):
# test_file2 is only reachable through item1; soft-deleting it doesn't sever the files
# M2M row, so this would regress to serving test_file2 as if item1 were still live if
# _accessible_files ever stops excluding soft-deleted items again.
self.f['item1'].delete()
reply = client.get(
f"/media/{self.f['hash2'][:2]}/{self.f['hash2'][2:4]}/{self.f['hash2'][4:6]}/{self.f['hash2'][6:]}",
self.f['local_user1'])
self.assertEqual(reply.status_code, 404)
@override_settings(SERVE_X_ACCEL_REDIRECT=True)
def test_profile_picture_url(self):
self.f['local_user1'].profile_picture = self.f['test_file3']
self.f['local_user1'].save()
reply = client.get(
f"/media/{self.f['hash3'][:2]}/{self.f['hash3'][2:4]}/{self.f['hash3'][4:6]}/{self.f['hash3'][6:]}",
self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
@override_settings(SERVE_X_ACCEL_REDIRECT=True)
def test_profile_picture_url_friend(self):
self.f['local_user1'].profile_picture = self.f['test_file3']
self.f['local_user1'].save()
reply = client.get(
f"/media/{self.f['hash3'][:2]}/{self.f['hash3'][2:4]}/{self.f['hash3'][4:6]}/{self.f['hash3'][6:]}",
self.f['local_user2'])
self.assertEqual(reply.status_code, 200)
def test_profile_picture_url_not_friend(self):
self.f['local_user1'].profile_picture = self.f['test_file3']
self.f['local_user1'].save()
reply = client.get(
f"/media/{self.f['hash3'][:2]}/{self.f['hash3'][2:4]}/{self.f['hash3'][4:6]}/{self.f['hash3'][6:]}",
self.f['ext_user1'])
self.assertEqual(reply.status_code, 404)
class GroupOwnedMediaUrlTestCase(FilesTestMixin, UserTestMixin, GroupTestMixin, ToolshedTestCase):
"""_accessible_files() only checked connected_items__owner (personal items) before, never
connected_items__owner_group - a group-owned item's own files were unreachable via /media/ or
/thumbnails/ for every member, including ones who could see and edit the item itself."""
def setUp(self):
super().setUp()
self.prepare_files()
self.prepare_users()
self.prepare_groups()
self.f['group1'].members.add(self.f['local_user2'].public_identity)
self.f['group_item'] = InventoryItem.create_for_owner(
owner_group=self.f['group1'], owned_quantity=1, name='group-drill')
self.f['group_item'].files.add(self.f['test_file1'])
@override_settings(SERVE_X_ACCEL_REDIRECT=True)
def test_group_member_can_view_group_item_file(self):
reply = client.get(
f"/media/{self.f['hash1'][:2]}/{self.f['hash1'][2:4]}/{self.f['hash1'][4:6]}/{self.f['hash1'][6:]}",
self.f['local_user2'])
self.assertEqual(reply.status_code, 200)
def test_non_member_cannot_view_group_item_file(self):
reply = client.get(
f"/media/{self.f['hash1'][:2]}/{self.f['hash1'][2:4]}/{self.f['hash1'][4:6]}/{self.f['hash1'][6:]}",
self.f['ext_user1'])
self.assertEqual(reply.status_code, 404)
class ThumbnailUrlTestCase(FilesTestMixin, UserTestMixin, InventoryTestMixin, ToolshedTestCase):
def setUp(self):
super().setUp()
self.prepare_files()
self.prepare_users()
self.prepare_categories()
self.prepare_tags()
self.prepare_properties()
self.prepare_inventory()
# Each test uses a distinct seeded image (own hash/cache path) since InMemoryStorage
# isn't reset between test methods, so a shared image risks one test's cached thumbnail leaking into another's assertions.
seed = zlib.crc32(self._testMethodName.encode()) % 256
buffer = io.BytesIO()
Image.new('RGB', (800, 600), (seed, 255 - seed, 128)).save(buffer, 'PNG')
image_bytes = buffer.getvalue()
self.f['image_hash'] = sha256(image_bytes, encoder=HexEncoder).decode('utf-8')
self.f['image_file'] = File.objects.create(
mime_type='image/png', data=base64.b64encode(image_bytes).decode('utf-8'))
self.f['item1'].files.add(self.f['image_file'])
def _thumb_url(self, size, image_hash=None):
h = image_hash or self.f['image_hash']
return f"/media/{size}/{h[:2]}/{h[2:4]}/{h[4:6]}/{h[6:]}/"
def _thumb_rel_path(self, size):
h = self.f['image_hash']
return os.path.join('thumbnails', str(size), h[:2], h[2:4], h[4:6], h[6:] + '.jpg')
def test_thumbnail_sizes_available(self):
# Fixed size allow-list this suite exercises - update both if media_urls.py's THUMBNAIL_SIZES changes.
self.assertEqual(THUMBNAIL_SIZES, (32, 64, 256))
@override_settings(SERVE_X_ACCEL_REDIRECT=False)
def test_thumbnail_generates_resized_jpeg(self):
self.assertFalse(default_storage.exists(self._thumb_rel_path(64)))
reply = client.get(self._thumb_url(64), self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.headers['Content-Type'], 'image/jpeg')
generated = Image.open(io.BytesIO(reply.content))
self.assertEqual(generated.format, 'JPEG')
# Aspect-ratio-preserving fit within a 64x64 box, not a crop to exactly 64x64.
self.assertLessEqual(max(generated.size), 64)
self.assertAlmostEqual(generated.size[0] / generated.size[1], 800 / 600, places=2)
@override_settings(SERVE_X_ACCEL_REDIRECT=False)
def test_thumbnail_flattens_transparency_instead_of_going_black(self):
# Regression test for an 'LA' source with zeroed transparent-region luminance. See docs/implementation.md#rgba-flattening-avoids-revealing-black-under-transparent-pixels.
half_transparent = Image.new('LA', (200, 200))
pixels = half_transparent.load()
for x in range(200):
for y in range(200):
if x < 100:
pixels[x, y] = (0, 0) # transparent, zeroed-out luminance underneath
else:
pixels[x, y] = (255, 255) # fully opaque, bright content
buffer = io.BytesIO()
half_transparent.save(buffer, 'PNG')
image_bytes = buffer.getvalue()
image_hash = sha256(image_bytes, encoder=HexEncoder).decode('utf-8')
image_file = File.objects.create(
mime_type='image/png', data=base64.b64encode(image_bytes).decode('utf-8'))
self.f['item1'].files.add(image_file)
reply = client.get(self._thumb_url(64, image_hash=image_hash), self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
generated = Image.open(io.BytesIO(reply.content)).convert('L')
# The opaque (right) half must stay bright; a naive RGB conversion would blacken it too.
self.assertGreater(generated.getpixel((generated.width - 1, generated.height // 2)), 200)
self.assertNotEqual(generated.getextrema(), (0, 0))
@override_settings(SERVE_X_ACCEL_REDIRECT=False)
def test_thumbnail_served_from_cache_on_second_request(self):
client.get(self._thumb_url(64), self.f['local_user1'])
rel_path = self._thumb_rel_path(64)
with default_storage.open(rel_path, 'rb') as f:
cached_bytes = f.read()
# Overwrites the cache with a marker so a correct implementation must serve it back, not regenerate.
default_storage.delete(rel_path)
default_storage.save(rel_path, ContentFile(cached_bytes + b'MARKER'))
reply = client.get(self._thumb_url(64), self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertTrue(reply.content.endswith(b'MARKER'))
def test_thumbnail_invalid_size(self):
reply = client.get(self._thumb_url(100), self.f['local_user1'])
self.assertEqual(reply.status_code, 404)
self.assertFalse(default_storage.exists(self._thumb_rel_path(100)))
def test_thumbnail_not_found(self):
reply = client.get(self._thumb_url(64, image_hash='0' * 64), self.f['local_user1'])
self.assertEqual(reply.status_code, 404)
def test_thumbnail_anonymous(self):
reply = anonymous_client.get(self._thumb_url(64))
self.assertEqual(reply.status_code, 403)
def test_thumbnail_not_friend(self):
# local_user1/local_user2 are friends here (see prepare_inventory), so the denied case needs a stranger instead.
reply = client.get(self._thumb_url(64), self.f['ext_user1'])
self.assertEqual(reply.status_code, 404)
self.assertFalse(default_storage.exists(self._thumb_rel_path(64)))
def test_thumbnail_conditional_get(self):
reply = client.get(self._thumb_url(64), self.f['local_user1'])
etag = reply.headers['ETag']
self.assertEqual(etag, f"{self.f['image_hash']}_64")
reply = client.get(self._thumb_url(64), self.f['local_user1'], HTTP_IF_NONE_MATCH=etag)
self.assertEqual(reply.status_code, 304)
@override_settings(SERVE_X_ACCEL_REDIRECT=True)
def test_thumbnail_x_accel_redirect(self):
reply = client.get(self._thumb_url(64), self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
h = self.f['image_hash']
self.assertEqual(reply.headers['X-Accel-Redirect'],
f"/redirect_media/thumbnails/64/{h[:2]}/{h[2:4]}/{h[4:6]}/{h[6:]}.jpg")

24
backend/hostadmin/serializers.py
View file

@ -41,10 +41,6 @@ class DomainSerializer(serializers.ModelSerializer):
class CategorySerializer(serializers.ModelSerializer): class CategorySerializer(serializers.ModelSerializer):
parent = SlugPathField(slug_field='name', queryset=Category.objects.all(), required=False) parent = SlugPathField(slug_field='name', queryset=Category.objects.all(), required=False)
handle = serializers.SerializerMethodField()
def get_handle(self, obj):
return obj.get_handle()
def validate(self, attrs): def validate(self, attrs):
if 'name' in attrs: if 'name' in attrs:
@ -60,17 +56,13 @@ class CategorySerializer(serializers.ModelSerializer):
class Meta: class Meta:
model = Category model = Category
fields = ['name', 'description', 'parent', 'origin', 'handle'] fields = ['name', 'description', 'parent', 'origin']
read_only_fields = ['origin', 'handle'] read_only_fields = ['origin']
ref_name = 'HostAdminCategory' ref_name = 'HostAdminCategory'
class PropertySerializer(serializers.ModelSerializer): class PropertySerializer(serializers.ModelSerializer):
category = SlugPathField(slug_field='name', queryset=Category.objects.all(), required=False) category = SlugPathField(slug_field='name', queryset=Category.objects.all(), required=False)
handle = serializers.SerializerMethodField()
def get_handle(self, obj):
return obj.get_handle()
def validate(self, attrs): def validate(self, attrs):
if 'name' in attrs: if 'name' in attrs:
@ -87,17 +79,13 @@ class PropertySerializer(serializers.ModelSerializer):
class Meta: class Meta:
model = Property model = Property
fields = ['name', 'description', 'category', 'unit_symbol', 'unit_name', 'unit_name_plural', 'base2_prefix', fields = ['name', 'description', 'category', 'unit_symbol', 'unit_name', 'unit_name_plural', 'base2_prefix',
'dimensions', 'origin', 'handle'] 'dimensions', 'origin']
read_only_fields = ['origin', 'handle'] read_only_fields = ['origin']
ref_name = 'HostAdminProperty' ref_name = 'HostAdminProperty'
class TagSerializer(serializers.ModelSerializer): class TagSerializer(serializers.ModelSerializer):
category = SlugPathField(slug_field='name', queryset=Category.objects.all(), required=False) category = SlugPathField(slug_field='name', queryset=Category.objects.all(), required=False)
handle = serializers.SerializerMethodField()
def get_handle(self, obj):
return obj.get_handle()
def validate(self, attrs): def validate(self, attrs):
if 'name' in attrs: if 'name' in attrs:
@ -113,6 +101,6 @@ class TagSerializer(serializers.ModelSerializer):
class Meta: class Meta:
model = Tag model = Tag
fields = ['name', 'description', 'category', 'origin', 'handle'] fields = ['name', 'description', 'category', 'origin']
read_only_fields = ['origin', 'handle'] read_only_fields = ['origin']
ref_name = 'HostAdminTag' ref_name = 'HostAdminTag'

8
backend/hostadmin/tests.py
View file

@ -28,7 +28,7 @@ class DomainApiTestCase(UserTestMixin, ToolshedTestCase):
self.prepare_users() self.prepare_users()
def test_get_domains(self): def test_get_domains(self):
response = client.get('/api/v1/domains/', self.f['local_user1']) response = client.get('/api/domains/', self.f['local_user1'])
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response.json(), ['example.com']) self.assertEqual(response.json(), ['example.com'])
@ -97,7 +97,7 @@ class CategoryApiTestCase(UserTestMixin, CategoryTestMixin, ToolshedTestCase):
self.prepare_categories() self.prepare_categories()
def test_get_categories(self): def test_get_categories(self):
response = client.get('/api/v1/categories/', self.f['local_user1']) response = client.get('/api/categories/', self.f['local_user1'])
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response.json(), self.assertEqual(response.json(),
["cat1", "cat2", "cat3", "cat1/subcat1", ["cat1", "cat2", "cat3", "cat1/subcat1",
@ -215,7 +215,7 @@ class TagApiTestCase(UserTestMixin, CategoryTestMixin, TagTestMixin, ToolshedTes
self.prepare_tags() self.prepare_tags()
def test_get_tags(self): def test_get_tags(self):
response = client.get('/api/v1/tags/', self.f['local_user1']) response = client.get('/api/tags/', self.f['local_user1'])
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response.json(), ["tag1", "tag2", "tag3"]) self.assertEqual(response.json(), ["tag1", "tag2", "tag3"])
@ -275,7 +275,7 @@ class PropertyApiTestCase(UserTestMixin, CategoryTestMixin, PropertyTestMixin, T
self.prepare_properties() self.prepare_properties()
def test_get_properties(self): def test_get_properties(self):
response = client.get('/api/v1/properties/', self.f['local_user1']) response = client.get('/api/properties/', self.f['local_user1'])
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(len(response.json()), 3) self.assertEqual(len(response.json()), 3)
self.assertEqual(response.json()[0]['name'], 'prop1') self.assertEqual(response.json()[0]['name'], 'prop1')

1
backend/requirements.txt
View file

@ -25,7 +25,6 @@ MarkupSafe==2.1.3
openapi-codec==1.3.2 openapi-codec==1.3.2
packaging==23.1 packaging==23.1
pycparser==2.21 pycparser==2.21
Pillow==10.4.0
PyNaCl==1.5.0 PyNaCl==1.5.0
python-dotenv==1.0.0 python-dotenv==1.0.0
pytz==2023.3 pytz==2023.3

59
backend/shared_data/base.json
View file

@ -1,59 +0,0 @@
{
"categories": [
{ "name": "hardware"},
{ "name": "material"},
{ "name": "tools"}
],
"properties": [
{ "name": "angle", "unit_symbol": "°", "unit_name": "degree", "unit_name_plural": "degrees" },
{ "name": "area", "unit_symbol": "m²", "unit_name": "square meter", "unit_name_plural": "square meters" },
{ "name": "current", "unit_symbol": "A", "unit_name": "ampere", "unit_name_plural": "amperes" },
{ "name": "diameter", "unit_symbol": "m", "unit_name": "meter", "unit_name_plural": "meters" },
{ "name": "energy", "unit_symbol": "J", "unit_name": "joule", "unit_name_plural": "joules" },
{ "name": "frequency", "unit_symbol": "Hz", "unit_name": "hertz", "unit_name_plural": "hertz" },
{ "name": "height", "unit_symbol": "m", "unit_name": "meter", "unit_name_plural": "meters" },
{ "name": "length", "unit_symbol": "m", "unit_name": "meter", "unit_name_plural": "meters" },
{ "name": "memory", "unit_symbol": "B", "unit_name": "byte", "unit_name_plural": "bytes", "base2_prefix": true },
{ "name": "power", "unit_symbol": "W", "unit_name": "watt", "unit_name_plural": "watts" },
{ "name": "price", "unit_symbol": "€", "unit_name": "euro", "unit_name_plural": "euros" },
{ "name": "speed", "unit_symbol": "m/s", "unit_name": "meter per second", "unit_name_plural": "meters per second" },
{ "name": "temperature", "unit_symbol": "°C", "unit_name": "degree Celsius", "unit_name_plural": "degrees Celsius" },
{ "name": "time", "unit_symbol": "s", "unit_name": "second", "unit_name_plural": "seconds" },
{ "name": "voltage", "unit_symbol": "V", "unit_name": "volt", "unit_name_plural": "volts" },
{ "name": "volume", "unit_symbol": "l", "unit_name": "liter", "unit_name_plural": "liters" },
{ "name": "weight", "unit_symbol": "g", "unit_name": "gram", "unit_name_plural": "grams" },
{ "name": "width", "unit_symbol": "m", "unit_name": "meter", "unit_name_plural": "meters" }
],
"tags": [
{"name": "bolt", "category": "hardware"},
{"name": "chisel", "category": "tools"},
{"name": "clamp", "category": "tools"},
{"name": "drill", "category": "tools"},
{"name": "ear plugs", "category": "tools"},
{"name": "extension cord", "category": "tools"},
{"name": "flashlight", "category": "tools"},
{"name": "gloves", "category": "tools"},
{"name": "goggles", "category": "tools"},
{"name": "hammer", "category": "tools"},
{"name": "level", "category": "tools"},
{"name": "mask", "category": "tools"},
{"name": "nail", "category": "hardware"},
{"name": "nut", "category": "hardware"},
{"name": "paint brush", "category": "tools"},
{"name": "paint roller", "category": "tools"},
{"name": "paint tray", "category": "tools"},
{"name": "pliers", "category": "tools"},
{"name": "power strip", "category": "tools"},
{"name": "sander", "category": "tools"},
{"name": "saw", "category": "tools"},
{"name": "screw", "category": "hardware"},
{"name": "screwdriver", "category": "tools"},
{"name": "soldering iron", "category": "tools"},
{"name": "stapler", "category": "tools"},
{"name": "tape measure", "category": "tools"},
{"name": "tool"},
{"name": "vise", "category": "tools"},
{"name": "washer", "category": "hardware"},
{"name": "wrench", "category": "tools"}
]
}

92
backend/shared_data/ee.json
View file

@ -1,92 +0,0 @@
{
"depends": [ "git:base" ],
"categories": [
{ "name": "electronics", "parent": "hardware"},
{ "name": "electronics", "parent": "tools"},
{ "name": "bus", "parent": "hardware/electronics"},
{ "name": "mcu", "parent": "hardware/electronics"},
{ "name": "wireless", "parent": "hardware/electronics"}
],
"tags": [
{"name": "smt", "category": "hardware/electronics"},
{"name": "tht", "category": "hardware/electronics"},
{"name": "adapter", "category": "hardware/electronics"},
{"name": "amperemeter", "category": "tools/electronics"},
{"name": "cable", "category": "hardware/electronics"},
{"name": "camera", "category": "tools/electronics"},
{"name": "connector", "category": "hardware/electronics"},
{"name": "flux", "category": "hardware/electronics"},
{"name": "microscope", "category": "tools/electronics"},
{"name": "multimeter", "category": "tools/electronics"},
{"name": "oscilloscope", "category": "tools/electronics"},
{"name": "power supply", "category": "hardware/electronics"},
{"name": "solder", "category": "hardware/electronics"},
{"name": "soldering", "category": "tools/electronics"},
{"name": "voltmeter", "category": "tools/electronics"},
{"name": "charger", "category": "hardware/electronics"},
{"name": "actuator", "category": "hardware/electronics"},
{"name": "battery", "category": "hardware/electronics"},
{"name": "capacitor", "category": "hardware/electronics"},
{"name": "diode", "category": "hardware/electronics"},
{"name": "display", "category": "hardware/electronics"},
{"name": "encoder", "category": "hardware/electronics"},
{"name": "fuse", "category": "hardware/electronics"},
{"name": "inductor", "category": "hardware/electronics"},
{"name": "inverter", "category": "hardware/electronics"},
{"name": "lcd", "category": "hardware/electronics"},
{"name": "led", "category": "hardware/electronics"},
{"name": "motor", "category": "hardware/electronics"},
{"name": "oscillator", "category": "hardware/electronics"},
{"name": "potentiometer", "category": "hardware/electronics"},
{"name": "relay", "category": "hardware/electronics"},
{"name": "resistor", "category": "hardware/electronics"},
{"name": "sensor", "category": "hardware/electronics"},
{"name": "servo", "category": "hardware/electronics"},
{"name": "stepper", "category": "hardware/electronics"},
{"name": "switch", "category": "hardware/electronics"},
{"name": "thermistor", "category": "hardware/electronics"},
{"name": "thermocouple", "category": "hardware/electronics"},
{"name": "transformer", "category": "hardware/electronics"},
{"name": "transistor", "category": "hardware/electronics"},
{"name": "bluetooth", "category": "hardware/electronics/wireless"},
{"name": "gps", "category": "hardware/electronics/wireless"},
{"name": "gsm", "category": "hardware/electronics/wireless"},
{"name": "lora", "category": "hardware/electronics/wireless"},
{"name": "nfc", "category": "hardware/electronics/wireless"},
{"name": "rfid", "category": "hardware/electronics/wireless"},
{"name": "thread", "category": "hardware/electronics/wireless"},
{"name": "wifi", "category": "hardware/electronics/wireless"},
{"name": "zigbee", "category": "hardware/electronics/wireless"},
{"name": "zwave", "category": "hardware/electronics/wireless"},
{"name": "can", "category": "hardware/electronics/bus"},
{"name": "ethernet", "category": "hardware/electronics/bus"},
{"name": "i2c", "category": "hardware/electronics/bus"},
{"name": "lin", "category": "hardware/electronics/bus"},
{"name": "spi", "category": "hardware/electronics/bus"},
{"name": "uart", "category": "hardware/electronics/bus"},
{"name": "usb", "category": "hardware/electronics/bus"},
{"name": "arduino", "category": "hardware/electronics/mcu"},
{"name": "atmega", "category": "hardware/electronics/mcu"},
{"name": "attiny", "category": "hardware/electronics/mcu"},
{"name": "beaglebone", "category": "hardware/electronics/mcu"},
{"name": "esp32", "category": "hardware/electronics/mcu"},
{"name": "esp8266", "category": "hardware/electronics/mcu"},
{"name": "nucleo", "category": "hardware/electronics/mcu"},
{"name": "raspberry", "category": "hardware/electronics/mcu"},
{"name": "stm32", "category": "hardware/electronics/mcu"},
{"name": "stm8", "category": "hardware/electronics/mcu"},
{"name": "6502", "category": "hardware/electronics/mcu"},
{"name": "8051", "category": "hardware/electronics/mcu"},
{"name": "arm", "category": "hardware/electronics/mcu"},
{"name": "avr", "category": "hardware/electronics/mcu"},
{"name": "cortex", "category": "hardware/electronics/mcu"},
{"name": "m68k", "category": "hardware/electronics/mcu"},
{"name": "mips", "category": "hardware/electronics/mcu"},
{"name": "pic", "category": "hardware/electronics/mcu"},
{"name": "powerpc", "category": "hardware/electronics/mcu"},
{"name": "risc-v", "category": "hardware/electronics/mcu"},
{"name": "x86", "category": "hardware/electronics/mcu"},
{"name": "xtensa", "category": "hardware/electronics/mcu"},
{"name": "z80", "category": "hardware/electronics/mcu"}
]
}

99
backend/shared_data/ee_packages.json
View file

@ -1,99 +0,0 @@
{
"depends": [ "git:ee" ],
"categories": [
{ "name": "packages", "parent": "hardware/electronics", "description": "Component Packages"}
],
"tags": [
{"name": "0201", "category": "hardware/electronics/packages"},
{"name": "0402", "category": "hardware/electronics/packages"},
{"name": "0603", "category": "hardware/electronics/packages"},
{"name": "0805", "category": "hardware/electronics/packages"},
{"name": "1206", "category": "hardware/electronics/packages"},
{"name": "1210", "category": "hardware/electronics/packages"},
{"name": "1812", "category": "hardware/electronics/packages"},
{"name": "MELF", "category": "hardware/electronics/packages"},
{"name": "MiniMELF", "category": "hardware/electronics/packages"},
{"name": "MicroMELF", "category": "hardware/electronics/packages"},
{"name": "SMC", "category": "hardware/electronics/packages"},
{"name": "SMB", "category": "hardware/electronics/packages"},
{"name": "SMA", "category": "hardware/electronics/packages"},
{"name": "GF1", "category": "hardware/electronics/packages"},
{"name": "DIP-x", "category": "hardware/electronics/packages",
"description": "Dual Inline Package, needs pin count property to be unambiguous"},
{"name": "SOD", "category": "hardware/electronics/packages"},
{"name": "SOD-123", "category": "hardware/electronics/packages"},
{"name": "SOD-323", "category": "hardware/electronics/packages"},
{"name": "SOD-523", "category": "hardware/electronics/packages"},
{"name": "SOD-923", "category": "hardware/electronics/packages"},
{"name": "SOT", "category": "hardware/electronics/packages"},
{"name": "SOT23", "category": "hardware/electronics/packages"},
{"name": "SOT23-3", "category": "hardware/electronics/packages"},
{"name": "SOT323", "category": "hardware/electronics/packages"},
{"name": "SOT416", "category": "hardware/electronics/packages"},
{"name": "SOT23-5", "category": "hardware/electronics/packages"},
{"name": "SOT353", "category": "hardware/electronics/packages"},
{"name": "SOT553", "category": "hardware/electronics/packages"},
{"name": "SOT23-6", "category": "hardware/electronics/packages"},
{"name": "SOT363", "category": "hardware/electronics/packages"},
{"name": "SOT563", "category": "hardware/electronics/packages"},
{"name": "SOT23-8", "category": "hardware/electronics/packages"},
{"name": "SOT54", "category": "hardware/electronics/packages", "alias":"TO-92"},
{"name": "SOT143", "category": "hardware/electronics/packages"},
{"name": "SOT343", "category": "hardware/electronics/packages"},
{"name": "SOT490", "category": "hardware/electronics/packages"},
{"name": "SOT89-3", "category": "hardware/electronics/packages"},
{"name": "SOT89-5", "category": "hardware/electronics/packages"},
{"name": "SOT223-4", "category": "hardware/electronics/packages"},
{"name": "SOT223-5", "category": "hardware/electronics/packages"},
{"name": "SOT223-8", "category": "hardware/electronics/packages"},
{"name": "TO-3", "category": "hardware/electronics/packages"},
{"name": "TO-5", "category": "hardware/electronics/packages"},
{"name": "TO-8", "category": "hardware/electronics/packages"},
{"name": "TO-18", "category": "hardware/electronics/packages"},
{"name": "TO-39", "category": "hardware/electronics/packages"},
{"name": "TO-66", "category": "hardware/electronics/packages"},
{"name": "TO-92", "category": "hardware/electronics/packages"},
{"name": "TO-220", "category": "hardware/electronics/packages"},
{"name": "TO-247", "category": "hardware/electronics/packages"},
{"name": "TO-251", "category": "hardware/electronics/packages"},
{"name": "TO-252", "category": "hardware/electronics/packages"},
{"name": "TO-263", "category": "hardware/electronics/packages"},
{"name": "TO-264", "category": "hardware/electronics/packages"},
{"name": "TO-268", "category": "hardware/electronics/packages"},
{"name": "TO-269", "category": "hardware/electronics/packages"},
{"name": "SOIC-x", "category": "hardware/electronics/packages", "alias": "SO-x",
"description": "Small Outline Integrated Circuit, needs pin count property to be unambiguous"},
{"name": "SOJ-x", "category": "hardware/electronics/packages",
"description": "Small Outline J-leaded, needs pin count property to be unambiguous"},
{"name": "MSOP-x", "category": "hardware/electronics/packages",
"description": "Mini Small Outline Package, needs pin count property to be unambiguous"},
{"name": "SSOP-x", "category": "hardware/electronics/packages",
"description": "Shrink Small Outline Package, needs pin count property to be unambiguous"},
{"name": "SOP-x", "category": "hardware/electronics/packages",
"description": "Small Outline Package, needs pin count property to be unambiguous"},
{"name": "TSOP-x", "category": "hardware/electronics/packages",
"description": "Thin Small Outline Package, needs pin count property to be unambiguous"},
{"name": "TSSOP-x", "category": "hardware/electronics/packages",
"description": "Thin Shrink Small Outline Package, needs pin count property to be unambiguous"},
{"name": "QFP-x", "category": "hardware/electronics/packages",
"description": "Quad Flat Package, needs pin count property to be unambiguous"},
{"name": "TQFP-x", "category": "hardware/electronics/packages",
"description": "Thin Quad Flat Package, needs pin count property to be unambiguous"},
{"name": "LQFP-x", "category": "hardware/electronics/packages",
"description": "Low-profile Quad Flat Package, needs pin count property to be unambiguous"},
{"name": "DFN-x", "category": "hardware/electronics/packages",
"description": "Dual Flat No-leaded, needs pin count property to be unambiguous"},
{"name": "QFN-x", "category": "hardware/electronics/packages",
"description": "Quad Flat No-leaded, needs pin count property to be unambiguous"},
{"name": "TQFN-x", "category": "hardware/electronics/packages",
"description": "Thin Quad Flat No-leaded, needs pin count property to be unambiguous"},
{"name": "LQFN-x", "category": "hardware/electronics/packages",
"description": "Low-profile Quad Flat No-leaded, needs pin count property to be unambiguous"},
{"name": "UQFN-x", "category": "hardware/electronics/packages",
"description": "Ultra-thin Quad Flat No-leaded, needs pin count property to be unambiguous"}
],
"properties": [
{ "name": "pin count", "unit_symbol": "", "unit_name": "", "unit_name_plural": "" }
],
"url": "https://en.wikipedia.org/wiki/List_of_integrated_circuit_packaging_types"
}

62
backend/shared_data/electrical.json
View file

@ -1,62 +0,0 @@
{
"depends": ["git:base"],
"categories": [
{ "name": "electrical"},
{ "name": "connectors"},
{ "name": "power", "parent": "connectors" }
],
"tags": [
{ "name": "braker", "category": "electrical" },
{ "name": "cable", "category": "electrical" },
{ "name": "connector", "category": "electrical" },
{ "name": "plug", "category": "connectors" },
{ "name": "socket", "category": "connectors" },
{ "name": "power", "category": "connectors" },
{ "name": "C1", "category": "connectors/power" },
{ "name": "C2", "category": "connectors/power" },
{ "name": "C3", "category": "connectors/power" },
{ "name": "C4", "category": "connectors/power" },
{ "name": "C5", "category": "connectors/power" },
{ "name": "C6", "category": "connectors/power" },
{ "name": "C7", "category": "connectors/power" },
{ "name": "C7P", "category": "connectors/power" },
{ "name": "C8", "category": "connectors/power" },
{ "name": "C8P", "category": "connectors/power" },
{ "name": "C9", "category": "connectors/power" },
{ "name": "C10", "category": "connectors/power" },
{ "name": "C11", "category": "connectors/power" },
{ "name": "C12", "category": "connectors/power" },
{ "name": "C13", "category": "connectors/power" },
{ "name": "C14", "category": "connectors/power" },
{ "name": "C15", "category": "connectors/power" },
{ "name": "C15A", "category": "connectors/power" },
{ "name": "C16", "category": "connectors/power" },
{ "name": "C16A", "category": "connectors/power" },
{ "name": "C17", "category": "connectors/power" },
{ "name": "C18", "category": "connectors/power" },
{ "name": "C19", "category": "connectors/power" },
{ "name": "C20", "category": "connectors/power" },
{ "name": "C21", "category": "connectors/power" },
{ "name": "C22", "category": "connectors/power" },
{ "name": "C23", "category": "connectors/power" },
{ "name": "C24", "category": "connectors/power" },
{ "name": "Type A", "category": "connectors/power", "description": "NEMA 1-15, U.S. 2 pin" },
{ "name": "Type B", "category": "connectors/power", "description": "NEMA 5-15, U.S. 3 pin" },
{ "name": "Type C", "category": "connectors/power", "description": "CEE 7/16, Europlug" },
{ "name": "Type D", "category": "connectors/power", "description": "BS 546, India 5A/15A" },
{ "name": "Type E", "category": "connectors/power", "description": "CEE 7/5, French 2 pin" },
{ "name": "Type F", "category": "connectors/power", "description": "CEE 7/4, Schuko" },
{ "name": "Type E/F", "category": "connectors/power", "description": "CEE 7/7, Schuko/French hybrid" },
{ "name": "Type G", "category": "connectors/power", "description": "BS 1363, U.K. 3 pin" },
{ "name": "Type H", "category": "connectors/power", "description": "SI 32 Israel"},
{ "name": "Type I", "category": "connectors/power", "description": "AS/NZS 3112, Australia 3 pin" },
{ "name": "Type J", "category": "connectors/power", "description": "SEV 1011, Swiss 3 pin" },
{ "name": "Type K", "category": "connectors/power", "description": "DS 60884-2-D1, Danish 3 pin" },
{ "name": "Type L", "category": "connectors/power", "description": "CEI 23-16/VII, Italian 3 pin" }
],
"properties": [
{ "name": "max current", "description": "Current rating"},
{ "name": "max power", "description": "Power rating" },
{ "name": "grid frequency", "description": "Typical frequency" }
]
}

82
backend/shared_data/it.json
View file

@ -1,82 +0,0 @@
{
"depends": [ "git:electrical" ],
"categories": [
{ "name": "pc", "description": "PC related" },
{ "name": "usb", "parent": "connectors" }
],
"tags": [
{ "name": "case", "category": "pc" },
{ "name": "cooler", "category": "pc" },
{ "name": "cpu" },
{ "name": "drone" },
{ "name": "fan", "category": "pc" },
{ "name": "gpu" },
{ "name": "hdd" },
{ "name": "headset" },
{ "name": "hub" },
{ "name": "iot" },
{ "name": "keyboard" },
{ "name": "laptop" },
{ "name": "memory" },
{ "name": "microphone" },
{ "name": "monitor" },
{ "name": "motherboard" },
{ "name": "mouse" },
{ "name": "pc"},
{ "name": "power supply" },
{ "name": "printer" },
{ "name": "router" },
{ "name": "scanner" },
{ "name": "server" },
{ "name": "smartphone" },
{ "name": "smartwatch" },
{ "name": "speaker" },
{ "name": "ssd" },
{ "name": "switch" },
{ "name": "tablet" },
{ "name": "watercooling" , "category": "pc"},
{ "name": "webcam" },
{ "name": "workstation" },
{ "name": "USB", "category": "connectors" },
{ "name": "Type A", "category": "connectors/usb" },
{ "name": "Type B", "category": "connectors/usb" },
{ "name": "Type C", "category": "connectors/usb" },
{ "name": "Micro", "category": "connectors/usb" },
{ "name": "Mini", "category": "connectors/usb" },
{ "name": "USB 2.0", "category": "connectors/usb" },
{ "name": "USB 3.0", "category": "connectors/usb" },
{ "name": "USB 3.1", "category": "connectors/usb" },
{ "name": "USB 3.2", "category": "connectors/usb" },
{ "name": "OTG", "category": "connectors/usb" },
{ "name": "thunderbolt", "category": "connectors/usb" },
{ "name": "24pin", "category": "connectors" },
{ "name": "8pin", "category": "connectors" },
{ "name": "atx", "category": "connectors" },
{ "name": "chinch", "category": "connectors" },
{ "name": "displayport", "category": "connectors" },
{ "name": "dvi", "category": "connectors" },
{ "name": "eps", "category": "connectors" },
{ "name": "floppy", "category": "connectors" },
{ "name": "hdmi", "category": "connectors" },
{ "name": "ide", "category": "connectors" },
{ "name": "jack", "category": "connectors" },
{ "name": "m.2", "category": "connectors" },
{ "name": "molex", "category": "connectors" },
{ "name": "p4", "category": "connectors" },
{ "name": "pcie", "category": "connectors" },
{ "name": "qsfp", "category": "connectors" },
{ "name": "qsfp+", "category": "connectors" },
{ "name": "qsfp28", "category": "connectors" },
{ "name": "rj11", "category": "connectors" },
{ "name": "rj45", "category": "connectors" },
{ "name": "sas", "category": "connectors" },
{ "name": "sata", "category": "connectors" },
{ "name": "scsi", "category": "connectors" },
{ "name": "sfp", "category": "connectors" },
{ "name": "sfp+", "category": "connectors" },
{ "name": "sfp28", "category": "connectors" },
{ "name": "toslink", "category": "connectors" },
{ "name": "vga", "category": "connectors" },
{ "name": "xlr", "category": "connectors" }
]
}

30
backend/shared_data/screws.json
View file

@ -1,30 +0,0 @@
{
"depends": [ "git:base" ],
"categories": [
{ "name": "screws", "parent": "hardware"}
],
"tags": [
{"name": "m1", "category": "screws"},
{"name": "m2", "category": "screws"},
{"name": "m2.5", "category": "screws"},
{"name": "m3", "category": "screws"},
{"name": "m4", "category": "screws"},
{"name": "m5", "category": "screws"},
{"name": "m6", "category": "screws"},
{"name": "m8", "category": "screws"},
{"name": "m10", "category": "screws"},
{"name": "m12", "category": "screws"},
{"name": "m16", "category": "screws"},
{"name": "torx", "category": "screws"},
{"name": "hex", "category": "screws"},
{"name": "phillips", "category": "screws"},
{"name": "pozidriv", "category": "screws"},
{"name": "slotted", "category": "screws"},
{"name": "socket", "category": "screws"},
{"name": "flat", "category": "screws"},
{"name": "pan", "category": "screws"},
{"name": "button", "category": "screws"},
{"name": "countersunk", "category": "screws"},
{"name": "round", "category": "screws"}
]
}

68
backend/shared_data/tools.json
View file

@ -1,68 +0,0 @@
{
"depends": ["git:base"],
"categories": [
{ "name": "powertools", "parent": "tools" }
],
"tags": [
{ "name": "3d printer"},
{ "name": "air compressor", "category": "powertools"},
{ "name": "air filter"},
{ "name": "automotive", "category": "tools" },
{ "name": "bandsaw"},
{ "name": "belt sander"},
{ "name": "bench grinder"},
{ "name": "circular saw", "category": "powertools"},
{ "name": "concrete", "category": "tools" },
{ "name": "construction", "category": "tools" },
{ "name": "corded", "category": "powertools" },
{ "name": "cordless", "category": "powertools" },
{ "name": "disc sander"},
{ "name": "drill press"},
{ "name": "drywall", "category": "tools" },
{ "name": "dust collector"},
{ "name": "dust mask"},
{ "name": "ear protection"},
{ "name": "electrical", "category": "tools" },
{ "name": "eye protection"},
{ "name": "fire extinguisher"},
{ "name": "first aid"},
{ "name": "generator", "category": "powertools"},
{ "name": "glue gun", "category": "powertools"},
{ "name": "grinder", "category": "powertools"},
{ "name": "hammer drill", "category": "powertools"},
{ "name": "handheld", "category": "tools" },
{ "name": "heat gun", "category": "powertools"},
{ "name": "impact driver", "category": "powertools"},
{ "name": "jigsaw"},
{ "name": "jointer"},
{ "name": "ladder"},
{ "name": "laser cutter"},
{ "name": "lathe"},
{ "name": "masonry", "category": "tools" },
{ "name": "mechanical", "category": "tools" },
{ "name": "metalworking", "category": "tools" },
{ "name": "mill"},
{ "name": "multitool"},
{ "name": "nailgun", "category": "powertools"},
{ "name": "oscillating tool", "category": "powertools"},
{ "name": "painting", "category": "tools" },
{ "name": "planer"},
{ "name": "plasma cutter"},
{ "name": "plumbing", "category": "tools" },
{ "name": "powerplane", "category": "powertools"},
{ "name": "pressure washer", "category": "powertools"},
{ "name": "roofing", "category": "tools" },
{ "name": "router table"},
{ "name": "router", "category": "powertools"},
{ "name": "sawhorse"},
{ "name": "sawzall", "category": "powertools"},
{ "name": "screwgun", "category": "powertools"},
{ "name": "stationary", "category": "tools" },
{ "name": "tablesaw"},
{ "name": "tile", "category": "tools" },
{ "name": "welder", "category": "powertools"},
{ "name": "woodworking", "category": "tools" },
{ "name": "workbench"},
{ "name": "worklight"}
]
}

61
backend/toolshed/admin.py
View file

@ -1,40 +1,11 @@
from django.contrib import admin from django.contrib import admin
from toolshed.models import ( from toolshed.models import InventoryItem, Property, Tag, Category
InventoryItem, ItemProperty, ItemTag, Property, Tag, Category, StorageLocation, WorkflowInstance,
)
class ItemTagInline(admin.TabularInline):
model = ItemTag
extra = 0
autocomplete_fields = ('tag',)
class ItemPropertyInline(admin.TabularInline):
model = ItemProperty
extra = 0
autocomplete_fields = ('property',)
class InventoryItemAdmin(admin.ModelAdmin): class InventoryItemAdmin(admin.ModelAdmin):
list_display = ('name', 'description', 'category', 'availability_policy', 'visibility_policy', 'owned_quantity', list_display = ('name', 'description', 'category', 'availability_policy', 'owned_quantity', 'owner')
'owner', 'owner_group', 'storage_location', 'get_tags', 'get_properties') search_fields = ('name', 'description', 'category', 'availability_policy', 'owned_quantity', 'owner')
search_fields = ('name', 'description', 'category__name', 'availability_policy', 'visibility_policy',
'owner__username', 'owner_group__name', 'storage_location__name', 'tags__name',
'itemproperty__property__name')
inlines = (ItemTagInline, ItemPropertyInline)
def get_queryset(self, request):
return super().get_queryset(request).prefetch_related('tags', 'itemproperty_set__property')
@admin.display(description='Tags')
def get_tags(self, obj):
return ', '.join(tag.name for tag in obj.tags.all())
@admin.display(description='Properties')
def get_properties(self, obj):
return ', '.join(f'{ip.property.name}={ip.value}' for ip in obj.itemproperty_set.all())
admin.site.register(InventoryItem, InventoryItemAdmin) admin.site.register(InventoryItem, InventoryItemAdmin)
@ -42,7 +13,7 @@ admin.site.register(InventoryItem, InventoryItemAdmin)
class PropertyAdmin(admin.ModelAdmin): class PropertyAdmin(admin.ModelAdmin):
list_display = ('name', 'description', 'category', 'unit_symbol', 'base2_prefix', 'dimensions', 'origin') list_display = ('name', 'description', 'category', 'unit_symbol', 'base2_prefix', 'dimensions', 'origin')
search_fields = ('name', 'description', 'category__name', 'unit_symbol', 'origin') search_fields = ('name', 'description', 'category', 'unit_symbol', 'base2_prefix', 'dimensions', 'origin')
admin.site.register(Property, PropertyAdmin) admin.site.register(Property, PropertyAdmin)
@ -50,7 +21,7 @@ admin.site.register(Property, PropertyAdmin)
class TagAdmin(admin.ModelAdmin): class TagAdmin(admin.ModelAdmin):
list_display = ('name', 'description', 'category', 'origin') list_display = ('name', 'description', 'category', 'origin')
search_fields = ('name', 'description', 'category__name', 'origin') search_fields = ('name', 'description', 'category', 'origin')
admin.site.register(Tag, TagAdmin) admin.site.register(Tag, TagAdmin)
@ -58,27 +29,7 @@ admin.site.register(Tag, TagAdmin)
class CategoryAdmin(admin.ModelAdmin): class CategoryAdmin(admin.ModelAdmin):
list_display = ('name', 'description', 'parent', 'origin') list_display = ('name', 'description', 'parent', 'origin')
search_fields = ('name', 'description', 'parent__name', 'origin') search_fields = ('name', 'description', 'parent', 'origin')
admin.site.register(Category, CategoryAdmin) admin.site.register(Category, CategoryAdmin)
class StorageLocationAdmin(admin.ModelAdmin):
list_display = ('name', 'description', 'category', 'parent', 'owner', 'visibility_policy')
search_fields = ('name', 'description', 'category__name', 'parent__name', 'owner__username',
'visibility_policy')
list_filter = ('category', 'owner', 'visibility_policy')
admin.site.register(StorageLocation, StorageLocationAdmin)
class WorkflowInstanceAdmin(admin.ModelAdmin):
list_display = ('slug', 'state', 'owner', 'created_at', 'updated_at')
search_fields = ('slug', 'owner__username')
list_filter = ('state', 'created_at', 'owner')
readonly_fields = ('created_at', 'updated_at')
admin.site.register(WorkflowInstance, WorkflowInstanceAdmin)

151
backend/toolshed/api/files.py
View file

@ -1,101 +1,49 @@
from django.db.models import Q
from django.urls import path from django.urls import path
from rest_framework import status from rest_framework import status
from rest_framework.decorators import api_view, permission_classes, authentication_classes from rest_framework.decorators import api_view, permission_classes, authentication_classes
from rest_framework.permissions import IsAuthenticated from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response from rest_framework.response import Response
from authentication.signature_auth import SignatureAuthentication, SignatureAuthenticationLocal from authentication.signature_auth import SignatureAuthenticationLocal
from files.models import File from files.models import File
from files.serializers import FileSerializer from files.serializers import FileSerializer
from toolshed.models import InventoryItem, WorkflowInstance from toolshed.models import InventoryItem
def _get_authorized_item(identity, item_id):
"""Owner-or-group-scoped item lookup; returns None if identity may not act on it."""
if identity.user.exists():
try:
return InventoryItem.objects.get(owner=identity.user.get(), id=item_id)
except InventoryItem.DoesNotExist:
pass
# Checked one group at a time, not owner_group__in=<all>, since id is only unique within one group's own items and a combined query could raise MultipleObjectsReturned on a collision.
for group in identity.member_of_groups.all():
item = InventoryItem.objects.filter(owner_group=group, id=item_id).first()
if item:
return item
return None
@api_view(['GET']) @api_view(['GET'])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthenticationLocal]) @authentication_classes([SignatureAuthenticationLocal])
def list_all_files(request, format=None): def list_all_files(request, format=None): # /files/
# request.user is a ToolshedUser here; reach group membership via public_identity. files = File.objects.select_related().filter(connected_items__owner=request.user).distinct()
files = File.objects.select_related().filter(
Q(connected_items__owner=request.user) |
Q(connected_items__owner_group__in=request.user.public_identity.member_of_groups.all()),
connected_items__is_deleted=False
).distinct()
return Response(FileSerializer(files, many=True).data) return Response(FileSerializer(files, many=True).data)
def get_item_files(request, item_id): def get_item_files(request, item_id):
item = _get_authorized_item(request.user, item_id) try:
if item is None: item = InventoryItem.objects.get(id=item_id, owner=request.user)
files = item.files.all()
return Response(FileSerializer(files, many=True).data)
except InventoryItem.DoesNotExist:
return Response(status=status.HTTP_404_NOT_FOUND) return Response(status=status.HTTP_404_NOT_FOUND)
files = item.files.all()
return Response(FileSerializer(files, many=True).data)
def post_item_file(request, item_id): def post_item_file(request, item_id):
item = _get_authorized_item(request.user, item_id)
if item is None:
return Response(status=status.HTTP_404_NOT_FOUND)
if 'file_hash' in request.data:
# Attaches an already-staged file by hash instead of re-uploading it. See docs/implementation.md#staged-files-are-identified-by-hash-alone.
if not request.user.user.exists():
return Response(status=status.HTTP_404_NOT_FOUND)
try:
file = File.objects.filter(hash=request.data['file_hash'],
staged_by_workflows__owner=request.user.user.get()).distinct().get()
except File.DoesNotExist:
return Response(status=status.HTTP_404_NOT_FOUND)
item.files.add(file)
return Response(FileSerializer(file).data, status=status.HTTP_201_CREATED)
serializer = FileSerializer(data=request.data)
if serializer.is_valid():
file = serializer.save()
item.files.add(file)
return Response(serializer.data, status=status.HTTP_201_CREATED)
return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
def get_staged_files(request, workflow_id):
try: try:
workflow = WorkflowInstance.objects.get(id=workflow_id, owner=request.user) item = InventoryItem.objects.get(id=item_id, owner=request.user)
# Hash alone is enough to discover what another session/device already staged. See docs/implementation.md#staged-files-are-identified-by-hash-alone.
return Response(list(workflow.staged_files.values_list('hash', flat=True)))
except WorkflowInstance.DoesNotExist:
return Response(status=status.HTTP_404_NOT_FOUND)
def post_staged_file(request, workflow_id):
try:
workflow = WorkflowInstance.objects.get(id=workflow_id, owner=request.user)
serializer = FileSerializer(data=request.data) serializer = FileSerializer(data=request.data)
if serializer.is_valid(): if serializer.is_valid():
file = serializer.save() file = serializer.save()
workflow.staged_files.add(file) item.files.add(file)
return Response({'hash': file.hash}, status=status.HTTP_201_CREATED) return Response(serializer.data, status=status.HTTP_201_CREATED)
return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST) return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
except WorkflowInstance.DoesNotExist: except InventoryItem.DoesNotExist:
return Response(status=status.HTTP_404_NOT_FOUND) return Response(status=status.HTTP_404_NOT_FOUND)
@api_view(['POST', 'GET']) @api_view(['POST', 'GET'])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication]) @authentication_classes([SignatureAuthenticationLocal])
def item_files(request, item_id, format=None): def item_files(request, item_id, format=None): # /item_files/
if request.method == 'GET': if request.method == 'GET':
return get_item_files(request, item_id) return get_item_files(request, item_id)
elif request.method == 'POST': elif request.method == 'POST':
@ -105,71 +53,15 @@ def item_files(request, item_id, format=None):
@api_view(['DELETE']) @api_view(['DELETE'])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthenticationLocal]) @authentication_classes([SignatureAuthenticationLocal])
def delete_file(request, file_id, format=None): def delete_item_file(request, item_id, file_id, format=None): # /item_files/
try:
file = File.objects.get(id=file_id)
except File.DoesNotExist:
return Response(status=status.HTTP_404_NOT_FOUND)
# Only detach from items this identity is actually authorized to act on - a file is
# content-addressed and can be shared by other users'/groups' items via the same hash,
# so it must never be removed from connections this request has no authority over.
authorized_items = file.connected_items.filter(
Q(owner=request.user) | Q(owner_group__in=request.user.public_identity.member_of_groups.all())
)
if not authorized_items.exists():
return Response(status=status.HTTP_404_NOT_FOUND)
for item in authorized_items:
item.files.remove(file)
if file.connected_items.count() == 0 and file.profile_picture_users.count() == 0 \
and file.staged_by_workflows.count() == 0:
file.file.delete(save=False)
file.delete()
return Response(status=status.HTTP_204_NO_CONTENT)
@api_view(['DELETE'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication])
def delete_item_file(request, item_id, file_id, format=None):
item = _get_authorized_item(request.user, item_id)
if item is None:
return Response(status=status.HTTP_404_NOT_FOUND)
try: try:
item = InventoryItem.objects.get(id=item_id, owner=request.user)
file = item.files.get(id=file_id) file = item.files.get(id=file_id)
except File.DoesNotExist: item.files.remove(file_id)
return Response(status=status.HTTP_404_NOT_FOUND) if file.connected_items.count() == 0:
item.files.remove(file_id)
if file.connected_items.count() == 0 and file.profile_picture_users.count() == 0 \
and file.staged_by_workflows.count() == 0:
file.file.delete(save=False)
file.delete()
return Response(status=status.HTTP_204_NO_CONTENT)
@api_view(['POST', 'GET'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthenticationLocal])
def staged_files(request, workflow_id, format=None):
if request.method == 'GET':
return get_staged_files(request, workflow_id)
elif request.method == 'POST':
return post_staged_file(request, workflow_id)
@api_view(['DELETE'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthenticationLocal])
def delete_staged_file(request, workflow_id, file_hash, format=None):
try:
workflow = WorkflowInstance.objects.get(id=workflow_id, owner=request.user)
file = workflow.staged_files.get(hash=file_hash)
workflow.staged_files.remove(file)
if file.connected_items.count() == 0 and file.profile_picture_users.count() == 0 \
and file.staged_by_workflows.count() == 0:
file.file.delete(save=False)
file.delete() file.delete()
return Response(status=status.HTTP_204_NO_CONTENT) return Response(status=status.HTTP_204_NO_CONTENT)
except WorkflowInstance.DoesNotExist: except InventoryItem.DoesNotExist:
return Response(status=status.HTTP_404_NOT_FOUND) return Response(status=status.HTTP_404_NOT_FOUND)
except File.DoesNotExist: except File.DoesNotExist:
return Response(status=status.HTTP_404_NOT_FOUND) return Response(status=status.HTTP_404_NOT_FOUND)
@ -177,9 +69,6 @@ def delete_staged_file(request, workflow_id, file_hash, format=None):
urlpatterns = [ urlpatterns = [
path('files/', list_all_files), path('files/', list_all_files),
path('files/<int:file_id>/', delete_file),
path('item_files/<int:item_id>/', item_files), path('item_files/<int:item_id>/', item_files),
path('item_files/<int:item_id>/<int:file_id>/', delete_item_file), path('item_files/<int:item_id>/<int:file_id>/', delete_item_file),
path('staged_files/<int:workflow_id>/', staged_files),
path('staged_files/<int:workflow_id>/<str:file_hash>/', delete_staged_file),
] ]

12
backend/toolshed/api/friend.py
View file

@ -19,13 +19,13 @@ class Friends(APIView, ViewSetMixin):
authentication_classes = [SignatureAuthentication] authentication_classes = [SignatureAuthentication]
permission_classes = [IsAuthenticated] permission_classes = [IsAuthenticated]
def get(self, request, format=None): def get(self, request, format=None): # /api/friends/ #
user = request.user user = request.user
friends = user.friends.all() friends = user.friends.all()
serializer = FriendSerializer(friends, many=True) serializer = FriendSerializer(friends, many=True)
return Response(serializer.data) return Response(serializer.data)
def post(self, request, format=None): def post(self, request, format=None): # /api/friends/
# only for local users # only for local users
try: try:
user = request.user user = request.user
@ -47,7 +47,7 @@ class Friends(APIView, ViewSetMixin):
class FriendsRequests(APIView, ViewSetMixin): class FriendsRequests(APIView, ViewSetMixin):
def get(self, request, format=None): def get(self, request, format=None): # /api/friendrequests/
raw_request = request.body.decode('utf-8') raw_request = request.body.decode('utf-8')
if user := authenticate_request_against_local_users(request, raw_request): if user := authenticate_request_against_local_users(request, raw_request):
friends_requests = user.friend_requests_incoming.all() friends_requests = user.friend_requests_incoming.all()
@ -56,7 +56,7 @@ class FriendsRequests(APIView, ViewSetMixin):
else: else:
return Response(status=status.HTTP_401_UNAUTHORIZED, data={'status': 'unauthorized'}) return Response(status=status.HTTP_401_UNAUTHORIZED, data={'status': 'unauthorized'})
def post(self, request, format=None): def post(self, request, format=None): # /api/friendrequests/
raw_request = request.body.decode('utf-8') raw_request = request.body.decode('utf-8')
if 'befriender' not in request.data or 'befriendee' not in request.data: if 'befriender' not in request.data or 'befriendee' not in request.data:
return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': 'missing parameters'}) return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': 'missing parameters'})
@ -131,7 +131,7 @@ class FriendsRequests(APIView, ViewSetMixin):
@api_view(['DELETE']) @api_view(['DELETE'])
@authentication_classes([SignatureAuthenticationLocal]) @authentication_classes([SignatureAuthenticationLocal])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
def dropFriend(request, pk, format=None): def dropFriend(request, pk, format=None): # /api/friends/<id>/
user = request.user user = request.user
friend = get_object_or_404(user.friends, pk=pk) friend = get_object_or_404(user.friends, pk=pk)
user.friends.remove(friend) user.friends.remove(friend)
@ -142,7 +142,7 @@ def dropFriend(request, pk, format=None):
@api_view(['DELETE']) @api_view(['DELETE'])
@authentication_classes([SignatureAuthenticationLocal]) @authentication_classes([SignatureAuthenticationLocal])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
def deleteFriendRequest(request, pk, format=None): def deleteFriendRequest(request, pk, format=None): # /api/friendrequests/<id>/
user = request.user user = request.user
get_object_or_404(user.friend_requests_incoming, pk=pk).delete() get_object_or_404(user.friend_requests_incoming, pk=pk).delete()
user.save() user.save()

197
backend/toolshed/api/group.py
View file

@ -1,197 +0,0 @@
import secrets
from django.urls import path
from rest_framework import status
from rest_framework.decorators import api_view, authentication_classes, permission_classes
from rest_framework.generics import get_object_or_404
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from rest_framework.views import APIView
from rest_framework.viewsets import ViewSetMixin
from authentication.models import Group, GroupInvite, GroupInviteIncoming, GroupMembership, KnownIdentity, \
ToolshedUser
from authentication.signature_auth import SignatureAuthentication, SignatureAuthenticationLocal, \
authenticate_request_against_local_users, split_grouphandle_or_throw, split_userhandle_or_throw, \
verify_incoming_group_invite
from toolshed.serializers import GroupSerializer, GroupInviteIncomingSerializer, GroupMembershipSerializer
class Groups(APIView, ViewSetMixin):
authentication_classes = [SignatureAuthentication]
permission_classes = [IsAuthenticated]
def get(self, request, format=None):
groups = request.user.member_of_groups.all()
return Response(GroupSerializer(groups, many=True).data)
def post(self, request, format=None):
name = request.data.get('name')
if not name:
return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': 'name is required'})
if Group.objects.filter(name=name, domain=request.user.domain).exists():
return Response(status=status.HTTP_409_CONFLICT, data={'status': 'a group with this name already exists'})
group = Group.objects.create(name=name, domain=request.user.domain)
group.members.add(request.user)
return Response(status=status.HTTP_201_CREATED, data=GroupSerializer(group).data)
class GroupDetail(APIView, ViewSetMixin):
authentication_classes = [SignatureAuthentication]
permission_classes = [IsAuthenticated]
def get(self, request, handle, format=None):
try:
name, domain = split_userhandle_or_throw(handle)
except ValueError:
return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': 'invalid group handle'})
group = get_object_or_404(Group, name=name, domain=domain)
if not group.is_member(request.user):
return Response(status=status.HTTP_404_NOT_FOUND)
return Response(GroupSerializer(group).data)
@api_view(['DELETE'])
@authentication_classes([SignatureAuthentication])
@permission_classes([IsAuthenticated])
def removeGroupMember(request, handle, identity_id, format=None):
try:
name, domain = split_userhandle_or_throw(handle)
except ValueError:
return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': 'invalid group handle'})
group = get_object_or_404(Group, name=name, domain=domain)
if not group.is_member(request.user):
return Response(status=status.HTTP_404_NOT_FOUND)
member = get_object_or_404(group.members, pk=identity_id)
if group.members.count() <= 1:
return Response(status=status.HTTP_400_BAD_REQUEST,
data={'status': "cannot remove the group's last member"})
group.members.remove(member)
return Response(status=status.HTTP_204_NO_CONTENT)
@api_view(['POST'])
@authentication_classes([SignatureAuthentication])
@permission_classes([IsAuthenticated])
def createGroupInvite(request, handle, format=None):
try:
name, domain = split_userhandle_or_throw(handle)
except ValueError:
return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': 'invalid group handle'})
group = get_object_or_404(Group, name=name, domain=domain)
if not group.is_member(request.user):
return Response(status=status.HTTP_404_NOT_FOUND)
invitee = request.data.get('invitee')
if not invitee:
return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': 'invitee is required'})
try:
invitee_username, invitee_domain = split_userhandle_or_throw(invitee)
except ValueError:
return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': 'invalid invitee handle'})
if group.members.filter(username=invitee_username, domain=invitee_domain).exists():
return Response(status=status.HTTP_208_ALREADY_REPORTED, data={'status': 'already a member'})
secret = secrets.token_hex(64)
GroupInvite.objects.create(group=group, invitee_username=invitee_username, invitee_domain=invitee_domain,
secret=secret)
return Response(status=status.HTTP_201_CREATED, data={'secret': secret, 'status': 'pending'})
class GroupInvitesIncoming(APIView, ViewSetMixin):
def get(self, request, format=None):
raw_request = request.body.decode('utf-8')
if not (user := authenticate_request_against_local_users(request, raw_request)):
return Response(status=status.HTTP_401_UNAUTHORIZED, data={'status': 'unauthorized'})
invites = user.group_invites_incoming.all()
return Response(GroupInviteIncomingSerializer(invites, many=True).data)
def post(self, request, format=None):
raw_request = request.body.decode('utf-8')
for field in ('group', 'inviter', 'inviter_key', 'invitee', 'secret'):
if field not in request.data:
return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': f'missing {field}'})
if not verify_incoming_group_invite(request, raw_request, 'inviter', 'inviter_key'):
return Response(status=status.HTTP_401_UNAUTHORIZED, data={'status': 'unauthorized'})
try:
group_name, group_domain = split_grouphandle_or_throw(request.data['group'])
inviter_username, inviter_domain = split_userhandle_or_throw(request.data['inviter'])
invitee_username, invitee_domain = split_userhandle_or_throw(request.data['invitee'])
except ValueError:
return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': 'invalid handle'})
try:
invitee_user = ToolshedUser.objects.get(username=invitee_username, domain=invitee_domain)
except ToolshedUser.DoesNotExist:
return Response(status=status.HTTP_404_NOT_FOUND, data={'status': 'invitee is not local to this server'})
GroupInviteIncoming.objects.create(
group_name=group_name, group_domain=group_domain,
inviter_username=inviter_username, inviter_domain=inviter_domain,
invitee_user=invitee_user, secret=request.data['secret'])
return Response(status=status.HTTP_201_CREATED, data={'status': 'delivered'})
@api_view(['DELETE'])
@authentication_classes([SignatureAuthenticationLocal])
@permission_classes([IsAuthenticated])
def declineGroupInvite(request, pk, format=None):
get_object_or_404(request.user.group_invites_incoming, pk=pk).delete()
return Response(status=status.HTTP_204_NO_CONTENT)
@api_view(['POST'])
@authentication_classes([SignatureAuthenticationLocal])
@permission_classes([IsAuthenticated])
def recordGroupMembership(request, pk, format=None):
invite = get_object_or_404(request.user.group_invites_incoming, pk=pk)
membership, _ = GroupMembership.objects.get_or_create(
user=request.user, group_name=invite.group_name, group_domain=invite.group_domain)
invite.delete()
return Response(status=status.HTTP_201_CREATED, data=GroupMembershipSerializer(membership).data)
class GroupMemberships(APIView, ViewSetMixin):
authentication_classes = [SignatureAuthenticationLocal]
permission_classes = [IsAuthenticated]
def get(self, request, format=None):
memberships = request.user.group_memberships.all()
return Response(GroupMembershipSerializer(memberships, many=True).data)
@api_view(['POST'])
def acceptGroupInvite(request, format=None):
raw_request = request.body.decode('utf-8')
for field in ('group', 'invitee', 'invitee_key', 'secret'):
if field not in request.data:
return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': f'missing {field}'})
if not verify_incoming_group_invite(request, raw_request, 'invitee', 'invitee_key'):
return Response(status=status.HTTP_401_UNAUTHORIZED, data={'status': 'unauthorized'})
try:
group_name, group_domain = split_grouphandle_or_throw(request.data['group'])
invitee_username, invitee_domain = split_userhandle_or_throw(request.data['invitee'])
except ValueError:
return Response(status=status.HTTP_400_BAD_REQUEST, data={'status': 'invalid handle'})
try:
group = Group.objects.get(name=group_name, domain=group_domain)
except Group.DoesNotExist:
return Response(status=status.HTTP_404_NOT_FOUND, data={'status': 'no such group here'})
invite = GroupInvite.objects.filter(group=group, invitee_username=invitee_username,
invitee_domain=invitee_domain, secret=request.data['secret'])
if not invite.exists():
return Response(status=status.HTTP_404_NOT_FOUND, data={'status': 'no matching invite'})
identity, _ = KnownIdentity.objects.get_or_create(
username=invitee_username, domain=invitee_domain, public_key=request.data['invitee_key'])
group.members.add(identity)
invite.delete()
return Response(status=status.HTTP_201_CREATED, data={'status': 'accepted'})
urlpatterns = [
path('groups/', Groups.as_view(), name='groups'),
path('groups/<str:handle>/', GroupDetail.as_view(), name='group_detail'),
path('groups/<str:handle>/members/<int:identity_id>/', removeGroupMember, name='remove_group_member'),
path('groups/<str:handle>/invites/', createGroupInvite, name='create_group_invite'),
path('groupinvites/', GroupInvitesIncoming.as_view(), name='group_invites_incoming'),
path('groupinvites/<int:pk>/', declineGroupInvite, name='decline_group_invite'),
path('groupinvites/<int:pk>/accept/', recordGroupMembership, name='record_group_membership'),
path('group_invites/accept/', acceptGroupInvite, name='accept_group_invite'),
path('groupmemberships/', GroupMemberships.as_view(), name='group_memberships'),
]

70
backend/toolshed/api/idmap.py
View file

@ -1,70 +0,0 @@
from django.urls import path
from rest_framework.decorators import api_view, authentication_classes, permission_classes
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from rest_framework.views import APIView
from rest_framework.viewsets import ViewSetMixin
from authentication.models import KnownIdentity, ToolshedUser, Group
from authentication.signature_auth import SignatureAuthentication
from toolshed.models import InventoryItem, StorageLocation
from toolshed.serializers import FriendSerializer, GroupIdMapSerializer
class IdMap(APIView, ViewSetMixin):
authentication_classes = [SignatureAuthentication]
permission_classes = [IsAuthenticated]
def get(self, request, format=None):
identity = request.user
identities = identity.friends.all() | KnownIdentity.objects.filter(pk=identity.pk)
groups = identity.member_of_groups.all()
return Response({
'identities': FriendSerializer(identities, many=True).data,
'groups': GroupIdMapSerializer(groups, many=True).data,
})
@api_view(['GET'])
@authentication_classes([SignatureAuthentication])
@permission_classes([IsAuthenticated])
def resolve_short_id(request, kind, owner_id, local_id):
"""Resolves a short id's (kind, owner_id, local_id) against this backend's own numbering. See docs/implementation.md#domain-qualified-short-id-resolution."""
if kind in ('item', 'storage_location'):
try:
owner = KnownIdentity.objects.get(pk=owner_id).user.get()
except (KnownIdentity.DoesNotExist, ToolshedUser.DoesNotExist):
return Response(status=404)
if owner not in request.user.friends_or_self():
return Response(status=403)
model = InventoryItem if kind == 'item' else StorageLocation
try:
obj = model.objects.get(owner=owner, id=local_id)
except model.DoesNotExist:
return Response(status=404)
is_owner = request.user.user.filter(pk=owner.pk).exists()
if not is_owner and (getattr(obj, 'availability_policy', 'share') == 'private'
or obj.visibility_policy == 'private'):
return Response(status=403)
return Response({'handle': f'{owner.username}@{owner.domain}', 'id': obj.id})
if kind in ('group_item', 'group_storage_location'):
try:
group = Group.objects.get(pk=owner_id)
except Group.DoesNotExist:
return Response(status=404)
if not group.is_member(request.user):
return Response(status=403)
model = InventoryItem if kind == 'group_item' else StorageLocation
try:
obj = model.objects.get(owner_group=group, id=local_id)
except model.DoesNotExist:
return Response(status=404)
return Response({'handle': str(group), 'id': obj.id})
return Response(status=400)
urlpatterns = [
path('idmap/', IdMap.as_view(), name='idmap'),
path('resolve_short_id/<str:kind>/<int:owner_id>/<int:local_id>/', resolve_short_id,
name='resolve_short_id'),
]

35
backend/toolshed/api/info.py
View file

@ -7,12 +7,20 @@ from hostadmin.models import Domain
from authentication.signature_auth import SignatureAuthentication from authentication.signature_auth import SignatureAuthentication
from toolshed.models import Tag, Property, Category, InventoryItem from toolshed.models import Tag, Property, Category, InventoryItem
from toolshed.serializers import CategorySerializer, PropertySerializer from toolshed.serializers import CategorySerializer, PropertySerializer
from backend.settings import TOOLSHED_VERSION
@api_view(['GET']) @api_view(['GET'])
@permission_classes([]) @permission_classes([])
@authentication_classes([]) @authentication_classes([])
def list_domains(request, format=None): def get_version(request, format=None): # /version/
return Response({'version': TOOLSHED_VERSION})
@api_view(['GET'])
@permission_classes([])
@authentication_classes([])
def list_domains(request, format=None): # /domains/
domains = [domain.name for domain in Domain.objects.filter(open_registration=True)] domains = [domain.name for domain in Domain.objects.filter(open_registration=True)]
return Response(domains) return Response(domains)
@ -20,7 +28,7 @@ def list_domains(request, format=None):
@api_view(['GET']) @api_view(['GET'])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication]) @authentication_classes([SignatureAuthentication])
def list_tags(format=None): def list_tags(format=None): # /tags/
tags = [tag.name for tag in Tag.objects.all()] tags = [tag.name for tag in Tag.objects.all()]
return Response(tags) return Response(tags)
@ -28,50 +36,41 @@ def list_tags(format=None):
@api_view(['GET']) @api_view(['GET'])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication]) @authentication_classes([SignatureAuthentication])
def list_properties(request, format=None): def list_properties(request, format=None): # /properties/
return Response(PropertySerializer(Property.objects.all(), many=True).data) return Response(PropertySerializer(Property.objects.all(), many=True).data)
@api_view(['GET']) @api_view(['GET'])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication]) @authentication_classes([SignatureAuthentication])
def list_categories(request, format=None): def list_categories(request, format=None): # /categories/
return Response(CategorySerializer(Category.objects.all(), many=True).data) return Response(CategorySerializer(Category.objects.all(), many=True).data)
@api_view(['GET']) @api_view(['GET'])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication]) @authentication_classes([SignatureAuthentication])
def list_availability_policies(request, format=None): def list_availability_policies(request, format=None): # /availability_policies/
return Response(InventoryItem.AVAILABILITY_POLICY_CHOICES) return Response(InventoryItem.AVAILABILITY_POLICY_CHOICES)
@api_view(['GET']) @api_view(['GET'])
@permission_classes([IsAuthenticated]) @permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication]) @authentication_classes([SignatureAuthentication])
def list_visibility_policies(request, format=None): def combined_info(request, format=None): # /info/
return Response(InventoryItem.VISIBILITY_POLICY_CHOICES)
@api_view(['GET'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication])
def combined_info(request, format=None):
tags = [tag.name for tag in Tag.objects.all()] tags = [tag.name for tag in Tag.objects.all()]
properties = PropertySerializer(Property.objects.all(), many=True).data properties = PropertySerializer(Property.objects.all(), many=True).data
categories = [str(category) for category in Category.objects.all()] categories = [str(category) for category in Category.objects.all()]
policies = InventoryItem.AVAILABILITY_POLICY_CHOICES policies = InventoryItem.AVAILABILITY_POLICY_CHOICES
visibility_policies = InventoryItem.VISIBILITY_POLICY_CHOICES
domains = [domain.name for domain in Domain.objects.filter(open_registration=True)] domains = [domain.name for domain in Domain.objects.filter(open_registration=True)]
return Response( return Response(
{'tags': tags, 'properties': properties, 'availability_policies': policies, {'tags': tags, 'properties': properties, 'availability_policies': policies, 'categories': categories,
'visibility_policies': visibility_policies, 'categories': categories, 'domains': domains}) 'domains': domains})
urlpatterns = [ urlpatterns = [
path('version/', get_version, name='version'),
path('availability_policies/', list_availability_policies, name='availability_policies'), path('availability_policies/', list_availability_policies, name='availability_policies'),
path('visibility_policies/', list_visibility_policies, name='visibility_policies'),
path('properties/', list_properties, name='propertylist'), path('properties/', list_properties, name='propertylist'),
path('categories/', list_categories, name='categorylist'), path('categories/', list_categories, name='categorylist'),
path('domains/', list_domains, name='domainlist'), path('domains/', list_domains, name='domainlist'),

204
backend/toolshed/api/inventory.py
View file

@ -1,16 +1,16 @@
from django.db import transaction from django.db import transaction
from django.urls import path from django.urls import path
from rest_framework import routers, viewsets, status from rest_framework import routers, viewsets
from rest_framework.decorators import authentication_classes, api_view, permission_classes, action from rest_framework.decorators import authentication_classes, api_view, permission_classes
from rest_framework.exceptions import NotFound, PermissionDenied, ValidationError
from rest_framework.permissions import IsAuthenticated from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response from rest_framework.response import Response
from authentication.models import ToolshedUser, KnownIdentity from authentication.models import ToolshedUser, KnownIdentity
from authentication.signature_auth import SignatureAuthentication, resolve_owner_handle from authentication.signature_auth import SignatureAuthentication
from files.models import File from toolshed.models import InventoryItem, StorageLocation
from toolshed.models import InventoryItem, StorageLocation, WorkflowInstance from toolshed.serializers import InventoryItemSerializer, StorageLocationSerializer
from toolshed.serializers import InventoryItemSerializer, StorageLocationSerializer, WorkflowInstanceSerializer
router = routers.SimpleRouter()
def inventory_items(identity): def inventory_items(identity):
@ -22,10 +22,9 @@ def inventory_items(identity):
except ToolshedUser.DoesNotExist: except ToolshedUser.DoesNotExist:
pass pass
for friend in identity.friends.all(): for friend in identity.friends.all():
friend_user = friend.user.first() if friend_user := friend.user.get():
if friend_user:
for item in friend_user.inventory_items.all(): for item in friend_user.inventory_items.all():
if item.availability_policy != 'private' and item.visibility_policy != 'private': if item.availability_policy != 'private':
yield item yield item
@ -33,83 +32,24 @@ class InventoryItemViewSet(viewsets.ModelViewSet):
serializer_class = InventoryItemSerializer serializer_class = InventoryItemSerializer
authentication_classes = [SignatureAuthentication] authentication_classes = [SignatureAuthentication]
permission_classes = [IsAuthenticated] permission_classes = [IsAuthenticated]
# Detail routes address an item by its owner-scoped id, not the internal row id. See docs/implementation.md#owner-handle-scoped-routes.
lookup_field = 'id'
lookup_url_kwarg = 'pk'
def get_queryset(self): def get_queryset(self):
# Every route is scoped by the owner handle URL param (own/friend/group); no handle-less fallback exists. See docs/implementation.md#owner-handle-scoped-routes. if type(self.request.user) == KnownIdentity and self.request.user.user.exists():
if type(self.request.user) != KnownIdentity: return InventoryItem.objects.filter(owner=self.request.user.user.get())
return InventoryItem.objects.none() return InventoryItem.objects.none()
identity = self.request.user
try:
owner_user, owner_group = resolve_owner_handle(self.kwargs.get('handle', ''))
except ValueError:
raise ValidationError('invalid owner handle')
if owner_group:
if not owner_group.is_member(identity):
return InventoryItem.objects.none()
queryset = InventoryItem.objects.filter(owner_group=owner_group)
elif owner_user:
if owner_user not in identity.friends_or_self():
return InventoryItem.objects.none()
queryset = InventoryItem.objects.filter(owner=owner_user)
if not identity.user.filter(pk=owner_user.pk).exists():
queryset = queryset.exclude(availability_policy='private').exclude(visibility_policy='private')
else:
return InventoryItem.objects.none()
# InventoryItemSerializer touches owner/owner_group/category/storage_location (FKs) and
# tags/files/itemproperty_set (M2M/reverse-FK) for every row - without this, listing N
# items costs ~5N extra queries (each a full network round trip once every other API
# call also goes through the federation ServerSet layer, not just local Django).
return queryset.select_related('owner', 'owner_group', 'category', 'storage_location').prefetch_related(
'tags', 'files', 'itemproperty_set__property')
def perform_create(self, serializer): def perform_create(self, serializer):
try:
owner_user, owner_group = resolve_owner_handle(self.kwargs.get('handle', ''))
except ValueError:
raise ValidationError('invalid owner handle')
with transaction.atomic(): with transaction.atomic():
if owner_group: serializer.save(owner=self.request.user.user.get()).clean()
if not owner_group.is_member(self.request.user):
raise PermissionDenied('Not a member of this group')
serializer.save(owner=None, owner_group=owner_group).clean()
elif owner_user:
if not self.request.user.user.filter(pk=owner_user.pk).exists():
raise PermissionDenied('Cannot create items for another user')
serializer.save(owner=owner_user).clean()
else:
raise NotFound('No such owner')
@staticmethod
def _is_authorized(request, instance):
if instance.owner_id:
return request.user.user.filter(pk=instance.owner_id).exists()
return instance.owner_group.is_member(request.user)
def perform_update(self, serializer): def perform_update(self, serializer):
# get_queryset() may return a friend's read-only item; reject explicitly rather than silently no-op. See docs/implementation.md#perform-update-rejects-non-owned-items-explicitly.
if not self._is_authorized(self.request, serializer.instance):
raise PermissionDenied('Not authorized to modify this item')
with transaction.atomic(): with transaction.atomic():
serializer.save().clean() if serializer.instance.owner == self.request.user.user.get():
serializer.save().clean()
def perform_destroy(self, instance): def perform_destroy(self, instance):
if not self._is_authorized(self.request, instance): if instance.owner == self.request.user.user.get():
raise PermissionDenied('Not authorized to delete this item') instance.delete()
instance.delete()
def matches_query(item, query):
query = query.lower()
if query in item.name.lower():
return True
if item.description and query in item.description.lower():
return True
if any(query in tag.name.lower() for tag in item.tags.all()):
return True
return False
@api_view(['GET']) @api_view(['GET'])
@ -118,8 +58,7 @@ def matches_query(item, query):
def search_inventory_items(request): def search_inventory_items(request):
query = request.query_params.get('query') query = request.query_params.get('query')
if query: if query:
matching_items = [item for item in inventory_items(request.user) if matches_query(item, query)] return Response(InventoryItemSerializer(inventory_items(request.user), many=True).data)
return Response(InventoryItemSerializer(matching_items, many=True).data)
return Response({'error': 'No query provided.'}, status=400) return Response({'error': 'No query provided.'}, status=400)
@ -127,114 +66,15 @@ class StorageLocationViewSet(viewsets.ModelViewSet):
serializer_class = StorageLocationSerializer serializer_class = StorageLocationSerializer
authentication_classes = [SignatureAuthentication] authentication_classes = [SignatureAuthentication]
permission_classes = [IsAuthenticated] permission_classes = [IsAuthenticated]
# Detail routes address a location by its owner-scoped id, not the internal row id. See docs/implementation.md#owner-handle-scoped-routes.
lookup_field = 'id'
lookup_url_kwarg = 'pk'
def get_queryset(self):
# See docs/implementation.md#owner-handle-scoped-routes; unlike items, StorageLocation has no availability_policy, but visibility_policy='private' still hides a location from friends.
if type(self.request.user) != KnownIdentity:
return StorageLocation.objects.none()
identity = self.request.user
try:
owner_user, owner_group = resolve_owner_handle(self.kwargs.get('handle', ''))
except ValueError:
raise ValidationError('invalid owner handle')
if owner_group:
if not owner_group.is_member(identity):
return StorageLocation.objects.none()
return StorageLocation.objects.filter(owner_group=owner_group)
if owner_user:
if owner_user not in identity.friends_or_self():
return StorageLocation.objects.none()
queryset = StorageLocation.objects.filter(owner=owner_user)
if not identity.user.filter(pk=owner_user.pk).exists():
queryset = queryset.exclude(visibility_policy='private')
return queryset
return StorageLocation.objects.none()
def perform_create(self, serializer):
try:
owner_user, owner_group = resolve_owner_handle(self.kwargs.get('handle', ''))
except ValueError:
raise ValidationError('invalid owner handle')
with transaction.atomic():
if owner_group:
if not owner_group.is_member(self.request.user):
raise PermissionDenied('Not a member of this group')
serializer.save(owner=None, owner_group=owner_group).clean()
elif owner_user:
if not self.request.user.user.filter(pk=owner_user.pk).exists():
raise PermissionDenied('Cannot create locations for another user')
serializer.save(owner=owner_user).clean()
else:
raise NotFound('No such owner')
@staticmethod
def _is_authorized(request, instance):
if instance.owner_id:
return request.user.user.filter(pk=instance.owner_id).exists()
return instance.owner_group.is_member(request.user)
def perform_update(self, serializer):
# See docs/implementation.md#perform-update-rejects-non-owned-items-explicitly.
if not self._is_authorized(self.request, serializer.instance):
raise PermissionDenied('Not authorized to modify this location')
with transaction.atomic():
serializer.save().clean()
def perform_destroy(self, instance):
if not self._is_authorized(self.request, instance):
raise PermissionDenied('Not authorized to delete this location')
instance.delete()
class WorkflowInstanceViewSet(viewsets.ModelViewSet):
serializer_class = WorkflowInstanceSerializer
authentication_classes = [SignatureAuthentication]
permission_classes = [IsAuthenticated]
def get_queryset(self): def get_queryset(self):
if type(self.request.user) == KnownIdentity and self.request.user.user.exists(): if type(self.request.user) == KnownIdentity and self.request.user.user.exists():
return WorkflowInstance.objects.filter(owner=self.request.user.user.get()) return StorageLocation.objects.filter(owner=self.request.user.user.get())
return WorkflowInstance.objects.none() return StorageLocation.objects.none()
def perform_create(self, serializer):
with transaction.atomic():
serializer.save(owner=self.request.user.user.get())
def perform_update(self, serializer):
with transaction.atomic():
if serializer.instance.owner == self.request.user.user.get():
serializer.save()
def perform_destroy(self, instance):
if instance.owner == self.request.user.user.get():
staged_file_ids = list(instance.staged_files.values_list('id', flat=True))
instance.delete()
for file in File.objects.filter(id__in=staged_file_ids):
if file.connected_items.count() == 0 and file.profile_picture_users.count() == 0 \
and file.staged_by_workflows.count() == 0:
file.file.delete(save=False)
file.delete()
@action(detail=True, methods=['post'])
def update_step(self, request, pk=None):
# get_queryset already scopes to the request user's own workflows, so get_object 404s for anyone else's.
instance = self.get_object()
with transaction.atomic():
if 'current_step' in request.data:
instance.current_step = request.data['current_step']
if 'payload' in request.data:
instance.payload = request.data['payload']
instance.save()
return Response(self.get_serializer(instance).data)
router = routers.SimpleRouter() router.register(r'inventory_items', InventoryItemViewSet, basename='inventory_items')
router.register(r'inventory_items/(?P<handle>[^/]+)', InventoryItemViewSet, basename='inventory_items') router.register(r'storage_locations', StorageLocationViewSet, basename='storage_locations')
router.register(r'storage_locations/(?P<handle>[^/]+)', StorageLocationViewSet, basename='storage_locations')
router.register(r'workflows', WorkflowInstanceViewSet, basename='workflows')
urlpatterns = router.urls + [ urlpatterns = router.urls + [
path('search/', search_inventory_items, name='search_inventory_items'), path('search/', search_inventory_items, name='search_inventory_items'),

253
backend/toolshed/api/offlinedata.py
View file

@ -1,253 +0,0 @@
from django.http import HttpResponse
from django.urls import path
from rest_framework.decorators import api_view, permission_classes, authentication_classes
from rest_framework.permissions import IsAuthenticated
from rest_framework.response import Response
from authentication.signature_auth import SignatureAuthentication
from toolshed.offlinedata import (
inventory_rows, friend_rows, location_rows, inventory_files, rows_to_csv,
import_locations, import_friends, import_inventory,
profile_data, profile_picture_files, settings_data, import_profile, import_settings,
delete_user_data, delete_user_account,
)
def local_user_or_none(identity):
"""Resolve the local ToolshedUser associated with an authenticated KnownIdentity, if any.
Returns None if the identity belongs to an external/remote user with no local account.
"""
if identity is None:
return None
return identity.user.first()
def user_data(user):
import io
import json
import zipfile
zip_buffer = io.BytesIO()
with zipfile.ZipFile(zip_buffer, "a", zipfile.ZIP_DEFLATED, False) as zip_file:
zip_file.writestr('profile.json', json.dumps(profile_data(user)))
zip_file.writestr('settings.json', json.dumps(settings_data(user)))
inventory_csv = b''.join(rows_to_csv(inventory_rows(user)))
zip_file.writestr('inventory.csv', inventory_csv)
friends_csv = b''.join(rows_to_csv(friend_rows(user)))
zip_file.writestr('friends.csv', friends_csv)
locations_csv = b''.join(rows_to_csv(location_rows(user)))
zip_file.writestr('locations.csv', locations_csv)
written_files = set()
for arcname, data in profile_picture_files(user):
zip_file.writestr(arcname, data)
written_files.add(arcname)
for arcname, data in inventory_files(user):
if arcname in written_files:
continue
zip_file.writestr(arcname, data)
written_files.add(arcname)
return zip_buffer.getvalue()
def parse_user_data(data):
import io
import zipfile
with zipfile.ZipFile(io.BytesIO(data), "r") as zip_file:
for file_name in zip_file.namelist():
yield file_name, zip_file.read(file_name)
def import_files(zip_file):
"""Fault-tolerant extraction of the 'files/' subfolder into File objects.
Returns a dict mapping the zip arcname to the created/existing File instance. Entries that
fail to read or save are silently skipped so a single corrupt attachment doesn't abort the import.
"""
from hashlib import sha256
import mimetypes
from django.core.files.base import ContentFile
from django.db import transaction
from files.models import File
result = {}
for name in zip_file.namelist():
if not name.startswith('files/') or name == 'files/':
continue
try:
with transaction.atomic():
data = zip_file.read(name)
content_hash = sha256(data).hexdigest()
mime_type, _ = mimetypes.guess_type(name)
file_obj = File.objects.filter(hash=content_hash).first()
if file_obj is None:
file_obj = File.objects.create(
file=ContentFile(data, content_hash),
mime_type=mime_type or 'application/octet-stream',
hash=content_hash,
)
result[name] = file_obj
except Exception as error:
print(f'Skipping file "{name}" during import: {error}')
return result
def import_user_data(user, data):
"""Fault-tolerant import of an export zip produced by `user_data()`.
Any of 'profile.json', 'settings.json', 'inventory.csv', 'friends.csv', 'locations.csv' or
the 'files/' subfolder may be missing; whatever is present is imported and everything else
is silently skipped.
Each section is imported inside its own transaction savepoint (`transaction.atomic()`), so a
DB-level failure in one section (e.g. a profile.json whose email collides with another
account) can't leave the connection in a broken/aborted-transaction state that would
otherwise silently take down every subsequent section (including the inventory items and
their properties) with an opaque "current transaction is aborted" error.
"""
import io
import zipfile
from django.db import transaction
summary = {'profile': False, 'settings': 0, 'locations': 0, 'friends': 0, 'inventory_items': 0, 'files': 0,
'errors': []}
with zipfile.ZipFile(io.BytesIO(data), 'r') as zip_file:
names = set(zip_file.namelist())
available_files = import_files(zip_file)
summary['files'] = len(available_files)
if 'profile.json' in names:
try:
with transaction.atomic():
summary['profile'] = import_profile(user, zip_file.read('profile.json'), available_files)
except Exception as error:
summary['errors'].append(f'Could not import profile.json: {error}')
if 'settings.json' in names:
try:
with transaction.atomic():
summary['settings'] = import_settings(user, zip_file.read('settings.json'))
except Exception as error:
summary['errors'].append(f'Could not import settings.json: {error}')
if 'locations.csv' in names:
try:
with transaction.atomic():
summary['locations'] = import_locations(user, zip_file.read('locations.csv'))
except Exception as error:
summary['errors'].append(f'Could not import locations.csv: {error}')
if 'friends.csv' in names:
try:
with transaction.atomic():
summary['friends'] = import_friends(user, zip_file.read('friends.csv'))
except Exception as error:
summary['errors'].append(f'Could not import friends.csv: {error}')
if 'inventory.csv' in names:
try:
with transaction.atomic():
summary['inventory_items'], inventory_errors = import_inventory(
user, zip_file.read('inventory.csv'), available_files)
summary['errors'].extend(inventory_errors)
except Exception as error:
summary['errors'].append(f'Could not import inventory.csv: {error}')
return summary
def _extract_zip_bytes(zip_payload):
"""Normalize the incoming 'zip' request payload (uploaded file, base64 string, or raw bytes)."""
import base64
if hasattr(zip_payload, 'read'):
return zip_payload.read()
if isinstance(zip_payload, str):
try:
return base64.b64decode(zip_payload, validate=True)
except Exception:
return zip_payload.encode('utf-8')
return zip_payload
@api_view(['POST'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication])
def import_data(request, format=None):
local_user = local_user_or_none(request.user)
if local_user is None:
return Response({'detail': 'This endpoint is only available to local users'}, status=403)
zip_payload = request.data.get('zip')
if not zip_payload:
return Response(status=400)
try:
zip_bytes = _extract_zip_bytes(zip_payload)
summary = import_user_data(local_user, zip_bytes)
except Exception as error:
return Response({'detail': f'Could not read zip file: {error}'}, status=400)
return Response(summary, status=200)
@api_view(['GET'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication])
def export_data(request, format=None):
local_user = local_user_or_none(request.user)
if local_user is None:
return Response({'detail': 'This endpoint is only available to local users'}, status=403)
return HttpResponse(user_data(local_user), content_type='application/zip', status=200)
@api_view(['DELETE'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication])
def delete_data(request, format=None):
"""Wipe all of the local user's data (everything included in the export), but keep the account.
This is *not* account deletion/closure - the user stays logged in and can keep using the
account (with all their data reset to a blank slate) afterwards.
"""
local_user = local_user_or_none(request.user)
if local_user is None:
return Response({'detail': 'This endpoint is only available to local users'}, status=403)
summary = delete_user_data(local_user)
return Response(summary, status=200)
@api_view(['DELETE'])
@permission_classes([IsAuthenticated])
@authentication_classes([SignatureAuthentication])
def delete_account(request, format=None):
"""Permanently close the local user's account, after wiping all of its data.
Unlike `delete_data()`, this also removes the account itself - the user can no longer log
in afterwards. Their public identity is kept so remote friends/history referencing it stay
intact, but the local ToolshedUser row is gone.
"""
local_user = local_user_or_none(request.user)
if local_user is None:
return Response({'detail': 'This endpoint is only available to local users'}, status=403)
summary = delete_user_account(local_user)
return Response(summary, status=200)
urlpatterns = [
path('export/', export_data, name='export_data'),
path('import/', import_data, name='import_data'),
path('account_data/', delete_data, name='delete_data'),
path('account/', delete_account, name='delete_account'),
]

17
backend/toolshed/api/version.py
View file

@ -1,17 +0,0 @@
from django.urls import path
from rest_framework.decorators import api_view, permission_classes, authentication_classes
from rest_framework.response import Response
from backend.settings import TOOLSHED_VERSION, GIT_COMMIT
@api_view(['GET'])
@permission_classes([])
@authentication_classes([])
def get_api_version(request, format=None):
return Response({'version': TOOLSHED_VERSION, 'commit': GIT_COMMIT})
urlpatterns = [
path('', get_api_version, name='version'),
]

28
backend/toolshed/migrations/0007_workflowinstance.py
View file

@ -1,28 +0,0 @@
# Generated by Django 4.2.2 on 2025-09-26 10:20
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
('toolshed', '0006_alter_tag_options_alter_category_name_and_more'),
]
operations = [
migrations.CreateModel(
name='WorkflowInstance',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('name', models.CharField(max_length=255)),
('state', models.CharField(max_length=255)),
('payload', models.JSONField(blank=True, default=dict)),
('created_at', models.DateTimeField(auto_now_add=True)),
('updated_at', models.DateTimeField(auto_now=True)),
('owner', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='workflows', to=settings.AUTH_USER_MODEL)),
],
),
]

19
backend/toolshed/migrations/0008_alter_inventoryitem_storage_location.py
View file

@ -1,19 +0,0 @@
# Generated by Django 4.2.2 on 2026-07-23 02:02
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0007_workflowinstance'),
]
operations = [
migrations.AlterField(
model_name='inventoryitem',
name='storage_location',
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='inventory_items', to='toolshed.storagelocation'),
),
]

21
backend/toolshed/migrations/0009_alter_workflowinstance_payload.py
View file

@ -1,21 +0,0 @@
# Generated manually: WorkflowInstance.payload changes from a native JSONField
# to a plain opaque TextField. The frontend now serializes/deserializes the
# JSON itself; the backend just stores whatever string it receives.
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0008_alter_inventoryitem_storage_location'),
]
operations = [
migrations.AlterField(
model_name='workflowinstance',
name='payload',
field=models.TextField(blank=True, default=''),
),
]

18
backend/toolshed/migrations/0010_rename_name_workflowinstance_slug.py
View file

@ -1,18 +0,0 @@
# Generated by Django 4.2.2 on 2026-08-09 13:08
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0009_alter_workflowinstance_payload'),
]
operations = [
migrations.RenameField(
model_name='workflowinstance',
old_name='name',
new_name='slug',
),
]

19
backend/toolshed/migrations/0011_workflowinstance_staged_files.py
View file

@ -1,19 +0,0 @@
# Generated by Django 4.2.2
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('files', '0001_initial'),
('toolshed', '0010_rename_name_workflowinstance_slug'),
]
operations = [
migrations.AddField(
model_name='workflowinstance',
name='staged_files',
field=models.ManyToManyField(blank=True, related_name='staged_by_workflows', to='files.file'),
),
]

27
backend/toolshed/migrations/0012_inventoryitem_owner_group_alter_inventoryitem_owner.py
View file

@ -1,27 +0,0 @@
# Generated by Django 4.2.2 on 2026-08-19 13:44
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
('authentication', '0005_group_groupinviteincoming_groupinvite_and_more'),
('toolshed', '0011_workflowinstance_staged_files'),
]
operations = [
migrations.AddField(
model_name='inventoryitem',
name='owner_group',
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='inventory_items', to='authentication.group'),
),
migrations.AlterField(
model_name='inventoryitem',
name='owner',
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='inventory_items', to=settings.AUTH_USER_MODEL),
),
]

37
backend/toolshed/migrations/0013_owneritemsequence.py
View file

@ -1,37 +0,0 @@
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
('authentication', '0005_group_groupinviteincoming_groupinvite_and_more'),
('toolshed', '0012_inventoryitem_owner_group_alter_inventoryitem_owner'),
]
operations = [
migrations.CreateModel(
name='OwnerItemSequence',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('last_id', models.PositiveIntegerField(default=0)),
('owner', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE,
related_name='+', to=settings.AUTH_USER_MODEL)),
('owner_group', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE,
related_name='+', to='authentication.group')),
],
),
migrations.AddConstraint(
model_name='owneritemsequence',
constraint=models.UniqueConstraint(condition=models.Q(('owner__isnull', False)), fields=('owner',),
name='owneritemsequence_unique_owner'),
),
migrations.AddConstraint(
model_name='owneritemsequence',
constraint=models.UniqueConstraint(condition=models.Q(('owner_group__isnull', False)),
fields=('owner_group',),
name='owneritemsequence_unique_owner_group'),
),
]

21
backend/toolshed/migrations/0014_rename_id_inventoryitem_internal_id.py
View file

@ -1,21 +0,0 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0013_owneritemsequence'),
]
operations = [
migrations.RenameField(
model_name='inventoryitem',
old_name='id',
new_name='internal_id',
),
migrations.AlterField(
model_name='inventoryitem',
name='internal_id',
field=models.AutoField(primary_key=True, serialize=False),
),
]

16
backend/toolshed/migrations/0015_inventoryitem_id.py
View file

@ -1,16 +0,0 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0014_rename_id_inventoryitem_internal_id'),
]
operations = [
migrations.AddField(
model_name='inventoryitem',
name='id',
field=models.PositiveIntegerField(editable=False, null=True),
),
]

42
backend/toolshed/migrations/0016_backfill_inventoryitem_id.py
View file

@ -1,42 +0,0 @@
from django.db import migrations
def collapse_ids(apps, schema_editor):
"""Collapse each owner/owner_group's InventoryItem ids from the sparse global range they
had before this migration down to a continuous 1..N range, in original creation order
(internal_id order), including soft-deleted rows since they still occupy a slot in that
scope's history. Then seed OwnerItemSequence so future allocation continues right after."""
InventoryItem = apps.get_model('toolshed', 'InventoryItem')
OwnerItemSequence = apps.get_model('toolshed', 'OwnerItemSequence')
scope = None
next_id = 0
counts = {}
for item in InventoryItem.objects.order_by('owner_id', 'owner_group_id', 'internal_id'):
key = (item.owner_id, item.owner_group_id)
if key != scope:
scope = key
next_id = 0
next_id += 1
item.id = next_id
item.save(update_fields=['id'])
counts[key] = next_id
for (owner_id, owner_group_id), count in counts.items():
OwnerItemSequence.objects.update_or_create(
owner_id=owner_id, owner_group_id=owner_group_id, defaults={'last_id': count})
def noop_reverse(apps, schema_editor):
pass
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0015_inventoryitem_id'),
]
operations = [
migrations.RunPython(collapse_ids, noop_reverse),
]

21
backend/toolshed/migrations/0017_inventoryitem_id_not_null_and_unique.py
View file

@ -1,21 +0,0 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0016_backfill_inventoryitem_id'),
]
operations = [
migrations.AlterField(
model_name='inventoryitem',
name='id',
field=models.PositiveIntegerField(editable=False),
),
migrations.AddConstraint(
model_name='inventoryitem',
constraint=models.UniqueConstraint(fields=('owner', 'owner_group', 'id'),
name='inventoryitem_unique_owner_scoped_id'),
),
]

23
backend/toolshed/migrations/0018_ownerstoragelocationsequence.py
View file

@ -1,23 +0,0 @@
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
('toolshed', '0017_inventoryitem_id_not_null_and_unique'),
]
operations = [
migrations.CreateModel(
name='OwnerStorageLocationSequence',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('last_id', models.PositiveIntegerField(default=0)),
('owner', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='+',
to=settings.AUTH_USER_MODEL, unique=True)),
],
),
]

21
backend/toolshed/migrations/0019_rename_id_storagelocation_internal_id.py
View file

@ -1,21 +0,0 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0018_ownerstoragelocationsequence'),
]
operations = [
migrations.RenameField(
model_name='storagelocation',
old_name='id',
new_name='internal_id',
),
migrations.AlterField(
model_name='storagelocation',
name='internal_id',
field=models.AutoField(primary_key=True, serialize=False),
),
]

16
backend/toolshed/migrations/0020_storagelocation_id.py
View file

@ -1,16 +0,0 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0019_rename_id_storagelocation_internal_id'),
]
operations = [
migrations.AddField(
model_name='storagelocation',
name='id',
field=models.PositiveIntegerField(editable=False, null=True),
),
]

40
backend/toolshed/migrations/0021_backfill_storagelocation_id.py
View file

@ -1,40 +0,0 @@
from django.db import migrations
def collapse_ids(apps, schema_editor):
"""Collapse each owner's StorageLocation ids from the sparse global range they had before this
migration down to a continuous 1..N range, in original creation order (internal_id order). Then
seed OwnerStorageLocationSequence so future allocation continues right after."""
StorageLocation = apps.get_model('toolshed', 'StorageLocation')
OwnerStorageLocationSequence = apps.get_model('toolshed', 'OwnerStorageLocationSequence')
scope = None
next_id = 0
counts = {}
for location in StorageLocation.objects.order_by('owner_id', 'internal_id'):
key = location.owner_id
if key != scope:
scope = key
next_id = 0
next_id += 1
location.id = next_id
location.save(update_fields=['id'])
counts[key] = next_id
for owner_id, count in counts.items():
OwnerStorageLocationSequence.objects.update_or_create(owner_id=owner_id, defaults={'last_id': count})
def noop_reverse(apps, schema_editor):
pass
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0020_storagelocation_id'),
]
operations = [
migrations.RunPython(collapse_ids, noop_reverse),
]

20
backend/toolshed/migrations/0022_storagelocation_id_not_null_and_unique.py
View file

@ -1,20 +0,0 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0021_backfill_storagelocation_id'),
]
operations = [
migrations.AlterField(
model_name='storagelocation',
name='id',
field=models.PositiveIntegerField(editable=False),
),
migrations.AddConstraint(
model_name='storagelocation',
constraint=models.UniqueConstraint(fields=('owner', 'id'), name='storagelocation_unique_owner_scoped_id'),
),
]

63
backend/toolshed/migrations/0023_remove_storagelocation_storagelocation_unique_owner_scoped_id_and_more.py
View file

@ -1,63 +0,0 @@
# Generated by Django 4.2.2 on 2026-08-26 18:55
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('authentication', '0006_groupmembership'),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
('toolshed', '0022_storagelocation_id_not_null_and_unique'),
]
operations = [
migrations.RemoveConstraint(
model_name='storagelocation',
name='storagelocation_unique_owner_scoped_id',
),
migrations.AddField(
model_name='ownerstoragelocationsequence',
name='owner_group',
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='+', to='authentication.group'),
),
migrations.AddField(
model_name='storagelocation',
name='owner_group',
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='storage_locations', to='authentication.group'),
),
migrations.AlterField(
model_name='owneritemsequence',
name='id',
field=models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
),
migrations.AlterField(
model_name='ownerstoragelocationsequence',
name='id',
field=models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
),
migrations.AlterField(
model_name='ownerstoragelocationsequence',
name='owner',
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='+', to=settings.AUTH_USER_MODEL),
),
migrations.AlterField(
model_name='storagelocation',
name='owner',
field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='storage_locations', to=settings.AUTH_USER_MODEL),
),
migrations.AddConstraint(
model_name='ownerstoragelocationsequence',
constraint=models.UniqueConstraint(condition=models.Q(('owner__isnull', False)), fields=('owner',), name='ownerstoragelocationsequence_unique_owner'),
),
migrations.AddConstraint(
model_name='ownerstoragelocationsequence',
constraint=models.UniqueConstraint(condition=models.Q(('owner_group__isnull', False)), fields=('owner_group',), name='ownerstoragelocationsequence_unique_owner_group'),
),
migrations.AddConstraint(
model_name='storagelocation',
constraint=models.UniqueConstraint(fields=('owner', 'owner_group', 'id'), name='storagelocation_unique_owner_scoped_id'),
),
]

18
backend/toolshed/migrations/0024_workflowinstance_current_step.py
View file

@ -1,18 +0,0 @@
# Generated by Django 4.2.2
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0023_remove_storagelocation_storagelocation_unique_owner_scoped_id_and_more'),
]
operations = [
migrations.AddField(
model_name='workflowinstance',
name='current_step',
field=models.PositiveIntegerField(default=1),
),
]

23
backend/toolshed/migrations/0025_inventoryitem_visibility_policy_and_more.py
View file

@ -1,23 +0,0 @@
# Generated by Django 4.2.2 on 2026-08-31 23:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('toolshed', '0024_workflowinstance_current_step'),
]
operations = [
migrations.AddField(
model_name='inventoryitem',
name='visibility_policy',
field=models.CharField(choices=[('public', 'Public'), ('friends', 'Friends'), ('private', 'Private')], default='friends', max_length=20),
),
migrations.AddField(
model_name='storagelocation',
name='visibility_policy',
field=models.CharField(choices=[('public', 'Public'), ('friends', 'Friends'), ('private', 'Private')], default='friends', max_length=20),
),
]

151
backend/toolshed/models.py
View file

@ -1,9 +1,9 @@
from django.db import models, transaction from django.db import models
from django.core.validators import MinValueValidator, MaxValueValidator from django.core.validators import MinValueValidator
from django_softdelete.models import SoftDeleteModel from django_softdelete.models import SoftDeleteModel
from rest_framework.exceptions import ValidationError from rest_framework.exceptions import ValidationError
from authentication.models import ToolshedUser, KnownIdentity, Group from authentication.models import ToolshedUser, KnownIdentity
from files.models import File from files.models import File
@ -26,10 +26,6 @@ class Category(SoftDeleteModel):
parent = str(self.parent) + "/" if self.parent else "" parent = str(self.parent) + "/" if self.parent else ""
return parent + self.name return parent + self.name
def get_handle(self):
"""Return a fully qualified handle like 'git:base#category:tools'"""
return f"{self.origin}#category:{self.name}"
class Property(models.Model): class Property(models.Model):
name = models.CharField(max_length=255) name = models.CharField(max_length=255)
@ -54,10 +50,6 @@ class Property(models.Model):
def __str__(self): def __str__(self):
return self.name return self.name
def get_handle(self):
"""Return a fully qualified handle like 'git:base#property:length'"""
return f"{self.origin}#property:{self.name}"
class Tag(models.Model): class Tag(models.Model):
name = models.CharField(max_length=255) name = models.CharField(max_length=255)
@ -77,42 +69,6 @@ class Tag(models.Model):
def __str__(self): def __str__(self):
return self.name return self.name
def get_handle(self):
"""Return a fully qualified handle like 'git:tools#tag:drill'"""
return f"{self.origin}#tag:{self.name}"
class OwnerItemSequence(models.Model):
"""Tracks the last InventoryItem id handed out per owner/owner_group scope for sequential,
gapless allocation (see InventoryItem.create_for_owner); exactly one of owner/owner_group is
set, mirroring InventoryItem's own split."""
owner = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, null=True, blank=True, related_name='+')
owner_group = models.ForeignKey(Group, on_delete=models.CASCADE, null=True, blank=True, related_name='+')
last_id = models.PositiveIntegerField(default=0)
class Meta:
constraints = [
models.UniqueConstraint(fields=['owner'], condition=models.Q(owner__isnull=False),
name='owneritemsequence_unique_owner'),
models.UniqueConstraint(fields=['owner_group'], condition=models.Q(owner_group__isnull=False),
name='owneritemsequence_unique_owner_group'),
]
@classmethod
def allocate(cls, *, owner=None, owner_group=None):
with transaction.atomic():
seq, _ = cls.objects.select_for_update().get_or_create(owner=owner, owner_group=owner_group)
seq.last_id += 1
seq.save(update_fields=['last_id'])
return seq.last_id
VISIBILITY_POLICY_CHOICES = (
('public', 'Public'),
('friends', 'Friends'),
('private', 'Private'),
)
class InventoryItem(SoftDeleteModel): class InventoryItem(SoftDeleteModel):
AVAILABILITY_POLICY_CHOICES = ( AVAILABILITY_POLICY_CHOICES = (
@ -122,49 +78,24 @@ class InventoryItem(SoftDeleteModel):
('share', 'Share'), ('share', 'Share'),
('private', 'Private'), ('private', 'Private'),
) )
VISIBILITY_POLICY_CHOICES = VISIBILITY_POLICY_CHOICES
internal_id = models.AutoField(primary_key=True)
# Externally visible id, sequential/gapless within owner/owner_group's own items (see
# OwnerItemSequence), never internal_id; always allocate via create_for_owner, not .objects.create().
id = models.PositiveIntegerField(editable=False)
published = models.BooleanField(default=False) published = models.BooleanField(default=False)
name = models.CharField(max_length=255, null=True, blank=True) name = models.CharField(max_length=255, null=True, blank=True)
description = models.TextField(null=True, blank=True) description = models.TextField(null=True, blank=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE, null=True, related_name='inventory_items') category = models.ForeignKey(Category, on_delete=models.CASCADE, null=True, related_name='inventory_items')
availability_policy = models.CharField(max_length=20, choices=AVAILABILITY_POLICY_CHOICES, default='private') availability_policy = models.CharField(max_length=20, choices=AVAILABILITY_POLICY_CHOICES, default='private')
visibility_policy = models.CharField(max_length=20, choices=VISIBILITY_POLICY_CHOICES, default='private')
owned_quantity = models.IntegerField(default=1, validators=[MinValueValidator(0)]) owned_quantity = models.IntegerField(default=1, validators=[MinValueValidator(0)])
owner = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, null=True, blank=True, owner = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, related_name='inventory_items')
related_name='inventory_items')
owner_group = models.ForeignKey(Group, on_delete=models.CASCADE, null=True, blank=True,
related_name='inventory_items')
created_at = models.DateTimeField(auto_now_add=True) created_at = models.DateTimeField(auto_now_add=True)
tags = models.ManyToManyField(Tag, through='ItemTag', related_name='inventory_items') tags = models.ManyToManyField(Tag, through='ItemTag', related_name='inventory_items')
properties = models.ManyToManyField(Property, through='ItemProperty') properties = models.ManyToManyField(Property, through='ItemProperty')
files = models.ManyToManyField(File, related_name='connected_items') files = models.ManyToManyField(File, related_name='connected_items')
storage_location = models.ForeignKey('StorageLocation', on_delete=models.SET_NULL, null=True, blank=True, storage_location = models.ForeignKey('StorageLocation', on_delete=models.CASCADE, null=True, blank=True,
related_name='inventory_items') related_name='inventory_items')
class Meta:
constraints = [
models.UniqueConstraint(fields=['owner', 'owner_group', 'id'],
name='inventoryitem_unique_owner_scoped_id'),
]
def clean(self): def clean(self):
if (self.name is None or self.name == "") and self.files.count() == 0: if (self.name is None or self.name == "") and self.files.count() == 0:
raise ValidationError("Name or at least one file must be set") raise ValidationError("Name or at least one file must be set")
if (self.owner is None) == (self.owner_group is None):
raise ValidationError("Exactly one of owner or owner_group must be set")
@classmethod
def create_for_owner(cls, *, owner=None, owner_group=None, **kwargs):
"""The only supported way to create an InventoryItem: atomically allocates the next id
for this owner/owner_group scope."""
with transaction.atomic():
next_id = OwnerItemSequence.allocate(owner=owner, owner_group=owner_group)
return cls.objects.create(owner=owner, owner_group=owner_group, id=next_id, **kwargs)
class ItemProperty(models.Model): class ItemProperty(models.Model):
@ -178,84 +109,14 @@ class ItemTag(models.Model):
inventory_item = models.ForeignKey(InventoryItem, on_delete=models.CASCADE) inventory_item = models.ForeignKey(InventoryItem, on_delete=models.CASCADE)
class OwnerStorageLocationSequence(models.Model):
"""Tracks the last StorageLocation id handed out per owner/owner_group scope for sequential,
gapless allocation (see StorageLocation.create_for_owner); exactly one of owner/owner_group is
set, mirroring OwnerItemSequence/InventoryItem's own split."""
owner = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, null=True, blank=True, related_name='+')
owner_group = models.ForeignKey(Group, on_delete=models.CASCADE, null=True, blank=True, related_name='+')
last_id = models.PositiveIntegerField(default=0)
class Meta:
constraints = [
models.UniqueConstraint(fields=['owner'], condition=models.Q(owner__isnull=False),
name='ownerstoragelocationsequence_unique_owner'),
models.UniqueConstraint(fields=['owner_group'], condition=models.Q(owner_group__isnull=False),
name='ownerstoragelocationsequence_unique_owner_group'),
]
@classmethod
def allocate(cls, *, owner=None, owner_group=None):
with transaction.atomic():
seq, _ = cls.objects.select_for_update().get_or_create(owner=owner, owner_group=owner_group)
seq.last_id += 1
seq.save(update_fields=['last_id'])
return seq.last_id
class StorageLocation(models.Model): class StorageLocation(models.Model):
VISIBILITY_POLICY_CHOICES = VISIBILITY_POLICY_CHOICES
internal_id = models.AutoField(primary_key=True)
# Externally visible id, sequential/gapless within the owner/owner_group's own locations (see
# OwnerStorageLocationSequence), never internal_id; always allocate via create_for_owner, not
# .objects.create().
id = models.PositiveIntegerField(editable=False)
name = models.CharField(max_length=255) name = models.CharField(max_length=255)
description = models.TextField(null=True, blank=True) description = models.TextField(null=True, blank=True)
category = models.ForeignKey(Category, on_delete=models.CASCADE, null=True, blank=True, category = models.ForeignKey(Category, on_delete=models.CASCADE, null=True, blank=True,
related_name='storage_locations') related_name='storage_locations')
parent = models.ForeignKey('self', on_delete=models.CASCADE, null=True, blank=True, related_name='children') parent = models.ForeignKey('self', on_delete=models.CASCADE, null=True, blank=True, related_name='children')
owner = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, null=True, blank=True, owner = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, related_name='storage_locations')
related_name='storage_locations')
owner_group = models.ForeignKey(Group, on_delete=models.CASCADE, null=True, blank=True,
related_name='storage_locations')
visibility_policy = models.CharField(max_length=20, choices=VISIBILITY_POLICY_CHOICES, default='private')
class Meta:
constraints = [
models.UniqueConstraint(fields=['owner', 'owner_group', 'id'],
name='storagelocation_unique_owner_scoped_id'),
]
def __str__(self): def __str__(self):
parent = str(self.parent) + "/" if self.parent else "" parent = str(self.parent) + "/" if self.parent else ""
return parent + self.name return parent + self.name
def clean(self):
if (self.owner is None) == (self.owner_group is None):
raise ValidationError("Exactly one of owner or owner_group must be set")
@classmethod
def create_for_owner(cls, *, owner=None, owner_group=None, **kwargs):
"""The only supported way to create a StorageLocation: atomically allocates the next id
for this owner/owner_group's scope."""
with transaction.atomic():
next_id = OwnerStorageLocationSequence.allocate(owner=owner, owner_group=owner_group)
return cls.objects.create(owner=owner, owner_group=owner_group, id=next_id, **kwargs)
class WorkflowInstance(models.Model):
slug = models.CharField(max_length=255)
state = models.CharField(max_length=255)
current_step = models.PositiveIntegerField(default=1)
payload = models.TextField(default='', blank=True) # an opaque, frontend-serialized JSON string on the backend.
owner = models.ForeignKey(ToolshedUser, on_delete=models.CASCADE, related_name='workflows')
staged_files = models.ManyToManyField(File, related_name='staged_by_workflows', blank=True)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
def __str__(self):
return f"{self.slug} ({self.state})"

582
backend/toolshed/offlinedata.py
View file

@ -1,582 +0,0 @@
"""Data helpers for building/importing a user's offline export; deal with toolshed models and CSV row shapes only, and know nothing about the zip container format or request auth - see toolshed/api/offlinedata.py for that."""
def inventory_rows(user):
"""Generator that yields the given user's inventory items as flattened dicts, one per row."""
import mimetypes
from toolshed.models import InventoryItem
items = (InventoryItem.objects
.filter(owner=user)
.select_related('category', 'storage_location')
.prefetch_related('tags', 'itemproperty_set__property', 'files'))
for item in items:
file_paths = []
for f in item.files.all():
extension = mimetypes.guess_extension(f.mime_type) or ''
file_paths.append(f'files/{f.hash}{extension}')
yield {
'id': item.id,
'name': item.name or '',
'description': item.description or '',
'category': item.category.get_handle() if item.category else '',
'availability_policy': item.availability_policy,
'visibility_policy': item.visibility_policy,
'owned_quantity': item.owned_quantity,
'storage_location': str(item.storage_location) if item.storage_location else '',
'tags': ', '.join(tag.get_handle() for tag in item.tags.all()),
'properties': _encode_properties_cell(item.itemproperty_set.all()),
'files': ', '.join(file_paths),
'created_at': item.created_at.isoformat() if item.created_at else '',
}
def friend_rows(user):
"""Generator that yields the given user's friends (known identities) as flattened dicts, one per row."""
friends = user.friends.all()
for friend in friends:
yield {
'username': friend.username,
'domain': friend.domain,
'handle': f'{friend.username}@{friend.domain}',
'public_key': friend.public_key,
}
def location_path(location):
"""Recursively build the '/'-joined path of a StorageLocation, following its parent chain."""
if location.parent:
return location_path(location.parent) + '/' + location.name
return location.name
def location_rows(user):
"""Generator that yields the given user's storage locations as flattened dicts, one per row."""
from toolshed.models import StorageLocation
locations = StorageLocation.objects.filter(owner=user).select_related('category', 'parent')
for location in locations:
yield {
'id': location.id,
'name': location.name,
'description': location.description or '',
'category': str(location.category) if location.category else '',
'parent': str(location.parent) if location.parent else '',
'path': location_path(location),
'visibility_policy': location.visibility_policy,
}
def inventory_files(user):
"""Yields (arcname, data) for each unique file attached to the user's inventory items, deduplicated by hash. See docs/implementation.md#file-naming-convention-in-exports for the naming scheme."""
import mimetypes
from toolshed.models import InventoryItem
seen_hashes = set()
items = InventoryItem.objects.filter(owner=user).prefetch_related('files')
for item in items:
for file in item.files.all():
if file.hash in seen_hashes:
continue
seen_hashes.add(file.hash)
extension = mimetypes.guess_extension(file.mime_type) or ''
arcname = f'files/{file.hash}{extension}'
file.file.open('rb')
try:
data = file.file.read()
finally:
file.file.close()
yield arcname, data
def profile_data(user):
"""Return the given user's profile as a dict, matching profile.json in the export zip."""
import mimetypes
data = {
'username': user.username,
'domain': user.domain,
'email': user.email,
'first_name': user.first_name or '',
'last_name': user.last_name or '',
'profile_picture': None,
}
if user.profile_picture:
extension = mimetypes.guess_extension(user.profile_picture.mime_type) or ''
data['profile_picture'] = f'files/{user.profile_picture.hash}{extension}'
return data
def profile_picture_files(user):
"""Generator that yields (arcname, data) for the user's profile picture, if one is set. Kept separate from `inventory_files()` so the picture is included even for users with no inventory items or whose picture isn't attached to any item."""
import mimetypes
if not user.profile_picture:
return
extension = mimetypes.guess_extension(user.profile_picture.mime_type) or ''
arcname = f'files/{user.profile_picture.hash}{extension}'
user.profile_picture.file.open('rb')
try:
data = user.profile_picture.file.read()
finally:
user.profile_picture.file.close()
yield arcname, data
def settings_data(user):
"""Return the given user's account-level preferences as a {key: value} dict for settings.json. Only account preferences are exported; device-level preferences stay in the browser's local storage since they describe the device, not the account."""
return {pref.key: pref.value for pref in user.preferences.all()}
def import_profile(user, data, available_files):
"""Fault-tolerant import of profile.json, updating the user's editable profile fields. See docs/implementation.md#profile-import-semantics for which fields are applied and why."""
import json
try:
profile = json.loads(data.decode('utf-8'))
if not isinstance(profile, dict):
return False
if 'first_name' in profile:
user.first_name = profile.get('first_name') or ''
if 'last_name' in profile:
user.last_name = profile.get('last_name') or ''
if profile.get('email'):
user.email = profile['email']
picture_path = profile.get('profile_picture')
if picture_path and picture_path in available_files:
user.profile_picture = available_files[picture_path]
user.save()
return True
except Exception as error:
print(f'Skipping profile.json: {error}')
return False
def import_settings(user, data):
"""Fault-tolerant import of settings.json; each top-level key/value pair is upserted as an AccountPreference, and unreadable data or a non-object payload is skipped rather than aborting the whole import."""
import json
from authentication.models import AccountPreference
try:
settings = json.loads(data.decode('utf-8'))
if not isinstance(settings, dict):
return 0
imported = 0
for key, value in settings.items():
try:
AccountPreference.objects.update_or_create(user=user, key=key, defaults={'value': value})
imported += 1
except Exception as error:
print(f'Skipping setting "{key}": {error}')
return imported
except Exception as error:
print(f'Skipping settings.json: {error}')
return 0
def delete_user_data(user):
"""Permanently delete everything that `user_data()` exports, keeping the account itself intact. Returns a summary dict describing what was removed. See docs/implementation.md#account-data-deletion for exactly what's removed and why the account itself survives."""
from django.db import transaction
from toolshed.models import InventoryItem, StorageLocation
summary = {'inventory_items': 0, 'locations': 0, 'settings': 0, 'friends': 0, 'files': 0}
with transaction.atomic():
candidate_file_ids = set(
InventoryItem.global_objects.filter(owner=user).values_list('files__id', flat=True))
candidate_file_ids.discard(None)
if user.profile_picture_id:
candidate_file_ids.add(user.profile_picture_id)
for item in InventoryItem.global_objects.filter(owner=user):
item.hard_delete()
summary['inventory_items'] += 1
summary['locations'], _ = StorageLocation.objects.filter(owner=user).delete()
summary['settings'], _ = user.preferences.all().delete()
summary['friends'] = user.public_identity.friends.count()
user.public_identity.friends.clear()
user.profile_picture = None
user.save(update_fields=['profile_picture'])
summary['files'] = _delete_orphaned_files(candidate_file_ids)
return summary
def delete_user_account(user):
"""Permanently delete the local user's account, after wiping all of its data via `delete_user_data()`. Returns a summary dict with `account` set to True. See docs/implementation.md#account-data-deletion for why the underlying KnownIdentity is kept."""
from django.db import transaction
with transaction.atomic():
summary = delete_user_data(user)
user.delete()
summary['account'] = True
return summary
def _delete_orphaned_files(file_ids):
"""Delete File rows (and their underlying blobs) in `file_ids` that are no longer referenced. Returns the number of files deleted. See docs/implementation.md#account-data-deletion for the orphan definition."""
from files.models import File
deleted = 0
for file_obj in File.objects.filter(id__in=file_ids):
if file_obj.connected_items.exists() or file_obj.profile_picture_users.exists() \
or file_obj.staged_by_workflows.exists():
continue
file_obj.file.delete(save=False)
file_obj.delete()
deleted += 1
return deleted
def rows_to_csv(rows, fieldnames=None, encoding='utf-8'):
"""Generator that consumes an iterable of dicts and yields encoded CSV data chunk by chunk."""
import csv
class _Echo:
"""A file-like object whose write() just returns what was passed, for streaming csv.writer output."""
def write(self, value):
return value
rows = iter(rows)
try:
first_row = next(rows)
except StopIteration:
return
if fieldnames is None:
fieldnames = list(first_row.keys())
writer = csv.DictWriter(_Echo(), fieldnames=fieldnames)
yield writer.writeheader().encode(encoding)
yield writer.writerow(first_row).encode(encoding)
for row in rows:
yield writer.writerow(row).encode(encoding)
def _read_csv_rows(data, encoding='utf-8'):
"""Decode CSV bytes and yield rows as dicts keyed by the header labels (not column offsets)."""
import csv
import io
reader = csv.DictReader(io.StringIO(data.decode(encoding)))
for row in reader:
yield row
def _field_is_optional(model, field_name):
"""Return True if a field may be omitted: it's a relation, has a default, or allows null/blank."""
try:
field = model._meta.get_field(field_name)
except Exception:
return True
if getattr(field, 'many_to_many', False) or getattr(field, 'one_to_many', False):
return True
return bool(getattr(field, 'null', False) or getattr(field, 'blank', False) or field.has_default())
def get_or_create_category(path):
"""Resolve or create a Category from a '/'-separated path such as 'Electronics/Cables'."""
from toolshed.models import Category
parent = None
category = None
for part in (p for p in path.split('/') if p):
category, _ = Category.objects.get_or_create(name=part, parent=parent, defaults={'origin': 'import'})
parent = category
return category
def import_locations(user, data):
"""Fault-tolerant import of locations.csv into StorageLocations owned by `user`. See docs/implementation.md#location-import-ordering-and-savepoints for row ordering and error-isolation rules."""
from django.db import transaction
from toolshed.models import StorageLocation
rows = list(_read_csv_rows(data))
rows.sort(key=lambda row: (row.get('path') or row.get('name') or '').count('/'))
resolved_by_path = {}
imported = 0
for row in rows:
try:
with transaction.atomic():
name = (row.get('name') or '').strip()
if not name and not _field_is_optional(StorageLocation, 'name'):
continue # required column missing, skip this row
path = (row.get('path') or name).strip()
parent = None
if '/' in path:
parent = resolved_by_path.get(path.rsplit('/', 1)[0])
category = None
category_path = row.get('category')
if category_path:
category = get_or_create_category(category_path)
defaults = {
'description': row.get('description', '') or '',
'category': category,
'visibility_policy': row.get('visibility_policy') or 'private',
}
try:
location = StorageLocation.objects.get(owner=user, name=name, parent=parent)
for field, value in defaults.items():
setattr(location, field, value)
location.save(update_fields=list(defaults.keys()))
except StorageLocation.DoesNotExist:
location = StorageLocation.create_for_owner(owner=user, name=name, parent=parent, **defaults)
resolved_by_path[path] = location
imported += 1
except Exception as error:
print(f'Skipping location row {row}: {error}')
return imported
def import_friends(user, data):
"""Fault-tolerant import of friends.csv, adding valid entries to the user's known friends."""
from django.db import transaction
from authentication.models import KnownIdentity
imported = 0
for row in _read_csv_rows(data):
try:
with transaction.atomic():
username = row.get('username')
domain = row.get('domain')
handle = row.get('handle')
if (not username or not domain) and handle and '@' in handle:
username, domain = handle.split('@', 1)
public_key = row.get('public_key')
if not username or not domain or not public_key:
continue # required fields missing, skip this row
identity, _ = KnownIdentity.objects.get_or_create(
username=username, domain=domain,
defaults={'public_key': public_key},
)
user.public_identity.friends.add(identity)
imported += 1
except Exception as error:
print(f'Skipping friend row {row}: {error}')
return imported
class _HandleNotFound(Exception):
"""Raised when a fully qualified handle (e.g. 'git:base#tag:drill') can't be resolved, aborting the row rather than creating a new entity. See docs/implementation.md#handle-resolution-semantics."""
def _resolve_handle(value, entity_type, model):
"""Resolve a fully qualified handle (e.g. 'git:base#tag:drill') to an *existing* model instance; never creates one. See docs/implementation.md#handle-resolution-semantics for the rationale."""
origin, rest = value.split('#', 1)
if ':' in rest:
found_type, name = rest.split(':', 1)
if found_type != entity_type:
raise _HandleNotFound(
f"expected a {entity_type} handle but got '{value}' (type '{found_type}')")
else:
name = rest
try:
return model.objects.get(origin=origin, name=name)
except model.DoesNotExist:
raise _HandleNotFound(f"{entity_type} '{value}' does not exist, skipping item")
def _quote_value_if_needed(value):
"""Wrap `value` in double quotes (CSV-style, doubling embedded quotes) if it contains a comma or quote character. See docs/implementation.md#properties-csv-encoding."""
if any(ch in value for ch in ',"'):
return '"' + value.replace('"', '""') + '"'
return value
def _encode_properties_cell(item_properties):
"""Encode an item's properties as a comma-separated "handle=value" list for the 'properties' CSV cell. See docs/implementation.md#properties-csv-encoding and `_parse_properties_cell()` for the reader side."""
entries = [
f"{ip.property.get_handle()}={_quote_value_if_needed(ip.value or '')}"
for ip in item_properties
]
return ', '.join(entries)
def _split_quoted_comma_list(raw_value):
"""Split a comma-separated list into entries, honouring double-quoted substrings (CSV-style) so a quoted value's own commas aren't mistaken for separators. See docs/implementation.md#properties-csv-encoding for the quoting/whitespace rules this implements."""
entries = []
current = []
in_quotes = False
i = 0
length = len(raw_value)
while i < length:
char = raw_value[i]
if char == '"':
if in_quotes and i + 1 < length and raw_value[i + 1] == '"':
current.append('"')
i += 2
continue
in_quotes = not in_quotes
i += 1
continue
if char == ',' and not in_quotes:
entries.append(''.join(current))
current = []
i += 1
if i < length and raw_value[i] == ' ':
i += 1 # skip the single space of the ", " separator
continue
current.append(char)
i += 1
entries.append(''.join(current))
return [entry for entry in entries if entry]
def _parse_properties_cell(raw_value, resolve_property):
"""Parse the 'properties' CSV cell into a list of (Property, value) tuples. See docs/implementation.md#properties-csv-encoding for how the cell is encoded by `_encode_properties_cell()`."""
raw_value = (raw_value or '').strip()
if not raw_value:
return []
properties = []
for prop_entry in _split_quoted_comma_list(raw_value):
if '=' not in prop_entry:
continue
prop_name, value = prop_entry.split('=', 1)
prop = resolve_property(prop_name.strip())
if prop:
properties.append((prop, value))
return properties
def import_inventory(user, data, available_files):
"""Fault-tolerant import of inventory.csv into InventoryItems owned by `user`. See docs/implementation.md#inventory-import-semantics for file/handle resolution and error-isolation rules."""
from django.db import transaction
from toolshed.models import Category, InventoryItem, ItemProperty, StorageLocation, Tag, Property
def resolve_category_from_csv(value):
"""Resolve a category from CSV - supports both old format and fully qualified handles"""
if not value:
return None
if '#' in value:
return _resolve_handle(value, 'category', Category)
# Fallback to old path-based lookup
return get_or_create_category(value)
def resolve_tag_from_csv(value):
"""Resolve a tag from CSV - supports both old format and fully qualified handles"""
if not value:
return None
if '#' in value:
return _resolve_handle(value, 'tag', Tag)
# Fallback to old name-only lookup or create
return Tag.objects.get_or_create(name=value, category=None, defaults={'origin': 'import'})[0]
def resolve_property_from_csv(value):
"""Resolve a property from CSV - supports both old format and fully qualified handles"""
if not value:
return None
if '#' in value:
return _resolve_handle(value, 'property', Property)
# Fallback to old name-only lookup or create
return Property.objects.get_or_create(name=value, category=None, defaults={'origin': 'import'})[0]
imported = 0
errors = []
for row in _read_csv_rows(data):
name = (row.get('name') or '').strip()
try:
with transaction.atomic():
file_paths = [p.strip() for p in (row.get('files') or '').split(',') if p.strip()]
files = [available_files[p] for p in file_paths if p in available_files]
if not name and not files:
continue # InventoryItem requires a name or at least one file
category = resolve_category_from_csv((row.get('category') or '').strip())
tags = []
for tag_name in (row.get('tags') or '').split(','):
tag_name = tag_name.strip()
if tag_name:
tag = resolve_tag_from_csv(tag_name)
if tag:
tags.append(tag)
properties = _parse_properties_cell(row.get('properties') or '', resolve_property_from_csv)
storage_location = None
location_path_value = row.get('storage_location')
if location_path_value:
storage_location = StorageLocation.objects.filter(
owner=user, name=location_path_value.rsplit('/', 1)[-1]).first()
try:
owned_quantity = int(row.get('owned_quantity') or 1)
except (TypeError, ValueError):
owned_quantity = 1
item = InventoryItem.create_for_owner(
owner=user,
name=name or None,
description=row.get('description', '') or '',
category=category,
availability_policy=row.get('availability_policy') or 'private',
visibility_policy=row.get('visibility_policy') or 'private',
owned_quantity=owned_quantity,
storage_location=storage_location,
)
for tag in tags:
item.tags.add(tag, through_defaults={})
for prop, value in properties:
ItemProperty.objects.create(inventory_item=item, property=prop, value=value)
for file in files:
item.files.add(file)
imported += 1
except _HandleNotFound as error:
message = f"Skipping item '{name or 'unnamed'}': {error}"
print(message)
errors.append(message)
except Exception as error:
message = f'Skipping inventory row {row}: {error}'
print(message)
errors.append(message)
return imported, errors

229
backend/toolshed/serializers.py
View file

@ -1,62 +1,19 @@
from django.core.exceptions import ObjectDoesNotExist
from django.db.models import Q
from rest_framework import serializers from rest_framework import serializers
from authentication.models import KnownIdentity, ToolshedUser, FriendRequestIncoming, Group, GroupInviteIncoming, \ from authentication.models import KnownIdentity, ToolshedUser, FriendRequestIncoming
GroupMembership from authentication.serializers import OwnerSerializer
from authentication.serializers import OwnerSerializer, GroupOwnerSerializer
from files.models import File from files.models import File
from files.serializers import FileSerializer from files.serializers import FileSerializer
from toolshed.models import Category, Property, ItemProperty, InventoryItem, Tag, StorageLocation, WorkflowInstance from toolshed.models import Category, Property, ItemProperty, InventoryItem, Tag, StorageLocation
def parse_handle(handle):
"""Parse a fully qualified handle like 'git:base#property:length' into (origin, entity_type, name)"""
if '#' not in handle:
# Fallback to old format (just name)
return None, None, handle
origin, rest = handle.split('#', 1)
if ':' not in rest:
return origin, None, rest
entity_type, name = rest.split(':', 1)
return origin, entity_type, name
def resolve_category_handle(handle):
"""Resolve a fully qualified handle to a Category object"""
origin, entity_type, name = parse_handle(handle)
if origin and entity_type == 'category':
return Category.objects.get(origin=origin, name=name)
# Fallback to name-only lookup
return Category.objects.get(name=handle.split('/')[-1])
def resolve_property_handle(handle):
"""Resolve a fully qualified handle to a Property object"""
origin, entity_type, name = parse_handle(handle)
if origin and entity_type == 'property':
return Property.objects.get(origin=origin, name=name)
# Fallback to name-only lookup
return Property.objects.get(name=handle)
def resolve_tag_handle(handle):
"""Resolve a fully qualified handle to a Tag object"""
origin, entity_type, name = parse_handle(handle)
if origin and entity_type == 'tag':
return Tag.objects.get(origin=origin, name=name)
# Fallback to name-only lookup
return Tag.objects.get(name=handle)
class FriendSerializer(serializers.ModelSerializer): class FriendSerializer(serializers.ModelSerializer):
handle = serializers.SerializerMethodField() username = serializers.SerializerMethodField()
class Meta: class Meta:
model = KnownIdentity model = KnownIdentity
fields = ['id', 'handle', 'public_key'] fields = ['id', 'username', 'public_key']
def get_handle(self, obj): def get_username(self, obj):
return obj.username + '@' + obj.domain return obj.username + '@' + obj.domain
@ -71,149 +28,35 @@ class FriendRequestSerializer(serializers.ModelSerializer):
return obj.befriender_username + '@' + obj.befriender_domain return obj.befriender_username + '@' + obj.befriender_domain
class GroupMemberSerializer(serializers.ModelSerializer):
handle = serializers.SerializerMethodField()
class Meta:
model = KnownIdentity
fields = ['id', 'handle', 'public_key']
def get_handle(self, obj):
return obj.username + '@' + obj.domain
class GroupSerializer(serializers.ModelSerializer):
handle = serializers.SerializerMethodField()
members = GroupMemberSerializer(many=True, read_only=True)
class Meta:
model = Group
fields = ['id', 'name', 'domain', 'handle', 'members']
read_only_fields = ['domain', 'handle', 'members']
def get_handle(self, obj):
return str(obj)
class GroupIdMapSerializer(serializers.ModelSerializer):
handle = serializers.SerializerMethodField()
class Meta:
model = Group
fields = ['id', 'name', 'domain', 'handle']
def get_handle(self, obj):
return str(obj)
class GroupInviteIncomingSerializer(serializers.ModelSerializer):
group = serializers.SerializerMethodField()
inviter = serializers.SerializerMethodField()
class Meta:
model = GroupInviteIncoming
fields = ['id', 'group', 'inviter', 'secret']
def get_group(self, obj):
return f"#{obj.group_name}@{obj.group_domain}"
def get_inviter(self, obj):
return obj.inviter_username + '@' + obj.inviter_domain
class GroupMembershipSerializer(serializers.ModelSerializer):
handle = serializers.SerializerMethodField()
class Meta:
model = GroupMembership
fields = ['id', 'group_name', 'group_domain', 'handle', 'created_at']
def get_handle(self, obj):
return f"#{obj.group_name}@{obj.group_domain}"
class PropertySerializer(serializers.ModelSerializer): class PropertySerializer(serializers.ModelSerializer):
category = serializers.SerializerMethodField() category = serializers.SlugRelatedField(queryset=Category.objects.all(), slug_field='name')
handle = serializers.SerializerMethodField()
def get_category(self, obj):
return resolve_category_handle(obj.category.get_handle()) if obj.category else None
def get_handle(self, obj):
return obj.get_handle()
class Meta: class Meta:
model = Property model = Property
fields = ['name', 'description', 'category', 'unit_symbol', 'unit_name', 'unit_name_plural', 'base2_prefix', 'handle'] fields = ['name', 'description', 'category', 'unit_symbol', 'unit_name', 'unit_name_plural', 'base2_prefix']
class CategorySerializer(serializers.ModelSerializer): class CategorySerializer(serializers.ModelSerializer):
handle = serializers.SerializerMethodField()
class Meta: class Meta:
model = Category model = Category
fields = ['name', 'handle'] fields = ['name']
def get_handle(self, obj):
return obj.get_handle()
def to_representation(self, instance): def to_representation(self, instance):
return instance.name return str(instance)
def to_internal_value(self, data): def to_internal_value(self, data):
return resolve_category_handle(data.split("/")[-1]) return Category.objects.get(name=data.split("/")[-1])
class OwnerScopedPrimaryKeyRelatedField(serializers.PrimaryKeyRelatedField):
"""Resolves/represents by the owner-scoped `id` rather than the model's internal pk, scoped to
the requesting user - StorageLocation.parent points at another StorageLocation, whose publicly
visible identity is now the owner-scoped id (see StorageLocation.create_for_owner), not
internal_id."""
def use_pk_only_optimization(self):
# False: to_representation needs the owner-scoped `id`, not just the internal pk that the
# PKOnlyObject optimization would otherwise limit us to.
return False
def get_queryset(self):
queryset = super().get_queryset()
request = self.context.get('request')
if request is None or type(request.user) != KnownIdentity:
return queryset.none()
identity = request.user
owner_filter = Q(owner_group__in=identity.member_of_groups.all())
if identity.user.exists():
owner_filter |= Q(owner=identity.user.get())
return queryset.filter(owner_filter)
def to_internal_value(self, data):
queryset = self.get_queryset()
try:
if isinstance(data, bool):
raise TypeError
return queryset.get(id=data)
except ObjectDoesNotExist:
self.fail('does_not_exist', pk_value=data)
except (TypeError, ValueError):
self.fail('incorrect_type', data_type=type(data).__name__)
def to_representation(self, value):
return value.id
class StorageLocationSerializer(serializers.ModelSerializer): class StorageLocationSerializer(serializers.ModelSerializer):
owner = OwnerSerializer(read_only=True) owner = OwnerSerializer(read_only=True)
owner_group = GroupOwnerSerializer(read_only=True) category = CategorySerializer(required=False, allow_null=True)
category = serializers.CharField(required=False, allow_null=True, allow_blank=True)
parent = OwnerScopedPrimaryKeyRelatedField(queryset=StorageLocation.objects.all(), required=False,
allow_null=True)
path = serializers.SerializerMethodField() path = serializers.SerializerMethodField()
class Meta: class Meta:
model = StorageLocation model = StorageLocation
fields = ['id', 'name', 'description', 'path', 'category', 'owner', 'owner_group', 'parent', fields = ['id', 'name', 'description', 'path', 'category', 'owner']
'visibility_policy'] read_only_fields = ['path']
read_only_fields = ['id', 'path']
@staticmethod @staticmethod
def get_path(obj): def get_path(obj):
@ -221,56 +64,38 @@ class StorageLocationSerializer(serializers.ModelSerializer):
return StorageLocationSerializer.get_path(obj.parent) + "/" + obj.name return StorageLocationSerializer.get_path(obj.parent) + "/" + obj.name
return obj.name return obj.name
def create(self, validated_data):
return StorageLocation.create_for_owner(**validated_data)
class ItemPropertySerializer(serializers.ModelSerializer): class ItemPropertySerializer(serializers.ModelSerializer):
property = PropertySerializer(read_only=True) property = PropertySerializer(read_only=True)
handle = serializers.SerializerMethodField()
class Meta: class Meta:
model = ItemProperty model = ItemProperty
fields = ['property', 'value', 'handle'] fields = ['property', 'value']
def get_handle(self, obj):
return obj.property.get_handle()
def to_representation(self, instance): def to_representation(self, instance):
return {'value': instance.value, 'name': instance.property.name} return {'value': instance.value, 'name': instance.property.name}
def to_internal_value(self, data): def to_internal_value(self, data):
prop = resolve_property_handle(data.get('name') or data.get('handle')) prop = Property.objects.get(name=data['name'])
value = data['value'] value = data['value']
return {'property': prop, 'value': value} return {'property': prop, 'value': value}
class InventoryItemSerializer(serializers.ModelSerializer): class InventoryItemSerializer(serializers.ModelSerializer):
owner = OwnerSerializer(read_only=True) owner = OwnerSerializer(read_only=True)
owner_group = GroupOwnerSerializer(read_only=True) tags = serializers.SlugRelatedField(many=True, required=False, queryset=Tag.objects.all(), slug_field='name')
tags = serializers.SerializerMethodField()
tags_input = serializers.ListField(child=serializers.CharField(), write_only=True, required=False)
properties = ItemPropertySerializer(many=True, required=False, source='itemproperty_set') properties = ItemPropertySerializer(many=True, required=False, source='itemproperty_set')
category = CategorySerializer(required=False, allow_null=True) category = CategorySerializer(required=False, allow_null=True)
files = FileSerializer(many=True, read_only=True) files = FileSerializer(many=True, read_only=True)
storage_location = OwnerScopedPrimaryKeyRelatedField(queryset=StorageLocation.objects.all(), required=False,
allow_null=True)
class Meta: class Meta:
model = InventoryItem model = InventoryItem
fields = ['id', 'name', 'description', 'owner', 'owner_group', 'category', 'availability_policy', fields = ['id', 'name', 'description', 'owner', 'category', 'availability_policy', 'owned_quantity', 'owner',
'visibility_policy', 'owned_quantity', 'tags', 'tags_input', 'properties', 'files', 'tags', 'properties', 'files', 'storage_location']
'storage_location']
read_only_fields = ['id']
def get_tags(self, obj):
return [tag.name for tag in obj.tags.all()]
def to_internal_value(self, data): def to_internal_value(self, data):
files = data.pop('files', []) files = data.pop('files', [])
tags_input = data.pop('tags_input', data.pop('tags', []))
ret = super().to_internal_value(data) ret = super().to_internal_value(data)
ret['tags'] = [resolve_tag_handle(tag) for tag in tags_input]
ret['files'] = files ret['files'] = files
return ret return ret
@ -278,7 +103,7 @@ class InventoryItemSerializer(serializers.ModelSerializer):
tags = validated_data.pop('tags', []) tags = validated_data.pop('tags', [])
props = validated_data.pop('itemproperty_set', []) props = validated_data.pop('itemproperty_set', [])
files = validated_data.pop('files', []) files = validated_data.pop('files', [])
item = InventoryItem.create_for_owner(**validated_data) item = InventoryItem.objects.create(**validated_data)
for tag in tags: for tag in tags:
item.tags.add(tag, through_defaults={}) item.tags.add(tag, through_defaults={})
for prop in props: for prop in props:
@ -313,17 +138,3 @@ class InventoryItemSerializer(serializers.ModelSerializer):
ItemProperty.objects.create(inventory_item=item, property=prop['property'], value=prop['value']) ItemProperty.objects.create(inventory_item=item, property=prop['property'], value=prop['value'])
item.save() item.save()
return item return item
class WorkflowInstanceSerializer(serializers.ModelSerializer):
owner = serializers.StringRelatedField(read_only=True)
# Only the hash is needed to identify a staged file, unlike InventoryItemSerializer.files. See docs/implementation.md#staged-files-are-identified-by-hash-alone.
staged_files = serializers.SerializerMethodField()
class Meta:
model = WorkflowInstance
fields = ['id', 'slug', 'state', 'current_step', 'payload', 'owner', 'staged_files', 'created_at', 'updated_at']
read_only_fields = ['owner', 'staged_files', 'created_at', 'updated_at']
def get_staged_files(self, obj):
return list(obj.staged_files.values_list('hash', flat=True))

55
backend/toolshed/tests/fixtures.py
View file

@ -1,5 +1,4 @@
from toolshed.models import Category, Tag, Property, InventoryItem, ItemProperty, StorageLocation, WorkflowInstance from toolshed.models import Category, Tag, Property, InventoryItem, ItemProperty, StorageLocation
import json
class CategoryTestMixin: class CategoryTestMixin:
@ -35,12 +34,12 @@ class InventoryTestMixin(CategoryTestMixin, TagTestMixin, PropertyTestMixin):
def prepare_inventory(self): def prepare_inventory(self):
self.f['local_user1'].friends.add(self.f['local_user2'].public_identity) self.f['local_user1'].friends.add(self.f['local_user2'].public_identity)
self.f['item1'] = InventoryItem.create_for_owner( self.f['item1'] = InventoryItem.objects.create(
owner=self.f['local_user1'], owned_quantity=1, name='test1', description='test', category=self.f['cat1'], owner=self.f['local_user1'], owned_quantity=1, name='test1', description='test', category=self.f['cat1'],
availability_policy='friends', visibility_policy='friends') availability_policy='friends')
self.f['item2'] = InventoryItem.create_for_owner( self.f['item2'] = InventoryItem.objects.create(
owner=self.f['local_user1'], owned_quantity=1, name='test2', description='test2', category=self.f['cat1'], owner=self.f['local_user1'], owned_quantity=1, name='test2', description='test2', category=self.f['cat1'],
availability_policy='friends', visibility_policy='friends') availability_policy='friends')
self.f['item2'].tags.add(self.f['tag1'], through_defaults={}) self.f['item2'].tags.add(self.f['tag1'], through_defaults={})
self.f['item2'].tags.add(self.f['tag2'], through_defaults={}) self.f['item2'].tags.add(self.f['tag2'], through_defaults={})
ItemProperty.objects.create(inventory_item=self.f['item2'], property=self.f['prop1'], value='value1').save() ItemProperty.objects.create(inventory_item=self.f['item2'], property=self.f['prop1'], value='value1').save()
@ -49,41 +48,9 @@ class InventoryTestMixin(CategoryTestMixin, TagTestMixin, PropertyTestMixin):
class LocationTestMixin: class LocationTestMixin:
def prepare_locations(self): def prepare_locations(self):
self.f['loc1'] = StorageLocation.create_for_owner(name='loc1', owner=self.f['local_user1'], self.f['loc1'] = StorageLocation.objects.create(name='loc1', owner=self.f['local_user1'])
visibility_policy='friends') self.f['loc2'] = StorageLocation.objects.create(name='loc2', owner=self.f['local_user1'],
self.f['loc2'] = StorageLocation.create_for_owner(name='loc2', owner=self.f['local_user1'], category=self.f['cat1'])
category=self.f['cat1'], visibility_policy='friends') self.f['loc3'] = StorageLocation.objects.create(name='loc3', owner=self.f['local_user1'], parent=self.f['loc1'])
self.f['loc3'] = StorageLocation.create_for_owner(name='loc3', owner=self.f['local_user1'], parent=self.f['loc1'], self.f['loc4'] = StorageLocation.objects.create(name='loc4', owner=self.f['local_user1'], parent=self.f['loc1'],
visibility_policy='friends') category=self.f['cat1'])
self.f['loc4'] = StorageLocation.create_for_owner(name='loc4', owner=self.f['local_user1'], parent=self.f['loc1'],
category=self.f['cat1'], visibility_policy='friends')
class WorkflowTestMixin:
def prepare_workflows(self):
# `payload` is an opaque, frontend-serialized JSON string on the backend.
self.f['workflow1'] = WorkflowInstance.objects.create(
name='workflow1',
state='initial',
payload=json.dumps({}),
owner=self.f['local_user1']
)
self.f['workflow2'] = WorkflowInstance.objects.create(
name='workflow1',
state='upload',
payload=json.dumps({'files': ['ef35c4a9b2d1c4f1a3e6f7d8c9b0a1b2']}),
owner=self.f['local_user1']
)
self.f['workflow3'] = WorkflowInstance.objects.create(
name='workflow1',
state='describe',
payload=json.dumps({'files': ['ef35c4a9b2d1c4f1a3e6f7d8c9b0a1b2', 'a1b2c3d4e5f60718293a4b5c6d7e8f90', 'b1c2d3e4f5a60718293b4c5d6e7f8090'],
'descriptions': ['file 1 description']}),
owner=self.f['local_user1']
)
self.f['workflow_user2'] = WorkflowInstance.objects.create(
name='workflow2',
state='initial',
payload=json.dumps({}),
owner=self.f['local_user2']
)

27
backend/toolshed/tests/test_api.py
View file

@ -19,53 +19,42 @@ class CombinedApiTestCase(UserTestMixin, CategoryTestMixin, TagTestMixin, Proper
def test_version_anonymous(self): def test_version_anonymous(self):
response = anonymous_client.get('/api/version/') response = anonymous_client.get('/api/version/')
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response.json(), {'version': settings.TOOLSHED_VERSION, 'commit': settings.GIT_COMMIT}) self.assertEqual(response.json(), {'version': settings.TOOLSHED_VERSION})
def test_version_authenticated(self): def test_version_authenticated(self):
response = client.get('/api/version/', self.f['local_user1']) response = client.get('/api/version/', self.f['local_user1'])
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response.json(), {'version': settings.TOOLSHED_VERSION, 'commit': settings.GIT_COMMIT}) self.assertEqual(response.json(), {'version': settings.TOOLSHED_VERSION})
def test_domains_anonymous(self): def test_domains_anonymous(self):
response = anonymous_client.get('/api/v1/domains/') response = anonymous_client.get('/api/domains/')
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response.json(), ['example.com']) self.assertEqual(response.json(), ['example.com'])
def test_domains_authenticated(self): def test_domains_authenticated(self):
response = client.get('/api/v1/domains/', self.f['local_user1']) response = client.get('/api/domains/', self.f['local_user1'])
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response.json(), ['example.com']) self.assertEqual(response.json(), ['example.com'])
def test_policy_api_anonymous(self): def test_policy_api_anonymous(self):
response = anonymous_client.get('/api/v1/availability_policies/') response = anonymous_client.get('/api/availability_policies/')
self.assertEqual(response.status_code, 403) self.assertEqual(response.status_code, 403)
def test_policy_api(self): def test_policy_api(self):
response = client.get('/api/v1/availability_policies/', self.f['local_user1']) response = client.get('/api/availability_policies/', self.f['local_user1'])
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response.json(), [['sell', 'Sell'], ['rent', 'Rent'], ['lend', 'Lend'], ['share', 'Share'], self.assertEqual(response.json(), [['sell', 'Sell'], ['rent', 'Rent'], ['lend', 'Lend'], ['share', 'Share'],
['private', 'Private']]) ['private', 'Private']])
def test_visibility_policy_api_anonymous(self):
response = anonymous_client.get('/api/v1/visibility_policies/')
self.assertEqual(response.status_code, 403)
def test_visibility_policy_api(self):
response = client.get('/api/v1/visibility_policies/', self.f['local_user1'])
self.assertEqual(response.status_code, 200)
self.assertEqual(response.json(), [['public', 'Public'], ['friends', 'Friends'], ['private', 'Private']])
def test_combined_api_anonymous(self): def test_combined_api_anonymous(self):
response = anonymous_client.get('/api/v1/info/') response = anonymous_client.get('/api/info/')
self.assertEqual(response.status_code, 403) self.assertEqual(response.status_code, 403)
def test_combined_api(self): def test_combined_api(self):
response = client.get('/api/v1/info/', self.f['local_user1']) response = client.get('/api/info/', self.f['local_user1'])
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(response.json()['availability_policies'], [['sell', 'Sell'], ['rent', 'Rent'], ['lend', 'Lend'], self.assertEqual(response.json()['availability_policies'], [['sell', 'Sell'], ['rent', 'Rent'], ['lend', 'Lend'],
['share', 'Share'], ['private', 'Private']]) ['share', 'Share'], ['private', 'Private']])
self.assertEqual(response.json()['visibility_policies'],
[['public', 'Public'], ['friends', 'Friends'], ['private', 'Private']])
self.assertEqual(response.json()['categories'], self.assertEqual(response.json()['categories'],
['cat1', 'cat2', 'cat3', 'cat1/subcat1', 'cat1/subcat2', 'cat1/subcat1/subcat1', ['cat1', 'cat2', 'cat3', 'cat1/subcat1', 'cat1/subcat2', 'cat1/subcat1/subcat1',
'cat1/subcat1/subcat2']) 'cat1/subcat1/subcat2'])

2
backend/toolshed/tests/test_category.py
View file

@ -32,7 +32,7 @@ class CategoryApiTestCase(CategoryTestMixin, UserTestMixin, ToolshedTestCase):
self.prepare_categories() self.prepare_categories()
def test_get_categories(self): def test_get_categories(self):
reply = client.get('/api/v1/categories/', self.f['local_user1']) reply = client.get('/api/categories/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 7) self.assertEqual(len(reply.json()), 7)
self.assertEqual(reply.json()[0], 'cat1') self.assertEqual(reply.json()[0], 'cat1')

179
backend/toolshed/tests/test_files.py
View file

@ -1,8 +1,7 @@
from django.test import Client from django.test import Client
from authentication.models import Group, ToolshedUser from authentication.tests import SignatureAuthClient, UserTestMixin, ToolshedTestCase
from authentication.tests import SignatureAuthClient, UserTestMixin, GroupTestMixin, ToolshedTestCase
from files.tests import FilesTestMixin from files.tests import FilesTestMixin
from toolshed.models import File, InventoryItem from toolshed.models import File
from toolshed.tests import InventoryTestMixin from toolshed.tests import InventoryTestMixin
@ -25,11 +24,11 @@ class FileApiTestCase(UserTestMixin, FilesTestMixin, InventoryTestMixin, Toolshe
self.f['item2'].files.add(self.f['test_file1']) self.f['item2'].files.add(self.f['test_file1'])
def test_files_anonymous(self): def test_files_anonymous(self):
response = anonymous_client.get(f"/api/v1/item_files/{self.f['item1'].id}/") response = anonymous_client.get(f"/api/item_files/{self.f['item1'].id}/")
self.assertEqual(response.status_code, 403) self.assertEqual(response.status_code, 403)
def test_list_all_files(self): def test_list_all_files(self):
response = client.get(f"/api/v1/files/", self.f['local_user1']) response = client.get(f"/api/files/", self.f['local_user1'])
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(len(response.json()), 2) self.assertEqual(len(response.json()), 2)
self.assertEqual(response.json()[0]['mime_type'], 'text/plain') self.assertEqual(response.json()[0]['mime_type'], 'text/plain')
@ -39,19 +38,8 @@ class FileApiTestCase(UserTestMixin, FilesTestMixin, InventoryTestMixin, Toolshe
self.assertEqual(response.json()[1]['name'], self.assertEqual(response.json()[1]['name'],
f"/media/{self.f['hash2'][:2]}/{self.f['hash2'][2:4]}/{self.f['hash2'][4:6]}/{self.f['hash2'][6:]}") f"/media/{self.f['hash2'][:2]}/{self.f['hash2'][2:4]}/{self.f['hash2'][4:6]}/{self.f['hash2'][6:]}")
def test_list_all_files_excludes_files_only_connected_via_deleted_item(self):
# test_file2 is only reachable through item1; test_file1 is also reachable through item2,
# which stays live. Soft-deleting item1 doesn't sever its files M2M rows, so this would
# regress to listing test_file2 as if it were still owned if the join-based filter ever
# stops excluding soft-deleted items again.
self.f['item1'].delete()
response = client.get(f"/api/v1/files/", self.f['local_user1'])
self.assertEqual(response.status_code, 200)
hashes = [f['hash'] for f in response.json()]
self.assertEqual(hashes, [self.f['hash1']])
def test_files(self): def test_files(self):
response = client.get(f"/api/v1/item_files/{self.f['item1'].id}/", self.f['local_user1']) response = client.get(f"/api/item_files/{self.f['item1'].id}/", self.f['local_user1'])
self.assertEqual(response.status_code, 200) self.assertEqual(response.status_code, 200)
self.assertEqual(len(response.json()), 2) self.assertEqual(len(response.json()), 2)
self.assertEqual(response.json()[0]['mime_type'], 'text/plain') self.assertEqual(response.json()[0]['mime_type'], 'text/plain')
@ -62,11 +50,11 @@ class FileApiTestCase(UserTestMixin, FilesTestMixin, InventoryTestMixin, Toolshe
f"/media/{self.f['hash2'][:2]}/{self.f['hash2'][2:4]}/{self.f['hash2'][4:6]}/{self.f['hash2'][6:]}") f"/media/{self.f['hash2'][:2]}/{self.f['hash2'][2:4]}/{self.f['hash2'][4:6]}/{self.f['hash2'][6:]}")
def test_files_not_found(self): def test_files_not_found(self):
response = client.get(f"/api/v1/item_files/99999/", self.f['local_user1']) response = client.get(f"/api/item_files/99999/", self.f['local_user1'])
self.assertEqual(response.status_code, 404) self.assertEqual(response.status_code, 404)
def test_post_file(self): def test_post_file(self):
response = client.post(f"/api/v1/item_files/{self.f['item1'].id}/", self.f['local_user1'], response = client.post(f"/api/item_files/{self.f['item1'].id}/", self.f['local_user1'],
{'data': self.f['encoded_content4'], 'mime_type': 'text/plain'}) {'data': self.f['encoded_content4'], 'mime_type': 'text/plain'})
self.assertEqual(response.status_code, 201) self.assertEqual(response.status_code, 201)
self.assertEqual(File.objects.count(), 4) self.assertEqual(File.objects.count(), 4)
@ -79,7 +67,7 @@ class FileApiTestCase(UserTestMixin, FilesTestMixin, InventoryTestMixin, Toolshe
def test_post_file_duplicate(self): def test_post_file_duplicate(self):
self.assertEqual(File.objects.count(), 3) self.assertEqual(File.objects.count(), 3)
self.assertEqual(self.f['item1'].files.count(), 2) self.assertEqual(self.f['item1'].files.count(), 2)
response = client.post(f"/api/v1/item_files/{self.f['item1'].id}/", self.f['local_user1'], response = client.post(f"/api/item_files/{self.f['item1'].id}/", self.f['local_user1'],
{'data': self.f['encoded_content3'], 'mime_type': 'text/plain'}) {'data': self.f['encoded_content3'], 'mime_type': 'text/plain'})
self.assertEqual(response.status_code, 201) self.assertEqual(response.status_code, 201)
self.assertEqual(File.objects.count(), 3) self.assertEqual(File.objects.count(), 3)
@ -90,112 +78,69 @@ class FileApiTestCase(UserTestMixin, FilesTestMixin, InventoryTestMixin, Toolshe
f"{self.f['hash3'][:2]}/{self.f['hash3'][2:4]}/{self.f['hash3'][4:6]}/{self.f['hash3'][6:]}") f"{self.f['hash3'][:2]}/{self.f['hash3'][2:4]}/{self.f['hash3'][4:6]}/{self.f['hash3'][6:]}")
def test_post_file_invalid(self): def test_post_file_invalid(self):
response = client.post(f"/api/v1/item_files/{self.f['item1'].id}/", self.f['local_user1'], response = client.post(f"/api/item_files/{self.f['item1'].id}/", self.f['local_user1'],
{'data': self.f['encoded_content4']}) {'data': self.f['encoded_content4']})
self.assertEqual(response.status_code, 400) self.assertEqual(response.status_code, 400)
def test_post_file_not_found_item(self): def test_post_file_not_found_item(self):
response = client.post(f"/api/v1/item_files/99999/", self.f['local_user1'], response = client.post(f"/api/item_files/99999/", self.f['local_user1'],
{'data': self.f['encoded_content3'], 'mime_type': 'text/plain'}) {'data': self.f['encoded_content3'], 'mime_type': 'text/plain'})
self.assertEqual(response.status_code, 404) self.assertEqual(response.status_code, 404)
self.assertEqual(File.objects.count(), 3) self.assertEqual(File.objects.count(), 3)
def test_post_file_not_authenticated(self): def test_post_file_not_authenticated(self):
response = anonymous_client.post(f"/api/v1/item_files/{self.f['item1'].id}/", response = anonymous_client.post(f"/api/item_files/{self.f['item1'].id}/",
{'data': self.f['encoded_content3'], 'mime_type': 'text/plain'}) {'data': self.f['encoded_content3'], 'mime_type': 'text/plain'})
self.assertEqual(response.status_code, 403) self.assertEqual(response.status_code, 403)
self.assertEqual(File.objects.count(), 3) self.assertEqual(File.objects.count(), 3)
def test_post_file_not_authorized(self): def test_post_file_not_authorized(self):
response = client.post(f"/api/v1/item_files/{self.f['item1'].id}/", self.f['local_user2'], response = client.post(f"/api/item_files/{self.f['item1'].id}/", self.f['local_user2'],
{'data': self.f['encoded_content3'], 'mime_type': 'text/plain'}) {'data': self.f['encoded_content3'], 'mime_type': 'text/plain'})
self.assertEqual(response.status_code, 404) self.assertEqual(response.status_code, 404)
self.assertEqual(File.objects.count(), 3) self.assertEqual(File.objects.count(), 3)
def test_delete_file(self): def test_delete_file(self):
response = client.delete(f"/api/v1/item_files/{self.f['item1'].id}/{self.f['test_file1'].id}/", response = client.delete(f"/api/item_files/{self.f['item1'].id}/{self.f['test_file1'].id}/",
self.f['local_user1']) self.f['local_user1'])
self.assertEqual(response.status_code, 204) self.assertEqual(response.status_code, 204)
self.assertEqual(File.objects.count(), 3) self.assertEqual(File.objects.count(), 3)
self.assertEqual(self.f['item1'].files.count(), 1) self.assertEqual(self.f['item1'].files.count(), 1)
def test_delete_file_last_use(self): def test_delete_file_last_use(self):
response = client.delete(f"/api/v1/item_files/{self.f['item1'].id}/{self.f['test_file2'].id}/", response = client.delete(f"/api/item_files/{self.f['item1'].id}/{self.f['test_file2'].id}/",
self.f['local_user1']) self.f['local_user1'])
self.assertEqual(response.status_code, 204) self.assertEqual(response.status_code, 204)
self.assertEqual(File.objects.count(), 2) self.assertEqual(File.objects.count(), 2)
self.assertEqual(self.f['item1'].files.count(), 1) self.assertEqual(self.f['item1'].files.count(), 1)
def test_delete_file_not_found(self): def test_delete_file_not_found(self):
response = client.delete(f"/api/v1/item_files/{self.f['item1'].id}/99999/", self.f['local_user1']) response = client.delete(f"/api/item_files/{self.f['item1'].id}/99999/", self.f['local_user1'])
self.assertEqual(response.status_code, 404) self.assertEqual(response.status_code, 404)
self.assertEqual(File.objects.count(), 3) self.assertEqual(File.objects.count(), 3)
self.assertEqual(self.f['item1'].files.count(), 2) self.assertEqual(self.f['item1'].files.count(), 2)
def test_delete_file_not_found_item(self): def test_delete_file_not_found_item(self):
response = client.delete(f"/api/v1/item_files/99999/{self.f['test_file1'].id}/", self.f['local_user1']) response = client.delete(f"/api/item_files/99999/{self.f['test_file1'].id}/", self.f['local_user1'])
self.assertEqual(response.status_code, 404) self.assertEqual(response.status_code, 404)
self.assertEqual(File.objects.count(), 3) self.assertEqual(File.objects.count(), 3)
self.assertEqual(self.f['item1'].files.count(), 2) self.assertEqual(self.f['item1'].files.count(), 2)
def test_delete_file_not_owner(self): def test_delete_file_not_owner(self):
response = client.delete(f"/api/v1/item_files/{self.f['item1'].id}/{self.f['test_file1'].id}/", response = client.delete(f"/api/item_files/{self.f['item1'].id}/{self.f['test_file1'].id}/",
self.f['local_user2']) self.f['local_user2'])
self.assertEqual(response.status_code, 404) self.assertEqual(response.status_code, 404)
self.assertEqual(File.objects.count(), 3) self.assertEqual(File.objects.count(), 3)
self.assertEqual(self.f['item1'].files.count(), 2) self.assertEqual(self.f['item1'].files.count(), 2)
def test_delete_file_anonymous(self): def test_delete_file_anonymous(self):
response = anonymous_client.delete(f"/api/v1/item_files/{self.f['item1'].id}/{self.f['test_file1'].id}/") response = anonymous_client.delete(f"/api/item_files/{self.f['item1'].id}/{self.f['test_file1'].id}/")
self.assertEqual(response.status_code, 403) self.assertEqual(response.status_code, 403)
self.assertEqual(File.objects.count(), 3) self.assertEqual(File.objects.count(), 3)
self.assertEqual(self.f['item1'].files.count(), 2) self.assertEqual(self.f['item1'].files.count(), 2)
def test_delete_file_top_level(self):
response = client.delete(f"/api/v1/files/{self.f['test_file2'].id}/", self.f['local_user1'])
self.assertEqual(response.status_code, 204)
self.assertEqual(File.objects.count(), 2)
self.assertEqual(self.f['item1'].files.count(), 1)
def test_delete_file_top_level_removes_from_all_owned_items(self):
response = client.delete(f"/api/v1/files/{self.f['test_file1'].id}/", self.f['local_user1'])
self.assertEqual(response.status_code, 204)
self.assertEqual(File.objects.count(), 2)
self.assertEqual(self.f['item1'].files.count(), 1)
self.assertEqual(self.f['item2'].files.count(), 0)
def test_delete_file_top_level_not_found(self):
response = client.delete(f"/api/v1/files/99999/", self.f['local_user1'])
self.assertEqual(response.status_code, 404)
self.assertEqual(File.objects.count(), 3)
def test_delete_file_top_level_not_owner(self):
response = client.delete(f"/api/v1/files/{self.f['test_file1'].id}/", self.f['local_user2'])
self.assertEqual(response.status_code, 404)
self.assertEqual(File.objects.count(), 3)
self.assertEqual(self.f['item1'].files.count(), 2)
def test_delete_file_top_level_anonymous(self):
response = anonymous_client.delete(f"/api/v1/files/{self.f['test_file1'].id}/")
self.assertEqual(response.status_code, 403)
self.assertEqual(File.objects.count(), 3)
def test_delete_file_top_level_only_detaches_own_items_when_shared_by_hash(self):
other_item = InventoryItem.create_for_owner(
owner=self.f['local_user2'], owned_quantity=1, name='other-user-item',
availability_policy='private')
other_item.files.add(self.f['test_file1'])
response = client.delete(f"/api/v1/files/{self.f['test_file1'].id}/", self.f['local_user1'])
self.assertEqual(response.status_code, 204)
self.assertEqual(self.f['item1'].files.count(), 1)
self.assertEqual(self.f['item2'].files.count(), 0)
self.assertEqual(other_item.files.count(), 1)
self.assertEqual(File.objects.filter(id=self.f['test_file1'].id).count(), 1)
def test_get_inventory(self): def test_get_inventory(self):
reply = client.get('/api/v1/inventory_items/{}/'.format(self.f['local_user1']), self.f['local_user1']) reply = client.get('/api/inventory_items/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 2) self.assertEqual(len(reply.json()), 2)
self.assertEqual(reply.json()[0]['name'], 'test1') self.assertEqual(reply.json()[0]['name'], 'test1')
@ -211,89 +156,3 @@ class FileApiTestCase(UserTestMixin, FilesTestMixin, InventoryTestMixin, Toolshe
self.assertEqual(reply.json()[0]['files'][0]['mime_type'], 'text/plain') self.assertEqual(reply.json()[0]['files'][0]['mime_type'], 'text/plain')
self.assertEqual(reply.json()[0]['files'][1]['mime_type'], 'text/plain') self.assertEqual(reply.json()[0]['files'][1]['mime_type'], 'text/plain')
self.assertEqual(reply.json()[1]['files'][0]['mime_type'], 'text/plain') self.assertEqual(reply.json()[1]['files'][0]['mime_type'], 'text/plain')
class GroupOwnedFileApiTestCase(UserTestMixin, GroupTestMixin, FilesTestMixin, ToolshedTestCase):
def setUp(self):
super().setUp()
self.prepare_users()
self.prepare_groups()
self.prepare_files()
self.f['group1'].members.add(self.f['local_user2'].public_identity)
self.f['group_item'] = InventoryItem.create_for_owner(
owner_group=self.f['group1'], owned_quantity=1, name='group-drill', availability_policy='private')
self.f['group_item'].files.add(self.f['test_file1'])
def test_get_group_item_files(self):
response = client.get(f"/api/v1/item_files/{self.f['group_item'].id}/", self.f['local_user1'])
self.assertEqual(response.status_code, 200)
self.assertEqual(len(response.json()), 1)
def test_other_member_can_post_file(self):
response = client.post(f"/api/v1/item_files/{self.f['group_item'].id}/", self.f['local_user2'],
{'data': self.f['encoded_content4'], 'mime_type': 'text/plain'})
self.assertEqual(response.status_code, 201)
self.assertEqual(self.f['group_item'].files.count(), 2)
def test_remote_member_without_local_account_can_post_file(self):
self.f['group1'].members.add(self.f['ext_user1'].public_identity)
response = client.post(f"/api/v1/item_files/{self.f['group_item'].id}/", self.f['ext_user1'],
{'data': self.f['encoded_content4'], 'mime_type': 'text/plain'})
self.assertEqual(response.status_code, 201)
self.assertEqual(self.f['group_item'].files.count(), 2)
def test_remote_member_without_local_account_can_get_files(self):
self.f['group1'].members.add(self.f['ext_user1'].public_identity)
response = client.get(f"/api/v1/item_files/{self.f['group_item'].id}/", self.f['ext_user1'])
self.assertEqual(response.status_code, 200)
def test_non_member_cannot_post_file(self):
response = client.post(f"/api/v1/item_files/{self.f['group_item'].id}/", self.f['ext_user1'],
{'data': self.f['encoded_content4'], 'mime_type': 'text/plain'})
self.assertEqual(response.status_code, 404)
self.assertEqual(self.f['group_item'].files.count(), 1)
def test_non_member_cannot_get_files(self):
response = client.get(f"/api/v1/item_files/{self.f['group_item'].id}/", self.f['ext_user1'])
self.assertEqual(response.status_code, 404)
def test_other_member_can_delete_file(self):
response = client.delete(f"/api/v1/item_files/{self.f['group_item'].id}/{self.f['test_file1'].id}/",
self.f['local_user2'])
self.assertEqual(response.status_code, 204)
self.assertEqual(self.f['group_item'].files.count(), 0)
def test_non_member_cannot_delete_file(self):
response = client.delete(f"/api/v1/item_files/{self.f['group_item'].id}/{self.f['test_file1'].id}/",
self.f['ext_user1'])
self.assertEqual(response.status_code, 404)
self.assertEqual(self.f['group_item'].files.count(), 1)
def test_other_member_can_delete_file_top_level(self):
response = client.delete(f"/api/v1/files/{self.f['test_file1'].id}/", self.f['local_user2'])
self.assertEqual(response.status_code, 204)
self.assertEqual(self.f['group_item'].files.count(), 0)
def test_non_member_cannot_delete_file_top_level(self):
outsider = ToolshedUser.objects.create_user('testuser3', 'test3@abc.de', 'testpassword4',
domain=self.f['example_com'].name)
response = client.delete(f"/api/v1/files/{self.f['test_file1'].id}/", outsider)
self.assertEqual(response.status_code, 404)
self.assertEqual(self.f['group_item'].files.count(), 1)
def test_item_files_when_id_collides_across_two_groups(self):
group2 = Group.objects.create(name='group2', domain=self.f['example_com'].name)
group2.members.add(self.f['local_user2'].public_identity)
group2_item = InventoryItem.create_for_owner(
owner_group=group2, owned_quantity=1, name='group2-drill', availability_policy='private')
self.assertEqual(self.f['group_item'].id, group2_item.id)
response = client.get(f"/api/v1/item_files/{self.f['group_item'].id}/", self.f['local_user2'])
self.assertEqual(response.status_code, 200)
def test_group_item_files_listed_in_all_files(self):
response = client.get('/api/v1/files/', self.f['local_user2'])
self.assertEqual(response.status_code, 200)
self.assertEqual(len(response.json()), 1)
self.assertEqual(response.json()[0]['id'], self.f['test_file1'].id)

82
backend/toolshed/tests/test_friend.py
View file

@ -60,33 +60,45 @@ class FriendApiTestCase(UserTestMixin, ToolshedTestCase):
self.f['ext_user1'].friends.add(self.f['local_user1'].public_identity) self.f['ext_user1'].friends.add(self.f['local_user1'].public_identity)
def test_friend_list_internal1(self): def test_friend_list_internal1(self):
reply = client.get('/api/v1/friends/', self.f['local_user1']) reply = client.get('/api/friends/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 2) self.assertEqual(len(reply.json()), 2)
self.assertEqual(reply.json()[0]['handle'], str(self.f['local_user2'])) self.assertEqual(reply.json()[0]['username'], str(self.f['local_user2']))
self.assertEqual(reply.json()[1]['handle'], str(self.f['ext_user1'])) self.assertEqual(reply.json()[1]['username'], str(self.f['ext_user1']))
def test_friend_list_internal2(self): def test_friend_list_internal2(self):
reply = client.get('/api/v1/friends/', self.f['local_user2']) reply = client.get('/api/friends/', self.f['local_user2'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 1) self.assertEqual(len(reply.json()), 1)
self.assertEqual(reply.json()[0]['handle'], str(self.f['local_user1'])) self.assertEqual(reply.json()[0]['username'], str(self.f['local_user1']))
def test_friend_list_external(self): def test_friend_list_external(self):
reply = client.get('/api/v1/friends/', self.f['ext_user1']) reply = client.get('/api/friends/', self.f['ext_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 1) self.assertEqual(len(reply.json()), 1)
self.assertEqual(reply.json()[0]['handle'], str(self.f['local_user1'])) self.assertEqual(reply.json()[0]['username'], str(self.f['local_user1']))
def test_friend_delete(self): def test_friend_delete(self):
reply = client.delete('/api/v1/friends/{}/'.format(self.f['local_user2'].public_identity.id), reply = client.delete('/api/friends/{}/'.format(self.f['local_user2'].public_identity.id),
self.f['local_user1']) self.f['local_user1'])
self.assertEqual(reply.status_code, 204) self.assertEqual(reply.status_code, 204)
self.assertEqual(self.f['local_user1'].friends.count(), 1) self.assertEqual(self.f['local_user1'].friends.count(), 1)
# Friend request/accept protocol walkthrough. See # what ~should~ happen:
# docs/implementation.md#friend-request-and-accept-protocol-flow. # 1. user x@A sends a friend request to user y@B
# 1.1. x@A's client sends a POST request to A/api/friendrequests/ with body {from: x@A, to: y@B}
# 1.2. A's backend creates a FriendRequestOutgoing object, containing x@A's identity and y@B's name
# 1.3. x@A's client sends a POST request to B/api/friendrequests/ with body
# {from: x@A, to: y@B, public_key: x@A's public key}
# 1.4. B's backend creates a FriendRequestIncoming object, containing y@B's and x@A's identities
# 2. user y@B accepts the friend request
# 2.1. y@B's client sends a POST request to A/api/friendsrequests/ with body
# {from: x@A, to: y@B, public_key: y@B's public key}
# 2.2. A's backend matches the data to the FriendRequestOutgoing object, deletes both and creates a Friend object,
# containing x@A's and y@B's identities
# 2.3. y@B's client sends a POST request to B/api/friends/ containing the id of the FriendRequestIncoming object
# 2.4. B's backend creates a Friend object, using the identities from the FriendRequestIncoming object
class FriendRequestListTestCase(UserTestMixin, ToolshedTestCase): class FriendRequestListTestCase(UserTestMixin, ToolshedTestCase):
@ -101,29 +113,29 @@ class FriendRequestListTestCase(UserTestMixin, ToolshedTestCase):
self.friendrequest1.save() self.friendrequest1.save()
def test_friend_request_withouth_auth(self): def test_friend_request_withouth_auth(self):
reply = Client().get('/api/v1/friendrequests/') reply = Client().get('/api/friendrequests/')
self.assertEqual(reply.status_code, 401) self.assertEqual(reply.status_code, 401)
def test_friend_request_empty(self): def test_friend_request_empty(self):
reply = client.get('/api/v1/friendrequests/', self.f['local_user2']) reply = client.get('/api/friendrequests/', self.f['local_user2'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json(), []) self.assertEqual(reply.json(), [])
def test_friend_request_list(self): def test_friend_request_list(self):
reply = client.get('/api/v1/friendrequests/', self.f['local_user1']) reply = client.get('/api/friendrequests/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 1) self.assertEqual(len(reply.json()), 1)
self.assertEqual(reply.json()[0]['befriender'], str(self.f['ext_user2'])) self.assertEqual(reply.json()[0]['befriender'], str(self.f['ext_user2']))
self.assertEqual(reply.json()[0]['befriender_public_key'], self.f['ext_user2'].public_key()) self.assertEqual(reply.json()[0]['befriender_public_key'], self.f['ext_user2'].public_key())
def test_delete_friend_request(self): def test_delete_friend_request(self):
reply = client.delete('/api/v1/friendrequests/{}/'.format(self.friendrequest1.id), reply = client.delete('/api/friendrequests/{}/'.format(self.friendrequest1.id),
self.f['local_user1']) self.f['local_user1'])
self.assertEqual(reply.status_code, 204) self.assertEqual(reply.status_code, 204)
self.assertEqual(FriendRequestIncoming.objects.count(), 0) self.assertEqual(FriendRequestIncoming.objects.count(), 0)
def test_delete_friend_request_not_found(self): def test_delete_friend_request_not_found(self):
reply = client.delete('/api/v1/friendrequests/999/', self.f['local_user1']) reply = client.delete('/api/friendrequests/999/', self.f['local_user1'])
self.assertEqual(reply.status_code, 404) self.assertEqual(reply.status_code, 404)
self.assertEqual(FriendRequestIncoming.objects.count(), 1) self.assertEqual(FriendRequestIncoming.objects.count(), 1)
@ -140,7 +152,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
def test_post_request(self): def test_post_request(self):
befriender = self.f['ext_user1'] befriender = self.f['ext_user1']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
reply = client.post('/api/v1/friendrequests/', befriender, { reply = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriender_key': befriender.public_key(), 'befriender_key': befriender.public_key(),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
@ -158,7 +170,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
def test_post_request_local(self): def test_post_request_local(self):
befriender = self.f['local_user2'] befriender = self.f['local_user2']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
reply = client.post('/api/v1/friendrequests/', befriender, { reply = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
# 'secret': 'secret2' # 'secret': 'secret2'
@ -173,14 +185,14 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
# self.assertEqual(incoming.secret, 'secret2') # self.assertEqual(incoming.secret, 'secret2')
def test_post_request_withouth_auth(self): def test_post_request_withouth_auth(self):
reply = Client().post('/api/v1/friendrequests/') reply = Client().post('/api/friendrequests/')
self.assertEqual(reply.status_code, 400) self.assertEqual(reply.status_code, 400)
def test_post_request_broken_header(self): def test_post_request_broken_header(self):
befriender = self.f['ext_user1'] befriender = self.f['ext_user1']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
broken_client = SignatureAuthClient(header_prefix='broken ') broken_client = SignatureAuthClient(header_prefix='broken ')
reply = broken_client.post('/api/v1/friendrequests/', befriender, { reply = broken_client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriender_key': befriender.public_key(), 'befriender_key': befriender.public_key(),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
@ -191,7 +203,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
def test_post_request_missing_key(self): def test_post_request_missing_key(self):
befriender = self.f['ext_user1'] befriender = self.f['ext_user1']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
reply = client.post('/api/v1/friendrequests/', befriender, { reply = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
'secret': 'secret2' 'secret': 'secret2'
@ -201,7 +213,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
def test_post_request_missing_key_none(self): def test_post_request_missing_key_none(self):
befriender = self.f['ext_user1'] befriender = self.f['ext_user1']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
reply = client.post('/api/v1/friendrequests/', befriender, { reply = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
'befriender_key': None, 'befriender_key': None,
@ -212,7 +224,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
def test_post_request_breaking_key(self): def test_post_request_breaking_key(self):
befriender = self.f['ext_user1'] befriender = self.f['ext_user1']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
reply = client.post('/api/v1/friendrequests/', befriender, { reply = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
'secret': 'secret2', 'secret': 'secret2',
@ -223,7 +235,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
def test_post_request_wrong_befriender(self): def test_post_request_wrong_befriender(self):
befriender = self.f['ext_user1'] befriender = self.f['ext_user1']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
reply = client.post('/api/v1/friendrequests/', befriender, { reply = client.post('/api/friendrequests/', befriender, {
'befriender': str(self.f['local_user2']), 'befriender': str(self.f['local_user2']),
'befriender_key': befriender.public_key(), 'befriender_key': befriender.public_key(),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
@ -235,7 +247,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
befriender = self.f['ext_user1'] befriender = self.f['ext_user1']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
bad_signature = SignatureAuthClient(bad_signature=True) bad_signature = SignatureAuthClient(bad_signature=True)
reply = bad_signature.post('/api/v1/friendrequests/', befriender, { reply = bad_signature.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriender_key': befriender.public_key(), 'befriender_key': befriender.public_key(),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
@ -246,7 +258,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
def test_post_request_self(self): def test_post_request_self(self):
befriender = self.f['local_user1'] befriender = self.f['local_user1']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
reply = client.post('/api/v1/friendrequests/', befriender, { reply = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriender_key': befriender.public_key(), 'befriender_key': befriender.public_key(),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
@ -257,7 +269,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
def test_post_request_befreindee_not_found(self): def test_post_request_befreindee_not_found(self):
befriender = self.f['ext_user1'] befriender = self.f['ext_user1']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
reply = client.post('/api/v1/friendrequests/', befriender, { reply = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriender_key': befriender.public_key(), 'befriender_key': befriender.public_key(),
'befriendee': 'nonexistent@' + befriendee.domain, 'befriendee': 'nonexistent@' + befriendee.domain,
@ -268,7 +280,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
def test_post_request_missing_secret(self): def test_post_request_missing_secret(self):
befriender = self.f['ext_user1'] befriender = self.f['ext_user1']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
reply = client.post('/api/v1/friendrequests/', befriender, { reply = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriender_key': befriender.public_key(), 'befriender_key': befriender.public_key(),
'befriendee': str(befriendee) 'befriendee': str(befriendee)
@ -281,7 +293,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
request = FriendRequestIncoming.objects.filter(befriender_username=befriender.username, request = FriendRequestIncoming.objects.filter(befriender_username=befriender.username,
befriender_domain=befriender.domain, befriender_domain=befriender.domain,
befriendee_user=befriendee).first() befriendee_user=befriendee).first()
reply = client.post('/api/v1/friends/', befriendee, { reply = client.post('/api/friends/', befriendee, {
'friend_request_id': request.id, 'friend_request_id': request.id,
'secret': request.secret 'secret': request.secret
}) })
@ -294,7 +306,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
request = FriendRequestIncoming.objects.filter(befriender_username=befriender.username, request = FriendRequestIncoming.objects.filter(befriender_username=befriender.username,
befriender_domain=befriender.domain, befriender_domain=befriender.domain,
befriendee_user=befriendee).first() befriendee_user=befriendee).first()
reply = client.post('/api/v1/friends/', befriendee, { reply = client.post('/api/friends/', befriendee, {
'friend_request_id': request.id, 'friend_request_id': request.id,
'secret': request.secret 'secret': request.secret
}) })
@ -304,7 +316,7 @@ class FriendRequestIncomingTestCase(UserTestMixin, ToolshedTestCase):
def test_accept_request_not_found(self): def test_accept_request_not_found(self):
befriender = self.f['ext_user2'] befriender = self.f['ext_user2']
befriendee = self.f['local_user1'] befriendee = self.f['local_user1']
reply = client.post('/api/v1/friends/', befriendee, { reply = client.post('/api/friends/', befriendee, {
'friend_request_id': 999, 'friend_request_id': 999,
'secret': 'secret1' 'secret': 'secret1'
}) })
@ -326,7 +338,7 @@ class FriendRequestOutgoingTestCase(UserTestMixin, ToolshedTestCase):
def test_post_outgoing_friend_request(self): def test_post_outgoing_friend_request(self):
befriender = self.f['local_user1'] befriender = self.f['local_user1']
befriendee = self.f['ext_user1'] befriendee = self.f['ext_user1']
reply = client.post('/api/v1/friendrequests/', befriender, { reply = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
}) })
@ -343,7 +355,7 @@ class FriendRequestOutgoingTestCase(UserTestMixin, ToolshedTestCase):
def test_accept_request(self): def test_accept_request(self):
befriender = self.f['ext_user1'] befriender = self.f['ext_user1']
befriendee = self.f['local_user2'] befriendee = self.f['local_user2']
reply = client.post('/api/v1/friendrequests/', befriender, { reply = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriender_key': befriender.public_key(), 'befriender_key': befriender.public_key(),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
@ -367,12 +379,12 @@ class FriendRequestCombinedTestCase(UserTestMixin, ToolshedTestCase):
def test_friend_request_combined(self): def test_friend_request_combined(self):
befriender = self.f['local_user1'] befriender = self.f['local_user1']
befriendee = self.f['local_user2'] befriendee = self.f['local_user2']
reply1 = client.post('/api/v1/friendrequests/', befriender, { reply1 = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
}) })
secret = reply1.json()['secret'] secret = reply1.json()['secret']
reply2 = client.post('/api/v1/friendrequests/', befriender, { reply2 = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriender_key': befriender.public_key(), 'befriender_key': befriender.public_key(),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
@ -389,7 +401,7 @@ class FriendRequestCombinedTestCase(UserTestMixin, ToolshedTestCase):
befriender = self.f['local_user1'] befriender = self.f['local_user1']
befriendee = self.f['local_user2'] befriendee = self.f['local_user2']
befriender.friends.add(befriendee.public_identity) befriender.friends.add(befriendee.public_identity)
reply1 = client.post('/api/v1/friendrequests/', befriender, { reply1 = client.post('/api/friendrequests/', befriender, {
'befriender': str(befriender), 'befriender': str(befriender),
'befriendee': str(befriendee), 'befriendee': str(befriendee),
}) })

304
backend/toolshed/tests/test_group.py
View file

@ -1,304 +0,0 @@
from django.test import Client
from authentication.tests import SignatureAuthClient, UserTestMixin, GroupTestMixin, ToolshedTestCase, \
DummyExternalUser
from authentication.models import Group, GroupInvite, GroupInviteIncoming, GroupMembership, KnownIdentity
client = SignatureAuthClient()
class GroupModelTestCase(UserTestMixin, GroupTestMixin, ToolshedTestCase):
def setUp(self):
super().setUp()
self.prepare_users()
self.prepare_groups()
def test_group_str(self):
self.assertEqual(str(self.f['group1']), '#group1@' + self.f['example_com'].name)
def test_is_member(self):
self.assertTrue(self.f['group1'].is_member(self.f['local_user1'].public_identity))
self.assertFalse(self.f['group1'].is_member(self.f['local_user2'].public_identity))
class GroupApiTestCase(UserTestMixin, GroupTestMixin, ToolshedTestCase):
def setUp(self):
super().setUp()
self.prepare_users()
def test_create_group(self):
reply = client.post('/api/v1/groups/', self.f['local_user1'], {'name': 'workshop'})
self.assertEqual(reply.status_code, 201)
group = Group.objects.get(name='workshop')
self.assertEqual(group.domain, self.f['local_user1'].domain)
self.assertTrue(group.is_member(self.f['local_user1'].public_identity))
def test_create_group_duplicate_name(self):
client.post('/api/v1/groups/', self.f['local_user1'], {'name': 'workshop'})
reply = client.post('/api/v1/groups/', self.f['local_user1'], {'name': 'workshop'})
self.assertEqual(reply.status_code, 409)
def test_create_group_missing_name(self):
reply = client.post('/api/v1/groups/', self.f['local_user1'], {})
self.assertEqual(reply.status_code, 400)
def test_list_groups(self):
self.prepare_groups()
reply = client.get('/api/v1/groups/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 1)
reply2 = client.get('/api/v1/groups/', self.f['local_user2'])
self.assertEqual(reply2.status_code, 200)
self.assertEqual(len(reply2.json()), 0)
def test_group_detail_member(self):
self.prepare_groups()
group = self.f['group1']
reply = client.get('/api/v1/groups/{}@{}/'.format(group.name, group.domain), self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json()['handle'], str(group))
self.assertEqual(len(reply.json()['members']), 1)
def test_group_detail_non_member(self):
self.prepare_groups()
group = self.f['group1']
reply = client.get('/api/v1/groups/{}@{}/'.format(group.name, group.domain), self.f['local_user2'])
self.assertEqual(reply.status_code, 404)
def test_group_detail_no_such_group(self):
reply = client.get('/api/v1/groups/nonexistent@example.com/', self.f['local_user1'])
self.assertEqual(reply.status_code, 404)
def test_remove_member(self):
self.prepare_groups()
group = self.f['group1']
group.members.add(self.f['local_user2'].public_identity)
reply = client.delete('/api/v1/groups/{}@{}/members/{}/'.format(
group.name, group.domain, self.f['local_user2'].public_identity.id), self.f['local_user1'])
self.assertEqual(reply.status_code, 204)
self.assertEqual(group.members.count(), 1)
def test_remove_last_member_blocked(self):
self.prepare_groups()
group = self.f['group1']
reply = client.delete('/api/v1/groups/{}@{}/members/{}/'.format(
group.name, group.domain, self.f['local_user1'].public_identity.id), self.f['local_user1'])
self.assertEqual(reply.status_code, 400)
self.assertEqual(group.members.count(), 1)
def test_remove_member_non_member_denied(self):
self.prepare_groups()
group = self.f['group1']
group.members.add(self.f['local_user2'].public_identity)
reply = client.delete('/api/v1/groups/{}@{}/members/{}/'.format(
group.name, group.domain, self.f['local_user1'].public_identity.id), self.f['ext_user1'])
self.assertEqual(reply.status_code, 404)
self.assertEqual(group.members.count(), 2)
class GroupInviteApiTestCase(UserTestMixin, GroupTestMixin, ToolshedTestCase):
def setUp(self):
super().setUp()
self.prepare_users()
self.prepare_groups()
def test_invite_local_member_full_flow(self):
group = self.f['group1']
inviter = self.f['local_user1']
invitee = self.f['local_user2']
reply1 = client.post('/api/v1/groups/{}@{}/invites/'.format(group.name, group.domain), inviter,
{'invitee': str(invitee)})
self.assertEqual(reply1.status_code, 201)
secret = reply1.json()['secret']
self.assertEqual(GroupInvite.objects.count(), 1)
reply2 = client.post('/api/v1/groupinvites/', inviter, {
'group': str(group), 'inviter': str(inviter), 'inviter_key': inviter.public_key(),
'invitee': str(invitee), 'secret': secret
})
self.assertEqual(reply2.status_code, 201)
incoming = GroupInviteIncoming.objects.get()
self.assertEqual(incoming.invitee_user, invitee)
reply3 = client.get('/api/v1/groupinvites/', invitee)
self.assertEqual(reply3.status_code, 200)
self.assertEqual(len(reply3.json()), 1)
self.assertEqual(reply3.json()[0]['group'], str(group))
self.assertEqual(reply3.json()[0]['inviter'], str(inviter))
reply4 = client.post('/api/v1/group_invites/accept/', invitee, {
'group': str(group), 'invitee': str(invitee), 'invitee_key': invitee.public_key(), 'secret': secret
})
self.assertEqual(reply4.status_code, 201)
self.assertTrue(group.is_member(invitee.public_identity))
self.assertEqual(GroupInvite.objects.count(), 0)
reply5 = client.post('/api/v1/groupinvites/{}/accept/'.format(incoming.id), invitee)
self.assertEqual(reply5.status_code, 201)
self.assertEqual(reply5.json()['handle'], str(group))
self.assertEqual(GroupInviteIncoming.objects.count(), 0)
membership = GroupMembership.objects.get(user=invitee)
self.assertEqual(membership.group_name, group.name)
self.assertEqual(membership.group_domain, group.domain)
def test_invite_non_member_denied(self):
group = self.f['group1']
reply = client.post('/api/v1/groups/{}@{}/invites/'.format(group.name, group.domain), self.f['local_user2'],
{'invitee': str(self.f['ext_user1'])})
self.assertEqual(reply.status_code, 404)
self.assertEqual(GroupInvite.objects.count(), 0)
def test_invite_already_member(self):
group = self.f['group1']
group.members.add(self.f['local_user2'].public_identity)
reply = client.post('/api/v1/groups/{}@{}/invites/'.format(group.name, group.domain), self.f['local_user1'],
{'invitee': str(self.f['local_user2'])})
self.assertEqual(reply.status_code, 208)
def test_decline_invite(self):
invite = GroupInviteIncoming.objects.create(
group_name=self.f['group1'].name, group_domain=self.f['group1'].domain,
inviter_username=self.f['local_user1'].username, inviter_domain=self.f['local_user1'].domain,
invitee_user=self.f['local_user2'], secret='some-secret')
reply = client.delete('/api/v1/groupinvites/{}/'.format(invite.id), self.f['local_user2'])
self.assertEqual(reply.status_code, 204)
self.assertEqual(GroupInviteIncoming.objects.count(), 0)
def test_accept_wrong_secret(self):
group = self.f['group1']
inviter = self.f['local_user1']
invitee = self.f['local_user2']
client.post('/api/v1/groups/{}@{}/invites/'.format(group.name, group.domain), inviter, {'invitee': str(invitee)})
reply = client.post('/api/v1/group_invites/accept/', invitee, {
'group': str(group), 'invitee': str(invitee), 'invitee_key': invitee.public_key(),
'secret': 'not-the-right-secret'
})
self.assertEqual(reply.status_code, 404)
self.assertFalse(group.is_member(invitee.public_identity))
def test_accept_remote_member(self):
group = self.f['group1']
inviter = self.f['local_user1']
invitee = DummyExternalUser('newmember', 'remote.example', known=False)
reply1 = client.post('/api/v1/groups/{}@{}/invites/'.format(group.name, group.domain), inviter,
{'invitee': str(invitee)})
secret = reply1.json()['secret']
reply = client.post('/api/v1/group_invites/accept/', invitee, {
'group': str(group), 'invitee': str(invitee), 'invitee_key': invitee.public_key(), 'secret': secret
})
self.assertEqual(reply.status_code, 201)
identity = KnownIdentity.objects.get(username='newmember', domain='remote.example')
self.assertTrue(group.is_member(identity))
# This lands on the group's own home backend, not the invitee's -- the invitee here isn't
# even a local ToolshedUser on this backend, so there's nothing to point at locally.
self.assertEqual(GroupMembership.objects.count(), 0)
def test_accept_bad_signature(self):
group = self.f['group1']
inviter = self.f['local_user1']
invitee = self.f['local_user2']
reply1 = client.post('/api/v1/groups/{}@{}/invites/'.format(group.name, group.domain), inviter,
{'invitee': str(invitee)})
secret = reply1.json()['secret']
bad_signature_client = SignatureAuthClient(bad_signature=True)
reply = bad_signature_client.post('/api/v1/group_invites/accept/', invitee, {
'group': str(group), 'invitee': str(invitee), 'invitee_key': invitee.public_key(), 'secret': secret
})
self.assertEqual(reply.status_code, 401)
self.assertFalse(group.is_member(invitee.public_identity))
class GroupMembershipApiTestCase(UserTestMixin, GroupTestMixin, ToolshedTestCase):
"""Covers the pointer index kept on a member's own home backend (GroupMembership), which is
what lets that backend remember foreign group membership, analogous to how a friendship ends
up recorded on both sides rather than only on the group's authoritative backend."""
def setUp(self):
super().setUp()
self.prepare_users()
self.prepare_groups()
def test_record_membership_for_remote_group(self):
# local_user2's own home backend has no Group row at all for this group -- it's hosted
# entirely on another domain -- yet it should still remember the membership once the
# invitee's client confirms the accept against the group's home backend succeeded.
invitee = self.f['local_user2']
incoming = GroupInviteIncoming.objects.create(
group_name='remoteworkshop', group_domain='other.example',
inviter_username='someone', inviter_domain='other.example',
invitee_user=invitee, secret='some-secret')
reply = client.post('/api/v1/groupinvites/{}/accept/'.format(incoming.id), invitee)
self.assertEqual(reply.status_code, 201)
self.assertEqual(reply.json()['handle'], '#remoteworkshop@other.example')
self.assertEqual(GroupInviteIncoming.objects.count(), 0)
membership = GroupMembership.objects.get(user=invitee)
self.assertEqual(membership.group_name, 'remoteworkshop')
self.assertEqual(membership.group_domain, 'other.example')
def test_record_membership_wrong_user_denied(self):
incoming = GroupInviteIncoming.objects.create(
group_name='remoteworkshop', group_domain='other.example',
inviter_username='someone', inviter_domain='other.example',
invitee_user=self.f['local_user2'], secret='some-secret')
reply = client.post('/api/v1/groupinvites/{}/accept/'.format(incoming.id), self.f['local_user1'])
self.assertEqual(reply.status_code, 404)
self.assertEqual(GroupMembership.objects.count(), 0)
self.assertEqual(GroupInviteIncoming.objects.count(), 1)
def test_decline_does_not_record_membership(self):
incoming = GroupInviteIncoming.objects.create(
group_name='remoteworkshop', group_domain='other.example',
inviter_username='someone', inviter_domain='other.example',
invitee_user=self.f['local_user2'], secret='some-secret')
reply = client.delete('/api/v1/groupinvites/{}/'.format(incoming.id), self.f['local_user2'])
self.assertEqual(reply.status_code, 204)
self.assertEqual(GroupMembership.objects.count(), 0)
def test_record_membership_already_a_member_is_idempotent(self):
# Re-invited (or re-accepting) into a group we already have a pointer for shouldn't blow
# up on the unique_together constraint, and shouldn't duplicate the pointer either.
invitee = self.f['local_user2']
GroupMembership.objects.create(
user=invitee, group_name='remoteworkshop', group_domain='other.example')
incoming = GroupInviteIncoming.objects.create(
group_name='remoteworkshop', group_domain='other.example',
inviter_username='someone', inviter_domain='other.example',
invitee_user=invitee, secret='some-secret')
reply = client.post('/api/v1/groupinvites/{}/accept/'.format(incoming.id), invitee)
self.assertEqual(reply.status_code, 201)
self.assertEqual(GroupMembership.objects.filter(user=invitee).count(), 1)
self.assertEqual(GroupInviteIncoming.objects.count(), 0)
def test_list_memberships(self):
GroupMembership.objects.create(
user=self.f['local_user1'], group_name='remoteworkshop', group_domain='other.example')
reply = client.get('/api/v1/groupmemberships/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 1)
self.assertEqual(reply.json()[0]['handle'], '#remoteworkshop@other.example')
reply2 = client.get('/api/v1/groupmemberships/', self.f['local_user2'])
self.assertEqual(reply2.status_code, 200)
self.assertEqual(len(reply2.json()), 0)
def test_list_memberships_unauthorized(self):
GroupMembership.objects.create(
user=self.f['local_user1'], group_name='remoteworkshop', group_domain='other.example')
reply = client.get('/api/v1/groupmemberships/', self.f['ext_user1'])
# authenticate() returns bare None (not raise) for a caller with no local ToolshedUser, so
# DRF falls through to IsAuthenticated denying an anonymous request -- 403, not 401 (same
# as any other SignatureAuthenticationLocal-only endpoint, see e.g. dropFriend).
self.assertEqual(reply.status_code, 403)

163
backend/toolshed/tests/test_idmap.py
View file

@ -1,163 +0,0 @@
from django.test import Client
from authentication.tests import SignatureAuthClient, UserTestMixin, GroupTestMixin, ToolshedTestCase
from toolshed.models import InventoryItem, StorageLocation
from toolshed.tests import InventoryTestMixin, LocationTestMixin
client = SignatureAuthClient()
class IdMapTestCase(UserTestMixin, GroupTestMixin, ToolshedTestCase):
def setUp(self):
super().setUp()
self.prepare_users()
def test_idmap_includes_self(self):
reply = client.get('/api/v1/idmap/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
identities = reply.json()['identities']
self.assertIn(str(self.f['local_user1']), [i['handle'] for i in identities])
def test_idmap_includes_friends(self):
self.f['local_user1'].friends.add(self.f['local_user2'].public_identity)
reply = client.get('/api/v1/idmap/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
identities = reply.json()['identities']
self.assertIn(str(self.f['local_user2']), [i['handle'] for i in identities])
def test_idmap_excludes_non_friends(self):
reply = client.get('/api/v1/idmap/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
identities = reply.json()['identities']
self.assertNotIn(str(self.f['local_user2']), [i['handle'] for i in identities])
def test_idmap_includes_member_groups(self):
self.prepare_groups()
reply = client.get('/api/v1/idmap/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
groups = reply.json()['groups']
self.assertIn(str(self.f['group1']), [g['handle'] for g in groups])
def test_idmap_excludes_non_member_groups(self):
self.prepare_groups()
reply = client.get('/api/v1/idmap/', self.f['local_user2'])
self.assertEqual(reply.status_code, 200)
groups = reply.json()['groups']
self.assertNotIn(str(self.f['group1']), [g['handle'] for g in groups])
def test_idmap_unauthenticated(self):
reply = Client().get('/api/v1/idmap/')
self.assertEqual(reply.status_code, 403)
class ResolveShortIdApiTestCase(UserTestMixin, InventoryTestMixin, GroupTestMixin, LocationTestMixin,
ToolshedTestCase):
def setUp(self):
super().setUp()
self.prepare_users()
self.prepare_categories()
self.prepare_tags()
self.prepare_properties()
self.prepare_inventory()
self.prepare_groups()
self.prepare_locations()
self.owner_id = self.f['local_user1'].public_identity.pk
def test_resolve_item_as_owner(self):
reply = client.get(f'/api/v1/resolve_short_id/item/{self.owner_id}/{self.f["item1"].id}/',
self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json(), {'handle': 'testuser1@example.com', 'id': self.f['item1'].id})
def test_resolve_item_as_friend(self):
reply = client.get(f'/api/v1/resolve_short_id/item/{self.owner_id}/{self.f["item1"].id}/',
self.f['local_user2'])
self.assertEqual(reply.status_code, 200)
def test_resolve_item_not_friend(self):
reply = client.get(f'/api/v1/resolve_short_id/item/{self.owner_id}/{self.f["item1"].id}/',
self.f['ext_user1'])
self.assertEqual(reply.status_code, 403)
def test_resolve_item_private_not_owner(self):
private_item = InventoryItem.create_for_owner(
owner=self.f['local_user1'], owned_quantity=1, name='secret', availability_policy='private')
reply = client.get(f'/api/v1/resolve_short_id/item/{self.owner_id}/{private_item.id}/',
self.f['local_user2'])
self.assertEqual(reply.status_code, 403)
reply = client.get(f'/api/v1/resolve_short_id/item/{self.owner_id}/{private_item.id}/',
self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
def test_resolve_item_visibility_private_not_owner(self):
private_item = InventoryItem.create_for_owner(
owner=self.f['local_user1'], owned_quantity=1, name='secret', availability_policy='friends',
visibility_policy='private')
reply = client.get(f'/api/v1/resolve_short_id/item/{self.owner_id}/{private_item.id}/',
self.f['local_user2'])
self.assertEqual(reply.status_code, 403)
reply = client.get(f'/api/v1/resolve_short_id/item/{self.owner_id}/{private_item.id}/',
self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
def test_resolve_item_unknown_owner(self):
reply = client.get('/api/v1/resolve_short_id/item/999999/1/', self.f['local_user2'])
self.assertEqual(reply.status_code, 404)
def test_resolve_item_unknown_local_id(self):
reply = client.get(f'/api/v1/resolve_short_id/item/{self.owner_id}/999999/', self.f['local_user2'])
self.assertEqual(reply.status_code, 404)
def test_resolve_unsupported_kind(self):
# workflow/category/file/group aren't wired up yet - see
# docs/handles-and-shortids.md#domain-qualified-short-id's note on the remaining gap.
reply = client.get(f'/api/v1/resolve_short_id/workflow/{self.owner_id}/1/', self.f['local_user2'])
self.assertEqual(reply.status_code, 400)
def test_resolve_storage_location_as_owner(self):
reply = client.get(f'/api/v1/resolve_short_id/storage_location/{self.owner_id}/{self.f["loc1"].id}/',
self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json(), {'handle': 'testuser1@example.com', 'id': self.f['loc1'].id})
def test_resolve_storage_location_visibility_private_not_owner(self):
private_location = StorageLocation.create_for_owner(
owner=self.f['local_user1'], name='secret-loc', visibility_policy='private')
reply = client.get(f'/api/v1/resolve_short_id/storage_location/{self.owner_id}/{private_location.id}/',
self.f['local_user2'])
self.assertEqual(reply.status_code, 403)
reply = client.get(f'/api/v1/resolve_short_id/storage_location/{self.owner_id}/{private_location.id}/',
self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
def test_resolve_storage_location_unknown_local_id(self):
reply = client.get(f'/api/v1/resolve_short_id/storage_location/{self.owner_id}/999999/',
self.f['local_user1'])
self.assertEqual(reply.status_code, 404)
def test_resolve_group_item_as_member(self):
item = InventoryItem.create_for_owner(
owner_group=self.f['group1'], owned_quantity=1, name='drill', availability_policy='private')
reply = client.get(f'/api/v1/resolve_short_id/group_item/{self.f["group1"].pk}/{item.id}/',
self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json(), {'handle': str(self.f['group1']), 'id': item.id})
def test_resolve_group_item_non_member_denied(self):
item = InventoryItem.create_for_owner(
owner_group=self.f['group1'], owned_quantity=1, name='drill', availability_policy='private')
reply = client.get(f'/api/v1/resolve_short_id/group_item/{self.f["group1"].pk}/{item.id}/',
self.f['local_user2'])
self.assertEqual(reply.status_code, 403)
def test_resolve_group_storage_location_as_member(self):
location = StorageLocation.create_for_owner(owner_group=self.f['group1'], name='shelf')
reply = client.get(
f'/api/v1/resolve_short_id/group_storage_location/{self.f["group1"].pk}/{location.id}/',
self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json(), {'handle': str(self.f['group1']), 'id': location.id})
def test_resolve_unknown_group(self):
reply = client.get('/api/v1/resolve_short_id/group_item/999999/1/', self.f['local_user1'])
self.assertEqual(reply.status_code, 404)

363
backend/toolshed/tests/test_inventory.py
View file

@ -1,8 +1,7 @@
from authentication.models import Group from authentication.tests import SignatureAuthClient, UserTestMixin, ToolshedTestCase
from authentication.tests import SignatureAuthClient, UserTestMixin, GroupTestMixin, ToolshedTestCase
from files.tests import FilesTestMixin from files.tests import FilesTestMixin
from toolshed.models import InventoryItem, Category, StorageLocation from toolshed.models import InventoryItem, Category
from toolshed.tests import InventoryTestMixin, CategoryTestMixin, TagTestMixin, PropertyTestMixin, LocationTestMixin from toolshed.tests import InventoryTestMixin
client = SignatureAuthClient() client = SignatureAuthClient()
@ -16,10 +15,9 @@ class InventoryApiTestCase(UserTestMixin, InventoryTestMixin, ToolshedTestCase):
self.prepare_tags() self.prepare_tags()
self.prepare_properties() self.prepare_properties()
self.prepare_inventory() self.prepare_inventory()
self.own_handle = str(self.f['local_user1'])
def test_get_inventory(self): def test_get_inventory(self):
reply = client.get('/api/v1/inventory_items/{}/'.format(self.own_handle), self.f['local_user1']) reply = client.get('/api/inventory_items/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 2) self.assertEqual(len(reply.json()), 2)
self.assertEqual(reply.json()[0]['name'], 'test1') self.assertEqual(reply.json()[0]['name'], 'test1')
@ -39,7 +37,7 @@ class InventoryApiTestCase(UserTestMixin, InventoryTestMixin, ToolshedTestCase):
self.assertEqual(reply.json()[1]['availability_policy'], 'friends') self.assertEqual(reply.json()[1]['availability_policy'], 'friends')
def test_post_new_item(self): def test_post_new_item(self):
reply = client.post('/api/v1/inventory_items/{}/'.format(self.own_handle), self.f['local_user1'], { reply = client.post('/api/inventory_items/', self.f['local_user1'], {
'availability_policy': 'rent', 'availability_policy': 'rent',
'category': 'cat2', 'category': 'cat2',
'name': 'test3', 'name': 'test3',
@ -62,7 +60,7 @@ class InventoryApiTestCase(UserTestMixin, InventoryTestMixin, ToolshedTestCase):
self.assertEqual([p.value for p in item.itemproperty_set.all()], ['value3', 'value4']) self.assertEqual([p.value for p in item.itemproperty_set.all()], ['value3', 'value4'])
def test_post_new_item2(self): def test_post_new_item2(self):
reply = client.post('/api/v1/inventory_items/{}/'.format(self.own_handle), self.f['local_user1'], { reply = client.post('/api/inventory_items/', self.f['local_user1'], {
'availability_policy': 'share', 'availability_policy': 'share',
'name': 'test3', 'name': 'test3',
'description': 'test', 'description': 'test',
@ -81,7 +79,7 @@ class InventoryApiTestCase(UserTestMixin, InventoryTestMixin, ToolshedTestCase):
self.assertEqual([p for p in item.properties.all()], []) self.assertEqual([p for p in item.properties.all()], [])
def test_post_new_item_empty(self): def test_post_new_item_empty(self):
reply = client.post('/api/v1/inventory_items/{}/'.format(self.own_handle), self.f['local_user1'], { reply = client.post('/api/inventory_items/', self.f['local_user1'], {
'availability_policy': 'rent', 'availability_policy': 'rent',
'owned_quantity': 1, 'owned_quantity': 1,
'image': '', 'image': '',
@ -90,7 +88,7 @@ class InventoryApiTestCase(UserTestMixin, InventoryTestMixin, ToolshedTestCase):
self.assertEqual(InventoryItem.objects.count(), 2) self.assertEqual(InventoryItem.objects.count(), 2)
def test_post_new_item3(self): def test_post_new_item3(self):
reply = client.post('/api/v1/inventory_items/{}/'.format(self.own_handle), self.f['local_user1'], { reply = client.post('/api/inventory_items/', self.f['local_user1'], {
'availability_policy': 'private', 'availability_policy': 'private',
'name': 'test3', 'name': 'test3',
'description': 'test', 'description': 'test',
@ -110,7 +108,7 @@ class InventoryApiTestCase(UserTestMixin, InventoryTestMixin, ToolshedTestCase):
self.assertEqual([p for p in item.properties.all()], []) self.assertEqual([p for p in item.properties.all()], [])
def test_put_item(self): def test_put_item(self):
reply = client.put('/api/v1/inventory_items/{}/1/'.format(self.own_handle), self.f['local_user1'], { reply = client.put('/api/inventory_items/1/', self.f['local_user1'], {
'availability_policy': 'sell', 'availability_policy': 'sell',
'name': 'test4', 'name': 'test4',
'description': 'new description', 'description': 'new description',
@ -133,7 +131,7 @@ class InventoryApiTestCase(UserTestMixin, InventoryTestMixin, ToolshedTestCase):
self.assertEqual([p.value for p in item.itemproperty_set.all()], ['value5', 'value6', 'value7']) self.assertEqual([p.value for p in item.itemproperty_set.all()], ['value5', 'value6', 'value7'])
def test_patch_item(self): def test_patch_item(self):
reply = client.patch('/api/v1/inventory_items/{}/1/'.format(self.own_handle), self.f['local_user1'], { reply = client.patch('/api/inventory_items/1/', self.f['local_user1'], {
'description': 'new description2', 'description': 'new description2',
'category': 'cat1', 'category': 'cat1',
'owned_quantity': 100, 'owned_quantity': 100,
@ -153,7 +151,7 @@ class InventoryApiTestCase(UserTestMixin, InventoryTestMixin, ToolshedTestCase):
self.assertEqual([p.value for p in item.itemproperty_set.all()], ['value8']) self.assertEqual([p.value for p in item.itemproperty_set.all()], ['value8'])
def test_patch_item2(self): def test_patch_item2(self):
reply = client.patch('/api/v1/inventory_items/{}/1/'.format(self.own_handle), self.f['local_user1'], { reply = client.patch('/api/inventory_items/1/', self.f['local_user1'], {
'description': 'new description2', 'description': 'new description2',
'category': None, 'category': None,
'owned_quantity': 100, 'owned_quantity': 100,
@ -172,14 +170,14 @@ class InventoryApiTestCase(UserTestMixin, InventoryTestMixin, ToolshedTestCase):
self.assertEqual([p for p in item.properties.all()], []) self.assertEqual([p for p in item.properties.all()], [])
def test_delete_item(self): def test_delete_item(self):
reply = client.delete('/api/v1/inventory_items/{}/1/'.format(self.own_handle), self.f['local_user1']) reply = client.delete('/api/inventory_items/1/', self.f['local_user1'])
self.assertEqual(reply.status_code, 204) self.assertEqual(reply.status_code, 204)
self.assertEqual(InventoryItem.objects.count(), 1) self.assertEqual(InventoryItem.objects.count(), 1)
self.assertEqual(InventoryItem.objects.get(id=2).name, 'test2') self.assertEqual(InventoryItem.objects.get(id=2).name, 'test2')
self.assertEqual(InventoryItem.objects.filter(name='test1').count(), 0) self.assertEqual(InventoryItem.objects.filter(name='test1').count(), 0)
def test_search_items(self): def test_search_items(self):
reply = client.get('/api/v1/search/?query=test', self.f['local_user1']) reply = client.get('/api/search/?query=test', self.f['local_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 2) self.assertEqual(len(reply.json()), 2)
self.assertEqual(reply.json()[0]['name'], 'test1') self.assertEqual(reply.json()[0]['name'], 'test1')
@ -199,86 +197,22 @@ class InventoryApiTestCase(UserTestMixin, InventoryTestMixin, ToolshedTestCase):
self.assertEqual(reply.json()[1]['availability_policy'], 'friends') self.assertEqual(reply.json()[1]['availability_policy'], 'friends')
def test_search_items2(self): def test_search_items2(self):
reply = client.get('/api/v1/search/?query=test', self.f['local_user2']) reply = client.get('/api/search/?query=test', self.f['local_user2'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 2) self.assertEqual(len(reply.json()), 2)
self.assertEqual(reply.json()[0]['name'], 'test1') self.assertEqual(reply.json()[0]['name'], 'test1')
self.assertEqual(reply.json()[1]['name'], 'test2') self.assertEqual(reply.json()[1]['name'], 'test2')
def test_search_items_excludes_visibility_private(self):
InventoryItem.create_for_owner(
owner=self.f['local_user1'], owned_quantity=1, name='test-secret', availability_policy='friends',
visibility_policy='private')
reply = client.get('/api/v1/search/?query=test', self.f['local_user2'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 2)
self.assertNotIn('test-secret', [item['name'] for item in reply.json()])
def test_search_items_fail(self): def test_search_items_fail(self):
reply = client.get('/api/v1/search/', self.f['local_user1']) reply = client.get('/api/search/', self.f['local_user1'])
self.assertEqual(reply.status_code, 400) self.assertEqual(reply.status_code, 400)
self.assertEqual(reply.json()['error'], 'No query provided.') self.assertEqual(reply.json()['error'], 'No query provided.')
def test_search_items_fail2(self): def test_search_items_fail2(self):
reply = client.get('/api/v1/search/?query=test', self.f['ext_user1']) reply = client.get('/api/search/?query=test', self.f['ext_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 0) self.assertEqual(len(reply.json()), 0)
def test_get_shared_item_as_friend(self):
reply = client.get('/api/v1/inventory_items/testuser1@example.com/' + str(self.f['item1'].id) + '/',
self.f['local_user2'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json()['name'], 'test1')
self.assertEqual(reply.json()['owner'], 'testuser1@example.com')
def test_get_shared_item_as_owner(self):
reply = client.get('/api/v1/inventory_items/testuser1@example.com/' + str(self.f['item1'].id) + '/',
self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(reply.json()['name'], 'test1')
def test_get_shared_item_not_friend(self):
# Non-visible resolves the same way a non-member's group access does now (see
# GroupOwnedInventoryApiTestCase.test_non_member_cannot_see_or_edit) - 404, not a
# separate 403 special case for "not a friend".
reply = client.get('/api/v1/inventory_items/testuser1@example.com/' + str(self.f['item1'].id) + '/',
self.f['ext_user1'])
self.assertEqual(reply.status_code, 404)
def test_get_shared_item_private(self):
private_item = InventoryItem.create_for_owner(
owner=self.f['local_user1'], owned_quantity=1, name='secret', availability_policy='private')
reply = client.get('/api/v1/inventory_items/testuser1@example.com/' + str(private_item.id) + '/',
self.f['local_user2'])
self.assertEqual(reply.status_code, 404)
reply = client.get('/api/v1/inventory_items/testuser1@example.com/' + str(private_item.id) + '/',
self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
def test_get_shared_item_visibility_private(self):
private_item = InventoryItem.create_for_owner(
owner=self.f['local_user1'], owned_quantity=1, name='secret', availability_policy='friends',
visibility_policy='private')
reply = client.get('/api/v1/inventory_items/testuser1@example.com/' + str(private_item.id) + '/',
self.f['local_user2'])
self.assertEqual(reply.status_code, 404)
reply = client.get('/api/v1/inventory_items/testuser1@example.com/' + str(private_item.id) + '/',
self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
def test_get_shared_item_unknown_handle(self):
reply = client.get('/api/v1/inventory_items/nobody@example.com/' + str(self.f['item1'].id) + '/',
self.f['local_user2'])
self.assertEqual(reply.status_code, 404)
def test_get_shared_item_unknown_id(self):
reply = client.get('/api/v1/inventory_items/testuser1@example.com/99999/', self.f['local_user2'])
self.assertEqual(reply.status_code, 404)
def test_get_shared_item_bad_handle(self):
reply = client.get('/api/v1/inventory_items/testuser1/' + str(self.f['item1'].id) + '/', self.f['local_user2'])
self.assertEqual(reply.status_code, 400)
class TestInventoryItemWithFileApiTestCase(UserTestMixin, FilesTestMixin, InventoryTestMixin, ToolshedTestCase): class TestInventoryItemWithFileApiTestCase(UserTestMixin, FilesTestMixin, InventoryTestMixin, ToolshedTestCase):
def setUp(self): def setUp(self):
@ -289,10 +223,9 @@ class TestInventoryItemWithFileApiTestCase(UserTestMixin, FilesTestMixin, Invent
self.prepare_properties() self.prepare_properties()
self.prepare_files() self.prepare_files()
self.prepare_inventory() self.prepare_inventory()
self.own_handle = str(self.f['local_user1'])
def test_post_item_with_file_id(self): def test_post_item_with_file_id(self):
reply = client.post('/api/v1/inventory_items/{}/'.format(self.own_handle), self.f['local_user1'], { reply = client.post('/api/inventory_items/', self.f['local_user1'], {
'name': 'test4', 'name': 'test4',
'description': 'test', 'description': 'test',
'category': 'cat1', 'category': 'cat1',
@ -315,7 +248,7 @@ class TestInventoryItemWithFileApiTestCase(UserTestMixin, FilesTestMixin, Invent
self.assertEqual([f for f in item.files.all()], [self.f['test_file1']]) self.assertEqual([f for f in item.files.all()], [self.f['test_file1']])
def test_post_item_with_encoded_file(self): def test_post_item_with_encoded_file(self):
reply = client.post('/api/v1/inventory_items/{}/'.format(self.own_handle), self.f['local_user1'], { reply = client.post('/api/inventory_items/', self.f['local_user1'], {
'name': 'test4', 'name': 'test4',
'description': 'test', 'description': 'test',
'category': 'cat1', 'category': 'cat1',
@ -338,7 +271,7 @@ class TestInventoryItemWithFileApiTestCase(UserTestMixin, FilesTestMixin, Invent
self.assertEqual([f for f in item.files.all()], [self.f['test_file3']]) self.assertEqual([f for f in item.files.all()], [self.f['test_file3']])
def test_post_item_with_file_id_fail(self): def test_post_item_with_file_id_fail(self):
reply = client.post('/api/v1/inventory_items/{}/'.format(self.own_handle), self.f['local_user1'], { reply = client.post('/api/inventory_items/', self.f['local_user1'], {
'name': 'test4', 'name': 'test4',
'description': 'test', 'description': 'test',
'category': 'cat1', 'category': 'cat1',
@ -350,7 +283,7 @@ class TestInventoryItemWithFileApiTestCase(UserTestMixin, FilesTestMixin, Invent
self.assertEqual(reply.status_code, 400) self.assertEqual(reply.status_code, 400)
def test_post_item_with_encoded_file_fail(self): def test_post_item_with_encoded_file_fail(self):
reply = client.post('/api/v1/inventory_items/{}/'.format(self.own_handle), self.f['local_user1'], { reply = client.post('/api/inventory_items/', self.f['local_user1'], {
'name': 'test4', 'name': 'test4',
'description': 'test', 'description': 'test',
'category': 'cat1', 'category': 'cat1',
@ -359,258 +292,4 @@ class TestInventoryItemWithFileApiTestCase(UserTestMixin, FilesTestMixin, Invent
'properties': [{'name': 'prop1', 'value': 'value1'}, {'name': 'prop2', 'value': 'value2'}], 'properties': [{'name': 'prop1', 'value': 'value1'}, {'name': 'prop2', 'value': 'value2'}],
'files': [{'data': self.f['encoded_content3']}] 'files': [{'data': self.f['encoded_content3']}]
}) })
self.assertEqual(reply.status_code, 400) self.assertEqual(reply.status_code, 400)
class GroupOwnedInventoryApiTestCase(UserTestMixin, GroupTestMixin, CategoryTestMixin, TagTestMixin,
PropertyTestMixin, FilesTestMixin, LocationTestMixin, ToolshedTestCase):
def setUp(self):
super().setUp()
self.prepare_users()
self.prepare_groups()
self.prepare_categories()
self.prepare_tags()
self.prepare_properties()
self.prepare_files()
self.prepare_locations()
self.f['group1'].members.add(self.f['local_user2'].public_identity)
self.group_handle = '+' + str(self.f['group1'])[1:]
def create_group_item(self, name='drill'):
return client.post('/api/v1/inventory_items/{}/'.format(self.group_handle), self.f['local_user1'], {
'name': name, 'owned_quantity': 1, 'availability_policy': 'private',
})
def test_create_group_owned_item(self):
reply = self.create_group_item()
self.assertEqual(reply.status_code, 201)
item = InventoryItem.objects.get(name='drill')
self.assertIsNone(item.owner)
self.assertEqual(item.owner_group, self.f['group1'])
self.assertEqual(reply.json()['owner_group'], str(self.f['group1']))
self.assertIsNone(reply.json()['owner'])
def test_create_group_owned_item_non_member_denied(self):
reply = client.post('/api/v1/inventory_items/{}/'.format(self.group_handle), self.f['ext_user1'], {
'name': 'drill', 'owned_quantity': 1, 'availability_policy': 'private',
})
self.assertEqual(reply.status_code, 403)
self.assertEqual(InventoryItem.objects.count(), 0)
def test_other_member_can_edit(self):
item_id = self.create_group_item().json()['id']
reply = client.patch('/api/v1/inventory_items/{}/{}/'.format(self.group_handle, item_id), self.f['local_user2'], {
'name': 'drill-renamed'
})
self.assertEqual(reply.status_code, 200)
self.assertEqual(InventoryItem.objects.get(id=item_id).name, 'drill-renamed')
def test_other_member_can_delete(self):
item_id = self.create_group_item().json()['id']
reply = client.delete('/api/v1/inventory_items/{}/{}/'.format(self.group_handle, item_id), self.f['local_user2'])
self.assertEqual(reply.status_code, 204)
self.assertEqual(InventoryItem.objects.filter(id=item_id).count(), 0)
def test_delete_group_item_when_id_collides_with_own_personal_item(self):
own_handle = str(self.f['local_user2'])
personal_reply = client.post('/api/v1/inventory_items/{}/'.format(own_handle), self.f['local_user2'], {
'name': 'personal-drill', 'owned_quantity': 1, 'availability_policy': 'private',
})
self.assertEqual(personal_reply.json()['id'], 1)
item_id = self.create_group_item().json()['id']
self.assertEqual(item_id, 1)
reply = client.delete('/api/v1/inventory_items/{}/{}/'.format(self.group_handle, item_id), self.f['local_user2'])
self.assertEqual(reply.status_code, 204)
self.assertEqual(InventoryItem.objects.filter(name='drill').count(), 0)
self.assertEqual(InventoryItem.objects.filter(name='personal-drill').count(), 1)
def test_remote_member_without_local_account_can_edit(self):
# A remote member (KnownIdentity, no ToolshedUser row) must still act on group-owned
# items - not unauthorized just because .user.exists() is False.
self.f['group1'].members.add(self.f['ext_user1'].public_identity)
item_id = self.create_group_item().json()['id']
reply = client.get('/api/v1/inventory_items/{}/{}/'.format(self.group_handle, item_id), self.f['ext_user1'])
self.assertEqual(reply.status_code, 200)
reply = client.patch('/api/v1/inventory_items/{}/{}/'.format(self.group_handle, item_id), self.f['ext_user1'], {
'name': 'drill-renamed-by-remote-member'
})
self.assertEqual(reply.status_code, 200)
self.assertEqual(InventoryItem.objects.get(id=item_id).name, 'drill-renamed-by-remote-member')
def test_non_member_cannot_see_or_edit(self):
item_id = self.create_group_item().json()['id']
reply = client.get('/api/v1/inventory_items/{}/{}/'.format(self.group_handle, item_id), self.f['ext_user1'])
self.assertEqual(reply.status_code, 404)
def test_group_items_excluded_from_personal_list(self):
self.create_group_item()
reply = client.get('/api/v1/inventory_items/{}/'.format(str(self.f['local_user1'])), self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 0)
def test_group_items_listed_by_owner_handle(self):
self.create_group_item()
reply = client.get('/api/v1/inventory_items/{}/'.format(self.group_handle), self.f['local_user2'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 1)
self.assertEqual(reply.json()[0]['name'], 'drill')
def test_group_items_not_listed_for_non_member(self):
self.create_group_item()
reply = client.get('/api/v1/inventory_items/{}/'.format(self.group_handle), self.f['ext_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 0)
def test_create_group_owned_item_with_full_fields(self):
# Parity with InventoryApiTestCase.test_post_new_item: tags/properties/category attach
# to a group-owned item the same way as a personal one.
reply = client.post('/api/v1/inventory_items/{}/'.format(self.group_handle), self.f['local_user1'], {
'availability_policy': 'rent',
'category': 'cat2',
'name': 'drill',
'description': 'test',
'owned_quantity': 3,
'tags': ['tag1', 'tag2'],
'properties': [{'name': 'prop1', 'value': 'value1'}, {'name': 'prop2', 'value': 'value2'}],
})
self.assertEqual(reply.status_code, 201)
item = InventoryItem.objects.get(name='drill')
self.assertIsNone(item.owner)
self.assertEqual(item.owner_group, self.f['group1'])
self.assertEqual(item.availability_policy, 'rent')
self.assertEqual(item.category, Category.objects.get(name='cat2'))
self.assertEqual(item.owned_quantity, 3)
self.assertEqual([t for t in item.tags.all()], [self.f['tag1'], self.f['tag2']])
self.assertEqual([p for p in item.properties.all()], [self.f['prop1'], self.f['prop2']])
self.assertEqual([p.value for p in item.itemproperty_set.all()], ['value1', 'value2'])
def test_create_group_owned_item_empty_fails(self):
# Parity with InventoryApiTestCase.test_post_new_item_empty: clean()'s name-or-files validation still applies.
reply = client.post('/api/v1/inventory_items/{}/'.format(self.group_handle), self.f['local_user1'], {
'availability_policy': 'private', 'owned_quantity': 1,
})
self.assertEqual(reply.status_code, 400)
self.assertEqual(InventoryItem.objects.count(), 0)
def test_create_group_owned_item_nonexistent_group(self):
reply = client.post('/api/v1/inventory_items/+nonexistent@example.com/', self.f['local_user1'], {
'name': 'drill', 'owned_quantity': 1, 'availability_policy': 'private',
})
self.assertEqual(reply.status_code, 404)
self.assertEqual(InventoryItem.objects.count(), 0)
def test_group_items_listed_for_nonexistent_group(self):
reply = client.get('/api/v1/inventory_items/+nonexistent@example.com/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 0)
def test_put_group_item(self):
# Parity with InventoryApiTestCase.test_put_item, but as a PUT by a different member than
# the creator, to exercise the _is_authorized branch in perform_update for PUT too.
item_id = self.create_group_item().json()['id']
reply = client.put('/api/v1/inventory_items/{}/{}/'.format(self.group_handle, item_id), self.f['local_user2'], {
'availability_policy': 'sell',
'name': 'drill-4000',
'description': 'new description',
'owned_quantity': 100,
'tags': ['tag1', 'tag3'],
'properties': [{'name': 'prop1', 'value': 'value5'}],
})
self.assertEqual(reply.status_code, 200)
item = InventoryItem.objects.get(id=item_id)
self.assertEqual(item.owner_group, self.f['group1'])
self.assertEqual(item.availability_policy, 'sell')
self.assertEqual(item.name, 'drill-4000')
self.assertEqual(item.description, 'new description')
self.assertEqual(item.owned_quantity, 100)
self.assertEqual([t for t in item.tags.all()], [self.f['tag1'], self.f['tag3']])
self.assertEqual([p.value for p in item.itemproperty_set.all()], ['value5'])
def test_patch_group_item_clears_fields(self):
# Parity with InventoryApiTestCase.test_patch_item2 - clearing category/tags/properties.
reply = client.post('/api/v1/inventory_items/{}/'.format(self.group_handle), self.f['local_user1'], {
'name': 'drill', 'owned_quantity': 1, 'availability_policy': 'private',
'category': 'cat1', 'tags': ['tag1'],
})
item_id = reply.json()['id']
reply = client.patch('/api/v1/inventory_items/{}/{}/'.format(self.group_handle, item_id), self.f['local_user2'], {
'category': None, 'tags': [], 'properties': []
})
self.assertEqual(reply.status_code, 200)
item = InventoryItem.objects.get(id=item_id)
self.assertEqual(item.category, None)
self.assertEqual([t for t in item.tags.all()], [])
def test_group_item_storage_location(self):
reply = client.post('/api/v1/inventory_items/{}/'.format(self.group_handle), self.f['local_user1'], {
'name': 'drill', 'owned_quantity': 1, 'availability_policy': 'private',
'storage_location': self.f['loc1'].id,
})
self.assertEqual(reply.status_code, 201)
item = InventoryItem.objects.get(name='drill')
self.assertEqual(item.storage_location, self.f['loc1'])
def test_post_group_item_with_file_id(self):
# Parity with TestInventoryItemWithFileApiTestCase.test_post_item_with_file_id.
reply = client.post('/api/v1/inventory_items/{}/'.format(self.group_handle), self.f['local_user1'], {
'name': 'drill', 'owned_quantity': 1, 'availability_policy': 'private',
'files': [self.f['test_file1'].id],
})
self.assertEqual(reply.status_code, 201)
item = InventoryItem.objects.get(name='drill')
self.assertEqual([f for f in item.files.all()], [self.f['test_file1']])
def test_post_group_item_with_encoded_file(self):
reply = client.post('/api/v1/inventory_items/{}/'.format(self.group_handle), self.f['local_user1'], {
'name': 'drill', 'owned_quantity': 1, 'availability_policy': 'private',
'files': [{'data': self.f['encoded_content3'], 'mime_type': 'text/plain'}],
})
self.assertEqual(reply.status_code, 201)
item = InventoryItem.objects.get(name='drill')
self.assertEqual([f for f in item.files.all()], [self.f['test_file3']])
def test_group_items_excluded_from_search(self):
# Group-owned items are reachable only via the group's own detail page for MVP (see
# docs/design-in-progress/groups-mvp.md) - search must not surface them either.
self.create_group_item(name='searchable-drill')
InventoryItem.create_for_owner(owner=self.f['local_user1'], owned_quantity=1, name='searchable-personal')
reply = client.get('/api/v1/search/?query=searchable', self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
names = [item['name'] for item in reply.json()]
self.assertEqual(names, ['searchable-personal'])
class InventoryItemIdAllocationTestCase(UserTestMixin, ToolshedTestCase):
"""InventoryItem.id is sequential, gapless, and never reused within each owner/owner_group's
own items, allocated independently per scope (see OwnerItemSequence)."""
def setUp(self):
super().setUp()
self.prepare_users()
def test_ids_are_sequential_and_independent_per_owner(self):
user1_items = [InventoryItem.create_for_owner(owner=self.f['local_user1'], name=f'u1-{i}')
for i in range(3)]
user2_items = [InventoryItem.create_for_owner(owner=self.f['local_user2'], name=f'u2-{i}')
for i in range(2)]
self.assertEqual([item.id for item in user1_items], [1, 2, 3])
self.assertEqual([item.id for item in user2_items], [1, 2])
def test_deleted_item_id_is_never_reused(self):
item1 = InventoryItem.create_for_owner(owner=self.f['local_user1'], name='first')
item2 = InventoryItem.create_for_owner(owner=self.f['local_user1'], name='second')
self.assertEqual((item1.id, item2.id), (1, 2))
item2.delete() # soft delete - item2's row (and its id) stays in the table
item3 = InventoryItem.create_for_owner(owner=self.f['local_user1'], name='third')
self.assertEqual(item3.id, 3)
self.assertFalse(InventoryItem.objects.filter(owner=self.f['local_user1'], id=2).exists())
self.assertTrue(InventoryItem.global_objects.filter(owner=self.f['local_user1'], id=2).exists())
def test_group_scope_has_independent_sequence(self):
group = Group.objects.create(name='alloc-test-group', domain=self.f['example_com'].name)
personal_item = InventoryItem.create_for_owner(owner=self.f['local_user1'], name='personal')
group_item = InventoryItem.create_for_owner(owner_group=group, name='group-owned')
self.assertEqual(personal_item.id, 1)
self.assertEqual(group_item.id, 1)

227
backend/toolshed/tests/test_locations.py
View file

@ -1,6 +1,6 @@
from authentication.tests import SignatureAuthClient, UserTestMixin, GroupTestMixin, ToolshedTestCase from authentication.tests import SignatureAuthClient, UserTestMixin, ToolshedTestCase
from files.tests import FilesTestMixin from files.tests import FilesTestMixin
from toolshed.models import InventoryItem, Category, StorageLocation from toolshed.models import InventoryItem, Category
from toolshed.tests import InventoryTestMixin, LocationTestMixin from toolshed.tests import InventoryTestMixin, LocationTestMixin
client = SignatureAuthClient() client = SignatureAuthClient()
@ -16,7 +16,6 @@ class LocationApiTestCase(UserTestMixin, InventoryTestMixin, LocationTestMixin,
self.prepare_properties() self.prepare_properties()
self.prepare_locations() self.prepare_locations()
self.prepare_inventory() self.prepare_inventory()
self.own_handle = str(self.f['local_user1'])
def test_locations(self): def test_locations(self):
self.assertEqual("loc1", str(self.f['loc1'])) self.assertEqual("loc1", str(self.f['loc1']))
@ -31,7 +30,7 @@ class LocationApiTestCase(UserTestMixin, InventoryTestMixin, LocationTestMixin,
self.assertEqual(self.f['loc1'], self.f['loc4'].parent) self.assertEqual(self.f['loc1'], self.f['loc4'].parent)
def test_get_inventory(self): def test_get_inventory(self):
reply = client.get('/api/v1/inventory_items/{}/'.format(self.own_handle), self.f['local_user1']) reply = client.get('/api/inventory_items/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 2) self.assertEqual(len(reply.json()), 2)
self.assertEqual(reply.json()[0]['name'], 'test1') self.assertEqual(reply.json()[0]['name'], 'test1')
@ -51,7 +50,7 @@ class LocationApiTestCase(UserTestMixin, InventoryTestMixin, LocationTestMixin,
self.assertEqual(reply.json()[1]['availability_policy'], 'friends') self.assertEqual(reply.json()[1]['availability_policy'], 'friends')
def test_get_inventory_item(self): def test_get_inventory_item(self):
reply = client.get('/api/v1/storage_locations/{}/'.format(self.own_handle), self.f['local_user1']) reply = client.get('/api/storage_locations/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 4) self.assertEqual(len(reply.json()), 4)
self.assertEqual(reply.json()[0]['name'], 'loc1') self.assertEqual(reply.json()[0]['name'], 'loc1')
@ -70,221 +69,3 @@ class LocationApiTestCase(UserTestMixin, InventoryTestMixin, LocationTestMixin,
self.assertEqual(reply.json()[3]['description'], None) self.assertEqual(reply.json()[3]['description'], None)
self.assertEqual(reply.json()[3]['category'], 'cat1') self.assertEqual(reply.json()[3]['category'], 'cat1')
self.assertEqual(reply.json()[3]['path'], 'loc1/loc4') self.assertEqual(reply.json()[3]['path'], 'loc1/loc4')
def test_post_new_location(self):
reply = client.post('/api/v1/storage_locations/{}/'.format(self.own_handle), self.f['local_user1'], {
'name': 'loc5',
'description': 'a new location',
})
self.assertEqual(reply.status_code, 201)
self.assertEqual(StorageLocation.objects.count(), 5)
location = StorageLocation.objects.get(name='loc5')
self.assertEqual(location.description, 'a new location')
self.assertEqual(location.owner, self.f['local_user1'])
self.assertEqual(location.parent, None)
self.assertEqual(reply.json()['path'], 'loc5')
def test_post_new_nested_location(self):
reply = client.post('/api/v1/storage_locations/{}/'.format(self.own_handle), self.f['local_user1'], {
'name': 'loc5',
'parent': self.f['loc3'].id,
})
self.assertEqual(reply.status_code, 201)
location = StorageLocation.objects.get(name='loc5')
self.assertEqual(location.parent, self.f['loc3'])
self.assertEqual(reply.json()['path'], 'loc1/loc3/loc5')
def test_patch_location(self):
reply = client.patch(
'/api/v1/storage_locations/{}/{}/'.format(self.own_handle, self.f['loc2'].id), self.f['local_user1'], {
'name': 'loc2-renamed',
'parent': self.f['loc1'].id,
})
self.assertEqual(reply.status_code, 200)
location = StorageLocation.objects.get(id=self.f['loc2'].id)
self.assertEqual(location.name, 'loc2-renamed')
self.assertEqual(location.parent, self.f['loc1'])
self.assertEqual(reply.json()['path'], 'loc1/loc2-renamed')
def test_delete_location(self):
reply = client.delete(
'/api/v1/storage_locations/{}/{}/'.format(self.own_handle, self.f['loc4'].id), self.f['local_user1'])
self.assertEqual(reply.status_code, 204)
self.assertEqual(StorageLocation.objects.count(), 3)
self.assertEqual(StorageLocation.objects.filter(id=self.f['loc4'].id).count(), 0)
def test_delete_location_with_items_sets_null(self):
item = InventoryItem.create_for_owner(
owner=self.f['local_user1'], name='located_item', storage_location=self.f['loc3'])
reply = client.delete(
'/api/v1/storage_locations/{}/{}/'.format(self.own_handle, self.f['loc3'].id), self.f['local_user1'])
self.assertEqual(reply.status_code, 204)
item.refresh_from_db()
self.assertIsNone(item.storage_location)
self.assertEqual(InventoryItem.objects.filter(id=item.id).count(), 1)
def test_locations_are_owner_scoped(self):
reply = client.get('/api/v1/storage_locations/{}/'.format(str(self.f['local_user2'])), self.f['local_user2'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 0)
def test_get_shared_location_visibility_private(self):
private_location = StorageLocation.create_for_owner(
owner=self.f['local_user1'], name='secret-loc', visibility_policy='private')
reply = client.get(
'/api/v1/storage_locations/{}/{}/'.format(self.own_handle, private_location.id), self.f['local_user2'])
self.assertEqual(reply.status_code, 404)
reply = client.get(
'/api/v1/storage_locations/{}/{}/'.format(self.own_handle, private_location.id), self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
def test_list_locations_excludes_visibility_private_for_friend(self):
StorageLocation.create_for_owner(owner=self.f['local_user1'], name='secret-loc', visibility_policy='private')
reply = client.get('/api/v1/storage_locations/{}/'.format(self.own_handle), self.f['local_user2'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 4)
self.assertNotIn('secret-loc', [loc['name'] for loc in reply.json()])
def test_cannot_delete_other_users_location(self):
# local_user2 is a friend of local_user1 (see prepare_inventory), so local_user1's own
# handle resolves and the location is visible - friends can read but never write, so this
# is a 403 (found, not authorized), not a 404.
reply = client.delete(
'/api/v1/storage_locations/{}/{}/'.format(self.own_handle, self.f['loc1'].id), self.f['local_user2'])
self.assertEqual(reply.status_code, 403)
self.assertEqual(StorageLocation.objects.filter(id=self.f['loc1'].id).count(), 1)
class GroupOwnedLocationApiTestCase(UserTestMixin, GroupTestMixin, ToolshedTestCase):
def setUp(self):
super().setUp()
self.prepare_users()
self.prepare_groups()
self.f['group1'].members.add(self.f['local_user2'].public_identity)
# The '+' prefix is the '#'->'+' escape resolve_owner_handle expects for a group handle in
# the owner_handle URL path parameter - the same escape encodeHandleForUrl uses.
self.group_handle = '+' + str(self.f['group1'])[1:]
def create_group_location(self, name='shelf'):
return client.post('/api/v1/storage_locations/{}/'.format(self.group_handle), self.f['local_user1'], {
'name': name,
})
def test_create_group_owned_location(self):
reply = self.create_group_location()
self.assertEqual(reply.status_code, 201)
location = StorageLocation.objects.get(name='shelf')
self.assertIsNone(location.owner)
self.assertEqual(location.owner_group, self.f['group1'])
self.assertEqual(reply.json()['owner_group'], str(self.f['group1']))
self.assertIsNone(reply.json()['owner'])
def test_create_group_owned_location_non_member_denied(self):
reply = client.post('/api/v1/storage_locations/{}/'.format(self.group_handle), self.f['ext_user1'], {
'name': 'shelf',
})
self.assertEqual(reply.status_code, 403)
self.assertEqual(StorageLocation.objects.count(), 0)
def test_other_member_can_edit(self):
location_id = self.create_group_location().json()['id']
reply = client.patch(
'/api/v1/storage_locations/{}/{}/'.format(self.group_handle, location_id), self.f['local_user2'], {
'name': 'shelf-renamed'
})
self.assertEqual(reply.status_code, 200)
self.assertEqual(StorageLocation.objects.get(id=location_id).name, 'shelf-renamed')
def test_other_member_can_delete(self):
location_id = self.create_group_location().json()['id']
reply = client.delete(
'/api/v1/storage_locations/{}/{}/'.format(self.group_handle, location_id), self.f['local_user2'])
self.assertEqual(reply.status_code, 204)
self.assertEqual(StorageLocation.objects.filter(id=location_id).count(), 0)
def test_delete_group_location_when_id_collides_with_own_personal_location(self):
# Mirrors InventoryItemViewSet's own regression test - id is only unique within its own
# owner/owner_group scope, so a member's personal location and their group's location can
# land on the same id. Every route being scoped by an explicit owner_handle - never a
# bare, unscoped id - means this can no longer raise MultipleObjectsReturned.
own_handle = str(self.f['local_user2'])
personal_reply = client.post('/api/v1/storage_locations/{}/'.format(own_handle), self.f['local_user2'],
{'name': 'personal-shelf'})
self.assertEqual(personal_reply.json()['id'], 1)
location_id = self.create_group_location().json()['id']
self.assertEqual(location_id, 1)
reply = client.delete(
'/api/v1/storage_locations/{}/{}/'.format(self.group_handle, location_id), self.f['local_user2'])
self.assertEqual(reply.status_code, 204)
self.assertEqual(StorageLocation.objects.filter(name='shelf').count(), 0)
self.assertEqual(StorageLocation.objects.filter(name='personal-shelf').count(), 1)
def test_remote_member_without_local_account_can_edit(self):
# A remote member (KnownIdentity, no ToolshedUser row) must still act on group-owned
# locations - not unauthorized just because .user.exists() is False.
self.f['group1'].members.add(self.f['ext_user1'].public_identity)
location_id = self.create_group_location().json()['id']
reply = client.get(
'/api/v1/storage_locations/{}/{}/'.format(self.group_handle, location_id), self.f['ext_user1'])
self.assertEqual(reply.status_code, 200)
reply = client.patch(
'/api/v1/storage_locations/{}/{}/'.format(self.group_handle, location_id), self.f['ext_user1'], {
'name': 'shelf-renamed-by-remote-member'
})
self.assertEqual(reply.status_code, 200)
self.assertEqual(StorageLocation.objects.get(id=location_id).name, 'shelf-renamed-by-remote-member')
def test_non_member_cannot_see_or_edit(self):
location_id = self.create_group_location().json()['id']
reply = client.get(
'/api/v1/storage_locations/{}/{}/'.format(self.group_handle, location_id), self.f['ext_user1'])
self.assertEqual(reply.status_code, 404)
def test_group_locations_excluded_from_personal_list(self):
self.create_group_location()
reply = client.get('/api/v1/storage_locations/{}/'.format(str(self.f['local_user1'])), self.f['local_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 0)
def test_group_locations_listed_by_owner_handle(self):
self.create_group_location()
reply = client.get('/api/v1/storage_locations/{}/'.format(self.group_handle), self.f['local_user2'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 1)
self.assertEqual(reply.json()[0]['name'], 'shelf')
def test_group_locations_not_listed_for_non_member(self):
self.create_group_location()
reply = client.get('/api/v1/storage_locations/{}/'.format(self.group_handle), self.f['ext_user1'])
self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 0)
def test_group_location_can_be_parent_for_group_member(self):
parent_id = self.create_group_location('shelf').json()['id']
reply = client.post('/api/v1/storage_locations/{}/'.format(self.group_handle), self.f['local_user2'], {
'name': 'bin', 'parent': parent_id,
})
self.assertEqual(reply.status_code, 201)
self.assertEqual(reply.json()['path'], 'shelf/bin')
def test_own_personal_location_can_be_parent_of_group_location(self):
# OwnerScopedPrimaryKeyRelatedField (see serializers.py) scopes `parent` to every
# location the requester can act on - own personal ones plus any group's - with no
# further check that it matches the new location's own owner_group. Same behavior
# InventoryItem.storage_location already had before group ownership existed here; not
# something this feature narrows.
personal = StorageLocation.create_for_owner(name='mine', owner=self.f['local_user1'])
reply = client.post('/api/v1/storage_locations/{}/'.format(self.group_handle), self.f['local_user1'], {
'name': 'bin', 'parent': personal.id,
})
self.assertEqual(reply.status_code, 201)
def test_group_location_not_valid_parent_for_non_member(self):
parent_id = self.create_group_location('shelf').json()['id']
reply = client.post(
'/api/v1/storage_locations/{}/'.format(str(self.f['ext_user1'])), self.f['ext_user1'], {
'name': 'bin', 'parent': parent_id,
})
self.assertEqual(reply.status_code, 400)

259
backend/toolshed/tests/test_offlinedata.py
View file

@ -1,259 +0,0 @@
from django.core.files.base import ContentFile
from django.test import Client
from authentication.models import AccountPreference, ToolshedUser
from authentication.tests import UserTestMixin, SignatureAuthClient, ToolshedTestCase
from files.models import File
from toolshed.models import InventoryItem, ItemProperty, StorageLocation
from toolshed.offlinedata import import_inventory, inventory_rows, rows_to_csv
from toolshed.tests import CategoryTestMixin, LocationTestMixin, PropertyTestMixin, TagTestMixin
anonymous_client = Client()
client = SignatureAuthClient()
class _DeleteTestDataMixin(UserTestMixin, CategoryTestMixin, LocationTestMixin):
"""Shared fixture setup for the delete-data and delete-account test cases."""
def setUp(self):
super().setUp()
self.prepare_users()
self.prepare_categories()
self.prepare_locations()
self.f['local_user1'].friends.add(self.f['local_user2'].public_identity)
self.f['shared_file'] = File.objects.create(
file=ContentFile(b'shared', 'shared'), mime_type='text/plain', hash='shared')
self.f['orphan_file'] = File.objects.create(
file=ContentFile(b'orphan', 'orphan'), mime_type='text/plain', hash='orphan')
self.f['item1'] = InventoryItem.create_for_owner(
owner=self.f['local_user1'], owned_quantity=1, name='item1', category=self.f['cat1'])
self.f['item1'].files.add(self.f['orphan_file'])
self.f['item_other_user'] = InventoryItem.create_for_owner(
owner=self.f['local_user2'], owned_quantity=1, name='item2', category=self.f['cat1'])
self.f['item_other_user'].files.add(self.f['shared_file'])
self.f['item1'].files.add(self.f['shared_file'])
AccountPreference.objects.create(user=self.f['local_user1'], key='theme', value='dark')
self.f['local_user1'].profile_picture = self.f['orphan_file']
self.f['local_user1'].save()
class DeleteDataTestCase(_DeleteTestDataMixin, ToolshedTestCase):
def test_delete_data_anonymous(self):
response = anonymous_client.delete('/api/v1/account_data/')
self.assertEqual(response.status_code, 403)
def test_delete_data_removes_all_owned_data_but_keeps_account(self):
response = client.delete('/api/v1/account_data/', self.f['local_user1'])
self.assertEqual(response.status_code, 200)
summary = response.json()
self.assertEqual(summary['inventory_items'], 1)
self.assertEqual(summary['locations'], 4)
self.assertEqual(summary['settings'], 1)
self.assertEqual(summary['friends'], 1)
# the account itself survives - this wipes data, it doesn't close the account
self.f['local_user1'].refresh_from_db()
self.assertTrue(ToolshedUser.objects.filter(username='testuser1').exists())
self.assertIsNone(self.f['local_user1'].profile_picture)
self.assertFalse(InventoryItem.global_objects.filter(owner_id=self.f['local_user1'].id).exists())
self.assertFalse(StorageLocation.objects.filter(owner_id=self.f['local_user1'].id).exists())
self.assertFalse(AccountPreference.objects.filter(user_id=self.f['local_user1'].id).exists())
self.assertEqual(self.f['local_user1'].public_identity.friends.count(), 0)
# orphaned file (only referenced by the deleted user/items) is gone
self.assertFalse(File.objects.filter(hash='orphan').exists())
# file still referenced by the other user's item survives
self.assertTrue(File.objects.filter(hash='shared').exists())
# the other user's data and identity/friend relation to the deleted identity are untouched
self.f['local_user2'].refresh_from_db()
self.assertTrue(InventoryItem.objects.filter(owner=self.f['local_user2']).exists())
class DeleteAccountTestCase(_DeleteTestDataMixin, ToolshedTestCase):
def test_delete_account_anonymous(self):
response = anonymous_client.delete('/api/v1/account/')
self.assertEqual(response.status_code, 403)
def test_delete_account_removes_data_and_closes_account(self):
user1_id = self.f['local_user1'].id
identity_id = self.f['local_user1'].public_identity_id
response = client.delete('/api/v1/account/', self.f['local_user1'])
self.assertEqual(response.status_code, 200)
summary = response.json()
self.assertEqual(summary['inventory_items'], 1)
self.assertEqual(summary['locations'], 4)
self.assertEqual(summary['settings'], 1)
self.assertEqual(summary['friends'], 1)
self.assertTrue(summary['account'])
# the account itself is gone
self.assertFalse(ToolshedUser.objects.filter(id=user1_id).exists())
self.assertFalse(InventoryItem.global_objects.filter(owner_id=user1_id).exists())
self.assertFalse(StorageLocation.objects.filter(owner_id=user1_id).exists())
self.assertFalse(AccountPreference.objects.filter(user_id=user1_id).exists())
# the underlying identity is kept, so remote friends/history referencing it stay intact
from authentication.models import KnownIdentity
self.assertTrue(KnownIdentity.objects.filter(id=identity_id).exists())
# orphaned file (only referenced by the deleted user/items) is gone
self.assertFalse(File.objects.filter(hash='orphan').exists())
# file still referenced by the other user's item survives
self.assertTrue(File.objects.filter(hash='shared').exists())
# the other user's data and identity/friend relation to the deleted identity are untouched
self.f['local_user2'].refresh_from_db()
self.assertTrue(InventoryItem.objects.filter(owner=self.f['local_user2']).exists())
self.assertEqual(self.f['local_user2'].public_identity.friends.count(), 0)
class ImportInventoryPropertiesTestCase(UserTestMixin, CategoryTestMixin, TagTestMixin, PropertyTestMixin,
ToolshedTestCase):
"""Properties must round-trip through export/import even when their value contains a comma or '=' sign, which a naive "handle=value, handle2=value2" encoding would otherwise misinterpret as a separator."""
def setUp(self):
super().setUp()
self.prepare_users()
self.prepare_categories()
self.prepare_tags()
self.prepare_properties()
def test_property_values_with_comma_and_equals_round_trip(self):
item = InventoryItem.create_for_owner(owner=self.f['local_user1'], name='widget')
ItemProperty.objects.create(inventory_item=item, property=self.f['prop1'], value='10cm, 20cm')
ItemProperty.objects.create(inventory_item=item, property=self.f['prop2'], value='a=b')
csv_bytes = b''.join(rows_to_csv(list(inventory_rows(self.f['local_user1']))))
imported, errors = import_inventory(self.f['local_user2'], csv_bytes, available_files={})
self.assertEqual(errors, [])
self.assertEqual(imported, 1)
new_item = InventoryItem.objects.get(owner=self.f['local_user2'], name='widget')
values = {ip.property.name: ip.value for ip in new_item.itemproperty_set.select_related('property')}
self.assertEqual(values, {'prop1': '10cm, 20cm', 'prop2': 'a=b'})
def test_legacy_comma_equals_format_is_still_importable(self):
handle1 = self.f['prop1'].get_handle()
handle2 = self.f['prop2'].get_handle()
csv_data = (
'name,properties\r\n'
f'legacy widget,"{handle1}=value1, {handle2}=value2"\r\n'
).encode('utf-8')
imported, errors = import_inventory(self.f['local_user1'], csv_data, available_files={})
self.assertEqual(errors, [])
self.assertEqual(imported, 1)
item = InventoryItem.objects.get(owner=self.f['local_user1'], name='legacy widget')
values = {ip.property.name: ip.value for ip in item.itemproperty_set.select_related('property')}
self.assertEqual(values, {'prop1': 'value1', 'prop2': 'value2'})
def test_item_without_properties_imports_cleanly(self):
item = InventoryItem.create_for_owner(owner=self.f['local_user1'], name='bare item')
csv_bytes = b''.join(rows_to_csv(list(inventory_rows(self.f['local_user1']))))
imported, errors = import_inventory(self.f['local_user2'], csv_bytes, available_files={})
self.assertEqual(errors, [])
self.assertEqual(imported, 1)
new_item = InventoryItem.objects.get(owner=self.f['local_user2'], name='bare item')
self.assertEqual(list(new_item.itemproperty_set.all()), [])
def test_category_and_tags_round_trip(self):
item = InventoryItem.create_for_owner(
owner=self.f['local_user1'], name='cat and tags item', category=self.f['cat1'])
item.tags.add(self.f['tag1'], self.f['tag2'], through_defaults={})
csv_bytes = b''.join(rows_to_csv(list(inventory_rows(self.f['local_user1']))))
imported, errors = import_inventory(self.f['local_user2'], csv_bytes, available_files={})
self.assertEqual(errors, [])
self.assertEqual(imported, 1)
new_item = InventoryItem.objects.get(owner=self.f['local_user2'], name='cat and tags item')
self.assertEqual(new_item.category, self.f['cat1'])
self.assertEqual(sorted(t.name for t in new_item.tags.all()), ['tag1', 'tag2'])
def test_unknown_property_handle_skips_item_with_error(self):
csv_data = (
'name,properties\r\n'
'ghost widget,test#property:doesnotexist=x\r\n'
).encode('utf-8')
imported, errors = import_inventory(self.f['local_user1'], csv_data, available_files={})
self.assertEqual(imported, 0)
self.assertEqual(len(errors), 1)
self.assertIn('doesnotexist', errors[0])
self.assertFalse(InventoryItem.objects.filter(owner=self.f['local_user1'], name='ghost widget').exists())
def test_property_value_with_quote_character_round_trips(self):
item = InventoryItem.create_for_owner(owner=self.f['local_user1'], name='quoted widget')
ItemProperty.objects.create(inventory_item=item, property=self.f['prop1'], value='12" screen')
csv_bytes = b''.join(rows_to_csv(list(inventory_rows(self.f['local_user1']))))
imported, errors = import_inventory(self.f['local_user2'], csv_bytes, available_files={})
self.assertEqual(errors, [])
self.assertEqual(imported, 1)
new_item = InventoryItem.objects.get(owner=self.f['local_user2'], name='quoted widget')
values = {ip.property.name: ip.value for ip in new_item.itemproperty_set.select_related('property')}
self.assertEqual(values, {'prop1': '12" screen'})
class ExportImportApiRoundTripTestCase(UserTestMixin, CategoryTestMixin, TagTestMixin, PropertyTestMixin,
ToolshedTestCase):
"""End-to-end coverage of the /api/v1/export/ + /api/v1/import/ endpoints (as actually used by clients), rather than calling the internal helper functions directly - this is what a real export/import round trip between two accounts looks like."""
def setUp(self):
super().setUp()
self.prepare_users()
self.prepare_categories()
self.prepare_tags()
self.prepare_properties()
def test_export_then_import_preserves_category_tags_and_properties(self):
import base64
item = InventoryItem.create_for_owner(
owner=self.f['local_user1'], name='drill', description='cordless drill',
category=self.f['cat1'], availability_policy='friends', visibility_policy='public', owned_quantity=2)
item.tags.add(self.f['tag1'], self.f['tag2'], through_defaults={})
ItemProperty.objects.create(inventory_item=item, property=self.f['prop1'], value='10cm, 20cm')
ItemProperty.objects.create(inventory_item=item, property=self.f['prop2'], value='a=b')
export_reply = client.get('/api/v1/export/', self.f['local_user1'])
self.assertEqual(export_reply.status_code, 200)
zip_bytes = export_reply.content
import_reply = client.post('/api/v1/import/', self.f['local_user2'],
{'zip': base64.b64encode(zip_bytes).decode('ascii')})
self.assertEqual(import_reply.status_code, 200)
summary = import_reply.json()
self.assertEqual(summary['inventory_items'], 1)
self.assertEqual(summary['errors'], [])
new_item = InventoryItem.objects.get(owner=self.f['local_user2'], name='drill')
self.assertEqual(new_item.category, self.f['cat1'])
self.assertEqual(new_item.visibility_policy, 'public')
self.assertEqual(sorted(t.name for t in new_item.tags.all()), ['tag1', 'tag2'])
values = {ip.property.name: ip.value for ip in new_item.itemproperty_set.select_related('property')}
self.assertEqual(values, {'prop1': '10cm, 20cm', 'prop2': 'a=b'})

2
backend/toolshed/tests/test_property.py
View file

@ -41,7 +41,7 @@ class PropertyApiTestCase(PropertyTestMixin, CategoryTestMixin, UserTestMixin, T
self.prepare_properties() self.prepare_properties()
def test_get_properties(self): def test_get_properties(self):
reply = client.get('/api/v1/properties/', self.f['local_user1']) reply = client.get('/api/properties/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 3) self.assertEqual(len(reply.json()), 3)
self.assertEqual(reply.json()[0]['name'], 'prop1') self.assertEqual(reply.json()[0]['name'], 'prop1')

2
backend/toolshed/tests/test_tag.py
View file

@ -41,7 +41,7 @@ class TagApiTestCase(TagTestMixin, CategoryTestMixin, UserTestMixin, ToolshedTes
self.prepare_tags() self.prepare_tags()
def test_get_tags(self): def test_get_tags(self):
reply = client.get('/api/v1/tags/', self.f['local_user1']) reply = client.get('/api/tags/', self.f['local_user1'])
self.assertEqual(reply.status_code, 200) self.assertEqual(reply.status_code, 200)
self.assertEqual(len(reply.json()), 3) self.assertEqual(len(reply.json()), 3)
self.assertEqual(reply.json()[0], 'tag1') self.assertEqual(reply.json()[0], 'tag1')

48
backend/toolshed/tests/test_workflow_api.py
View file

@ -1,48 +0,0 @@
import json
from django.test import Client
from django.urls import reverse
from rest_framework import status
from authentication.tests import UserTestMixin, SignatureAuthClient, ToolshedTestCase
from toolshed.tests import WorkflowTestMixin
from toolshed.models import WorkflowInstance
anonymous_client = Client()
client = SignatureAuthClient()
class WorkflowInstanceApiTestCase(UserTestMixin, WorkflowTestMixin, ToolshedTestCase):
"""Comprehensive test cases for the Workflow API"""
def setUp(self):
super().setUp()
self.prepare_users()
self.prepare_workflows()
def test_get_workflow_instances(self):
reply = client.get('/api/v1/workflows/', self.f['local_user1'])
self.assertEqual(reply.status_code, status.HTTP_200_OK)
self.assertEqual(len(reply.data), 3)
self.assertEqual(reply.data[0]['name'], 'workflow1')
self.assertEqual(reply.data[1]['name'], 'workflow1')
self.assertEqual(reply.data[2]['name'], 'workflow1')
self.assertEqual(reply.data[0]['state'], 'initial')
self.assertEqual(reply.data[1]['state'], 'upload')
self.assertEqual(reply.data[2]['state'], 'describe')
self.assertEqual(json.loads(reply.data[0]['payload']), {})
self.assertEqual(json.loads(reply.data[1]['payload']), {'files': ['ef35c4a9b2d1c4f1a3e6f7d8c9b0a1b2']})
self.assertEqual(json.loads(reply.data[2]['payload']), {'files': ['ef35c4a9b2d1c4f1a3e6f7d8c9b0a1b2',
'a1b2c3d4e5f60718293a4b5c6d7e8f90',
'b1c2d3e4f5a60718293b4c5d6e7f8090'],
'descriptions': ['file 1 description']})
def test_get_workflow_instances_user2(self):
reply = client.get('/api/v1/workflows/', self.f['local_user2'])
self.assertEqual(reply.status_code, status.HTTP_200_OK)
self.assertEqual(len(reply.data), 1)
self.assertEqual(reply.data[0]['name'], 'workflow2')
self.assertEqual(reply.data[0]['state'], 'initial')
self.assertEqual(json.loads(reply.data[0]['payload']), {})

320
cli-client/toolshed-client.py
View file

@ -1,320 +0,0 @@
#!/usr/bin/env python3
import argparse
import base64
import os
import sys
import requests
from nacl.signing import SigningKey
from json import dumps, loads
COMMANDS = {
'getinventory': {'path': '/api/v1/inventory_items/{handle}/', 'method': 'get'},
'additem': {'path': '/api/v1/inventory_items/{handle}/', 'method': 'post'},
'delitem': {'path': '/api/v1/inventory_items/{handle}/{internal_id}/', 'method': 'delete'},
}
class ApiError(Exception):
"""A problem talking to the backend - network/TLS failure, or a response that isn't the JSON
we expected. Deliberately distinct from ValueError (bad input) so main() can report both with
a plain message instead of a traceback, without conflating "you gave me something invalid"
with "the server didn't behave"."""
class ToolshedApi:
user = None
host = None
signing_key = None
def __init__(self, user, host, key, ca_cert=None):
if host is None:
raise ValueError("No host configured - set TOOLSHED_HOST or pass --host (e.g. a.localhost:8000)")
if user is None:
raise ValueError("No user configured - set TOOLSHED_USER or pass --user (e.g. you@a.localhost)")
if key is None:
raise ValueError("No signing key configured - set TOOLSHED_KEY or pass --key")
if len(key) != 64:
raise ValueError("TOOLSHED_KEY must be 64 hex characters, got {} characters".format(len(key)))
try:
signing_key = SigningKey(bytes.fromhex(key))
except ValueError:
raise ValueError("TOOLSHED_KEY must be a hex-encoded Ed25519 private key")
if ca_cert is not None and not os.path.isfile(ca_cert):
raise ValueError("CA cert file not found: {}".format(ca_cert))
self.user = user
self.host = host
self.signing_key = signing_key
self.verify = ca_cert if ca_cert is not None else True
self._spec = None
def _url(self, target):
return "https://" + self.host + target
def _ssl_error(self, error):
hint = "" if self.verify is not True else \
" - if this is a dev server with a self-signed cert, pass --ca-cert/TOOLSHED_CA_CERT"
return ApiError("TLS error talking to {}: {}{}".format(self.host, error, hint))
def _send(self, method, target, json_body=None):
url = self._url(target)
signed_body = dumps(json_body).encode('utf-8') if json_body is not None else b''
signature = self.signing_key.sign(url.encode('utf-8') + signed_body).signature.hex()
headers = {"Authorization": "Signature " + self.user + ":" + signature}
try:
return requests.request(method, url, headers=headers, json=json_body, verify=self.verify)
except requests.exceptions.SSLError as error:
raise self._ssl_error(error)
except requests.exceptions.ConnectionError as error:
raise ApiError("Could not reach {} - is the host/port correct and reachable? ({})".format(
self.host, error))
except requests.exceptions.Timeout:
raise ApiError("Request to {} timed out".format(self.host))
@staticmethod
def _parse_json(response):
try:
return response.json()
except ValueError:
raise ApiError("Expected a JSON response from {} but got {} {}: {}".format(
response.url, response.status_code, response.reason, response.text[:300]))
def get_spec(self):
if self._spec is None:
try:
response = requests.get(self._url("/docs/?format=openapi"), verify=self.verify)
except requests.exceptions.SSLError as error:
raise self._ssl_error(error)
except requests.exceptions.ConnectionError as error:
raise ApiError("Could not reach {} - is the host/port correct and reachable? ({})".format(
self.host, error))
self._spec = self._parse_json(response)
return self._spec
def get(self, target):
return self._parse_json(self._send('GET', target))
def post(self, target, data):
return self._parse_json(self._send('POST', target, json_body=data))
def delete(self, target):
response = self._send('DELETE', target)
if not response.content:
return {"deleted": response.ok}
return self._parse_json(response)
def get_raw(self, target):
"""Like get(), but returns the raw response body instead of parsing it as JSON - for
endpoints like /api/v1/export/ that hand back a zip file, not a JSON document."""
response = self._send('GET', target)
if not response.ok:
raise ApiError("{} {} from {}: {}".format(
response.status_code, response.reason, response.url, response.text[:300]))
return response.content
def resolve_schema(spec, schema):
if '$ref' in schema:
return spec['definitions'][schema['$ref'].split('/')[-1]]
return schema
def operation_parameters(spec, path, method):
"""Path- and body-parameters for an operation, per the endpoint's own OpenAPI spec entry."""
path_item = spec['paths'][path]
params = path_item.get('parameters', []) + path_item[method].get('parameters', [])
path_params = [p for p in params if p.get('in') == 'path']
body_param = next((p for p in params if p.get('in') == 'body'), None)
return path_params, body_param
def parse_kv_args(cmd_args):
"""['name=Drill', 'owned_quantity=2'] -> {'name': 'Drill', 'owned_quantity': '2'}"""
pairs = {}
for arg in cmd_args:
if '=' in arg:
key, value = arg.split('=', 1)
pairs[key] = value
return pairs
def resolve_path_params(api, path_params, cmd_args, json_input):
values = {}
positional = [arg for arg in cmd_args if '=' not in arg]
kv = parse_kv_args(cmd_args)
for param in path_params:
name = param['name']
if name == 'handle':
values[name] = api.user
continue
if json_input is not None:
value = json_input.get(name, json_input.get('id'))
if value is None:
raise ValueError("Missing required path parameter '{}' in --json stdin input".format(name))
elif name in kv:
value = kv[name]
elif 'id' in kv:
value = kv['id']
elif positional:
value = positional.pop(0)
else:
value = input("{}: ".format(name))
values[name] = value
return values
def resolve_body(spec, body_param, cmd_args, json_input):
if body_param is None:
return None
if json_input is not None:
return json_input
kv = parse_kv_args(cmd_args)
if kv:
return kv
schema = resolve_schema(spec, body_param['schema'])
writable_fields = [name for name, prop in schema.get('properties', {}).items() if not prop.get('readOnly')]
body = {}
for field in writable_fields:
value = input("{}: ".format(field))
if value != '':
body[field] = value
return body
def build_request(api, spec, command, cmd_args, json_input):
path_params, body_param = operation_parameters(spec, command['path'], command['method'])
values = resolve_path_params(api, path_params, cmd_args, json_input)
url = command['path']
for name, value in values.items():
url = url.replace('{' + name + '}', str(value))
body = resolve_body(spec, body_param, cmd_args, json_input)
return url, body
def run_command(api, cmd, cmd_args, json_input):
if cmd == 'export':
path = cmd_args[0] if cmd_args else 'toolshed-export.zip'
data = api.get_raw("/api/v1/export/")
with open(path, 'wb') as f:
f.write(data)
return {'exported_to': path, 'bytes': len(data)}
elif cmd == 'import':
if not cmd_args:
raise ValueError("import requires the path to a previously exported zip file")
path = cmd_args[0]
with open(path, 'rb') as f:
data = f.read()
return api.post("/api/v1/import/", {"zip": base64.b64encode(data).decode('ascii')})
command = COMMANDS.get(cmd)
if command is None:
raise ValueError("Unknown command: " + cmd)
spec = api.get_spec()
url, body = build_request(api, spec, command, cmd_args, json_input)
if command['method'] == 'get':
return api.get(url)
elif command['method'] == 'post':
return api.post(url, body or {})
elif command['method'] == 'delete':
return api.delete(url)
else:
raise ValueError("Unsupported method: " + command['method'])
def read_json_input():
if sys.stdin.isatty():
return {}
raw = sys.stdin.read()
return loads(raw) if raw.strip() else {}
def stringify_cell(value):
if value is None:
return ''
if isinstance(value, (list, tuple)):
return ', '.join(stringify_cell(v) for v in value)
if isinstance(value, dict):
return dumps(value)
return str(value)
def format_table(rows):
if not rows:
return '(empty)'
columns = []
for row in rows:
for key in row.keys():
if key not in columns:
columns.append(key)
cells = [[stringify_cell(row.get(column)) for column in columns] for row in rows]
widths = [max([len(columns[i])] + [len(cell[i]) for cell in cells] + [3]) for i in range(len(columns))]
def format_row(values):
return '| ' + ' | '.join(value.ljust(widths[i]) for i, value in enumerate(values)) + ' |'
lines = [format_row(columns), '|-' + '-|-'.join('-' * width for width in widths) + '-|']
for cell in cells:
lines.append(format_row(cell))
return '\n'.join(lines)
def print_result(result, as_json):
if as_json:
print(dumps(result, indent=2))
elif isinstance(result, list) and all(isinstance(item, dict) for item in result):
print(format_table(result))
elif isinstance(result, dict):
print(format_table([result]))
else:
print(result)
def main():
host = os.environ.get('TOOLSHED_HOST')
user = os.environ.get('TOOLSHED_USER')
key = os.environ.get('TOOLSHED_KEY')
ca_cert = os.environ.get('TOOLSHED_CA_CERT')
parser = argparse.ArgumentParser(description='Toolshed API client')
parser.add_argument('--host', help='Toolshed host')
parser.add_argument('--user', help='Toolshed user')
parser.add_argument('--key', help='Toolshed key')
parser.add_argument('--ca-cert',
help='CA cert file to verify the server against, for a dev/self-signed host '
'(e.g. frontend/.local/RootCA.crt) - normal system CAs are used otherwise')
parser.add_argument('--json', action='store_true',
help='Read input as JSON from stdin and print output as JSON, instead of prompting/printing')
parser.add_argument('cmd', help='Command')
parser.add_argument('args', nargs='*', help="Command arguments, as key=value pairs (e.g. name=Drill)")
args = parser.parse_args()
if args.host is not None:
host = args.host
if args.user is not None:
user = args.user
if args.key is not None:
key = args.key
if args.ca_cert is not None:
ca_cert = args.ca_cert
try:
api = ToolshedApi(user, host, key, ca_cert)
json_input = read_json_input() if args.json else None
result = run_command(api, args.cmd, args.args, json_input)
except (ValueError, ApiError, OSError) as error:
print("Error: {}".format(error), file=sys.stderr)
sys.exit(1)
print_result(result, args.json)
if __name__ == '__main__':
main()

5
deploy/dev/Dockerfile.backend
View file

@ -7,11 +7,10 @@ ENV PYTHONUNBUFFERED 1
# Set work directory # Set work directory
WORKDIR /code WORKDIR /code
RUN mkdir /git
# Install dependencies # Install dependencies
COPY requirements.txt /code/ COPY requirements.txt /code/
RUN pip install --no-cache-dir -r requirements.txt RUN pip install --no-cache-dir -r requirements.txt
RUN touch /mnt/db.sqlite3 && touch /mnt/testdata.py && mkdir /mnt/userfiles
# Run the application # Run the application
CMD ["sh", "-c", "python manage.py migrate && python manage.py runserver 0.0.0.0:8000 --insecure"] CMD ["python", "manage.py", "runserver", "0.0.0.0:8000", "--insecure"]

2
deploy/dev/Dockerfile.frontend
View file

@ -3,13 +3,11 @@ FROM node:14
# Set work directory # Set work directory
WORKDIR /app WORKDIR /app
RUN mkdir /git
# Install app dependencies # Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied # A wildcard is used to ensure both package.json AND package-lock.json are copied
COPY package.json ./ COPY package.json ./
COPY extras/ ./extras/
RUN npm install RUN npm install
CMD [ "npm", "run", "dev", "--", "--host"] CMD [ "npm", "run", "dev", "--", "--host"]

13
deploy/dev/Dockerfile.proxy
View file

@ -1 +1,14 @@
FROM nginx:bookworm FROM nginx:bookworm
# snakeoil for localhost
RUN apt-get update && \
apt-get install -y openssl && \
openssl genrsa -des3 -passout pass:x -out server.pass.key 2048 && \
openssl rsa -passin pass:x -in server.pass.key -out server.key && \
rm server.pass.key && \
openssl req -new -key server.key -out server.csr \
-subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=localhost" && \
openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt &&\
mv server.crt /etc/nginx/nginx.crt && \
mv server.key /etc/nginx/nginx.key \

15
deploy/dev/dns_server.py
View file

@ -3,9 +3,6 @@ import socketserver
import urllib.parse import urllib.parse
import dnslib import dnslib
import base64 import base64
import socket
UPSTREAM_DNS = ("8.8.8.8", 53)
try: try:
@ -14,13 +11,6 @@ try:
if record["name"] == qname and record["type"] == qtype and "value" in record: if record["name"] == qname and record["type"] == qtype and "value" in record:
return record["value"] return record["value"]
def resolve_recursive(raw_query):
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
sock.settimeout(5)
sock.sendto(raw_query, UPSTREAM_DNS)
data, _ = sock.recvfrom(4096)
return data
class DnsHttpRequestHandler(http.server.BaseHTTPRequestHandler): class DnsHttpRequestHandler(http.server.BaseHTTPRequestHandler):
def do_GET(self): def do_GET(self):
@ -50,16 +40,15 @@ try:
print("SRV record") print("SRV record")
reply = dnslib.SRV(record["priority"], record["weight"], record["port"], record["target"]) reply = dnslib.SRV(record["priority"], record["weight"], record["port"], record["target"])
response.add_answer(dnslib.RR(dns.q.qname, dns.q.qtype, rdata=reply)) response.add_answer(dnslib.RR(dns.q.qname, dns.q.qtype, rdata=reply))
pack = response.pack()
else: else:
print(f"Recursively resolving {dns.q.qname}") response.header.rcode = dnslib.RCODE.NXDOMAIN
pack = resolve_recursive(raw)
print(response) print(response)
self.send_response(200) self.send_response(200)
self.send_header("Content-type", "application/dns-message") self.send_header("Content-type", "application/dns-message")
self.end_headers() self.end_headers()
pack = response.pack()
self.wfile.write(pack) self.wfile.write(pack)
return return
except Exception as e: except Exception as e:

110
deploy/dev/docker-compose.yml
View file

@ -1,110 +0,0 @@
version: '3.8'
name: dev
services:
backend-a:
build:
context: ../../backend/
dockerfile: ../deploy/dev/Dockerfile.backend
environment:
TOOLSHED_DB_PATH: /mnt/db.sqlite3
TOOLSHED_USERFILES_PATH: /mnt/userfiles
TOOLSHED_SETUP_PATH: /mnt/testdata.py
volumes:
- ../../backend:/code
- ../../.git:/git:ro
- ./instance_a/a.env:/code/.env
- ./instance_a/testdata.py:/mnt/testdata.py
- ./instance_a/a.sqlite3:/mnt/db.sqlite3
- ./instance_a/userfiles:/mnt/userfiles
expose:
- 8000
command: bash -c "python configure.py; python configure.py testdata; python manage.py runserver 0.0.0.0:8000 --insecure"
backend-b:
build:
context: ../../backend/
dockerfile: ../deploy/dev/Dockerfile.backend
environment:
TOOLSHED_DB_PATH: /mnt/db.sqlite3
TOOLSHED_USERFILES_PATH: /mnt/userfiles
TOOLSHED_SETUP_PATH: /mnt/testdata.py
volumes:
- ../../backend:/code
- ../../.git:/git:ro
- ./instance_b/b.env:/code/.env
- ./instance_b/testdata.py:/mnt/testdata.py
- ./instance_b/b.sqlite3:/mnt/db.sqlite3
- ./instance_b/userfiles:/mnt/userfiles
expose:
- 8000
command: bash -c "python configure.py; python configure.py testdata; python manage.py runserver 0.0.0.0:8000 --insecure"
frontend:
build:
context: ../../frontend/
dockerfile: ../deploy/dev/Dockerfile.frontend
volumes:
- ../../frontend:/app
- ../../.git:/git:ro
- /app/node_modules
expose:
- 5173
command: bash -c "npm install && npm run dev -- --host"
wiki:
build:
context: ../../
dockerfile: deploy/dev/Dockerfile.wiki
volumes:
- ../../mkdocs.yml:/wiki/mkdocs.yml
- ../../docs:/wiki/docs
expose:
- 8001
command: mkdocs serve --dev-addr=0.0.0.0:8001
proxy-a:
build:
context: ./
dockerfile: Dockerfile.proxy
volumes:
- ./instance_a/nginx-a.dev.conf:/etc/nginx/nginx.conf:ro
- ./instance_a/dns.json:/var/www/dns.json:ro
- ./instance_a/domains.json:/var/www/domains.json:ro
- ./instance_a/userfiles:/var/www/userfiles:ro
# A stable, CA-signed cert (see frontend/.local/make_localhost.sh) covering localhost plus
# every loopback IP a dev proxy binds to below, instead of Dockerfile.proxy generating a
# fresh throwaway self-signed one on every image build - that regenerated cert invalidated
# any trust exception you'd added in your browser on the previous build. Trust
# frontend/.local/RootCA.crt once (see docs/development.md) and it keeps working across
# rebuilds.
- ../../frontend/.local/localhost.crt:/etc/nginx/nginx.crt:ro
- ../../frontend/.local/localhost.key:/etc/nginx/nginx.key:ro
ports:
- "127.0.0.1:8080:8080"
- "127.0.0.3:5353:5353"
proxy-b:
build:
context: ./
dockerfile: Dockerfile.proxy
volumes:
- ./instance_b/nginx-b.dev.conf:/etc/nginx/nginx.conf:ro
- ./instance_b/userfiles:/var/www/userfiles:ro
- ../../frontend/.local/localhost.crt:/etc/nginx/nginx.crt:ro
- ../../frontend/.local/localhost.key:/etc/nginx/nginx.key:ro
ports:
- "127.0.0.2:8080:8080"
dns:
build:
context: ./
dockerfile: Dockerfile.dns
volumes:
- ./zone.json:/dns/zone.json
expose:
- 8053
networks:
default:
aliases:
- toolshed-dns

2
deploy/dev/instance_a/a.env
View file

@ -2,8 +2,6 @@
# SECURITY WARNING: don't run with debug turned on in production! # SECURITY WARNING: don't run with debug turned on in production!
DEBUG=True DEBUG=True
SERVE_X_ACCEL_REDIRECT=False
# SECURITY WARNING: keep the secret key used in production secret! # SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY='e*lm&*!j0_stqaiod$1zob(vs@aq6+n-i$1%!rek)_v9n^ue$3' SECRET_KEY='e*lm&*!j0_stqaiod$1zob(vs@aq6+n-i$1%!rek)_v9n^ue$3'

58
deploy/dev/instance_a/nginx-a.dev.conf
View file

@ -1,8 +1,6 @@
events {} events {}
http { http {
client_max_body_size 128M;
upstream backend { upstream backend {
server backend-a:8000; server backend-a:8000;
} }
@ -16,7 +14,7 @@ http {
} }
upstream dns { upstream dns {
server toolshed-dns:8053; server dns:8053;
} }
server { server {
@ -47,34 +45,8 @@ http {
proxy_pass http://backend; proxy_pass http://backend;
} }
location /media {
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Port $server_port;
proxy_pass http://backend;
}
location /djangoadmin {
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Port $server_port;
proxy_pass http://backend;
}
location /docs { location /docs {
proxy_set_header Host $host:$server_port; proxy_pass http://backend/docs;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Port $server_port;
proxy_pass http://backend;
} }
location /static { location /static {
@ -108,35 +80,17 @@ http {
# DoH server # DoH server
server { server {
listen 5353 ssl; listen 5353 ssl;
server_name localhost 127.0.0.3; server_name localhost;
ssl_certificate /etc/nginx/nginx.crt; ssl_certificate /etc/nginx/nginx.crt;
ssl_certificate_key /etc/nginx/nginx.key; ssl_certificate_key /etc/nginx/nginx.key;
# Ensure CORS headers are present even when nginx generates 5xx responses.
add_header 'Access-Control-Allow-Origin' '*' always;
add_header 'Access-Control-Allow-Methods' 'GET, OPTIONS' always;
add_header 'Access-Control-Allow-Headers' 'Accept, Content-Type, Origin, User-Agent' always;
add_header 'Access-Control-Expose-Headers' 'Content-Type' always;
error_page 500 502 503 504 = @doh_error;
location /dns-query { location /dns-query {
if ($request_method = OPTIONS) {
add_header 'Access-Control-Allow-Origin' '*' always;
add_header 'Access-Control-Allow-Methods' 'GET, OPTIONS' always;
add_header 'Access-Control-Allow-Headers' 'Accept, Content-Type, Origin, User-Agent' always;
add_header 'Access-Control-Max-Age' 86400 always;
add_header 'Content-Length' 0;
add_header 'Content-Type' 'text/plain; charset=utf-8';
return 204;
}
proxy_pass http://dns; proxy_pass http://dns;
} # allow any origin
add_header 'Access-Control-Allow-Origin' '*';
add_header 'Access-Control-Allow-Methods' 'GET, OPTIONS';
location @doh_error {
default_type text/plain;
return 502 'DoH upstream unavailable';
} }
} }
} }

2
deploy/dev/instance_b/b.env
View file

@ -1,8 +1,6 @@
# SECURITY WARNING: don't run with debug turned on in production! # SECURITY WARNING: don't run with debug turned on in production!
DEBUG=True DEBUG=True
SERVE_X_ACCEL_REDIRECT=True
# SECURITY WARNING: keep the secret key used in production secret! # SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY='7ccxjje%q@@0*z+r&-$fy3(rj9n)%$!sk-k++-&rb=_u(wpjbe' SECRET_KEY='7ccxjje%q@@0*z+r&-$fy3(rj9n)%$!sk-k++-&rb=_u(wpjbe'

24
deploy/dev/instance_b/nginx-b.dev.conf
View file

@ -1,8 +1,6 @@
events {} events {}
http { http {
client_max_body_size 128M;
upstream backend { upstream backend {
server backend-b:8000; server backend-b:8000;
} }
@ -37,28 +35,6 @@ http {
proxy_pass http://backend; proxy_pass http://backend;
} }
location /media {
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Port $server_port;
proxy_pass http://backend;
}
location /redirect_media/ {
internal;
alias /var/www/userfiles/;
# This location serves the file directly, bypassing Django (and therefore
# django-cors-headers) entirely - it's the target of the X-Accel-Redirect
# response from files/media_urls.py, used when SERVE_X_ACCEL_REDIRECT=True.
# CORS headers must be added here explicitly since nothing else will.
add_header 'Access-Control-Allow-Origin' '*' always;
add_header 'Access-Control-Allow-Methods' 'GET, OPTIONS' always;
add_header 'Access-Control-Allow-Headers' 'Authorization, Accept, Content-Type, Origin, User-Agent' always;
}
location /docs { location /docs {
proxy_pass http://backend/docs; proxy_pass http://backend/docs;
} }

2
deploy/dev/zone.json
View file

@ -21,4 +21,4 @@
"target": "127.0.0.2." "target": "127.0.0.2."
} }
} }
] ]

78
deploy/docker-compose.override.yml Normal file
View file

@ -0,0 +1,78 @@
version: '3.8'
services:
backend-a:
build:
context: ../backend/
dockerfile: ../deploy/dev/Dockerfile.backend
volumes:
- ../backend:/code
- ../deploy/dev/instance_a/a.env:/code/.env
- ../deploy/dev/instance_a/a.sqlite3:/code/db.sqlite3
expose:
- 8000
command: bash -c "python configure.py; python configure.py testdata; python manage.py runserver 0.0.0.0:8000 --insecure"
backend-b:
build:
context: ../backend/
dockerfile: ../deploy/dev/Dockerfile.backend
volumes:
- ../backend:/code
- ../deploy/dev/instance_b/b.env:/code/.env
- ../deploy/dev/instance_b/b.sqlite3:/code/db.sqlite3
expose:
- 8000
command: bash -c "python configure.py; python configure.py testdata; python manage.py runserver 0.0.0.0:8000 --insecure"
frontend:
build:
context: ../frontend/
dockerfile: ../deploy/dev/Dockerfile.frontend
volumes:
- ../frontend:/app:ro
- /app/node_modules
expose:
- 5173
command: npm run dev -- --host
wiki:
build:
context: ../
dockerfile: deploy/dev/Dockerfile.wiki
volumes:
- ../mkdocs.yml:/wiki/mkdocs.yml
- ../docs:/wiki/docs
expose:
- 8001
command: mkdocs serve --dev-addr=0.0.0.0:8001
proxy-a:
build:
context: ./
dockerfile: dev/Dockerfile.proxy
volumes:
- ./dev/instance_a/nginx-a.dev.conf:/etc/nginx/nginx.conf:ro
- ./dev/instance_a/dns.json:/var/www/dns.json:ro
- ./dev/instance_a/domains.json:/var/www/domains.json:ro
ports:
- "127.0.0.1:8080:8080"
- "127.0.0.3:5353:5353"
proxy-b:
build:
context: ./
dockerfile: dev/Dockerfile.proxy
volumes:
- ./dev/instance_b/nginx-b.dev.conf:/etc/nginx/nginx.conf:ro
ports:
- "127.0.0.2:8080:8080"
dns:
build:
context: ./dev/
dockerfile: Dockerfile.dns
volumes:
- ./dev/zone.json:/dns/zone.json
expose:
- 8053

3
deploy/prod/.gitignore vendored
View file

@ -1,3 +0,0 @@
.secrets/
inventory.yml
.frontend-build/

33
deploy/prod/Dockerfile.backend
View file

@ -1,33 +0,0 @@
# Production image for the Django backend.
# Runs migrations then serves the app with gunicorn on port 8000.
# Static files are collected at build time into /app/staticfiles and
# served by the backend itself behind the host nginx reverse proxy.
FROM python:3.11-slim
# The build context here is just backend/ (no .git), so settings.py's own
# `git rev-parse` fallback can't find a repo - the actual commit is passed
# in from the real checkout via this build-arg instead (see playbook.yml).
ARG GIT_COMMIT=unknown
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
DJANGO_SETTINGS_MODULE=backend.settings \
GIT_COMMIT=$GIT_COMMIT
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --upgrade pip \
&& pip install --no-cache-dir -r requirements.txt gunicorn
COPY . .
# collectstatic only needs Django settings to import cleanly, not a real
# secret; the actual SECRET_KEY is injected at container runtime via
# --env-file and overrides this.
RUN SECRET_KEY=build-time-placeholder python manage.py collectstatic --noinput
EXPOSE 8000
CMD ["sh", "-c", "python manage.py migrate --noinput && exec gunicorn backend.wsgi:application --bind 0.0.0.0:8000 --workers 3"]

26
deploy/prod/Dockerfile.frontend
View file

@ -1,26 +0,0 @@
# Build-only image for the Vue frontend.
# It is never run as a service: ansible builds this image once, runs it
# with the host output directory bind-mounted at /output, the container
# copies the compiled static build into it, and exits. Nginx on the host
# then serves that directory directly.
FROM node:20-alpine AS build
WORKDIR /app
# The build context here is just frontend/ (no .git), so vite.config.js's
# own `git rev-parse` fallback can't find a repo - the actual commit is
# passed in from the real checkout via this build-arg instead (see
# playbook.yml).
ARG GIT_COMMIT=unknown
ENV GIT_COMMIT=$GIT_COMMIT
COPY package.json package-lock.json ./
#COPY extras/ ./extras/
RUN npm ci
COPY . .
RUN npm run build
FROM alpine AS export
COPY --from=build /app/dist /dist
VOLUME /output
CMD ["sh", "-c", "rm -rf /output/* && cp -a /dist/. /output/"]

17
deploy/prod/Dockerfile.wiki
View file

@ -1,17 +0,0 @@
# Build-only image for the project wiki (mkdocs).
# It is never run as a service: ansible builds this image once, runs it
# with the host output directory bind-mounted at /output, the container
# copies the built static site into it, and exits. Nginx on the host
# then serves that directory directly, the same way it does the frontend.
FROM python:3.11-slim AS build
WORKDIR /wiki
RUN pip install --no-cache-dir mkdocs
COPY mkdocs.yml ./
COPY docs/ ./docs/
RUN mkdocs build
FROM alpine AS export
COPY --from=build /wiki/site /site
VOLUME /output
CMD ["sh", "-c", "rm -rf /output/* && cp -a /site/. /output/"]

224
deploy/prod/README.md
View file

@ -1,224 +0,0 @@
# Toolshed production deployment — manual steps
`playbook.yml` automates installing docker.io and nginx (plus certbot, and
obtaining/renewing a TLS certificate with it, on hosts that manage their own
— see `behind_tls_proxy` below), building the backend, frontend and wiki
images, exporting the frontend/wiki static builds and the backend's
`collectstatic` output for nginx to serve directly (nginx also serves
user-uploaded files directly, via an X-Accel-Redirect Django issues after its
own permission check — see `location /redirect_media/` in `playbook.yml`),
writing the small `/local/domains` and `/local/dns` fixture files the
frontend fetches directly (registration domain list and DoH resolver
preference — see `toolshed_register_domains`/`toolshed_doh_resolvers` in
`playbook.yml`), configuring nginx, and installing the `toolshed-backend`
systemd service. It does **not** set up the target server or DNS. Those are
manual, one-time steps and are covered here. Seeding the backend's shared
reference data is also a manual, one-time step — see
[First superuser & shared reference data](#5-first-superuser--shared-reference-data).
## 1. Server & firewall
- A Debian/Ubuntu host reachable over SSH.
- Copy `inventory.example.yml` to `inventory.yml` (git-ignored, since it
holds real hostnames/IPs) and fill in your host(s) — see
[Per-deployment configuration](#2-per-deployment-configuration).
- Inbound TCP 80 open in the firewall/security group. Also open 443 unless
`behind_tls_proxy: true` — and keep both open permanently, not just for the
initial deploy: certbot's renewal timer needs 80 for the ACME HTTP-01
challenge and 443 for HTTPS traffic for as long as this host is live.
## 2. Per-deployment configuration
Each entry under `hosts:` in `inventory.yml` is its own independent
deployment (its own repo checkout, database, domain, systemd service and
Django `SECRET_KEY` — nothing is shared between hosts). Set these as
host_vars directly on each host entry, not via `-e` on the command line,
so a single `inventory.yml` can hold several unrelated deployments safely:
```yaml
toolshed:
hosts:
my-server:
ansible_host: 203.0.113.10
ansible_user: deploy
toolshed_domain: toolshed.webdomain.tld
toolshed_handle_domain: yourtoolshed.tld # optional, see below
toolshed_repo_url: git@example.com:your-org/toolshed.git
behind_tls_proxy: false
```
- `toolshed_domain` — the **web domain**: the nginx `server_name`, Django
`ALLOWED_HOSTS`, and the hostname(s) you'll point a TLS cert at — e.g.
`toolshed.webdomain.tld`. Required, no default. May be a single domain (as
above) or a list, e.g. to also answer on a `www.` alias:
```yaml
toolshed_domain:
- toolshed.webdomain.tld
- www.toolshed.webdomain.tld
```
The Let's Encrypt certificate covers all of them, named on disk after
whichever one is listed first. This is not necessarily the same as the
**handle domain** your users log in with (the part after `@` in
`user@yourtoolshed.tld`) — see [DNS](#3-dns) for how those two relate.
- `toolshed_handle_domain` — the **handle domain**, only needed when it's
different from `toolshed_domain`. Omit it when the two are the same (it
then defaults to `toolshed_domain`). Like `toolshed_domain`, it may be a
single domain or a list, e.g. if this deployment accepts registrations for
more than one handle domain. It doesn't affect nginx/Django at all (they
only ever accept `toolshed_domain` as the `Host` header) — it's used
solely to populate the `/local/domains` registration fixture (see
`toolshed_register_domains` in `playbook.yml`); publishing the SRV record for
each handle domain is a separate, manual DNS step either way.
- `toolshed_repo_url` — the git remote the playbook checks out and builds
from. Required, no default.
- `toolshed_version` — the branch, tag or commit to check out and build.
Optional, defaults to `stable`.
- `behind_tls_proxy` — `true` if TLS for this host is already terminated by
something in front of it (e.g. an external reverse proxy or load
balancer) that forwards plain HTTP here; `false` if this nginx has to
terminate TLS itself. This controls two things:
- Whether nginx trusts an upstream `X-Forwarded-Proto` header or sets its
own — get this wrong and Django's `SECURE_PROXY_SSL_HEADER` check
(`backend/backend/settings.py`) will treat every request as insecure or,
flipped the other way, treat plain HTTP as secure.
- Whether the playbook manages TLS at all. When `false`, it automatically
obtains a Let's Encrypt certificate via certbot and switches nginx over
to it — nothing to do manually beyond DNS (below). certbot's own systemd
timer keeps renewing it afterwards, independent of the playbook.
- `toolshed_letsencrypt_email` — required whenever `behind_tls_proxy` is
`false`; the account email certbot registers the certificate under
(used only for renewal-failure notices). Ignored otherwise.
- `http_port` — optional, defaults to `80`. Only relevant when
`behind_tls_proxy: true` and whatever's in front of this host forwards to
a nonstandard port instead of 80.
- `doh_resolvers` — optional, defaults to `["1.1.1.1", "8.8.8.8"]` (the same
hardcoded fallback the frontend itself uses, see `frontend/src/dns.js`).
DNS-over-HTTPS resolvers the frontend uses to look up a handle domain's
`_toolshed-server._tcp` SRV record before it has a cached preference.
Written to `/local/dns` at deploy time; only worth overriding as a
host_var (or `-e doh_resolvers='["9.9.9.9"]'`) if you want this
deployment to prefer a specific resolver.
## 3. DNS
There are two distinct domains at play here, and it's easy to conflate them:
- **Web domain** — the machine's actual hostname: nginx `server_name`,
Django `ALLOWED_HOSTS`, your TLS cert, what's in `toolshed_domain`. This is
what an A/AAAA record has to resolve to the server's IP for.
- **Handle domain** — the part after the `@` in a username, e.g.
`user@yourtoolshed.tld`. Toolshed usernames don't encode a server address
directly; the frontend resolves the handle domain to a server via an SRV
record, `_toolshed-server._tcp.<handle domain>.` (see
`frontend/src/store.js`, `lookupServer`), which always points at the web
domain — nginx/Django never see the handle domain as a `Host` header.
`toolshed_handle_domain` (see [Per-deployment
configuration](#2-per-deployment-configuration)) only feeds the
`/local/domains` registration fixture; publishing the actual SRV record is
still a separate, manual DNS step, covered below.
The SRV lookup happens for every login, not just federation with other
servers, so **every** deployment needs it published for its own handle
domain — even a standalone server that only ever serves itself.
These two domains can be **the same** or **completely different**, and
that's exactly the choice between an A record and an SRV record:
- **Same domain**: if `yourtoolshed.tld` is both the web domain and the
handle domain, it needs both an A record (so the domain itself resolves to
the server) and an SRV record that happens to point back at itself.
- **Different domains**: the handle domain only needs the SRV record — no A
record of its own — pointing at whatever web domain the server actually
lives at. This is useful when the handle you give out (short, brandable,
independent of hosting) shouldn't have to match wherever the box is
actually deployed (a subdomain of a shared hosting provider, an internal
service name, etc.).
**a) A/AAAA record — web domain → server IP:**
```sh
dig <your-web-domain> A
```
**b) SRV record — handle domain → web domain + port.** Use port 443: the
federation protocol is HTTPS-only.
```sh
dig _toolshed-server._tcp.<your-handle-domain> SRV
```
For example, with a handle domain of `yourtoolshed.tld` and a web domain of
`toolshed.webdomain.tld`:
```
$ dig _toolshed-server._tcp.yourtoolshed.tld srv
_toolshed-server._tcp.yourtoolshed.tld. 300 IN SRV 10 10 443 toolshed.webdomain.tld.
$ dig toolshed.webdomain.tld A
toolshed.webdomain.tld. 300 IN A 203.0.113.10
```
If you instead want `yourtoolshed.tld` itself to be the web domain too, its
SRV record just points at itself (`... SRV 10 10 443 yourtoolshed.tld.`) and
it additionally needs its own A record.
## 4. Secrets
`toolshed_secret_key` is generated once per host by the playbook (via the
`password` lookup, keyed by the host's inventory name) and stored as
`.secrets/<inventory-hostname>_secret_key` on the *control* machine, not on
the target. Back these files up — losing one invalidates all sessions and
signed cookies for that deployment on its next redeploy. They're git-ignored
on purpose; never commit them.
## 5. First superuser & shared reference data
The production backend image only runs `migrate` and `collectstatic` at
startup (see `Dockerfile.backend`) — unlike the dev compose setup, it never
runs the interactive `configure.py`. Two things dev gets "for free" from that
script therefore need doing manually, once, after a host's backend container
is first up (run these on the target host itself, or prefix with
`ssh <that-host>`):
- **Superuser account:**
```sh
docker exec -it toolshed-backend python manage.py createsuperuser
```
- **Shared reference data** (the standard categories/properties/tags
shipped in `backend/shared_data/*.json` — tools, electrical, screws, IT,
etc.): without this step a fresh deployment starts with none of them.
Run `configure.py` interactively (the `-it` flags matter — the script's
prompts only appear with a real tty) and answer "yes" when it asks to
import them:
```sh
docker exec -it toolshed-backend python configure.py
```
The other prompts it asks first (create `.env`, create a database) are
harmless to answer "yes" to as well: the container already gets its real
`SECRET_KEY`/`ALLOWED_HOSTS`/db path from the environment (the systemd unit
passes them via `--env-file`, see the "Write backend environment file" task
in `playbook.yml`), those checks just look for files at paths relative to
`/app` that don't exist in this container, and re-running `migrate` against
the real database is idempotent. You can say "no" to the superuser prompt
here if you already created one above.
## 6. Running the playbook
Always target one host at a time with `--limit` — running against the whole
`toolshed` group in one invocation would apply every host's own
`toolshed_domain`/`toolshed_repo_url` correctly (they're per-host vars, see
[Per-deployment configuration](#2-per-deployment-configuration)), but rolls
out all deployments back-to-back in one run, which is rarely what you want:
```sh
ansible-playbook -i inventory.yml playbook.yml --limit my-server
```
Re-run it to roll out a new version to that host. It deploys whatever
`toolshed_version` is set for that host (`stable` by default) — set the
host_var for a persistent change, or pass `-e toolshed_version=<branch/tag/commit>`
for a one-off deploy of something else.

53
deploy/prod/inventory.example.yml
View file

@ -1,53 +0,0 @@
---
# Copy this file to inventory.yml (git-ignored) and fill in your real
# hosts. Each entry under hosts: is an independent deployment - see the
# README's "Per-deployment configuration" section for what each var means.
toolshed:
hosts:
my-server:
ansible_host: 203.0.113.10
ansible_user: deploy
# toolshed_domain is the "web domain" - see the README's DNS section
# for how this relates to the separate "handle domain" your users
# log in with (user@yourtoolshed.tld). May be a single domain (as
# here) or a list, e.g. to also answer on a "www." alias:
# toolshed_domain:
# - toolshed.webdomain.tld
# - www.toolshed.webdomain.tld
# The Let's Encrypt certificate is requested for all of them, named
# after whichever one is listed first.
toolshed_domain: toolshed.webdomain.tld
# Optional - only needed if the handle domain differs from the web
# domain above. Omit it entirely when they're the same. Like
# toolshed_domain, this may be a single domain or a list, e.g. if this
# deployment accepts registrations for more than one handle domain:
# toolshed_handle_domain:
# - yourtoolshed.tld
# - alt.yourtoolshed.tld
toolshed_handle_domain: yourtoolshed.tld
toolshed_repo_url: git@example.com:your-org/toolshed.git
# Optional - branch, tag or commit to deploy. Defaults to "stable".
toolshed_version: stable
# true if something in front of this host already terminates TLS
# (reverse proxy/load balancer), false if this nginx must do it itself.
behind_tls_proxy: false
# Required whenever behind_tls_proxy is false: the playbook obtains
# its own Let's Encrypt certificate via certbot, which needs an
# account email for renewal notices.
toolshed_letsencrypt_email: admin@example.com
# A second, unrelated deployment behind an existing TLS-terminating
# proxy - remove this if you only run one instance. Here the handle
# domain and web domain are the same, so toolshed_handle_domain is
# simply omitted, and toolshed_letsencrypt_email isn't needed since
# this nginx never handles TLS itself.
my-other-server:
ansible_host: my-other-server.example.com
ansible_user: deploy
toolshed_domain: toolshed.example.com
toolshed_repo_url: git@example.com:your-org/toolshed.git
behind_tls_proxy: true
# Only needed if the proxy in front forwards to something other than
# port 80 on this host.
http_port: 8080

677
deploy/prod/playbook.yml
View file

@ -1,677 +0,0 @@
---
# Production deploy for toolshed.
#
# - installs docker.io and nginx on the target (plus certbot, unless
# behind_tls_proxy is true)
# - checks out the source and builds the backend and frontend docker images
# - runs the frontend image once to export its static build, which nginx
# then serves directly (the frontend image is never run as a service)
# - configures nginx (inline template, no separate .conf file) and, unless
# behind_tls_proxy is true, obtains/renews a Let's Encrypt certificate via
# certbot and switches nginx over to it automatically - no manual TLS step
# - installs and manages a systemd service that runs the backend container
#
# Usage (each host is its own independent deployment - always target one
# at a time, never the whole "toolshed" group in one run):
# ansible-playbook -i inventory.yml playbook.yml --limit my-server
#
# toolshed_repo_url, toolshed_domain, toolshed_handle_domain (optional,
# either may be a single domain or a list of domains), toolshed_version
# (optional, defaults to "stable"), behind_tls_proxy and
# toolshed_letsencrypt_email (required unless behind_tls_proxy is true) are
# per-deployment and must be set as host_vars in inventory.yml (copy
# inventory.example.yml) rather than here or via -e, so that each host in
# the "toolshed" group can point at its own repo/domain/branch. They're read
# with `mandatory`/`default()` below instead of being declared in play
# `vars:`, since play vars always take precedence over inventory host_vars
# and would otherwise silently override whatever is set per-host.
- name: Deploy toolshed
hosts: toolshed
become: true
vars:
toolshed_src_dir: /opt/toolshed/src
toolshed_data_dir: /opt/toolshed/data
toolshed_dist_dir: /var/www/toolshed
toolshed_backend_image: toolshed-backend
toolshed_frontend_image: toolshed-frontend-builder
toolshed_wiki_image: toolshed-wiki-builder
toolshed_backend_container: toolshed-backend
toolshed_backend_port: 8000
toolshed_wiki_dist_dir: /var/www/toolshed-wiki
toolshed_local_dir: /var/www/toolshed-local
# The frontend build runs on the controller (see "Build frontend builder
# docker image (controller)" below) rather than the target host, so its
# scratch checkout and build output live here instead of under
# toolshed_src_dir/toolshed_dist_dir. Keyed by inventory_hostname so
# concurrent deploys to different hosts never collide.
toolshed_frontend_build_src_dir: "{{ playbook_dir }}/.frontend-build/{{ inventory_hostname }}/src"
toolshed_frontend_build_dist_dir: "{{ playbook_dir }}/.frontend-build/{{ inventory_hostname }}/dist"
# Django's collectstatic output (admin/drf-yasg assets etc.), exported
# from the built backend image so nginx can serve it directly instead of
# proxying to gunicorn for every asset request.
toolshed_static_dir: /var/www/toolshed-static
# Domain(s) this server accepts registrations for (the "handle domain" -
# see the README's DNS section). toolshed_handle_domain may be a single
# domain or a list; when unset it falls back to toolshed_domain (whole
# list, if that's a list too). Served as a static /local/domains fixture
# that the frontend's registration/pairing forms fetch to populate their
# domain dropdown (frontend/src/views/Register.vue, Pairing.vue) -
# without it that dropdown is just empty.
toolshed_handle_domain_or_default: "{{ toolshed_handle_domain | default(toolshed_domain) }}"
toolshed_register_domains: >-
{{ ([toolshed_handle_domain_or_default]
if toolshed_handle_domain_or_default is string
else toolshed_handle_domain_or_default) | unique }}
# DoH resolvers the frontend falls back to for SRV lookups when it has
# no cached preference yet, served as a static /local/dns fixture. These
# match the frontend's own hardcoded fallback (frontend/src/dns.js), so
# this mostly makes the choice explicit and per-host overridable (e.g.
# -e doh_resolvers='["9.9.9.9"]') rather than changing behavior.
toolshed_doh_resolvers: "{{ doh_resolvers | default(['1.1.1.1', '8.8.8.8']) }}"
# Docker tags can't contain "/", but toolshed_version is a git ref and
# branch names like "jedi/proto/frontend" do - sanitize before using it
# as an image tag. The raw value is still used as-is for the actual git
# checkout, where slashes are fine.
toolshed_image_tag: "{{ (toolshed_version | default('stable')) | replace('/', '-') }}"
toolshed_debug: "False"
# Plain HTTP listen port. Only relevant behind an external proxy that
# forwards to something other than 80 (see http_port in inventory.yml);
# when this nginx terminates TLS itself, the public port is always 443.
toolshed_http_port: "{{ http_port | default(80) }}"
toolshed_letsencrypt_webroot: /var/www/letsencrypt
# Nginx sets its own X-Forwarded-Proto from $scheme when it terminates
# TLS itself. Behind an external TLS-terminating proxy, $scheme at this
# nginx is always "http" (the proxy already stripped TLS one hop
# earlier), so overwriting the header with $scheme would tell Django
# every request is insecure. In that case pass through the proxy's own
# header instead.
toolshed_x_forwarded_proto: >-
{{ '$http_x_forwarded_proto' if (behind_tls_proxy | default(false) | bool) else '$scheme' }}
# Only the web domain (toolshed_domain) - nginx server_name, Django
# ALLOWED_HOSTS, and the cert certbot requests. The handle domain
# (toolshed_handle_domain) is resolved by clients via its own SRV record
# and doesn't necessarily have an A record pointing at this host at all
# (see the README's DNS section), so it can't reliably serve an HTTP-01
# challenge or ever show up as this nginx's Host header.
#
# toolshed_domain may be a single domain or a list (e.g. a bare domain
# plus a "www." alias). certbot names the Let's Encrypt certificate's
# live/ directory after whichever domain is passed first via -d, so
# toolshed_hostnames[0] (below) is used wherever the playbook needs to
# reference that directory by name.
toolshed_domain_checked: >-
{{ toolshed_domain | mandatory('toolshed_domain must be set as a host_var for ' ~ inventory_hostname) }}
toolshed_hostnames: >-
{{ ([toolshed_domain_checked]
if toolshed_domain_checked is string
else toolshed_domain_checked) | unique }}
# Generated once per host on the controller and reused on every
# subsequent run against that host, keyed by inventory_hostname so
# separate deployments never end up sharing a Django SECRET_KEY.
toolshed_secret_key: >-
{{ lookup('ansible.builtin.password',
playbook_dir ~ '/.secrets/' ~ inventory_hostname ~ '_secret_key length=64 chars=ascii_letters,digits') }}
# Rendered twice against the same var (see the tasks below): once before
# a certificate exists (serves the site plainly over toolshed_http_port,
# or over 80/plain-HTTP forever if behind_tls_proxy), and once after
# certbot has obtained one, at which point the plain HTTP vhost switches
# to a redirect and a 443 vhost with the real content appears. Whichever
# of those two states applies, toolshed_cert (a registered `stat` result,
# undefined/false until it's checked) decides which one renders - this
# is the "another nginx config" from a single inline template, driven by
# behind_tls_proxy and certificate state rather than a separate file.
toolshed_nginx_conf: |
upstream toolshed_backend {
server 127.0.0.1:{{ toolshed_backend_port }};
}
{% macro toolshed_locations() %}
location /api {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto {{ toolshed_x_forwarded_proto }};
proxy_pass http://toolshed_backend;
}
location /auth {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto {{ toolshed_x_forwarded_proto }};
proxy_pass http://toolshed_backend;
}
# Django (SignatureAuthentication + per-file friend/owner checks,
# see files/media_urls.py) decides whether the request is allowed
# at all; it never streams the bytes itself here (SERVE_X_ACCEL_REDIRECT
# is on), it just answers with an X-Accel-Redirect to the internal
# location below, which nginx follows and serves directly from disk.
location /media {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto {{ toolshed_x_forwarded_proto }};
proxy_pass http://toolshed_backend;
}
# Only reachable via the X-Accel-Redirect above, never directly by
# clients (`internal`) - this is what makes it safe for nginx to
# serve these bytes itself without reimplementing the access
# checks Django already did in the /media location.
location /redirect_media/ {
internal;
alias {{ toolshed_data_dir }}/userfiles/;
# Django would normally set this itself (CORS_ALLOW_ALL_ORIGINS,
# see settings.py) but never gets to run for a request nginx
# serves directly - see the comment in files/media_urls.py.
add_header Access-Control-Allow-Origin * always;
}
location /djangoadmin {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto {{ toolshed_x_forwarded_proto }};
proxy_pass http://toolshed_backend;
}
location /docs {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto {{ toolshed_x_forwarded_proto }};
proxy_pass http://toolshed_backend;
}
location /static/ {
alias {{ toolshed_static_dir }}/;
try_files $uri =404;
}
location /wiki/ {
alias {{ toolshed_wiki_dist_dir }}/;
try_files $uri $uri/ =404;
}
location = /wiki {
return 301 /wiki/;
}
# Static fixtures the frontend fetches directly (registration
# domain list, DoH resolver preference) - see toolshed_register_domains
# and toolshed_doh_resolvers above.
location /local/ {
alias {{ toolshed_local_dir }}/;
try_files $uri.json =404;
add_header Content-Type application/json;
}
# Vue-router history mode: fall back to index.html for
# any path that isn't a real static file.
location / {
try_files $uri $uri/ /index.html;
}
{% endmacro %}
{% if behind_tls_proxy | default(false) | bool %}
server {
listen {{ toolshed_http_port }};
listen [::]:{{ toolshed_http_port }};
server_name {{ toolshed_hostnames | join(' ') }};
client_max_body_size 128M;
root {{ toolshed_dist_dir }};
index index.html;
{{ toolshed_locations() }}
}
{% else %}
{% set tls_active = toolshed_cert.stat.exists | default(false) %}
server {
listen {{ toolshed_http_port }};
listen [::]:{{ toolshed_http_port }};
server_name {{ toolshed_hostnames | join(' ') }};
location /.well-known/acme-challenge/ {
root {{ toolshed_letsencrypt_webroot }};
}
{% if tls_active %}
location / {
return 301 https://$host$request_uri;
}
{% else %}
client_max_body_size 128M;
root {{ toolshed_dist_dir }};
index index.html;
{{ toolshed_locations() }}
{% endif %}
}
{% if tls_active %}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name {{ toolshed_hostnames | join(' ') }};
ssl_certificate /etc/letsencrypt/live/{{ toolshed_hostnames[0] }}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/{{ toolshed_hostnames[0] }}/privkey.pem;
client_max_body_size 128M;
root {{ toolshed_dist_dir }};
index index.html;
{{ toolshed_locations() }}
}
{% endif %}
{% endif %}
tasks:
- name: Install docker.io, nginx and rsync
ansible.builtin.apt:
name:
- docker.io
- nginx
# rsync is what the frontend dist sync (further down) relies on -
# it's the ansible.posix.synchronize module's transport.
- rsync
state: present
update_cache: true
- name: Install certbot
ansible.builtin.apt:
name: certbot
state: present
when: not (behind_tls_proxy | default(false) | bool)
- name: Ensure docker is running and enabled
ansible.builtin.systemd:
name: docker
state: started
enabled: true
- name: Ensure nginx is running and enabled
ansible.builtin.systemd:
name: nginx
state: started
enabled: true
- name: Checkout toolshed source
ansible.builtin.git:
repo: "{{ toolshed_repo_url | mandatory('toolshed_repo_url must be set as a host_var for ' ~ inventory_hostname) }}"
dest: "{{ toolshed_src_dir }}"
version: "{{ toolshed_version | default('stable') }}"
force: true
# frontend/extras is registered as a submodule but unused and its
# pinned commit isn't fetchable from upstream - don't let a broken
# submodule block the checkout.
recursive: false
register: toolshed_checkout
- name: Create toolshed system user
ansible.builtin.user:
name: toolshed
system: true
shell: /usr/sbin/nologin
home: "{{ toolshed_data_dir }}"
create_home: false
register: toolshed_user
- name: Create backend data directories
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: toolshed
group: toolshed
mode: "0750"
loop:
- "{{ toolshed_data_dir }}"
- "{{ toolshed_data_dir }}/userfiles"
# nginx's `location /redirect_media/` (below) reads user-uploaded files
# straight off disk as www-data - group membership plus the 0750 mode
# above/FILE_UPLOAD_PERMISSIONS (backend/backend/settings.py) is what
# makes that readable without loosening it to world-readable.
- name: Allow nginx to read backend user files
ansible.builtin.user:
name: www-data
groups: toolshed
append: true
# New group membership only takes effect for processes started (or
# forked) after this - nginx's already-running workers won't see it
# until reloaded.
notify: reload nginx
- name: Create frontend static output directory
ansible.builtin.file:
path: "{{ toolshed_dist_dir }}"
state: directory
owner: "www-data"
group: "www-data"
mode: "0750"
- name: Create backend static output directory
ansible.builtin.file:
path: "{{ toolshed_static_dir }}"
state: directory
owner: www-data
group: www-data
mode: "0750"
- name: Write backend environment file
ansible.builtin.copy:
dest: "{{ toolshed_data_dir }}/backend.env"
# Root-owned and unreadable by the toolshed user on purpose: this is
# read by the docker daemon (root) via --env-file at container
# start and injected directly as env vars, so the containerized app
# - which runs as the toolshed user, see the systemd unit below -
# never needs filesystem access to its own SECRET_KEY.
owner: root
group: root
mode: "0600"
content: |
DEBUG={{ toolshed_debug }}
SECRET_KEY={{ toolshed_secret_key }}
ALLOWED_HOSTS={{ toolshed_hostnames | join(',') }}
SERVE_X_ACCEL_REDIRECT=True
TOOLSHED_DB_PATH=/data/db.sqlite3
TOOLSHED_USERFILES_PATH=/data/userfiles
notify: restart backend
- name: Build backend docker image
ansible.builtin.command:
cmd: >-
docker build -t {{ toolshed_backend_image }}:{{ toolshed_image_tag }}
--build-arg GIT_COMMIT={{ toolshed_checkout.after[:7] }}
-f {{ toolshed_src_dir }}/deploy/prod/Dockerfile.backend {{ toolshed_src_dir }}/backend
changed_when: true
notify: restart backend
- name: Tag backend image as latest
ansible.builtin.command:
cmd: docker tag {{ toolshed_backend_image }}:{{ toolshed_image_tag }} {{ toolshed_backend_image }}:latest
changed_when: true
notify: restart backend
# Dockerfile.backend runs collectstatic at build time, baking the result
# into the image at /app/staticfiles - copy it out to the host so nginx
# can serve it directly instead of proxying every asset request to
# gunicorn. No Django settings/DB access needed, so this can run as a
# one-off command against the image rather than the container.
- name: Export backend static files
ansible.builtin.command:
cmd: >-
docker run --rm -v {{ toolshed_static_dir }}:/output
{{ toolshed_backend_image }}:latest
sh -c "cp -a /app/staticfiles/. /output/"
changed_when: true
- name: Fix ownership of exported backend static files
ansible.builtin.file:
path: "{{ toolshed_static_dir }}"
owner: www-data
group: www-data
recurse: true
- name: Install systemd unit for the backend container
ansible.builtin.copy:
dest: /etc/systemd/system/toolshed-backend.service
owner: root
group: root
mode: "0644"
content: |
[Unit]
Description=Toolshed backend (Django) container
After=docker.service network-online.target
Requires=docker.service
Wants=network-online.target
[Service]
TimeoutStartSec=0
Restart=always
ExecStartPre=-/usr/bin/docker stop {{ toolshed_backend_container }}
ExecStartPre=-/usr/bin/docker rm {{ toolshed_backend_container }}
ExecStart=/usr/bin/docker run --rm --name {{ toolshed_backend_container }} \
--user {{ toolshed_user.uid }}:{{ toolshed_user.group }} \
--env-file {{ toolshed_data_dir }}/backend.env \
-v {{ toolshed_data_dir }}:/data \
-p 127.0.0.1:{{ toolshed_backend_port }}:8000 \
{{ toolshed_backend_image }}:latest
ExecStop=/usr/bin/docker stop {{ toolshed_backend_container }}
[Install]
WantedBy=multi-user.target
notify: restart backend
# Installed before the bootstrap nginx flush_handlers below (which
# flushes every pending handler, not just reload nginx) - otherwise a
# fresh host would flush "restart backend" before this unit file exists
# and fail with "Could not find the requested service".
- name: Ensure toolshed-backend service is enabled and started
ansible.builtin.systemd:
name: toolshed-backend
daemon_reload: true
enabled: true
state: started
# The next few tasks build the frontend on the controller instead of the
# target host: `npm run build` pulls in bootstrap+jquery+vue+moment+
# js-nacl+qrcode, and esbuild's rendering/minification pass for that
# bundle needs more memory than small/memory-constrained target hosts
# (e.g. LXC containers without usable swap) reliably have. Only the
# resulting static dist/ is shipped to the target - the docker image
# itself never runs there. This assumes docker is already usable on the
# controller (not managed by this playbook, since "Install docker.io,
# nginx and rsync" above targets the remote host only).
- name: Checkout toolshed source (controller, for frontend build)
ansible.builtin.git:
repo: "{{ toolshed_repo_url | mandatory('toolshed_repo_url must be set as a host_var for ' ~ inventory_hostname) }}"
dest: "{{ toolshed_frontend_build_src_dir }}"
version: "{{ toolshed_version | default('stable') }}"
force: true
recursive: false
register: toolshed_frontend_checkout
delegate_to: localhost
become: false
- name: Build frontend builder docker image (controller)
ansible.builtin.command:
cmd: >-
docker build -t {{ toolshed_frontend_image }}:{{ toolshed_image_tag }}
--build-arg GIT_COMMIT={{ toolshed_frontend_checkout.after[:7] }}
-f {{ toolshed_frontend_build_src_dir }}/deploy/prod/Dockerfile.frontend {{ toolshed_frontend_build_src_dir }}/frontend
changed_when: true
delegate_to: localhost
become: false
- name: Create local frontend dist scratch directory (controller)
ansible.builtin.file:
path: "{{ toolshed_frontend_build_dist_dir }}"
state: directory
mode: "0755"
delegate_to: localhost
become: false
- name: Run frontend builder once to export the static build (controller)
ansible.builtin.command:
cmd: docker run --rm -v {{ toolshed_frontend_build_dist_dir }}:/output {{ toolshed_frontend_image }}:{{ toolshed_image_tag }}
changed_when: true
delegate_to: localhost
become: false
- name: Sync built frontend dist to the target host
ansible.posix.synchronize:
src: "{{ toolshed_frontend_build_dist_dir }}/"
dest: "{{ toolshed_dist_dir }}/"
delete: true
delegate_to: localhost
become: false
- name: Fix ownership of exported frontend build
ansible.builtin.file:
path: "{{ toolshed_dist_dir }}"
owner: www-data
group: www-data
recurse: true
- name: Create wiki static output directory
ansible.builtin.file:
path: "{{ toolshed_wiki_dist_dir }}"
state: directory
owner: www-data
group: www-data
mode: "0755"
- name: Build wiki builder docker image
ansible.builtin.command:
cmd: >-
docker build -t {{ toolshed_wiki_image }}:{{ toolshed_image_tag }}
-f {{ toolshed_src_dir }}/deploy/prod/Dockerfile.wiki {{ toolshed_src_dir }}
changed_when: true
- name: Run wiki builder once to export the static site
ansible.builtin.command:
cmd: docker run --rm -v {{ toolshed_wiki_dist_dir }}:/output {{ toolshed_wiki_image }}:{{ toolshed_image_tag }}
changed_when: true
- name: Fix ownership of exported wiki build
ansible.builtin.file:
path: "{{ toolshed_wiki_dist_dir }}"
owner: www-data
group: www-data
recurse: true
- name: Create local fixtures directory
ansible.builtin.file:
path: "{{ toolshed_local_dir }}"
state: directory
owner: www-data
group: www-data
mode: "0755"
- name: Write registration domain list fixture
ansible.builtin.copy:
dest: "{{ toolshed_local_dir }}/domains.json"
owner: www-data
group: www-data
mode: "0644"
content: "{{ toolshed_register_domains | to_nice_json }}"
- name: Write DoH resolver fixture
ansible.builtin.copy:
dest: "{{ toolshed_local_dir }}/dns.json"
owner: www-data
group: www-data
mode: "0644"
content: "{{ toolshed_doh_resolvers | to_nice_json }}"
- name: Create ACME HTTP-01 challenge webroot
ansible.builtin.file:
path: "{{ toolshed_letsencrypt_webroot }}"
state: directory
owner: www-data
group: www-data
mode: "0755"
when: not (behind_tls_proxy | default(false) | bool)
- name: Check for an existing Let's Encrypt certificate
ansible.builtin.stat:
path: "/etc/letsencrypt/live/{{ toolshed_hostnames[0] }}/fullchain.pem"
register: toolshed_cert
when: not (behind_tls_proxy | default(false) | bool)
- name: Configure nginx site for toolshed (bootstrap)
ansible.builtin.copy:
dest: /etc/nginx/sites-available/toolshed.conf
owner: root
group: root
mode: "0644"
content: "{{ toolshed_nginx_conf }}"
notify: reload nginx
- name: Remove default nginx site
ansible.builtin.file:
path: /etc/nginx/sites-enabled/default
state: absent
notify: reload nginx
- name: Enable toolshed nginx site
ansible.builtin.file:
src: /etc/nginx/sites-available/toolshed.conf
dest: /etc/nginx/sites-enabled/toolshed.conf
state: link
notify: reload nginx
# Certbot's webroot check (below) needs nginx already serving the
# bootstrap config from the tasks above, so force the reload now
# instead of waiting for the end of the play.
- name: Apply the bootstrap nginx config now
ansible.builtin.meta: flush_handlers
- name: Ensure the certbot renewal deploy-hook directory exists
ansible.builtin.file:
path: /etc/letsencrypt/renewal-hooks/deploy
state: directory
mode: "0755"
when: not (behind_tls_proxy | default(false) | bool)
- name: Reload nginx after certbot renews a certificate
ansible.builtin.copy:
dest: /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
owner: root
group: root
mode: "0755"
content: |
#!/bin/sh
systemctl reload nginx
when: not (behind_tls_proxy | default(false) | bool)
- name: Obtain or renew the Let's Encrypt certificate
ansible.builtin.command:
cmd: >-
certbot certonly --webroot -w {{ toolshed_letsencrypt_webroot }}
-d {{ toolshed_hostnames | join(' -d ') }}
--non-interactive --agree-tos
-m {{ toolshed_letsencrypt_email | mandatory('toolshed_letsencrypt_email must be set as a host_var for ' ~ inventory_hostname ~ ' since behind_tls_proxy is false there') }}
register: toolshed_certbot
changed_when: "'Certificate not yet due for renewal' not in toolshed_certbot.stdout"
when: not (behind_tls_proxy | default(false) | bool)
- name: Re-check the certificate now that certbot has run
ansible.builtin.stat:
path: "/etc/letsencrypt/live/{{ toolshed_hostnames[0] }}/fullchain.pem"
register: toolshed_cert
when: not (behind_tls_proxy | default(false) | bool)
- name: Configure nginx site for toolshed (final)
ansible.builtin.copy:
dest: /etc/nginx/sites-available/toolshed.conf
owner: root
group: root
mode: "0644"
content: "{{ toolshed_nginx_conf }}"
notify: reload nginx
handlers:
- name: validate nginx config
ansible.builtin.command: nginx -t
listen: reload nginx
changed_when: false
- name: reload nginx
ansible.builtin.systemd:
name: nginx
state: reloaded
listen: reload nginx
- name: restart backend
ansible.builtin.systemd:
name: toolshed-backend
daemon_reload: true
state: restarted
listen: restart backend

17
docs/development.md
View file

@ -97,22 +97,9 @@ Start the fullstack application:
docker-compose -f deploy/docker-compose.override.yml up --build docker-compose -f deploy/docker-compose.override.yml up --build
``` ```
Run backend tests in Docker:
``` bash
docker compose -f deploy/docker-compose.override.yml run --rm backend-a bash -lc "python configure.py && python manage.py test"
```
This will start an instance of the frontend and wiki, a limited DoH (DNS over HTTPS) server and **two** instances of the backend. This will start an instance of the frontend and wiki, a limited DoH (DNS over HTTPS) server and **two** instances of the backend.
The two backend instances are set up to use the domains `a.localhost` and `b.localhost`, the local DoH The two backend instances are set up to use the domains `a.localhost` and `b.localhost`, the local DoH
server is used to direct the frontend to the correct backend instance. server is used to direct the frontend to the correct backend instance.
The frontend is configured to act as if it was served from the domain `a.localhost`. The frontend is configured to act as if it was served from the domain `a.localhost`.
Access the frontend at `http://localhost:8080/`, backend at `http://localhost:8080/api/v1/`, api docs Access the frontend at `http://localhost:8080/`, backend at `http://localhost:8080/api/`, api docs
at `http://localhost:8080/docs/` and the wiki at `http://localhost:8080/wiki/`. at `http://localhost:8080/docs/` and the wiki at `http://localhost:8080/wiki/`.
The dev proxies terminate TLS with `frontend/.local/localhost.crt`, signed by the dev-only CA at
`frontend/.local/RootCA.crt` (regenerate both with `frontend/.local/make_ca.sh` and
`make_localhost.sh` if they've expired). Import `RootCA.crt` into your OS/browser trust store once;
without that, requests the frontend makes itself (federation calls, the DoH lookup) will fail with
opaque network errors even though clicking through the browser's own cert warning for the page you
navigated to by hand looked fine.

95
docs/federation.md
View file

@ -20,97 +20,4 @@ uses it to verify access to the friend's inventory. While accepting a friend req
their own public key to the friend's server. This way both users can access each other's inventory. their own public key to the friend's server. This way both users can access each other's inventory.
The protocol is based on a simple HTTPS API exchanging JSON data that is signed with the user's private key. By default The protocol is based on a simple HTTPS API exchanging JSON data that is signed with the user's private key. By default
Toolshed servers provide a documentation of the API at [/docs/api](/docs/api). Toolshed servers provide a documentation of the API at [/docs/api](/docs/api).
## Unique Handles
Federation only works if every server can talk about the same thing without a central authority to ask. Toolshed's
answer is that every kind of entity that needs to be referenced across servers gets a handle: a name that is unique
within its own scope and that carries, as part of itself, enough information to say where it is authoritative. This
keeps servers independent of each other while still letting them agree on what they're talking about.
### Users (and Groups)
A user's handle is their username paired with the domain their account belongs to, written the way an email address
is, e.g. `user@toolsheddomain.tld`. Uniqueness is only required within a single domain, not across all of Toolshed,
so two different domains can each have their own "alice" without conflict, the same way two different email
providers can each have an "alice" mailbox. The domain half of the handle is what makes the name globally
unambiguous, and it is also what tells any other backend where to look to find out who's currently authoritative for
that identity, i.e. which backend holds the account and can vouch for its public key.
Groups aren't implemented yet, but they're intended to fit the same idea: a group would get its own handle on the
domain of the server that hosts it, the same way a user does, so that group membership and group-owned data could be
referenced by other servers without needing a separate mechanism. A group handle is written with a leading `#`, e.g.
`#groupname@toolsheddomain.tld`, so that group and user handles occupy visibly distinct spaces on the same domain
and a name can't be squatted as one to collide with the other. See [groups.md](design-in-progress/groups.md) for
details.
### Servers
The domain half of a handle, e.g. `toolsheddomain.tld`, is an authority record, not a location. Owning a domain
just means being able to say which backend is currently authoritative for handles under it; it says nothing about
where that backend is hosted, who operates it, or how many other domains it might also be authoritative for. A
single backend can just as easily host entities for one domain or for many unrelated ones at once, there's no
assumption anywhere in the model that a domain and a backend are the same thing, or that the relationship is one to
one.
The frontend application is a third, separate thing again. The app a user loads isn't necessarily served by, or
even related to, the backend that ends up handling their requests: when given a handle, the frontend looks up which
backend is currently authoritative for that handle's domain and talks to that backend directly from then on. So
using the frontend at one domain to log into a backend authoritative for a completely different domain isn't a
special case, it's the normal path, since "where the app was loaded from" and "which backend answers for a given
handle" were never the same question to begin with. Servers, in the cryptographic sense described below, don't have
an identity of their own beyond the handles they're currently authoritative for; a backend is, conceptually, just
wherever a given domain's handles happen to resolve to right now.
### Tags, Properties, and Categories
Inventory items aren't just described in free text, they can be classified with tags, properties, and categories,
and those get handles too, written as an origin followed by the kind and name, e.g. `origin#tag:drill` or
`origin#category:power-tools`. This lets the same short name (e.g. a "drill" tag) exist independently under
different origins without colliding, while a handle as a whole unambiguously says which taxonomy an entry belongs
to.
An origin isn't necessarily a server; it's whatever the classification is considered to have come from, which could
be a shared, canonical reference dataset that multiple servers import and reuse, just as easily as it could be a
server's own locally-invented taxonomy. This lets independently-run servers converge on a shared vocabulary where it
matters, without forcing every server to invent its own from scratch or requiring a central body to define one.
Handles are resolved strictly: a reference to an origin or entity a server doesn't know about is left unresolved
rather than being guessed at or silently merged into something that looks similar. This mirrors the rest of
Toolshed's federation philosophy, nothing is combined across servers implicitly; agreement always has to be
traceable to an explicit, shared handle.
### Items
Inventory items don't get a handle of their own the way tags or categories do, because they don't need one: every
item belongs to exactly one user, so a simple local identifier is already enough to tell two items apart within that
user's inventory. Combined with the owner's user handle, that local identifier is automatically unique across all of
Toolshed too, since no two users share a handle. Unlike a tag or category, an item isn't meant to be the same entity
reused across servers, it describes something one specific person actually owns, so there's no shared-origin concept
to design for here, ownership alone already provides the scope.
## Cryptography
Handles say who or what is being referred to; cryptography is what lets a server trust that the entity behind a
handle really is who it claims to be, without needing to ask a central authority. Every user handle has exactly one
asymmetric keypair backing it: a private key that never leaves the user's control, and a public key that gets handed
out freely as part of establishing that handle elsewhere.
A server first learns a public key at the moment it has reason to trust it: for its own users, that's registration;
for a friend's handle, that's the friend-request/accept exchange described above. From then on, a public key is
permanently paired with the handle it arrived with, never with a server. This is why friending is really a
key-exchange ceremony rather than just a social action, accepting a request is the moment a server starts trusting a
new handle's signature.
Every request made on a user's behalf is signed with that user's private key, and whichever server receives it
verifies the signature against the public key it holds for that handle. This is what makes it safe for a request to
travel to a server that isn't the user's home server: the receiving server doesn't need to trust the network path or
the sender, only the signature.
It's worth being explicit about what this layer of cryptography is for and what it isn't. Signing establishes
authenticity and integrity, that a request genuinely came from the handle it claims to, unaltered, not
confidentiality. The data itself isn't encrypted by the protocol; keeping it private in transit is what the
underlying HTTPS layer is for. Only user handles carry a keypair; tags, categories, properties, and item handles are
just names, their trustworthiness comes entirely from being reachable only through a signed request from the user
handle that owns or created them, not from any cryptographic identity of their own.

109
docs/glossary-todo.md
View file

@ -1,109 +0,0 @@
# Glossary terminology TODO
Working list from a repo-wide audit of where code/docs use a different word for a concept that
[glossary.md](glossary.md) already gives a canonical name. Nothing here has been changed yet, this
is a collection point before any renaming/edit work starts. Grouped by glossary term; only real
inconsistencies are listed, not every correct usage that was checked and cleared.
## Backend
- `frontend/src/federation.js` — the whole module (`class ServerSet`, `add(server)`, every
request method) talks about "server" throughout where the concept is a **Backend**.
- `frontend/src/store.js` — `getHomeServers`, `getFriendServers`, `getAllKnownServers`,
`setAllFriendsServers`/`all_friends_servers`, `home_servers`, `lookupServer` (~line 313, 325,
332, 359, 76-77, 107-108, 334).
- `frontend/src/views/Friends.vue:16,43` — user-visible table column labeled "Server".
- `docs/design-in-progress/tags.md` (lines 10, 20, 45, 76-78, 117, 204, 228-229) — "server" used
throughout for what the glossary calls Backend (same looseness federation.md already has, but
worth normalizing here too since tags.md is in active editing).
- `issues.md:58` — "federated home servers" / "user identity" conflation.
- `deploy/dev/docker-compose.yml` (`instance_a`/`instance_b`) and `docs/development.md:106-108` —
"instance"/"backend instance" as a near-synonym for Backend.
- `cli-client/toolshed-client.py:11-70`, `README.md:92` — `--host`/`self.host` for "which backend
to talk to".
## Discovery
- `frontend/src/store.js:313` — `lookupServer` action *is* the discovery operation, never named
"discovery".
- `frontend/src/store.js:347` — `could not resolve server for friend` — "resolve" used instead.
- `docs/design-in-progress/items-labels.md:99`, `docs/development.md:108` — describe the
discovery operation via "resolves"/"direct the frontend to the correct backend" without naming
it (minor, but candidates for a one-word tightening).
## Handle / User handle
- `frontend/src/store.js` — several action params destructured as `{username}` that actually carry
a full handle: `lookupServer` (313), `getFriendServers` (359), `fetchFriendProfile` (401-405),
`login` (276-282).
- `frontend/src/views/Login.vue` (lines 24, 27-28, 82-83, 102-105, 115-117) — form label/variable
"Username" for a field that must be a full user handle (`user@domain`, per its own validation
message at line 103).
- `frontend/src/router.js:51` — route param `/inventory/shared/:user/:id` uses `:user` for what's
meant to eventually be a full handle; contrast with the sibling route at line 61 which already
correctly uses `:handle`. (Already called out by items-labels.md itself, so low-risk to leave
as-is, but listed for completeness.)
## Availability policy, Friend/Friendship, Signature/Signing, Strict resolution, Actor, Targeted sharing
No real inconsistencies found — implemented code already uses the glossary's own terms
consistently (`availability_policy` field name throughout backend+frontend; `friend`/`befriend`
consistently; `Signature`/`sign`/`verify` consistently; `_HandleNotFound`/`_resolve_handle` in
`backend/toolshed/offlinedata.py` implement strict resolution faithfully without needing to name
it; Actor and Targeted sharing are unimplemented with no competing name anywhere).
- Checked and cleared, not a real conflict: `frontend/src/neigbors.js`'s `NeighborsCache`/
"neighbor" vocabulary — refers to unreachable backend *domains* during discovery, not to
friendship, despite reading like a synonym at a glance.
## Keypair / Private key / Public key
- Wire-format drift on the one field that actually crosses the network: `befriender_key` is used
for a public key at `frontend/src/store.js:435,449` and `backend/toolshed/api/friend.py:107`,
while the model field, serializer field, and UI all call the same value
`befriender_public_key`/`public_key` (`backend/authentication/models.py:144`,
`backend/toolshed/serializers.py:65`, `backend/toolshed/api/friend.py:118`,
`frontend/src/views/Friends.vue:81`).
- `cli-client/toolshed-client.py` (`--key`, `TOOLSHED_KEY`, `self.signing_key`, ~lines 12, 52, 57)
and `README.md:92` — never say "private key," just "key"/"Toolshed key", even though it's
specifically the private half.
## Origin
- `backend/configure.py:130` ("Identifier set {} already imported, skipping") and the model
`ImportedIdentifierSets` (`backend/hostadmin/models.py:13-19`) — call an imported origin dataset
an "identifier set".
- `issues.md:200-202` — Instance Admin TODO list: "identifier-sets" for **Origin** (matches
`ImportedIdentifierSets`/`configure.py`: one identifier set per `shared_data/*.json` file, named
`git:<file>`, i.e. exactly an Origin) and bare "identifiers" for **Classifier** (not Classification
handle — a classification handle is the `origin#type:name` pointer, not the entity it names).
## Alias
- `backend/shared_data/ee_packages.json:40,64` — two tags already carry an `"alias"` field in real
data (e.g. `SOT54` → `alias: "TO-92"`), but shaped as a bare name string, not an
`origin#type:name` handle pointer, and with no unit-conversion concept. It's silently dropped on
import today (`Tag`/`TagSerializer` have no `alias` field). Not a different-word issue, but the
design doc (which calls Alias "Proposed") doesn't acknowledge this pre-existing, inert
precedent — worth reconciling either the data or the doc.
## Tag / Property / Category
- `frontend/src/components/workflow/workflows/BulkItemImportWorkflow.vue:495` — CSV
column-auto-mapping heuristic treats `"type"` as a synonym for Category:
`lowerColumn.includes('category') || lowerColumn.includes('type')`.
## Item Label
- `frontend/src/components/workflow/workflows/FotoFirstBulkImportWorkflow.vue:567-572,822` and
`FotoFirstBulkImportWorkflow2.vue:593-598,847` — checkbox "Generate QR codes for items" /
`importOptions.generate_qr_codes` names exactly the Item Label concept but never uses that term
(and the option is currently unwired — declared and defaulted `true` but never read elsewhere).
## Item URL / Local id / Domain / Frontend / Definition fingerprint / Fragmentation / Handle collision / Classification handle
No real inconsistencies found — each already uses consistent, glossary-matching vocabulary
(`id`/`item_id` for Local id; `origin` kept cleanly separate from `domain` everywhere it's used;
`get_handle()` consistently for Classification handle; no competing names found anywhere for
Fragmentation, Handle collision, or Definition fingerprint, which also doesn't collide with the
unrelated `File.hash` content-hash field despite both being called "hash").

Some files were not shown because too many files have changed in this diff Show more