build: the themes ship compiled, libsass is no longer installed

A theme change compiled Bootswatch with libsass (34 MB, upstream
deprecated) into two tracked files, which updates, rollbacks and new
Docker containers then reset. The 27 themes now ship as
static/css/themes/<theme>.min.css, built by "manage.py build_themes";
the Settings page only saves the theme name and every page links that
file (zephyr when no settings are in context, as on the 500 page).

libsass moves to dev/requirements.txt; a test checks that each shipped
file is its theme compiled from dev/scss. The "Theme SASS Path" setting
is removed, and the theme default "flaty", a typo that never applied,
becomes zephyr, the theme those installs showed. The update and
rollback steps restore an older version's rewritten theme files only
where they are modified.

.github/workflows/test.yml

@@ -154,12 +154,12 @@ jobs:
           done
 
       - name: Import Compiled Dependencies
-        run: docker exec wvc /srv/webvirtcloud/venv/bin/python3 -c "import libvirt, ldap, lxml.etree, psycopg, sass"
+        run: docker exec wvc /srv/webvirtcloud/venv/bin/python3 -c "import libvirt, ldap, lxml.etree, psycopg"
 
       # Static URLs carry no version: browsers must revalidate them
       - name: Static Files Revalidated
         run: |
-          url=http://127.0.0.1:8080/static/css/wvc-main.min.css
+          url=http://127.0.0.1:8080/static/css/themes/zephyr.min.css
           header() { tr -d '\r' | awk -v h="$1" 'tolower($0) ~ "^" h ":" {sub(/^[^:]*: */, ""); print}'; }
           h=$(curl -s -m 5 -D - -o /dev/null "$url")
           [ "$(echo "$h" | header cache-control)" = no-cache ] || { echo "$h"; exit 1; }
@@ -168,7 +168,7 @@ jobs:
           h=$(curl -s -m 5 -D - -o /dev/null -H "If-None-Match: $etag" "$url")
           echo "$h" | head -1 | grep -q " 304" || { echo "$h"; exit 1; }
           [ "$(echo "$h" | header cache-control)" = no-cache ]
-          docker exec wvc sh -c 'echo "/* changed */" >> /srv/webvirtcloud/staticfiles/css/wvc-main.min.css'
+          docker exec wvc sh -c 'echo "/* changed */" >> /srv/webvirtcloud/staticfiles/css/themes/zephyr.min.css'
           h=$(curl -s -m 5 -D - -o /dev/null -H "If-None-Match: $etag" "$url")
           echo "$h" | head -1 | grep -q " 200" || { echo "$h"; exit 1; }
           new=$(echo "$h" | header etag)

README.md

@@ -457,7 +457,7 @@ Move the `environment=` values of that config into `/etc/default/webvirtcloud` (
    { [ ! -f /etc/default/webvirtcloud ] || cp -p /etc/default/webvirtcloud "$b"/; } &&
    git rev-parse HEAD > "$b/commit" &&
    git ls-files -z --others --exclude-standard -- '*/migrations/*.py' | tar --null -T - --remove-files -cf "$b/local-migrations.tar" &&
-   git checkout -- static/css/wvc-main.min.css dev/scss/wvc-main.scss &&
+   git checkout -- $(git ls-files -m -- static/css dev/scss) &&
    git checkout master && git pull
    ```
 
@@ -537,7 +537,7 @@ b=/root/webvirtcloud-backup-YYYYMMDD-HHMMSS
 systemctl stop webvirt-cloud webvirt-novncd webvirt-socketiod webvirt-worker
 cd /srv/webvirtcloud &&
 ! systemctl is-active --quiet webvirt-cloud webvirt-novncd webvirt-socketiod webvirt-worker &&
-git checkout -- static/css/wvc-main.min.css dev/scss/wvc-main.scss &&
+git checkout -- $(git ls-files -m -- static/css dev/scss) &&
 git checkout "$(cat "$b/commit")" &&
 tar -xf "$b/local-migrations.tar" &&
 venv/bin/python3 -c "import os,sqlite3,sys; [os.remove(p) for p in ('db.sqlite3-journal','db.sqlite3-wal','db.sqlite3-shm') if os.path.exists(p)]; sqlite3.connect(sys.argv[1]).backup(sqlite3.connect('db.sqlite3'))" "$b/db.sqlite3" &&
@@ -589,7 +589,7 @@ A version from before v3.0.0 has no worker: `systemctl disable --now webvirt-wor
 - A running VM without a guest agent shows the IPv4 address the host learned by ARP (the lookup always failed before).
 - After a migration the VM's NVRAM file is deleted on the source, when it is in libvirt's `/var/lib/libvirt/qemu/nvram` and the destination does not share that directory (a VM created there later with the same name would start with those UEFI variables).
 - gstfsd finds the guest's operating system the way libguestfs inspects it, so a root on LVM works, and it always answers (an error when it finds no or several operating systems). Update `/usr/local/bin/gstfsd` on every compute (see [Adding a Compute Node](#adding-a-compute-node), step 3).
-- A theme picked on the Settings page rewrites two tracked files; the update and rollback steps put back the shipped theme. Pick yours again afterwards, also after Docker recreates the container.
+- The themes ship compiled in `static/css/themes`; picking one on the Settings page only saves its name, so updates, rollbacks and recreated Docker containers keep it, and the "Theme SASS Path" setting is gone. libsass is no longer installed; `manage.py build_themes` recompiles the themes in a development setup.
 - With the bundled nginx config, browsers revalidate static files on each load (304 when unchanged), so an update or a theme change shows without a hard reload. Files cached before this update can stay until a hard reload (Ctrl+F5) of each affected page, or until the site's browser cache is cleared once; an external caching proxy may need a purge.
 - External snapshots use libvirt's snapshot API and need libvirt 9.9 or newer on the compute hosts. They work on UEFI VMs and running, paused or shut-off VMs, can save memory too or, on a running VM with a connected QEMU guest agent, freeze the guest's file systems (the default there), and a revert keeps disks added later (it detaches them). A snapshot is refused when a storage pool with the VM's disks has less free space than a share of those disks (10%, at least 1 GiB; both on the Settings page) plus, for the pool of the memory file, the VM's memory; this is checked before the snapshot only, nothing is reserved. A VM's snapshots form one chain: after a revert, delete the newer snapshots before taking another. Snapshot names no longer get an `s1.` prefix; existing ones keep theirs and can still be reverted and deleted, except that a snapshot an older version took of a UEFI VM can only be deleted. On NFS, libvirt removes merged snapshot files as the qemu user, who needs write access to the image directory; files it could not remove are named in a warning after the delete or revert. Deleting a VM with its data removes its snapshot files too, except a template it grew from and files another VM uses.
 - A storage volume a VM still needs (a disk, a backing file or a snapshot's file) is no longer deleted from the storage page or the API; the delete is refused with the reason. Only VMs on the same compute are checked.

appsettings/management/commands/build_themes.py

@@ -0,0 +1,27 @@
+from pathlib import Path
+
+import sass
+from django.conf import settings
+from django.core.management.base import BaseCommand
+
+from appsettings.views import THEMES_DIR
+
+SCSS = Path(settings.BASE_DIR, "dev", "scss")
+
+
+def compile_theme(theme):
+    return sass.compile(
+        string=f"@import '{SCSS}/wvc-themes/{theme}/variables';\n"
+        f"@import '{SCSS}/bootstrap-overrides.scss';\n"
+        f"@import '{SCSS}/wvc-themes/{theme}/bootswatch';",
+        output_style="compressed",
+    )
+
+
+class Command(BaseCommand):
+    help = "Compile the themes in dev/scss/wvc-themes into static/css/themes (needs libsass, dev/requirements.txt)."
+
+    def handle(self, *args, **options):
+        THEMES_DIR.mkdir(exist_ok=True)
+        for theme in sorted(p.name for p in (SCSS / "wvc-themes").iterdir() if p.is_dir()):
+            (THEMES_DIR / f"{theme}.min.css").write_text(compile_theme(theme))

appsettings/migrations/0019_prebuilt_themes.py

@@ -0,0 +1,15 @@
+from django.db import migrations
+
+
+def forward(apps, schema_editor):
+    setting = apps.get_model("appsettings", "AppSettings")
+    db_alias = schema_editor.connection.alias
+    setting.objects.using(db_alias).filter(key="SASS_DIR").delete()
+    # the default was a typo that never applied: these installs showed the shipped zephyr
+    setting.objects.using(db_alias).filter(key="BOOTSTRAP_THEME", value="flaty").update(value="zephyr")
+
+
+class Migration(migrations.Migration):
+    dependencies = [("appsettings", "0018_worker_settings")]
+
+    operations = [migrations.RunPython(forward, migrations.RunPython.noop)]

appsettings/templates/appsettings.html

@@ -28,14 +28,6 @@
             </div>
         </form>
         {% if request.user.is_superuser %}
-        <form method="post" action="" role="form" aria-label="Edit sass directory settings form">{% csrf_token %}
-            <div class="row mb-1">
-                <label class="col-sm-3 col-form-label">{% trans sass_dir.name %}</label>
-                <div class="col-sm-6">
-                    <input class="form-control" name="{{ sass_dir.key }}" value="{{ sass_dir.value }}" onchange="this.form.submit()" title="{% trans sass_dir.description %}"/>
-                </div>
-            </div>
-        </form>
         <form method="post" action="" role="form" aria-label="Edit theme settings form">{% csrf_token %}
             <div class="row mb-1">
                 <label class="col-sm-3 col-form-label">{% trans bootstrap_theme.name %}</label>
@@ -45,7 +37,6 @@
                         <option {% if bootstrap_theme.value == theme %}selected{% endif %} value="{{ theme }}">{{ theme }}</option>
                     {% endfor %}
                     </select>
-                    <span class="text-muted">{% trans "After change please full refresh page with 'Ctrl + F5' "%}</span>
                 </div>
             </div>
         </form>

appsettings/tests/test_theme_name.py

@@ -1,99 +1,30 @@
-"""The theme name posted on the settings page must be one of the themes in
-SASS_DIR/wvc-themes: it is written into @import paths and the compiled CSS."""
-
-import os
-import shutil
-import tempfile
-from pathlib import Path
+"""The settings page picks one of the prebuilt themes in static/css/themes;
+every page links the chosen theme's file."""
 
 from appsettings.models import AppSettings
-from django.conf import settings
 from django.contrib.auth.models import User
-from django.contrib.messages import get_messages
-from django.test import TestCase, override_settings
+from django.test import TestCase
 from django.urls import reverse
+from lxml import html
 
 
 class ThemeNameTests(TestCase):
     def setUp(self):
         self.client.force_login(User.objects.create_superuser("th-admin", "th@example.com", "pw"))
-        self.sass = tempfile.TemporaryDirectory()
-        self.addCleanup(self.sass.cleanup)
-        os.makedirs(os.path.join(self.sass.name, "wvc-themes", "flatly"))
-        AppSettings.objects.filter(key="SASS_DIR").update(value=self.sass.name)
         self.before = AppSettings.objects.get(key="BOOTSTRAP_THEME").value
 
+    def theme_link(self, url):
+        (href,) = html.fromstring(self.client.get(url).content).xpath("//link[@id='wvc_css']/@href")
+        return href
+
     def test_a_name_that_is_not_a_theme_is_refused(self):
-        for theme in ("../../etc", "flatly';@import '/etc/passwd", "nope", ""):
+        for theme in ("../../etc", "flatly.min.css", "nope", ""):
             with self.subTest(theme=theme):
                 self.client.post(reverse("appsettings"), {"BOOTSTRAP_THEME": theme})
-                self.assertFalse(os.path.exists(os.path.join(self.sass.name, "wvc-main.scss")))
                 self.assertEqual(AppSettings.objects.get(key="BOOTSTRAP_THEME").value, self.before)
 
-
-class SassDirTests(TestCase):
-    """SASS_DIR must be a directory inside the project: the page lists it and
-    writes wvc-main.scss into it."""
-
-    def setUp(self):
-        self.client.force_login(User.objects.create_superuser("sd-admin", "sd@example.com", "pw"))
-        self.before = AppSettings.objects.get(key="SASS_DIR").value
-
-    def test_a_directory_outside_the_project_is_refused(self):
-        outside = tempfile.TemporaryDirectory()
-        self.addCleanup(outside.cleanup)
-        for value in (outside.name, "/etc", "../", "dev/scss/../../..", "no/such/dir"):
-            with self.subTest(value=value):
-                self.client.post(reverse("appsettings"), {"SASS_DIR": value})
-                self.assertEqual(AppSettings.objects.get(key="SASS_DIR").value, self.before)
-
-    def test_the_project_scss_directory_is_accepted(self):
-        self.client.post(reverse("appsettings"), {"SASS_DIR": "dev/scss"})
-        self.assertEqual(AppSettings.objects.get(key="SASS_DIR").value, "dev/scss")
-
-    def test_a_stored_directory_outside_the_project_is_not_used(self):
-        outside = tempfile.TemporaryDirectory()
-        self.addCleanup(outside.cleanup)
-        os.makedirs(os.path.join(outside.name, "wvc-themes", "flatly"))
-        AppSettings.objects.filter(key="SASS_DIR").update(value=outside.name)
+    def test_pages_link_the_chosen_theme(self):
+        self.assertEqual(self.theme_link(reverse("appsettings")), "/static/css/themes/zephyr.min.css")
         self.client.post(reverse("appsettings"), {"BOOTSTRAP_THEME": "flatly"})
-        self.assertFalse(os.path.exists(os.path.join(outside.name, "wvc-main.scss")))
-
-
-class ThemeCssTests(TestCase):
-    """A theme change also rewrites the collected wvc-main.min.css: nginx
-    serves STATIC_ROOT, not static/."""
-
-    def setUp(self):
-        self.client.force_login(User.objects.create_superuser("tc-admin", "tc@example.com", "pw"))
-        project = tempfile.TemporaryDirectory()
-        self.addCleanup(project.cleanup)
-        self.project = Path(project.name)
-        shutil.copytree(Path(settings.BASE_DIR, "dev", "scss"), self.project / "dev" / "scss")
-        (self.project / "static" / "css").mkdir(parents=True)
-        cwd = os.getcwd()
-        os.chdir(self.project)
-        self.addCleanup(os.chdir, cwd)
-        AppSettings.objects.filter(key="SASS_DIR").update(value="dev/scss")
-
-    def change_theme(self, static_root):
-        with override_settings(BASE_DIR=self.project, STATIC_ROOT=static_root):
-            response = self.client.post(reverse("appsettings"), {"BOOTSTRAP_THEME": "flatly"})
-        self.assertIn("Theme is changed. Now: flatly", [str(m) for m in get_messages(response.wsgi_request)])
         self.assertEqual(AppSettings.objects.get(key="BOOTSTRAP_THEME").value, "flatly")
-        source = (self.project / "static" / "css" / "wvc-main.min.css").read_text()
-        self.assertIn(".btn", source)
-        return source
-
-    def test_the_collected_css_is_the_new_theme(self):
-        served = self.project / "staticfiles" / "css" / "wvc-main.min.css"
-        served.parent.mkdir(parents=True)
-        served.write_text("old theme")
-        self.assertEqual(served.read_text(), "old theme")
-        self.assertEqual(self.change_theme(self.project / "staticfiles"), served.read_text())
-
-    def test_without_collected_css_only_the_source_is_written(self):
-        (self.project / "staticfiles").mkdir()
-        self.change_theme(self.project / "staticfiles")
-        self.assertEqual(list((self.project / "staticfiles").iterdir()), [])
-        self.change_theme(None)
+        self.assertEqual(self.theme_link(reverse("appsettings")), "/static/css/themes/flatly.min.css")

appsettings/tests/test_themes.py

@@ -1,33 +1,17 @@
-"""The themes in dev/scss/wvc-themes compile with the pinned libsass, and the
-shipped wvc-main.min.css is the default theme compiled from these sources."""
+"""static/css/themes holds every theme in dev/scss/wvc-themes, compiled from
+these sources with the pinned libsass (manage.py build_themes)."""
 
-from pathlib import Path
-
-import sass
-from django.conf import settings
 from django.test import SimpleTestCase
 
-SCSS = Path(settings.BASE_DIR, "dev", "scss")
-
-
-def compile_theme(theme):
-    # the imports the settings view compiles
-    return sass.compile(
-        string=f"@import '{SCSS}/wvc-themes/{theme}/variables';\n"
-        f"@import '{SCSS}/bootstrap-overrides.scss';\n"
-        f"@import '{SCSS}/wvc-themes/{theme}/bootswatch';",
-        output_style="compressed",
-    )
+from appsettings.management.commands.build_themes import SCSS, compile_theme
+from appsettings.views import THEMES_DIR
 
 
 class ThemeTests(SimpleTestCase):
-    def test_every_theme_compiles(self):
+    def test_the_shipped_themes_are_compiled_from_these_sources(self):
         themes = sorted(p.name for p in (SCSS / "wvc-themes").iterdir() if p.is_dir())
         self.assertIn("zephyr", themes)
+        self.assertEqual(sorted(p.name for p in THEMES_DIR.iterdir()), [f"{theme}.min.css" for theme in themes])
         for theme in themes:
             with self.subTest(theme=theme):
-                self.assertIn(".btn", compile_theme(theme))
-
-    def test_the_shipped_css_is_zephyr_from_these_sources(self):
-        shipped = Path(settings.BASE_DIR, "static", "css", "wvc-main.min.css").read_text()
-        self.assertEqual(shipped, compile_theme("zephyr"))
+                self.assertEqual((THEMES_DIR / f"{theme}.min.css").read_text(), compile_theme(theme), "run manage.py build_themes")

appsettings/views.py

@@ -1,7 +1,5 @@
-import os
 from pathlib import Path
 
-import sass
 from admin.decorators import superuser_only
 from django.conf import settings
 from django.contrib import messages
@@ -14,11 +12,8 @@ from operations.utils import BOUNDS
 from appsettings.models import AppSettings
 
 
-def sass_dir_in_project(value):
-    """True if the SASS_DIR value is an existing directory inside the project:
-    the page lists its themes and writes wvc-main.scss into it."""
-    path = (Path(settings.BASE_DIR) / value).resolve()
-    return path.is_dir() and path.is_relative_to(Path(settings.BASE_DIR).resolve())
+# the themes "manage.py build_themes" compiles; base.html links the chosen one
+THEMES_DIR = Path(settings.BASE_DIR, "static", "css", "themes")
 
 
 @superuser_only
@@ -27,87 +22,24 @@ def appsettings(request):
     :param request:
     :return:
     """
-    main_css = "wvc-main.min.css"
-    sass_dir = AppSettings.objects.get(key="SASS_DIR")
     bootstrap_theme = AppSettings.objects.get(key="BOOTSTRAP_THEME")
-    themes_list = []
-    if not sass_dir_in_project(sass_dir.value):
-        messages.error(request, _("SASS directory path must be a directory inside the project: %(dir)s") % {"dir": sass_dir.value})
-    else:
-        try:
-            themes_list = os.listdir(sass_dir.value + "/wvc-themes")
-        except FileNotFoundError as err:
-            messages.error(request, err)
-            addlogmsg(request.user.username, "-", "", err)
+    themes_list = sorted(p.name.removesuffix(".min.css") for p in THEMES_DIR.glob("*.min.css"))
 
-    # Bootstrap settings related with filesystems, because of that they are excluded from other settings
+    # the theme has its own form; the other settings are listed one by one
     appsettings = AppSettings.objects.exclude(
         description__startswith="Bootstrap"
     ).order_by("name")
 
     if request.method == "POST":
-        if "SASS_DIR" in request.POST:
-            if not sass_dir_in_project(request.POST.get("SASS_DIR", "")):
-                messages.error(
-                    request,
-                    _("SASS directory path must be a directory inside the project: %(dir)s")
-                    % {"dir": request.POST.get("SASS_DIR", "")},
-                )
-                return HttpResponseRedirect(request.get_full_path())
-            try:
-                sass_dir.value = request.POST.get("SASS_DIR", "")
-                sass_dir.save()
-
-                msg = _("SASS directory path is changed. Now: %(dir)s") % {
-                    "dir": sass_dir.value
-                }
-                messages.success(request, msg)
-            except Exception as err:
-                msg = err
-                messages.error(request, msg)
-
-            addlogmsg(request.user.username, "-", "", msg)
-            return HttpResponseRedirect(request.get_full_path())
-
         if "BOOTSTRAP_THEME" in request.POST:
             theme = request.POST.get("BOOTSTRAP_THEME", "")
-            # the name goes into @import paths: only the themes in SASS_DIR
             if theme not in themes_list:
                 messages.error(request, _("Unknown theme: %(theme)s") % {"theme": theme})
                 return HttpResponseRedirect(request.get_full_path())
-            scss_var = f"@import '{sass_dir.value}/wvc-themes/{theme}/variables';"
-            # scss_boot = f"@import '{sass_dir.value}/bootstrap/bootstrap.scss';"
-            scss_boot = f"@import '{sass_dir.value}/bootstrap-overrides.scss';"
-            scss_bootswatch = (
-                f"@import '{sass_dir.value}/wvc-themes/{theme}/bootswatch';"
-            )
-
-            try:
-                with open(sass_dir.value + "/wvc-main.scss", "w") as main:
-                    main.write(
-                        scss_var + "\n" + scss_boot + "\n" + scss_bootswatch + "\n"
-                    )
-
-                css_compressed = sass.compile(
-                    string=scss_var + "\n" + scss_boot + "\n" + scss_bootswatch,
-                    output_style="compressed",
-                )
-                with open("static/css/" + main_css, "w") as css:
-                    css.write(css_compressed)
-                # nginx serves the collected copy; collectstatic later takes it from static/
-                served = Path(settings.STATIC_ROOT or "", "css")
-                if settings.STATIC_ROOT and served.is_dir():
-                    (served / main_css).write_text(css_compressed)
-
-                bootstrap_theme.value = theme
-                bootstrap_theme.save()
-
-                msg = _("Theme is changed. Now: %(theme)s") % {"theme": theme}
-                messages.success(request, msg)
-            except Exception as err:
-                msg = err
-                messages.error(request, msg)
-
+            bootstrap_theme.value = theme
+            bootstrap_theme.save()
+            msg = _("Theme is changed. Now: %(theme)s") % {"theme": theme}
+            messages.success(request, msg)
             addlogmsg(request.user.username, "-", "", msg)
             return HttpResponseRedirect(request.get_full_path())
 

conf/requirements.txt

@@ -16,7 +16,6 @@ djangorestframework==3.18.3
 drf-nested-routers==0.95.3
 drf-spectacular[sidecar]==0.30.0
 gunicorn==26.2.0
-libsass==0.23.0
 libvirt-python>=9.0.0
 lxml>=6.1
 paramiko==5.0.0

console/templates/console-base.html

@@ -6,7 +6,8 @@
     <meta charset="utf-8">
     <link rel="shortcut icon" href="{% static "favicon.ico" %}">
     <meta name="viewport" content="width=device-width, initial-scale=1.0">
-    <link rel="stylesheet" href="{% static "css/wvc-main.min.css" %}" id="wvc_css">
+    {# error pages render without app_settings: they get zephyr #}
+    {% with theme=app_settings.BOOTSTRAP_THEME|default:"zephyr" %}<link rel="stylesheet" href="{% static "css/themes/"|add:theme|add:".min.css" %}" id="wvc_css">{% endwith %}
     <link href="{% static "css/webvirtcloud.css" %}" rel="stylesheet">
 
     <style>

dev/requirements.txt

@@ -1,5 +1,7 @@
 -r ../conf/requirements.txt
 coverage==7.16.2
+# compiles the themes (manage.py build_themes)
+libsass==0.23.0
 django-debug-toolbar==8.0.0
 django-debug-toolbar-template-profiler==2.1.0
 pip-audit==2.10.1

dev/scss/wvc-main.scss

@@ -1,3 +0,0 @@
-@import 'dev/scss/wvc-themes/zephyr/variables';
-@import 'dev/scss/bootstrap-overrides.scss';
-@import 'dev/scss/wvc-themes/zephyr/bootswatch';

templates/base.html

@@ -12,7 +12,8 @@
     <title>{% block title %}{% endblock %}</title>
 
     <!-- Bootstrap Core CSS -->
-    <link href="{% static "css/wvc-main.min.css" %}" rel="stylesheet" id="wvc_css">
+    {# error pages render without app_settings: they get zephyr #}
+    {% with theme=app_settings.BOOTSTRAP_THEME|default:"zephyr" %}<link href="{% static "css/themes/"|add:theme|add:".min.css" %}" rel="stylesheet" id="wvc_css">{% endwith %}
 
     <!-- WebVirtCloud CSS -->
     <link href="{% static "css/webvirtcloud.css" %}" rel="stylesheet">