Skip to content

DEPLOYMENT.md

Latest commit

 

History

History
345 lines (245 loc) · 15.8 KB

DEPLOYMENT.md

File metadata and controls

345 lines (245 loc) · 15.8 KB
Raw

Deployment Guide

This document covers the Makeability Lab website's production infrastructure, deployment pipeline, and server administration.

Table of Contents

Server Overview

The Makeability Lab website runs on two UW CSE servers:

Server URL Purpose
Test https://makeabilitylab-test.cs.washington.edu Staging environment for testing changes
Production https://makeabilitylab.cs.washington.edu Live public-facing website

Each server has its own:

  • PostgreSQL database
  • File storage backend
  • Log files

Important: Content added to the test server does not affect production, and vice versa. They are completely independent environments.

Deployment Pipeline

Deployments are automated via GitHub webhooks:

graph LR
    A[Push to<br/>master] --> B(Webhook fires)
    B --> C[makeabilitylab-test<br/>auto-deploys]

    D[Push a tag<br/>e.g. 2.1.0] --> E(Webhook fires)
    E --> F[makeabilitylab<br/>production]
Loading

Deploying to Test

Any push to master automatically deploys to the test server:

git push origin master

Deploying to Production

Production deployments require a version tag:

git tag 2.1.0
git push --tags

Verifying Deployment

Confirm the deployment succeeded with /version.json, which reports the running build (the /logs/buildlog.txt URL this section used to point at now 404s — see "Log Files" below):

curl -s https://makeabilitylab-test.cs.washington.edu/version.json | python3 -m json.tool

Match git_sha against the commit you pushed — not built_at, which has read fresh while the host served stale code. The build log itself is only available in the deploy email sent to maintainers on every push.

Versioning

We use Semantic Versioning for production releases.

Change Type Version Component Example
First release Start at 1.0.0 1.0.0
Bug fixes, minor patches Increment PATCH (third digit) 1.0.0 → 1.0.1
New features (backward compatible) Increment MINOR (second digit) 1.0.1 → 1.1.0
Breaking changes Increment MAJOR (first digit) 1.1.0 → 2.0.0

View current and past versions on the Releases page.

Creating a Release

  1. Ensure all changes are merged to master and tested on the test server

  2. Confirm the Python test suite passes locally:

    docker exec makeabilitylabwebsite-website-1 python manage.py test website --settings=makeabilitylab.settings_test

    (See Running the Test Suite in CONTRIBUTING.md for what the suite covers and how to add to it.)

  3. Determine the appropriate version number based on the changes

  4. Create and push the tag:

    git tag 2.1.0
git push --tags

  5. Verify the deploy landed via /version.json — check git_sha, not built_at

Server Configuration

The production server was configured by UW CSE's IT team (Jason Howe).

Configuration File

Django reads database credentials and secret keys from config.ini. This file:

  • Is not stored in Git (for security)
  • Is mounted as a Docker volume on the production server
  • Contains PostgreSQL connection strings and Django secret keys

Environment Variables

Production-specific settings are configured in settings.py using values from config.ini. Local development uses different defaults specified in docker-compose-local-dev.yml.

Static Files vs. Dynamic Requests (Apache routing)

On both servers, Apache sits in front of the Django container. It serves any URL that maps to a real file directly, and only proxies to Django (over plain HTTP) for paths that have no matching file. This has a few non-obvious consequences:

  • /robots.txt is a static file — it is the top-level robots.txt committed in the repo root, served by Apache from the project checkout. To change crawler rules or the advertised sitemap, edit that file and deploy. A Django view/route for /robots.txt would be dead code on the servers (it only runs under local runserver, which diverges from production).
  • /sitemap.xml is dynamic — no such file exists, so Apache proxies it to Django's django.contrib.sitemaps (see website/sitemaps.py), which builds the XML from the database on each request.
  • Django sees requests as HTTP, not HTTPS. Apache terminates TLS and proxies to Django over plain HTTP, so request.scheme is http. Any code that builds absolute URLs from the request (e.g. the sitemap) must force https explicitly — the sitemaps do this via protocol = "https".
  • The test server is never indexed. Apache stamps X-Robots-Tag: noindex, nofollow on every response from the test host, so staging stays out of search engines regardless of its robots.txt. (Production pages carry no such header — verify with curl -sI https://makeabilitylab.cs.washington.edu/ | grep -i x-robots-tag, which should return nothing.)

Search Engine Indexing (Sitemap & Google Search Console)

The production sitemap is dynamically generated from the database (website/sitemaps.py, served at /sitemap.xml) and advertised in the repo-root robots.txt. New people, news items, publications, projects, etc. appear in it automatically — there is nothing to regenerate or re-upload when content changes. (Sitemap/robots work landed in #1252; the related prod-deploy stall it surfaced is #1313.)

Quick health check (anytime — all should be true):

curl -sI https://makeabilitylab.cs.washington.edu/sitemap.xml | head -1   # 200, served by Django (WSGIServer)
curl -s  https://makeabilitylab.cs.washington.edu/robots.txt              # allow-all + a "Sitemap:" line
curl -s  https://makeabilitylab.cs.washington.edu/sitemap.xml | grep -c '<loc>'  # count of URLs (~700+)

The X-Robots-Tag: noindex that the sitemap file returns is intentional and harmless — it keeps the XML out of search results without affecting the URLs listed inside.

Current status: already registered & verified

The production site is already a verified property in Google Search Console (https://makeabilitylab.cs.washington.edu/, URL-prefix), with the sitemap.xml submitted on 2026-06-17 ("Sitemap submitted successfully — Google will periodically process it and look for changes"). Ownership was verified via the site's pre-existing Google Analytics property (the Analytics snippet served on every page), so no verification file lives in the repo. You do not need to re-do any of the steps below under normal operation — see "Ongoing maintenance" for what little there is. The steps are retained only for re-setup (e.g. registering a new property or recovering after the Search Console / Analytics account access is lost).

Setting up from scratch (only if re-registering)

You only do this once per property (not per content change):

  1. Go to Google Search Console → Add property → URL prefix (not Domain — that needs a DNS record we can't add for cs.washington.edu).
  2. Enter https://makeabilitylab.cs.washington.edu/ exactly.
  3. Verify ownership. Easiest if it works: the Google Analytics method (prod already serves an Analytics snippet). Otherwise use the HTML file method — commit Google's google<token>.html to the repo root (Apache serves it statically, exactly like robots.txt) and ship it to prod with a SemVer tag, then click Verify. Leave the verification asset (GA snippet or HTML file) in place permanently — removing it un-verifies the property.
  4. In the left sidebar → Sitemaps → enter sitemap.xml → Submit. Status moves to Success once Google fetches it.

Ongoing maintenance: essentially none

  • Per new person / news item / publication: do nothing. The dynamic sitemap updates itself and Google re-crawls /sitemap.xml on its own schedule (days–weeks).
  • Re-submit only if the sitemap URL changes or you restructure the site's URL scheme.
  • Optional: glance at Search Console's Pages (indexing) report ~quarterly for crawl errors, or use URL Inspection → Request Indexing to fast-track an important new page.

Debugging & Logging

Log Files

Not all logs live in the same place. Only the Django application log is on the shared CSE filesystem, and it is the only one we can still read directly. The build and web-server logs live on the Docker host (grabthar / docker-test2), which we can't SSH into.

The web /logs/ URL is gone. It used to expose these files over HTTP. Every path under it now 404s on both prod and test (verified 2026-07-28): the response comes back as Server: gunicorn with Django's custom 404 template, meaning Apache has no /logs/ alias any more and the request falls through to Django. It is not coming back — web access to logs is no longer needed and UW CSE IT has trouble maintaining it. Use SSH.

Log Description Where to find it
debug.log Django application logs On the shared filesystem — read via SSH (see below). Rotated debug.log.1 … .6 sit alongside it.
buildlog.txt Deployment build output Not on the shared filesystem (so not under www/). It lives on the Docker host and is emailed to maintainers on every push — that email is the only copy you can get.
httpd-access.log HTTP request logs On the Docker host — not reachable; ask UW CSE IT if you need it.
httpd-error.log HTTP error logs On the Docker host — not reachable; ask UW CSE IT if you need it.

Where debug.log is written, and what happens if that fails (#1283)

The path is derived in makeabilitylab/settings.py:

LOG_DIR  = $ML_LOG_DIR, defaulting to <BASE_DIR>/media   # /code/media in the container
LOG_FILE = $LOG_DIR/debug.log                            # /code/media/debug.log

LOG_DIR must stay inside MEDIA_ROOT, because that is the directory bind-mounted out to the shared CSE filesystem — it's what makes debug.log readable over SSH at /cse/web/research/makelab/www/debug.log (prod) and www-test/debug.log (test).

Note that MEDIA_ROOT is a web-served tree, so keep the log conservative: we log at INFO rather than DEBUG whenever DEBUG is off, and nothing personal or sensitive should ever be written to it.

  • ML_LOG_DIR is an optional environment override for hosts that don't use /code. It is not set on prod, test, or local dev, and shouldn't need to be. If you point it outside MEDIA_ROOT, the log stops being bind-mounted to the shared filesystem and you lose SSH access to it — which, with no shell on these servers, means losing all access. Only do that if you also arrange a mount for the new location.
  • If the log directory can't be created or written, Django does not crash (it used to: LOGGING is evaluated at django.setup(), so a bad path killed startup before a single request). The file handler degrades to a NullHandler instead — meaning the server runs fine but writes no logs at all.

Because a degraded state is otherwise invisible (there is no console access on these servers), it is surfaced two ways:

  1. /version.json → "log_to_file": false and "log_file": "<path that failed>".
  2. The /admin/ dashboard → a warning callout, shown to superusers only.

Verifying logging after a deploy

log_to_file: true only means the directory was writable at startup, so check both the flag and that records are actually landing on disk:

HOST=https://makeabilitylab-test.cs.washington.edu   # or the prod host
curl -s $HOST/version.json | python3 -m json.tool
# expect: "log_to_file": true, "log_file": "/code/media/debug.log",
#         and a "git_sha" matching the commit you pushed

Match git_sha — not built_at, which has shown fresh on a stuck auto-deploy serving stale code. Then confirm records are really being written (this has to be SSH; there is no web path to the log):

ssh makelab1                                          # or makelab2 / recycle
ls -l /cse/web/research/makelab/www-test/debug.log     # www/ for prod
tail -5 /cse/web/research/makelab/www-test/debug.log   # timestamps after the deploy

The log rotates at 5 MB, so check debug.log.1 too when hunting a container-start sequence. Rotation is currently unreliable under multiple Gunicorn workers — see issue #1439 — so a rotated file may be much smaller than 5 MB and may be missing records.

Reading debug.log over SSH

This is the only way to read the application log. SSH access is read-mostly: you can read the shared CSE filesystem, but there is no Docker or manage.py access on the hosts that run the stack.

  1. SSH to a host with the shared filesystem mounted:

    ssh makelab1     # or makelab2, or recycle.cs.washington.edu

  2. Navigate to the log directory:

    # Test server
cd /cse/web/research/makelab/www-test

# Production server
cd /cse/web/research/makelab/www

  3. View recent log entries (a rotated debug.log.1 may also be present):

    # Last 100 lines
tail -n 100 debug.log

# Save to file
tail -n 100 debug.log > last100lines.log

# Follow log in real-time
tail -f debug.log

    To pull the log down to your machine for analysis:

    scp jonf@recycle.cs.washington.edu:/cse/web/research/makelab/www/debug.log ~/Downloads/prod-debug.log

Windows Network Drive Access

If you have Windows directory mapping configured, logs are accessible at:

O:\cse\web\research\makelab\www        # Production
O:\cse\web\research\makelab\www-test   # Test

Data Management

Uploaded Files

Files uploaded via the Django admin (publications, talks, images, etc.) are stored in the /media folder:

Server Path
Test /cse/web/research/makelab/www-test/media
Production /cse/web/research/makelab/www/media

To browse uploaded files:

ssh recycle.cs.washington.edu
cd /cse/web/research/makelab/www/media  # or www-test for test server
ls -la

Database Access

The production PostgreSQL database runs on grabthar.cs.washington.edu.

Note: Direct database access is rarely needed. In the uncommon case you need to query PostgreSQL directly, you must connect through recycle.cs.washington.edu:

ssh recycle.cs.washington.edu
# Then connect to PostgreSQL from there

For routine data management, use the Django admin interface instead.