Remote Install (Production Server)

Last Updated: 2026-10-09 | For: System administrators | Version: Current

This runbook describes image deployment with an Apache edge and AWS Elastic IP cutover. Use the actual host, domain, image revision, and private configuration for your deployment; the named site below is an example of this topology. Deployment Guide explains the supported runtime and verification boundary.

Outcome

After this guide:

1. New production host is fully bootstrapped before traffic cutover.
2. App runs in Docker Compose (mofacts, authenticated mongodb, replica-set initialization, and authenticated redis).
3. Apache + Let's Encrypt serves HTTPS on mofacts.optimallearning.org.
4. Elastic IP reassociation is used for fast cutover and rollback.

Deployment Model

• App container: mofacts on 3000.
• DB container: authenticated mongodb, with replica set mofacts-rs and database MoFACT-meteor3, on the internal network.
• Redis: authenticated internal service for coordination.
• Runtime settings: private read-only host mount; no credentials baked into the image.
• Edge proxy: Apache on host (80/443) to 127.0.0.1:3000.
• TLS: Certbot (python3-certbot-apache).
• Traffic switch: Elastic IP reassociation.

Core assets from the canonical deployment workflow:

• Compose: deploy/docker-compose.yml
• Production template: deploy/.env.self-hosted.example; production uses a private populated env file
• Deploy validator: deploy/server-deploy-validate.sh

Critical Sequencing

Do this in order:

1. Build/push image.
2. Bootstrap new host completely (Docker + Apache + TLS + app running).
3. Validate host.
4. Reassociate Elastic IP last.

Do not move Elastic IP before step 2 is complete.

Prerequisites

• A host sized for your workload, database, and configured password hashing; a historical staging instance size does not establish production capacity.
• Security group inbound:
  • 22 from admin IPs
  • 80 from anywhere
  • 443 from anywhere
• SSH key for instance login.
• Domain in DNS (mofacts.optimallearning.org).
• Docker image pushed to the deployment registry, for example <registry>/mofacts-mini:<tag>.

Host Bootstrap (One-Time Per New Server)

1) Install Docker + Compose

curl -fsSL https://get.docker.com | sudo sh
sudo apt-get update
sudo apt-get install -y docker-compose-plugin
sudo systemctl enable --now docker
sudo usermod -aG docker ubuntu

2) Prepare runtime directory

sudo mkdir -p /var/www/mofacts
sudo chown ubuntu:ubuntu /var/www/mofacts

3) Copy runtime files to server

Preserve the deployment directory layout under /var/www/mofacts, including deploy/docker-compose.yml, deploy/mongodb/, deploy/mongo-init/, and the deployment/backup scripts used by your operating procedure. The Compose file resolves its script mounts relative to its own directory; copying only the YAML is insufficient.

Create private settings and env files from the self-hosted templates. Configure the image, public ROOT_URL, loopback MOFACTS_HTTP_BIND=127.0.0.1:3000, settings/env mounts, backup path, root and app database credentials, and private replica-set keyfile. Set the database name to MoFACT-meteor3 and the replica-set name to mofacts-rs. Use the app user's authenticated MONGO_URL, including authSource=MoFACT-meteor3&replicaSet=mofacts-rs; URL-encode credentials. MongoDB and Redis remain internal. Configure the private Redis password and audit-ingest secret according to the templates.

Set METEOR_SETTINGS_HOST_PATH to the private settings file. Compose mounts it at /run/mofacts/settings.json and sets METEOR_SETTINGS_FILE. Do not use the removed METEOR_SETTINGS_PATH or an image-embedded settings file. See Settings Reference.

For a new replica set, the repository's New-MongoReplicaSetKeyfile.ps1 generates a private keyfile without overwriting an existing one. Retain it securely. Converting an existing standalone database requires the maintenance/backup procedure in deploy/README.md; do not treat initialization of an empty database as verification of an existing-data conversion.

4) Start containers

cd /var/www/mofacts/deploy
sudo docker compose --env-file .env.self-hosted -f docker-compose.yml pull
sudo docker compose --env-file .env.self-hosted -f docker-compose.yml up -d

5) Validate app locally

cd /var/www/mofacts/deploy
sudo docker compose --env-file .env.self-hosted -f docker-compose.yml ps
sudo docker logs --tail 120 mofacts
curl -I http://localhost:3000

Edge Layer (Staging-Parity Apache Setup)

Use Apache to match staging behavior.

1) Install Apache + Certbot and required modules

sudo apt-get update
sudo apt-get install -y apache2 certbot python3-certbot-apache
sudo a2enmod proxy proxy_http proxy_wstunnel ssl rewrite headers

2) Configure HTTP vhost

/etc/apache2/sites-available/000-default.conf:

<VirtualHost *:80>
  ServerName mofacts.optimallearning.org
  ServerAlias mofacts.optimallearning.org

  ProxyPass / http://127.0.0.1:3000/
  ProxyPassReverse / http://127.0.0.1:3000/
  ProxyPreserveHost on

  RewriteEngine on
  RewriteCond %{SERVER_NAME} =mofacts.optimallearning.org
  RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

Apply:

sudo apache2ctl configtest
sudo systemctl restart apache2

3) Issue TLS cert

sudo certbot --apache -d mofacts.optimallearning.org \
  --non-interactive --agree-tos --register-unsafely-without-email --redirect

Expected cert files:

• /etc/letsencrypt/live/mofacts.optimallearning.org/fullchain.pem
• /etc/letsencrypt/live/mofacts.optimallearning.org/privkey.pem

4) Ensure Meteor proxy directives exist on HTTPS vhost

Check /etc/apache2/sites-enabled/000-default-le-ssl.conf contains:

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"
Header always set Content-Security-Policy "default-src 'self'; script-src 'self' 'wasm-unsafe-eval' https://www.youtube.com/iframe_api https://www.youtube.com/s/player/; style-src 'self' https://fonts.googleapis.com; style-src-attr 'unsafe-inline'; img-src 'self' data: blob: https://d2xvti2irp4c7t.cloudfront.net https://i.ytimg.com/vi/; font-src 'self' data: https://fonts.gstatic.com; connect-src 'self' wss://mofacts.optimallearning.org https://noembed.com/embed https://cdn.plyr.io/3.8.4/plyr.svg; media-src 'self' blob: data: https://d2xvti2irp4c7t.cloudfront.net https://cloud.google.com https://docs.cloud.google.com; worker-src 'self' blob:; frame-src 'self' https://www.youtube-nocookie.com https://d2xvti2irp4c7t.cloudfront.net https://plot.ly https://cran.r-project.org; frame-ancestors 'none'; object-src 'none'; base-uri 'self'; form-action 'self'"

ProxyPass /websocket ws://127.0.0.1:3000/websocket
ProxyPassMatch ^/sockjs/(.*)/websocket ws://127.0.0.1:3000/sockjs/$1/websocket

ProxyPass / http://127.0.0.1:3000/
ProxyPassReverse / http://127.0.0.1:3000/
ProxyPreserveHost on

SetEnv proxy-initial-not-pooled 1
SetEnv proxy-nokeepalive 1

Reload:

sudo apache2ctl configtest
sudo systemctl reload apache2

The bounded expression interpreter is the only supported runtime for authored probability formulas and adaptive rules. Run the admin Deployment Readiness check and require tdf.expressions to pass before deployment. The CSP must not include unsafe-eval; invalid expressions fail explicitly instead of entering a JavaScript execution path.

The production CSP permits WebAssembly compilation for Anki import through the wasm-unsafe-eval source in script-src; the separate unsafe-eval permission remains prohibited. Voice previews now load samples directly from https://docs.cloud.google.com; allow that host in media-src. The displayed production policy also retains its existing https://cloud.google.com media allowance. A redirect through that host is no longer required by the preview implementation.

Use C:\dev\mofacts_config\production\apache\000-default-le-ssl.conf as the canonical production policy. YouTube video lessons need the iframe API and versioned player scripts in script-src, thumbnail images in img-src, and Plyr's title metadata and control-icon requests in connect-src, using the exact HTTPS paths shown above. Review the versioned icon URL when upgrading Plyr. Playback remains inside the www.youtube-nocookie.com iframe. A frame allowance alone leaves lessons at “Loading video…”. After applying the policy, verify the served header and test player readiness, playback, timestamp questions, and the completion gate.

5) Validate edge

sudo ss -ltnp | egrep ':80|:443|:3000'
curl -I http://localhost:80
curl -I https://localhost --insecure
curl -I https://mofacts.optimallearning.org

Ongoing Deploy (No Source Build On Server)

From server:

cd /var/www/mofacts/deploy
set -a
source ./.env.self-hosted
set +a
bash ./server-deploy-validate.sh --compose-file docker-compose.yml --service mofacts --image <registry>/mofacts-mini:<tag>

What this does:

1. Preflight checks (docker, compose, file presence).
2. Pull target image.
3. Update app service via compose override.
4. Verify container reaches running.
5. Attempt rollback to the prior image if the container fails to reach running.

The script defaults to a different Compose filename, so pass the canonical file explicitly. It does not load .env.self-hosted itself; the example exports the existing private environment first. running is a process state, not functional readiness. Use --require-readiness --readiness-command with a reviewed command for required functional checks. A readiness-command failure occurs after startup and does not automatically roll back; the operator owns that recovery decision.

Cutover (Elastic IP)

1. Confirm new host passes local + domain checks.
2. Reassociate production Elastic IP to new instance.
3. Re-test:
   • https://mofacts.optimallearning.org
   • OAuth sign-in
   • admin pages and core learner path

If reassociation fails with "already associated", enable Allow reassociation or disassociate first, then associate.

Fast Rollback

If production fails:

1. Reassociate Elastic IP back to old instance immediately.
2. Keep old instance running until new host is stable.
3. Fix and repeat cutover.

Post-Cutover Checklist

1. Monitor logs and restart count for 24 hours.
2. Stop old instance first (do not terminate immediately).
3. Terminate old instance only after stability window and resource cleanup review.

AWS Cleanup Notes

• Terminated instances are not restartable.
• You may still need to clean up separately:
  • unattached EBS volumes
  • snapshots
  • unused Elastic IPs
• Snapshot deletion may fail if referenced by an AMI; deregister AMI first.

Troubleshooting

• redirect_uri_mismatch (Google): callback URL mismatch.
  • Ensure authorized redirect URI includes:
    • https://mofacts.optimallearning.org/_oauth/google
• Microsoft OAuth callback mismatch:
  • In Azure App Registration -> Authentication -> Web redirect URIs, ensure:
    • https://mofacts.optimallearning.org/_oauth/microsoft
  • If staging shares the same app registration, also keep:
    • https://staging.optimallearning.org/_oauth/microsoft
• 503 from Apache while container running: app container crash loop or proxy target unavailable.
• Startup crash Match error: Expected string, got undefined: invalid/missing email entries in settings/user data.
• ssh ... port 22 timed out: wrong IP or SG inbound SSH missing.

Security Reminders

• Do not commit .env or secrets.
• Rotate secrets if they are exposed.
• Keep Mongo internal-only unless explicit external access is required.
• Patch host packages and base images regularly.

See Also

• Settings.json Reference
• Troubleshooting