SecureServer — wp-engine.sh

Also known as WP-Engine — Multi-Tenant Hardened WordPress Infrastructure.

A single, self-contained Bash script that takes a bare-metal server, Proxmox VM, Proxmox LXC, or base container from an open, unhardened state to a secure, multi-tenant WordPress hosting platform with observability dashboards — in two commands: --bootstrap and create.

It eliminates container sprawl by combining a centralized, tuned MariaDB 11.4 instance and a centralized Redis 7 object cache with completely isolated tenant environments — alongside a host-level security perimeter (hardened high-port SSH, UFW with container-bypass prevention, and Fail2ban) and a full Prometheus + Grafana observability stack.

Work proceeds in two phases: --bootstrap hardens the host and stands up the platform, then create spins up unlimited WordPress sites — each with its own database, credentials, Redis cache index, and TLS routing.

---

Table of Contents

1. Project Overview
2. Requirements
3. Quick Start
4. CLI Reference & Options
5. Step-by-Step: What --bootstrap Does
6. Step-by-Step: What create Does
7. Option Guides
8. Architecture: Networks, Ports, Layout
9. Credentials & State Files
10. Monitoring Dashboards
11. Day-2 Operations
12. Security Hardening Summary
13. Known Limitations & Post-Install TODOs
14. Testing the Script

---

1. Project Overview

Problem: A freshly provisioned Linux server is open to the internet — SSH brute force, WordPress exploit scanners, open databases, no firewall, no visibility.

Solution: wp-engine.sh performs two phases:

Phase	Command	Purpose
Phase 1 — Host Hardening & Platform	./wp-engine.sh --bootstrap	Installs packages, container runtime, hardened SSH, UFW firewall, Fail2ban jails, container networks, ingress reverse proxy, and the Grafana/Prometheus monitoring stack.
Phase 2 — Site Provisioning	./wp-engine.sh create <slug> <domain> ...	Creates an isolated WordPress site (container, database, DB user, Redis cache index, TLS route, WP-CLI automated install).

Key design properties:

• Multi-tenant — many WordPress sites share one host, one MariaDB instance (isolated DB per site), and one Redis instance (isolated DB index per site).
• Proxy-agnostic — choose Nginx Proxy Manager, Traefik v3, or Caddy at bootstrap time.
• Runtime-agnostic — Docker (official packages) or Podman (distro packages).
• Persistent state — choices are persisted to /server/websites/.engine-settings, so subsequent create runs need no re-prompting.
• Automated WordPress install — wp core install, permalinks, and Redis object cache are wired up via WP-CLI containers; no browser wizard required.

Supporting design documents live in docs/:

• docs/Runtime_Architecture_Network_Topology.md — Mermaid diagram of traffic flow, networks, and security layers.
• docs/Filesystem_and_Directory_Structure.md — Mermaid diagram of /server/websites/ layout.

---

2. Requirements

Requirement	Detail
OS	Clean Debian 11/12 or Ubuntu 22.04/24.04 LTS installation (bare-metal, Proxmox VM, or Proxmox LXC) with apt and systemd (the script calls apt-get, systemctl, ufw). Not compatible with RHEL/Alpine/Windows hosts.
Privileges	Must be run as root (the script exits if EUID != 0).
Network	Outbound internet access to pull packages and container images (Docker Hub, download.docker.com, etc.).
Inbound ports	80/443 reachable from the internet (for HTTP/HTTPS + ACME TLS challenges), plus your chosen SSH port.
DNS	A/AAAA records for each site domain pointing at the server's public IP before running create (needed for certificate issuance).
Resources	Recommended ≥ 2 GB RAM (MariaDB buffer pool is pre-set to 2G in 99-performance.cnf) and ≥ 20 GB disk.

---

3. Quick Start

# 0. Copy the script to the target host and make it executable
chmod +x wp-engine.sh

# 1. Harden the host + deploy proxy & monitoring (interactive prompts)
./wp-engine.sh --bootstrap

# 2. Provision your first WordPress site
./wp-engine.sh create teetimeprints teetimeprints.com mkell admin@teetimeprints.com

Fully unattended bootstrap (no prompts — ideal for cloud-init / Ansible / automation):

# Docker + Caddy on custom SSH port 48222
./wp-engine.sh --bootstrap --ssh-port 48222 --engine docker --proxy caddy

# Docker + Traefik v3 on custom SSH port 52822
./wp-engine.sh --bootstrap --ssh-port 52822 --engine docker --proxy traefik

# Podman + Nginx Proxy Manager (NPM) on custom SSH port 48222
./wp-engine.sh --bootstrap --ssh-port 48222 --engine podman --proxy npm

Install system-wide so it can be run from anywhere:

install -m 0755 wp-engine.sh /usr/local/bin/wp-engine.sh
# ...or download it directly:
curl -fsSL https://your-repo/wp-engine.sh -o /usr/local/bin/wp-engine.sh
chmod +x /usr/local/bin/wp-engine.sh

[!WARNING] Do not close your current SSH terminal after running --bootstrap. Open a secondary terminal and verify that you can connect over your chosen port before disconnecting:

ssh -p <PORT> user@<HOST_IP>

---

4. CLI Reference & Options

Usage:
  Interactive Bootstrap:
    ./wp-engine.sh --bootstrap

  Unattended Bootstrap:
    ./wp-engine.sh --bootstrap --ssh-port <port> --engine <docker|podman> --proxy <npm|traefik|caddy>

  Provision Site:
    ./wp-engine.sh create <slug> <domain> [admin_user] [admin_email]

Examples:
  ./wp-engine.sh --bootstrap --ssh-port 48222 --engine docker --proxy caddy
  ./wp-engine.sh create teetimeprints teetimeprints.com mkell admin@teetimeprints.com

Actions

Action	Description
--bootstrap	Runs the 6-step host provisioning sequence, then deploys the monitoring stack. Prompts for any option not supplied via flags.
create <slug> <domain> [admin_user] [admin_email]	Provisions a new WordPress site. If the ingress proxy isn't running yet, bootstrap runs automatically; if the shared DB/cache core isn't up, it is initialized automatically.
(no/unknown action)	Prints the usage text above and exits with code 1.

Flags (used with --bootstrap)

Flag	Values	Default	Effect
--ssh-port	1–65535	48222	Moves SSH off port 22; written to sshd_config.d/99-hardened.conf, the ssh.socket unit (if present), UFW, and the Fail2ban sshd jail.
--engine	docker | podman	docker	Container runtime. Docker installs from Docker's official apt repo; Podman installs podman, podman-compose, podman-docker from distro repos.
--proxy	npm | traefik | caddy	npm	Ingress reverse proxy. Determines the proxy compose file, extra firewall openings, TLS automation, and how sites get routed.

Note: Flags are read before the persisted state is loaded, so they take precedence for that run and are saved into /server/websites/.engine-settings. Subsequent create commands reuse the saved choices without prompting.

create arguments

Position	Argument	Default	Description
1	slug	(required)	Directory/container name, e.g. teetimeprints. Hyphens are converted to underscores for DB identifiers.
2	domain	(required)	Public FQDN, e.g. teetimeprints.com. Used for TLS routing and the WP site URL.
3	admin_user	admin	WordPress administrator username.
4	admin_email	admin@<domain>	WordPress administrator e-mail.

Additional create examples:

# Minimal — admin user defaults to `admin`, e-mail to admin@myblog.com
./wp-engine.sh create myblog myblog.com

# Custom admin credentials
./wp-engine.sh create teetimeprints teetimeprints.com mkell admin@teetimeprints.com

---

5. Step-by-Step: What --bootstrap Does

Phase A — Bare-host provisioning & hardening (6 steps)

[1/6] Installing Host Packages Installs apt-transport-https ca-certificates curl gnupg lsb-release ufw fail2ban nftables openssl jq.

[2/6] Setting up Container Engine

• Docker: adds Docker's official GPG key + apt repo, installs docker-ce, docker-ce-cli, containerd.io, docker-buildx-plugin, docker-compose-plugin, enables the service (skipped if Docker already present).
• Podman: apt-get install podman podman-compose podman-docker, enables the service.

[3/6] Hardening SSH on Port <port> Writes /etc/ssh/sshd_config.d/99-hardened.conf:

Port <chosen port>
PermitRootLogin prohibit-password
PasswordAuthentication yes
X11Forwarding no
MaxAuthTries 4
ClientAliveInterval 300
ClientAliveCountMax 2

Also re-points ssh.socket's ListenStream to the new port when socket activation is in use, then restarts SSH.

[4/6] Configuring Firewall & Isolation Rules

• UFW policy: deny incoming / allow outgoing.
• Allowed inbound: SSH port, 80, 443, 3000 (Grafana), plus 81 if NPM or 8080 if Traefik.
• For Docker, appends a DOCKER-USER chain to /etc/ufw/after.rules so UFW rules actually apply to container traffic (blocks direct SYN scans into 172.16.0.0/12, rate-limited logging).
• ufw --force enable && ufw reload.

[5/6] Scaffolding Networks & Fail2ban Jails

• Creates the directory skeleton (/server/websites/{proxy,core,template,sites,monitoring}).
• Creates three external container networks: proxy-net, shared-backend, monitoring-net.
• Writes Fail2ban jails:
  • sshd — your SSH port, maxretry 3, findtime 600, bantime 3600, banaction ufw.
  • ingress-wordpress — watches /server/websites/proxy/logs/*.log for brute force on /wp-login.php and /xmlrpc.php, plus scans for .env, .git, wp-config.php, eval-stdin.php; maxretry 4, bantime 86400 (24 h), banaction ufw.
• Enables and restarts Fail2ban.

[6/6] Launching Ingress Proxy — writes and starts the proxy compose file (differences per proxy are covered in Option Guides), then deploys monitoring.

Phase B — Observability stack (deploy_monitoring)

Deploys /server/websites/monitoring/docker-compose.yml on the isolated monitoring-net:

Service	Image	Purpose	Exposure
prometheus	prom/prometheus:v2.54.0	Time-series DB; scrapes node_exporter + cAdvisor every 15 s	internal
node_exporter	prom/node-exporter:v1.8.2	Host CPU/RAM/disk metrics (/proc, /sys, / mounted read-only)	internal
cadvisor	gcr.io/cadvisor/cadvisor:v0.49.1	Per-container cgroup resource usage	internal
grafana	grafana/grafana:11.2.0	Dashboards; Prometheus datasource auto-provisioned; sign-up disabled	:3000

A random Grafana admin password is generated (openssl rand) and appended to the state file as GRAFANA_ADMIN_PASS.

---

6. Step-by-Step: What create Does

./wp-engine.sh create <slug> <domain> [admin_user] [admin_email]

1. Load state — reads /server/websites/.engine-settings (runs bootstrap prompts only if it doesn't exist yet).
2. Auto-heal — if no ingress_proxy container is running, bootstrap runs automatically; if core_mariadb is missing, the shared core is initialized.
3. Allocate resources
   • Next Redis DB index from NEXT_REDIS_INDEX (incremented in core/.env).
   • DB name wp_<slug> and DB user usr_<slug> (hyphens → underscores).
   • Random DB password and random WordPress admin password (openssl rand).
4. Create the database — CREATE DATABASE, CREATE USER, GRANT ALL scoped to that site's DB only, executed inside core_mariadb.
5. Write site files
   • config/wordpress.ini — PHP limits: memory_limit 512M, upload_max_filesize/post_max_size 128M, max_execution_time 300, max_input_vars 5000, OPcache enabled (256 MB, 20 000 files).
   • docker-compose.yml — wordpress:6.7-apache container attached to proxy-net and shared-backend, with WORDPRESS_CONFIG_EXTRA injecting:
     • Redis connection (core_redis, port 6379, per-site password, per-site DB index, per-site cache key salt)
     • DISALLOW_FILE_EDIT (no theme/plugin editing from wp-admin)
     • FORCE_SSL_ADMIN
     • X-Forwarded-Proto HTTPS detection behind the proxy
     • Traefik routing labels (only when --proxy traefik)
6. Start the site — chown 33:33 on the docroot, then compose up -d.
7. Wire up routing
   • Caddy: appends a domain { reverse_proxy slug_wp:80 ... } block to the Caddyfile and hot-reloads Caddy.
   • Traefik: handled by the labels from step 5 (router on websecure, cert resolver myresolver).
   • NPM: prints a reminder — create the Proxy Host manually in the NPM UI.
8. Automated WordPress bootstrap (WP-CLI container)
   • Waits for wp-config.php to appear in the docroot.
   • wp core install --url=https://<domain> --admin_user=... --admin_password=... --admin_email=... --skip-email
   • wp rewrite structure '/%postname%/' --hard
   • wp plugin install redis-cache --activate → wp redis enable
   • chown -R 33:33 data/wp/wp-content
9. Prints a summary — domain, admin user, admin password (shown only once — copy it now), DB name, DB user, Redis index, directory path, plus any proxy-specific next step.

---

7. Option Guides

7.1 Container runtime

	Docker (default)	Podman
Install source	Docker official apt repo (docker-ce + compose plugin)	Distro repos (podman, podman-compose, podman-docker)
Compose command	docker compose	podman-compose
UFW integration	Gets the DOCKER-USER after.rules patch (required, since Docker bypasses UFW by default)	Not applied (Podman networking works differently with nftables/UFW)
Root	Script still requires root for host hardening	Same — host services are hardened system-wide

7.2 Ingress proxy

	NPM (default)	Traefik v3	Caddy
Image	jc21/nginx-proxy-manager	traefik:v3.1	caddy:2-alpine
Management	Web UI on :81 (opened in UFW)	Docker labels + insecure API dashboard on :8080 (opened in UFW)	Text Caddyfile
TLS certificates	Requested per host in the UI (Let's Encrypt)	ACME TLS-ALPN challenge via myresolver labels	Fully automatic Let's Encrypt
Site onboarding	Manual: create Proxy Host → https://domain → forward to <slug>_wp:80	Automatic: labels emitted by create	Automatic: create appends a site block + reloads
Access log	bind-mounted proxy/logs/	proxy/logs/access.log (--accesslog)	proxy/logs/access.log via log {} block
Extra ports opened	81	8080	none
Best for	Users who want a GUI	Label-driven, declarative setups	Minimal, zero-touch SSL

The proxy choice affects the generated proxy/docker-compose.yml, the UFW allow rules, and how create wires routing — all persisted to the state file.

7.3 SSH port

Any non-standard port (default 48222). Applied consistently in four places: sshd_config.d, ssh.socket, the UFW allow rule, and the Fail2ban sshd jail — so ban actions target the right port.

---

8. Architecture: Networks, Ports, Layout

High-level traffic flow (see docs/ for the full Mermaid diagrams):

                        Internet / WAN
                              │
                              ▼
            ┌──────────────────────────────────┐
            │   Host Firewall (UFW / nftables) │
            │   + Fail2ban Bare-Host Jails     │
            └─────────────────┬────────────────┘
                              │ Ports 80 / 443
                              ▼
            ┌──────────────────────────────────┐
            │ Ingress: Caddy / Traefik / NPM   │ ── Access Logs ──► Host Fail2ban
            └──────────────┬───────────────────┘
                           │
       ┌───────────────────┴───────────────────┐
       ▼                                       ▼
┌──────────────┐                       ┌──────────────┐
│ WordPress    │                       │ Observability│
│  - PHP 8.3   │                       │  - Grafana   │ (Port 3000)
│  - WP-CLI    │                       │  - Prometheus│
│  - OPcache   │                       │  - Node Exp  │
└──────┬───────┘                       │  - cAdvisor  │
       │                               └──────────────┘
  ┌────┴────────────┐
  ▼                 ▼
┌───────────┐ ┌───────────┐
│ MariaDB   │ │ Redis     │
│ (Shared   │ │ (Shared   │
│  Core)    │ │  Cache)   │
└───────────┘ └───────────┘

Networks

Network	Scope	Members
proxy-net	external	ingress proxy + all WordPress containers (public-facing side)
shared-backend	external, isolated	all WordPress containers + core_mariadb + core_redis (no host ports published)
monitoring-net	external, isolated	Prometheus, node_exporter, cAdvisor, Grafana

Ports

Port	Service	Firewall
22 → 48222 (default)	SSH	allowed (custom port)
80 / 443	Ingress proxy (HTTP/HTTPS + ACME)	allowed
81	Nginx Proxy Manager admin UI	allowed only if --proxy npm
8080	Traefik dashboard	allowed only if --proxy traefik
3000	Grafana	allowed
3306 / 6379	MariaDB / Redis	not published — container network only

Filesystem layout (/server/websites/)

/server/websites/
├── .engine-settings          # Persisted choices: ENGINE, PROXY_TYPE, SSH_PORT, GRAFANA_ADMIN_PASS
├── core/                     # Shared data tier
│   ├── .env                  # DB_ROOT_PASSWORD, REDIS_PASSWORD, NEXT_REDIS_INDEX (chmod 600)
│   ├── docker-compose.yml    # core_mariadb (MariaDB 11.4) + core_redis (Redis 7)
│   ├── config/mariadb/99-performance.cnf
│   └── data/{mariadb,redis}/
├── proxy/                    # Ingress tier
│   ├── docker-compose.yml    # ingress_proxy container
│   ├── Caddyfile             # (Caddy only) auto-appended per site
│   ├── letsencrypt/          # acme.json / certs
│   └── logs/access.log       # scanned by Fail2ban `ingress-wordpress`
├── monitoring/               # Observability tier
│   ├── docker-compose.yml    # prometheus, node_exporter, cadvisor, grafana
│   ├── prometheus/prometheus.yml
│   └── grafana/provisioning/datasources/prometheus.yaml
├── sites/<slug>/             # One directory per WordPress tenant
│   ├── docker-compose.yml    # <slug>_wp container (proxy-net + shared-backend)
│   ├── config/wordpress.ini  # OPcache & upload limits
│   └── data/wp/              # Full WordPress docroot (uid 33)
└── template/                 # Reserved for site templates

Host-level configs written by the script:

• /etc/ssh/sshd_config.d/99-hardened.conf
• /etc/ufw/after.rules (DOCKER-USER protection block)
• /etc/fail2ban/jail.d/sshd.local
• /etc/fail2ban/jail.d/ingress-wordpress.local
• /etc/fail2ban/filter.d/ingress-wordpress.conf

---

9. Credentials & State Files

Secret	Where it lives	Notes
Engine / proxy / SSH port	/server/websites/.engine-settings	plain, non-secret
Grafana admin password	/server/websites/.engine-settings → GRAFANA_ADMIN_PASS	login: admin @ http://<host>:3000
MariaDB root password	/server/websites/core/.env → DB_ROOT_PASSWORD	chmod 600; used for creating per-site DBs
Redis password	/server/websites/core/.env → REDIS_PASSWORD	injected into every site's wp-config
Per-site DB password	injected into the site's compose environment	random per site
WordPress admin password	printed to stdout only, at the end of create	not persisted anywhere — save it immediately
Redis DB index	core/.env → NEXT_REDIS_INDEX counter	one index allocated per site (0, 1, 2, …)

⚠️ Re-running --bootstrap appends a new GRAFANA_ADMIN_PASS line; when the state file is sourced, the last line wins, effectively rotating the Grafana password.

---

10. Monitoring Dashboards

After bootstrap:

1. Open http://<HOST_IP>:3000 (Grafana).

2. Log in with admin / the password generated during bootstrap and saved in the state file:

   grep GRAFANA_ADMIN_PASS /server/websites/.engine-settings
   

3. The Prometheus datasource is pre-provisioned and set as default. Import dashboards via Dashboards → Import — recommended IDs:

   ID	Dashboard	Provides
   1860	Node Exporter Full	Real-time CPU, RAM, disk I/O, network throughput, load averages
   14282	cAdvisor Docker Monitoring	Per-container resource usage; spot CPU/RAM throttling across tenant sites

Data collected:

• node_exporter → host CPU, memory, disk I/O, network.
• cAdvisor → per-container CPU/memory of every WordPress, DB, proxy, and monitoring container.
• Prometheus → 15 s scrape interval, persistent prometheus_data volume.

---

11. Day-2 Operations

# Add another site (no prompts — state already saved)
./wp-engine.sh create kellcreations kellcreations.com admin admin@kellcreations.com

# Check running containers
docker ps            # or: podman ps

# Site lifecycle
cd /server/websites/sites/<slug>
docker compose up -d / stop / restart / logs -f

# Core stack & proxy
cd /server/websites/core  && docker compose ps
cd /server/websites/proxy && docker compose logs -f ingress

# WP-CLI inside a site (one-off)
docker run --rm --user 33:33 --network shared-backend \
  -v /server/websites/sites/<slug>/data/wp:/var/www/html wordpress:cli wp plugin list

# Logs for a specific site
docker logs -f <slug>_wp            # or: podman logs -f <slug>_wp

# Unban an IP address
fail2ban-client set ingress-wordpress unbanip <IP_ADDRESS>

# Security status
ufw status verbose
fail2ban-client status
fail2ban-client status ingress-wordpress
fail2ban-client status sshd
fail2ban-client get ingress-wordpress banned

Backups (not automated by the script — recommended):

• /server/websites/core/data/mariadb/ — all site databases (stop MariaDB or use mariadb-dump for consistency).
• /server/websites/sites/*/data/wp/ — WordPress docroots (uploads, themes, plugins).
• /server/websites/core/.env and .engine-settings — credentials/state.

---

12. Security Hardening Summary

Layer	Control
Access	SSH moved off port 22; root login limited to key auth (PermitRootLogin prohibit-password); MaxAuthTries 4; X11 forwarding off; idle timeout via ClientAliveInterval/ClientAliveCountMax
Perimeter	UFW default-deny inbound; explicit allow-list (SSH, 80, 443, 3000, +81/8080 per proxy)
Container escape path	DOCKER-USER nftables chain in /etc/ufw/after.rules so UFW governs Docker-bridged traffic; direct scans into RFC1918 container ranges are logged + dropped
Brute force	Fail2ban sshd jail (3 attempts → 1 h ban) and ingress-wordpress jail (4 attempts on wp-login.php/xmlrpc.php/sensitive-file scans → 24 h ban via UFW), fed by proxy access logs
WordPress	DISALLOW_FILE_EDIT, FORCE_SSL_ADMIN, HTTPS detection behind proxy, randomized admin/DB passwords, per-site scoped DB grants (no shared credentials)
Secrets	Root/Redis passwords generated with openssl rand; core/.env set to chmod 600; Grafana sign-up disabled
Isolation	Each site gets its own DB, DB user, Redis index, cache key salt, docroot, and compose project — on a shared-but-segmented backend network with no published host ports
Observability	Centralized access logs → Fail2ban; Prometheus metrics for capacity/anomaly detection

---

13. Known Limitations & Post-Install TODOs

Behaviors of the current script — review and adjust to your threat model:

1. NPM requires a manual step — after create, log in to http://<host>:81 (NPM default login) and add a Proxy Host forwarding https://<domain> → <slug>_wp:80 with an SSL certificate. Change the NPM default password immediately.
2. Traefik ACME email is a placeholder — the generated compose file hardcodes admin@example.com. Edit /server/websites/proxy/docker-compose.yml before the first certificate request.
3. Traefik dashboard is exposed — --api.insecure=true on :8080 and UFW allows it publicly. Restrict or close it if you don't need it.
4. NPM admin UI (:81) is exposed publicly when --proxy npm is selected. Consider restricting it to your IP.
5. Grafana (:3000) is served over plain HTTP with no TLS and is firewall-open. Put it behind the proxy with HTTPS, or restrict the firewall rule to your IP.
6. PasswordAuthentication yes remains enabled in the SSH hardening block — convenient for first-time migration, but switch to keys-only (no) once your key is installed.
7. Single MariaDB/Redis instance — innodb_buffer_pool_size = 2G is a fixed default; tune core/config/mariadb/99-performance.cnf for smaller/larger hosts.
8. Distro scope — apt/systemd only. Non-Debian hosts need manual adaptation of steps 1 and 4.
9. No automated backups or WAF — add ModSecurity/Cloudflare or scheduled backups per your requirements.
10. WP admin password is ephemeral — captured only in the terminal output of create; store it in your password manager right away.

---

14. Testing the Script

The script expects root + systemd + apt + a privileged container runtime, so it can't execute directly on Windows. The repo ships a staged test harness plus a dev container that reproduces a real host.

14.1 Test stages — tests/smoke.sh

STAGE=	Needs	What it verifies
static (default)	Linux (WSL is fine), no root	bash -n syntax on both scripts, shellcheck -S error, usage/argument handling, non-root rejection via setpriv
bootstrap	root + systemd + privileged container	Full unattended --bootstrap (Docker + Caddy), then asserts: state file, UFW active, both Fail2ban jails, all 5 containers running, Grafana health, proxy on :80, SSH on the hardened port
site	same as bootstrap	Auto-bootstraps if needed, runs create smoketest smoketest.test, then asserts: site dir, WP container, wp-config.php, scoped DB wp_smoketest, Caddyfile route, HTTP response through the proxy, wp core is-installed
all	same as bootstrap	static + bootstrap + site
cleanup	root	Removes the disposable test site (container, DB, user, directory)

bash tests/smoke.sh                          # safe static checks (works in WSL / CI)
STAGE=all bash tests/smoke.sh                # full end-to-end run (dev container)
STAGE=cleanup bash tests/smoke.sh            # remove the test site
SSH_PORT=2222 TEST_SLUG=demo bash tests/smoke.sh   # override defaults

Exit codes: 0 all passed · 1 failures (each printed as [FAIL]) · 2 wrong environment for that stage. Long stages are wrapped in timeout guards (BOOTSTRAP_TIMEOUT=900, SITE_TIMEOUT=900 — override via env).

14.2 Dev container — .devcontainer/

A privileged Ubuntu 24.04 container with systemd as PID 1, preloaded with shellcheck, openssh-server, and the tooling the script's hardening path exercises. Docker/UFW/Fail2ban are deliberately not preinstalled — --bootstrap installs them itself, which is part of what's being tested. Ports 80, 443, 81, 8080, 3000, and 48222 are forwarded to the host for manual poking.

Option A — VS Code Dev Containers (requires Docker Desktop or any Docker engine + the Dev Containers extension):

1. Open the folder in VS Code
2. Command Palette → Dev Containers: Reopen in Container
3. The postCreateCommand runs the static suite automatically
4. In the terminal: STAGE=all bash tests/smoke.sh

Option B — plain Docker CLI:

docker build -t wp-engine-dev .devcontainer
docker run --rm -d --name wp-engine-dev --privileged \
  -p 80:80 -p 443:443 -p 81:81 -p 8080:8080 -p 3000:3000 -p 48222:48222 \
  -v "$PWD":/workspaces/SecureServer wp-engine-dev
docker exec -it wp-engine-dev bash

# inside the container:
STAGE=all bash tests/smoke.sh
STAGE=cleanup bash tests/smoke.sh

Option C — Podman (verified working on this workstation):

# One-time machine setup: rootful, enough RAM for MariaDB, repo drive shared
podman machine init --now --rootful -m 6144 --cpus 4 -v "D:\:/mnt/d"

podman build -t wp-engine-dev .devcontainer
# Named volume REQUIRED: nested dockerd can't use the container's own overlay
# as upperdir ("not supported as upperdir"); the volume binds real ext4 instead.
podman volume create docker-storage
podman run --rm -d --name wp-engine-dev --privileged \
  -p 80:80 -p 443:443 -p 81:81 -p 8080:8080 -p 3000:3000 -p 48222:48222 \
  -v docker-storage:/var/lib/docker \
  -v "D:/programming/Servers/SecureServer:/workspaces/SecureServer" wp-engine-dev

# run the suites inside it
podman exec wp-engine-dev bash -c 'cd /workspaces/SecureServer && STAGE=static bash tests/smoke.sh'
podman exec wp-engine-dev bash -c 'STAGE=all bash /workspaces/SecureServer/tests/smoke.sh'

14.3 Notes & caveats

• First full run pulls ~1–2 GB of images (Docker CE, WordPress, MariaDB, Grafana, Prometheus, …) and takes several minutes.
• The site stage uses smoketest.test (RFC 2606 reserved TLD) — no real DNS needed; checks send Host: headers directly to the proxy, so ACME issuance is never triggered.
• A full STAGE=all run reconfigures the container's SSH port and firewall — that's expected inside the throwaway container; never run it on a production host.
• Rebuild the container for a clean slate: docker rm -f wp-engine-dev and re-run (host state lives in the container's writable layer; the mounted repo is untouched).
• CI option: the static stage runs anywhere with bash + shellcheck and could be wired into Forgejo Actions later.

---

14.4 Troubleshooting the Podman-on-Windows setup

Hard-won fixes from bringing this environment up (recorded so the next machine doesn't have to rediscover them):

1. podman machine init fails with failed to pull … machine-os: The system cannot find the path specified — known bug in Podman ≤ 6.0.2 (podman#29639). Download the machine-os blob manually from quay.io/podman/machine-os (auth token → manifest → layer blob, verify SHA256) and place it at the cache path podman expects: %USERPROFILE%\.local\share\containers\podman\machine\wsl\cache\<manifest-digest>.tar.zst — the next podman machine init treats it as a cache hit.
2. Containers fail with crun: controller 'pids' is not available under …/non-systemd/… — the machine booted with WSL's init instead of systemd, so cgroups aren't managed. Append [boot] systemd=true to /etc/wsl.conf inside the machine (wsl -d podman-machine-default -u root), then wsl --terminate podman-machine-default + podman machine start.
3. machine start fails with ssh error: machine not in running state — usually a stale win-sshproxy.exe from a previous attempt holding the SSH port, which triggers a port reassignment and leaves sshd listening on the old port while the config says the new one. Fix: kill all win-sshproxy/gvproxy processes, podman machine stop, wsl --terminate podman-machine-default, verify both show Stopped, then podman machine start.
4. CreateFile \\.\pipe\docker_engine: All pipe instances are busy — Podman Desktop is holding the Docker-compatible pipe; Podman clients are unaffected. A clean machine stop/start cycle usually reclaims it.
5. Machine dies mid-run (e.g., during a Docker CE install) — WSL OOM. Ensure %USERPROFILE%\.wslconfig has a swap (e.g. swap=8GB under [wsl2]).
6. machine start reports an SSH transition error but podman version shows a healthy Server — verify with podman ps / podman run quay.io/podman/hello; if those work, the machine is usable.

---

14.5 Bugs the suite caught (fixed)

Every one of these broke wp-engine.sh on stock Ubuntu 24.04 and would have failed on a real host — exactly what the harness exists for:

1. SSH socket port rewrite — Ubuntu 24.04's ssh.socket lists per-family entries (0.0.0.0:22, [::]:22); the blanket sed produced two identical ListenStream=48222 lines → systemd EADDRINUSE → bootstrap aborted. Fixed with family-preserving substitutions.
2. Missing ufw-docker-logging-deny chain declaration in the /etc/ufw/after.rules DOCKER-USER block → iptables-restore error at line 47 → ufw enable failed.
3. Empty Caddyfile (touch) → caddy run fails adaptation with EOF and crash-loops. Now seeded with a comment + :80 health-respond block.
4. MariaDB first-boot race — mariadb-admin ping reports success before root-password init completes; the first create got ERROR 1045. Replaced with an authenticated SELECT 1 wait loop.
5. CREATE USER ... '@'\%' created the user with a literal \% host while GRANT targeted % → ERROR 1133. (This one meant site provisioning never worked at all.)
6. WP-CLI containers missing WORDPRESS_DB_* env — wp-config.php falls back to host mysql → "Error establishing a database connection".
7. Missing wp binary prefix in all four WP-CLI invocations → exec: core: not found (exit 127).
8. WP-CLI containers missing WORDPRESS_CONFIG_EXTRA — wp redis enable fell back to 127.0.0.1.
9. Compose $VAR interpolation ate $_SERVER in the injected PHP block (isset(['HTTP_X_FORWARDED_PROTO'])) — broke HTTPS detection behind the proxy on every site. Fixed with $$ escaping.

Harness fixes (in tests/smoke.sh): Grafana/proxy assertions got 60 s retry windows, the :80 check no longer false-passes on curl exit quirks, the DB check uses root credentials from core/.env, and the wp core is-installed check passes the site's full env (multi-line WORDPRESS_CONFIG_EXTRA included).

---

Repository Contents

wp-engine.sh                                   # The installer/provisioner script (executable)
README.md                                      # This file
.gitignore / .gitattributes                    # Secrets excluded; *.sh forced to LF
.devcontainer/
│   Dockerfile                                 # Privileged Ubuntu 24.04 + systemd test host
│   devcontainer.json                          # VS Code Dev Container config (ports, shellcheck)
tests/
│   smoke.sh                                   # Staged test harness (static|bootstrap|site|all|cleanup)
docs/
│   Runtime_Architecture_Network_Topology.md   # Architecture & network Mermaid diagram
│   Filesystem_and_Directory_Structure.md      # Filesystem layout Mermaid diagram