Initial commit: wp-engine hardening script, README, and architecture docs

2026-10-03 13:39:34 -04:00 · 2026-10-03 13:39:34 -04:00 · 08fc8f66d0
commit 08fc8f66d0
5 changed files with 1506 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@ -0,0 +1,16 @@
 # OS / editor noise
 .DS_Store
 Thumbs.db
 *.swp
 *~
 .vscode/
 .idea/
 # Secrets & runtime state — never commit
 .engine-settings
 .env
 *.env
 *.log
 # Transient
 *.tmp
--- a/README.md
+++ b/README.md
@ -0,0 +1,513 @@
 # SecureServer — `wp-engine.sh`
 **Also known as WP-Engine — Multi-Tenant Hardened WordPress Infrastructure.**
 A single, self-contained Bash script that takes a **bare-metal server, Proxmox VM, Proxmox LXC, or base container** from an open, unhardened state to a **secure, multi-tenant WordPress hosting platform** with **observability dashboards** — in two commands: `--bootstrap` and `create`.
 It eliminates container sprawl by combining a **centralized, tuned MariaDB 11.4 instance** and a **centralized Redis 7 object cache** with completely isolated tenant environments — alongside a host-level security perimeter (hardened high-port SSH, UFW with container-bypass prevention, and Fail2ban) and a full Prometheus + Grafana observability stack.
 Work proceeds in two phases: `--bootstrap` hardens the host and stands up the platform, then `create` spins up unlimited WordPress sites — each with its own database, credentials, Redis cache index, and TLS routing.
 ---
 ## Table of Contents
 1. [Project Overview](#1-project-overview)
 2. [Requirements](#2-requirements)
 3. [Quick Start](#3-quick-start)
 4. [CLI Reference & Options](#4-cli-reference--options)
 5. [Step-by-Step: What `--bootstrap` Does](#5-step-by-step-what---bootstrap-does)
 6. [Step-by-Step: What `create` Does](#6-step-by-step-what-create-does)
 7. [Option Guides](#7-option-guides)
 8. [Architecture: Networks, Ports, Layout](#8-architecture-networks-ports-layout)
 9. [Credentials & State Files](#9-credentials--state-files)
 10. [Monitoring Dashboards](#10-monitoring-dashboards)
 11. [Day-2 Operations](#11-day-2-operations)
 12. [Security Hardening Summary](#12-security-hardening-summary)
 13. [Known Limitations & Post-Install TODOs](#13-known-limitations--post-install-todos)
 ---
 ## 1. Project Overview
 **Problem:** A freshly provisioned Linux server is open to the internet — SSH brute force, WordPress exploit scanners, open databases, no firewall, no visibility.
 **Solution:** `wp-engine.sh` performs two phases:
 | Phase | Command | Purpose |
 |---|---|---|
 | **Phase 1 — Host Hardening & Platform** | `./wp-engine.sh --bootstrap` | Installs packages, container runtime, hardened SSH, UFW firewall, Fail2ban jails, container networks, ingress reverse proxy, and the Grafana/Prometheus monitoring stack. |
 | **Phase 2 — Site Provisioning** | `./wp-engine.sh create <slug> <domain> ...` | Creates an isolated WordPress site (container, database, DB user, Redis cache index, TLS route, WP-CLI automated install). |
 **Key design properties:**
 - **Multi-tenant** — many WordPress sites share one host, one MariaDB instance (isolated DB per site), and one Redis instance (isolated DB index per site).
 - **Proxy-agnostic** — choose Nginx Proxy Manager, Traefik v3, or Caddy at bootstrap time.
 - **Runtime-agnostic** — Docker (official packages) or Podman (distro packages).
 - **Persistent state** — choices are persisted to `/server/websites/.engine-settings`, so subsequent `create` runs need no re-prompting.
 - **Automated WordPress install** — `wp core install`, permalinks, and Redis object cache are wired up via WP-CLI containers; no browser wizard required.
 Supporting design documents live in [`docs/`](docs/):
 - [`docs/Runtime_Architecture_Network_Topology.md`](docs/Runtime_Architecture_Network_Topology.md) — Mermaid diagram of traffic flow, networks, and security layers.
 - [`docs/Filesystem_and_Directory_Structure.md`](docs/Filesystem_and_Directory_Structure.md) — Mermaid diagram of `/server/websites/` layout.
 ---
 ## 2. Requirements
 | Requirement | Detail |
 |---|---|
 | **OS** | Clean Debian 11/12 or Ubuntu 22.04/24.04 LTS installation (bare-metal, Proxmox VM, or Proxmox LXC) with `apt` and `systemd` (the script calls `apt-get`, `systemctl`, `ufw`). Not compatible with RHEL/Alpine/Windows hosts. |
 | **Privileges** | Must be run as `root` (the script exits if `EUID != 0`). |
 | **Network** | Outbound internet access to pull packages and container images (Docker Hub, download.docker.com, etc.). |
 | **Inbound ports** | `80`/`443` reachable from the internet (for HTTP/HTTPS + ACME TLS challenges), plus your chosen SSH port. |
 | **DNS** | `A`/`AAAA` records for each site domain pointing at the server's public IP **before** running `create` (needed for certificate issuance). |
 | **Resources** | Recommended ≥ 2 GB RAM (MariaDB buffer pool is pre-set to `2G` in `99-performance.cnf`) and ≥ 20 GB disk. |
 ---
 ## 3. Quick Start
 ```bash
 # 0. Copy the script to the target host and make it executable
 chmod +x wp-engine.sh
 # 1. Harden the host + deploy proxy & monitoring (interactive prompts)
 ./wp-engine.sh --bootstrap
 # 2. Provision your first WordPress site
 ./wp-engine.sh create teetimeprints teetimeprints.com mkell admin@teetimeprints.com
 ```
 **Fully unattended bootstrap** (no prompts — ideal for cloud-init / Ansible / automation):
 ```bash
 # Docker + Caddy on custom SSH port 48222
 ./wp-engine.sh --bootstrap --ssh-port 48222 --engine docker --proxy caddy
 # Docker + Traefik v3 on custom SSH port 52822
 ./wp-engine.sh --bootstrap --ssh-port 52822 --engine docker --proxy traefik
 # Podman + Nginx Proxy Manager (NPM) on custom SSH port 48222
 ./wp-engine.sh --bootstrap --ssh-port 48222 --engine podman --proxy npm
 ```
 **Install system-wide** so it can be run from anywhere:
 ```bash
 install -m 0755 wp-engine.sh /usr/local/bin/wp-engine.sh
 # ...or download it directly:
 curl -fsSL https://your-repo/wp-engine.sh -o /usr/local/bin/wp-engine.sh
 chmod +x /usr/local/bin/wp-engine.sh
 ```
 > [!WARNING]
 > **Do not close your current SSH terminal** after running `--bootstrap`. Open a secondary terminal and verify that you can connect over your chosen port before disconnecting:
 > ```bash
 > ssh -p <PORT> user@<HOST_IP>
 > ```
 ---
 ## 4. CLI Reference & Options
 ```
 Usage:
  Interactive Bootstrap:
    ./wp-engine.sh --bootstrap
  Unattended Bootstrap:
    ./wp-engine.sh --bootstrap --ssh-port <port> --engine <docker|podman> --proxy <npm|traefik|caddy>
  Provision Site:
    ./wp-engine.sh create <slug> <domain> [admin_user] [admin_email]
 Examples:
  ./wp-engine.sh --bootstrap --ssh-port 48222 --engine docker --proxy caddy
  ./wp-engine.sh create teetimeprints teetimeprints.com mkell admin@teetimeprints.com
 ```
 ### Actions
 | Action | Description |
 |---|---|
 | `--bootstrap` | Runs the 6-step host provisioning sequence, then deploys the monitoring stack. Prompts for any option not supplied via flags. |
 | `create <slug> <domain> [admin_user] [admin_email]` | Provisions a new WordPress site. If the ingress proxy isn't running yet, bootstrap runs automatically; if the shared DB/cache core isn't up, it is initialized automatically. |
 | *(no/unknown action)* | Prints the usage text above and exits with code 1. |
 ### Flags (used with `--bootstrap`)
 | Flag | Values | Default | Effect |
 |---|---|---|---|
 | `--ssh-port` | `1–65535` | `48222` | Moves SSH off port 22; written to `sshd_config.d/99-hardened.conf`, the `ssh.socket` unit (if present), UFW, and the Fail2ban `sshd` jail. |
 | `--engine` | `docker` \| `podman` | `docker` | Container runtime. Docker installs from Docker's official apt repo; Podman installs `podman`, `podman-compose`, `podman-docker` from distro repos. |
 | `--proxy` | `npm` \| `traefik` \| `caddy` | `npm` | Ingress reverse proxy. Determines the proxy compose file, extra firewall openings, TLS automation, and how sites get routed. |
 > **Note:** Flags are read before the persisted state is loaded, so they take precedence for that run and are saved into `/server/websites/.engine-settings`. Subsequent `create` commands reuse the saved choices without prompting.
 ### `create` arguments
 | Position | Argument | Default | Description |
 |---|---|---|---|
 | 1 | `slug` | *(required)* | Directory/container name, e.g. `teetimeprints`. Hyphens are converted to underscores for DB identifiers. |
 | 2 | `domain` | *(required)* | Public FQDN, e.g. `teetimeprints.com`. Used for TLS routing and the WP site URL. |
 | 3 | `admin_user` | `admin` | WordPress administrator username. |
 | 4 | `admin_email` | `admin@<domain>` | WordPress administrator e-mail. |
 Additional `create` examples:
 ```bash
 # Minimal — admin user defaults to `admin`, e-mail to admin@myblog.com
 ./wp-engine.sh create myblog myblog.com
 # Custom admin credentials
 ./wp-engine.sh create teetimeprints teetimeprints.com mkell admin@teetimeprints.com
 ```
 ---
 ## 5. Step-by-Step: What `--bootstrap` Does
 ### Phase A — Bare-host provisioning & hardening (6 steps)
 **`[1/6] Installing Host Packages`**
 Installs `apt-transport-https ca-certificates curl gnupg lsb-release ufw fail2ban nftables openssl jq`.
 **`[2/6] Setting up Container Engine`**
 - *Docker:* adds Docker's official GPG key + apt repo, installs `docker-ce`, `docker-ce-cli`, `containerd.io`, `docker-buildx-plugin`, `docker-compose-plugin`, enables the service (skipped if Docker already present).
 - *Podman:* `apt-get install podman podman-compose podman-docker`, enables the service.
 **`[3/6] Hardening SSH on Port <port>`**
 Writes `/etc/ssh/sshd_config.d/99-hardened.conf`:
 ```
 Port <chosen port>
 PermitRootLogin prohibit-password
 PasswordAuthentication yes
 X11Forwarding no
 MaxAuthTries 4
 ClientAliveInterval 300
 ClientAliveCountMax 2
 ```
 Also re-points `ssh.socket`'s `ListenStream` to the new port when socket activation is in use, then restarts SSH.
 **`[4/6] Configuring Firewall & Isolation Rules`**
 - UFW policy: **deny incoming / allow outgoing**.
 - Allowed inbound: SSH port, `80`, `443`, `3000` (Grafana), plus `81` if NPM or `8080` if Traefik.
 - For Docker, appends a **`DOCKER-USER` chain** to `/etc/ufw/after.rules` so UFW rules actually apply to container traffic (blocks direct SYN scans into `172.16.0.0/12`, rate-limited logging).
 - `ufw --force enable && ufw reload`.
 **`[5/6] Scaffolding Networks & Fail2ban Jails`**
 - Creates the directory skeleton (`/server/websites/{proxy,core,template,sites,monitoring}`).
 - Creates three external container networks: `proxy-net`, `shared-backend`, `monitoring-net`.
 - Writes Fail2ban jails:
  - **`sshd`** — your SSH port, `maxretry 3`, `findtime 600`, `bantime 3600`, `banaction ufw`.
  - **`ingress-wordpress`** — watches `/server/websites/proxy/logs/*.log` for brute force on `/wp-login.php` and `/xmlrpc.php`, plus scans for `.env`, `.git`, `wp-config.php`, `eval-stdin.php`; `maxretry 4`, `bantime 86400` (24 h), `banaction ufw`.
 - Enables and restarts Fail2ban.
 **`[6/6] Launching Ingress Proxy`** — writes and starts the proxy compose file (differences per proxy are covered in [Option Guides](#7-option-guides)), then deploys monitoring.
 ### Phase B — Observability stack (`deploy_monitoring`)
 Deploys `/server/websites/monitoring/docker-compose.yml` on the isolated `monitoring-net`:
 | Service | Image | Purpose | Exposure |
 |---|---|---|---|
 | `prometheus` | `prom/prometheus:v2.54.0` | Time-series DB; scrapes node_exporter + cAdvisor every 15 s | internal |
 | `node_exporter` | `prom/node-exporter:v1.8.2` | Host CPU/RAM/disk metrics (`/proc`, `/sys`, `/` mounted read-only) | internal |
 | `cadvisor` | `gcr.io/cadvisor/cadvisor:v0.49.1` | Per-container cgroup resource usage | internal |
 | `grafana` | `grafana/grafana:11.2.0` | Dashboards; Prometheus datasource auto-provisioned; sign-up disabled | **`:3000`** |
 A random Grafana admin password is generated (`openssl rand`) and appended to the state file as `GRAFANA_ADMIN_PASS`.
 ---
 ## 6. Step-by-Step: What `create` Does
 `./wp-engine.sh create <slug> <domain> [admin_user] [admin_email]`
 1. **Load state** — reads `/server/websites/.engine-settings` (runs bootstrap prompts only if it doesn't exist yet).
 2. **Auto-heal** — if no `ingress_proxy` container is running, bootstrap runs automatically; if `core_mariadb` is missing, the shared core is initialized.
 3. **Allocate resources**
   - Next Redis DB index from `NEXT_REDIS_INDEX` (incremented in `core/.env`).
   - DB name `wp_<slug>` and DB user `usr_<slug>` (hyphens → underscores).
   - Random DB password and random WordPress admin password (`openssl rand`).
 4. **Create the database** — `CREATE DATABASE`, `CREATE USER`, `GRANT ALL` scoped to that site's DB only, executed inside `core_mariadb`.
 5. **Write site files**
   - `config/wordpress.ini` — PHP limits: `memory_limit 512M`, `upload_max_filesize/post_max_size 128M`, `max_execution_time 300`, `max_input_vars 5000`, OPcache enabled (256 MB, 20 000 files).
   - `docker-compose.yml` — `wordpress:6.7-apache` container attached to `proxy-net` **and** `shared-backend`, with `WORDPRESS_CONFIG_EXTRA` injecting:
     - Redis connection (`core_redis`, port 6379, per-site password, per-site DB index, per-site cache key salt)
     - `DISALLOW_FILE_EDIT` (no theme/plugin editing from wp-admin)
     - `FORCE_SSL_ADMIN`
     - `X-Forwarded-Proto` HTTPS detection behind the proxy
     - Traefik routing labels (only when `--proxy traefik`)
 6. **Start the site** — `chown 33:33` on the docroot, then `compose up -d`.
 7. **Wire up routing**
   - *Caddy:* appends a `domain { reverse_proxy slug_wp:80 ... }` block to the `Caddyfile` and hot-reloads Caddy.
   - *Traefik:* handled by the labels from step 5 (router on `websecure`, cert resolver `myresolver`).
   - *NPM:* prints a reminder — create the Proxy Host manually in the NPM UI.
 8. **Automated WordPress bootstrap (WP-CLI container)**
   - Waits for `wp-config.php` to appear in the docroot.
   - `wp core install --url=https://<domain> --admin_user=... --admin_password=... --admin_email=... --skip-email`
   - `wp rewrite structure '/%postname%/' --hard`
   - `wp plugin install redis-cache --activate` → `wp redis enable`
   - `chown -R 33:33 data/wp/wp-content`
 9. **Prints a summary** — domain, admin user, **admin password (shown only once — copy it now)**, DB name, DB user, Redis index, directory path, plus any proxy-specific next step.
 ---
 ## 7. Option Guides
 ### 7.1 Container runtime
 | | **Docker** (default) | **Podman** |
 |---|---|---|
 | Install source | Docker official apt repo (`docker-ce` + compose plugin) | Distro repos (`podman`, `podman-compose`, `podman-docker`) |
 | Compose command | `docker compose` | `podman-compose` |
 | UFW integration | Gets the `DOCKER-USER` after.rules patch (required, since Docker bypasses UFW by default) | Not applied (Podman networking works differently with nftables/UFW) |
 | Root | Script still requires root for host hardening | Same — host services are hardened system-wide |
 ### 7.2 Ingress proxy
 | | **NPM** (default) | **Traefik v3** | **Caddy** |
 |---|---|---|---|
 | Image | `jc21/nginx-proxy-manager` | `traefik:v3.1` | `caddy:2-alpine` |
 | Management | Web UI on **`:81`** (opened in UFW) | Docker labels + insecure API dashboard on **`:8080`** (opened in UFW) | Text `Caddyfile` |
 | TLS certificates | Requested per host in the UI (Let's Encrypt) | ACME **TLS-ALPN challenge** via `myresolver` labels | **Fully automatic** Let's Encrypt |
 | Site onboarding | **Manual**: create Proxy Host → `https://domain` → forward to `<slug>_wp:80` | **Automatic**: labels emitted by `create` | **Automatic**: `create` appends a site block + reloads |
 | Access log | bind-mounted `proxy/logs/` | `proxy/logs/access.log` (`--accesslog`) | `proxy/logs/access.log` via `log {}` block |
 | Extra ports opened | `81` | `8080` | none |
 | Best for | Users who want a GUI | Label-driven, declarative setups | Minimal, zero-touch SSL |
 The proxy choice affects the generated `proxy/docker-compose.yml`, the UFW allow rules, and how `create` wires routing — all persisted to the state file.
 ### 7.3 SSH port
 Any non-standard port (default `48222`). Applied consistently in four places: `sshd_config.d`, `ssh.socket`, the UFW allow rule, and the Fail2ban `sshd` jail — so ban actions target the right port.
 ---
 ## 8. Architecture: Networks, Ports, Layout
 High-level traffic flow (see `docs/` for the full Mermaid diagrams):
 ```text
                        Internet / WAN
                              │
                              ▼
            ┌──────────────────────────────────┐
            │   Host Firewall (UFW / nftables) │
            │   + Fail2ban Bare-Host Jails     │
            └─────────────────┬────────────────┘
                              │ Ports 80 / 443
                              ▼
            ┌──────────────────────────────────┐
            │ Ingress: Caddy / Traefik / NPM   │ ── Access Logs ──► Host Fail2ban
            └──────────────┬───────────────────┘
                           │
       ┌───────────────────┴───────────────────┐
       ▼                                       ▼
 ┌──────────────┐                       ┌──────────────┐
 │ WordPress    │                       │ Observability│
 │  - PHP 8.3   │                       │  - Grafana   │ (Port 3000)
 │  - WP-CLI    │                       │  - Prometheus│
 │  - OPcache   │                       │  - Node Exp  │
 └──────┬───────┘                       │  - cAdvisor  │
       │                               └──────────────┘
  ┌────┴────────────┐
  ▼                 ▼
 ┌───────────┐ ┌───────────┐
 │ MariaDB   │ │ Redis     │
 │ (Shared   │ │ (Shared   │
 │  Core)    │ │  Cache)   │
 └───────────┘ └───────────┘
 ```
 ### Networks
 | Network | Scope | Members |
 |---|---|---|
 | `proxy-net` | external | ingress proxy + all WordPress containers (public-facing side) |
 | `shared-backend` | external, isolated | all WordPress containers + `core_mariadb` + `core_redis` (no host ports published) |
 | `monitoring-net` | external, isolated | Prometheus, node_exporter, cAdvisor, Grafana |
 ### Ports
 | Port | Service | Firewall |
 |---|---|---|
 | 22 → **48222** (default) | SSH | allowed (custom port) |
 | 80 / 443 | Ingress proxy (HTTP/HTTPS + ACME) | allowed |
 | 81 | Nginx Proxy Manager admin UI | allowed **only if** `--proxy npm` |
 | 8080 | Traefik dashboard | allowed **only if** `--proxy traefik` |
 | 3000 | Grafana | allowed |
 | 3306 / 6379 | MariaDB / Redis | **not published** — container network only |
 ### Filesystem layout (`/server/websites/`)
 ```
 /server/websites/
 ├── .engine-settings          # Persisted choices: ENGINE, PROXY_TYPE, SSH_PORT, GRAFANA_ADMIN_PASS
 ├── core/                     # Shared data tier
 │   ├── .env                  # DB_ROOT_PASSWORD, REDIS_PASSWORD, NEXT_REDIS_INDEX (chmod 600)
 │   ├── docker-compose.yml    # core_mariadb (MariaDB 11.4) + core_redis (Redis 7)
 │   ├── config/mariadb/99-performance.cnf
 │   └── data/{mariadb,redis}/
 ├── proxy/                    # Ingress tier
 │   ├── docker-compose.yml    # ingress_proxy container
 │   ├── Caddyfile             # (Caddy only) auto-appended per site
 │   ├── letsencrypt/          # acme.json / certs
 │   └── logs/access.log       # scanned by Fail2ban `ingress-wordpress`
 ├── monitoring/               # Observability tier
 │   ├── docker-compose.yml    # prometheus, node_exporter, cadvisor, grafana
 │   ├── prometheus/prometheus.yml
 │   └── grafana/provisioning/datasources/prometheus.yaml
 ├── sites/<slug>/             # One directory per WordPress tenant
 │   ├── docker-compose.yml    # <slug>_wp container (proxy-net + shared-backend)
 │   ├── config/wordpress.ini  # OPcache & upload limits
 │   └── data/wp/              # Full WordPress docroot (uid 33)
 └── template/                 # Reserved for site templates
 ```
 Host-level configs written by the script:
 - `/etc/ssh/sshd_config.d/99-hardened.conf`
 - `/etc/ufw/after.rules` (DOCKER-USER protection block)
 - `/etc/fail2ban/jail.d/sshd.local`
 - `/etc/fail2ban/jail.d/ingress-wordpress.local`
 - `/etc/fail2ban/filter.d/ingress-wordpress.conf`
 ---
 ## 9. Credentials & State Files
 | Secret | Where it lives | Notes |
 |---|---|---|
 | Engine / proxy / SSH port | `/server/websites/.engine-settings` | plain, non-secret |
 | Grafana admin password | `/server/websites/.engine-settings` → `GRAFANA_ADMIN_PASS` | login: `admin` @ `http://<host>:3000` |
 | MariaDB **root** password | `/server/websites/core/.env` → `DB_ROOT_PASSWORD` | `chmod 600`; used for creating per-site DBs |
 | Redis password | `/server/websites/core/.env` → `REDIS_PASSWORD` | injected into every site's `wp-config` |
 | Per-site DB password | injected into the site's compose environment | random per site |
 | **WordPress admin password** | **printed to stdout only, at the end of `create`** | **not persisted anywhere — save it immediately** |
 | Redis DB index | `core/.env` → `NEXT_REDIS_INDEX` counter | one index allocated per site (0, 1, 2, …) |
 > ⚠️ Re-running `--bootstrap` appends a **new** `GRAFANA_ADMIN_PASS` line; when the state file is sourced, the **last** line wins, effectively rotating the Grafana password.
 ---
 ## 10. Monitoring Dashboards
 After bootstrap:
 1. Open `http://<HOST_IP>:3000` (Grafana).
 2. Log in with `admin` / the password generated during bootstrap and saved in the state file:
   ```bash
   grep GRAFANA_ADMIN_PASS /server/websites/.engine-settings
   ```
 3. The **Prometheus** datasource is pre-provisioned and set as default. Import dashboards via **Dashboards → Import** — recommended IDs:
   | ID | Dashboard | Provides |
   |---|---|---|
   | `1860` | Node Exporter Full | Real-time CPU, RAM, disk I/O, network throughput, load averages |
   | `14282` | cAdvisor Docker Monitoring | Per-container resource usage; spot CPU/RAM throttling across tenant sites |
 Data collected:
 - **node_exporter** → host CPU, memory, disk I/O, network.
 - **cAdvisor** → per-container CPU/memory of every WordPress, DB, proxy, and monitoring container.
 - **Prometheus** → 15 s scrape interval, persistent `prometheus_data` volume.
 ---
 ## 11. Day-2 Operations
 ```bash
 # Add another site (no prompts — state already saved)
 ./wp-engine.sh create kellcreations kellcreations.com admin admin@kellcreations.com
 # Check running containers
 docker ps            # or: podman ps
 # Site lifecycle
 cd /server/websites/sites/<slug>
 docker compose up -d / stop / restart / logs -f
 # Core stack & proxy
 cd /server/websites/core  && docker compose ps
 cd /server/websites/proxy && docker compose logs -f ingress
 # WP-CLI inside a site (one-off)
 docker run --rm --user 33:33 --network shared-backend \
  -v /server/websites/sites/<slug>/data/wp:/var/www/html wordpress:cli wp plugin list
 # Logs for a specific site
 docker logs -f <slug>_wp            # or: podman logs -f <slug>_wp
 # Unban an IP address
 fail2ban-client set ingress-wordpress unbanip <IP_ADDRESS>
 # Security status
 ufw status verbose
 fail2ban-client status
 fail2ban-client status ingress-wordpress
 fail2ban-client status sshd
 fail2ban-client get ingress-wordpress banned
 ```
 **Backups** (not automated by the script — recommended):
 - `/server/websites/core/data/mariadb/` — all site databases (stop MariaDB or use `mariadb-dump` for consistency).
 - `/server/websites/sites/*/data/wp/` — WordPress docroots (uploads, themes, plugins).
 - `/server/websites/core/.env` and `.engine-settings` — credentials/state.
 ---
 ## 12. Security Hardening Summary
 | Layer | Control |
 |---|---|
 | **Access** | SSH moved off port 22; root login limited to key auth (`PermitRootLogin prohibit-password`); `MaxAuthTries 4`; X11 forwarding off; idle timeout via `ClientAliveInterval`/`ClientAliveCountMax` |
 | **Perimeter** | UFW default-deny inbound; explicit allow-list (SSH, 80, 443, 3000, +81/8080 per proxy) |
 | **Container escape path** | `DOCKER-USER` nftables chain in `/etc/ufw/after.rules` so UFW governs Docker-bridged traffic; direct scans into RFC1918 container ranges are logged + dropped |
 | **Brute force** | Fail2ban `sshd` jail (3 attempts → 1 h ban) and `ingress-wordpress` jail (4 attempts on `wp-login.php`/`xmlrpc.php`/sensitive-file scans → 24 h ban via UFW), fed by proxy access logs |
 | **WordPress** | `DISALLOW_FILE_EDIT`, `FORCE_SSL_ADMIN`, HTTPS detection behind proxy, randomized admin/DB passwords, per-site scoped DB grants (no shared credentials) |
 | **Secrets** | Root/Redis passwords generated with `openssl rand`; `core/.env` set to `chmod 600`; Grafana sign-up disabled |
 | **Isolation** | Each site gets its own DB, DB user, Redis index, cache key salt, docroot, and compose project — on a shared-but-segmented backend network with **no published host ports** |
 | **Observability** | Centralized access logs → Fail2ban; Prometheus metrics for capacity/anomaly detection |
 ---
 ## 13. Known Limitations & Post-Install TODOs
 Behaviors of the current script — review and adjust to your threat model:
 1. **NPM requires a manual step** — after `create`, log in to `http://<host>:81` (NPM default login) and add a Proxy Host forwarding `https://<domain>` → `<slug>_wp:80` with an SSL certificate. Change the NPM default password immediately.
 2. **Traefik ACME email is a placeholder** — the generated compose file hardcodes `admin@example.com`. Edit `/server/websites/proxy/docker-compose.yml` before the first certificate request.
 3. **Traefik dashboard is exposed** — `--api.insecure=true` on `:8080` and UFW allows it publicly. Restrict or close it if you don't need it.
 4. **NPM admin UI (`:81`) is exposed publicly** when `--proxy npm` is selected. Consider restricting it to your IP.
 5. **Grafana (`:3000`) is served over plain HTTP** with no TLS and is firewall-open. Put it behind the proxy with HTTPS, or restrict the firewall rule to your IP.
 6. **`PasswordAuthentication yes`** remains enabled in the SSH hardening block — convenient for first-time migration, but switch to keys-only (`no`) once your key is installed.
 7. **Single MariaDB/Redis instance** — `innodb_buffer_pool_size = 2G` is a fixed default; tune `core/config/mariadb/99-performance.cnf` for smaller/larger hosts.
 8. **Distro scope** — apt/systemd only. Non-Debian hosts need manual adaptation of steps 1 and 4.
 9. **No automated backups or WAF** — add ModSecurity/Cloudflare or scheduled backups per your requirements.
 10. **WP admin password is ephemeral** — captured only in the terminal output of `create`; store it in your password manager right away.
 ---
 ## Repository Contents
 ```
 wp-engine.sh                                   # The installer/provisioner script (executable)
 docs/Runtime_Architecture_Network_Topology.md  # Architecture & network Mermaid diagram
 docs/Filesystem_and_Directory_Structure.md     # Filesystem layout Mermaid diagram
 README.md                                      # This file
 ```
--- a/docs/Filesystem_and_Directory_Structure.md
+++ b/docs/Filesystem_and_Directory_Structure.md
@ -0,0 +1,99 @@
 # Filesystem & Directory Structure
 ```mermaid
 graph TD
    Root["/server/websites/"]
    %% Root Configuration & State Files
    StateFile[".engine-settings\n(Persisted: RUNTIME, PROXY_TYPE, SSH_PORT, GRAFANA_PASS)"]
    ScriptExec["/usr/local/bin/wp-engine.sh\n(Self-contained installer & provisioner)"]
    Root --- StateFile
    Root --- ScriptExec
    %% Core Shared Stack
    subgraph CoreDir["/server/websites/core/"]
        CoreEnv[".env (DB_ROOT_PASSWORD, REDIS_PASSWORD, NEXT_REDIS_INDEX)"]
        CoreCompose["docker-compose.yml (core_mariadb + core_redis)"]
        CoreConfig["config/mariadb/99-performance.cnf"]
        CoreDataMariaDB[("data/mariadb/ (Persistent MySQL data)")]
        CoreDataRedis[("data/redis/ (Persistent Redis dump)")]
        CoreEnv --- CoreCompose
        CoreCompose --- CoreConfig
        CoreCompose --- CoreDataMariaDB
        CoreCompose --- CoreDataRedis
    end
    Root --> CoreDir
    %% Proxy Stack
    subgraph ProxyDir["/server/websites/proxy/"]
        ProxyCompose["docker-compose.yml (ingress_proxy)"]
        ProxyData["data/ (NPM state / Caddy data)"]
        ProxyLE["letsencrypt/ (acme.json / certs)"]
        ProxyLogs["logs/access.log (Scanned by bare-host Fail2ban)"]
        ProxyCaddy["Caddyfile (If Caddy selected; dynamically updated)"]
        ProxyCompose --- ProxyData
        ProxyCompose --- ProxyLE
        ProxyCompose --- ProxyLogs
        ProxyCompose --- ProxyCaddy
    end
    Root --> ProxyDir
    %% Monitoring Stack
    subgraph MonDir["/server/websites/monitoring/"]
        MonCompose["docker-compose.yml (Prometheus, Node Exp, cAdvisor, Grafana)"]
        PromConfig["prometheus/prometheus.yml"]
        GrafanaProv["grafana/provisioning/datasources/prometheus.yaml"]
        MonData["prometheus_data / grafana_data (Named Volumes)"]
        MonCompose --- PromConfig
        MonCompose --- GrafanaProv
        MonCompose --- MonData
    end
    Root --> MonDir
    %% Template Stack
    subgraph TemplateDir["/server/websites/template/"]
        SiteTemplate["docker-compose.site.yml (Standard WP Compose definition)"]
    end
    Root --> TemplateDir
    %% Tenant Sites
    subgraph SitesDir["/server/websites/sites/"]
        subgraph SiteA["mysite/"]
            SiteAEnv[".env (Scoped DB user, password, Redis DB index)"]
            SiteACompose["docker-compose.yml"]
            SiteAConfig["config/wordpress.ini (OPcache, upload limits)"]
            SiteAData[("data/wp/ (Full WordPress document root)")]
            SiteAEnv --- SiteACompose
            SiteACompose --- SiteAConfig
            SiteACompose --- SiteAData
        end
        subgraph SiteB["teetimeprints/"]
            SiteBCompose["... (Identical isolated structure)"]
        end
    end
    Root --> SitesDir
    %% Host-level configurations
    subgraph HostConfig["Host-Level Configs (/etc/)"]
        F2BJail["/etc/fail2ban/jail.d/ingress-wordpress.local"]
        F2BFilter["/etc/fail2ban/filter.d/ingress-wordpress.conf"]
        UFWChain["/etc/ufw/after.rules (DOCKER-USER isolation chain)"]
        SSHDDropin["/etc/ssh/sshd_config.d/99-hardened.conf"]
    end
    ScriptExec -.->|Deploys & Controls| HostConfig
    %% Styling
    classDef dir fill:#eceff1,stroke:#455a64,stroke-width:2px,color:#263238;
    classDef compose fill:#e1f5fe,stroke:#0288d1,stroke-width:1px,color:#01579b;
    classDef data fill:#fff8e1,stroke:#ffa000,stroke-width:1px,color:#ff6f00;
    class Root,CoreDir,ProxyDir,MonDir,TemplateDir,SitesDir,SiteA,SiteB dir;
    class CoreCompose,ProxyCompose,MonCompose,SiteTemplate,SiteACompose compose;
    class CoreDataMariaDB,CoreDataRedis,SiteAData,MonData data;
    ```
--- a/docs/Runtime_Architecture_Network_Topology.md
+++ b/docs/Runtime_Architecture_Network_Topology.md
@ -0,0 +1,84 @@
 # Runtime Architecture & Network Topology
 ```mermaid
 flowchart TD
    %% Internet & External Ingress
    User([WAN / Client Traffic]) -->|Port 80 / 443| IngressRule
    Attacker([Malicious Actor / Brute Force]) -.->|wp-login / xmlrpc / exploits| IngressRule
    %% Bare Host Hardening & Security Layer
    subgraph BareHost["Bare Proxmox / Debian Host"]
        IngressRule["Host Firewall: nftables / UFW\n(Deny All Incoming except 80, 443, 3000, 48222)"]
        SSHD["Hardened OpenSSH\n(Port 48222)"]
        F2B["Fail2ban Engine\n(jail: ingress-wordpress, sshd)"]
        LogWatcher[("/server/websites/proxy/logs/*.log")]
        IngressRule -->|Port 48222| SSHD
        LogWatcher -->|Inspects Access Logs| F2B
        F2B -->|Bans IP via UFW / nftables| IngressRule
    end
    %% Ingress Proxy Tier
    subgraph NetProxy["Docker / Podman Network: proxy-net (External)"]
        IngressProxy["Ingress Reverse Proxy\n(Caddy / Traefik v3 / NPM)"]
        IngressRule -->|Forward 80/443| IngressProxy
        IngressProxy -.->|Writes Client Access Logs| LogWatcher
    end
    %% WordPress Multi-Tenant Tier
    subgraph NetBackend["Docker / Podman Network: shared-backend (Isolated)"]
        subgraph Tenant1["Tenant: Site 1 (e.g. teetimeprints)"]
            WP1["WordPress 6.7 Container\n(PHP 8.3 / Apache)"]
            RedisDropin1["Redis Object Cache Drop-in\n(Key Salt: teetimeprints_)"]
            WP1 --- RedisDropin1
        end
        subgraph Tenant2["Tenant: Site N (e.g. kellcreations)"]
            WPN["WordPress Container\n(PHP 8.3 / Apache)"]
            RedisDropinN["Redis Object Cache Drop-in\n(Key Salt: siteN_)"]
            WPN --- RedisDropinN
        end
        %% Centralized Core Data Tier
        subgraph CoreData["Centralized Core Stack"]
            CoreDB[("core_mariadb\n(MariaDB 11.4)\n- Isolated DB per site\n- Scoped DB user per site\n- Buffer Pool: 2GB")]
            CoreRedis[("core_redis\n(Redis 7 Alpine)\n- DB Index 0..15 per site\n- Maxmemory: 512MB / LRU")]
        end
        WP1 -->|MySQL: Port 3306| CoreDB
        WP1 -->|TCP: Port 6379 / DB 0| CoreRedis
        WPN -->|MySQL: Port 3306| CoreDB
        WPN -->|TCP: Port 6379 / DB N| CoreRedis
    end
    %% Proxy to WP App Connections
    IngressProxy -->|Reverse Proxy / HTTPS Term| WP1
    IngressProxy -->|Reverse Proxy / HTTPS Term| WPN
    %% Observability Stack
    subgraph NetMon["Docker / Podman Network: monitoring-net (Isolated)"]
        NodeExp["node_exporter:9100\n(Host CPU, RAM, Disk I/O)"]
        CAdvisor["cadvisor:8080\n(Per-container cgroup usage)"]
        Prometheus[("prometheus:9090\n(Time Series DB)")]
        Grafana["grafana:3000\n(Pre-provisioned Dashboards)"]
        NodeExp -->|Scraped| Prometheus
        CAdvisor -->|Scraped| Prometheus
        Prometheus -->|Datasource| Grafana
    end
    IngressRule -->|Port 3000| Grafana
    %% Styling
    classDef security fill:#ffebee,stroke:#c62828,stroke-width:2px,color:#b71c1c;
    classDef proxy fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#1b5e20;
    classDef app fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#0d47a1;
    classDef db fill:#fff3e0,stroke:#e65100,stroke-width:2px,color:#e65100;
    classDef monitor fill:#f3e5f5,stroke:#6a1b9a,stroke-width:2px,color:#4a148c;
    class F2B,IngressRule,SSHD security;
    class IngressProxy proxy;
    class WP1,WPN app;
    class CoreDB,CoreRedis db;
    class Grafana,Prometheus,NodeExp,CAdvisor monitor;
    ```
--- a/wp-engine.sh
+++ b/wp-engine.sh
@ -0,0 +1,794 @@
 #!/usr/bin/env bash
 set -euo pipefail
 BASE_DIR="/server/websites"
 PROXY_DIR="${BASE_DIR}/proxy"
 CORE_DIR="${BASE_DIR}/core"
 TEMPLATE_DIR="${BASE_DIR}/template"
 SITES_DIR="${BASE_DIR}/sites"
 MONITORING_DIR="${BASE_DIR}/monitoring"
 STATE_FILE="${BASE_DIR}/.engine-settings"
 PROXY_NETWORK="proxy-net"
 BACKEND_NETWORK="shared-backend"
 MONITORING_NETWORK="monitoring-net"
 if [ "$EUID" -ne 0 ]; then
  echo "[!] Error: Root privileges required." >&2
  exit 1
 fi
 # ==============================================================================
 # PROMPT & CONFIGURATION HELPERS
 # ==============================================================================
 prompt_config() {
  if [ -z "${SSH_PORT_ARG:-}" ]; then
    read -rp "Enter non-standard SSH port [Default: 48222]: " SSH_PORT_ARG
    SSH_PORT_ARG=${SSH_PORT_ARG:-48222}
  fi
  if [ -z "${ENGINE_ARG:-}" ]; then
    echo "Select Container Runtime:"
    echo "  1) Docker"
    echo "  2) Podman"
    read -rp "Choice [1-2, Default: 1]: " engine_choice
    case "${engine_choice}" in
      2) ENGINE_ARG="podman" ;;
      *) ENGINE_ARG="docker" ;;
    esac
  fi
  if [ -z "${PROXY_ARG:-}" ]; then
    echo "Select Ingress Proxy:"
    echo "  1) Nginx Proxy Manager (NPM - Web UI on :81)"
    echo "  2) Traefik v3 (Label discovery)"
    echo "  3) Caddy (Automated SSL & lightweight Caddyfile)"
    read -rp "Choice [1-3, Default: 1]: " proxy_choice
    case "${proxy_choice}" in
      2) PROXY_ARG="traefik" ;;
      3) PROXY_ARG="caddy" ;;
      *) PROXY_ARG="npm" ;;
    esac
  fi
  mkdir -p "${BASE_DIR}"
  cat <<EOF > "${STATE_FILE}"
 ENGINE="${ENGINE_ARG}"
 PROXY_TYPE="${PROXY_ARG}"
 SSH_PORT="${SSH_PORT_ARG}"
 EOF
 }
 load_state() {
  if [ -f "${STATE_FILE}" ]; then
    # shellcheck source=/dev/null
    source "${STATE_FILE}"
  else
    echo "[!] System uninitialized. Running setup prompts..."
    prompt_config
    # shellcheck source=/dev/null
    source "${STATE_FILE}"
  fi
 }
 compose_exec() {
  local dir="$1"
  shift
  if [ "${ENGINE}" = "podman" ]; then
    (cd "${dir}" && podman-compose "$@")
  else
    (cd "${dir}" && docker compose "$@")
  fi
 }
 cli_exec() {
  if [ "${ENGINE}" = "podman" ]; then
    podman "$@"
  else
    docker "$@"
  fi
 }
 # ==============================================================================
 # 1. BARE-HOST PROVISIONING & HARDENING
 # ==============================================================================
 run_host_bootstrap() {
  echo "=========================================================="
  echo " [1/6] Installing Host Packages"
  echo "=========================================================="
  apt-get update -y
  apt-get install -y apt-transport-https ca-certificates curl gnupg \
    lsb-release ufw fail2ban nftables openssl jq
  echo "=========================================================="
  echo " [2/6] Setting up Container Engine: ${ENGINE}"
  echo "=========================================================="
  if [ "${ENGINE}" = "podman" ]; then
    apt-get install -y podman podman-compose podman-docker
    systemctl enable --now podman
  else
    if ! command -v docker &>/dev/null; then
      install -m 0755 -d /etc/apt/keyrings
      curl -fsSL "https://download.docker.com/linux/$(. /etc/os-release && echo "$ID")/gpg" | gpg --dearmor --yes -o /etc/apt/keyrings/docker.gpg
      chmod a+r /etc/apt/keyrings/docker.gpg
      echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/$(. /etc/os-release && echo "$ID") $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
      apt-get update -y
      apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
      systemctl enable --now docker
    fi
  fi
  echo "=========================================================="
  echo " [3/6] Hardening SSH on Port ${SSH_PORT}"
  echo "=========================================================="
  if systemctl is-active --quiet ssh.socket; then
    sed -i -E "s/^ListenStream=.*/ListenStream=${SSH_PORT}/" /lib/systemd/system/ssh.socket || true
    systemctl daemon-reload
    systemctl restart ssh.socket
  fi
  mkdir -p /etc/ssh/sshd_config.d/
  cat <<EOF > /etc/ssh/sshd_config.d/99-hardened.conf
 Port ${SSH_PORT}
 PermitRootLogin prohibit-password
 PasswordAuthentication yes
 X11Forwarding no
 MaxAuthTries 4
 ClientAliveInterval 300
 ClientAliveCountMax 2
 EOF
  systemctl restart ssh || systemctl restart sshd
  echo "=========================================================="
  echo " [4/6] Configuring Firewall & Isolation Rules"
  echo "=========================================================="
  ufw default deny incoming
  ufw default allow outgoing
  ufw allow "${SSH_PORT}"/tcp comment 'Hardened SSH'
  ufw allow 80/tcp comment 'HTTP Ingress'
  ufw allow 443/tcp comment 'HTTPS Ingress'
  [ "${PROXY_TYPE}" = "npm" ] && ufw allow 81/tcp comment 'NPM Admin'
  [ "${PROXY_TYPE}" = "traefik" ] && ufw allow 8080/tcp comment 'Traefik Dashboard'
  ufw allow 3000/tcp comment 'Grafana Dashboard'
  if [ "${ENGINE}" = "docker" ] && ! grep -q "BEGIN UFW AND DOCKER" /etc/ufw/after.rules; then
    cat <<'EOF' >> /etc/ufw/after.rules
 # ================================
 # BEGIN UFW AND DOCKER PROTECTION
 # ================================
 *filter
 :ufw-user-forward - [0:0]
 :DOCKER-USER - [0:0]
 -A DOCKER-USER -j ufw-user-forward
 -A DOCKER-USER -j RETURN -s 10.0.0.0/8
 -A DOCKER-USER -j RETURN -s 172.16.0.0/12
 -A DOCKER-USER -j RETURN -s 192.168.0.0/16
 -A DOCKER-USER -p udp -m udp --sport 53 --dport 1024:65535 -j RETURN
 -A DOCKER-USER -j ufw-docker-logging-deny -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -d 172.16.0.0/12
 -A DOCKER-USER -j RETURN
 -A ufw-docker-logging-deny -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix "[UFW DOCKER BLOCK] "
 -A ufw-docker-logging-deny -j DROP
 COMMIT
 # ================================
 # END UFW AND DOCKER PROTECTION
 # ================================
 EOF
  fi
  ufw --force enable
  ufw reload
  echo "=========================================================="
  echo " [5/6] Scaffolding Networks & Fail2ban Jails"
  echo "=========================================================="
  mkdir -p "${PROXY_DIR}/logs" "${CORE_DIR}/data/mariadb" "${CORE_DIR}/data/redis" "${CORE_DIR}/config/mariadb"
  mkdir -p "${TEMPLATE_DIR}" "${SITES_DIR}" "${MONITORING_DIR}"
  cli_exec network inspect "${PROXY_NETWORK}" >/dev/null 2>&1 || cli_exec network create "${PROXY_NETWORK}"
  cli_exec network inspect "${BACKEND_NETWORK}" >/dev/null 2>&1 || cli_exec network create "${BACKEND_NETWORK}"
  cli_exec network inspect "${MONITORING_NETWORK}" >/dev/null 2>&1 || cli_exec network create "${MONITORING_NETWORK}"
  touch "${PROXY_DIR}/logs/access.log"
  cat <<EOF > /etc/fail2ban/jail.d/sshd.local
 [sshd]
 enabled   = true
 port      = ${SSH_PORT}
 filter    = sshd
 maxretry  = 3
 findtime  = 600
 bantime   = 3600
 banaction = ufw
 EOF
  cat <<'EOF' > /etc/fail2ban/filter.d/ingress-wordpress.conf
 [Definition]
 failregex = ^<HOST>.*"(POST|GET) /wp-login\.php.*" (401|403|200)
            ^<HOST>.*"(POST|GET) /xmlrpc\.php.*" (401|403|200)
            ^<HOST>.*"GET .*(?:\.env|\.git|wp-config\.php|eval-stdin\.php).*".*
 ignoreregex =
 EOF
  cat <<EOF > /etc/fail2ban/jail.d/ingress-wordpress.local
 [ingress-wordpress]
 enabled   = true
 port      = http,https
 filter    = ingress-wordpress
 logpath   = ${PROXY_DIR}/logs/*.log
 maxretry  = 4
 findtime  = 600
 bantime   = 86400
 banaction = ufw
 EOF
  systemctl enable --now fail2ban
  systemctl restart fail2ban
  echo "=========================================================="
  echo " [6/6] Launching Ingress Proxy: ${PROXY_TYPE}"
  echo "=========================================================="
  if [ "${PROXY_TYPE}" = "npm" ]; then
    mkdir -p "${PROXY_DIR}/data" "${PROXY_DIR}/letsencrypt"
    cat <<EOF > "${PROXY_DIR}/docker-compose.yml"
 services:
  ingress:
    image: 'jc21/nginx-proxy-manager:latest'
    container_name: ingress_proxy
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
      - ./logs:/data/logs
    networks:
      - ${PROXY_NETWORK}
 networks:
  ${PROXY_NETWORK}:
    external: true
 EOF
  elif [ "${PROXY_TYPE}" = "traefik" ]; then
    mkdir -p "${PROXY_DIR}/letsencrypt"
    touch "${PROXY_DIR}/letsencrypt/acme.json"
    chmod 600 "${PROXY_DIR}/letsencrypt/acme.json"
    cat <<EOF > "${PROXY_DIR}/docker-compose.yml"
 services:
  ingress:
    image: traefik:v3.1
    container_name: ingress_proxy
    restart: unless-stopped
    command:
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entryPoints.web.address=:80"
      - "--entryPoints.websecure.address=:443"
      - "--certificatesresolvers.myresolver.acme.tlschallenge=true"
      - "--certificatesresolvers.myresolver.acme.email=admin@example.com"
      - "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
      - "--accesslog=true"
      - "--accesslog.filepath=/logs/access.log"
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./letsencrypt:/letsencrypt
      - ./logs:/logs
    networks:
      - ${PROXY_NETWORK}
 networks:
  ${PROXY_NETWORK}:
    external: true
 EOF
  elif [ "${PROXY_TYPE}" = "caddy" ]; then
    mkdir -p "${PROXY_DIR}/data" "${PROXY_DIR}/config"
    touch "${PROXY_DIR}/Caddyfile"
    cat <<EOF > "${PROXY_DIR}/docker-compose.yml"
 services:
  ingress:
    image: caddy:2-alpine
    container_name: ingress_proxy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./data:/data
      - ./config:/config
      - ./logs:/logs
    networks:
      - ${PROXY_NETWORK}
 networks:
  ${PROXY_NETWORK}:
    external: true
 EOF
  fi
  compose_exec "${PROXY_DIR}" up -d
  deploy_monitoring
 }
 # ==============================================================================
 # 2. GRAFANA & OBSERVABILITY DEPLOYMENT
 # ==============================================================================
 deploy_monitoring() {
  echo "=========================================================="
  echo " Deploying Observability Stack (Grafana + Prometheus + cAdvisor)"
  echo "=========================================================="
  mkdir -p "${MONITORING_DIR}/prometheus" "${MONITORING_DIR}/grafana/provisioning/datasources"
  # Prometheus Scrape Configuration
  cat <<'EOF' > "${MONITORING_DIR}/prometheus/prometheus.yml"
 global:
  scrape_interval: 15s
  evaluation_interval: 15s
 scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node_exporter:9100']
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']
 EOF
  # Grafana Datasource Auto-provisioning
  cat <<'EOF' > "${MONITORING_DIR}/grafana/provisioning/datasources/prometheus.yaml"
 apiVersion: 1
 datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
 EOF
  GRAFANA_SEC_PASS=$(openssl rand -base64 16 | tr -dc 'a-zA-Z0-9')
  echo "GRAFANA_ADMIN_PASS=${GRAFANA_SEC_PASS}" >> "${STATE_FILE}"
  cat <<EOF > "${MONITORING_DIR}/docker-compose.yml"
 services:
  prometheus:
    image: prom/prometheus:v2.54.0
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    networks:
      - ${MONITORING_NETWORK}
  node_exporter:
    image: prom/node-exporter:v1.8.2
    container_name: node_exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
    networks:
      - ${MONITORING_NETWORK}
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    container_name: cadvisor
    restart: unless-stopped
    privileged: true
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    networks:
      - ${MONITORING_NETWORK}
  grafana:
    image: grafana/grafana:11.2.0
    container_name: grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_SEC_PASS}
      - GF_USERS_ALLOW_SIGN_UP=false
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
    networks:
      - ${MONITORING_NETWORK}
 volumes:
  prometheus_data:
  grafana_data:
 networks:
  ${MONITORING_NETWORK}:
    external: true
 EOF
  compose_exec "${MONITORING_DIR}" up -d
  echo "==> Monitoring stack operational. Grafana UI: http://<HOST_IP>:3000"
 }
 # ==============================================================================
 # 3. SHARED DATABASE & OBJECT CACHE CORE
 # ==============================================================================
 init_shared_core() {
  echo "==> Initializing centralized MariaDB & Redis services..."
  mkdir -p "${CORE_DIR}/data/mariadb" "${CORE_DIR}/data/redis" "${CORE_DIR}/config/mariadb"
  CORE_DB_ROOT_PW=$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9')
  CORE_REDIS_PW=$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9')
  cat <<EOF > "${CORE_DIR}/.env"
 DB_ROOT_PASSWORD=${CORE_DB_ROOT_PW}
 REDIS_PASSWORD=${CORE_REDIS_PW}
 NEXT_REDIS_INDEX=0
 EOF
  chmod 600 "${CORE_DIR}/.env"
  cat <<'EOF' > "${CORE_DIR}/config/mariadb/99-performance.cnf"
 [mysqld]
 character-set-server           = utf8mb4
 collation-server               = utf8mb4_unicode_ci
 innodb_buffer_pool_size        = 2G
 innodb_log_file_size           = 256M
 innodb_flush_log_at_trx_commit = 2
 innodb_flush_method            = O_DIRECT
 max_connections                = 300
 query_cache_type               = 0
 query_cache_size               = 0
 EOF
  cat <<EOF > "${CORE_DIR}/docker-compose.yml"
 services:
  core_db:
    image: mariadb:11.4
    container_name: core_mariadb
    restart: unless-stopped
    environment:
      MYSQL_ROOT_PASSWORD: \${DB_ROOT_PASSWORD}
    volumes:
      - ./data/mariadb:/var/lib/mysql
      - ./config/mariadb/99-performance.cnf:/etc/mysql/conf.d/99-performance.cnf:ro
    networks:
      - ${BACKEND_NETWORK}
    healthcheck:
      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
      interval: 10s
      timeout: 5s
      retries: 5
  core_redis:
    image: redis:7-alpine
    container_name: core_redis
    restart: unless-stopped
    command: >
      redis-server 
      --requirepass \${REDIS_PASSWORD}
      --maxmemory 512mb 
      --maxmemory-policy allkeys-lru 
      --save ""
    volumes:
      - ./data/redis:/data
    networks:
      - ${BACKEND_NETWORK}
 networks:
  ${BACKEND_NETWORK}:
    external: true
 EOF
  compose_exec "${CORE_DIR}" up -d
  echo "==> Waiting for MariaDB health status..."
  until cli_exec exec core_mariadb mariadb-admin ping -u root -p"${CORE_DB_ROOT_PW}" --silent >/dev/null 2>&1; do
    sleep 2
  done
  echo "==> Core database and cache online."
 }
 # ==============================================================================
 # 4. WORDPRESS SITE SCALING & PROVISIONING
 # ==============================================================================
 provision_site() {
  local SITE_SLUG="${1:-}"
  local DOMAIN="${2:-}"
  local ADMIN_USER="${3:-admin}"
  local ADMIN_EMAIL="${4:-admin@${DOMAIN}}"
  if [ -z "${SITE_SLUG}" ] || [ -z "${DOMAIN}" ]; then
    echo "Usage: $0 create <site-slug> <domain.com> [admin_user] [admin_email]"
    exit 1
  fi
  load_state
  if ! cli_exec ps --format '{{.Names}}' | grep -q "ingress_proxy"; then
    run_host_bootstrap
  fi
  if [ ! -f "${CORE_DIR}/.env" ] || ! cli_exec ps --format '{{.Names}}' | grep -q "core_mariadb"; then
    init_shared_core
  fi
  # shellcheck source=/dev/null
  source "${CORE_DIR}/.env"
  local SITE_DIR="${SITES_DIR}/${SITE_SLUG}"
  if [ -d "${SITE_DIR}" ]; then
    echo "[!] Error: Destination directory '${SITE_DIR}' already exists." >&2
    exit 1
  fi
  local REDIS_INDEX="${NEXT_REDIS_INDEX:-0}"
  local NEW_NEXT_INDEX=$((REDIS_INDEX + 1))
  sed -i "s/^NEXT_REDIS_INDEX=.*/NEXT_REDIS_INDEX=${NEW_NEXT_INDEX}/" "${CORE_DIR}/.env"
  local DB_NAME="wp_${SITE_SLUG//-/_}"
  local DB_USER="usr_${SITE_SLUG//-/_}"
  local DB_PASS=$(openssl rand -base64 24 | tr -dc 'a-zA-Z0-9')
  local WP_ADMIN_PASS=$(openssl rand -base64 18 | tr -dc 'a-zA-Z0-9!@#%^&*')
  echo "==> Provisioning database credentials..."
  cli_exec exec -i core_mariadb mariadb -u root -p"${DB_ROOT_PASSWORD}" <<EOF
 CREATE DATABASE IF NOT EXISTS \`${DB_NAME}\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
 CREATE USER IF NOT EXISTS '${DB_USER}'@'\%' IDENTIFIED BY '${DB_PASS}';
 GRANT ALL PRIVILEGES ON \`${DB_NAME}\`.* TO '${DB_USER}'@'%';
 FLUSH PRIVILEGES;
 EOF
  mkdir -p "${SITE_DIR}/data/wp" "${SITE_DIR}/config"
  cat <<'EOF' > "${SITE_DIR}/config/wordpress.ini"
 file_uploads = On
 memory_limit = 512M
 upload_max_filesize = 128M
 post_max_size = 128M
 max_execution_time = 300
 max_input_vars = 5000
 opcache.enable = 1
 opcache.memory_consumption = 256
 opcache.max_accelerated_files = 20000
 opcache.revalidate_freq = 60
 EOF
  # Generate compose file based on chosen proxy
  if [ "${PROXY_TYPE}" = "traefik" ]; then
    cat <<EOF > "${SITE_DIR}/docker-compose.yml"
 services:
  wordpress:
    image: wordpress:6.7-apache
    container_name: ${SITE_SLUG}_wp
    restart: unless-stopped
    environment:
      WORDPRESS_DB_HOST: core_mariadb:3306
      WORDPRESS_DB_NAME: ${DB_NAME}
      WORDPRESS_DB_USER: ${DB_USER}
      WORDPRESS_DB_PASSWORD: ${DB_PASS}
      WORDPRESS_TABLE_PREFIX: wp_
      WORDPRESS_CONFIG_EXTRA: |
        define('WP_REDIS_HOST', 'core_redis');
        define('WP_REDIS_PORT', 6379);
        define('WP_REDIS_PASSWORD', '${REDIS_PASSWORD}');
        define('WP_REDIS_DATABASE', ${REDIS_INDEX});
        define('WP_CACHE_KEY_SALT', '${SITE_SLUG}_');
        define('WP_CACHE', true);
        define('DISALLOW_FILE_EDIT', true);
        define('FORCE_SSL_ADMIN', true);
        if (isset(\$_SERVER['HTTP_X_FORWARDED_PROTO']) && \$_SERVER['HTTP_X_FORWARDED_PROTO'] === 'https') {
            \$_SERVER['HTTPS'] = 'on';
        }
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.${SITE_SLUG}.rule=Host(\`${DOMAIN}\`)"
      - "traefik.http.routers.${SITE_SLUG}.entrypoints=websecure"
      - "traefik.http.routers.${SITE_SLUG}.tls.certresolver=myresolver"
      - "traefik.http.services.${SITE_SLUG}.loadbalancer.server.port=80"
    volumes:
      - ./data/wp:/var/www/html
      - ./config/wordpress.ini:/usr/local/etc/php/conf.d/wordpress.ini:ro
    networks:
      - ${PROXY_NETWORK}
      - ${BACKEND_NETWORK}
 networks:
  ${PROXY_NETWORK}:
    external: true
  ${BACKEND_NETWORK}:
    external: true
 EOF
  else
    # NPM and Caddy use standard ingress-connected containers
    cat <<EOF > "${SITE_DIR}/docker-compose.yml"
 services:
  wordpress:
    image: wordpress:6.7-apache
    container_name: ${SITE_SLUG}_wp
    restart: unless-stopped
    environment:
      WORDPRESS_DB_HOST: core_mariadb:3306
      WORDPRESS_DB_NAME: ${DB_NAME}
      WORDPRESS_DB_USER: ${DB_USER}
      WORDPRESS_DB_PASSWORD: ${DB_PASS}
      WORDPRESS_TABLE_PREFIX: wp_
      WORDPRESS_CONFIG_EXTRA: |
        define('WP_REDIS_HOST', 'core_redis');
        define('WP_REDIS_PORT', 6379);
        define('WP_REDIS_PASSWORD', '${REDIS_PASSWORD}');
        define('WP_REDIS_DATABASE', ${REDIS_INDEX});
        define('WP_CACHE_KEY_SALT', '${SITE_SLUG}_');
        define('WP_CACHE', true);
        define('DISALLOW_FILE_EDIT', true);
        define('FORCE_SSL_ADMIN', true);
        if (isset(\$_SERVER['HTTP_X_FORWARDED_PROTO']) && \$_SERVER['HTTP_X_FORWARDED_PROTO'] === 'https') {
            \$_SERVER['HTTPS'] = 'on';
        }
    volumes:
      - ./data/wp:/var/www/html
      - ./config/wordpress.ini:/usr/local/etc/php/conf.d/wordpress.ini:ro
    networks:
      - ${PROXY_NETWORK}
      - ${BACKEND_NETWORK}
 networks:
  ${PROXY_NETWORK}:
    external: true
  ${BACKEND_NETWORK}:
    external: true
 EOF
  fi
  chown -R 33:33 "${SITE_DIR}/data/wp"
  compose_exec "${SITE_DIR}" up -d
  # Automatic Caddy routing generation
  if [ "${PROXY_TYPE}" = "caddy" ]; then
    echo "==> Injecting ${DOMAIN} entry into Caddyfile..."
    cat <<EOF >> "${PROXY_DIR}/Caddyfile"
 ${DOMAIN} {
    reverse_proxy ${SITE_SLUG}_wp:80
    log {
        output file /logs/access.log
    }
 }
 EOF
    compose_exec "${PROXY_DIR}" exec ingress caddy reload --config /etc/caddy/Caddyfile || true
  fi
  echo "==> Awaiting core WordPress file deployment..."
  until [ -f "${SITE_DIR}/data/wp/wp-config.php" ]; do
    sleep 2
  done
  echo "==> Running automated WP-CLI bootstrap..."
  cli_exec run --rm \
    --user 33:33 \
    --network "${BACKEND_NETWORK}" \
    -v "${SITE_DIR}/data/wp:/var/www/html" \
    wordpress:cli \
    core install \
      --url="https://${DOMAIN}" \
      --title="${SITE_SLUG}" \
      --admin_user="${ADMIN_USER}" \
      --admin_password="${WP_ADMIN_PASS}" \
      --admin_email="${ADMIN_EMAIL}" \
      --skip-email
  cli_exec run --rm \
    --user 33:33 \
    --network "${BACKEND_NETWORK}" \
    -v "${SITE_DIR}/data/wp:/var/www/html" \
    wordpress:cli \
    rewrite structure '/%postname%/' --hard
  cli_exec run --rm \
    --user 33:33 \
    --network "${BACKEND_NETWORK}" \
    -v "${SITE_DIR}/data/wp:/var/www/html" \
    wordpress:cli \
    plugin install redis-cache --activate
  cli_exec run --rm \
    --user 33:33 \
    --network "${BACKEND_NETWORK}" \
    -v "${SITE_DIR}/data/wp:/var/www/html" \
    wordpress:cli \
    redis enable
  chown -R 33:33 "${SITE_DIR}/data/wp/wp-content"
  echo ""
  echo "=========================================================="
  echo " Site Provisioned Successfully!"
  echo "=========================================================="
  echo " Domain:         https://${DOMAIN}"
  echo " Administrator:  ${ADMIN_USER}"
  echo " Password:       ${WP_ADMIN_PASS}"
  echo " Database:       ${DB_NAME}"
  echo " DB User:        ${DB_USER}"
  echo " Redis Index:    ${REDIS_INDEX}"
  echo " Directory:      ${SITE_DIR}"
  if [ "${PROXY_TYPE}" = "npm" ]; then
    echo " Action Required in NPM: Forward https://${DOMAIN} ->${SITE_SLUG}_wp:80"
  elif [ "${PROXY_TYPE}" = "traefik" ]; then
    echo " Traefik Status: Automatically routed via Docker labels."
  elif [ "${PROXY_TYPE}" = "caddy" ]; then
    echo " Caddy Status: Block appended to Caddyfile and reloaded automatically."
  fi
  echo "=========================================================="
 }
 # ==============================================================================
 # CLI DISPATCHER
 # ==============================================================================
 ACTION="${1:-}"
 shift || true
 SSH_PORT_ARG=""
 ENGINE_ARG=""
 PROXY_ARG=""
 while [[ $# -gt 0 ]]; do
  case "$1" in
    --ssh-port)
      SSH_PORT_ARG="$2"
      shift 2
      ;;
    --engine)
      ENGINE_ARG="$2"
      shift 2
      ;;
    --proxy)
      PROXY_ARG="$2"
      shift 2
      ;;
    *)
      break
      ;;
  esac
 done
 case "${ACTION}" in
  --bootstrap)
    prompt_config
    load_state
    run_host_bootstrap
    ;;
  create)
    provision_site "$@"
    ;;
  *)
    echo "Usage:"
    echo "  Interactive Bootstrap:"
    echo "    $0 --bootstrap"
    echo ""
    echo "  Unattended Bootstrap:"
    echo "    $0 --bootstrap --ssh-port <port> --engine <docker|podman> --proxy <npm|traefik|caddy>"
    echo ""
    echo "  Provision Site:"
    echo "    $0 create <slug> <domain> [admin_user] [admin_email]"
    echo ""
    echo "Examples:"
    echo "  $0 --bootstrap --ssh-port 48222 --engine docker --proxy caddy"
    echo "  $0 create teetimeprints teetimeprints.com mkell admin@teetimeprints.com"
    exit 1
    ;;
 esac