mtkell
/
chronogrid
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
chronogrid/docs/ARCHITECTURE/01-OSAD-system-architecture.md

20 KiB
Raw Blame History

Operational and System Architecture Document (OSAD)

Project Name: ChronoGrid P2P (Distributed Time, Note & Project Management Engine)

Repository Host: git.kellsupport.com

Target Platforms: Windows, Linux, Android, Headless Peer Server

Core Framework: Flutter (Dart FFI) + Rust Native Core Core Engine

Architecture Model: Local-First, Serverless Peer-to-Peer (P2P), CRDT-Driven Synchronization

Status: Active Specification

Refer to system blueprints for Flutter presentation layer, Rust Core engine (chronogrid-core), and P2P Mesh Network (chronogrid-net).

1. System Overview & Architectural Vision

ChronoGrid P2P is a distributed, local-first productivity operating system. It provides comprehensive project planning, time blocking, and note management across mobile, desktop, and headless server nodes without relying on centralized cloud providers, SaaS databases, or multi-tenant servers.

Data sovereignty, cryptographic security, and offline execution form the foundation of the system architecture. Every instance—whether running on Windows, Linux, Android, or as a background service—operates as an autonomous node holding an encrypted, fully functional local database state. Synchronizations between nodes occur directly over local network discovery (MDNS) and WAN P2P mesh protocols (libp2p / Iroh) using Conflict-free Replicated Data Types (CRDTs) to guarantee eventual state convergence.

+-----------------------------------------------------------------------------------+
|                            FLUTTER PRESENTATION LAYER                             |
|          (Windows Win32 FFI / Linux GTK Desktop / Android Jetpack Engine)          |
+-----------------------------------------------------------------------------------+
                                         │
                        Dart FFI / C-ABI Binding Bridge
                                         │
+-----------------------------------------------------------------------------------+
|                        RUST NATIVE P2P ENGINE (Core Core)                          |
|  +------------------------+  +------------------------+  +---------------------+  |
|  |    Local Store Engine   |  |     CRDT Sync Engine   |  |   Automation Engine |  |
|  |  (SQLite + SQLCipher)  |  | (Automerge / Delta-LWW)|  |   (Embedded Rust)   |  |
|  +------------------------+  +------------------------+  +---------------------+  |
|  +------------------------------------------------------------------------------+  |
|  |                         P2P Mesh Networking Layer                            |  |
|  |  (libp2p / QUIC / Noise XK Cipher Handshake / NAT Hole-Punching / Gossipsub) |  |
|  +------------------------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------+

2. High-Level System Architecture & Component Breakdown

The architecture isolates the UI from data orchestration and networking using a modular, layered Rust core wrapped by a Flutter presentation layer.

               +--------------------------------------------+
               |        Flutter GUI (Windows/Linux/Android) |
               +--------------------------------------------+
                                     │
                        (Dart Stream / FFI Bridge)
                                     │
               +--------------------------------------------+
               |            Local App State Store           |
               +--------------------------------------------+
                 │                  │                     │
                 ▼                  ▼                     ▼
        +------------------+ +-------------------+ +-------------------+
        |  SQLite Storage  | |  CRDT State Engine| | Local AI Workflow |
        |   (SQLCipher)    | |    (Automerge)    | |  (Ollama / ONNX)  |
        +------------------+ +-------------------+ +-------------------+
                 │                  │                     │
                 +------------------┼---------------------+
                                    │
                                    ▼
               +--------------------------------------------+
               |           Libp2p Mesh Network              |
               | (mDNS / Hole Punching / Noise Protocol)    |
               +--------------------------------------------+
                     │                        │
                     ▼                        ▼
        +------------------------+ +------------------------+
        | Peer Node (Mobile/Desk)| | Headless Peer Server   |
        +------------------------+ +------------------------+

Layer 1: Flutter Presentation Layer (UI Shell)

  • Framework: Flutter 3.x configured with C-FFI direct memory communication.
  • Platforms: Windows (Win32), Linux (GTK3), Android (NDK/ARM64).
  • State Management: BLoC / Reactive Streams listening directly to native Rust event emitters.
  • UI Features:
  • Interactive Kanban boards, interactive Gantt timelines, and multi-column list views.
  • Eisenhower Priority Matrix view (Urgent vs. Important quadrant auto-sorting).
  • Integrated Calendar Time-Blocking Grid with drag-and-drop task duration sizing.
  • Markdown Document & Quick Jot Editor with bi-directional task linking.

Layer 2: Embedded Rust Core Engine (chronogrid-core)

The entire storage, cryptographic, sync, and networking logic is compiled into a native dynamic library (.dll on Windows, .so on Linux, .so on Android) and cross-compiled for headless binary deployments.

  • SQLCipher Engine: Manages local persistence with AES-256-GCM database-level encryption.
  • CRDT Merger Engine: Reconciles offline delta updates across devices without central coordination.
  • Automation & Rules Engine: Evaluates local incoming jots, emails, and triggers using an embedded lightweight evaluation engine.
  • Local AI Processor: Executes local model inferences (via embedded ONNX runtime or Ollama HTTP bindings) to extract metadata, assign scores, and categorize tasks without external API calls.

Layer 3: P2P Network Mesh (chronogrid-net)

  • Transport: QUIC over UDP and TCP with Noise Protocol (XK pattern) transport encryption.
  • Discovery: Local mDNS for zero-configuration local area network (LAN) peer discovery; Kademlia DHT and NAT Traversal (STUN/TURN/ICE hole punching) for Wide Area Network (WAN) peer discovery.
  • Gossipsub Protocol: Efficient message propagation for real-time state changes and sync notifications across personal node clusters.

3. Data Storage, Security, & Cryptography Model

Zero-Trust Local Storage

Data on disk is never stored in unencrypted plain text.

  1. Master Key Generation: During initial app setup, a 256-bit Master Key is derived from user credentials using Argon2id (Memory: 64MB, Iterations: 3, Parallelism: 4).
  2. Key Storage:
  • Windows: Windows Data Protection API (DPAPI) / Credential Manager.
  • Linux: Secret Service API (Freedesktop SecretService via Secret Storage).
  • Android: Android KeyStore System with hardware-backed Trusted Execution Environment (TEE).
  • Headless Server: Systemd credential store or encrypted keyfile payload.
  1. Database Encryption: SQLite encrypted end-to-end via SQLCipher 4.x using AES-256-CBC with HMAC-SHA512 per-page authentication.

Peer-to-Peer Encryption & Access Control

Syncing between personal nodes utilizes end-to-end asymmetric encryption:

  • Node Identity: Every node generates an Ed25519 keypair upon initialization. The public key acts as the NodeId.
  • Swarm Key & Cluster Pairing: Personal devices join a private mesh cluster using a shared, user-generated 256-bit Swarm Key. Unauthenticated nodes cannot decode network signals or handshake packets.
  • Wire Protocol Security: Network channels use Ephemeral Diffie-Hellman (X25519) key exchange combined with ChaCha20-Poly1305 AEAD payload ciphering.

4. Conflict-Free Replicated Synchronization Engine (CRDT)

To eliminate central server databases while supporting multi-device concurrent offline edits, ChronoGrid uses state-based and delta-based CRDT topologies.

       [ Device A (Offline) ]                   [ Device B (Offline) ]
      Creates Task T1 (v1.0)                   Edits Note N1 (v1.1)
                 │                                        │
                 ▼                                        ▼
      [ Local CRDT Append ]                    [ Local CRDT Append ]
                 │                                        │
                 └─────────────── Direct P2P ─────────────┘
                             (Reconnection Sync)
                                     │
                                     ▼
                       [ CRDT Deterministic Merge ]
                      State Converges Automatically

Data Modeling & Conflict Resolution

  1. Tasks & Notes (Observed-Remove Set / LWW-Register):
  • Fields like task title, body notes, priority rating, and status use Last-Write-Wins Element-Set (LWW-Element-Set) registers based on Hybrid Logical Clocks (HLCs).
  • HLCs prevent clock skew anomalies across distinct physical hardware platforms.
  1. Project Hierarchies & Trees (Sequence CRDT):
  • Sub-task orders and document tree layouts use fractional indexing combined with tree-CRDTs to prevent circular reference cycles when moving folders offline.
  1. Delta Sync Protocol:
  • Nodes keep an append-only state transition log.
  • When two peers connect, they exchange vector clocks. Only missing state deltas (changesets) are packaged, encrypted, and streamed across the QUIC network socket.

5. Integrated Time, Note, & Project Architecture

The application implements the complete productivity pipeline—note ingestion, task extraction, Eisenhower matrix sorting, and calendar time-blocking—as local data transformations inside the CRDT engine.

+-----------------------------------------------------------------------------------+
|                            INBOUND INGESTION ENGINE                               |
|        (Local IMAP Watcher / System Hotkeys / Local Files / Audio Jots)           |
+-----------------------------------------------------------------------------------+
                                         │
                                         ▼
+-----------------------------------------------------------------------------------+
|                         LOCAL AI & AUTOMATION ENGINE                              |
|           (Local LLM Inference / Keyword Extractor / Natural Language NLP)        |
|                                                                                   |
|  * Parses Due Dates & Durations    * Assigns Project & Namespace Tag             |
|  * Calculates Eisenhower Priorities * Converts Raw Jots to Structured Docs        |
+-----------------------------------------------------------------------------------+
                                         │
                                         ▼
+-----------------------------------------------------------------------------------+
|                        LOCAL CRDT PRODUCTIVITY DATABASE                           |
|  +---------------------+  +---------------------+  +---------------------------+  |
|  |  Multi-Level Tree   |  |  Eisenhower Matrix  |  |   Calendar Time-Blocking  |  |
|  | (Projects/Sub-Tasks)|  |  (Quadrant 1 - 4)   |  |   (CalDAV & Native Grid)  |  |
|  +---------------------+  +---------------------+  +---------------------------+  |
+-----------------------------------------------------------------------------------+

Multi-Project & Note Ecosystem

  • Namespaces & Hierarchies: Deep multi-level support for distinct spaces (e.g., Personal, Engineering, Creations).
  • Note-to-Task Pipeline: Unstructured daily jots are captured via global shortcuts or voice recordings. The internal processing pipe converts embedded action items into indexed tasks while maintaining bidirectional hyperlinks to the parent note.

Automated Eisenhower Priority Engine

Tasks automatically compute an Urgency-Importance Vector Score ($S$) based on internal properties:

S = (W_i \cdot P_{user}) + (W_u \cdot f(\text{Due Date} - t_{current})) + (W_a \cdot \text{Age})

Where:

  • P_{user} is the user-defined priority weight ($0.0 \text{ to } 1.0$).
  • f(\Delta t) is an exponential decay function mapping time proximity to urgency.
  • W_i, W_u, W_a are normalized weights for importance, urgency, and age accumulation.

The resulting vector automatically places the task into one of four Eisenhower Quadrants on the dynamic dashboard.

Local Calendar & Automated Ingestion System

  • Local Ingestion Daemon: The headless/background engine includes built-in watchers for:

  • Local File System / PDF / Markdown: Drops into monitored watch folders automatically parse into raw notes.

  • Direct TLS IMAP Engine: Runs locally inside the Rust core to securely poll personal or corporate mail servers directly over TLS, extracting flagged emails into actionable project tasks without third-party cloud bridges.

  • CalDAV Local Server & Client: The application embeds a local CalDAV service end-point on 127.0.0.1. Local calendar software (Outlook, Thunderbird, mobile calendars) can sync directly with the app locally.

6. Self-Hosted Headless Peer Node Architecture

To ensure high availability for sync without relying on public cloud servers, the core application compiles down to a lightweight, daemonized headless binary (chronogrid-daemon).

                              +--------------------+
                              |  WAN P2P Network   |
                              +--------------------+
                                 │              │
                    QUIC / Noise │              │ QUIC / Noise
                                 ▼              ▼
                    +----------------+      +------------------+
                    | Desktop Client |      |  Mobile Android  |
                    | (Windows/Linux)|      |      Client      |
                    +----------------+      +------------------+
                                 │              │
                                 │ QUIC / Noise │
                                 ▼              ▼
                              +--------------------+
                              | Headless Peer Node |
                              | (Always-On Relay & |
                              |   Storage Vault)   |
                              +--------------------+

Headless Server Features

  • Role: Acts as an always-on, high-availability peer node, relay node, and backup vault for encrypted CRDT states.
  • Storage: Holds the encrypted SQLCipher database. Even if hosted on a remote untrusted VPS, data remains unreadable without the user's Swarm Master Key.
  • Headless Services:
  • Runs local automated email polling (IMAP) continuously in the background.
  • Serves as a WebRTC/libp2p Circuit Relay v2 to facilitate NAT hole-punching between mobile devices on cellular networks and desktop environments.
  • Provides a lightweight local Web-GUI shell over HTTPS for browser-based task management.

7. Operational & Development Setup Guide

Repository Structure (git.kellsupport.com)

chronogrid/
├── Cargo.toml                    # Workspace setup for native rust crates
├── apps/
│   ├── chronogrid_flutter/       # Flutter UI source code (Windows, Linux, Android)
│   └── chronogrid_daemon/        # Headless server CLI binary
├── crates/
│   ├── chronogrid_core/          # Database, SQLCipher, and CRDT engine
│   ├── chronogrid_net/           # libp2p network mesh and discovery
│   ├── chronogrid_crypto/       # Key derivation, Argon2id, and cipher logic
│   └── chronogrid_ai/           # Local ONNX runtime and NLP engine
└── scripts/
    ├── build_android_libs.sh     # Android NDK cross-compilation pipeline
    └── build_desktop_ffi.py      # FFI bindings generator for Flutter

Flutter Development Setup (VS Code on Windows)

Prerequisites

  • Flutter SDK (3.22.x or later)
  • VS Code with extensions: Dart, Flutter, rust-analyzer, C/C++
  • LLVM / Clang and MSVC Build Tools (via Visual Studio Installer)
  • Rust toolchain (rustup target add x86_64-pc-windows-msvc aarch64-linux-android x86_64-unknown-linux-gnu)
  • flutter_rust_bridge_codegen CLI for direct Dart/Rust bindings.

Step 1: Environment Initialization

Clone the repository from the internal source control server:

git clone https://git.kellsupport.com/kell-engineering/chronogrid.git
cd chronogrid

Step 2: Compile Native Core Libraries

From the root workspace directory, generate the C/FFI dynamic link libraries and Flutter bindings:

# Generate C/FFI bindings for Flutter
cargo install flutter_rust_bridge_codegen
flutter_rust_bridge_codegen create --rust-input crates/chronogrid_core/src/api.rs --dart-output apps/chronogrid_flutter/lib/src/native/bridge_generated.dart

# Build Windows debug library
cargo build --package chronogrid_core

Step 3: Run the Flutter Application

Open apps/chronogrid_flutter in VS Code and launch the debug target:

cd apps/chronogrid_flutter
flutter pub get
flutter run -d windows

Step 4: Cross-Compiling for Headless Linux Server

To deploy an always-on peer server node:

cargo build --release --package chronogrid_daemon --target x86_64-unknown-linux-gnu

Deploy the resulting binary (target/x86_64-unknown-linux-gnu/release/chronogrid-daemon) to the server environment along with its systemd service file:

[Unit]
Description=ChronoGrid Headless Peer Node
After=network.target

[Service]
Type=simple
User=chronogrid
WorkingDirectory=/var/lib/chronogrid
ExecStart=/usr/local/bin/chronogrid-daemon --config /etc/chronogrid/config.toml
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

8. Master Development Phase Roadmap

+-----------------------------------------------------------------------------------+
|                               PHASED ROADMAP                                      |
|                                                                                   |
| [Phase 1: Core Engine] ──> [Phase 2: Networking] ──> [Phase 3: Flutter UI Shell]  |
| (Rust, Crypt, CRDT)        (P2P Mesh, Noise, QUIC)   (Windows, Linux, Android)    |
|                                                                 │                 |
|                                                                 ▼                 |
| [Phase 5: Release & Test] <── [Phase 4: Automation & AI] <──────┘                 |
| (Cross-Platform Builds)        (IMAP, Local NLP, Logic)                           |
+-----------------------------------------------------------------------------------+

Phase 1: Native Storage & CRDT Core Foundation

  • Implement chronogrid_crypto for key derivation and key storage integrations.
  • Set up SQLite with SQLCipher integration inside chronogrid_core.
  • Build the state transition engine and CRDT vector-clock resolution logic using Automerge primitives.

Phase 2: P2P Networking Mesh Integration

  • Implement libp2p network protocol stack inside chronogrid_net supporting mDNS, Gossipsub, and QUIC transports.
  • Build encrypted Noise XK protocol handshakes and Swarm Key authentication mechanisms.
  • Implement delta state exchange protocols over binary streams.

Phase 3: Cross-Platform Flutter Presentation Layer

  • Construct the Flutter desktop and mobile shell apps.
  • Integrate flutter_rust_bridge FFI bindings for zero-copy state streaming between Dart and Rust.
  • Build UI modules: Kanban boards, drag-and-drop Calendar grid, Note editor, and Eisenhower quadrant dashboard.

Phase 4: Local Automation & AI Engine Pipeline

  • Implement native IMAP background polling directly within chronogrid_core.
  • Embed local NLP metadata extraction engine for automatic task scoring and duration estimation.
  • Create local CalDAV bridge server for device calendar interoperability.

Phase 5: Headless Node & Deployment Optimization

  • Build chronogrid_daemon headless executable.
  • Perform multi-device cross-network latency, NAT traversal, and CRDT merge strain testing under high-volume offline updates.
  • Finalize build automation scripts on git.kellsupport.com for Windows (.msi), Linux (.AppImage/.deb), and Android (.apk).