journal/README.md

# Project Journal

A structured journaling system with encrypted monthly vaults, a Tauri desktop app, a web gateway server, CLI tools, and optional AI-assisted analysis.

## Repository Layout

```
journal/
├── Journal.Core/          .NET class library — all business logic and services
├── Journal.Sidecar/       Console app — stdin/stdout JSON protocol (Tauri sidecar bridge + CLI)
├── Journal.WebGateway/    ASP.NET Core app — HTTP wrapper for browser/web mode
├── Journal.SmokeTests/    Integration tests (~80 tests, no test framework dependency)
├── Journal.App/           SvelteKit + Tauri desktop app
│   ├── src/               SvelteKit frontend source
│   ├── src-tauri/         Rust Tauri shell (sidecar process manager)
│   └── static/            Static assets
├── scripts/               PowerShell dev, build, publish, and cache helpers
├── docs/                  Internal design docs
└── journal/               Runtime data directories (vault/, data/)
```

## Deployment Modes

The backend can run in three modes depending on the surface wired to it:

| Mode | Host | Frontend |
|------|------|----------|
| **Tauri desktop app** | `Journal.App` (Tauri + Rust) | SvelteKit embedded via Tauri WebView |
| **WebGateway server** | `Journal.WebGateway` (ASP.NET Core) | SvelteKit build served from `wwwroot` |
| **Sidecar CLI / stdin** | `Journal.Sidecar` (console) | None — raw JSON protocol |

All three modes share the same `Journal.Core` service layer and command protocol.

---

## Platform Support

- **Windows** — first-class (primary development target)
- **Linux** — first-class
- **macOS** — best effort

### Prerequisites

- [.NET 10 SDK](https://dotnet.microsoft.com/download)
- [Node.js](https://nodejs.org/) + npm (for `Journal.App` frontend)
- [Rust + Cargo](https://rustup.rs/) (for `Journal.Sidecar` Tauri desktop build)
- PowerShell 7+ (`pwsh`) recommended for scripts

---

## Quickstart

### Option A — Tauri Desktop App

Build the sidecar, then the Tauri app:

```powershell
cd Journal.App
npm install
npm run tauri build
```

Or via the publish scripts (recommended for clean environments):

```powershell
.\scripts\publish-sidecar.ps1 -Configuration Release -Runtime win-x64
.\scripts\publish-app.ps1 -Target tauri -TauriBundles none
```

Tauri auto-detects `Journal.Sidecar.exe` in the repository. On first launch it walks up from the working directory to find `Journal.Sidecar/` and resolves the built executable.

### Option B — WebGateway Server (browser mode)

Build the web UI bundle, then publish the gateway with web assets embedded:

```powershell
.\scripts\publish-app.ps1 -Target web
.\scripts\publish-webgateway.ps1 -Configuration Release -Runtime win-x64
```

Run the gateway:

```powershell
.\scripts\run-webgateway.ps1 -Urls http://0.0.0.0:5180
```

Open `http://localhost:5180` in your browser. The gateway automatically serves the SvelteKit build and proxies all `/api/command` calls to `Journal.Core`.

Quick health check:

```powershell
Invoke-RestMethod http://127.0.0.1:5180/api/health
```

### Option C — Sidecar / CLI only

```powershell
cd Journal.Core
dotnet build

# Run in stdin/stdout protocol mode
dotnet run --project Journal.Sidecar

# CLI subcommands
dotnet run --project Journal.Sidecar -- vault load --password <value>
dotnet run --project Journal.Sidecar -- vault save --password <value>
dotnet run --project Journal.Sidecar -- search "your query" --tag stress --start-date 2026-02-01
```

---

## C# Backend

### Projects

| Project | Type | Purpose |
|---------|------|---------|
| `Journal.Core` | Class library | Domain models, services, repositories, DTOs — all business logic |
| `Journal.Sidecar` | Console app | Stdin/stdout JSON protocol + vault/search CLI subcommands |
| `Journal.WebGateway` | ASP.NET Core | HTTP API wrapper; serves built SvelteKit UI from `wwwroot` |
| `Journal.SmokeTests` | Console app | ~80 integration tests (no xunit/nunit) |

### Solution File

```
Journal.slnx  (Visual Studio solution — Core + Sidecar + SmokeTests)
```

> `Journal.WebGateway` is not in the solution file; build/run it directly with `dotnet` or the `scripts/` wrappers.

### Architecture

```
Entry  (thin command dispatcher — shared by all three hosts)
  ├── Fragments/    IFragmentService      → FragmentService      → SQLiteFragmentRepository (SQLCipher)
  ├── Entries/      IEntryFileService     → EntryFileService     → DiskEntryFileRepository
  │                 IEntrySearchService   → EntrySearchService   (raw content + structured filters)
  │                 JournalParser         (date / section / checkbox / fragment parsing)
  ├── Lists/        IListService          → ListService          → SqliteListRepository
  ├── Todos/        ITodoService          → TodoService          → SqliteTodoRepository
  ├── Vault/        IVaultStorageService  → VaultStorageService  → IVaultCryptoService
  ├── Database/     IJournalDatabaseService (SQLCipher schema/key derivation/hydration)
  │                 IDatabaseSessionService (encrypted connection lifecycle after auth)
  ├── Ai/           IAiService            → PythonSidecarAiService | DisabledAiService
  ├── Speech/       ISpeechBridgeService  → PythonSidecarSpeechService | DisabledSpeechBridgeService
  ├── Sidecar/      PythonSidecarClient   (shared Python process I/O), SidecarCli
  ├── Logging/      CommandLogger, LogRedactor
  └── Config/       IJournalConfigService → JournalConfigService
```

Services live under `Journal.Core/Services/` in domain-specific subdirectories, each with its own namespace (e.g. `Journal.Core.Services.Ai`).

### Build

```powershell
# Build all projects
dotnet build

# Or use the resilient wrapper (handles proxy/NuGet quirks):
.\scripts\dotnet-min.ps1 build Journal.Sidecar/Journal.Sidecar.csproj
.\scripts\dotnet-min.ps1 build Journal.WebGateway/Journal.WebGateway.csproj
```

### Run Smoke Tests

```powershell
dotnet run --project Journal.SmokeTests
```

### Dependencies

- `Journal.Core` — `Microsoft.Data.Sqlite.Core`, `SQLitePCLRaw.bundle_e_sqlcipher`, `Microsoft.Extensions.DependencyInjection.Abstractions`
- `Journal.Sidecar` — `Microsoft.Extensions.DependencyInjection` + references `Journal.Core`
- `Journal.WebGateway` — `Microsoft.NET.Sdk.Web` + references `Journal.Core`
- `Journal.SmokeTests` — references `Journal.Core`

### Encryption

- **Vault**: AES-256-GCM with PBKDF2-HMAC-SHA256 key derivation (600k iterations) — wire format matches the Python implementation for cross-language parity
- **Database**: SQLCipher with PBKDF2-derived key
- Fragments and structured data are stored in the encrypted SQLCipher database; auth is required via `vault.load_all` or `db.hydrate_workspace`
- `DatabaseSessionService` holds the encryption password in memory after first auth and closes the connection on `vault.clear_data_directory`

### Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| `JOURNAL_PROJECT_ROOT` | auto-detected | Override project root (vault + data path resolution) |
| `JOURNAL_DATA_DIR` | `<root>/journal/data` | Override decrypted data directory |
| `JOURNAL_VAULT_DIR` | `<root>/journal/vault` | Override vault directory |
| `JOURNAL_AI_PROVIDER` | `none` | `none` or `python-sidecar` |
| `JOURNAL_PYTHON_EXE` | `python` | Python executable for AI/speech sidecar |
| `JOURNAL_AI_SIDECAR_PATH` | auto | Path to Python AI sidecar script |
| `JOURNAL_AI_TIMEOUT_MS` | 30000 | AI sidecar timeout |
| `JOURNAL_NLP_BACKEND` | `auto` | `auto`, `spacy`, or `fallback` |
| `JOURNAL_LOG_LEVEL` | `warning` | `trace`, `debug`, `information`, `warning`, `error`, `critical` |
| `JOURNAL_WEB_DIST` | auto | Override web UI dist path for WebGateway |

---

## Journal.WebGateway

An ASP.NET Core minimal API that wraps `Journal.Core` for browser use.

### Endpoints

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/health` | Health check |
| `POST` | `/api/command` | Send a JSON command to `Entry.HandleCommandAsync` |
| `GET` | `/api/web/status` | Reports web dist path and whether UI is available |
| `GET` | `/api/sidecar/root` | Returns current project root (auto-detected or custom) |
| `POST` | `/api/sidecar/root` | Override project root at runtime |
| `GET` | `/*` | Serves built SvelteKit UI from `wwwroot` (SPA fallback) |

### Web UI Resolution

On startup, `Journal.WebGateway` resolves the web dist in this order:

1. `JOURNAL_WEB_DIST` environment variable
2. `<AppContext.BaseDirectory>/wwwroot` (embedded in published output)
3. `Journal.App/build` (dev fallback — relative to repo root)

If no dist is found, `/` returns a JSON status message instead of the UI.

### Running WebGateway

```powershell
dotnet run --project Journal.WebGateway
# or
.\scripts\run-webgateway.ps1 -Urls http://0.0.0.0:5180 -ProjectRoot E:\path\to\journal
```

---

## Journal.App (Tauri + SvelteKit)

A Tauri 2 desktop application with a SvelteKit 5 / TypeScript frontend.

### Tech Stack

- **Frontend**: SvelteKit 5, TypeScript, Vite 6
- **Tauri shell**: Rust (Tauri 2), `tokio` for async process I/O
- **Backend bridge**: `Journal.Sidecar.exe` managed as a long-lived child process

### Tauri Sidecar Architecture

The Rust layer (`src-tauri/src/lib.rs`) manages a persistent `Journal.Sidecar.exe` child process:

- Sidecar is auto-started on first command and restarted if it dies
- Commands are sent as JSON lines to stdin, responses read from stdout
- `JOURNAL_PROJECT_ROOT` is set to the resolved repo root before spawning
- On Windows, the process is created with `CREATE_NO_WINDOW`

Tauri commands exposed to the frontend:

| Command | Description |
|---------|-------------|
| `sidecar_command` | Forward a `CommandEnvelope` to `Journal.Sidecar` and return parsed JSON |
| `get_sidecar_root` | Get the current resolved sidecar root path |
| `set_sidecar_root` | Override sidecar root path (saves to `settings.json`, restarts sidecar) |
| `get_ui_settings` | Load tag/fragment-type settings from `settings.json` |
| `set_ui_settings` | Persist tag/fragment-type settings |
| `shutdown` | Stop the sidecar and exit the app |

Sidecar path resolution order (relative to root):

1. `Journal.Sidecar.exe` in root
2. `publish/Journal.Sidecar.exe`
3. `Journal.Sidecar/bin/Debug/net10.0/Journal.Sidecar.exe`
4. `Journal.Sidecar/bin/Release/net10.0/win-x64/publish/Journal.Sidecar.exe`
5. Recursive scan of `Journal.Sidecar/`

### Frontend State

The frontend uses Svelte stores as the source of truth:

| Store | State | Purpose |
|-------|-------|---------|
| `entries.ts` | `entriesStore` | Journal entry list and drafts |
| `fragments.ts` | `fragmentsStore` | Fragment CRUD + parse/serialize helpers |
| `todos.ts` | `todoListsStore`, `todosStore` | Todo lists and items |
| `lists.ts` | `listsStore` | Generic lists |
| `settings.ts` | `settingsTags`, `settingsFragmentTypes` | Tag/type configuration |

**Store-First Rule**: components call store helpers for CRUD; they do not embed mutation or parsing logic directly.

### Dev Setup

```powershell
cd Journal.App
npm install
npm run dev       # SvelteKit dev server at http://localhost:1420
npm run tauri dev # Tauri dev mode (opens desktop window)
```

### Publishing

```powershell
# Web bundle only (for WebGateway)
.\scripts\publish-app.ps1 -Target web
# Output: Journal.App/build/

# Tauri raw exe (no installer)
.\scripts\publish-app.ps1 -Target tauri -TauriBundles none
# Output: Journal.App/src-tauri/target/release/journalapp.exe

# Tauri with NSIS installer
.\scripts\publish-app.ps1 -Target tauri -TauriBundles nsis
```

---

## Sidecar Protocol

`Journal.Sidecar` communicates over **stdin/stdout** using newline-delimited JSON. One JSON object in, one JSON object out.

### Command Format

```json
{
  "action": "fragments.create",
  "correlationId": null,
  "id": null,
  "type": null,
  "tag": null,
  "payload": { "type": "!TRIGGER", "description": "stomach drop" }
}
```

**Fields:**
- `action` — Operation to perform (e.g. `fragments.list`, `vault.load_all`)
- `correlationId` — Optional tracing ID (auto-generated if omitted)
- `id` — Target entity ID (for get/update/delete)
- `type` / `tag` — Filter parameters (for fragment search)
- `payload` — Request body, deserialized per action

### Response Format

Success:
```json
{ "ok": true, "data": { "id": "abc-123", "type": "!TRIGGER", "description": "...", "time": "...", "tags": [] } }
```

Error:
```json
{ "ok": false, "error": "Description is required" }
```

### Available Actions

| Action | Description | Key Requirements |
|--------|-------------|------------------|
| `fragments.list` | List all fragments | — |
| `fragments.get` | Get by ID | `id` |
| `fragments.create` | Create fragment | `payload` (CreateFragmentDto) |
| `fragments.update` | Update fragment | `id`, `payload` (UpdateFragmentDto) |
| `fragments.delete` | Delete fragment | `id` |
| `fragments.search` | Filter by type/tag | `type` and/or `tag` |
| `lists.list` | List all lists | — |
| `lists.get` | Get list by ID | `id` |
| `lists.create` | Create list | `payload` |
| `lists.update` | Update list | `id`, `payload` |
| `lists.delete` | Delete list | `id` |
| `todos.list` | List all todo lists | — |
| `todos.get` | Get todo list by ID | `id` |
| `todos.create` | Create todo list | `payload` |
| `todos.update` | Update todo list | `id`, `payload` |
| `todos.delete` | Delete todo list | `id` |
| `todos.items.create` | Add todo item | `payload` |
| `todos.items.update` | Update todo item | `id`, `payload` |
| `todos.items.delete` | Delete todo item | `id` |
| `entries.list` | List decrypted `.md` entries | optional `payload.dataDirectory` |
| `entries.load` | Load one entry file | `payload.filePath` |
| `entries.save` | Save/merge entry content | `payload.content`, optional `payload.filePath`, `payload.mode`, `payload.fileName` |
| `entries.delete` | Delete an entry file | `payload.filePath` |
| `templates.list` | List `.template.md` files | optional `payload.dataDirectory` |
| `templates.load` | Load a template | `payload.filePath` |
| `templates.save` | Save/create a template | `payload.name` |
| `templates.delete` | Delete a template | `payload.filePath` |
| `search.entries` | Search entries with filters | `payload.dataDirectory`, optional query/section/date/tags/types/checked/unchecked |
| `vault.initialize` | Ensure vault directory exists | `payload.password`, `payload.vaultDirectory` |
| `vault.load_all` | Decrypt all monthly vaults → data dir | `payload.password`, `payload.vaultDirectory`, `payload.dataDirectory` |
| `vault.save_current_month` | Encrypt only current month (optimized) | `payload.password`, `payload.vaultDirectory`, `payload.dataDirectory` |
| `vault.rebuild_all` | Rebuild all monthly vaults from data | `payload.password`, `payload.vaultDirectory`, `payload.dataDirectory` |
| `vault.clear_data_directory` | Wipe decrypted data directory | `payload.dataDirectory` |
| `db.status` | DB key/schema compatibility snapshot | `payload.password`, optional `payload.dataDirectory` |
| `db.initialize_schema` | Write SQL schema bootstrap file | optional `payload.dataDirectory` |
| `db.hydrate_workspace` | Bootstrap DB + set session password | `payload.password`, optional `payload.dataDirectory` |
| `config.get` | Return current config snapshot | — |
| `ai.health` | AI provider health status | — |
| `ai.summarize_entry` | Summarize one entry | `payload.content`, optional `payload.fileStem` |
| `ai.summarize_all` | Summarize multiple entries | `payload.entries[]` |
| `ai.chat` | Chat via AI provider bridge | `payload.prompt` |
| `ai.embed` | Generate embedding vector | `payload.content` |
| `speech.devices.list` | List audio input devices | — |
| `speech.transcribe` | Transcribe audio (base64) or text | `payload.audioBase64` or `payload.text` |

### Sidecar CLI Mode

In addition to stdin/stdout protocol, `Journal.Sidecar` supports direct CLI subcommands:

```powershell
# Load/decrypt all vaults into data workspace
dotnet run --project Journal.Sidecar -- vault load

# Save (rebuild) monthly vaults from decrypted markdown files
dotnet run --project Journal.Sidecar -- vault save

# Search entries (query + filters)
dotnet run --project Journal.Sidecar -- search "common text" --tag stress --type !TRIGGER --start-date 2026-02-01 --end-date 2026-02-28 --section Summary --checked "med taken"
```

**Password behavior:**
- Omit `--password` → prompts securely in terminal
- Pass `--password <value>` → non-interactive/automation mode

**Optional path overrides:**
- `--vault-dir <path>` / `--data-dir <path>`
- Env fallback: `JOURNAL_VAULT_DIR`, `JOURNAL_DATA_DIR`, `JOURNAL_APP_DIR`

**Search CLI flags:**
- positional `query` (optional)
- `--tag` / `-t` (repeatable)
- `--type` / `-y` (repeatable)
- `--start-date` / `-s` (`yyyy-MM-dd`)
- `--end-date` / `-e` (`yyyy-MM-dd`)
- `--section` / `-sec`
- `--checked` / `-chk` (repeatable)
- `--unchecked` / `-uchk` (repeatable)
- `--data-dir <path>` (optional override)

---

## Publishing

### Sidecar (self-contained executable)

```powershell
.\scripts\publish-sidecar.ps1 -Configuration Release -Runtime win-x64
# Output: output\Journal.Sidecar.exe (~70MB, all bundled)
```

Or raw `dotnet publish`:

```powershell
dotnet publish Journal.Sidecar/Journal.Sidecar.csproj -c Release -r win-x64 --self-contained -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true
```

To exclude debug symbols: add `-p:DebugType=none`

For a smaller build that requires .NET 10 on the target machine:

```powershell
dotnet publish Journal.Sidecar/Journal.Sidecar.csproj -c Release -r win-x64 -p:PublishSingleFile=true
```

### WebGateway (with embedded web UI)

```powershell
# Step 1: build web assets
.\scripts\publish-app.ps1 -Target web

# Step 2: publish gateway (copies web assets into wwwroot automatically)
.\scripts\publish-webgateway.ps1 -Configuration Release -Runtime win-x64
# Output: output\webgateway\  (with output\webgateway\wwwroot\ from Journal.App\build)
```

---

## DI Registration

`ServiceCollectionExtensions.AddFragmentServices()` wires everything. Any host calls:

```csharp
services.AddFragmentServices();
services.AddSingleton<Entry>();
```

Key registrations:
- `IDatabaseSessionService` → `DatabaseSessionService` (singleton)
- `IFragmentRepository` → `SqliteFragmentRepository` (singleton, SQLCipher-backed)
- `IFragmentService` → `FragmentService` (singleton)
- `IEntryFileRepository` → `DiskEntryFileRepository` (singleton)
- `IEntryFileService` → `EntryFileService` (singleton)
- `IListRepository` → `SqliteListRepository` (singleton)
- `IListService` → `ListService` (singleton)
- `ITodoRepository` → `SqliteTodoRepository` (singleton)
- `ITodoService` → `TodoService` (singleton)
- `IVaultCryptoService` → `VaultCryptoService` (singleton)
- `IVaultStorageService` → `VaultStorageService` (singleton)
- `IJournalDatabaseService` → `JournalDatabaseService` (singleton)
- `IAiService` → `PythonSidecarAiService` or `DisabledAiService` (per `JOURNAL_AI_PROVIDER`)
- `ISpeechBridgeService` → `PythonSidecarSpeechService` or `DisabledSpeechBridgeService`
- `IJournalConfigService` → `JournalConfigService` (singleton)
- `CommandLogger` (singleton)
- `SidecarCli` (singleton)

---

## Extending with New Modules

The `Command`/`Entry` pattern uses dot-notation actions. To add a module:

1. Create model, DTO, repository, and service in `Journal.Core/Services/<Domain>/`
2. Register services in `ServiceCollectionExtensions.cs`
3. Inject the service into `Entry.cs` and add cases to the `switch`
4. No changes needed to `App.cs`, `Journal.WebGateway/Program.cs`, or the Tauri Rust shell

---

## Scripts

See [`scripts/README.md`](scripts/README.md) for the full reference and [`scripts/WORKFLOWS.md`](scripts/WORKFLOWS.md) for copy-paste command recipes.

Quick reference:

| Script | Purpose |
|--------|---------|
| `dev-shell.ps1` | Dot-source to configure current shell with repo-local env vars |
| `dotnet-min.ps1` | `dotnet` wrapper with resilient NuGet defaults |
| `pip-min.ps1` | `pip` wrapper with repo-local cache and Windows compat mapping |
| `publish-app.ps1` | Build web bundle or Tauri desktop app |
| `publish-sidecar.ps1` | Publish `Journal.Sidecar` single-file exe to `output/` |
| `publish-webgateway.ps1` | Publish `Journal.WebGateway` with optional web assets |
| `run-webgateway.ps1` | Run `Journal.WebGateway` with controlled env and project root |
| `migration-gate.ps1` | End-to-end build + smoke + parity + API check gate |
| `nuget-export-cache.ps1` | Export NuGet cache to zip for offline/transfer use |
| `nuget-import-cache.ps1` | Import NuGet cache zip and validate restore |

---

## Notes

- Decrypted journal data in `journal/data/` is cleared on graceful shutdown (`vault.clear_data_directory`).
- The legacy Python placeholder file `_init_vault.vault` is treated as obsolete — the C# backend ignores and removes it during vault load.
- `Journal.WebGateway` is intentionally excluded from `Journal.slnx`; it is built/run independently via `dotnet` or the scripts wrappers.
- On Windows + Tauri, the sidecar process is spawned with `CREATE_NO_WINDOW` to suppress the console window.