diff --git a/README.md b/README.md index b07d408..b9744b1 100644 --- a/README.md +++ b/README.md @@ -1,264 +1,187 @@ -# Journal Backend (.NET) +# Project_Journal -A .NET 10 backend for the Project Journal app. Provides core journal functionality as a class library with a sidecar console app for Tauri integration and an optional HTTP API. +A structured journaling system with encrypted monthly vaults, desktop UI, CLI tools, and optional AI-assisted analysis. -## Project Structure +## Support Matrix -``` -backend/ -├── Journal.Core/ Class library — all business logic -│ ├── Models/ -│ │ ├── Fragment.cs Domain model (validated, owns Guid ID) -│ │ ├── Command.cs Stdin command shape for sidecar protocol -│ │ ├── ParsedSection.cs Parsed section model for entry parity work -│ │ ├── SectionTitles.cs Canonical section title list (Python parity) -│ │ └── JournalEntry.cs Entry domain (`date/raw_content/sections/fragments` + merge + markdown reconstruction) -│ ├── Dtos/ -│ │ └── FragmentDtos.cs Immutable records for API boundary -│ │ ├── FragmentDto Read (what goes out) -│ │ ├── CreateFragmentDto Create (what comes in) -│ │ └── UpdateFragmentDto Update (partial, all fields optional) -│ ├── Repositories/ -│ │ ├── IFragmentRepository.cs Interface (data access contract) -│ │ ├── InMemoryFragmentRepository.cs In-memory implementation (tests/dev) -│ │ └── FileFragmentRepository.cs File-backed implementation (default) -│ ├── Services/ -│ │ ├── IFragmentService.cs Interface (business logic contract) -│ │ ├── FragmentService.cs Validates, calls repo, maps to DTOs -│ │ ├── IEntrySearchService.cs Entry search contract (content parity) -│ │ ├── EntrySearchService.cs Searches decrypted `.md` entries by raw content query -│ │ ├── IJournalConfigService.cs Config contract for path/vault/AI/speech settings parity -│ │ ├── JournalConfigService.cs Env/default-backed config surface aligned with Python keys -│ │ ├── IAiService.cs AI bridge contract (optional provider) -│ │ ├── DisabledAiService.cs No-op AI provider for deterministic disabled mode -│ │ ├── PythonSidecarAiService.cs Local Python sidecar adapter (stdin/stdout JSON) -│ │ ├── SidecarCli.cs CLI runner (`vault` + `search`) used by Sidecar host -│ │ ├── JournalParser.cs Date + section + checkbox + fragment parser slices (Phase 2) -│ │ ├── IVaultCryptoService.cs Vault crypto contract -│ │ ├── VaultCryptoService.cs AES-256-GCM + PBKDF2 compatibility layer -│ │ ├── IVaultStorageService.cs Vault load/workflow contract -│ │ └── VaultStorageService.cs Monthly naming + load/decrypt/extract workflow -│ ├── Entry.cs Command dispatcher (stdin/stdout) -│ ├── ServiceCollectionExtensions.cs DI registration helper -│ └── Journal.Core.csproj -│ -├── Journal.Sidecar/ Console app — Tauri sidecar bridge -│ ├── App.cs Boots DI container, runs Entry.RunAsync() -│ └── Journal.Sidecar.csproj References Journal.Core -│ -├── Journal.Api/ Web API — HTTP endpoint wrapper (optional) -│ ├── Program.cs -│ └── Journal.Api.csproj -│ -└── README.md +- Python: `3.14` +- Platforms: Windows and Linux (first-class), macOS (best effort) +- Default profile: CPU +- Optional profiles: GPU, optional NLP backend + +## Dependency Profiles + +- `requirements_base.txt`: shared Journal runtime dependencies +- `requirements_cpu_only.txt`: base + CPU AI stack +- `requirements_gpu.txt`: base + GPU AI stack +- `requirements_nlp_optional.txt`: optional spaCy backend (auto-fallback if unavailable) + +## Quickstart + +### Linux (CPU default) + +```bash +cd Project_Journal +python3.14 -m venv .venv +source .venv/bin/activate +python -m pip install --upgrade pip +python -m pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements_cpu_only.txt ``` -## Architecture +### Linux (GPU optional) -Each layer only knows about the one below it: - -``` -Sidecar (stdin/stdout) ──┐ - ├──► Services (business logic) ──► Repositories (data access) -API (HTTP/JSON) ─────────┘ +```bash +cd Project_Journal +python3.14 -m venv .venv +source .venv/bin/activate +python -m pip install --upgrade pip +python -m pip install -r requirements_gpu.txt ``` -- **Models** — Domain objects with validation. The source of truth. -- **DTOs** — Immutable records that cross the API boundary. Internal logic never leaks out. -- **Repositories** — Where data lives. Current default is file-backed; can evolve to SQLite/EF Core without touching anything above. -- **Services** — Business rules, validation, orchestration. Doesn't know about HTTP or stdin. -- **Entry** — Transport adapter. Translates stdin/stdout JSON into service calls. - -## Dependencies - -- **Journal.Core** — `Microsoft.Extensions.DependencyInjection.Abstractions` (interface-only, lightweight) -- **Journal.Sidecar** — `Microsoft.Extensions.DependencyInjection` (full container implementation) + references `Journal.Core` -- **Journal.Api** — `Microsoft.AspNetCore.OpenApi` + ASP.NET shared framework - -## Building +### Windows PowerShell (CPU default) ```powershell -# Build everything (building Sidecar also rebuilds Core if changed) -dotnet build backend\Journal.Sidecar\Journal.Sidecar.csproj - -# Build just the library -dotnet build backend\Journal.Core\Journal.Core.csproj - -# Format code -dotnet format backend\Journal.Core\Journal.Core.csproj +cd Project_Journal +py -3.14 -m venv .venv +.\.venv\Scripts\Activate.ps1 +python -m pip install --upgrade pip +python -m pip install --extra-index-url https://download.pytorch.org/whl/cpu -r requirements_cpu_only.txt ``` -## Publishing +On Windows + Python 3.14, `pywebview` is intentionally skipped due upstream +`pythonnet` build compatibility. `run_desktop.py` will auto-fallback to opening +the app in your system browser. -Publish as a single-file self-contained executable (no .NET runtime install needed): +### Optional NLP backend (spaCy) + +```bash +python -m pip install -r requirements_nlp_optional.txt +python -m spacy download en_core_web_sm +``` + +If spaCy is missing or unsupported, Journal now auto-falls back to built-in NLP heuristics. +On current Python 3.14 environments, this optional install may be skipped due upstream spaCy compatibility. + +## Running + +### Desktop App + +```bash +python ./journal/run_desktop.py +``` + +### CLI + +```bash +python -m journal.cli.main --help +python -m journal.cli.main vault load +python -m journal.cli.main search "your query" +``` + +## NLP Backend Control + +Set `JOURNAL_NLP_BACKEND` to choose behavior: + +- `auto` (default): use spaCy when available, else fallback +- `spacy`: require spaCy backend and fail clearly if unavailable +- `fallback`: always use fallback heuristics + +Examples: + +```bash +export JOURNAL_NLP_BACKEND=fallback +python ./journal/run_desktop.py +``` ```powershell -dotnet publish backend\Journal.Sidecar\Journal.Sidecar.csproj -c Release -r win-x64 --self-contained -p:PublishSingleFile=true -p:IncludeNativeLibrariesForSelfExtract=true +$env:JOURNAL_NLP_BACKEND = "spacy" +python .\journal\run_desktop.py ``` -Output: `backend\Journal.Sidecar\bin\Release\net10.0\win-x64\publish\Journal.Sidecar.exe` (~70MB, everything bundled) +## Installer Script -To exclude debug symbols: add `-p:DebugType=none` +Use the Linux helper script: -For a smaller build that requires .NET 10 on the target machine: - -```powershell -dotnet publish backend\Journal.Sidecar\Journal.Sidecar.csproj -c Release -r win-x64 -p:PublishSingleFile=true +```bash +./installreqs.sh +./installreqs.sh --gpu +./installreqs.sh --with-nlp ``` -## Sidecar Protocol +## C# Backend -The sidecar communicates over stdin/stdout using JSON lines. One JSON line in, one JSON line out. -When run with no command-line args, this protocol mode is used by default. +The `backend/` directory contains a .NET 10 implementation that provides the same journal functionality as the Python layer, with encrypted vault support and an identical JSON command protocol. -## Sidecar CLI +### Projects -`Journal.Sidecar` also supports direct vault and search CLI commands: +- **Journal.Core** — shared library: domain models, services, repositories, DTOs +- **Journal.Api** — minimal ASP.NET Core web API (`/api/command` POST endpoint) +- **Journal.Sidecar** — console app (stdin/stdout JSON protocol or CLI with `vault` and `search` subcommands) +- **Journal.SmokeTests** — 70+ integration tests (no test framework dependency) -```powershell -# Load vaults into decrypted data workspace -dotnet run --project Journal.Sidecar/Journal.Sidecar.csproj -- vault load - -# Save (rebuild) monthly vaults from decrypted markdown files -dotnet run --project Journal.Sidecar/Journal.Sidecar.csproj -- vault save - -# Search entries (query + filters) -dotnet run --project Journal.Sidecar/Journal.Sidecar.csproj -- search "common text" --tag stress --type !TRIGGER --start-date 2026-02-01 --end-date 2026-02-28 --section Summary --checked "med taken" -``` - -Password prompt behavior: -- If `--password` is omitted, CLI prompts with `Vault password:` (hidden input in terminal mode). -- For automation/non-interactive use, pass `--password `. - -Optional path overrides: -- `--vault-dir ` -- `--data-dir ` -- Env fallback: `JOURNAL_VAULT_DIR`, `JOURNAL_DATA_DIR`, `JOURNAL_APP_DIR` - -Search CLI flags: -- positional `query` (optional) -- `--tag` / `-t` (repeatable) -- `--type` / `-y` (repeatable) -- `--start-date` / `-s` (`yyyy-MM-dd`) -- `--end-date` / `-e` (`yyyy-MM-dd`) -- `--section` / `-sec` -- `--checked` / `-chk` (repeatable) -- `--unchecked` / `-uchk` (repeatable) -- `--data-dir ` (optional override) - -## Config Keys (Parity Surface) - -`JournalConfigService` exposes and normalizes key settings expected from Python config: - -- Paths: `JOURNAL_PROJECT_ROOT`, `JOURNAL_APP_DIR`, `JOURNAL_DATA_DIR`, `JOURNAL_VAULT_DIR`, `JOURNAL_LOG_DIR`, `JOURNAL_PID_FILE`, `JOURNAL_SERVER_CONTROL_FILE` -- Vault format: `JOURNAL_MONTHLY_VAULT_FORMAT` (default `%Y-%m.vault`) -- AI endpoints/models: `CLOUDAI_API_KEY`, `CLOUDAI_API_URL`, `LLAMA_CPP_URL`, `LLAMA_CPP_MODEL`, `LLAMA_CPP_TIMEOUT`, `EMBEDDING_API_URL`, `EMBEDDING_MODEL_NAME`, `MODEL_CONTEXT_TOKENS`, `CHUNK_TOKEN_BUDGET` -- AI bridge mode: `JOURNAL_AI_PROVIDER` (`none` or `python-sidecar`), `JOURNAL_PYTHON_EXE`, `JOURNAL_AI_SIDECAR_PATH`, `JOURNAL_AI_TIMEOUT_MS` -- Speech/NLP: `MICROPHONE_DEVICE_INDEX`, `SPEECH_RECOGNITION_ENGINE`, `WHISPER_MODEL_SIZE`, `JOURNAL_NLP_BACKEND` - -### Command Format - -```json -{ - "action": "fragments.create", - "id": null, - "type": null, - "tag": null, - "payload": { "type": "!TRIGGER", "description": "stomach drop" } -} -``` - -**Fields:** -- `action` — The operation to perform (e.g. `fragments.list`, `fragments.create`) -- `id` — Target entity ID (for get/update/delete) -- `type` / `tag` — Filter parameters (for search) -- `payload` — Request body, deserialized into the appropriate DTO per action - -### Available Actions - -| Action | Description | Requires | -|--------|-------------|----------| -| `fragments.list` | List all fragments | — | -| `fragments.get` | Get fragment by ID | `id` | -| `fragments.create` | Create a new fragment | `payload` (CreateFragmentDto) | -| `fragments.update` | Update a fragment | `id`, `payload` (UpdateFragmentDto) | -| `fragments.delete` | Delete a fragment | `id` | -| `fragments.search` | Search by type/tag | `type` and/or `tag` | -| `entries.list` | List decrypted markdown entries in a data directory | optional `payload.dataDirectory` | -| `entries.load` | Load one entry file and return parsed metadata + raw content | `payload.filePath` | -| `entries.save` | Save/merge entry content to file (fragment append or full merge path) | `payload.content`, optional `payload.filePath`, `payload.mode` | -| `db.status` | Return DB key/schema compatibility status snapshot | `payload.password`, optional `payload.dataDirectory` | -| `db.initialize_schema` | Write SQL schema bootstrap (`journal_schema.sql`) for parity tables | optional `payload.dataDirectory` | -| `db.hydrate_workspace` | Perform C# DB hydration step for decrypted workspace (schema bootstrap + metadata) | `payload.password`, optional `payload.dataDirectory` | -| `config.get` | Return current backend config snapshot | — | -| `ai.health` | Return AI bridge health/provider status | — | -| `ai.summarize_entry` | Summarize one entry through AI provider | `payload.content`, optional `payload.fileStem` | -| `ai.summarize_all` | Summarize a set of entries through AI provider | `payload.entries[]` | -| `ai.chat` | Send chat prompt through AI provider bridge | `payload.prompt` | -| `ai.embed` | Generate embedding vector through AI provider bridge | `payload.content` | -| `search.entries` | Search decrypted entry content with optional parity filters | `payload.dataDirectory`, optional `payload.query`, `payload.section`, `payload.startDate`, `payload.endDate`, `payload.tags[]`, `payload.types[]`, `payload.checked[]`, `payload.unchecked[]` | -| `vault.initialize` | Ensure vault directory exists | `payload.password`, `payload.vaultDirectory` | -| `vault.load_all` | Load/decrypt all monthly vaults into data directory | `payload.password`, `payload.vaultDirectory`, `payload.dataDirectory` | -| `vault.save_current_month` | Save only current month vault (optimized path) | `payload.password`, `payload.vaultDirectory`, `payload.dataDirectory`, optional `payload.nowUtc` | -| `vault.rebuild_all` | Rebuild all monthly vaults from decrypted `.md` data | `payload.password`, `payload.vaultDirectory`, `payload.dataDirectory` | -| `vault.clear_data_directory` | Clear decrypted data directory and recreate it | `payload.dataDirectory` | - -### Response Format - -Success: -```json -{ "ok": true, "data": { "id": "abc-123", "type": "!TRIGGER", "description": "...", "time": "...", "tags": [] } } -``` - -Error: -```json -{ "ok": false, "error": "Description is required" } -``` - -## Extending with New Modules - -The `Command` class is generic — new modules use the same dot-notation pattern: +### Architecture ``` -vault.unlock → IVaultService (future) -vault.lock -entries.list → IEntryService (future) -entries.create -ai.health → IAiService (implemented bridge) -ai.summarize_* → IAiService (implemented bridge) -ai.chat → IAiService (implemented bridge) -ai.embed → IAiService (implemented bridge) -db.status → IJournalDatabaseService (in-progress DB parity) -search.query → ISearchService (future) +Entry (thin command dispatcher) + ├── IFragmentService → FragmentService → IFragmentRepository + ├── IEntryFileService → EntryFileService → IEntryFileRepository + ├── IEntrySearchService → EntrySearchService + ├── IVaultStorageService → VaultStorageService → IVaultCryptoService + ├── IJournalDatabaseService → JournalDatabaseService (SQLCipher) + ├── IAiService → PythonSidecarAiService | DisabledAiService + ├── ISpeechBridgeService → PythonSidecarSpeechService | DisabledSpeechBridgeService + ├── CommandLogger + └── IJournalConfigService → JournalConfigService ``` -To add a module: -1. Create model, DTO, repository, and service in `Journal.Core/` -2. Register the new service in `ServiceCollectionExtensions.cs` -3. Inject the service into `Entry.cs` and add cases to the action switch -4. No changes needed to `Command.cs` or `App.cs` +### Build & Run -## Dependency Injection - -`ServiceCollectionExtensions.cs` wires everything up. Any host (sidecar, API, tests) calls: - -```csharp -services.AddFragmentServices(); +```bash +cd backend +dotnet build ``` -This registers: -- `IFragmentRepository` → `FileFragmentRepository` (singleton — persisted fragment store) -- `IFragmentService` → `FragmentService` (transient — fresh instance per request) +Run the API server: -## Fragment Store Location +```bash +dotnet run --project Journal.Api +``` -`FileFragmentRepository` persists data to: +Run the sidecar (stdin/stdout mode): -- default: `.journal-sidecar/fragments.json` under current working directory -- override: `JOURNAL_FRAGMENT_STORE_PATH` environment variable +```bash +dotnet run --project Journal.Sidecar +``` -## Legacy Vault Compatibility Note +Sidecar CLI commands: -The legacy Python placeholder file `_init_vault.vault` is treated as obsolete. -During vault load, the C# backend ignores this file for decryption and removes it. -This preserves compatibility while migrating older vault directories forward. +```bash +dotnet run --project Journal.Sidecar -- vault load --password +dotnet run --project Journal.Sidecar -- vault save --password +dotnet run --project Journal.Sidecar -- search "your query" --tag stress --start-date 2026-02-01 +``` + +Run smoke tests: + +```bash +dotnet run --project Journal.SmokeTests +``` + +### Environment Variables + +- `JOURNAL_PROJECT_ROOT` — override project root detection +- `JOURNAL_DATA_DIR` / `JOURNAL_VAULT_DIR` — override data/vault paths +- `JOURNAL_AI_PROVIDER` — `none` (default) or `python-sidecar` +- `JOURNAL_PYTHON_EXE` — Python executable path (default: `python`) +- `JOURNAL_LOG_LEVEL` — `trace`, `debug`, `information`, `warning` (default), `error`, `critical` + +### Encryption + +- Vault: AES-256-GCM with PBKDF2-HMAC-SHA256 key derivation (600k iterations) +- Database: SQLCipher with PBKDF2-derived key +- Wire format matches the Python implementation for cross-language parity + +## Notes + +- Decrypted journal data in `journal/data` is cleared on graceful shutdown. +- Vault save/load commands remain unchanged.