SDT/docs/reliability-matrix-runbook.md

SDT Reliability Matrix Runbook

Purpose

Run and track SDT reliability verification across:

• Windows
• Linux
• macOS

with deterministic pass/fail outcomes for core orchestration flows.

Matrix Scope

Project-Type Coverage

• dotnet
• node/npm
• tauri/cargo

Validation Categories

• workflow planning/execution determinism
• prerequisite probe/install gating behavior
• versioned event JSON contract validity
• diagnostics bundle generation on failure
• deterministic stop reason and exit-code mapping

Execution Paths

CI (Preferred)

Use the GitHub Actions workflow:

• .github/workflows/reliability-matrix.yml

Triggers:

• pull requests
• pushes to main
• manual workflow_dispatch

Expected artifacts per OS:

• test-results.trx
• reliability-<os>.json

Local Spot Checks

Run:

dotnet build DevTool.csproj -c Release
dotnet test tests/DevTool.Tests/DevTool.Tests.csproj -c Release

Result Recording

After each CI matrix run:

1. Collect uploaded artifacts for all OS jobs.
2. Update docs/reliability-matrix-results.md with:
   • commit SHA
   • run id
   • per-OS status
   • notes on failures/flakes
3. If any OS fails:
   • open/attach issue with failing artifact references
   • do not mark matrix item complete in ROADMAP.md

Completion Criteria

Matrix verification is considered complete when:

1. A full CI run passes on all three OS runners.
2. Results are logged in docs/reliability-matrix-results.md.
3. ROADMAP.md matrix checkboxes are updated accordingly.