70 lines
1.5 KiB
Markdown
70 lines
1.5 KiB
Markdown
# SDT Reliability Matrix Runbook
|
|
|
|
## Purpose
|
|
Run and track SDT reliability verification across:
|
|
|
|
- Windows
|
|
- Linux
|
|
- macOS
|
|
|
|
with deterministic pass/fail outcomes for core orchestration flows.
|
|
|
|
## Matrix Scope
|
|
|
|
### Project-Type Coverage
|
|
- `dotnet`
|
|
- `node/npm`
|
|
- `tauri/cargo`
|
|
|
|
### Validation Categories
|
|
- workflow planning/execution determinism
|
|
- prerequisite probe/install gating behavior
|
|
- versioned event JSON contract validity
|
|
- diagnostics bundle generation on failure
|
|
- deterministic stop reason and exit-code mapping
|
|
|
|
## Execution Paths
|
|
|
|
### CI (Preferred)
|
|
Use the GitHub Actions workflow:
|
|
|
|
- `.github/workflows/reliability-matrix.yml`
|
|
|
|
Triggers:
|
|
- pull requests
|
|
- pushes to `main`
|
|
- manual `workflow_dispatch`
|
|
|
|
Expected artifacts per OS:
|
|
- `test-results.trx`
|
|
- `reliability-<os>.json`
|
|
|
|
### Local Spot Checks
|
|
Run:
|
|
|
|
```powershell
|
|
dotnet build DevTool.csproj -c Release
|
|
dotnet test tests/DevTool.Tests/DevTool.Tests.csproj -c Release
|
|
```
|
|
|
|
## Result Recording
|
|
After each CI matrix run:
|
|
|
|
1. Collect uploaded artifacts for all OS jobs.
|
|
2. Update `docs/reliability-matrix-results.md` with:
|
|
- commit SHA
|
|
- run id
|
|
- per-OS status
|
|
- notes on failures/flakes
|
|
3. If any OS fails:
|
|
- open/attach issue with failing artifact references
|
|
- do not mark matrix item complete in `ROADMAP.md`
|
|
|
|
## Completion Criteria
|
|
Matrix verification is considered complete when:
|
|
|
|
1. A full CI run passes on all three OS runners.
|
|
2. Results are logged in `docs/reliability-matrix-results.md`.
|
|
3. `ROADMAP.md` matrix checkboxes are updated accordingly.
|
|
|