sandpypi/ROADMAP.md

# Roadmap

This file is the single roadmap for the C# engine, the first falling-sand game, and the experimental chunk-engine program.

Status labels:

- `[x]` done
- `[-]` partial / in progress
- `[ ]` not started

## Current State

### Repo and runtime

- `[x]` C# code split into `Sand.Core`, `Sand.App`, and `Sand.Tests`
- `[x]` dense production backend with fixed-timestep stepping, pressure, thermal, explosions, and tool fields
- `[x]` main app backend seam with dense default and chunk experimental selection via `SAND_BACKEND`
- `[x]` benchmark harness under `Sand.Benchmarks`
- `[x]` chunk residency/prototype runtime under `Sand.ChunkPrototype`
- `[x]` imported `advchksys` kept as a support/prototype asset, not the production runtime substrate

### Chunk status

- `[x]` chunk residency scaffold exists
- `[x]` chunk backend is integrated into the main app as an experimental backend
- `[x]` chunk motion-regression tests cover wake-on-paint and shimmer issues found so far
- `[x]` active-move bottleneck instrumentation completed
- `[-]` high-motion chunk workloads are materially more stable in the real main app shell, and long-press gas overload no longer easily death-spirals the app, but mixed-scene runtime hotspots remain
- `[x]` chunk runtime can attribute active-frame cost to movement / runtime / activation / render buckets
- `[-]` live-app branch metrics now identify mixed-scene full-runtime work across solids, liquids, and gases as the main remaining single-thread hotspot after gas-stress stabilization
- `[-]` recent mixed-scene live runs show healthier attempt-to-move ratios and low render cost, so the remaining work is mostly true sim/runtime cost rather than frame catch-up collapse
- `[ ]` chunk backend competitive in the real main app shell
- `[ ]` chunk-local page-step architecture fully optimized
- `[ ]` chunk dirty-page render path fully paying off in the real app
- `[x]` chunk metrics/progress log added in [`docs/chunk-engine-progress.md`](/f:/Documents/Dev/sandpypi/docs/chunk-engine-progress.md)

## Parity and Engine Gaps

### Dense/runtime gaps

- `[-]` movement-property tuning from metadata (`velocity`, `friction`, `viscosity`, `heat_capacity`)
- `[-]` pressure/breakage tuning polish
- `[ ]` deterministic snapshot capture and comparison harness
- `[ ]` richer UI/debug overlays beyond current parity level
- `[ ]` accelerator implementation behind `ISimulationAccelerator`

### Chunk/runtime gaps

- `[ ]` chunk-local contiguous hot-path storage replacing the current prototype-level iteration costs
- `[ ]` chunk-local active-band stepping and better neighbor wake scheduling
- `[ ]` chunk-aware frame generation that clearly reduces bytes touched on localized scenes
- `[ ]` parity ladder for reactions, phase transitions, thermal, burning, and explosions
- `[ ]` deterministic dense-vs-chunk snapshot parity for ported subsystems

## Next Milestones

### Dense Track

- `[-]` continue movement-property and force tuning from metadata
- `[ ]` add snapshot capture/comparison harness for named scenes
- `[ ]` keep regression coverage growing around perf-sensitive fixes
- `[ ]` stage the first accelerator backend only after snapshot coverage exists

### Chunk Engine Program

#### Chunk Program A: Truthful measurement and tracking

- `[x]` add dated chunk progress / metrics log
- `[x]` add app-sized benchmark mode matching the main app sim dimensions
- `[x]` record first dense-vs-chunk app baseline
- `[x]` split chunk measurements into clearer step-core / activation / render buckets
- `[ ]` keep milestone-relevant benchmark and app results logged in the progress doc

#### Chunk Program B: Step-core stabilization

- `[x]` remove wake-on-paint settling artifacts
- `[x]` remove trapped liquid/gas shimmer regressions currently covered by tests
- `[x]` add chunk motion-regression suite
- `[-]` continue stabilizing gas/liquid/solid motion under mixed tool-field stress

#### Chunk Program C: Chunk-local architecture rewrite

- `[x]` remove LINQ / global particle sorting from the chunk step hot path
- `[x]` introduce chunk-local page storage types and scheduler scaffolding
- `[-]` step active pages using dirty-band guidance
- `[x]` wake neighbor pages on border writes
- `[ ]` reduce remaining page-lookup and page-render overhead in real app workloads

#### Chunk Program D: Rendering + stats

- `[-]` dirty-page / dirty-chunk frame generation path
- `[x]` extend shared chunk frame stats
- `[x]` surface chunk workload metrics in the app debug overlay
- `[ ]` make localized chunk rendering wins clearly visible in app-sized measurements

#### Chunk Program E: Active-Move Optimization Gate

- `[x]` add move-attempt / move-result / stalled-motion metrics
- `[x]` add runtime bucket timing for movement / runtime / activation / field decay / render
- `[x]` add active-motion benchmark scenes and app procedures
- `[-]` reduce chunk cost on `10k-15k` simultaneous move frames
- `[x]` verify chunk remains stable under gas-heavy active scenes enough to stop easy long-press collapse
- `[x]` document high-motion baselines and improvements in the progress log
- `[-]` use the stabilized gas/app baseline to reduce remaining mixed-scene full-runtime cost before resuming parity work

#### Chunk Program F: Feature parity ladder

- `[ ]` base motion parity improvements
- `[ ]` chunk force / pressure tuning
- `[ ]` common-material phase transitions
- `[ ]` core local reactions
- `[ ]` thermal subset
- `[ ]` burning/effect subset
- `[ ]` explosion / breakage subset

#### Chunk Program G: Promotion gate

- `[ ]` sparse app workload beats dense by `>=25%`
- `[ ]` mixed app workload beats dense by `>=10%`
- `[ ]` dense-like workload remains within `15%`
- `[ ]` snapshot suite passes for all ported subsystems

## Suggested Order

1. add active-move instrumentation and logging
2. establish dense vs chunk high-motion baselines in the real app shell
3. reduce chunk active-move overhead in the step core
4. confirm gains on gas-heavy and continuous-paint scenes
5. reduce remaining mixed-scene full-runtime hotspots from the stabilized single-thread baseline
6. plan multithreaded architecture from the stabilized single-thread baseline
7. only then resume chunk parity expansion and deeper engine storage promotion

## Commands

Dense app:

```powershell
Remove-Item Env:SAND_BACKEND -ErrorAction SilentlyContinue
dotnet run --project .\Sand.App\Sand.App.csproj -c Release
```

Chunk app:

```powershell
$env:SAND_BACKEND='chunk'
dotnet run --project .\Sand.App\Sand.App.csproj -c Release
```

Benchmarks:

```powershell
dotnet run --project .\Sand.Benchmarks\Sand.Benchmarks.csproj -c Release -- --mode app-sized
```

```powershell
dotnet run --project .\Sand.Benchmarks\Sand.Benchmarks.csproj -c Release -- --mode snapshot-scenes
```

## Notes

- Dense remains the production/reference backend while chunk continues as the experimental engine lane.
- The current chunk bottleneck is high simultaneous movement volume, not simple on-screen particle count.
- Gas-stress long-press collapse is now materially harder to trigger in the main app after overload control and gas-path tuning.
- The main remaining single-thread hotspot is mixed-scene full-runtime work across solids, liquids, and gases.
- Recent live-app runs show the stable-gas case is now mostly under control; the next optimization passes should target mixed runtime-chain cost rather than reopening gas-specific hacks.
- Recent mixed-scene runs also show the overload policy is holding: frame collapse is no longer primarily coming from runaway catch-up steps, but from the simulation work itself.
- Parity work is paused behind the active-move optimization gate.
- `moves/frame` and FPS are both required signals; benchmark-only wins are not enough by themselves.
- No particle-id-specific throttles are planned; chunk optimizations need to generalize across gas/runtime-heavy materials.
- Multithreaded architecture is a follow-on goal only after the single-thread chunk step is stable, measured, and behaviorally trustworthy.
- The chunk backend is now an engine program, not a one-off feasibility toy, but it still has to beat dense under real workloads before promotion.
- `advchksys` remains a support library and reference asset, not the final Sand engine storage contract.