sandpypi/ROADMAP.md

Roadmap

This file is the single roadmap for the C# engine, the first falling-sand game, and the experimental chunk-engine program.

Status labels:

• [x] done
• [-] partial / in progress
• [ ] not started

Current State

Repo and runtime

• [x] C# code split into Sand.Core, Sand.App, and Sand.Tests
• [x] dense production backend with fixed-timestep stepping, pressure, thermal, explosions, and tool fields
• [x] main app backend seam with dense default and chunk experimental selection via SAND_BACKEND
• [x] benchmark harness under Sand.Benchmarks
• [x] chunk residency/prototype runtime under Sand.ChunkPrototype
• [x] imported advchksys kept as a support/prototype asset, not the production runtime substrate

Chunk status

• [x] chunk residency scaffold exists
• [x] chunk backend is integrated into the main app as an experimental backend
• [x] chunk motion-regression tests cover wake-on-paint and shimmer issues found so far
• [x] active-move bottleneck instrumentation completed
• [-] high-motion chunk workloads are materially more stable in the real main app shell, and long-press gas overload no longer easily death-spirals the app, but mixed-scene runtime hotspots remain
• [x] chunk runtime can attribute active-frame cost to movement / runtime / activation / render buckets
• [-] live-app branch metrics now identify mixed-scene full-runtime work across solids, liquids, and gases as the main remaining single-thread hotspot after gas-stress stabilization
• [-] recent mixed-scene live runs show healthier attempt-to-move ratios and low render cost, so the remaining work is mostly true sim/runtime cost rather than frame catch-up collapse
• [ ] chunk backend competitive in the real main app shell
• [ ] chunk-local page-step architecture fully optimized
• [ ] chunk dirty-page render path fully paying off in the real app
• [x] chunk metrics/progress log added in docs/chunk-engine-progress.md

Parity and Engine Gaps

Dense/runtime gaps

• [-] movement-property tuning from metadata (velocity, friction, viscosity, heat_capacity)
• [-] pressure/breakage tuning polish
• [ ] deterministic snapshot capture and comparison harness
• [ ] richer UI/debug overlays beyond current parity level
• [ ] accelerator implementation behind ISimulationAccelerator

Chunk/runtime gaps

• [ ] chunk-local contiguous hot-path storage replacing the current prototype-level iteration costs
• [ ] chunk-local active-band stepping and better neighbor wake scheduling
• [ ] chunk-aware frame generation that clearly reduces bytes touched on localized scenes
• [ ] parity ladder for reactions, phase transitions, thermal, burning, and explosions
• [ ] deterministic dense-vs-chunk snapshot parity for ported subsystems

Next Milestones

Dense Track

• [-] continue movement-property and force tuning from metadata
• [ ] add snapshot capture/comparison harness for named scenes
• [ ] keep regression coverage growing around perf-sensitive fixes
• [ ] stage the first accelerator backend only after snapshot coverage exists

Chunk Engine Program

Chunk Program A: Truthful measurement and tracking

• [x] add dated chunk progress / metrics log
• [x] add app-sized benchmark mode matching the main app sim dimensions
• [x] record first dense-vs-chunk app baseline
• [x] split chunk measurements into clearer step-core / activation / render buckets
• [ ] keep milestone-relevant benchmark and app results logged in the progress doc

Chunk Program B: Step-core stabilization

• [x] remove wake-on-paint settling artifacts
• [x] remove trapped liquid/gas shimmer regressions currently covered by tests
• [x] add chunk motion-regression suite
• [-] continue stabilizing gas/liquid/solid motion under mixed tool-field stress

Chunk Program C: Chunk-local architecture rewrite

• [x] remove LINQ / global particle sorting from the chunk step hot path
• [x] introduce chunk-local page storage types and scheduler scaffolding
• [-] step active pages using dirty-band guidance
• [x] wake neighbor pages on border writes
• [ ] reduce remaining page-lookup and page-render overhead in real app workloads

Chunk Program D: Rendering + stats

• [-] dirty-page / dirty-chunk frame generation path
• [x] extend shared chunk frame stats
• [x] surface chunk workload metrics in the app debug overlay
• [ ] make localized chunk rendering wins clearly visible in app-sized measurements

Chunk Program E: Active-Move Optimization Gate

• [x] add move-attempt / move-result / stalled-motion metrics
• [x] add runtime bucket timing for movement / runtime / activation / field decay / render
• [x] add active-motion benchmark scenes and app procedures
• [-] reduce chunk cost on 10k-15k simultaneous move frames
• [x] verify chunk remains stable under gas-heavy active scenes enough to stop easy long-press collapse
• [x] document high-motion baselines and improvements in the progress log
• [-] use the stabilized gas/app baseline to reduce remaining mixed-scene full-runtime cost before resuming parity work

Chunk Program F: Feature parity ladder

• [ ] base motion parity improvements
• [ ] chunk force / pressure tuning
• [ ] common-material phase transitions
• [ ] core local reactions
• [ ] thermal subset
• [ ] burning/effect subset
• [ ] explosion / breakage subset

Chunk Program G: Promotion gate

• [ ] sparse app workload beats dense by >=25%
• [ ] mixed app workload beats dense by >=10%
• [ ] dense-like workload remains within 15%
• [ ] snapshot suite passes for all ported subsystems

Suggested Order

1. add active-move instrumentation and logging
2. establish dense vs chunk high-motion baselines in the real app shell
3. reduce chunk active-move overhead in the step core
4. confirm gains on gas-heavy and continuous-paint scenes
5. reduce remaining mixed-scene full-runtime hotspots from the stabilized single-thread baseline
6. plan multithreaded architecture from the stabilized single-thread baseline
7. only then resume chunk parity expansion and deeper engine storage promotion

Commands

Dense app:

Remove-Item Env:SAND_BACKEND -ErrorAction SilentlyContinue
dotnet run --project .\Sand.App\Sand.App.csproj -c Release

Chunk app:

$env:SAND_BACKEND='chunk'
dotnet run --project .\Sand.App\Sand.App.csproj -c Release

Benchmarks:

dotnet run --project .\Sand.Benchmarks\Sand.Benchmarks.csproj -c Release -- --mode app-sized

dotnet run --project .\Sand.Benchmarks\Sand.Benchmarks.csproj -c Release -- --mode snapshot-scenes

Notes

• Dense remains the production/reference backend while chunk continues as the experimental engine lane.
• The current chunk bottleneck is high simultaneous movement volume, not simple on-screen particle count.
• Gas-stress long-press collapse is now materially harder to trigger in the main app after overload control and gas-path tuning.
• The main remaining single-thread hotspot is mixed-scene full-runtime work across solids, liquids, and gases.
• Recent live-app runs show the stable-gas case is now mostly under control; the next optimization passes should target mixed runtime-chain cost rather than reopening gas-specific hacks.
• Recent mixed-scene runs also show the overload policy is holding: frame collapse is no longer primarily coming from runaway catch-up steps, but from the simulation work itself.
• Parity work is paused behind the active-move optimization gate.
• moves/frame and FPS are both required signals; benchmark-only wins are not enough by themselves.
• No particle-id-specific throttles are planned; chunk optimizations need to generalize across gas/runtime-heavy materials.
• Multithreaded architecture is a follow-on goal only after the single-thread chunk step is stable, measured, and behaviorally trustworthy.
• The chunk backend is now an engine program, not a one-off feasibility toy, but it still has to beat dense under real workloads before promotion.
• advchksys remains a support library and reference asset, not the final Sand engine storage contract.