sandpypi/chunk-engine-progress.md

Chunk Engine Progress

Purpose

This log tracks milestone-relevant chunk-engine progress, blockers, benchmark runs, and real-app comparison data. It is not meant to capture every casual test run.

Current headline status

• Dense is still the production/reference backend.
• The chunk backend is integrated into the main app and benchmark harness as the main experimental lane.
• Chunk now has page-local storage, active-page scheduling, dirty-chunk rendering, motion-regression coverage, and app-sized benchmark modes.
• High simultaneous move volume is now the primary chunk optimization blocker.
• Further chunk parity work is gated behind active-move instrumentation and optimization.
• Post-instrumentation baselines show chunk still ahead on settled/localized workloads, but behind dense on sustained active-move scenes.
• Live app branch-level counters now show the stable long-press gas case is materially more controlled than before.
• The main remaining single-thread hotspot is now mixed-scene full-runtime work across solids, liquids, and gases.
• New live-app stress runs show gas-specific retry/runtime work is harder to spike, while mixed material scenes still pay too much broad runtime-chain cost.

Open blockers

• Chunk still needs more branch-level metrics to fully explain attempt spikes across all material classes.
• Gas-heavy and continuous-paint scenes still need targeted high-move optimization.
• Mixed scenes still route too many runtime-heavy particles through broad lifecycle / thermal / reaction / pressure work in the same frame.
• Move-attempt cost and stalled-motion cost need clearer measurement and trend tracking.
• Stable gas and overload behavior are improved, but mixed-scene full-runtime cost still needs to come down before parity work resumes.
• Real-app chunk results still need to beat dense consistently, not only in selected scenes.
• Chunk parity remains paused behind the active-move optimization gate.

Manual app scenarios

active_sand_flood

• Setup: continuously pour a dense falling solid stream until the scene sustains 10k+ simultaneous moves.
• Measure: average FPS, minimum FPS, move attempts, successful moves, stalled movable cells.
• Acceptance: no wake/sleep artifact and move cost scales better than the recorded baseline.

mixed_pile

• Setup: create a layered sand/water pile over a floor and let it settle for 30 seconds.
• Measure: average FPS, minimum FPS, stability of settling.
• Acceptance: no obvious shimmer or frozen partial-settle behavior.

active_gas_burst

• Setup: inject a large gas burst in a confined chamber until gas dominates movement.
• Measure: average FPS, minimum FPS, move attempts, full-runtime step count.
• Acceptance: gas remains visually coherent and does not dominate frame time disproportionately.

continuous_mixed_paint

• Setup: sustain mixed solid/liquid/gas painting for 30 seconds.
• Measure: FPS plus movement/runtime bucket timings.
• Acceptance: no severe collapse when move volume spikes.

tool_stress

• Setup: apply wind/air/gravity tools continuously over a mixed-material scene for 30 seconds.
• Measure: FPS, visible stability, and chunk field activity.
• Acceptance: field-driven motion remains active without repainting.

Metrics table

Required columns for recorded runs:

• date
• commit or branch label
• backend
• build
• scene name
• world size
• particle count
• avg FPS
• min FPS
• avg step ms
• avg frame/build ms
• loaded chunks
• active chunks
• stepped chunks
• dirty chunks
• field cells
• move attempts
• successful moves
• swap attempts
• stalled movable cells
• movement-only fast-path count
• full-runtime step count
• activation ms
• movement ms
• runtime ms
• field decay ms
• render ms
• notes

Dated entries

2026-03-09

• Replaced the chunk prototype hot path with page-local storage/scheduling scaffolding and removed the old global sparse/LINQ step path.
• Added app-sized benchmark mode and snapshot mode to Sand.Benchmarks.
• Extended shared/app frame stats with chunk workload counters and surfaced them in the app debug overlay.
• Reordered the chunk program around an active-move optimization gate before further parity expansion.
• Added move-attempt, stalled-motion, fast-path/full-runtime, and coarse timing buckets to chunk step stats, app overlay, and benchmark output.
• Recorded the first post-instrumentation high-motion baseline below.
• Added row-count guided stepping, narrower border wakes, same-page occupancy fast paths, and limited gas-runtime throttling.
• Result: continuous_mixed_paint improved materially and active_sand_flood improved versus the prior local pass, but chunk is still behind dense on sustained flood and gas-burst scenes.
• Added branch-level move-attempt counters (vertical/diagonal/lateral) plus full-runtime counters by particle kind (solid/liquid/gas) to the chunk overlay.
• Live-app chunk gas stress with ultratanium now shows the runtime bottleneck clearly: a representative run reported moves 2357, attempts 9161, Att v 5467 d 2221 l 1073, full 5644, s 0 l 0 g 5644, run 11.52 ms, render 0.52 ms.
• Interpretation: render is not the primary limiter in the observed gas stress case; the next optimization pass should reduce general full-runtime gas exposure and repeated vertical gas probes without particle-specific throttles.
• Follow-up retry-reduction work made the worst spikes harder to trigger in the live app, but later ultratanium screenshots still showed large full-runtime gas counts once the gas mass interacted with terrain and boundaries.
• Interpretation update: the next generic gas optimization should focus on thermally stable gas near non-gas boundaries, not only open-cloud gas and not particle-id-specific behavior.

2026-03-10

• Added app-shell frame instrumentation and overload control so the main app now reports total frame, update, sim-loop, build, upload, draw, and fixed-step counts.
• Result: long-press gas abuse is materially harder to use to collapse the app, because the fixed-step loop no longer tries to chase every missed step forever under overload.
• Recent live-app screenshots now show a clearer split:
  • stable gas runs can remain responsive with Full 0, meaning the worst gas-runtime branch is no longer the default limiter
  • mixed scenes still fall back into expensive full-runtime work across solids, liquids, and gases
• Updated single-thread focus: stop treating gas-only stress as the only blocker and reduce the mixed-scene runtime chain before planning multithreading.
• Current code work is trimming the chunk full-runtime path so particles only pay for lifecycle, burn, emission, special, phase, reaction, pressure, and thermal domains when they actually use them.
• Latest mixed-scene live tests after runtime trimming showed a much healthier profile at roughly 44.7k particles:
  • FPS 23
  • moves 4936, attempts 5466, stalled 218
  • app frame 44.07 ms, sim 41.55 ms, steps 2
  • render 1.18 ms, draw 0.97 ms
• Interpretation update: the app is no longer mostly failing from catch-up debt or render/upload cost in this stress case. The remaining hotspot is honest mixed-scene simulation/runtime work.

Baseline measurements

date	commit/label	backend	build	scene	world	particles	avg FPS	min FPS	avg step ms	avg frame/build ms	loaded chunks	active chunks	stepped chunks	dirty chunks	field cells	move attempts	successful moves	swap attempts	stalled movable cells	movement-only fast-path count	full-runtime step count	activation ms	movement ms	runtime ms	field decay ms	render ms	notes
2026-03-09	local-working-tree	dense	Release	active_sand_flood	265x200	1920	433.46	n/a	2.265	0.042	0	0	0	0	0	0.0	0.0	0.0	0.0	0.0	0.0	0.000	0.000	0.000	0.000	0.042	dense baseline for sustained falling solids
2026-03-09	local-working-tree	chunk	Release	active_sand_flood	265x200	2520	200.45	n/a	4.806	0.182	21	21	21	21	0	1330.0	1330.0	0.0	0.0	1330.0	0.0	0.016	4.722	0.000	0.001	0.182	movement-only work dominates; chunk still behind dense here
2026-03-09	local-working-tree	dense	Release	active_gas_burst	265x200	1566	1150.01	n/a	0.848	0.022	0	0	0	0	0	0.0	0.0	0.0	0.0	0.0	0.0	0.000	0.000	0.000	0.000	0.022	dense baseline for sustained gas injection
2026-03-09	local-working-tree	chunk	Release	active_gas_burst	265x200	2130	290.73	n/a	3.406	0.034	16	16	10	10	0	1143.8	1147.3	0.0	0.0	142.8	1009.8	0.005	0.005	3.024	0.001	0.034	full-runtime gas path is the dominant blocker
2026-03-09	local-working-tree	dense	Release	continuous_mixed_paint	265x200	2216	697.89	n/a	1.411	0.022	0	0	0	0	0	0.0	0.0	0.0	0.0	0.0	0.0	0.000	0.000	0.000	0.000	0.022	dense baseline for sustained mixed painting
2026-03-09	local-working-tree	chunk	Release	continuous_mixed_paint	265x200	2196	323.38	n/a	3.016	0.076	23	23	23	23	0	1320.5	991.1	0.4	0.1	386.8	388.9	0.008	0.001	2.233	0.001	0.076	movement and runtime both contribute under mixed active paint

Next actions

• reduce move-heavy cost on active_sand_flood, active_gas_burst, and continuous_mixed_paint
• reduce mixed-scene full-runtime cost by narrowing the lifecycle / reaction / pressure / thermal chain per particle
• keep stable-gas wins intact while mixed-scene runtime work is reduced
• keep ultratanium as a stress particle, but avoid particle-id-specific throttles or hacks
• continue reducing chunk overhead on gas-heavy and continuous-paint workloads
• use the healthier mixed-scene overlay numbers as the single-thread baseline before any multithread planning
• return to same-page falling-solid movement overhead after the mixed-scene runtime chain is under better control
• add deterministic snapshot review for the named scenes after 120 and 300 steps
• keep further parity work paused until the active-move gate shows repeatable gains