Add sample-displacement/transparency corrections and standardize the FullProf verification suite

.github/workflows/nightly.yml → .github/workflows/nightly-test.yml

@@ -31,10 +31,32 @@ jobs:
       - name: Set up pixi
         uses: ./.github/actions/setup-pixi
 
-      - name: Run the full test suite (all tiers)
-        run: pixi run test-all
+      # Run each tier as its own step for per-suite visibility. The two
+      # tutorial steps must stay sequential — they share
+      # tmp/tutorials/projects/ — so they run as ordered steps rather than
+      # a single depends-on aggregate. `if: ${{ !cancelled() }}` lets a
+      # failure in one tier still report the others.
+      - name: Unit tests
+        run: pixi run unit-tests
+
+      - name: Functional tests
+        if: ${{ !cancelled() }}
+        run: pixi run functional-tests
+
+      - name: Integration tests
+        if: ${{ !cancelled() }}
+        run: pixi run integration-tests
+
+      - name: Tutorials as scripts + output check
+        if: ${{ !cancelled() }}
+        run: pixi run script-tests-checked
+
+      - name: Tutorials + verification notebooks + output check
+        if: ${{ !cancelled() }}
+        run: pixi run notebook-tests-checked
 
       - name: Run performance benchmarks
+        if: ${{ !cancelled() }}
         run: pixi run benchmarks
 
       - name: Upload benchmark results

.github/workflows/pypi-test.yml

@@ -10,17 +10,12 @@ on:
 # Allow only one concurrent workflow, skipping runs queued between the run
 # in-progress and latest queued. And cancel in-progress runs.
 concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
 
 permissions:
   contents: read
 
-# Set the environment variables to be used in all jobs defined in this workflow
-env:
-  CI_BRANCH: ${{ github.head_ref || github.ref_name }}
-  DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
-
 jobs:
   # Job 1: Test installation from PyPI on multiple OS
   pypi-package-tests:
@@ -77,10 +72,12 @@ jobs:
         run: pixi run python -m pytest ../tests/unit/ --color=yes -v
 
       - name: Run functional tests to verify the installation
+        if: ${{ !cancelled() }}
         working-directory: easydiffraction
         run: pixi run python -m pytest ../tests/functional/ --color=yes -v
 
       - name: Run integration tests to verify the installation
+        if: ${{ !cancelled() }}
         working-directory: easydiffraction
         run: pixi run python -m pytest ../tests/integration/ --color=yes -n auto
 

.github/workflows/test.yml

@@ -60,7 +60,7 @@ jobs:
       # Select the test cost tier by branch (see ADR test-suite-and-validation):
       # the pr tier runs on develop/master and pull requests; feature-branch
       # pushes run only the default (fast) tier. The nightly tier runs
-      # separately in nightly.yml.
+      # separately in nightly-test.yml.
       - name: Set test tier marker expression
         id: set-mark
         run: |

docs/dev/adrs/suggestions/lazy-pattern-recalculation.md

@@ -0,0 +1,106 @@
+# ADR: Lazy Pattern Recalculation
+
+## Status
+
+Proposed.
+
+## Date
+
+2026-06-09
+
+## Group
+
+Core model.
+
+## Context
+
+The calculated pattern of an experiment is exposed through the data
+category as plain array properties — `intensity_calc`, `intensity_bkg`,
+and the related totals — each built on access from the stored per-point
+descriptors (for example
+`np.fromiter(p.intensity_calc.value for p in _calc_items)`). Those
+stored values are only refreshed when something calls
+`_update_categories()`: today that happens inside `fit()`, the new
+`project.analysis.calculate()`, `as_cif`, and `display.pattern()`.
+
+As a consequence, reading a computed array directly after changing a
+model parameter returns **stale** values until the next explicit
+trigger:
+
+```python
+experiment.cell.length_a = 4.20      # model changed
+y = experiment.data.intensity_calc   # still the OLD pattern
+```
+
+Normal user flows hide this because plotting and CIF export call
+`_update_categories()` first, so they always render fresh data. The
+stale read only surfaces when code bypasses those high-level operations
+and reads the raw array — which is exactly the cross-engine verification
+pattern, and the reason `project.analysis.calculate()` was introduced as
+an explicit trigger.
+
+A more convenient API would recompute automatically when the data is
+read after a change, removing the need to remember an explicit call.
+
+## Decision (proposed)
+
+Introduce a per-experiment **dirty flag** for the calculated pattern:
+
+- Any change to a parameter that affects the pattern — cell, atom sites,
+  instrument, peak profile, linked-phase scale, calculator type,
+  constraints, aliases — marks the owning experiment's pattern dirty.
+- The **first** access to any computed array in the data category
+  (`intensity_calc`, `intensity_bkg`, totals, …) while the experiment is
+  dirty triggers a single full pattern recalculation, caches all
+  computed arrays, and clears the dirty flag.
+- **Subsequent** accesses to any computed array return the cached result
+  with no recalculation, until the next parameter change re-marks the
+  pattern dirty.
+
+The compute-once-per-change rule is the core requirement: needing both
+the calculated points and the background points (two array reads from
+the same data category) must trigger **one** recalculation, not two —
+only the first read after a change recomputes.
+
+`project.analysis.calculate()` remains available as an explicit trigger
+(and the explicit name documented in tutorials), but becomes optional:
+reading a computed array is always fresh on its own.
+
+## Consequences
+
+- Reading a computed array always reflects the current model; no
+  explicit `calculate()` call is required.
+- Robust dirty propagation is mandatory and is the hard part: **every**
+  pattern-affecting setter across structure, experiment, instrument,
+  peak, linked phases, constraints, and aliases must flip the flag. A
+  missed setter yields silently stale results — the worst failure mode
+  in a refinement tool — while over-flagging recomputes too often.
+- A property getter can now trigger an expensive diffraction
+  calculation; the per-change caching above keeps repeated reads cheap,
+  but the cache must stay coherent with the calculators' own internal
+  caching.
+- Structure changes are shared across experiments through linked phases,
+  so a structure edit must mark every linked experiment dirty.
+
+## Alternatives Considered
+
+- **Explicit `project.analysis.calculate()` (current).** Simple,
+  predictable, and consistent with `fit()`, but the user must remember
+  to call it before reading the raw array. Kept regardless of this ADR.
+- **Recompute on every array access (no caching).** Removes the dirty
+  flag but recomputes redundantly when several arrays are read in
+  sequence (calculated points then background) — too expensive.
+- **Lazy recompute with per-change caching (this proposal).**
+
+## Deferred Work / Open Questions
+
+- Where the dirty flag lives and how setters reach it: a hook on the
+  `Parameter`/descriptor base that notifies its owning experiment,
+  versus category-level invalidation.
+- Granularity: a single per-experiment pattern-dirty flag versus
+  finer-grained invalidation (for example structure-factor vs profile
+  terms).
+- Interaction with constraints and aliases, which mutate parameters
+  indirectly during `_update_categories()`.
+- Coordination with the existing calculator-level caches so the two
+  layers do not fight or double-cache.

docs/dev/issues/open.md

@@ -10,6 +10,55 @@ implemented, remove it from this file and update
 
 ---
 
+## 119. 🟡 Model Sample Absorption (Debye–Scherrer, μR)
+
+**Type:** Physics / Engine feature
+
+The calculators (`cryspy`, `crysfml`) apply no sample-absorption
+correction. For a cylindrical sample in Debye–Scherrer geometry this is
+an angle-dependent intensity factor that boosts high-angle peaks. The
+LaB₆ verification reference (`pd-neut-cwl_tch-fcj_lab6`) was refined in
+FullProf with `μR = 0.7`; the unmodelled correction is the _entire_
+intensity residual on the companion `pd-neut-cwl_tch-fcj_abs_lab6` page
+(≈5% profile difference), while the `μR = 0` page passes to corr 0.9999.
+
+**Correction (Hewat, Debye–Scherrer), validated to 4 decimals against
+FullProf output:**
+
+```
+A(θ) = exp( -(1.7133 − 0.0368·sin²θ)·μR + (0.0927 + 0.375·sin²θ)·μR² )
+```
+
+A Lobanov–Alte-da-Veiga form covers `μR > 3`.
+
+**Implementation sketch:**
+
+- Add a `μR` instrument parameter for CWL powder (Debye–Scherrer).
+- `crysfml`: CrysFML08 already implements this — reachable via
+  `Lorentz_abs_CW(..., ilor='DBS', cabs='HEWAT', tmv=μR)` through
+  pycrysfml.
+- `cryspy`: multiply each reflection's intensity by `A(θ_hkl)`,
+  analogous to the existing Lorentz factor (one extra term).
+
+**Note:** absorption is nearly degenerate with Biso + scale (its angle
+term is linear in `sin²θ`, like the Debye–Waller), so refining Biso can
+partly absorb it — but that biases Biso, so an explicit correction is
+preferable.
+
+**References:**
+
+- A. W. Hewat, _Acta Cryst._ A35 (1979) 248 — cylindrical absorption.
+- N. N. Lobanov & L. Alte da Veiga, 6th EPDIC, Abstract P12-16 (1998).
+- CrysFML08:
+  [`Src/CFML_Powder/Pow_Lorentz_Absorption.f90`](https://code.ill.fr/scientific-software/CrysFML2008/-/blob/master/Src/CFML_Powder/Pow_Lorentz_Absorption.f90),
+  `Lorentz_abs_CW`.
+- FullProf `μR`: `.pcr` Lambda line field 7; `iabscor = 2` selects
+  HEWAT.
+
+**Depends on:** adding a `μR` instrument parameter.
+
+---
+
 ## 3. 🟡 Rebuild Joint-Fit Weights on Every Fit
 
 **Type:** Fragility
@@ -2286,3 +2335,4 @@ parameters (cryspy/crysfml) before the second item can be wired through.
 | 116 | Add a static type checker to the quality gate     | 🟡 Med   | Tooling / Correctness        |
 | 117 | Live-notebook Plotly: loader vs native mimetype   | 🟢 Low   | Display / Architecture       |
 | 118 | Plotly empty rows in the VISA JupyterLab          | 🟢 Low   | Display / Environment        |
+| 119 | Model sample absorption (Debye–Scherrer μR)       | 🟡 Med   | Physics / Engine feature     |