feat(skill-memory): per-skill cross-session recall + historian auto-extraction

[iceteaSA]

Summary

Adds skill-memory — per-skill "motor memory" that gives a skill cross-session recall of its own hard-won lessons (gotchas, discoveries, fixes, workflow steps). When a skill's SKILL.md declares skill-memory: { enabled: true }, accumulated notes for that skill surface automatically in a <skill-memory> block appended to the skill tool's result on every load — and, with the historian extension, the historian writes those notes automatically during compaction (no agent action required).

It's fully opt-in per skill and cache-safe by construction: the block rides the tool-result tail (conversation), never the cached system/m[0] prefix, so it can't bust the prompt cache.

How it works

• Transparent recall — three-hook augmentation: tool.definition advertises an intent param; tool.execute.before stashes the intent (bounded TTL); the after-hook parses the skill's Base directory, reads its SKILL.md frontmatter, and formats the recall block. Lands in the tool RESULT (cache-safe).
• Write paths — ctx_skill_note (agent-authored) and the historian (auto-extracted). Both dedup on a normalized hash.
• Intent-scoped ranking — a recall cascade: model-matched embeddings → cosine blend over intent_embedding + delta_embedding (relevance/recency/hit weights tunable per skill via ranking_* frontmatter); FTS5 fallback over a content-linked skill_memory_fts vtable; flat recency×hit fallback.
• Historian auto-extraction — the historian sees TC: skill(<name>) markers in its chunk and emits a <skill_observations> block; both the OpenCode and Pi runners promote those post-commit as global notes (project_identity='*', source_type='historian'), recallable from any project.

Review units (4 commits)

The branch is organized into four coherent, reviewable phases:

1. P1 — transparent per-skill recall: skill_memory table migration, the three-hook augmentation, flat recall + storage, ctx_skill_note/ctx_skill_recall tools, opt-in distill-skill-memory dreamer task, TUI/ctx-status stats, docs.
2. P2 — embeddings + intent-scoped recall: delta_embedding + recall_count columns + skill_memory_fts FTS5 vtable; embed-on-write; the cosine/FTS recall cascade; a programmatic (no-LLM) reembed pre-step.
3. P3a — historian-extraction foundation: the TC: skill(<name>) marker keystone; origin_project + source_type columns; global-tier notes unified under project_identity='*' (collision-merge); partitionKey routing.
4. P3b — historian auto-extraction pipeline: prompt emits <skill_observations>, parser, validated-result threading, both runners promote via the shared promoteSkillObservations helper, plus an initializeDatabase self-heal net (re-creates skill_memory + ensureColumn so an upgraded DB recovers even if a migration row is lost).

Schema / migrations

Three migrations (numbered after the current master ceiling): skill_memory table, then delta_embedding/recall_count/FTS, then origin_project/source_type/* unification. LATEST_SUPPORTED_VERSION bumped in lockstep (the schema-version-fence test enforces it). Migration bodies are ensureColumn/IF NOT EXISTS idempotent.

Testing

Full plugin + Pi suites pass (one unrelated pre-existing full-suite ordering flake in tui-config.test.ts that passes in isolation), tsc clean both packages, lint clean, build produces all bundles. Dedicated coverage for the migrations (coexistence + fence), recall rungs, the tools, FTS triggers/backfill, the '*' collision-merge, and the historian promotion path on both runners. Verified working live: the historian auto-extraction writes genuine source_type='historian' notes, embeddings populate, and the read-side recall_count increments on surfacing.

Notes for reviewers

• Migration version numbers are placeholders relative to this fork's base — happy to renumber to whatever slots are free at merge time.
• The four commits are independently meaningful; if you'd prefer this as stacked PRs (P1 / P2 / historian), I can split it.

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Adds per-skill cross-session memory with embedding + FTS ranking and historian auto-extraction. Each skill result now includes a <skill-memory> block; ctx_skill_note/ctx_skill_recall manage notes, and the distill-skill-memory task pre-embeds and reports corpus health (TUI/ctx-status show stats).

• New Features

  • Transparent, opt-in per-skill recall: the skill tool accepts optional intent; before/after hooks stash intent and inject recall; enable via SKILL.md frontmatter (skill-memory: { enabled: true }). TUI/ctx-status show per-project totals and pinned counts.
  • Intent-scoped ranking: cosine blend over intent_embedding + delta_embedding with FTS5 fallback; recall_count tracked; a pre-step re-embeds NULL/stale vectors before the distill run.
  • Historian auto-extraction: TC: skill(<name>) markers → <skill_observations>; both OpenCode and Pi promote via promoteSkillObservations as global '*' notes (source_type='historian').
  • Distill task: distill-skill-memory (off by default) runs the re-embed pre-step and emits a read-only corpus health report; added to config schema, CLI setup, and dashboard.
  • Schema/self-heal: migrations v51–v53 add skill_memory + FTS and origin_project/source_type; initializeDatabase can rebuild skill_memory_fts; global '*' partition helpers and project-scoped stats added.

• Bug Fixes

  • Cross-partition recall: union the skill’s project partition with global '*' so project-local skills surface historian notes; writes/dedup stay exact-partition.
  • Frontmatter/provenance: reject ctx_skill_note when disabled; accept inline { enabled: true }; start-of-file anchored with BOM/whitespace tolerance and inline # handling; accept plain paths and file://; normalize Windows paths; keep the last, line-anchored “Base directory” match.
  • Guidance/observability/budget: drop ctx_memory cross-ref when memory is off; log recall errors; count per-note XML framing and clamp pinned budget.
  • Safety + misc: avoid init crash when the plugin is disabled; session-scoped intent stash (sessionId:callId); escape apostrophes in SQL; compiled TUI regenerated so the Skill Memory section renders.
  • Tests: replace global embedding mock.module with the provider-factory seam (_setTestProviderFactoryForProject + registerProjectEmbedding) to stop cross-file bleed and timeouts; stabilize embedding-related tests.

^{Written for commit 8b3af4e. Summary will update on new commits.}

Greptile Summary

This PR adds a skill-memory feature — opt-in, per-skill cross-session recall that injects a <skill-memory> block into the skill tool's result. Notes are stored in a new skill_memory SQLite table (migrations v51–v53), retrieved via a cosine/FTS5/flat-recency cascade, and auto-extracted by the historian from TC: skill(<name>) markers in the conversation transcript.

• Three-hook augmentation: tool.definition injects an optional intent param into the skill tool schema; tool.execute.before stashes intent keyed by ${sessionID}:${callID} with 60s TTL and 256-entry cap; tool.execute.after parses provenance from the skill output, reads SKILL.md frontmatter, and appends the recall block to the tool result — cache-safe because it touches only the conversation tail.
• Two write paths: ctx_skill_note (agent-authored) and promoteSkillObservations (historian extraction); both hash-dedup and reject disabled skills before writing.
• Historian pipeline: the compartment parser extracts <skill_observations> blocks emitted by the historian; both the OpenCode and Pi runners promote them as global '*'-partition notes; recall unions the skill's own partition with the global partition so project-local skills surface historian notes.

Confidence Score: 5/5

Safe to merge. All previously reported bugs are fixed with regression tests, the three migrations are idempotent, and the recall cascade degrades gracefully on DB or embedding failures.

All four previously flagged defects (intent-stash cross-session eviction, duplicate dynamic import, FTS self-heal gap, and orphaned historian notes for project-local skills) are addressed with targeted fixes and added regression tests. The new code handles failure modes explicitly — recall errors are logged and return empty string rather than throwing, historian promotion is best-effort per observation, and the FTS rebuild is guarded against repeated execution. The only remaining findings are style/clarity nits with no behavioral impact.

No files require special attention. The most complex new logic — partition routing in storage.ts and the intent stash lifecycle in hook-handlers.ts — are both well-tested and the invariants are clearly documented inline.

Important Files Changed

Filename	Overview
packages/plugin/src/features/magic-context/skill-memory/storage.ts	Core storage layer for skill_memory notes. Introduces recallPartitionPredicate (unions own partition + global '*') and all CRUD helpers. searchSkillMemoryFts calls recallPartitionPredicate twice with the same args — the clause and binds are computed separately rather than from a single destructure.
packages/plugin/src/features/magic-context/skill-memory/recall.ts	Shared recall core used by both the transparent after-hook and ctx_skill_recall. Implements the cosine → FTS5 → flat cascade, budgetFill, and XML block builder. Recall errors are now logged (fixing the previous bare-catch issue). Logic is sound.
packages/plugin/src/hooks/magic-context/hook-handlers.ts	Adds intent stash map (keyed sessionID:callID with 60s TTL / 256-cap), createToolExecuteBeforeHook to capture intent, and skill-memory injection to createToolExecuteAfterHook. Previously reported intentByCallId cross-session eviction and duplicate import bugs are both fixed.
packages/plugin/src/features/magic-context/migrations.ts	Adds migrations v51 (skill_memory table), v52 (delta_embedding + recall_count + FTS5), v53 (origin_project + source_type + global '*' unification). All idempotent via IF NOT EXISTS / columnExists guards. v53 collision-merge runs in a transaction.
packages/plugin/src/features/magic-context/skill-memory/promote.ts	Historian promotion helper. Correctly writes to tier='global', project_identity='*' with hash dedup. Previously reported orphan issue (project-local skills never recalling historian notes) is fixed via recallPartitionPredicate in the storage layer.
packages/plugin/src/tools/ctx-skill-note/tools.ts	Agent-authored note tool. Guards disabled-skill writes correctly (previously reported bug is fixed). Exact hash + semantic cosine dedup before insert. Correct partition routing for project vs. global tier.
packages/plugin/src/tools/ctx-skill-recall/tools.ts	Explicit recall tool with registry-first + disk-fallback resolution. Cold-start disk search covers the standard opencode paths but not ancestor walk-up or custom cfg.skills.paths, which is a known limitation noted in comments.
packages/plugin/src/features/magic-context/storage-db.ts	initializeDatabase self-heal creates skill_memory and skill_memory_fts (with triggers) and performs a guarded FTS rebuild if the index is empty but the table has rows. Previously reported missing-rebuild bug is fixed.
packages/plugin/src/features/magic-context/skill-memory/frontmatter.ts	Custom minimal YAML frontmatter parser for skill-memory config. Handles inline flow-mapping form, block form, BOM tolerance, comment stripping. Does not depend on a full YAML library; malformed configs return null.
packages/plugin/src/hooks/magic-context/compartment-parser.ts	Adds skill_observations block parsing alongside existing fact/event/primer parsing. Uses a pipe-delimited '* skillId
packages/plugin/src/features/magic-context/skill-memory/provenance.ts	Parses 'Base directory for this skill:' line from skill output (supports both file:// URL and plain path). Derives tier/source from path prefix heuristics. Registry keyed sessionID:skillID with per-session cleanup in onSessionDeleted.
packages/plugin/src/hooks/magic-context/hook.ts	Wires up intentByCallId map, skillLoadRegistry, and the new before/after hooks. Session cleanup prunes both maps by prefix. Exposes skillLoadRegistry on the return value so ctx_skill_note shares the same instance.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Model as LLM Model
    participant Hook as tool.execute hooks
    participant Registry as SkillLoadRegistry
    participant DB as SQLite (skill_memory)
    participant Historian as Historian Runner

    Note over Model,DB: Transparent recall path (per skill load)
    Model->>Hook: "tool.execute.before (skill, intent=...)"
    Hook->>Registry: stashIntent(sessionID:callID, intent)
    Model->>Hook: tool.execute.after (skill output)
    Hook->>Hook: parseSkillProvenance(output) → resolvedPath
    Hook->>Hook: readFileSync(SKILL.md) → parseFrontmatterConfig
    Hook->>Registry: set(sessionID:skillID, provenance+config)
    Hook->>DB: recallSkillMemoryBlock(skillId, intent, tier, projectIdentity)
    DB-->>Hook: cosine/FTS/flat recall notes
    Hook->>Model: "output += skill-memory block"

    Note over Model,Hook: Agent write path
    Model->>Hook: ctx_skill_note(skill, kind, delta, intent)
    Hook->>Registry: getSkillLoad(sessionID, skill)
    Hook->>DB: findExistingNote / insertSkillMemoryNote

    Note over Historian,DB: Historian extraction path
    Historian->>Historian: parse skill_observations from chunk
    Historian->>DB: "promoteSkillObservations → tier=global, project_identity=*"
    Note over DB: Recall unions own partition + global '*' partition

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Model as LLM Model
    participant Hook as tool.execute hooks
    participant Registry as SkillLoadRegistry
    participant DB as SQLite (skill_memory)
    participant Historian as Historian Runner

    Note over Model,DB: Transparent recall path (per skill load)
    Model->>Hook: "tool.execute.before (skill, intent=...)"
    Hook->>Registry: stashIntent(sessionID:callID, intent)
    Model->>Hook: tool.execute.after (skill output)
    Hook->>Hook: parseSkillProvenance(output) → resolvedPath
    Hook->>Hook: readFileSync(SKILL.md) → parseFrontmatterConfig
    Hook->>Registry: set(sessionID:skillID, provenance+config)
    Hook->>DB: recallSkillMemoryBlock(skillId, intent, tier, projectIdentity)
    DB-->>Hook: cosine/FTS/flat recall notes
    Hook->>Model: "output += skill-memory block"

    Note over Model,Hook: Agent write path
    Model->>Hook: ctx_skill_note(skill, kind, delta, intent)
    Hook->>Registry: getSkillLoad(sessionID, skill)
    Hook->>DB: findExistingNote / insertSkillMemoryNote

    Note over Historian,DB: Historian extraction path
    Historian->>Historian: parse skill_observations from chunk
    Historian->>DB: "promoteSkillObservations → tier=global, project_identity=*"
    Note over DB: Recall unions own partition + global '*' partition

Loading

Comments Outside Diff (1)

1. packages/plugin/src/features/magic-context/skill-memory/recall.ts, line 481-484 (link)

   Bare catch {} makes recall errors invisible

   Any exception thrown inside recallSkillMemoryBlock — a SQLite error, embedding provider crash, or a coding mistake — is silently swallowed and returns an empty string. From the caller's perspective this is indistinguishable from "no notes exist" or "skill-memory disabled". At minimum a sessionLog or console.warn call here would let a developer diagnose why the recall block is absent without attaching a debugger.

_{Reviews (15): Last reviewed commit: "fix(test): stop global embedding mock.mo..." | Re-trigger Greptile}

[cubic-dev-ai]

11 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 15 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

4 issues found across 78 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/dreamer/task-executor.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/dreamer/task-executor.ts:419">
P2: Re-embed pre-step errors are suppressed, allowing successful task completion reporting despite a failed prerequisite data maintenance step.</violation>
</file>

<sub>Note: This PR contains a large number of files. cubic only reviews up to 40 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 2 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 4 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/features/magic-context/skill-memory/reembed.test.ts">

<violation number="1" location="packages/plugin/src/features/magic-context/skill-memory/reembed.test.ts:53">
P3: Redundant `_setTestProviderFactoryForProject(null)` call before `installReembedTestProvider()` -- the `installReembedTestProvider()` lambda immediately overwrites the factory anyway. The `_resetProjectEmbeddingRegistryForTests()` already clears the factory to null, so this extra call is a no-op. Looks like a fragment from an earlier refactor; consider dropping the middle call to keep the setup intent clear.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P3: Redundant _setTestProviderFactoryForProject(null) call before installReembedTestProvider() -- the installReembedTestProvider() lambda immediately overwrites the factory anyway. The _resetProjectEmbeddingRegistryForTests() already clears the factory to null, so this extra call is a no-op. Looks like a fragment from an earlier refactor; consider dropping the middle call to keep the setup intent clear.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/plugin/src/features/magic-context/skill-memory/reembed.test.ts, line 53:

<comment>Redundant `_setTestProviderFactoryForProject(null)` call before `installReembedTestProvider()` -- the `installReembedTestProvider()` lambda immediately overwrites the factory anyway. The `_resetProjectEmbeddingRegistryForTests()` already clears the factory to null, so this extra call is a no-op. Looks like a fragment from an earlier refactor; consider dropping the middle call to keep the setup intent clear.</comment>

<file context>
@@ -1,71 +1,116 @@
+describe("reembed", () => {
+    test("reembedStaleSkillNotes fills NULL embeddings (bounded, idempotent)", async () => {
+        _resetProjectEmbeddingRegistryForTests();
+        _setTestProviderFactoryForProject(null);
+        installReembedTestProvider();
+
</file context>