From 1c4158c3737719ccc952047717df4de7bcbb6064 Mon Sep 17 00:00:00 2001 From: Bernt Date: Thu, 2 Jul 2026 12:19:10 +0000 Subject: [PATCH] docs: Field Trial Log + Decision KPIs + MEMORY update MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Created FIELD_TRIAL_LOG.md — observation protocol for real customer cases - Log entry template with 10 fields - Two example entries (accepted and rejected decisions) - Questions the log answers: adoption, accuracy, calibration, rejection analysis - Future KPIs: Decision Adoption Rate, Decision Accuracy, Time to Decision - Updated MEMORY.md with Control Intelligence & Decision Model section - Decision Pipeline v1.0 summary - Decision Object v1.0 (STRUCTURE FROZEN) - Status: VALIDATED FOR FIELD TRIALS - Three target customer cases - Future KPIs - Key principle: No more modeling until first real customer case - Added Decision Adoption Rate and Decision Accuracy as future product KPIs - Not implemented yet — start collecting data now, calculate later - Decision Adoption = accepted / total recommendations - Decision Accuracy = correct / total decisions Rationale: Stop modeling, start observing. First real customer case will teach more than the last twenty documents combined. --- MEMORY.md | 46 ++++++++++++++ docs/design/FIELD_TRIAL_LOG.md | 113 +++++++++++++++++++++++++++++++++ 2 files changed, 159 insertions(+) create mode 100644 docs/design/FIELD_TRIAL_LOG.md diff --git a/MEMORY.md b/MEMORY.md index 7f3fda9c6..5cd376c39 100644 --- a/MEMORY.md +++ b/MEMORY.md @@ -881,3 +881,49 @@ Mätvärden: - Leverabler: Engineering Readiness Score (viktad), Policy Enforcement, STOP Register (med risk), Release Gate, Meta-Rules - Exit Criteria: Varje STOP är eliminerad, har dispens, eller ersatts av kontrollerad process - EOS-regel: "EOS får aldrig blockera något utan att kunna förklara exakt varför och vilken minsta åtgärd som krävs för att häva blockeringen." + +--- + +## 11. CONTROL INTELLIGENCE & DECISION MODEL (2026-07-02) + +**LandveX produces Control Intelligence, not AI analysis.** + +AI is implementation. Control Intelligence is the product. + +### Decision Pipeline v1.0 +``` +Reality → Observation → Evidence → Finding → Decision → Action → Business Impact → Learning +``` + +**Seven steps with input/transformation/output/owner.** + +### Decision Object v1.0 (STRUCTURE FROZEN) +- 7 fields: Decision, Why, Evidence, Confidence, Consequence, Action, Business Impact +- Changes require v1.1 + migration note + revalidation +- Implementation/algorithms NOT frozen + +### Status: VALIDATED FOR FIELD TRIALS +- 8 scenarios reviewed (6 PASS, 2 OBSERVATION, 0 FAIL) +- Decision Invariance Test: PASS +- Evidence Variation Test: PASS +- Explainability Invariance Test: PASS +- Ready for real customer testing, not production truth + +### Three Target Customer Cases +1. Municipality — "Inspect or wait?" (Maintenance prioritization) +2. Property Owner — "Repair now or plan later?" (Cost vs risk) +3. Contractor/Operations — "Which action first?" (Operational planning) + +### Future KPIs (collect now, calculate later) +- **Decision Adoption Rate** = Accepted recommendations / Total recommendations +- **Decision Accuracy** = Correct decisions / Total decisions +- **Time to Decision** = Minutes from login to decision + +### Key Principle +**No more modeling until first real customer case has run through entire pipeline.** + +### Files +- `docs/design/DECISION_MODEL_v1.0.md` — Decision Model +- `docs/design/DECISION_PIPELINE_v1.0.md` — Decision Pipeline +- `docs/design/DECISION_MODEL_REVIEW.md` — Manual Review Results +- `docs/design/FIELD_TRIAL_LOG.md` — Observation Protocol diff --git a/docs/design/FIELD_TRIAL_LOG.md b/docs/design/FIELD_TRIAL_LOG.md new file mode 100644 index 000000000..c259b6243 --- /dev/null +++ b/docs/design/FIELD_TRIAL_LOG.md @@ -0,0 +1,113 @@ +# FIELD TRIAL LOG + +**Observation Protocol for Decision Model Validation** + +| | | +|---|---| +| **Version** | 1.0 | +| **Status** | ACTIVE | +| **Purpose** | Collect raw data from real customer cases | + +--- + +## Log Entry Template + +| Field | Example | Notes | +|-------|---------|-------| +| **Date** | 2026-07-15 | | +| **Customer Type** | Municipality / Property Owner / Contractor | | +| **Decision** | Inspect Road A12 | What the system recommended | +| **Decision Accepted** | Yes / No | Did the user accept? | +| **Action Executed** | Yes / No / Pending | Was the action carried out? | +| **Outcome** | Crack confirmed | What actually happened | +| **Time to Decision** | 3 minutes | How long from login to decision | +| **User Feedback** | "We trusted the recommendation." | Verbatim or summary | +| **Confidence** | 87% | System confidence at decision time | +| **Notes** | Free text | Anything else observed | + +--- + +## Example Entries + +### Entry 1 + +| Field | Value | +|-------|-------| +| Date | 2026-07-15 | +| Customer Type | Municipality | +| Decision | Inspect Road A12 | +| Decision Accepted | Yes | +| Action Executed | Yes | +| Outcome | Crack confirmed, repaired within 14 days | +| Time to Decision | 3 minutes | +| User Feedback | "We trusted the recommendation." | +| Confidence | 87% | +| Notes | User clicked through Explainability chain | + +### Entry 2 + +| Field | Value | +|-------|-------| +| Date | 2026-07-16 | +| Customer Type | Property Owner | +| Decision | Repair facade immediately | +| Decision Accepted | No | +| Action Executed | No | +| Outcome | Deferred, user chose to wait | +| Time to Decision | 5 minutes | +| User Feedback | "Too expensive right now." | +| Confidence | 92% | +| Notes | Confidence was high but user rejected due to cost | + +--- + +## Questions This Log Answers + +After collecting entries: + +- Were recommendations accepted? (Decision Adoption Rate) +- Were actions executed? (Execution Rate) +- Did outcomes match predictions? (Decision Accuracy) +- Was confidence reasonable? (Calibration) +- Which recommendations were ignored? (Rejection Analysis) +- How long did decisions take? (Time to Decision) + +--- + +## Future KPIs + +### Decision Adoption Rate + +``` +Recommendations accepted +──────────────────────── +Total recommendations +``` + +Example: 72 of 100 = **72%** + +### Decision Accuracy + +``` +Decisions that proved correct +───────────────────────────── +Total decisions +``` + +Example: 65 of 72 executed = **90%** + +**Note:** These require months of data. Start collecting now, calculate later. + +--- + +## Status + +**ACTIVE — Awaiting first customer case** + +--- + +## ändringshistoria + +| Version | Datum | Beskrivning | +|---------|-------|-------------| +| 1.0 | 2026-07-02 | Initial field trial log template |