docs: Field Trial Log + Decision KPIs + MEMORY update

- Created FIELD_TRIAL_LOG.md — observation protocol for real customer cases - Log entry template with 10 fields - Two example entries (accepted and rejected decisions) - Questions the log answers: adoption, accuracy, calibration, rejection analysis - Future KPIs: Decision Adoption Rate, Decision Accuracy, Time to Decision - Updated MEMORY.md with Control Intelligence & Decision Model section - Decision Pipeline v1.0 summary - Decision Object v1.0 (STRUCTURE FROZEN) - Status: VALIDATED FOR FIELD TRIALS - Three target customer cases - Future KPIs - Key principle: No more modeling until first real customer case - Added Decision Adoption Rate and Decision Accuracy as future product KPIs - Not implemented yet — start collecting data now, calculate later - Decision Adoption = accepted / total recommendations - Decision Accuracy = correct / total decisions Rationale: Stop modeling, start observing. First real customer case will teach more than the last twenty documents combined.
2026-07-02 12:19:10 +00:00
parent c8819109db
commit 1c4158c373
2 changed files with 159 additions and 0 deletions
@@ -0,0 +1,113 @@
+# FIELD TRIAL LOG
+
+**Observation Protocol for Decision Model Validation**
+
+| | |
+|---|---|
+| **Version** | 1.0 |
+| **Status** | ACTIVE |
+| **Purpose** | Collect raw data from real customer cases |
+
+---
+
+## Log Entry Template
+
+| Field | Example | Notes |
+|-------|---------|-------|
+| **Date** | 2026-07-15 | |
+| **Customer Type** | Municipality / Property Owner / Contractor | |
+| **Decision** | Inspect Road A12 | What the system recommended |
+| **Decision Accepted** | Yes / No | Did the user accept? |
+| **Action Executed** | Yes / No / Pending | Was the action carried out? |
+| **Outcome** | Crack confirmed | What actually happened |
+| **Time to Decision** | 3 minutes | How long from login to decision |
+| **User Feedback** | "We trusted the recommendation." | Verbatim or summary |
+| **Confidence** | 87% | System confidence at decision time |
+| **Notes** | Free text | Anything else observed |
+
+---
+
+## Example Entries
+
+### Entry 1
+
+| Field | Value |
+|-------|-------|
+| Date | 2026-07-15 |
+| Customer Type | Municipality |
+| Decision | Inspect Road A12 |
+| Decision Accepted | Yes |
+| Action Executed | Yes |
+| Outcome | Crack confirmed, repaired within 14 days |
+| Time to Decision | 3 minutes |
+| User Feedback | "We trusted the recommendation." |
+| Confidence | 87% |
+| Notes | User clicked through Explainability chain |
+
+### Entry 2
+
+| Field | Value |
+|-------|-------|
+| Date | 2026-07-16 |
+| Customer Type | Property Owner |
+| Decision | Repair facade immediately |
+| Decision Accepted | No |
+| Action Executed | No |
+| Outcome | Deferred, user chose to wait |
+| Time to Decision | 5 minutes |
+| User Feedback | "Too expensive right now." |
+| Confidence | 92% |
+| Notes | Confidence was high but user rejected due to cost |
+
+---
+
+## Questions This Log Answers
+
+After collecting entries:
+
+- Were recommendations accepted? (Decision Adoption Rate)
+- Were actions executed? (Execution Rate)
+- Did outcomes match predictions? (Decision Accuracy)
+- Was confidence reasonable? (Calibration)
+- Which recommendations were ignored? (Rejection Analysis)
+- How long did decisions take? (Time to Decision)
+
+---
+
+## Future KPIs
+
+### Decision Adoption Rate
+
+```
+Recommendations accepted
+────────────────────────
+Total recommendations
+```
+
+Example: 72 of 100 = **72%**
+
+### Decision Accuracy
+
+```
+Decisions that proved correct
+─────────────────────────────
+Total decisions
+```
+
+Example: 65 of 72 executed = **90%**
+
+**Note:** These require months of data. Start collecting now, calculate later.
+
+---
+
+## Status
+
+**ACTIVE — Awaiting first customer case**
+
+---
+
+## ändringshistoria
+
+| Version | Datum | Beskrivning |
+|---------|-------|-------------|
+| 1.0 | 2026-07-02 | Initial field trial log template |