Files

T

Bernt 1c4158c373 docs: Field Trial Log + Decision KPIs + MEMORY update

- Created FIELD_TRIAL_LOG.md — observation protocol for real customer cases
  - Log entry template with 10 fields
  - Two example entries (accepted and rejected decisions)
  - Questions the log answers: adoption, accuracy, calibration, rejection analysis
  - Future KPIs: Decision Adoption Rate, Decision Accuracy, Time to Decision

- Updated MEMORY.md with Control Intelligence & Decision Model section
  - Decision Pipeline v1.0 summary
  - Decision Object v1.0 (STRUCTURE FROZEN)
  - Status: VALIDATED FOR FIELD TRIALS
  - Three target customer cases
  - Future KPIs
  - Key principle: No more modeling until first real customer case

- Added Decision Adoption Rate and Decision Accuracy as future product KPIs
  - Not implemented yet — start collecting data now, calculate later
  - Decision Adoption = accepted / total recommendations
  - Decision Accuracy = correct / total decisions

Rationale: Stop modeling, start observing. First real customer case
will teach more than the last twenty documents combined.

2026-07-02 12:19:10 +00:00

2.7 KiB

Raw Blame History

FIELD TRIAL LOG

Observation Protocol for Decision Model Validation


Version	1.0
Status	ACTIVE
Purpose	Collect raw data from real customer cases

Log Entry Template

Field	Example	Notes
Date	2026-07-15
Customer Type	Municipality / Property Owner / Contractor
Decision	Inspect Road A12	What the system recommended
Decision Accepted	Yes / No	Did the user accept?
Action Executed	Yes / No / Pending	Was the action carried out?
Outcome	Crack confirmed	What actually happened
Time to Decision	3 minutes	How long from login to decision
User Feedback	"We trusted the recommendation."	Verbatim or summary
Confidence	87%	System confidence at decision time
Notes	Free text	Anything else observed

Example Entries

Entry 1

Field	Value
Date	2026-07-15
Customer Type	Municipality
Decision	Inspect Road A12
Decision Accepted	Yes
Action Executed	Yes
Outcome	Crack confirmed, repaired within 14 days
Time to Decision	3 minutes
User Feedback	"We trusted the recommendation."
Confidence	87%
Notes	User clicked through Explainability chain

Entry 2

Field	Value
Date	2026-07-16
Customer Type	Property Owner
Decision	Repair facade immediately
Decision Accepted	No
Action Executed	No
Outcome	Deferred, user chose to wait
Time to Decision	5 minutes
User Feedback	"Too expensive right now."
Confidence	92%
Notes	Confidence was high but user rejected due to cost

Questions This Log Answers

After collecting entries:

Were recommendations accepted? (Decision Adoption Rate)
Were actions executed? (Execution Rate)
Did outcomes match predictions? (Decision Accuracy)
Was confidence reasonable? (Calibration)
Which recommendations were ignored? (Rejection Analysis)
How long did decisions take? (Time to Decision)

Future KPIs

Decision Adoption Rate

Recommendations accepted
────────────────────────
Total recommendations

Example: 72 of 100 = 72%

Decision Accuracy

Decisions that proved correct
─────────────────────────────
Total decisions

Example: 65 of 72 executed = 90%

Note: These require months of data. Start collecting now, calculate later.

Status

ACTIVE — Awaiting first customer case

ändringshistoria

Version	Datum	Beskrivning
1.0	2026-07-02	Initial field trial log template

2.7 KiB Raw Blame History