Lost Demand Intelligence:
Seeing what analytics miss

I.

The Gold Signal: Reward what matters, penalize what doesn't

The core insight

Most search analytics reward clicks. Problem: a click after a perfect match (query = product) is low-value — the user already knew what they wanted. The system learned nothing.

The highest-value signal is the inverse: clicked_despite_no_match. User searched for "iPhone 14 Pro case leather", I didn't have it, but they clicked on "iPhone 14 silicone case" anyway. This tells me:

• They have genuine purchase intent (not browsing)
• They're flexible on specification (leather → silicone)
• I just learned a new product association

Platinum Signal: when discovery converts to purchase

Gold Signal captures intent (user clicked the alternative). When that click leads to a purchase, the record is upgraded to Platinum Signal — the strongest training label in the system.

The reward formula

LDI uses a weighted ensemble of behavioral signals with anti-bait penalties. Three signal categories drive the score:

• Match quality — high-confidence matches earn baseline reward; NO_MATCH with downstream click earns the Gold/Platinum premium
• Behavioral commitment — cart additions, purchases, and fast clicks compound the score upward
• Frustration penalties — multiple query refinements, long sessions without action, and bounces drag the score toward -1.0

Exact weights are tuned hyperparameters refined against the test suite. All scores normalize to [-1.0, +1.0] — directly consumable by reward-modeled fine-tuning pipelines.

Bounce detection runs as a batch sweep at JSONL export time, not on the hot path. Records older than 5 minutes with no click, cart-add, or purchase are flagged bounce: true in a single UPDATE. This keeps query classification free of background bookkeeping, at the cost of bounce labels being eventually consistent (not real-time).

II.

Semantic Validation: Filtering noise before it reaches the model

The problem with raw query logs

E-commerce search logs are 60-70% noise: keyboard mashing (asdfgh), incomplete prefixes (ip), food queries on an auto parts store (pizza), and hallucinated products (iPhone 30).

If you train on this data, you train on garbage. LDI implements a SemanticValidator that rejects low-quality queries before they enter the reward pipeline.

The 5-layer rejection pipeline

Every query passes through five sequential filters before it touches the reward engine:

1. Format checks — minimum length, keyboard mashing patterns, nonsense word lists
2. Domain heuristics — wrong-domain detection (e.g. food queries on an auto parts store)
3. Realism check — model number sanity catches hallucinations like iPhone 30 or Galaxy S99
4. Lost-demand qualification — NO_MATCH queries must carry both domain context (recognized brand or category) and at least one extractable feature; otherwise they are uninformative noise
5. Feature extraction — a per-domain attribute extractor (color, capacity, chassis code, fuel type, body type, etc.) populates missing_features for downstream training labels

Records that fail any layer are persisted with ai_ready: false and excluded from training exports by default.

Why NO_MATCH with features is valid

If a user searches "BMW E46 klocki Brembo" and I get NO_MATCH, that's not noise — that's genuine lost demand. The query has domain context (car brand + part term) and the feature extractor pulls out gen:e46. These records are flagged VALID_LOST_DEMAND and routed to the Gold/Platinum Signal pipeline.

III.

Universal Automotive Knowledge: ~980 entities, zero hallucinations

The domain knowledge layer

LDI was built first for automotive e-commerce. The UNIVERSAL_AUTOMOTIVE_KNOWLEDGE dictionary ships with:

• 185 brands — 107 car brands (from BMW to BYD, including Chinese EV), 54 motorcycle, 24 luxury
• 512 models — 370 car models (E46, W124, A4 B8, ...) plus 142 motorcycle models
• 190 part categories — klocki, amortyzator, filtr oleju, świece
• 84 technical terms — chassis codes, fuel types, body styles
• 9 product-code regex patterns — VW/Audi OEM, NGK plug codes, Bosch part numbers

Total: ~980 domain entities across 8 categories, all hot-loaded into memory at boot. The electronics adaptation swaps this dictionary for an equivalent UNIVERSAL_ELECTRONICS_KNOWLEDGE structure — same shape, different content.

Why this matters

When a user types "Brembo klocki E46", the system knows:

• Brembo = known OEM supplier (brake specialist)
• klocki = brake pads (part category)
• E46 = BMW 3-series 1998-2006 (car model)

This context enables fuzzy matching: even if I don't have "Brembo brake pads for E46" in the product database, I can suggest "ATE brake pads for E46" — and if the user clicks, that's a Gold Signal.

IV.

Real-time classification pipeline

Architecture flow

User query ("BMW E46 klocki Brembo")
    ↓
[1] source dispatcher (moto vs elektro)
    ↓
[2] per-domain MissingFeatureExtractor
       moto   → MotoFeatureExtractor    (chassis, engine, fuel, body, color)
       elektro → ElektroFeatureExtractor (capacity, ram, screen, variant, gen)
    ↓
    extracts: ["gen:e46"]
    ↓
[3] SemanticValidator (5-layer pipeline)
    rejects: keyboard | wrong-domain | unrealistic | NO_MATCH-without-features
    ↓
[4] Fuzzy matcher against in-memory product catalog → confidence_level
    ↓
    ┌──────────────────────────────┬──────────────────────────────┐
    │ HIGH / MEDIUM                │ NO_MATCH                     │
    │ standard product results     │ alternative suggestions      │
    └──────────────────────────────┴───────────────┬──────────────┘
                                                   │
                                                   ↓
                                       User clicks "Add to cart"?
                                                   │
                                       ┌───────────┴───────────┐
                                      YES                     NO
                                       │                       │
                                       ↓                       ↓
                              GOLD SIGNAL (0.8)        Lost-demand log
                                       │                (inventory team)
                            purchase webhook fires?
                                       │
                                       ↓ YES
                              PLATINUM SIGNAL (1.0)

Hot-path design

Classification runs on the keystroke debounce — there are no external API calls in the request path. Three things make this work:

• Product catalog and domain knowledge are hot-loaded into memory at boot
• Validator patterns are pre-compiled re.Pattern objects, not strings
• Fuzzy matching uses score_cutoff to short-circuit low-quality matches early
• All bookkeeping (bounce labelling, session consolidation) is deferred to batch sweeps at JSONL export time, not the hot path

V.

Training data export: JSONL for fine-tuning

AI-ready data pipeline

Every validated session is logged to JSONL with a normalized reward score [-1.0, +1.0]. This format is directly consumable by reinforcement learning pipelines.

{
  "query": "BMW E46 klocki Brembo",
  "intent_label": "hamulce",
  "confidence": "NO_MATCH",
  "source": "moto",
  "reward_signal": {
    "score": 0.8,
    "clicked_alternative": true,
    "purchased": false,
    "bounce": false
  },
  "missing_features": ["gen:e46"],
  "matched_product_id": null,
  "alternative_clicked_id": "ate-klocki-e46-przod",
  "ai_ready": true,
  "query_refinement_count": 0,
  "timestamp": "2026-06-17 10:30:00"
}

Data quality guarantees

VI.

Privacy by design: GDPR-compliant from the first byte

LDI is deployed in the EU. Privacy is enforced at the data layer, not bolted on later.

What is scrubbed before logging

• PII patterns — emails, phone numbers, and identifiable fragments are stripped from queries before any DB write (scrub_pii runs at the edge of the request handler, not after the fact)
• IP addresses — stored as salted hashes plus a masked octet representation, never the raw IP
• Session IDs — used internally for query consolidation but never exported in JSONL training data
• Organization context — derived from ASN (Autonomous System Number) lookups, not from cookies, fingerprinting, or third-party trackers

What the JSONL training export contains — and what it doesn't

The export ships query text, classification metadata, behavioral flags, and the reward score. It does not ship session IDs, IP hashes, organization names, or any user identifier. Two records from the same session are linked only by the surrounding behavioral signal, never by an identifier.

Right to erasure

All visitor data is keyed by salted session hash in a single relational store. Deletion requests resolve to a single DB query, not a multi-system reconciliation. No analytics warehouse, no third-party CDN cookies, no off-platform reporting.

VII.

Test results & cross-domain generalization

Cross-domain generalization

The same architecture was retargeted from automotive to consumer electronics (smartphones, laptops, audio) without retraining the underlying model or touching its hyperparameters. What did change was the domain knowledge layer — brands, categories, and the attribute extractor — which had to be rebuilt for the new vertical. This is not a zero-effort transfer: the validator, reward engine, and session consolidation are domain-agnostic by design, but each new vertical still needs its own domain knowledge layer built out.

Automotive test methodology

The test suite covers edge cases that break naive matchers:

• Typos in brand names (merecedes → Mercedes)
• Polish diacritics (klocki hamulcowe vs klocki hamulcowe)
• Mixed language queries (BMW oil filter 5W30)
• Hallucinated products (iPhone 30, Galaxy S99)
• Food domain leakage (pizza on auto parts store)
• OEM code formats (1J0 615 301 = VW brake disc)

Results breakdown

VIII.

Where LDI fits next to standard analytics

What standard analytics already do

GA4, Mixpanel, and most internal warehouses can log search terms, no-result events, and custom "lost demand" dimensions. The question is what comes out the other end — and what shape the data is in by the time an ML engineer or merchandiser sees it.

Where LDI is structurally different

• Classification at query time, not at analysis time. LDI labels every query as noise or valid lost demand before it lands in storage. General analytics log everything and push the filtering problem downstream to the analyst
• Reward-shaped output. Every record is a self-contained training example with a normalized reward score in [-1, +1]. General analytics ship events; LDI ships labels
• Domain-aware extractors. LDI knows E46 is a chassis code, 2.0 TDI is an engine spec, and iPhone 13 pro max decomposes into model + variant. General analytics see opaque strings
• Session consolidation. Three refinements of one intent collapse into a single labeled record with a refinement count. Standard event streams keep them as three separate rows the analyst has to stitch back together

None of this replaces GA4 — they solve different problems. LDI is a specialized layer that produces something general analytics cannot: training-ready labels for the queries that didn't convert.

Epilogue: Why this matters for e-commerce

Every e-commerce store has a "lost demand" problem. Users search for products that don't exist in the catalog. The search returns "no results" or irrelevant suggestions. The user leaves.

Standard analytics cannot capture this. They see a bounce. They don't see the intent.

LDI captures the intent. It logs what users wanted, not just what they bought. It identifies which "no results" queries are genuine demand signals (Gold Signal) vs. noise (keyboard mashing, wrong domain).

This data is the highest-value signal for inventory decisions and AI training. It tells you what to stock. It tells you what associations to learn. It's the signal that standard analytics miss.