電商重排評估:具新鮮度感知的檢索架構
Marketplace Reranking Evaluation: Freshness-Aware Retrieval Architecture
文章提出一套具時間感知的電商搜尋檢索與重排評估方法,以帶時間戳的搜尋意圖和商品目錄快照建立可重播測試集。評估分開計算候選召回率與前 10 項過期率,並用時間留出測試和分層數據集檢查資料洩漏、商品更新及重複候選等問題。方法適用前提是有可靠的生效時間戳並能還原商品目錄狀態;否則應把測試結果標示為現況基準。
Marketplace search has a quiet failure mode: a highly relevant item can still be the wrong result when its price, stock, or delivery promise is stale. My decision rule is to evaluate retrieval and reranking against time-aware evaluation sets, then choose chunk boundaries that preserve the attributes a shopper actually filters on. A larger index or a clever prompt cannot repair labels that ignore catalog time.
Short answer: build evaluation sets from timestamped shopper intents and catalog snapshots, score recall before reranking, and score freshness-sensitive ranking separately so a model cannot hide stale listings behind a single average metric.
Start With the Retrieval Contract
For a marketplace, the retrieval contract is more useful than a diagram of services. Given a query, a catalog snapshot, and a cut-off time, the system must return products that were eligible at that time. The contract names the inputs, the allowed evidence, and the output fields. It also makes an offline test reproducible: the same query and snapshot should produce the same candidate IDs.
I keep the first stage deliberately boring. A lexical retriever catches exact attributes such as “linen 160x200 duvet,” while a vector retriever catches paraphrases such as “cool summer bedding.” Their union becomes the candidate set. The reranker sees product text plus structured fields, not a page assembled from whatever happens to be current at request time.
This is where chunking matters. Split long descriptions at semantic boundaries, but copy stable identifiers and the effective timestamp into every chunk. A chunk that says “compatible with model K” without its product ID is hard to trace; a chunk that contains a review written after the evaluation cut-off leaks future information. The RAG paper describes retrieval as a way to ground generation in external passages; the same discipline applies when passages ground a product ranking decision.
How Should Retrieval Architecture and Evaluation Sets Handle Freshness?
Use two clocks. The event clock records when a shopper searched or clicked. The catalog clock records when a title, price, inventory field, or policy became effective. For each test case, freeze both. A query from Monday must not receive a product description published on Tuesday just because the current index is easier to load.
I store cases as JSON Lines so they can be reviewed in a pull request and replayed from Python:
from dataclasses import dataclass
from datetime import datetime
from typing import Sequence
@dataclass(frozen=True)
class EvalCase:
query: str
snapshot_at: datetime
relevant_ids: frozenset[str]
freshness_limit_hours: int
def recall_at_k(candidates: Sequence[str], relevant: frozenset[str], k: int) -> float:
if not relevant:
return 0.0
return len(set(candidates[:k]) & relevant) / len(relevant)
def freshness_violation(product_updated_at: datetime,
snapshot_at: datetime,
limit_hours: int) -> bool:
age_hours = (snapshot_at - product_updated_at).total_seconds() / 3600
return age_hours > limit_hours
def evaluate_case(case: EvalCase,
retrieved: Sequence[str],
updated_at_by_id: dict[str, datetime]) -> dict[str, float]:
stale = sum(
freshness_violation(updated_at_by_id[item_id], case.snapshot_at,
case.freshness_limit_hours)
for item_id in retrieved[:10]
if item_id in updated_at_by_id
)
return {
"recall@10": recall_at_k(retrieved, case.relevant_ids, 10),
"stale_top10_rate": stale / max(1, min(10, len(retrieved))),
}
The code intentionally returns two numbers. Recall answers “did retrieval find the right products?” The stale-rate answers “did ranking surface products that violate the time contract?” A single NDCG or click-through score can conceal the second failure, especially when old products have stronger historical engagement.
For labels, separate exact relevance from acceptable substitutes. “Black running shoes, size 42” should identify an in-stock product matching the size and color constraints, while a visually similar size 41 is a near miss, not a success. Record the label source: human judgment, add-to-cart behavior, purchase, or a rule derived from structured attributes. Keep those sources as separate slices so a noisy behavioral label does not overrule a hard inventory constraint.
The Dataset Shape That Finds Real Failures
Randomly sampling popular queries gives a comforting number and a weak test. I stratify by the things that change the ranking contract: head versus tail queries, attribute-heavy queries, new products, edits to existing products, and periods with stock volatility. Each slice gets a minimum case count before it contributes to a release decision.
Temporal holdouts are the important part. Train or tune on an earlier catalog window, then evaluate on a later window whose products and edits were not visible during tuning. This exposes stale embeddings, delayed indexing, and chunks that accidentally retain a previous price. It also mirrors how a marketplace changes in production.
A small manifest is enough to make the split auditable:
from pathlib import Path
import json
def write_manifest(path: Path, cases: list[EvalCase]) -> None:
rows = []
for case in cases:
rows.append({
"query": case.query,
"snapshot_at": case.snapshot_at.isoformat(),
"relevant_ids": sorted(case.relevant_ids),
"freshness_limit_hours": case.freshness_limit_hours,
})
path.write_text("\n".join(json.dumps(row) for row in rows) + "\n")
The manifest should travel with the index version and the chunking configuration. When a score moves, you can ask whether the corpus changed, labels changed, or the retrieval code changed. Without those three identifiers, an evaluation result is a screenshot rather than a test.
Failure Modes Worth a Dedicated Slice
The first failure is temporal leakage: the evaluator resolves product text from “now” instead of the case snapshot. The second is field loss during chunking, where a price or variant attribute is left in a neighboring chunk and becomes invisible to the retriever. The third is duplicate amplification: ten chunks from one product fill the candidate list and reduce product-level recall.
I also watch for query drift. A phrase such as “gift under 50” changes meaning when currency, season, or shipping region changes. Store locale and currency with the case, and make the reranker consume normalized structured values. Free-form text alone is a poor place to enforce a numeric ceiling.
Three words: measure the slices.
For every release, compare candidate recall, top-k stale rate, duplicate rate, and latency by slice. Set a guardrail for each metric before looking at aggregate improvements. If a reranker raises average relevance while stale-rate rises for newly edited products, the release is a regression even when the headline score goes up.
Operational Trade-offs and the Exit Rule
A time-aware corpus costs storage and test maintenance. Keeping every snapshot is unnecessary; retaining the snapshots that explain catalog edits and seasonal changes usually gives better signal per byte. Larger chunks preserve context but make lexical matching less precise. Smaller chunks improve matching but increase duplicate handling and join work. I prefer a modest chunk size with explicit product-level aggregation, then verify that choice on the temporal slices rather than by intuition.
This approach is not suitable when the business has no reliable effective timestamps or cannot reconstruct catalog state. In that case, start with a current-state benchmark and label the result as such; do not pretend it measures historical freshness. Stick with a simpler lexical baseline when the catalog is tiny, attributes are authoritative, and latency matters more than semantic recall. The right architecture is the one whose evaluation contract the team can actually replay.
In Python, I keep the evaluation harness next to the indexing code and run it in CI with a fixed random seed. The harness emits per-query JSON, not just a dashboard average, so a reviewer can inspect why a product moved. Your mileage may vary on the exact freshness window; the unresolved question is a business policy, and the test should make that policy explicit instead of guessing.
References
來源:DEV Community · dev.to