Vector product search

vector-product-search

Add vector retrieval beside lexical product search, with one embedding per purchasable unit and exact SKU lookup kept first. Use when adding embedding search, hybrid ranking, or re-embedding after catalog edits.

Version 0.3.2  ·  MIT  ·  Source  ·  Rule

Install this skill

Copy the skill and its rule into your project:

npx @sadrmanesh/commerce-agent-skills add vector-product-search

The skill goes to .agents/skills/ and the rule to .agents/rules/. See Command line for other directories.

Use this skill when lexical search is not enough for paraphrased queries. Facets, stock, and lexical ranking stay in the marketplace search skill. This skill adds a vector index. It does not replace SKU and GTIN lookup.

Do not claim a recall or conversion number. Choose a model, record its id and dimension, and measure queries you have labeled.

Procedure

1. Embed one text per purchasable unit. A variant with its own title or attributes gets its own vector. Copy the parent name into that text.
2. Use the same model, dimension, and similarity metric for indexing and querying. Record model_id and a hash of the embedded text on the document.
3. Keep price, stock, and facet ids as filter fields. Do not rely on the embedding to enforce "blue" or "in stock".
4. Resolve an exact normalized SKU or GTIN before any vector query. If it hits one unit, return that unit first.
5. Combine lexical and vector scores with a formula you write down. Re-embed when the text hash changes, not when only the price changes.

Normalization and hybrid scoring are in the index notes. Worked examples and checks are in evals/tasks.json.

What is embedded

Include the fields a shopper might paraphrase: title, brand, category names, and attribute labels. Run the same text normalization as lexical search, including Persian Yeh, Kaf, digits, and ZWNJ removal, before you embed.

Exclude price, stock counts, and SKUs from the embedded text. Price and stock change without the words changing, and a SKU is handled by exact match. If you later include them, the content hash must change when they change.

Query

1. Normalize the query.
2. Exact identifier match wins.
3. Otherwise run the lexical query and the vector query with the same filters (facet ids, stock policy, category).
4. Fuse the two ranked lists. A default you may start from is a weighted sum after each list is normalized to the same scale. Change the weight only when labeled queries say so.
5. Page with a keyset on the fused score plus unit_id.

Apply filters in both retrievals. A vector hit that ignores the selected color is a bug.

Re-embedding

• Store model_id, dimension, metric (cosine unless the index documents another), and text_hash.
• A catalog write recomputes the hash. If the hash or the model id differs, enqueue that unit.
• Switching models rebuilds the index. Do not query an index with a different model than the one that wrote it.
• Delete the vector when the unit is deleted. A parent rebuild replaces that parent's vectors as a set.

Gotchas

• Embedding only the parent returns one vector for every size and color.
• Cosine and dot product are not interchangeable unless the vectors are normalized and you meant that equivalence.
• A vector score must not outrank an exact SKU match.
• Indexing the raw Persian string and querying the normalized string misses. Normalize both.
• Do not log shopper queries next to identifiers you are not allowed to retain. The query text can contain an address or a name the shopper pasted.

Verification

• A SKU query returns that unit even when a nearer vector belongs to another product.
• A color filter removes other colors from the vector results, not only from the lexical results.
• Changing the price does not change text_hash. Changing the title does, and that unit is re-embedded.
• Deleting a variant removes its vector.

Evaluation

Worked examples and the checks they must satisfy are in evals/tasks.json.

References

Vector index beside lexical search

Document

Field	Role
unit_id	Purchasable unit. Primary key.
parent_id	Set for variants.
sku, gtin	Exact match. Not part of the embedded text.
text_hash	Hash of the normalized text that was embedded.
model_id	Embedding model identifier and its dimension.
embedding	Vector from that model only.
facet_ids, in_stock, price, currency	Filters and display. Not a reason to re-embed.

Embedded text

{title}
{brand}
{category names}
{attribute labels}

Normalize with the marketplace search normalizer before hashing and embedding. The hash is of the normalized text. If you change the normalizer, rebuild the index. Old hashes would not match the new text.

Fusion

Normalize each retrieval's scores onto [0, 1] within that result set, then:

fused = (lexical_weight * lexical) + (vector_weight * vector)

Start at lexical_weight = 0.6 and vector_weight = 0.4 if you have no labeled queries yet. This is a starting point, not a measured optimum. Units that appear in only one list use 0 for the missing score.

Exact SKU or GTIN matches skip fusion and are placed first.

Metric

Write down cosine similarity or the index's metric next to model_id. Query with that metric. Do not mix cosine in one environment and inner product in another unless both sides use normalized vectors and you have recorded that they match.

Evaluation

Fixtures from evals/tasks.json. See evaluation fixtures for how the checks work.

sku-before-vector

Return the product whose SKU matches the query before any nearest-vector result.

Must include: sku, vector

Must not include: embedding of the price

Worked example

Normalize the query and look up sku and gtin first.

Only if that lookup misses, run the vector query. The embedding text does not include the price.

reembed-on-title

Re-embed a variant when its title changes and skip re-embedding when only the price changes.

Must include: text_hash, model_id

Must not include: cosine and dot product

Worked example

Store model_id and text_hash next to the vector.

A title edit changes the hash and enqueues that unit. A price edit leaves the hash in place because price is a filter field.