Crawler

Crawler is Running

power_settings_newGlobal Controls

Crawler EnabledCrawl New ContentDuplicate Detection (Sync)

Blocks duplicate pages during crawler-sync using BigQuery vector search (text ≥ 0.95, multimodal ≥ 0.92). Embeddings are always generated regardless of this setting.

Location Estimation (Sync)

Estimates lat/lng for pages without coordinates during crawler-sync. Uses tiered approach: JSON-LD → structured address → city centroid → LLM (OpenRouter).

speedScaling

Extractor Concurrency

OpenRouter Timeout (ms)

routeTraffic Split

Enable Traffic Split

scienceDirect Sampling

Enable Direct Sampling

Sampling Probability (2%)

Confidence Threshold (90%)

psychologyLLM Model Overrides

No model overrides configured — using defaults.

flagBlocking Flags (13)

photo_catalogphoto_logophoto_watermarklisting_soldlisting_expiredlisting_deletedlisting_unavailablecontent_no_pricecontent_no_titlecontent_spamcontent_error_pagenot_itemduplicate_exact