Why every store needs its own adapter

Two store pages can show the same product and share almost nothing useful in their source. One hides data in a Next.js payload, another annotates HTML tiles, and another may return a browser gate instead of results. MLScraper handles that variety with provider adapters. This chapter shows what each adapter owns, the small product contract they share, and why isolating store knowledge makes failures easier to test and repair.

The rest of MLScraper wants a simple answer from every page. It needs a list of products and the URL of the next page, when one exists. It does not want Amazon selectors, Liverpool JSON paths, or Mercado Libre gate detection mixed into lifecycle code.

Each provider package performs that translation. It owns store route builders, documented option values, parsing assumptions, pagination details, and store-specific incomplete states. The shared motor owns the repeated algorithm around those details.

This is an adapter pattern in practical clothing. The provider side speaks a store’s language. The shared side speaks product observations. The narrow contract between them prevents store changes from spreading through the scheduler and data model.

Every provider motor implements scrape_page. The method receives fetched content and the current URL, then returns normalized dictionaries and an optional next URL.

The required fields are identifier, title, and price. The product URL is optional. next_url becomes None when pagination ends or when an incomplete condition stops the cycle.

Stable identifiers carry the most weight. The shared article model compares products by identifier, not title. A provider should choose the store’s durable listing key, such as an Amazon ASIN, instead of deriving identity from text that can change.

The provider does not write JSON, move products between lifecycle streams, calculate price-drop percentages, send Telegram messages, or schedule its next cycle. Those behaviors operate on the normalized contract and belong to shared or runtime code.

Incomplete behavior is the one deliberate side channel. A provider that recognizes a gate or malformed page can mark the motor incomplete before returning. That signal tells shared reconciliation that the product list is not trustworthy evidence of absence.

Amazon Mexico is the most conventional adapter in the project. Its parser finds div elements marked with data-component-type="s-search-result", reads the data-asin value, extracts a title and price, and builds a canonical product URL from the ASIN.

Cards without an ASIN, title, or price are skipped. A search page can include sponsored, incomplete, or decorative elements that resemble products, so the parser prefers omission over inventing a zero-value listing.

Pagination looks for the next-page link and joins relative paths against the Amazon Mexico domain. The provider returns that route to the shared motor, which handles session policy and page delays before fetching it.

Amazon route generation also belongs inside the package. Known seller and brand options map to documented refinement values. The builder controls query encoding and filter order, while the job factory only asks it to produce a route from structured options.

Mercado Libre delegates its parsing to a focused parser.py module because its result data can arrive through ordinary DOM cards or embedded Nordic state. The parser first checks for known blocked-page signals, then tries the supported result shapes.

The result object can carry items, a next URL, a blocked reason, or an incomplete reason. A recognized account verification or suspicious-traffic page marks the motor blocked and applies the provider cooldown. An unsupported or partial result shape marks the cycle incomplete without pretending the page contained zero products.

Pagination needs more than a next-link selector. Helpers read result counts, page size, current offset, and whether another page is available. They inject the next offset into several known URL shapes while preserving the route’s other pieces.

This provider demonstrates why the adapter boundary matters. Browser readiness, gate detection, DOM parsing, embedded state, and offset rules can change together inside one store. Shared lifecycle still receives the same identifier, title, price, URL, and completion signal.

Liverpool pages include a __NEXT_DATA__ script containing application state. The provider parses that JSON and reads product records from the expected query.data.mainContent.records path.

Each valid record exposes an internal ID, title, promotional price, and enough text to build a product URL. Pagination reads the reported page count and inserts the next page segment while preserving the existing query string.

Missing or empty __NEXT_DATA__ is not a valid empty catalogue. Invalid JSON and unexpected key structure also mark the cycle incomplete. Malformed individual records are skipped with a warning while valid siblings remain available.

Liverpool route construction is unusually involved. Generated searches and pages can require a seller-filtered token resolved from store metadata. Provider helpers validate that the response contains the expected Liverpool seller refinement and page ancestry. Unsupported combinations require an explicit URL instead of a guessed generated route.

That complexity stays under provider/liverpool/. The loader can resolve a documented page name, and the factory can request a route, without learning how Liverpool encrypts or organizes its navigation tokens.

Palacio de Hierro exposes Constructor-backed attributes on product tile elements. The adapter reads the identifier, product name, price, link, total result count, and configured page size from those attributes and surrounding section metadata.

If no valid product tiles appear, the provider marks the cycle incomplete. That choice is conservative because an empty selector result could mean a changed template. A complete zero-result catalogue would need a separately reliable signal before reconciliation should treat every saved product as missing.

Pagination advances the start query parameter by page size until the next offset would reach the reported total. Invalid page-size metadata falls back to the current default of 52, while a missing or invalid total prevents further pagination.

The route builder supports global query searches and documented page paths with page-scoped brand filters. It normalizes, sorts, deduplicates, and encodes those brands. Unsupported structured combinations fail early and leave explicit URL as the escape hatch.

A store controls both the page route and the page shape. Keeping those assumptions together makes provider behavior easier to review. A new category field is incomplete if its generated route reaches a page the parser cannot understand.

The loader should not become a universal URL builder. It validates shared shape and converts provider values. Factories choose explicit or generated routing and storage. Provider URL modules turn store concepts into actual paths and query strings.

Preview uses provider helpers too. That keeps the route shown to an operator consistent with the route a factory would give the motor. It also lets provider URL tests verify exact outputs without constructing motors or touching runtime data.

When route and parser work change together, tests can tell a coherent story. One test proves the builder emits a supported page. Another fixture proves the parser understands that page’s data shape. The shared motor tests remain independent of both.

Provider parser tests use small HTML or JSON fixtures and assert normalized items, pagination, and incomplete behavior. They do not start the scheduler or make live requests. A store change can therefore be reproduced with the smallest relevant input.

Motor-flow tests use fake provider implementations to exercise shared fetching, page traversal, reconciliation, callbacks, and block state. Job tests construct provider motors and check URL or storage results without scraping the stores.

This division makes a failing test informative. An Amazon card change should fail near Amazon parsing. A three-miss lifecycle regression should fail with a fake motor. A duplicate job ID should fail in loader tests. The test location points toward ownership before anyone opens production logs.

Package imports reinforce the rule. Shared code must not import from provider/ or scraper/. Providers depend on shared motor contracts, and scraper runtime code assembles concrete providers. Reversing those dependencies would make the reusable center depend on every edge.

The shared item does not preserve every badge, installment offer, seller score, image, shipping promise, or crossed-out price a store displays. Adding fields because one page exposes them would make every provider and persisted record carry concepts they cannot support consistently.

A new shared field needs a cross-provider meaning. The project would need to define whether it is required, how older JSON loads without it, how updates enter history, and whether notifications consume it. That is a data-model change rather than a parser convenience.

Provider-only metadata can stay inside parsing decisions when it helps choose the normalized value. An adapter may inspect several price elements to select the current numeric price without exposing all of those elements to lifecycle code.

The contract also avoids returning raw HTML. Persisting source pages would increase storage, potentially retain unrelated personal or session data, and bind history to store markup. Focused test fixtures are a safer place to keep the minimum source shape needed for regression.

This smaller contract makes provider comparison possible. Every adapter can answer which listing appeared, what it is called, what it costs, where it lives, and whether another results page follows. The rest of the service builds its guarantees from those facts.

A new store begins with a provider package, motor subclass, provider key, page parser, and route helpers. Its factory accepts the shared job fields, builds a relative storage path, and registers under a short key.

The provider then needs tests for factory construction, generated and explicit routing, parser output, pagination, and known empty or gated pages. Configuration examples document the supported public fields. Runtime policy can add a provider-specific fetch strategy, delay, cooldown, or concurrency limit.

The scheduler should not gain a branch that names the new store. It already groups motors by provider_key, runs provider loops, applies limits, refreshes health, and routes normalized events. If adding a provider requires changing that algorithm, the adapter contract may be leaking.

That is the practical payoff of the design. Store-specific code remains detailed because storefronts are detailed. The rest of the service stays small enough to reason about because those details cross one narrow page contract.

Continue with when a store blocks the scraper to see how fetch policy produces usable or incomplete page bodies. Then how a product becomes price history follows normalized items after they leave the provider boundary.