How a YAML job becomes a store URL

A scraper configuration can look harmless while changing the identity of years of saved observations. Rename one field and the service may write a new file. Mix a copied URL with generated filters and nobody knows which route should win. MLScraper keeps that ambiguity out of the runtime by giving every watch a small shared contract, provider-owned options, and a preview command that reveals the final URL before recurring work begins.

Every active watch lives under the jobs list in config/jobs.yaml. The shared part of a job has three fields. job_id is required, while url and query are optional inputs supported by every provider factory.

The provider key selects a factory. The factory uses the remaining fields to construct one motor with a final URL and a relative storage path. That motor carries the same job ID into logs, health context, persisted filenames, and notification payloads.

This makes job_id operational identity rather than a display label. Keep it stable after a watch begins collecting data. A cosmetic rename changes the slug used in the generated filename, so the next process can start from an empty file even though an earlier history still exists under the old name.

Uniqueness is scoped to one provider. Amazon and Liverpool can both have a job called Apple, because their storage directories and runtime groups differ. Two Amazon jobs cannot share that ID. The loader rejects provider-local duplicates before factories build motors.

A query answers what a person wants to search for. It does not always determine the entire route. Provider options can add seller, category, page, brand, state, or seeded size information. Those concepts belong to the store that defines them.

Structured jobs are the preferred path when MLScraper already models the route. Their fields are shorter than copied browser URLs, easier to review, and checked against provider catalogues.

Each provider exposes a deliberate option vocabulary that keeps store-specific route concepts out of the shared job contract and beside the code that understands them.

The loader converts supported strings into provider enums or resolved page objects. An unknown Amazon brand, conflicting Liverpool page aliases, invalid Palacio brand shape, or unsupported size combination causes that entry to be skipped with a warning. Other valid entries can still load.

Generated fields document what the code understands. They do not attempt to model every filter a store can produce. That limit prevents the shared YAML format from becoming a second routing language filled with one-off fields that only make sense to one storefront.

Sometimes a useful store route contains refinements the provider catalogue does not model. In that case, a job can carry a complete URL copied from a browser or another verified source.

A nonblank url has absolute precedence. The provider uses it unchanged and does not merge query, seller, page, category, brand, state, or size fields into it. Factories log a warning when explicit and structured inputs appear together, then ignore the structured route inputs.

That rule removes a subtle source of surprises. Without it, a reviewer would need to know whether the copied URL wins, generated filters replace its query string, or both paths combine. MLScraper makes the answer boring. The explicit string is the route.

The escape hatch trades validation for reach. A copied route can express store behavior the project does not understand, but the loader cannot explain whether its filters remain correct. Generated options cover fewer shapes and provide better review and test boundaries.

Use explicit URLs for genuinely unmodeled routes. Do not use them to avoid learning a supported provider option. Structured fields communicate intent, survive store URL formatting changes more cleanly, and give provider tests a smaller contract to protect.

The job loader owns document-level checks. It verifies that the YAML root contains a list, requires a nonblank provider and job ID, rejects legacy shared fields, finds duplicates, and coerces provider values it knows how to validate.

The factory owns construction. It receives the clean entry, chooses explicit or generated routing, builds a stable storage path, and creates the provider motor. The registry maps short provider keys to those factories and checks that each factory declares the shared job_id, url, and query parameters.

This separation keeps YAML parsing away from motor classes. It also keeps the loader from constructing store routes. The loader can say that nintendo is a known Amazon brand. The Amazon URL builder decides that the brand maps to a particular rh refinement value and encoding order.

Unknown keys currently pass through loader validation. Factories accept extra keyword arguments and use the fields relevant to their provider. That allows provider-specific growth without adding every optional constructor field to one central switch, though documented fields still need tests and examples.

Factory failures do not crash construction for every job. The registry logs an unknown provider or factory error and skips that entry. This behavior favors a running service with visible omissions over losing every valid watch because one entry cannot be built.

Every factory creates the final URL and relative JSON path in one place. The path helper normalizes accents, lowercases text, replaces unsafe runs with hyphens, and falls back to a short digest when a job ID contains no usable characters.

Provider directories prevent cross-store collisions. Qualifiers preserve selected generated route context for Amazon and Mercado Libre. Liverpool and Palacio currently use the stable job slug without appending every option, so the job ID remains the clearest identity for those watches.

Paths must stay relative to DATA_PATH. The file manager resolves them below the configured root and rejects parent traversal or absolute paths outside that root. A YAML field should never be able to turn a watch into an arbitrary filesystem write.

Routing and storage still represent different promises. The URL can evolve when provider generation changes, while a stable job ID keeps the history file in place. Review route changes with that distinction in mind. A better URL should not casually reset product memory.

Use preview after changing a query, seller, category, page, brand, size, or explicit URL. The command loads the same job file and calls the provider preview helper without constructing motors or starting FastAPI.

The lookup requires exactly one match. No match means the provider or job ID is wrong. Multiple matches indicate ambiguous configuration. A valid match returns either the explicit URL unchanged or the route generated from provider fields.

Liverpool deserves extra attention because some generated routes resolve seller-filtered segments through store metadata. Preview follows that same path, so it can surface missing resolver tokens, incorrect ancestors, or unsupported combinations before the scheduled service starts. Explicit URLs remain the bypass when a Liverpool route cannot be modeled safely.

Preview does not test the parser, network, or current inventory. It proves that the local configuration resolves to a particular route. That smaller promise makes failures easier to interpret. A bad preview is a configuration problem. A good preview followed by browser_blocked is a fetch problem.

A missing required field belongs to job validation. An unknown provider option belongs to provider coercion. A malformed generated route belongs to the provider URL builder. A valid route with changed markup belongs to the provider parser. An unexpected new storage file usually begins with job identity or factory path construction.

Use this review order for configuration changes so identity, routing, storage, and later parser expectations stay aligned before recurring work reaches the store.

1. Keep the existing job_id unless a new history is intentional.
2. Confirm the ID is unique within its provider.
3. Prefer structured options when the provider documents the route.
4. Remove structured route fields when an explicit URL should win.
5. Run preview and inspect the complete result.
6. Confirm the expected provider directory and filename.
7. Watch provider health after the next real cycle.

The repository’s job tests document these rules without reaching live stores. They cover enum coercion, invalid entries, duplicate IDs, explicit URL precedence, slugging, path generation, registry signatures, and each provider factory. URL tests cover refinement order, aliases, route constraints, and preview behavior.

Those tests are useful documentation because store routes are easy to misread in prose. A fixture with exact expected encoding tells a future change whether a builder preserved its contract. It also separates a known generated route from a copied string that the project intentionally treats as opaque.

A valid job proves that local intent can become a route and storage path. It cannot prove that the remote page still exists, that a seller exposes the same refinements, or that the provider parser understands the response. Keeping that promise narrow prevents preview from becoming a misleading integration test.

Generated routes can still age. A documented enum value may point to a filter the store later removes. A Liverpool resolver may stop returning a required token. Tests preserve the project’s known contract, while runtime health reveals when the source stops honoring it.

Explicit routes age differently. The project preserves them unchanged, so it cannot repair or reinterpret their filters. A copied URL may remain syntactically valid while leading to a login page, a redirect, or a different catalogue. The job owner must review that opaque input.

Storage identity deserves a separate migration decision. If a watch must receive a new job_id, move or transform its data deliberately rather than relying on the next scrape to reconstruct history. The current project does not include an automated history migration command.

Secrets do not belong in jobs. The supported providers use public page routes, and Telegram credentials live in a separate ignored file. Adding tokens or private query values to tracked YAML would blur configuration with secret management and risk exposing them in review.

These boundaries make configuration honest. It can validate structure, known options, precedence, and deterministic outputs. Network access, live page meaning, parser compatibility, and data migration remain separate concerns with separate evidence.

MLScraper does not wait for a running scheduler to discover every mistake. The loader rejects invalid structure, provider helpers constrain known options, factories make URL precedence explicit, path helpers normalize identity, and preview exposes the route on demand.

That sequence protects both runtime behavior and product history. A job is small, but it decides what page gets fetched, what provider interprets it, what filename receives observations, and what label appears in health and notifications. Treating the YAML as identity-bearing input makes those consequences reviewable.

Continue with what happens during one scrape cycle to see how valid motors are grouped and scheduled. If a previewed route later times out or returns a gate, when a store blocks the scraper explains the fetch and backoff paths.