Operating MLScraper and adding a store

The first useful skill in scraper operations is knowing when not to touch the parser. A missing product can begin with invalid YAML, a browser dependency, a provider queue, a blocked page, an incomplete cycle, or a changed storage filename. MLScraper exposes clues for each boundary. This final chapter turns those clues into a local operating routine, then uses the same boundaries to add a provider without teaching the scheduler another store.

The project targets Python 3.14 in pyproject.toml. Its current exception syntax also relies on that version, so an older interpreter is not a faithful environment for running the suite or service.

The VS Code dev container installs runtime dependencies, development tools, Playwright Chromium, and browser system packages. Outside the container, install the Python requirements and browser explicitly.

Start the FastAPI application from the repository root after previewing any jobs whose route inputs changed and confirming the intended data directory.

The service listens on port 80 and exposes GET /health. Starting it also starts recurring provider loops, so validate job routes before this step. Use preview-url when a configuration change only needs route inspection.

Runtime JSON defaults to ./data. Set DATA_PATH when experiments need an isolated directory. This is safer than deleting or hand-editing a working history after a test.

Telegram credentials live in an ignored local file. Copy the example only when notifications are wanted and keep the populated file out of review.

Fill in the bot token and destination chat ID without committing the file. When the file is missing or blank, the app logs a warning and disables sends. Scraping, persistence, and health continue.

This optional behavior is useful during development. A test run can build history without contacting a chat. It also keeps a notification configuration problem from becoming a scraper startup failure.

The first successful scrape of a new watch suppresses new-item messages regardless of credentials. Expecting silence during baseline creation avoids diagnosing intended behavior as a broken Telegram integration.

Call the endpoint before searching logs at random so the provider, queue, block, timing, and recent error fields narrow the investigation.

First read top-level state and motor count. starting means readiness has not completed. idle_no_motors means the process could not build valid work from the job catalogue. An ok top level only says at least one provider cycle completed, so continue into provider entries.

Next compare provider status, active jobs, queued jobs, and configured limit. A queue can be normal while all capacity is active. A queue with no active work needs logs and block state. Long cycle duration can come from pagination, page delays, browser waits, or slow responses.

Then inspect blocked jobs and reasons. Those fields identify fetch startup, navigation, readiness, gate, parse, or rate-limit conditions. The reason tells you whether to inspect dependencies, provider policy, a source response, or parser fixtures.

Finally, compare health with the relevant JSON file. The endpoint explains current execution, while the article record explains what earlier complete cycles saved. Neither source replaces the other.

Health aggregates state for repeated inspection. Logs provide the timeline around a specific event. Use them after health identifies the provider and likely boundary.

The runtime logs provider cycle start and finish behavior, configured limits, queue transitions, and escaped errors. Motors log page result counts, next-page presence, incomplete conditions, and saved totals. Providers log malformed records and changed page structures.

Avoid adding duplicate logs at every layer. A provider parser should explain a provider assumption. The motor should explain page-sequence completeness. The scheduler should explain provider cycles. Repeating the same exception everywhere makes the useful owner harder to find.

Saved records add another timeline through datetime, last_updated, history, status history, and hold counts. Copy the file before a forensic inspection and keep the original evidence intact.

The project uses the standard library unittest runner. Its tests are designed to avoid live store requests and tracked runtime data.

Optional checks cover types, lint, formatting, and the configured pre-commit hooks used by contributors before a change enters local review.

The suite is divided by behavior. Article tests cover history and states. Fetcher tests use fakes for retries, timeouts, block selectors, and browser behavior. Job tests cover coercion, routes, paths, and registry rules. Provider tests use small fixtures. Runtime tests patch sleeps and inspect grouping or concurrency.

Offline tests cannot prove a live store still exposes the same page. They prove how MLScraper behaves for documented inputs and failure shapes. When a store changes, capture the smallest safe fixture that reproduces the change and update the provider test before broad runtime edits.

Use temporary DATA_PATH values for manual motor experiments. A developer check should not advance holds, overwrite history, or create backups in the directory used by a long-running service.

A new provider should change the edge of the service rather than the recurring scheduler. Follow this sequence to preserve that boundary.

1. Create provider/<name>/ with an __init__.py and focused helper modules.
2. Add a motor subclass and assign a short, unique PROVIDER_KEY.
3. Implement scrape_page(body) to return normalized items and an optional next URL.
4. Put selectors, payload paths, pagination, route building, and provider options inside that package.
5. Register a factory that declares job_id, url, and query and builds a relative storage path.
6. Add loader coercion only for structured YAML values that need provider validation.
7. Add provider fetch, delay, cooldown, and concurrency policy to config/motors.yaml when defaults do not fit.
8. Document a safe example job and the supported option vocabulary.
9. Add factory, route, parser, pagination, and incomplete-page tests.
10. Run the offline suite before any controlled source check.

The normalized contract remains small enough for every provider to implement and for shared lifecycle code to consume directly without store-specific branches.

Mark page-level uncertainty as incomplete. An absent application payload, known gate, malformed response, or suspicious empty page should not return a confident empty catalogue unless the provider has a reliable signal that zero products is valid.

The provider should not import scheduler helpers, Telegram formatting, or file-manager internals. The motor already handles shared fetch dispatch, lifecycle, persistence, and callback context. Crossing those lines creates another path that future providers must imitate.

A selector edit needs a parser fixture with valid and malformed items. A pagination edit needs boundary cases around the last page. A URL option needs generated, explicit, invalid, and preview tests. A lifecycle change needs complete and incomplete motor flows.

A provider concurrency change needs runtime tests around limits and counters. A persistence change needs temporary files, traversal rejection, backup behavior, and recovery. A notification change needs numeric parsing, first-run behavior, threshold edges, and send fakes.

This approach keeps tests small enough to explain failure. Starting the full application for every change can hide the owner behind asynchronous tasks, network setup, and unrelated jobs.

After focused tests pass, preview affected jobs. Then use an isolated data path for controlled runtime verification. Watch health and logs before inspecting output. Live source behavior belongs at the end of the process because it is the least repeatable input.

MLScraper has no hosted dashboard, user authentication, database server, or general alert bus. JSON files suit one service process and human inspection, but they are not designed for multiple concurrent writers or large analytical queries.

The 400-second successful-cycle interval and 14-percent alert threshold are code constants. Provider lifecycle threshold defaults to three misses. Moving those decisions into validated configuration would make different watches easier to tune without editing Python.

Browser context is shared and long lived, while pages close after each fetch. A future shutdown path could explicitly close Playwright resources. Broader observability could add metrics or structured event output beyond the current health dictionary and logs.

Store compatibility remains ongoing work. Fixtures document known shapes, but stores control their markup, application data, route tokens, and access behavior. The project can reduce the damage of change and shorten diagnosis. It cannot remove that external uncertainty.

These are useful next steps precisely because the current boundaries make their likely homes visible. Configuration policy belongs near loaders and motor config. Another notification channel belongs behind broadcast routing. Another storage backend belongs behind repository behavior. A dashboard should consume state rather than move parsing into HTTP handlers.

Imagine Liverpool health shows normal scheduling, but logs report that __NEXT_DATA__ is missing. Existing products remain active because the provider marked the cycle incomplete. That safety signal means the immediate incident has not corrupted lifecycle state.

Capture the smallest page fragment needed to reproduce the missing or changed payload without storing unrelated source content. Add a provider test that proves the current parser marks the old shape incomplete, then add a focused fixture for the new supported shape.

Change Liverpool parsing inside its provider package. Do not relax shared reconciliation or teach the scheduler about __NEXT_DATA__. Run provider and motor tests to confirm valid records normalize correctly and incomplete pages still preserve saved state.

Use a temporary data root for any controlled runtime check. Preview the job route first, start one relevant watch, and inspect health before opening the output file. A successful complete cycle should clear the provider symptom and update only the expected articles.

If the page change also affects route generation, test that separately through URL helpers and preview. Parser and route failures can arrive together, but combining their fixes into one untestable live experiment makes regression evidence weaker.

This example scales to other providers. Begin from the health signal, reproduce the smallest owned assumption, change the edge package, protect the shared contract, and verify with isolated state before returning the job to ordinary scheduling.

Preview configuration before starting loops. Read health before editing parsers. Treat incomplete cycles as protection. Preserve runtime files as evidence. Reproduce store changes with small fixtures. Add providers at the edge and keep shared execution store neutral.

That routine captures the engineering value of MLScraper. The project cannot promise cooperative storefronts, but it can make configuration, fetching, parsing, state, persistence, and alerts independently understandable. When something fails, the next useful question has a bounded place to look.

Return to the MLScraper case-study overview for the complete reading map, or inspect the project through its GitHub link there. The repository is most approachable after following your first price watch and seeing how each operational boundary protects the same saved product history.