History
A change keeps its previous value
Price and title updates move the old values into capped history records before the latest observation replaces them.
The bargain I missed
I saved a deeply discounted product and returned five minutes later ready to buy it. It was already sold out. After seeing several more bargains disappear just as quickly, I started wondering how many I never saw at all.
That question became MLScraper. It now watches Amazon Mexico, Mercado Libre, Liverpool, and Palacio de Hierro, keeps local price history, and reports selected drops through Telegram. It worked so well that I eventually stopped using it because I was spending money on things I did not need.
Four providers run without sharing one failure state
FastAPI starts the service and exposes health. The scraper groups jobs by provider, applies a provider-specific concurrency limit, and schedules each group independently. One blocked store does not stop the others.
Flowchart
Start with one price watch, then open the service
Follow one watch before opening the internals
Configure an illustrative Amazon Mexico job, preview its URL, and trace its baseline, later price change, health report, and optional alert.
Read this chapter →See how YAML becomes a store URL
Learn which fields define a watch, how provider options build a route, and why previewing the result is safer than discovering a mistake inside the scheduler.
Read this chapter →Follow one scheduled scrape cycle
See how FastAPI starts the service, provider loops share concurrency, motor failures stay isolated, and health describes the work running in the background.
Read this chapter →Keep store parsing inside adapters
Compare the four page formats and the small product record that lets shared lifecycle code work without importing store-specific selectors.
Read this chapter →Retries and cooldowns solve different failures
A fetcher retries short-lived network failures inside one request. A provider loop backs off after a broader cycle failure. Keeping those timers separate slows the affected work without freezing other stores.
Browser mode waits for evidence that a page is usable. A returned HTML document may still be a login gate or an empty application shell. Recognized block signals appear in health, while incomplete cycles stop before reconciliation.
Follow the fetch and backoff paths →A listing becomes durable product state
Lifecycle
Three complete misses finish a product
One complete miss moves an active product on hold. The default threshold waits for three complete misses before moving it to finished.
Safety
Incomplete pages cannot report absences
A blocked or malformed page stops before missing-item reconciliation, so uncertain input does not rewrite valid product state.
Health is the first debugging surface
Health built for an operator
The readiness surface reports provider status, active and queued jobs, block reasons, cycle counts, timing, and recent errors. An operator can distinguish a live process from a healthy scrape cycle.
Alerts begin after the baseline
The first successful cycle records products without announcing every item as new. Later price drops of at least 14 percent can become Telegram messages when notifications are configured.
Every watch follows the same guarded cycle
- Load job
- Fetch pages
- Parse items
- Reconcile
- Report
The outcome
MLScraper turned a missed bargain into a local price-watching service I could trust day to day.
It watches four stores, keeps product history, handles blocked pages without corrupting state, and sends Telegram alerts only after a real price change. The project is still intentionally small. It runs as one process and writes local files, but it has the pieces that made it useful: provider boundaries, guarded reconciliation, operator health, and alerts that arrive before the deal disappears.
Technical deep dives
Follow the project beyond the landing page. Each publication focuses on one architectural boundary, implementation decision, or operational lesson.
Vault publications
- Your first MLScraper price watch12 min read · #python #scraping #tutorial #automation
Follow one illustrative Amazon Mexico watch from YAML and URL preview to saved product history, runtime health, and a later price alert.
- How a YAML job becomes a store URL11 min read · #python #configuration #scraping
Learn how MLScraper turns stable job identity and provider options into validated store routes, predictable storage paths, and safe previews.
- What happens during one scrape cycle11 min read · #python #fastapi #asyncio #architecture
See how FastAPI starts MLScraper, provider loops share concurrency safely, and health turns background work into observable state.
- Why every store needs its own adapter10 min read · #python #architecture #scraping #adapters
Compare four ecommerce page formats and learn how MLScraper isolates store-specific URLs and parsing behind one product contract.
- When a store blocks the scraper10 min read · #python #playwright #scraping #operations
Learn how MLScraper chooses HTTP or browser fetching, protects history after incomplete pages, and slows the right scope after failures.
- How a product becomes price history10 min read · #python #data #persistence #state-machines
Follow one stable product identifier through updates, temporary absence, recovery, JSON persistence, and MLScraper price-drop alerts.
- Operating MLScraper and adding a store10 min read · #python #testing #operations #scraping
Run MLScraper locally, diagnose health and saved state, exercise its offline tests, and extend the provider contract without runtime leaks.