Start with the unit of work.
The requirement spans a related set of eligible public pages, not a queue of isolated URL requests your application wants to orchestrate itself.
API d’exploration
Crawl API for coordinated multi-page web collection.
Collect related eligible public pages as one bounded workflow, with source scope, collection rules, and results aligned to your pipeline.
- Approved entry points The public source space to evaluate
- Coverage criteria The page families that matter
- Result requirements What usable means downstream
- Operating cadence One-time or recurring business need
Collection fit
When one page request becomes a collection workflow.
Use Crawl API when related pages need to be treated as one bounded collection. Your team still defines what complete, correct, and usable means for its business decision.
Give the collection one boundary.
Entry points, relevant page families, exclusions, and acceptance rules belong to one agreed collection brief.
Review the result as a whole.
Judge coverage, source context, exceptions, and downstream usability against the agreed result contract—not page retrieval alone.
Crawl API is a fit whenthe bounded page collection is the unit your pipeline needs to receive and evaluate.Compare the alternatives
The crawl brief
Define the collection boundary before the first run.
Bring the source and business requirements that determine whether a collection is eligible, bounded, reviewable, and useful.
Source eligibility
Name the public sources, collection purpose, and policy requirements your team has reviewed.
Entry points
Bring representative domains, sections, or known pages that define where evaluation starts.
Coverage criteria
Describe the page families that count and the evidence your team uses to assess coverage.
Exclusions and stop conditions
Make off-limits areas, irrelevant paths, and business stop conditions explicit before collection.
Business cadence
Explain whether the need is one-time or recurring so the operating model can be evaluated honestly.
Quality checks
Define the checks your team will use to decide whether the collection is complete enough and usable.
Planning inputs, not a parameter list. The crawl brief turns source scope, boundaries, exclusions, and result needs into a workable collection plan.
From scope to collection
A crawl job should have a
clear beginning, boundary, and handoff.
The operating model begins with a buyer-approved brief and ends with a result your team can review against explicit acceptance criteria.
- 01 · defineCrawl brief
Frame the source space.
Your team supplies the approved entry points, relevant page families, exclusions, intended use, and quality criteria.
- 02 · executeManaged collection
Run the bounded crawl.
WebScrapingAPI operates the agreed collection path and returns results with source context.
- 03 · reviewBuyer review
Validate the collection.
Your team assesses coverage, exceptions, source context, and downstream usability against its quality criteria.
Brief before build
A clear crawl brief keeps source scope, result shape, exception handling, and delivery aligned before your team builds around the workflow.
Define done
Specify the usable result before choosing a delivery format.
A multi-page collection is useful only when every result and exception can be interpreted inside the customer's downstream workflow.
What is one page result?
Agree on the minimum page-level object your pipeline needs to inspect, process, or reject.
How is source context retained?
Identify the source identity and collection context that must travel with each usable result.
How are skipped or failed pages represented?
Define which non-result states must remain visible so absence is not mistaken for evidence.
Where must the collection go next?
Describe the system, team, and acceptance step that receive the collection after the job boundary.
Result planning
Align result structure, delivery route, and retention window with the workflow before the first production run.
Implementation plan
Define the job shape before your team builds around it.
Start with the operating decisions below so the integration is based on the source scope and result your workflow needs.
Implementation decisionPlanning pointWhat becomes explicit
The production access surface and credential flow.
How entry points, boundaries, and collection requirements are represented.
The operating states and customer actions available at each state.
How the collection and its source context reach the receiving system.
Which exceptions are visible, who decides on another attempt, and how repeated work is treated.
The scope, operating constraints, and result-availability window.
Operating ownership
WebScrapingAPI operates the crawl. Your team owns collection intent and review.
A written boundary prevents a web-access product from being mistaken for a fully operated data-delivery program.
Exécution de ramassage gérée
- Collection and access execution for the scoped crawl
- Job, result, and error surface for the workflow
- Structured collection handoff
Intent, limites et révision
- Source eligibility, purpose, entry points, exclusions, and stop criteria
- Coverage and quality validation
- Parsing and schema unless included in the crawl plan
- Storage, retention, and downstream use
- Source-change monitoring outside the crawl workflow
Le flux de travail de ramassage
- Input and coverage behavior
- Result and handoff behavior
- Lifecycle, exception, and operating controls
- Limits, cadence, pricing, support, and security requirements
Candidate workloads
Built for questions that span a related page set.
Each workload starts with source, boundary, result, and operating-fit clarity. Crawl API supplies the collection layer, not the customer's final business conclusion.
Product choice
Choose by unit of work and operating ownership.
Crawl API occupies a specific middle ground: broader than one page request, but not a substitute for maintained structured records or a fully operated delivery program.
Route réseau
Les proxies
Infrastructure building blockUne pageVotre équipe gère le client, le navigateur, le parseur, les retries, la validation et le stockage.
API de scraping
Managed retrievalStatut d'une pageEnvoyer une demande de page admissible avec un accès géré et des contrôles de réponse documentés.
API de navigateur
Guided browser requestListe des pages connexesEnvoyez une demande soutenue par le navigateur avec des contrôles d'état et d'interaction documentés.
API d’exploration
Bounded crawl workflowEnregistrements soutenusPlanifiez un travail de collecte multi-pages à l'échelle unique avec des limites de source et des besoins de résultats définis à l'avance.
API de données
Target-specific data productProgramme opérationnelLa demande a maintenu des enregistrements structurés à partir d'une source et d'un schéma pris en charge.
Données gérées
Data delivered to specificationDéfinir les sources, le schéma, la cadence, la qualité et la destination alors que WebScrapingAPI exploite le programme récurrent.
Commercial fit
Size the collection you actually need.
Pricing depends on source scope, volume, cadence, result needs, and delivery model. Start with a representative collection so the plan matches the work.
Apportez la charge de travail, pas un nombre de pages générique.
Source set
Representative eligible entry points
Page families
What belongs in the collection
Expected breadth
The practical shape of the source space
Business cadence
One-time or recurring need
Result requirements
Context, exceptions, and acceptance
Receiving workflow
Where the collection goes next
FAQ
Crawl API questions for planning a collection.
These answers explain product fit, ownership, and how to prepare a useful crawl brief.
What is Crawl API?
Crawl API is designed as the multi-page collection layer for a bounded set of related eligible public pages. Start by sharing the source space, collection boundary, and result your pipeline needs.
When should I use Crawl API instead of Scraper API?
Evaluate Crawl API when the unit of work is a related page set that should be handled as one bounded collection. Use Scraper API when your application already knows each URL and needs one managed page request at a time.
How is Crawl API different from Browser API?
Crawl API is evaluated around a scoped collection spanning related pages. Browser API is one documented browser-backed request where your application specifies the page state, supported interactions, context, and response.
How is Crawl API different from Data API?
Crawl API starts from an approved source space and a bounded multi-page collection need. Data API returns maintained structured records only for supported sources and schemas.
How is Crawl API different from Managed Data?
With Crawl API, your team owns collection intent, quality review, downstream validation, and the operating work outside the crawl job. Managed Data moves recurring collection, extraction, quality, maintenance, and delivery to WebScrapingAPI.
What sites and content can I crawl?
Eligibility and coverage depend on the public source, your intended use, and the collection boundary. Share representative entry points, page families, exclusions, and success criteria.
How is crawl scope defined?
Start with approved entry points, the page families that count, explicit exclusions, stop conditions, and the criteria your team will use to judge coverage and usability.
Which output formats and delivery options are supported?
The supported result representation depends on the collection model. Request a crawl brief to align result structure, source context, delivery, and retention with your workflow.
Can I schedule, monitor, cancel, retry, or resume a crawl?
Job-lifecycle controls depend on the workload. We help define the operating behavior, error treatment, and responsibilities before your team builds against the workflow.
How are pricing, limits, and service commitments determined?
Pricing depends on source scope, page volume, cadence, result needs, and delivery model. Share a representative collection to receive a workload-specific plan.
Votre courte course
Turn a source space into a bounded collection plan.
Bring one representative domain, the pages that matter, the boundaries that must hold, and the result your pipeline needs. We will help choose the right product path.