Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ if __name__ == '__main__':

### More examples

Explore our [Examples](https://crawlee.dev/python/docs/examples) page in the Crawlee documentation for a wide range of additional use cases and demonstrations.
Explore the [Guides](https://crawlee.dev/python/docs/guides) section of the Crawlee documentation for a wide range of additional use cases and demonstrations.

## Features

Expand Down
7 changes: 4 additions & 3 deletions docs/quick-start/index.mdx → docs/01_quick-start/index.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: quick-start
title: Quick start
description: Build and run your first Crawlee crawler in a few minutes, and pick the crawler type that fits your project.
---

import ApiLink from '@site/src/components/ApiLink';
Expand All @@ -15,7 +16,7 @@ import PlaywrightCrawlerExample from '!!raw-loader!roa-loader!./code_examples/pl

import PlaywrightCrawlerHeadfulExample from '!!raw-loader!./code_examples/playwright_crawler_headful_example.py';

This short tutorial will help you start scraping with Crawlee in just a minute or two. For an in-depth understanding of how Crawlee works, check out the [Introduction](../introduction/index.mdx) section, which provides a comprehensive step-by-step guide to creating your first scraper.
This short tutorial will help you start scraping with Crawlee in just a minute or two. For an in-depth understanding of how Crawlee works, check out the [Introduction](../02_introduction/index.mdx) section, which provides a comprehensive step-by-step guide to creating your first scraper.

## Choose your crawler

Expand Down Expand Up @@ -61,7 +62,7 @@ If you plan to use the <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`
playwright install
```

For detailed installation instructions, see the [Setting up](../introduction/01_setting_up.mdx) documentation page.
For detailed installation instructions, see the [Setting up](../02_introduction/01_setting_up.mdx) documentation page.

## Crawling

Expand Down Expand Up @@ -128,6 +129,6 @@ If you want to change the storage directory, you can set the `CRAWLEE_STORAGE_DI

## Examples and further reading

For more examples showcasing various features of Crawlee, visit the [Examples](/docs/examples) section of the documentation. To get a deeper understanding of Crawlee and its components, read the step-by-step [Introduction](../introduction/index.mdx) guide.
For more examples showcasing various features of Crawlee, visit the [Guides](./guides) section of the documentation. To get a deeper understanding of Crawlee and its components, read the step-by-step [Introduction](../02_introduction/index.mdx) guide.

[//]: # (TODO: add related links once they are ready)
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: setting-up
title: Setting up
description: How to install Crawlee, set up your environment, and bootstrap a new project.
---

import ApiLink from '@site/src/components/ApiLink';
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: first-crawler
title: First crawler
description: Build your first Crawlee crawler - set up a request queue, write a request handler, and crawl your first page.
---

import ApiLink from '@site/src/components/ApiLink';
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: adding-more-urls
title: Adding more URLs
description: Grow the crawl by discovering and enqueuing new links, with filtering and deduplication handled for you.
---

import ApiLink from '@site/src/components/ApiLink';
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: real-world-project
title: Real-world project
description: Plan a real scraping project - choose the data to collect and analyze the target website before writing code.
---

import ApiLink from '@site/src/components/ApiLink';
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: crawling
title: Crawling
description: Crawl the example Warehouse store - visit the category listings and enqueue the product detail pages.
---

import ApiLink from '@site/src/components/ApiLink';
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: scraping
title: Scraping
description: Extract structured data such as titles, prices, and stock information from the product detail pages.
---

import ApiLink from '@site/src/components/ApiLink';
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: saving-data
title: Saving data
description: Persist the scraped results into a dataset and find them on disk.
---

import ApiLink from '@site/src/components/ApiLink';
Expand Down Expand Up @@ -88,19 +89,10 @@ A helper <ApiLink to="class/PushDataFunction">`context.push_data`</ApiLink> save

:::info Automatic dataset initialization

Each time you start Crawlee a default <ApiLink to="class/Dataset">`Dataset`</ApiLink> is automatically created, so there's no need to initialize it or create an instance first. You can create as many datasets as you want and even give them names. For more details see the <ApiLink to="class/Dataset#open">`Dataset.open`</ApiLink> function.
Each time you start Crawlee a default <ApiLink to="class/Dataset">`Dataset`</ApiLink> is automatically created, so there's no need to initialize it or create an instance first. You can create as many datasets as you want and even give them names. For more details see the [Storages](../concepts/storages#dataset) page and the <ApiLink to="class/Dataset#open">`Dataset.open`</ApiLink> function.

:::

{/* TODO: mention result storage guide once it's done

:::info Automatic dataset initialization

Each time you start Crawlee a default <ApiLink to="class/Dataset">`Dataset`</ApiLink> is automatically created, so there's no need to initialize it or create an instance first. You can create as many datasets as you want and even give them names. For more details see the [Result storage guide](../guides/result-storage#dataset) and the `Dataset.open()` function.

:::
*/}

## Finding saved data

Unless you changed the configuration that Crawlee uses locally, which would suggest that you knew what you were doing, and you didn't need this tutorial anyway, you'll find your data in the storage directory that Crawlee creates in the working directory of the running script:
Expand All @@ -111,16 +103,12 @@ Unless you changed the configuration that Crawlee uses locally, which would sugg

The above folder will hold all your saved data in numbered files, as they were pushed into the dataset. Each file represents one invocation of <ApiLink to="class/Dataset#push_data">`Dataset.push_data`</ApiLink> or one table row.

{/* TODO: add mention of "Result storage guide" once it's ready:

:::tip Single file data storage options

If you would like to store your data in a single big file, instead of many small ones, see the [Result storage guide](../guides/result-storage#key-value-store) for Key-value stores.
If you would like to store your data in a single big file, instead of many small ones, see how to [export the whole dataset](../concepts/storages#exporting-the-dataset) to JSON or CSV, or use a [key-value store](../concepts/storages#key-value-store).

:::

*/}

## Next steps

Next, you'll see some improvements that you can add to your crawler code that will make it more readable and maintainable in the long run.
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: refactoring
title: Refactoring
description: Clean up the crawler code with a router and separate handlers to keep the project maintainable.
---

import ApiLink from '@site/src/components/ApiLink';
Expand All @@ -21,7 +22,7 @@ You might be wondering about the **anti-blocking, bot-protection avoiding stealt

However, the default configuration, while powerful, may not cover every scenario.

If you want to learn more, browse the [Avoid getting blocked](../guides/avoid-blocking), [Proxy management](../guides/proxy-management) and [Session management](../guides/session-management) guides.
If you want to learn more, browse the [Avoid getting blocked](../guides/avoid-blocking), [Proxy management](../concepts/proxy-management) and [Session management](../concepts/session-management) guides.
*/}

To promote good coding practices, let's look at how you can use a <ApiLink to="class/Router">`Router`</ApiLink> class to better structure your crawler code.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
id: deployment
title: Running your crawler in the Cloud
sidebar_label: Running in the Cloud
description: Deploying Crawlee-python projects to the Apify platform
description: Deploy your Crawlee for Python project to the Apify platform and run it in the cloud.
---

import CodeBlock from '@theme/CodeBlock';
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
id: introduction
title: Introduction
description: A step-by-step tutorial that takes you from your first crawler to a production-ready scraper for a real website.
---

import ApiLink from '@site/src/components/ApiLink';
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: An overview of the core components of the Crawlee library and its a

import ApiLink from '@site/src/components/ApiLink';

Crawlee is a modern and modular web scraping framework. It is designed for both HTTP-only and browser-based scraping. In this guide, we will provide a high-level overview of its architecture and the main components that make up the system.
Crawlee is a modern and modular web scraping framework. It is designed for both HTTP-only and browser-based scraping. This page provides a high-level overview of its architecture and the main components that make up the system.

## Crawler

Expand Down Expand Up @@ -76,7 +76,7 @@ PlaywrightCrawler --|> StagehandCrawler

### HTTP crawlers

HTTP crawlers use HTTP clients to fetch pages and parse them with HTML parsing libraries. They are fast and efficient for sites that do not require JavaScript rendering. HTTP clients are Crawlee components that wrap around HTTP libraries like [httpx2](https://httpx2.pydantic.dev/), [curl-impersonate](https://github.com/lwthiker/curl-impersonate) or [impit](https://github.com/apify/impit) and handle HTTP communication for requests and responses. You can learn more about them in the [HTTP clients guide](./http-clients).
HTTP crawlers use HTTP clients to fetch pages and parse them with HTML parsing libraries. They are fast and efficient for sites that do not require JavaScript rendering. HTTP clients are Crawlee components that wrap around HTTP libraries like [httpx2](https://httpx2.pydantic.dev/), [curl-impersonate](https://github.com/lwthiker/curl-impersonate) or [impit](https://github.com/apify/impit) and handle HTTP communication for requests and responses. You can learn more about them on the [HTTP clients](./http-clients) page.

HTTP crawlers inherit from <ApiLink to="class/AbstractHttpCrawler">`AbstractHttpCrawler`</ApiLink> and there are five crawlers that belong to this category:

Expand All @@ -86,18 +86,18 @@ HTTP crawlers inherit from <ApiLink to="class/AbstractHttpCrawler">`AbstractHttp
- <ApiLink to="class/PydanticAiCrawler">`PydanticAiCrawler`</ApiLink> parses HTML with Parsel and uses an LLM to extract structured data into a validated Pydantic model.
- <ApiLink to="class/FileDownloadCrawler">`FileDownloadCrawler`</ApiLink> downloads files of any content type, optionally streaming large bodies in chunks.

You can learn more about HTTP crawlers in the [HTTP crawlers guide](./http-crawlers).
You can learn more about HTTP crawlers on the [HTTP crawlers](./http-crawlers) page.

### Browser crawlers

Browser crawlers use a real browser to render pages, enabling scraping of sites that require JavaScript. They manage browser instances, pages, and context lifecycles. Crawlee provides two browser crawlers:

- <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> utilizes the [Playwright](https://playwright.dev/) library and provides a high-level API for controlling and navigating browsers. You can learn more about it in the [Playwright crawler guide](./playwright-crawler).
- <ApiLink to="class/StagehandCrawler">`StagehandCrawler`</ApiLink> extends `PlaywrightCrawler` with AI-powered browser automation via [Stagehand](https://github.com/browserbase/stagehand). It adds natural-language methods (`act`, `extract`, `observe`, `execute`) directly on the page object. You can learn more about it in the [Stagehand crawler guide](./stagehand-crawler).
- <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> utilizes the [Playwright](https://playwright.dev/) library and provides a high-level API for controlling and navigating browsers. You can learn more about it on the [Playwright crawler](./playwright-crawler) page.
- <ApiLink to="class/StagehandCrawler">`StagehandCrawler`</ApiLink> extends `PlaywrightCrawler` with AI-powered browser automation via [Stagehand](https://github.com/browserbase/stagehand). It adds natural-language methods (`act`, `extract`, `observe`, `execute`) directly on the page object. You can learn more about it in the [Stagehand crawler guide](../guides/stagehand-crawler).

### Adaptive crawler

The <ApiLink to="class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink> sits between HTTP and browser crawlers. It can automatically decide whether to use HTTP or browser crawling for each request based on heuristics or user configuration. This allows for optimal performance and compatibility. It also provides a uniform interface for both crawling types (modes). You can learn more about adaptive crawling in the [Adaptive Playwright crawler guide](./adaptive-playwright-crawler).
The <ApiLink to="class/AdaptivePlaywrightCrawler">`AdaptivePlaywrightCrawler`</ApiLink> sits between HTTP and browser crawlers. It can automatically decide whether to use HTTP or browser crawling for each request based on heuristics or user configuration. This allows for optimal performance and compatibility. It also provides a uniform interface for both crawling types (modes). You can learn more about adaptive crawling on the [Adaptive Playwright crawler](./adaptive-playwright-crawler) page.

## Crawling contexts

Expand Down Expand Up @@ -208,7 +208,7 @@ Crawlee provides three built-in storage types for managing data:
- <ApiLink to="class/KeyValueStore">`KeyValueStore`</ApiLink> - Storage for arbitrary data like JSON documents, images or configs. It supports get and set operations with key-value pairs; updates are only possible by replacement.
- <ApiLink to="class/RequestQueue">`RequestQueue`</ApiLink> - A managed queue for pending and completed requests, with automatic deduplication and dynamic addition of new items. It is used to track URLs for crawling.

See the [Storages guide](./storages) for more details.
See the [Storages](./storages) page for more details.

```mermaid
---
Expand Down Expand Up @@ -294,7 +294,7 @@ StorageClient --|> ApifyStorageClient

Storage clients can be registered globally with the <ApiLink to="class/ServiceLocator">`ServiceLocator`</ApiLink> (you will learn more about the <ApiLink to="class/ServiceLocator">`ServiceLocator`</ApiLink> in the next section), passed directly to crawlers, or specified when opening individual storage instances. You can also create custom storage clients by implementing the <ApiLink to="class/StorageClient">`StorageClient`</ApiLink> interface.

See the [Storage clients guide](./storage-clients) for more details.
See the [Storage clients](./storage-clients) page for more details.

## Request router

Expand All @@ -312,7 +312,7 @@ The request routing in Crawlee supports:
- Failed request handlers - Handle requests that exceed retry limits.
- Pre-navigation hooks - Execute logic before navigating to URLs.

See the [Request router guide](./request-router) for detailed information and examples.
See the [Request router](./request-router) page for detailed information and examples.

## Service locator

Expand All @@ -324,7 +324,7 @@ The <ApiLink to="class/ServiceLocator">`ServiceLocator`</ApiLink> is a central r

Services can be registered globally through the `service_locator` singleton instance, passed to crawler constructors, or provided when opening individual storage instances. The service locator includes conflict prevention mechanisms to ensure configuration consistency and prevent accidental service conflicts during runtime.

See the [Service locator guide](./service-locator) for detailed information about service registration and configuration options.
See the [Service locator](./service-locator) page for detailed information about service registration and configuration options.

## Request loaders

Expand All @@ -342,7 +342,7 @@ Request loaders provide a subset of <ApiLink to="class/RequestQueue">`RequestQue

Request loaders are useful when you need to start with a predefined set of URLs. The tandem approach allows processing requests from static sources (like files or sitemaps) while maintaining the ability to add new requests dynamically.

See the [Request loaders guide](./request-loaders) for detailed information.
See the [Request loaders](./request-loaders) page for detailed information.

## Event manager

Expand All @@ -356,7 +356,7 @@ Crawlee provides several implementations of the event manager:

:::info

You can learn more about <ApiLink to="class/Snapshotter">`Snapshotter`</ApiLink> and <ApiLink to="class/AutoscaledPool">`AutoscaledPool`</ApiLink> and their configuration in the [Scaling crawlers guide](./scaling-crawlers).
You can learn more about <ApiLink to="class/Snapshotter">`Snapshotter`</ApiLink> and <ApiLink to="class/AutoscaledPool">`AutoscaledPool`</ApiLink> and their configuration on the [Scaling crawlers](./scaling-crawlers) page.

:::

Expand Down Expand Up @@ -412,7 +412,7 @@ The core component of session management in Crawlee is <ApiLink to="class/Sessio

:::info

You can learn more about fingerprints and how to avoid getting blocked in the [Avoid blocking guide](./avoid-blocking).
You can learn more about fingerprints and how to avoid getting blocked in the [Avoid blocking guide](../guides/avoid-blocking).

:::

Expand All @@ -431,7 +431,7 @@ The session pool provides automated session lifecycle management:

The pool operates as an async context manager, automatically initializing with sessions and cleaning up on exit. It ensures proper session management by rotating sessions based on usage count, expiration time, and custom rules while maintaining optimal pool size.

See the [Session management guide](./session-management) for more information.
See the [Session management](./session-management) page for more information.

## Statistics

Expand All @@ -441,8 +441,3 @@ The system includes error tracking through the <ApiLink to="class/ErrorTracker">

Statistics are logged at configurable intervals in both table and inline formats, with final summary data returned from the `crawler.run` method available through <ApiLink to="class/FinalStatistics">`FinalStatistics`</ApiLink>.

## Conclusion

In this guide, we provided a high-level overview of the core components of the Crawlee library and its architecture. We covered the main components like crawlers, crawling contexts, storages, request routers, service locator, request loaders, event manager, session management, and statistics. Check out other guides, the [API reference](https://crawlee.dev/python/api), and [Examples](../examples) for more details on how to use these components in your own projects.

If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee-python) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!
Loading