This is a step-by-step walkthrough of scraping a real e-commerce marketplace. We'll go from a single CLI command to a full dataset of over 12,000 product listings, cleaned and ready for analysis. The entire process takes about 15 minutes, including the crawl time.
Step 1: Reconnaissance
Before writing any code, we need to understand the target. What protection does it use? How is the data structured? Is it server-rendered or a single-page application? IntelliScrape can answer these questions with a single probe request.
The probe tells us everything we need. The site uses Cloudflare protection, curl_cffi (tier 1) handles it without issue, there are 48 products on the first page, and pagination reveals roughly 250 pages total. That gives us an estimate of around 12,000 products.
Step 2: Engine selection
For this target, tier 1 (curl_cffi) is sufficient. The site's Cloudflare protection isn't aggressive enough to require a browser engine. This is ideal because curl_cffi is the fastest engine with the lowest resource usage. Each request takes about 200 milliseconds, compared to 2 to 3 seconds for a browser engine.
Step 3: Crawl configuration
Now we configure the full crawl. We need to set the maximum number of pages, configure rate limiting to be respectful of the server, and choose our export format. IntelliScrape handles pagination automatically, detecting next-page links and following them without manual configuration.
The crawl runs at a rate of about one page every 1.5 seconds, which is respectful and keeps us under any reasonable rate limit. The total crawl time is approximately 6 minutes for 250 pages. Each page is fetched, parsed, and stored in memory as it's scraped.
Step 4: Data structure
IntelliScrape extracts structured data from each product page. For this e-commerce site, we get: product name, price, brand, category, description, image URL, SKU, and availability status. The extraction is automatic because IntelliScrape's parser identifies common product page patterns.
If you need custom fields, you can use CSS selectors or XPath expressions to target specific elements. The --selector flag lets you define exactly what to extract from each page.
Step 5: Data cleaning
The raw scrape gives us structured data, but it needs cleaning. Prices come in various formats: '$29.99', '29.99 USD', '$ 29.99', 'From $29.99'. We need to normalize these to a consistent numeric format. Brand names are sometimes embedded in the title and sometimes in a separate field. Category hierarchies need to be flattened.
IntelliScrape includes built-in cleaning functions for common data types. The clean_prices function normalizes price strings. The extract_brands function pulls brand names from titles. The flatten_categories function converts nested category trees into flat strings.
Step 6: Export and analysis
The final dataset is exported as a Parquet file, which is ideal for analysis with pandas, Polars, or any other data tool. The Parquet format preserves column types, compresses well, and is fast to read. You can also export to CSV, JSON, Excel, or SQLite depending on your workflow.
The entire process from reconnaissance to clean dataset took about 15 minutes. The crawl itself was 6 minutes, data cleaning was near instant, and the rest was configuration and inspection. For larger sites, the crawl time scales linearly with page count, so a 10,000-page site would take roughly 4 hours.
