documentation

IntelliScrape Docs

Three ways to use IntelliScrape: the web interface for quick scrapes, the Python library for full power, and the CLI for terminal workflows.

on this page

1. Website

Using the Web Interface How to Scrape JSON & Downloadable Files REST API Limitations

2. Python Library

Installation Quick Start Python API Reference Engine Tiers Export Formats Proxy Configuration

3. CLI

CLI Installation Basic Usage All Commands CLI Examples

section 1 — website

Using the Web Interface

The fastest way to try IntelliScrape. Go to intelliscrape.dev, paste a URL into the scrape input on the homepage, and hit Enter. The cleaned content appears in the result panel on the right.

how to scrape

Step-by-step

Enter a URL

Paste any publicly accessible URL into the input field on the homepage.

Submit

Press Enter or click the submit button. The request is sent to the hosted API.

View results

The cleaned text content appears in the result panel. Page title and meta description are extracted automatically.

Copy or expand

Use the copy button to grab the full content. Use the expand button to view it in full screen.

json & downloadable files

Getting JSON & Downloadable Files

The web interface returns cleaned text by default. For JSON, raw HTML, or downloadable files, use the REST API directly.

get json

The response is always JSON with url, content, and success fields.

curl -X POST https://intelliscrape.fastapicloud.dev/scrape \
  -H "Content-Type: application/json" \
  -d '{"{"}"url": "https://example.com"{"}"}'

get raw html

Pass "raw": true to get the full HTML instead of cleaned text.

curl -X POST https://intelliscrape.fastapicloud.dev/scrape \
  -H "Content-Type: application/json" \
  -d '{"{"}"url": "https://example.com", "raw": true{"}"}'

download as file

Pipe the JSON response to a file and extract content with jq.

curl -s -X POST https://intelliscrape.fastapicloud.dev/scrape \
  -H "Content-Type: application/json" \
  -d '{"{"}"url": "https://example.com"{"}"}' \
  | jq -r '.content' > output.txt

render javascript

Pass "render": true for JavaScript-heavy sites (React, Angular, Vue SPAs). Uses Playwright headless browser.

curl -X POST https://intelliscrape.fastapicloud.dev/scrape \
  -H "Content-Type: application/json" \
  -d '{"{"}"url": "https://spa-site.com", "render": true{"}"}'

need csv, excel, or sqlite?

The hosted API returns text only. For structured export to CSV, JSON (line-delimited), Excel, or SQLite, install the Python library. It includes a DataExporter that handles format conversion.

rest api

REST API

Base URL:

https://intelliscrape.fastapicloud.dev

POST/scrape

Scrape a URL and return cleaned text. Uses curl_cffi with TLS impersonation by default. Set render=true for JavaScript-heavy sites (SPAs, React, Angular, Vue).

parameters

urlstring (required)

Target URL. Must be a valid HTTP/HTTPS URL.

rawboolean (optional)

Return raw HTML instead of cleaned text. Default: false.

renderboolean (optional)

Enable JavaScript rendering via Playwright headless browser. Use for SPAs and JS-heavy sites. Default: false.

response

{"{"}
  "url": "https://example.com",
  "content": "Title: Example...\n\nBody text...",
  "success": true
{"}"}

errors

400Invalid URL, connection failed, or timeout.

422Missing url field in request body.

GET/health

Health check. Returns server status and version.

response

{"{"} "status": "ok", "version": "2.2.0" {"}"}

limitations

Web Interface Limitations

The web interface is designed for quick, lightweight scraping. It cannot:

high

Bypass anti-bot protection

No Cloudflare, DataDome, Akamai, or PerimeterX bypass. Uses plain HTTP requests without fingerprint spoofing.

high

Render JavaScript

SPAs, lazy-loaded content, and JS-heavy pages will return incomplete results. Use the render=true parameter in the API for JS rendering.

medium

Scrape e-commerce properly

Product listings, prices, and variant data often require JS rendering and anti-bot bypass. For e-commerce, install the library.

medium

No cookie persistence, form submission, or authenticated access.

medium

Crawl multiple pages

Single URL only. No link following, pagination, or sitemap crawling.

low

Export to CSV/Excel/SQLite

Text output only. Use the Python library for structured export.

low

Rotate proxies or IPs

All requests come from the hosted server IP.

scraping e-commerce or protected sites?

The web interface won't cut it. Install the Python library — it has a 4-tier engine that escalates from fast HTTP requests to full browser automation, handling anti-bot challenges, JavaScript rendering, and structured data extraction.

section 2 — python library

Installation

Requires Python 3.10+. Install from PyPI:

$ pip install intelliscrape

This installs the CLI, Python API, and all engine dependencies (curl_cffi, playwright-stealth, nodriver, camoufox).

quick start

Quick Start

from intelliscrape import IntelliScrape

scraper = IntelliScrape()

result = scraper.scrape("https://example.com")

print(result.content)

The engine is auto-selected. It starts with the fastest method (curl_cffi) and escalates only when the target pushes back.

python api reference

Python API Reference

IntelliScrape()

Main scraper class. Manages engine selection, retries, and session state.

scraper.scrape(url, engine?, raw?)

Scrape a single URL. Returns a result object with .content, .status_code, and .engine_used.

url — target URL (required)

engine — force: "curl_cffi" | "playwright" | "nodriver" | "camoufox"

raw — return raw HTML (default: false)

scraper.crawl(url, max_pages?, follow_links?, delay?)

Crawl a site. Follows internal links, respects robots.txt, applies rate limiting.

max_pages — stop after N pages (default: 50)

follow_links — follow internal links (default: true)

delay — seconds between requests (default: 1.0)

scraper.login(username, password)

Authenticate on a site. Cookies and session state are preserved for subsequent requests.

DataExporter.export(result, format, file)

Export scrape results. See Export Formats below.

engine tiers

Engine Tiers

IntelliScrape escalates through 4 tiers. Each request starts at tier 1 and moves up only when the target blocks it.

TIER 1

curl_cffi

TLS impersonation. Speaks like Chrome/Firefox/Safari. Handles ~70% of targets. Sub-second.

TIER 2

playwright-stealth

Headless browser with patched fingerprints. Renders JS-heavy pages.

TIER 3

nodriver (CDP)

Raw Chrome DevTools Protocol. No webdriver flags. For DataDome/PerimeterX hardcases.

TIER 4

camoufox

Custom Firefox with canvas/WebGL/audio spoofing. Full fingerprint rewrite. Last resort.

export formats

Export Formats

Use DataExporter.export() to save results in any format:

from intelliscrape import DataExporter

# CSV

DataExporter.export(result, format="csv", file="output.csv")

# JSON

DataExporter.export(result, format="json", file="output.json")

# Excel

DataExporter.export(result, format="xlsx", file="output.xlsx")

# SQLite

DataExporter.export(result, format="sqlite", file="output.db")

CSV

typed columns, UTF-8

JSON

line-delimited or nested

Excel

multi-sheet, formatted

SQLite

indexed, queryable

proxy configuration

Proxy Configuration

IntelliScrape supports HTTP, HTTPS, and SOCKS5 proxies. Pass them per-request:

# Per-request proxy

result = scraper.scrape(

"https://target.com",

proxy="http://user:pass@proxy:8080"

)

# Residential proxies

result = scraper.scrape(

"https://target.com",

proxy="socks5://user:pass@residential:1080"

)

section 3 — cli

CLI Reference

The CLI is installed automatically with the Python library.

$ pip install intelliscrape

basic usage

Basic Usage

$ intelliscrape https://example.com

// outputs cleaned text to stdout

all commands

All Commands

$ intelliscrape <url>

Scrape a single URL. Outputs cleaned text to stdout.

$ intelliscrape <url> --raw

Output raw HTML instead of cleaned text.

$ intelliscrape <url> --export csv -o file.csv

Export directly to CSV file.

$ intelliscrape <url> --export json -o file.json

Export directly to JSON file.

$ intelliscrape <url> --export xlsx -o file.xlsx

Export directly to Excel file.

$ intelliscrape <url> --export sqlite -o file.db

Export directly to SQLite database.

$ intelliscrape <url> --engine playwright

Force a specific engine (curl_cffi, playwright, nodriver, camoufox).

$ intelliscrape <url> --login --username user --password pass

Authenticate before scraping.

$ intelliscrape <url> --crawl --max-pages 50

Crawl a site, following internal links.

$ intelliscrape <url> --proxy http://proxy:8080

Route requests through a proxy.

$ intelliscrape --version

Show installed version.

cli examples

CLI Examples

scrape and export to csv

$ intelliscrape https://news.ycombinator.com --export csv -o hn.csv
  → engine: curl_cffi [tier 1/4]
  → cleared in 0.8s
  → hn.csv · 30 rows · 5 fields

crawl a documentation site

$ intelliscrape https://docs.python.org --crawl --max-pages 20 --export json -o docs.json
  → engine: curl_cffi [tier 1/4]
  → crawling 20 pages...
  → docs.json · 20 records

force browser engine for protected site

$ intelliscrape https://protected-site.com --engine playwright --export sqlite -o data.db
  → engine: playwright-stealth [tier 2/4]
  → challenge cleared in 2.3s
  → data.db · 148 records

← home view repo →