# IntelliScrape — Full Documentation for LLMs

> Anti-detection web scraping toolkit for 98% of the internet. CLI, Python API, hosted service. LGPL-2.1 licensed.

## Overview

IntelliScrape is a Python scraping toolkit that bypasses Cloudflare, DataDome, Akamai, and PerimeterX anti-bot systems. It ships as a CLI, a Python API, and a hosted FastAPI service. Export to CSV, JSON, Excel, or SQLite.

**Key Facts:**
- Language: Python 3.10+
- Package: `pip install intelliscrape`
- License: LGPL-2.1
- Target coverage: 98% of the internet
- Engine tiers: 4 (curl_cffi, playwright-stealth, nodriver, camoufox)
- Export formats: CSV, JSON, Excel, SQLite
- Repository: https://github.com/GuixJoy/IntelliScrape
- Website: https://intelliscrape.dev
- Hosted API: https://intelliscrape.fastapicloud.dev
- Contact: joy@intelliscrape.dev

## The Problem

Modern anti-bot systems have evolved far beyond simple IP blocking:

1. **Anti-bot systems got smarter.** Cloudflare Turnstile, DataDome, and Akamai fingerprint TLS, canvas, WebGL, and audio — not just IPs. A rotating proxy does nothing against that.

2. **Proxies alone aren't enough.** You need matched browser fingerprints, real cookie state, and human-shaped request patterns. Otherwise you're just paying for blocked traffic.

3. **Most scrapers break on JS-heavy sites.** SPAs, hydration, lazy-loaded data, infinite scroll. Anything with a real runtime needs a real browser — but only when it needs one.

## How It Works — Tiered Engine

IntelliScrape runs a tiered engine. It attempts the cheapest, fastest scraping method first and escalates only when a target pushes back — so 70% of requests never leave tier 1, and hardcases still land.

| Tier | Engine | Description |
|------|--------|-------------|
| 1 | **curl_cffi** | TLS impersonation. Speaks like Chrome/Firefox/Safari. Handles ~70% of targets. Sub-second. |
| 2 | **playwright-stealth** | Headless browser with patched fingerprints. Renders JS-heavy pages. |
| 3 | **nodriver (CDP)** | Raw Chrome DevTools Protocol. No webdriver flags. For DataDome/PerimeterX hardcases. |
| 4 | **camoufox** | Custom Firefox with canvas/WebGL/audio spoofing. Full fingerprint rewrite. Last resort. |

## Capabilities

### Anti-bot Bypass
Cloudflare, DataDome, Akamai, PerimeterX bypass. Canvas / WebGL / audio fingerprint spoofing. TLS impersonation across Chrome, Firefox, Safari, Edge profiles.

```
$ intelliscrape https://target.com
  → protection: cloudflare
  → cleared in 2.1s
```

### Session Management
Auth flows, form submission, search inputs, click paths, human-shaped delays. Sessions survive across pages.

```python
scraper.login(user='me', pw='***')
scraper.search('q=widgets')
```

### Resilience
Cookie persistence, smart retries with fallback engines, request/response interception. Resume where a run left off.

### Network / Proxy Support
IP rotation, residential proxy support, per-domain health checking. Backoff when a target starts throttling — not after.

### Export
Structured extraction from JSON-LD, meta tags, and DOM fallbacks. Export directly to CSV, JSON, Excel, or SQLite.

```python
DataExporter.export(result, format='xlsx', file='out.xlsx')
```

### Whole-Site Crawling
Pagination detection, sitemap parsing, media downloads, per-domain concurrency limits. Point it at a domain and get a dataset.

---

## Three Ways to Use

### 1. CLI

**Installation:**
```bash
pip install intelliscrape
```

**Basic Usage:**
```bash
intelliscrape https://example.com
```

**All Commands:**

| Command | Description |
|---------|-------------|
| `intelliscrape <url>` | Scrape a single URL. Outputs cleaned text to stdout. |
| `intelliscrape <url> --raw` | Output raw HTML instead of cleaned text. |
| `intelliscrape <url> --export csv -o file.csv` | Export directly to CSV file. |
| `intelliscrape <url> --export json -o file.json` | Export directly to JSON file. |
| `intelliscrape <url> --export xlsx -o file.xlsx` | Export directly to Excel file. |
| `intelliscrape <url> --export sqlite -o file.db` | Export directly to SQLite database. |
| `intelliscrape <url> --engine playwright` | Force a specific engine (curl_cffi, playwright, nodriver, camoufox). |
| `intelliscrape <url> --login --username user --password pass` | Authenticate before scraping. |
| `intelliscrape <url> --crawl --max-pages 50` | Crawl a site, following internal links. |
| `intelliscrape <url> --proxy http://proxy:8080` | Route requests through a proxy. |
| `intelliscrape --version` | Show installed version. |

**CLI Examples:**

```bash
# Scrape and export to CSV
intelliscrape https://news.ycombinator.com --export csv -o hn.csv
  → engine: curl_cffi [tier 1/4]
  → cleared in 0.8s
  → hn.csv · 30 rows · 5 fields

# Crawl a documentation site
intelliscrape https://docs.python.org --crawl --max-pages 20 --export json -o docs.json
  → engine: curl_cffi [tier 1/4]
  → crawling 20 pages...
  → docs.json · 20 records

# Force browser engine for protected site
intelliscrape https://protected-site.com --engine playwright --export sqlite -o data.db
  → engine: playwright-stealth [tier 2/4]
  → challenge cleared in 2.3s
  → data.db · 148 records
```

### 2. Python API

**Installation:**
```bash
pip install intelliscrape
```

**Quick Start:**
```python
from intelliscrape import IntelliScrape

scraper = IntelliScrape()
result = scraper.scrape("https://example.com")
print(result.content)
```

**API Reference:**

- **`IntelliScrape()`** — Main scraper class. Manages engine selection, retries, and session state.

- **`scraper.scrape(url, engine?, raw?)`** — Scrape a single URL. Returns a result object with `.content`, `.status_code`, and `.engine_used`.
  - `url` — target URL (required)
  - `engine` — force: `"curl_cffi"` | `"playwright"` | `"nodriver"` | `"camoufox"`
  - `raw` — return raw HTML (default: false)

- **`scraper.crawl(url, max_pages?, follow_links?, delay?)`** — Crawl a site. Follows internal links, respects robots.txt, applies rate limiting.
  - `max_pages` — stop after N pages (default: 50)
  - `follow_links` — follow internal links (default: true)
  - `delay` — seconds between requests (default: 1.0)

- **`scraper.login(username, password)`** — Authenticate on a site. Cookies and session state are preserved for subsequent requests.

- **`DataExporter.export(result, format, file)`** — Export scrape results.

**Full Example:**
```python
from intelliscrape import IntelliScrape, DataExporter

scraper = IntelliScrape()

# Basic scrape — engine auto-selected
result = scraper.scrape("https://protected-site.com")

# Force a specific engine
result = scraper.scrape(
    "https://hardened-target.com",
    engine="camoufox"
)

# Crawl a full site
results = scraper.crawl(
    "https://docs.python.org",
    max_pages=50,
    follow_links=True
)

# Export to any format
DataExporter.export(result, format="csv", file="output.csv")
DataExporter.export(result, format="json", file="output.json")
DataExporter.export(result, format="xlsx", file="output.xlsx")
DataExporter.export(result, format="sqlite", file="output.db")
```

**Proxy Configuration:**
```python
# Per-request proxy
result = scraper.scrape(
    "https://target.com",
    proxy="http://user:pass@proxy:8080"
)

# Residential proxies
result = scraper.scrape(
    "https://target.com",
    proxy="socks5://user:pass@residential:1080"
)
```

### 3. Hosted Service (Private Beta)

A FastAPI backend and a web UI, without running anything yourself. For teams that don't want to manage proxies, browsers, or fingerprint rotation.

**REST API:**

Base URL: `https://intelliscrape.fastapicloud.dev`

**POST /scrape** — Scrape a URL and return cleaned text.
- Body: `{"url": "https://example.com", "raw": false}`
- Response: `{"url": "...", "content": "...", "success": true}`
- Errors: 400 (invalid URL), 422 (missing url field)

**GET /health** — Health check.
- Response: `{"status": "ok", "version": "2.2.0"}`

**Example:**
```bash
curl -X POST https://intelliscrape.fastapicloud.dev/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

**Specs:**
- Managed proxies — residential pool included
- Concurrency — up to 200 parallel jobs
- Storage — 7-day result retention
- Dashboard — web UI for managing jobs

**Contact:** joy@intelliscrape.dev

---

## Web Interface

The fastest way to try IntelliScrape. Go to intelliscrape.dev, paste a URL into the scrape input on the homepage, and hit Enter.

**How it works:**
1. Enter a URL — Paste any publicly accessible URL into the input field.
2. Submit — Press Enter or click the submit button.
3. View results — The cleaned text content appears in the result panel.
4. Copy or expand — Use the copy button to grab the full content.

**Limitations (what the web interface cannot do):**
- Bypass anti-bot protection (no Cloudflare/DataDome/Akamai/PerimeterX bypass)
- Render JavaScript (SPAs, lazy-loaded content, JS-heavy pages)
- Scrape e-commerce properly (product listings, prices, variant data)
- Log in or maintain sessions
- Crawl multiple pages
- Export to CSV/Excel/SQLite (text only)
- Rotate proxies or IPs

For these capabilities, install the Python library.

---

## Export Formats

| Format | Description |
|--------|-------------|
| **CSV** | Typed columns, UTF-8 encoded |
| **JSON** | Line-delimited or nested objects |
| **Excel** | Multi-sheet, formatted (.xlsx) |
| **SQLite** | Indexed, queryable database |

---

## Pricing

### Open Source (Library)
- Price: $0 forever
- License: LGPL-2.1
- Full CLI & Python API
- All 4 engine tiers
- Cloudflare / DataDome / Akamai / PerimeterX bypass
- Exports: CSV, JSON, Excel, SQLite
- Bring your own proxies
- Community support on GitHub

### Hosted Service
- Price: TBD (private beta)
- Everything in Open Source
- Managed browser fleet, zero setup
- Residential proxy pool included
- REST API + web dashboard
- Up to 200 concurrent jobs
- Priority support, SLA

---

## Tech Stack

### Library
- Python 3.10+
- curl_cffi (TLS impersonation)
- playwright + playwright-stealth (headless browser)
- nodriver (Chrome DevTools Protocol)
- camoufox (custom Firefox)

### Backend (Hosted Service)
- FastAPI on FastAPI Cloud
- Neon PostgreSQL (operational metadata)
- Deployed from: https://github.com/GuixJoy/IntelliScrape

### Website
- TanStack Start (React SSR)
- TypeScript
- Tailwind CSS
- GSAP (animations)
- Framer Motion (hero background)
- Deployed on Vercel from: https://github.com/GuixJoy/intelliscrape-web-scraper

### Infrastructure
- Vercel — website hosting
- FastAPI Cloud — API hosting
- Neon — PostgreSQL database
- fingerprint-oss — browser fingerprint capture

---

## Privacy & Data Handling

### Open-Source Library
The IntelliScrape Python library collects no data. It runs entirely on your machine. No telemetry, no phone-home, no analytics. When you scrape a website with the library, the only entities that see that traffic are you, your network, and the target site.

### Website
This website (intelliscrape.dev) is hosted on Vercel. Vercel may collect basic access logs (IP, user agent, timestamp) for operational purposes. No cookies, no analytics trackers, no advertising pixels.

### Hosted Service
- Scrape results retained for up to 7 days, then deleted.
- No reading, analyzing, or selling of scrape results.
- Request metadata (URL, timestamp, status) logged for monitoring and abuse prevention.
- Fingerprint data used solely for anti-detection, not persisted beyond the session.

### No Third Parties
No data shared with third parties. Only external services are infrastructure providers (Vercel, Neon, FastAPI Cloud).

---

## Responsible Use

IntelliScrape is for accessing publicly available data. Users should:

1. **Respect robots.txt** — Always check and honor robots.txt directives.
2. **Rate-limit requests** — Don't hammer servers. Use reasonable delays.
3. **Know the law** — Regulations vary by jurisdiction. Understand local laws.
4. **Minimize data collection** — Only scrape what you actually need.

**Disclaimer:** IntelliScrape is provided "as is" without warranty. Users are solely responsible for ensuring their scraping activities comply with applicable terms of service and local laws.

---

## Blog Posts (SEO Content)

1. **Beating Cloudflare Turnstile in 2026** — What Turnstile checks, why Python scrapers fail, tiered engine approach. https://intelliscrape.dev/blog/cloudflare-turnstile-2026

2. **Why Your Python Scraper Gets Blocked Before Byte One** — JA3/JA4 fingerprinting, TLS handshake, curl_cffi solution. https://intelliscrape.dev/blog/tls-impersonation-notes

3. **Scraping an Entire E-Commerce Site in 15 Minutes** — Full walkthrough from reconnaissance to data analysis. https://intelliscrape.dev/blog/scraping-a-marketplace

4. **IntelliScrape 0.9 Release Notes** — Tiered engine, camoufox, open source commitment. https://intelliscrape.dev/blog/release-0-9

5. **Why Bot Detection Measures Your Mouse Movements** — Behavioral analysis, what signals matter, future of detection. https://intelliscrape.dev/blog/human-shaped-behavior

6. **Why We Made IntelliScrape Open Source Under LGPL 2.1** — Open source philosophy, LGPL vs GPL, business model. https://intelliscrape.dev/blog/why-open-source-lgpl

7. **How We Use Neon DB to Log Scrape Operations** — Neon PostgreSQL choice, what gets logged, privacy. https://intelliscrape.dev/blog/neon-db-logging

8. **How fingerprint-oss Captures Your Browser Identity** — Browser fingerprinting explained, fingerprint-oss internals. https://intelliscrape.dev/blog/fingerprint-oss-browser

---

## Author & Contact

- **Author:** Joy (GuixJoy)
- **Email:** joy@intelliscrape.dev
- **GitHub:** https://github.com/GuixJoy
- **Repository:** https://github.com/GuixJoy/IntelliScrape
- **Issues:** https://github.com/GuixJoy/IntelliScrape/issues

## License

LGPL-2.1 — free for commercial and personal use. See https://github.com/GuixJoy/IntelliScrape/blob/main/LICENSE