# IntelliScrape

> Anti-detection web scraping toolkit for 98% of the internet. CLI, Python API, and hosted service.

IntelliScrape is a Python scraping toolkit that bypasses Cloudflare, DataDome, Akamai, and PerimeterX anti-bot systems. It ships as a CLI, a Python API, and a hosted FastAPI service. Export to CSV, JSON, Excel, or SQLite.

## Links

- Website: https://intelliscrape.dev
- GitHub: https://github.com/GuixJoy/IntelliScrape
- PyPI: https://pypi.org/project/intelliscrape/
- License: LGPL-2.1
- Contact: joy@intelliscrape.dev
- Hosted API: https://intelliscrape.fastapicloud.dev
- Blog: https://intelliscrape.dev/blog
- Docs: https://intelliscrape.dev/docs
- Privacy Policy: https://intelliscrape.dev/privacy

## Quick Start

```bash
pip install intelliscrape
intelliscrape https://example.com
```

```python
from intelliscrape import IntelliScrape

scraper = IntelliScrape()
result = scraper.scrape("https://example.com")
print(result.content)
```

## Engine Tiers

IntelliScrape uses a 4-tier engine that starts fast and escalates only when blocked:

1. **curl_cffi** — TLS impersonation (Chrome/Firefox/Safari profiles). Handles ~70% of targets. Sub-second.
2. **playwright-stealth** — Headless browser with patched fingerprints. Renders JS-heavy pages.
3. **nodriver (CDP)** — Raw Chrome DevTools Protocol. No webdriver flags. For DataDome/PerimeterX.
4. **camoufox** — Custom Firefox with canvas/WebGL/audio spoofing. Full fingerprint rewrite. Last resort.

## Capabilities

- Anti-bot bypass: Cloudflare, DataDome, Akamai, PerimeterX
- Browser fingerprint spoofing: canvas, WebGL, audio, TLS
- Session management: login, cookies, form submission
- Whole-site crawling: pagination, sitemap, concurrency limits
- Export: CSV, JSON (line-delimited), Excel, SQLite
- Proxy support: HTTP, HTTPS, SOCKS5 (bring your own or managed residential pool)

## Three Ways to Use

### CLI
```bash
intelliscrape https://example.com --export csv -o data.csv
intelliscrape https://protected-site.com --engine camoufox
intelliscrape https://docs.python.org --crawl --max-pages 50
```

### Python API
```python
from intelliscrape import IntelliScrape, DataExporter

scraper = IntelliScrape()
result = scraper.scrape("https://target.com", engine="camoufox")
DataExporter.export(result, format="csv", file="output.csv")
```

### Hosted Service (Private Beta)
```bash
curl -X POST https://intelliscrape.fastapicloud.dev/scrape \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

## REST API

**Base URL:** `https://intelliscrape.fastapicloud.dev`

- `POST /scrape` — Scrape a URL. Body: `{"url": "...", "raw": false}`. Returns `{url, content, success}`.
- `GET /health` — Health check. Returns `{status, version}`.

## Pricing

- **Open Source (Library):** Free forever. LGPL-2.1. Self-hosted. Full CLI and Python API.
- **Hosted Service:** TBD. Private beta. Managed browser fleet, residential proxies, REST API, up to 200 concurrent jobs.

## Tech Stack

- **Library:** Python 3.10+, curl_cffi, playwright, nodriver, camoufox
- **Backend:** FastAPI on FastAPI Cloud, Neon PostgreSQL
- **Website:** TanStack Start (React SSR), TypeScript, Tailwind CSS, GSAP, Framer Motion
- **Hosting:** Vercel (website), FastAPI Cloud (API), Neon (database)
- **Fingerprinting:** fingerprint-oss (browser identity capture)

## Author

- Name: Joy (GuixJoy)
- Email: joy@intelliscrape.dev
- GitHub: https://github.com/GuixJoy

## License

LGPL-2.1 — free for commercial and personal use. See https://github.com/GuixJoy/IntelliScrape/blob/main/LICENSE

## Responsible Use

IntelliScrape is for accessing publicly available data. Users should:
- Respect robots.txt
- Rate-limit requests
- Understand local laws
- Minimize data collection

## Blog Posts

- Beating Cloudflare Turnstile in 2026: https://intelliscrape.dev/blog/cloudflare-turnstile-2026
- Why Your Python Scraper Gets Blocked Before Byte One: https://intelliscrape.dev/blog/tls-impersonation-notes
- Scraping an Entire E-Commerce Site in 15 Minutes: https://intelliscrape.dev/blog/scraping-a-marketplace
- IntelliScrape 0.9 Release Notes: https://intelliscrape.dev/blog/release-0-9
- Why Bot Detection Measures Your Mouse Movements: https://intelliscrape.dev/blog/human-shaped-behavior
- Why We Made IntelliScrape Open Source Under LGPL 2.1: https://intelliscrape.dev/blog/why-open-source-lgpl
- How We Use Neon DB to Log Scrape Operations: https://intelliscrape.dev/blog/neon-db-logging
- How fingerprint-oss Captures Your Browser Identity: https://intelliscrape.dev/blog/fingerprint-oss-browser