When you run a scrape operation with IntelliScrape, something happens behind the scenes that most users never see. Every operation is logged to a database. This isn't telemetry in the traditional sense. It's operational logging that helps us understand how the tool is used, debug issues, and improve performance. In this post, we'll explain why we log operations, what we log, and how Neon DB makes it work.
Why operational logging matters
Scraping is inherently unpredictable. Targets change their protection, page structures evolve, and anti bot systems update their detection rules. Without operational logging, we'd have no way to know when something breaks or why. When a user reports that a target that worked last week is now blocked, the logs tell us what changed.
Operational logging also helps with performance optimization. By analyzing aggregate data on engine selection, response times, and success rates, we can tune the tier escalation logic. If tier 1 (curl_cffi) handles 85% of targets successfully, and tier 2 handles 12%, we know where to focus optimization efforts.
Most importantly, operational logging creates accountability. Every scrape request, every engine selection, every success and failure is recorded. This means we can audit the tool's behavior and ensure it's being used responsibly.
Neon PostgreSQL setup
We chose Neon DB for the logging infrastructure because it's a serverless PostgreSQL service that scales automatically. Traditional PostgreSQL requires managing servers, storage, and backups. Neon handles all of that. The database scales to zero when there's no traffic, which keeps costs low during quiet periods.
Neon's architecture separates compute from storage, which means we can have a small compute instance handling low traffic and scale up instantly when demand spikes. The storage layer handles durability and replication automatically. We don't need to worry about disk space, backups, or replication lag.
What gets logged
Each log entry captures the essential details of a scrape operation. The target URL tells us what site was accessed. The engine used tells us which tier handled the request. The protection detected field records whether Cloudflare, Akamai, PerimeterX, or another anti bot system was identified. The status field records success, failure, or escalation. The duration tells us how long the operation took.
We intentionally do not log the scraped content. The actual HTML, JSON, or data extracted from target sites is never stored in the log database. We log metadata about the operation, not the operation's results. This is a deliberate privacy decision.
Privacy considerations
Privacy was a primary design constraint for the logging system. Many scraping tools collect detailed telemetry that can be used to profile users. We took a different approach: log only what's necessary for debugging and optimization, and never log content or personally identifiable information.
User identifiers are hashed before logging. Target URLs are stored but not associated with specific users in the analytics pipeline. The logging system has no access to scraped content. Even if the database were compromised, the attacker would get operational metadata, not user data or scraped content.
We also provide a --no-log flag that disables operational logging entirely. If you're scraping sensitive targets or have strict privacy requirements, you can run IntelliScrape without any data being sent to the logging system. The tool functions identically with or without logging.
How Neon handles the workload
Neon DB handles the logging workload well because it's mostly append-only. We write log entries but rarely update them. This is an ideal workload for PostgreSQL's write-ahead log. Neon's storage layer optimizes for this pattern automatically.
The analytics queries run against read replicas, so they don't affect write performance. When we need to analyze engine selection patterns or success rates, we query a replica that's been updated a few seconds behind the primary. This gives us real-time analytics without impacting the logging write path.
Future plans for analytics
We're building a dashboard that will show aggregate analytics for the IntelliScrape community. Engine success rates by target category, average response times, and protection detection accuracy. The dashboard will use anonymized, aggregated data. No individual scrape operations will be visible.
We're also working on a feature that will let users see their own scrape logs through the CLI. This will help users track their usage, debug failed scrapes, and optimize their crawl configurations. The user's own logs are always available to them, which aligns with the open source philosophy of transparency.
