When we started building IntelliScrape, we made a decision early: it would be open source. Not just source-available, not just open-core, but truly open source under a recognized license. We chose LGPL-2.1, and in this post we'll explain why that matters, what LGPL-2.1 actually means, and how the community benefits.
Why open source matters for scraping tools
Web scraping tools interact with websites in ways that can be misused. When a scraping tool is closed source, users have no way to verify what it actually does. Does it send telemetry? Does it modify the sites it scrapes? Does it include backdoors? These are not hypothetical concerns. Closed source scraping tools have been caught collecting user data and injecting ads into scraped content.
Open source solves this problem through transparency. Anyone can audit the code, verify the behavior, and contribute improvements. When IntelliScrape makes a request to a target site, you can verify exactly what headers are sent, what data is collected, and what happens to the results. There's no black box.
The scraping community specifically benefits from open source because scraping is inherently adversarial. Anti-bot systems evolve, and scrapers need to evolve in response. An open source tool can be forked, patched, and improved by anyone who encounters a new anti bot technique. A closed source tool depends entirely on the vendor's priorities and timeline.
Why LGPL-2.1 and not GPL
We considered several licenses. GPL would require any derivative work to also be GPL, which would limit adoption by companies with proprietary codebases. MIT would allow anyone to close source their modifications, which would fragment the community. LGPL-2.1 strikes the right balance.
LGPL-2.1 (GNU Lesser General Public License) means you can use IntelliScrape as a library in your own projects without open sourcing your code. If you modify IntelliScrape itself, those modifications must be shared under LGPL-2.1. But if you just use IntelliScrape as a dependency, your project can be under any license.
This is the key difference from GPL. Under GPL, using the library in your project would force your entire project to be GPL. Under LGPL-2.1, only modifications to the library itself need to be shared. Your application code, your business logic, your proprietary algorithms remain yours.
What's included under LGPL-2.1
The core scraping engine, all anti detection techniques, the CLI, the Python SDK, and the documentation are all under LGPL-2.1. You can use, modify, and distribute these components freely, as long as you follow the license terms.
The hosted service (the cloud API and web dashboard) is not open source. It's a proprietary service that uses the open source engine. You don't need the hosted service to use IntelliScrape. The open source version is fully functional on its own. The hosted service is for users who want managed infrastructure and don't want to run their own browser fleet.
Community benefits
Open source creates a feedback loop that benefits everyone. When a user encounters a new anti bot technique, they can contribute a patch to handle it. When a developer finds a performance issue, they can submit an optimization. When a security researcher discovers a vulnerability, they can report it responsibly because they have full access to the code.
The IntelliScrape community has already contributed significant improvements. The curl_cffi integration was contributed by a community member who needed better TLS impersonation. The Camoufox integration was developed in collaboration with the Camoufox project. These contributions wouldn't have been possible with a closed source tool.
How to contribute
Contributions are welcome. The development happens on GitHub, with issues for bug reports and feature requests. Pull requests are reviewed and merged by the core team. The contributing guide explains the code style, testing requirements, and review process.
You don't need to be a core developer to contribute. Documentation improvements, bug reports, and feature suggestions are all valuable. The best way to start is to use IntelliScrape, report any issues you encounter, and share your use cases. Understanding how people use the tool helps us prioritize development.
