This is the source code for Marginalia Search.
The aim of the project is to develop new and alternative discovery methods for the Internet. It's an experimental workshop as much as it is a public service, the overarching goal is to elevate the more human, non-commercial sides of the Internet.
A side-goal is to do this without requiring datacenters and enterprise hardware budgets, to be able to run this operation on affordable hardware with minimal operational overhead.
The long term plan is to refine the search engine so that it provide enough public value that the project can be funded through grants, donations and commercial API licenses (non-commercial share-alike is always free).
The system can both be run as a copy of Marginalia Search, or as a white-label search engine for your own data (either crawled or side-loaded). At present the logic isn't very configurable, and a lot of the judgements made are based on the Marginalia project's goals, but additional configurability is being worked on!
Here's a demo of the set-up and operation of the self-hostable barebones mode of the search engine: 🌎 https://www.youtube.com/watch?v=PNwMkenQQ24
To set up a local test environment, follow the instructions in 📄 run/readme.md!
Further documentation is available at 🌎 https://docs.marginalia.nu/.
Before compiling, it's necessary to run ⚙️ run/setup.sh. This will download supplementary model data that is necessary to run the code. These are also necessary to run the tests.
If you wish to hack on the code, check out 📄 doc/ide-configuration.md.
A production-like environment requires a lot of RAM and ideally enterprise SSDs for the index, as well as some additional terabytes of slower harddrives for storing crawl data. It can be made to run on smaller hardware by limiting size of the index.
The system will definitely run on a 32 Gb machine, possibly smaller, but at that size it may not perform very well as it relies on disk caching to be fast.
A local developer's deployment is possible with much smaller hardware (and index size).
📁 code/ - The Source Code. See 📄 code/readme.md for a further breakdown of the structure and architecture.
📁 run/ - Scripts and files used to run the search engine locally
📁 third-party/ - Third party code
📁 doc/ - Supplementary documentation
📄 CONTRIBUTING.md - How to contribute
📄 LICENSE.md - License terms
You can email kontakt@marginalia.nu with any questions or feedback.
The bulk of the project is available with AGPL 3.0, with exceptions. Some parts are co-licensed under MIT, third party code may have different licenses. See the appropriate readme.md / license.md.
Consider donating to the project.
This project was funded through the NGI0 Entrust Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 101069594.
The project received a sponsorship from SerpApi.
SerpApi's Search Index API: Real-time web data for AI agents. SerpApi's Search Index API delivers fresh, structured results in JSON and Markdown, alongside a core API for public search engine results across Google, DuckDuckGo, Bing, YouTube, Amazon, and more.