YaCy
| Company type | Open-source software project |
|---|---|
| Industry | Internet, Search engine |
| Founded | 2003 |
| Founder | Michael Christen |
Area served | Worldwide |
| Products | Distributed search engine software |
| Owner | Community-driven / Open-source |
| Website | yacy |
YaCy (pronounced “ya see”) is an open-source distributed search engine that utilizes peer-to-peer (P2P) technology, initially developed by Michael Christen in 2003. Implemented in Java, YaCy runs across hundreds of computers, referred to as YaCy-peers, as of June 2024.
Each YaCy-peer independently performs web crawling, indexes the gathered pages, and shares this information within a decentralized framework, thereby removing reliance on any central server and improving user privacy. In contrast to partially distributed search engines, YaCy’s architecture is fully decentralized, treating all peers with equal status. It can operate either in crawling mode or as a local proxy server that indexes the web pages accessed by the user. Search requests are processed through a local web server, providing results akin to conventional search engines while safeguarding privacy.
System components
The YaCy search engine is composed of four fundamental parts:
- Crawler
- A web robot that navigates through websites, retrieving and analyzing content. Each peer in the YaCy network can perform crawling and indexing through:
- Discovery: Identifying new pages via hyperlinks.
- Fetching: Downloading the actual page data.
- Parsing: Extracting text, metadata, and links from the content.
- Indexer
- Builds a reverse word index (RWI), linking each term with associated URLs and ranking metrics, storing words as hashed values.
- Search and administration interface
- A local HTTP servlet offers a web-based interface for managing searches and configuring settings.
- Data storage
- Houses the reverse word index database using a distributed hash table system.
Search-engine technology
- YaCy functions as a complete search solution including user interface, indexing capabilities, administration tools, and monitoring features.
- It collects web pages through its crawler; documents are parsed and indexed on the local machine. When connected to the peer network, these local indexes combine into a global shared index.
- Search queries access both local indexes and those distributed across peers within the YaCy network.
- The YaCy Grid represents the second-generation P2P search system, consisting of microservices communicating via the Master Connect Program (MCP).
- The YaCy Parser microservice is deployable through Docker containers and connects by default to an MCP instance running on localhost or another specified address.
YaCy platform architecture
YaCy integrates various networking methods, administrative controls, and indexing strategies including blacklisting, moderation, and community interaction. Its principal operational modules include:
- Community: Web forums, usage statistics, XML API
- Maintenance: Web server operations, indexing processes, crawler with load balancing, peer-to-peer communications
- Content management: Blacklist and filter mechanisms, search interface features, bookmark management, monitoring of search outcomes
Distribution
YaCy is available in packaged formats for Linux, Windows, macOS, as well as Docker images. It can also be installed on alternative systems through manual compilation or tarball extraction. Java 11 is a prerequisite with Temurin 11 recommended.
A Debian package is provided via a project-maintained repository but has not yet been incorporated into the official Debian package archives.
See also
- Dooble – an open-source web browser incorporating the YaCy Search Engine Tool
- List of search engines
- Comparison of search engines
- Seeks
Further reading
Overview of YaCy on LinuxReviews