lens

🔍 Lens

Search engine for the distributed web

Lens is an opt-in search engine and data collection tool to aid content discovery of the distributed web. It exposes a simple, minimal API for intelligently indexing and querying content on IPFS.

Features and Usage

Initially integrated with Temporal, Lens will allow users to optionally have the data they upload be searched and indexed and be awarded with RTC for participating in the data collection process. Users can then search for content using a simple-to-use API.

Searching through Lens will be facilitated through Temporal web. Optionally, we will have a service independent from Temporal which users can submit content to have it be indexed. This however, is not compensated with RTC. In order to receive the RTC, you must participate through Lens indexing within the Temporal web interface.

API

Lens exposes a simple API via gRPC. The definitions are in RTradeLtd/grpc.

The Lens API, summarized, currently consists of three core RPCs:

service LensV2 {
  rpc Index(IndexReq)   returns (IndexResp)  {}
  rpc Search(SearchReq) returns (SearchResp) {}
  rpc Remove(RemoveReq) returns (RemoveResp) {}
}

Golang bindings for the Lens API can be found in RTradeLtd/grpc.

Supported Formats

Only IPFS CIDs are supported, and they must be plaintext files. We attempt to determine the content type via mime type sniffing, and use that to determine whether or not we can analyze the content.

Please see the following table for supported content types that we can index. Note if the type is listed as <type>/* it means that any "sub type" of that mime type is supported.

Deployment

The recommended way to deploy a Lens instance is via the rtradetech/lens Docker image.

$> docker pull rtradetech/lens:latest

A docker-compose configuration is available that also starts up other prerequisites:

$> wget -O lens.yml https://raw.githubusercontent.com/RTradeLtd/Lens/master/lens.yml
$> LENS=latest BASE=/my/dir docker-compose -f lens.yml up

Development

This project requires:

• Go 1.11+
• dep
• Tesseract
• Tensorflow
• go-fitz

To fetch the codebase, use go get:

$> go get github.com/RTradeLtd/Lens

A rudimentary Makefile target make dep is available for installing the required dependencies.