agarwal-nitesh / file-content-search

Ingest and Search s3 file contents with elasticsearch client 8.9.1

Remote Storage Files - Meta/Content Search

External Dependencies

1. Postgres
2. ElasticSearch
3. Kafka

External dependency documentation

1. Postgres - Is to store the file url and last processed time (CDC)
2. ElasticSearch:
   • For maintaining inverted indices of file content tokens (single term)
   • How does it work?
     • Inverted index datastructure - inverted-index
     • Lucene the underlying component of ES uses a term dictionary to map a term to the document iterator
     • Lucene index file format
3. Kafka:
   • When file processing of a s3 or any source is triggered, all files from the source is read and is produced.
   • The file url is the key (so that it is consumed by a specific partition) Maintain Strong Ordering Guarantees

Swagger doc

http://localhost:8085/fs/swagger-ui/index.html

Architecture

PRD

PRD Notion
PRD MD