shiyong8101 / aifs

Local semantic search. Stupidly simple.

AI Filesystem

Local semantic search over folders. Why didn't this exist?

pip install aifs

from aifs import search

search("How does AI Filesystem work?", path="/path/to/folder")
search("It's not unlike how Spotlight works.") # Path defaults to CWD

How it works

Running aifs.search will chunk and embed all nested supported files (.txt, .docx, .pptx, .jpg, .png, .eml, .html, and .pdf) in path. It will then store these embeddings into an _.aifs file in path.

By storing the index, you only have to chunk/embed once. This makes semantic search very fast after the first time you search a path.

If a file has changed or been added, aifs.search will update or add those chunks. We still need to handle file deletions (we welcome PRs).

In detail:

1. If a folder hasn't been indexed, we first use unstructured to parse and chunk every file in the path.
2. Then we use chroma to embed the chunks locally and save them to a _.aifs file in path.
3. Finally, chroma is used again to semantically search the embeddings.

If an _.aifs file is found in a directory, it uses that instead of indexing it again. If some files have been updated, it will re-index those.

Why?

We built this to let open-interpreter quickly semantically search files/folders.