Markdown `AdapterStep` for chunking by heading
olirice opened this issue · comments
Context
Markdown is a common format for documents ingested into vector systems and has more exploitable structure than simple text.
This task is to create an vecs.adapters.base.AdapterStep that handles chunking markdown by heading.
Ideally it would also accept parameters for
- The maximum number of words in each chunk
e.g.
from vecs.adapters import MarkdownChunker
MarkdownChunker(
max_tokens=512
)
Will give this a go if wanted
According to the Markdown Guide
Markdown applications don’t agree on how to handle a missing space between the number signs (#) and the heading name. For compatibility, always put a space between the number signs and the heading name.
Should we count headings without this space as valid?
###like thisMy vote would be to require the space
you could reference https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/markdown_header_metadata for ideas