While semantic splitting is extremely valuable, in certain frameworks like Langchan, llama index, and docling you can often find yourself having the opposite problem when parsing markdown or html that your element parser has produced lots of very small elements and you need to figure out a merging strategy.
Often times this merging strategy is based on header level or some other heuristic.
Once it has been determined that chunks should be merged, strategies like inlining headers and merging metadata can be employed without fear of the inlined header ending up in the wrong chunk.
It would be really great if semchunk offered some helpers to help determine if merging two or more chunks is a good idea
While semantic splitting is extremely valuable, in certain frameworks like Langchan, llama index, and docling you can often find yourself having the opposite problem when parsing markdown or html that your element parser has produced lots of very small elements and you need to figure out a merging strategy.
Often times this merging strategy is based on header level or some other heuristic.
Once it has been determined that chunks should be merged, strategies like inlining headers and merging metadata can be employed without fear of the inlined header ending up in the wrong chunk.
It would be really great if semchunk offered some helpers to help determine if merging two or more chunks is a good idea