Skip to content

feat: add support for parsing and indexing plain text (.txt) files - #483

Open
denis-samatov wants to merge 1 commit into
VectifyAI:mainfrom
denis-samatov:feat/support-txt-files
Open

feat: add support for parsing and indexing plain text (.txt) files#483
denis-samatov wants to merge 1 commit into
VectifyAI:mainfrom
denis-samatov:feat/support-txt-files

Conversation

@denis-samatov

Copy link
Copy Markdown

Summary

  • Adds native support for parsing and indexing plain text (.txt) documents into PageIndex hierarchical tree structures.
  • Resolves Add support for parsing .txt files聽#222: "Add support for parsing .txt files".
  • Provides CLI integration via --txt_path in run_pageindex.py alongside --pdf_path and --md_path.

Changes

  1. pageindex/page_index_txt.py:
    • extract_nodes_from_txt: extracts document structure from plain text by identifying:
      • Chapter/Section/Part/Appendix labels (e.g. Chapter 1: Intro, Section 2)
      • Numbered headings (e.g. 1. Introduction, 1.2 Details)
      • Underlined headings (=== or ---)
      • Standalone ALL-CAPS section headers
      • Graceful fallback to paragraph blocks when no explicit headings exist.
    • txt_to_tree: generates standard PageIndex structure (title, node_id, line_num, text, nodes) with support for summaries, thinning, and document descriptions.
  2. pageindex/__init__.py:
    • Exposed txt_to_tree lazily in __all__ and _LAZY dictionary so top-level imports remain lightweight without eager subpackage loading.
  3. run_pageindex.py:
    • Added --txt_path CLI argument.
    • Updated file argument validation to enforce specifying exactly one of --pdf_path, --md_path, or --txt_path.
    • Wired --txt_path processing and output saving to results/<doc_name>_structure.json.
  4. tests/test_page_index_txt.py:
    • Added unit test suite covering numbered/chapter headers, underlined headers, all-caps headers, paragraph fallback, text content slicing, and end-to-end txt_to_tree execution.

Verification

  • Ran pytest tests/test_page_index_txt.py: 6 passed.
  • Ran pytest tests/test_package_surface.py: 11 passed (lazy import verification passed).
  • Ran full test suite: 436 passed, 54 skipped, 0 failures.

Closes #222

- Implement pageindex.page_index_txt with extract_nodes_from_txt and txt_to_tree
- Support chapter/section headings, numbered headings, underlined headings, and paragraph blocks
- Add --txt_path option to run_pageindex.py CLI
- Add test_page_index_txt.py with comprehensive unit tests

Closes VectifyAI#222
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for parsing .txt files

1 participant