Database dump parser with high-speed local search & compression
- High-speed local search
- CSV/TSV/TXT/JSON support
- Parquet index manager
- DuckDB
- ZSTD compression
- AES-GCM encryption
- Multi-core support
Just extract the database dumps to any directory and specify it in file config.py (DB_PATH)
All settings are located in config.py:
| Parameter | Default | Description |
|---|---|---|
DB_PATH |
~/databases |
Root directory containing database subfolders |
PARQUET_DIR |
parquet_store |
Directory for storing compressed/encrypted Parquet files and manifest |
REPORT_FILE |
report.txt |
Output log file for search results |
MERGE_FOLDER_PARQUET |
True |
Merge all files within a database folder into a single .eparquet / .parquet file |
STREAM_CHUNK_LINES |
250_000 |
Line buffer size for streaming normalization |
TEXT_CHECK_BYTES |
8192 |
Header sample size for binary and encoding detection |
MAX_CHAOS_THRESHOLD |
0.5 |
Maximum charset-normalizer chaos score allowed for text files |
COMPRESSION_CODEC |
ZSTD |
Parquet compression codec |
COMPRESSION_LEVEL |
9 |
Parquet compression level |
ENCRYPT_INDEX |
True |
Enable AES-GCM encryption for Parquet files and manifest |
ENCRYPTION_PASSWORD |
DB_PARSER_KEY env var |
Master password for index encryption |
SUPPORTED_EXTENSIONS |
{".csv", ".tsv", ".txt", ".json", ".jsonl"} |
Allowed file extensions during directory scans |
Set the encryption password in PowerShell before running:
$env:DB_PARSER_KEY = "password"Caution
This project does not contain personal data of any people. The examples above, including database dump names and obfuscated content fragments, are provided solely as an example as possible content to be processed by the utility.
Note
The utility supports acceleration due to multi-core mode support, while the identification and utilization of the maximum available number of processor cores is performed in automatic mode. Automatic transliteration of full name is also supported (only Russian language for now).
Run without arguments to launch the interactive search loop:
python db_parser.py- Enter a search value and select the search type (
1-NAME,2-PHONE,3-EMAIL,4-BIRTHDAY). - Enter
0to synchronize the Parquet index withDB_PATH. - Enter
qto exit.
Scan DB_PATH for new or modified files, normalize text to UTF-8, and update the Parquet store:
python db_parser.py syncDisplay all indexed databases, Parquet files, source tables, file sizes, and detected encodings:
python db_parser.py listRun a search query directly from the command line and append results to report.txt:
python db_parser.py search "Ivanov Ivan" --type name
python db_parser.py search "+79991234567" --type phone
python db_parser.py search "user@example.com" --type email
python db_parser.py search "01.01.1990" --type birthdayOptions:
value: Search string (required).-t,--type: Search mode (name,phone,email,birthday; default:name).
Import an external file or directory into the index:
# Add a single file to an existing or new database
python db_parser.py add "D:\dumps\part2.csv" --db clients
# Add an entire folder (uses folder name as DB name if --db is omitted)
python db_parser.py add "D:\dumps\new_leak_folder"Options:
path: Path to a source file or directory (required).--db: Target database name (optional; defaults to parent folder name for files or directory name for folders).
Remove an entire database or a specific source table from the index:
# Remove a single table and rebuild the merged Parquet file without it
python db_parser.py remove --db clients --table "part2.csv"
# Remove an entire database from the index
python db_parser.py remove --db clientsOptions:
--db: Database name (required).--table: Specific source filename to remove (optional; omits to delete the entire database).
Re-encrypt all .eparquet files and manifest.enc with a new password:
# Interactive password prompt
python db_parser.py passwd
# Non-interactive mode
python db_parser.py passwd --old-password "old-secret" --new-password "new-secret"