Skip to content

fix: resolve "Not in Wikidata" errors for valid verbs (#598, #738) - #742

Open
prince-0408 wants to merge 2 commits into
scribe-org:mainfrom
prince-0408:fix/conjugate-lookup-wikidata-598
Open

prince-0408 wants to merge 2 commits into
scribe-org:mainfrom
prince-0408:fix/conjugate-lookup-wikidata-598

Conversation

@prince-0408

Copy link
Copy Markdown
Collaborator

Description

This PR resolves #598 ("Conjugate command returns 'Not in Wikidata' for valid verbs") and #738 ("Redundant sequential SQLite file opens and identical SELECT queries").

Root Causes

  1. Missing Database Fallback in DatabaseFileManager: When keyboard users download their language packages, the data is saved in ${language}LanguageData.sqlite (which includes the verbs table). However, ConjugateDataManager exclusively queried getConjugateDatabase(), which looked only for ${language}ConjugateData.sqlite (downloaded only in the Conjugate standalone app). Since the file didn't exist for keyboard users, getConjugateDatabase() returned null, failing all valid verbs with "Not in Wikidata".
  2. Strict Single-Column & Case-Sensitive Matching: getVerbCursor() used WHERE $columnName = ? on only the infinitive column. Any uppercase or mixed-case verb (e.g. German capitalized forms, or uppercase typing), inflected verb form (e.g. typing "went" or "running"), or verbs with common grammatical prefixes ("to ", "se ", "sich ", "s'") immediately failed.
  3. Severe N+1 Disk I/O & Connection Leaks: getTheConjugateLabels looped across every single tense and form (up to 120 iterations per verb), opening and closing the database file sequentially on every call. Additionally, parseComplexForm invoked getConjugateDatabase() without closing the database connection.

Key Changes

  • Database Fallback & Case Normalization (DatabaseFileManager.kt):
    • Added an automatic fallback to getLanguageDatabase() when ${language}ConjugateData.sqlite is absent.
    • Normalized database filenames with .uppercase() to prevent filesystem casing mismatches on Linux/Android ext4 filesystems.
  • Single-Query Caching & Fallback Lookup Ladder (ConjugateDataManager.kt):
    • Opens the database connection once per lookup and caches the verb row into an in-memory map.
    • Queries candidate infinitive columns using case-insensitive SQL matching: LOWER(${SqlIdentifier.quote(col)}) = LOWER(?).
    • Supports prefix-stripping fallback for leading "to ", "se ", "sich ", and "s'".
    • Adds an inflected form fallback that searches verbs table columns if direct infinitive matching yields no results.
    • Reuses the existing open database connection in auxiliary parsing (parseComplexFormFromRow), eliminating database connection leaks.
    • Added getStructuredConjugations() to efficiently serve UI screens.
  • SQL Sanitization & Table Column Discovery (SQLiteExtensions.kt):
    • Added getTableColumns() helper with SqlIdentifier validation and quoting.
  • UI State Preservation & Fast Loading (ConjugationSelectionScreen.kt):
    • Uses manager.getStructuredConjugations() to load forms in a single pass.
    • Uses rememberSaveable for selectedTenseGroup to survive device rotations and fold/unfold events.
  • Unit Testing (ConjugateDataManagerTest.kt):
    • Added comprehensive JUnit 5 tests covering exact infinitive lookups, case-insensitivity (HABEN -> haben), prefix-stripping (to walk -> walk), inflected form lookups (went -> go), and missing database handling.
  • Changelog:

Closes #598
Closes #738

@prince-0408 prince-0408 changed the title fix: resolve "Not in Wikidata" errors and eliminate redundant sequent… fix: resolve "Not in Wikidata" errors for valid verbs (#598, #738) Oct 28, 2026
private lateinit var db: SQLiteDatabase
private lateinit var manager: ConjugateDataManager

private val sampleContract =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a verb column is missing the complex form parsing now returns just the auxiliary, like have instead of an empty string which would show wrong conjugations. The new fallback also changes what the Conjugate app search returns and the tests only use mocks so they never check the SQL or the actual fix.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review @Roniscend! Addressed in the latest commit!

… test against real SQLite

- Return empty string immediately in parseComplexFormFromRow if the verb column is missing or empty, avoiding displaying standalone auxiliaries like 'have'.
- Remove the getLanguageDatabase fallback in DatabaseFileManager.getConjugateDatabase to prevent unintentional query scope pollution in Conjugate app search.
- Replace MockK cursor stubs with Robolectric in-memory SQLite tables to verify actual SQL queries and schema behaviors.
@prince-0408
prince-0408 requested a review from Roniscend October 28, 2026 17:49
@andrewtavis

Copy link
Copy Markdown
Member

@ashb155, can you also take a look at this to see if it addresses #738?

@ashb155

ashb155 commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

yes it does, looks good to me @andrewtavis !

@andrewtavis

Copy link
Copy Markdown
Member

Thanks, @ashb155! I'll look into scribe-org/Scribe-Data#725 and then we can redeploy Scribe-Server and finalize this one. @angrezichatterbox, would be good for you to also take a look once the Scribe-Data/Scribe-Server processes are finalized :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Redundant sequential SQLite file opens and identical SELECT queries Conjugate command returns "Not in Wikidata" for valid verbs

4 participants