LanceDB table existence check fails when more than 10 memory tables exist

[xmoon2022]

Bug Description

When using SimpleMem with more than 10 LanceDB tables, reopening an existing memory table can fail with:

Table '<table_name>' already exists

The issue appears to come from the table existence check in LanceDBVectorStoreBackend._init_table():

if self.table_name not in self.db.table_names():
    self.table = self.db.create_table(self.table_name, schema=schema)
else:
    self.table = self.db.open_table(self.table_name)

In recent LanceDB versions, table_names() is deprecated and defaults to returning only 10 tables.

As a result, once the database contains more than 10 tables, an existing table that is not included in the first page may be incorrectly treated as nonexistent. SimpleMem then calls create_table(), which fails because the table actually already exists.

Reproduction

1. Create more than 10 SimpleMem memories with different table_name values.
2. Restart the process.
3. Reopen one of the tables that is not returned in the first 10 results of db.table_names().
4. SimpleMem attempts to create the table again.

Observed error:

Table 'todo5-test-memory' already exists

Removing the other memory tables so that fewer than 10 tables remain makes the same reopen operation work correctly.

Environment

• SimpleMem: 0.3.0
• LanceDB: 0.37.1
• Python: 3.13

Root Cause

LanceDB's current Python API defines table_names() with a default limit of 10.

LanceDB already implements full pagination for membership checks through Connection.__contains__() / _all_table_names().

This is also related to the table listing/pagination behavior discussed in lancedb/lancedb#2727.

Suggested Fix

Replace:

if self.table_name not in self.db.table_names():

with:

if self.table_name not in self.db:

The latter iterates through all table-list pages instead of checking only the first 10 tables.

After applying this change locally, an existing table can be reopened correctly even when the database contains more than 10 tables.

I would be happy to submit a small PR for this fix if desired.

[modelpath-dev]

I will take this issue. Please assign it to me.

The problem is in the LanceDBVectorStoreBackend._init_table() method. The table_names() function is deprecated and only returns 10 tables by default. This causes the existence check to fail when more than 10 tables exist. I will replace the check with if self.table_name not in self.db:. This uses the full pagination mechanism and should solve the issue. I will test this change to ensure it correctly reopens tables beyond the 10-table limit.

[modelpath-dev]

I am working on this issue and will implement the suggested fix. I will replace the existence check with if self.table_name not in self.db: to handle pagination correctly. I will test the change to ensure it resolves the problem with reopening tables when there are more than 10.

[zzx04-cool]

I'd like to work on this fix. The root cause is clear: db.table_names() defaults to a 10-table limit in recent LanceDB versions, so existing tables beyond the first page are incorrectly treated as nonexistent. Replacing the membership check with self.table_name not in self.db uses LanceDB's paginated __contains__ and resolves it. I'll open a PR with the fix and a regression test shortly.