fix(catalog): handle filesystem catalog existence on object stores

[Pandas886]

Purpose

Linked issue: close #527

The filesystem catalog could discover a database or table while listing an object-store prefix, then reject the same entry when checking the exact path. Existing Paimon warehouses often have objects below a prefix without a directory marker, so an exact HEAD request is not enough to determine whether a database exists.

Table existence also needs to be based on schema files rather than the table directory itself. This is the behavior used by the Java filesystem catalog.

Brief change log

• Add a directory-aware existence check to FileIO.
• Use directory semantics when checking filesystem catalog databases.
• Check schema-0 first, then fall back to other schema IDs when checking tables.
• Filter directories without a valid schema from list_tables().
• Cover empty directories, markerless prefixes, non-zero schema IDs, and incomplete table directories.

Tests

• cargo test -p paimon --all-targets --features fulltext,vortex
• cargo clippy --all-targets --workspace --features fulltext,vortex -- -D warnings
• cargo fmt --all -- --check
• cargo check -p paimon --features storage-s3
• Tested against a local MinIO container with an empty database, a markerless database and table, a table starting at schema-1, and an incomplete table directory.

API and Format

This adds FileIO::exists_dir(). There is no storage format change.

Documentation

No documentation changes are needed.

[JingsongLi]

PR changes “table existence” to “schema must exist,” and list_tables() will also filter out orphaned directories without a schema; however, drop_database(cascade=false) still checks the original subdirectories directly. As a result, the Catalog shows that the database has no tables, but it returns DatabaseNotEmpty due to an incomplete or orphaned directory. The Java implementation uses listTables(name) for this check; the semantics should be standardized.

[Pandas886]

Good catch. I updated drop_database() to use list_tables(name) for the non-cascade emptiness check, matching the Java catalog semantics. I also added a regression test covering a database that contains only an incomplete table directory: it is not listed as a table and the database can be dropped without cascade.