[server] local segment ttl

[gyang94]

Purpose

Linked issue: close #xxx

Brief change log

Tests

API and Format

Documentation

[Copilot]

Pull request overview

This PR wires the table-level table.log.ttl setting into server-side LogTablet creation and uses it to expire tier-protected inactive local log segments (kept for tiered storage) once they exceed TTL, with a new remote-log-focused regression test. It also adjusts a Flink IT base test to use a unique database name per test run.

Changes:

• Pass table log TTL (ms) through Replica → LogManager → LogTablet and use it when deciding whether tier-protected inactive local segments are deletable.
• Invoke local cleanup of “already copied to remote” segments on every remote retention pass (not only when the remote end offset increases).
• Add TieredLocalSegmentTtlTest and make Flink batch ITs use per-test database names.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
fluss-server/src/test/java/org/apache/fluss/server/log/remote/TieredLocalSegmentTtlTest.java	New test covering TTL-driven cleanup of tier-protected inactive local segments in tiered storage setups.
fluss-server/src/main/java/org/apache/fluss/server/replica/Replica.java	Plumbs table log TTL into log creation path.
fluss-server/src/main/java/org/apache/fluss/server/log/LogTablet.java	Adds TTL awareness to local deletable-segment selection and runs remote-copied local cleanup on each retention pass.
fluss-server/src/main/java/org/apache/fluss/server/log/LogManager.java	Extends getOrCreateLog/loadLog creation to pass TTL into LogTablet.
fluss-flink/fluss-flink-common/src/test/java/org/apache/fluss/flink/source/FlinkTableSourceBatchITCase.java	Uses a randomized database name and updates expectations accordingly.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[gyang94]

It is an existing edge case which may not trigger deleteSegmentsAlreadyExistsInRemote (no matter newly added TTL or existing local-segment file counts).
I'd suggest to handle this in another issue.

[swuferhong]

Hi, @gyang94 I don't think local TTL cleanup should be coupled with remote log manifest updates. Local TTL is part of the local log's retention semantics, so it would be better to run it as a periodic task in LogManager, scanning all LogTablets independently of whether the remote manifest has changed.
The remote log should only provide the safety watermark indicating which segments have been uploaded successfully. Local deletion should still be bounded by the highest uploaded offset, the high watermark, and the minimum retain offset. same comments: #3222 (comment)

[swuferhong]

Thanks for your contribution. I left some comments.

[swuferhong]

LGTM +1. Hi @wuchong could you take a look?

[zuston]

Thanks for working on this. After reviewing the changes, I have a couple of questions:

1. Without this PR, if we set table.log.tiered.local-segments = 1, would that achieve the same goal as this PR?
2. It looks like this PR does not roll the active segment for historical/inactive partitions.

[swuferhong]

Thanks for working on this. After reviewing the changes, I have a couple of questions:

1. Without this PR, if we set table.log.tiered.local-segments = 1, would that achieve the same goal as this PR?
2. It looks like this PR does not roll the active segment for historical/inactive partitions.

Hi, @zuston. Based on the offline discussion with @gyang94 , implementing TTL for the active segment in the short term is fairly tricky and requires a detailed design—but that doesn't mean it shouldn't be done. You can continue working on this in a follow-up PR.

Regarding whether it can be set to table.log.tiered.local-segments = 1: these are two completely different concepts. One is about how much local data to reserve as a local read cache, and the other is about how long data should be retained before TTL kicks in. Consider the following scenario: the user has an hourly-partitioned table where partitions are retained for 24 hours, the log TTL is set to 1 hour, and the number of local segments is 5 (to ensure that the most recent hour of appended data stays on the server). In this case, the historical 23 hours would incur a large amount of redundant storage—even though the user has configured the log TTL to 1 hour.

[zuston]

thanks for your quick reply. @swuferhong

the user has an hourly-partitioned table where partitions are retained for 24 hours, the log TTL is set to 1 hour, and the number of local segments is 5 (to ensure that the most recent hour of appended data stays on the server). In this case, the historical 23 hours would incur a large amount of redundant storage—even though the user has configured the log TTL to 1 hour.

make sense if setting the large number of local segments reservation.

[swuferhong]

thanks for your quick reply. @swuferhong

the user has an hourly-partitioned table where partitions are retained for 24 hours, the log TTL is set to 1 hour, and the number of local segments is 5 (to ensure that the most recent hour of appended data stays on the server). In this case, the historical 23 hours would incur a large amount of redundant storage—even though the user has configured the log TTL to 1 hour.

make sense if setting the large number of local segments reservation.

Because Fluss remote currently lacks column pruning capability, many users will still choose to configure table.log.tiered.local-segments to a large value. Regarding TTL for active segments, could you and @gyang94 come up with a proposal docs first?

[zuston]

Regarding TTL for active segments, could you and @gyang94 come up with a proposal docs first?

OK. I also want to cut off the storage cost for the long-term historical partition.