Skip to content

Add pluggable page-level AEAD encryption support#866

Draft
HTHou wants to merge 1 commit into
developfrom
codex/tsfile-tde-page-aead
Draft

Add pluggable page-level AEAD encryption support#866
HTHou wants to merge 1 commit into
developfrom
codex/tsfile-tde-page-aead

Conversation

@HTHou

@HTHou HTHou commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add an optional file encryption header carrying provider, profile, key, wrapped data key, and file identity metadata
  • introduce a provider-neutral AEAD SPI and registry without embedding a concrete cryptographic implementation
  • encrypt compressed page bodies for non-aligned, aligned, and table-model write paths, and decrypt them on the corresponding read paths
  • preserve existing unencrypted TsFile behavior and propagate encryption metadata through append, recovery, lazy loading, and sketch tooling
  • add compatibility, round-trip, tamper-detection, append, and recovery tests

Motivation

TsFile currently has legacy encryption interfaces but no self-describing file-level context for pluggable page-level authenticated encryption. This change establishes the format and I/O integration points while leaving algorithm and key-management implementations to external providers.

Compatibility

  • Existing unencrypted TsFiles keep their current layout and remain readable.
  • Encrypted files require a registered provider matching the identifiers stored in the encryption header.
  • Page metadata remains readable while compressed page bodies are protected with AEAD.
  • The encrypted-file format and public API are still under discussion and are not intended as a compatibility commitment in this draft.

Validation

  • ./mvnw spotless:check -P with-java -pl java/tsfile -am
  • ./mvnw test -P with-java -pl java/tsfile -am -Dtest='PageCryptoContextTest,FileEncryptionHeaderTest,TDEPageAeadTsFileTest,UnClosedTsFileReaderTest,ForceAppendTsFileWriterTest,RestorableTsFileIOWriterTest,TimePageWriterTest' -Dsurefire.failIfNoSpecifiedTests=false
  • 30 targeted tests passed with no failures or errors

Draft discussion points

  • finalize the encryption-header schema and format-version semantics
  • bind page authentication data to an unambiguous chunk/page identity
  • refine page-index ownership and the separation between legacy encryption parameters and file AEAD context
  • confirm append/recovery behavior and backward-compatibility expectations before stabilizing the format

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.37107% with 163 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.71%. Comparing base (86252e5) to head (76754e0).
⚠️ Report is 7 commits behind head on develop.

Files with missing lines Patch % Lines
...pache/tsfile/file/header/FileEncryptionHeader.java 73.56% 23 Missing ⚠️
...a/org/apache/tsfile/read/TsFileSequenceReader.java 63.46% 19 Missing ⚠️
...sfile/write/v4/AbstractTableModelTsFileWriter.java 37.03% 17 Missing ⚠️
...a/org/apache/tsfile/encrypt/PageCryptoContext.java 74.57% 15 Missing ⚠️
...g/apache/tsfile/read/reader/chunk/ChunkReader.java 60.00% 14 Missing ⚠️
...ain/java/org/apache/tsfile/write/TsFileWriter.java 59.37% 13 Missing ⚠️
...e/tsfile/write/writer/ForceAppendTsFileWriter.java 54.16% 11 Missing ⚠️
...org/apache/tsfile/write/writer/TsFileIOWriter.java 82.60% 8 Missing ⚠️
...che/tsfile/encrypt/EncryptionProviderRegistry.java 75.00% 7 Missing ⚠️
...ache/tsfile/read/reader/page/LazyLoadPageData.java 70.83% 7 Missing ⚠️
... and 10 more
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #866      +/-   ##
===========================================
+ Coverage    60.52%   60.71%   +0.19%     
===========================================
  Files          735      740       +5     
  Lines        48439    48930     +491     
  Branches      7679     7746      +67     
===========================================
+ Hits         29319    29710     +391     
- Misses       17718    17818     +100     
  Partials      1402     1402              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants