Skip to content

refactor: multiprocessing helper#222

Open
NguyenHoangSon96 wants to merge 3 commits into
mainfrom
refactor/multiprocessing-helper
Open

refactor: multiprocessing helper#222
NguyenHoangSon96 wants to merge 3 commits into
mainfrom
refactor/multiprocessing-helper

Conversation

@NguyenHoangSon96

@NguyenHoangSon96 NguyenHoangSon96 commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Closes #

Proposed Changes

  • Refactor MultiprocessingWriter class:
    - WriteApi will be created and used directly in the class for writing.
    - Tests are added for writing with multiprocessing.
    - Use DefaultContext.Process(target) to create a new process.
    - Users can now choose one of the start methods fork, spawn or forkserver when creating a new Process. The default will be spawn.

Checklist

  • CHANGELOG.md updated
  • Rebased/mergeable
  • A test has been added if appropriate
  • Tests pass
  • Commit messages are conventional
  • Sign CLA (if not already signed)

@NguyenHoangSon96 NguyenHoangSon96 self-assigned this Jul 14, 2026
@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.65%. Comparing base (b24d6c1) to head (845cd7c).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #222      +/-   ##
==========================================
- Coverage   87.21%   86.65%   -0.56%     
==========================================
  Files          28       28              
  Lines        2072     2068       -4     
==========================================
- Hits         1807     1792      -15     
- Misses        265      276      +11     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread tests/test_influxdb_client_3_integration.py Outdated
@NguyenHoangSon96
NguyenHoangSon96 force-pushed the refactor/multiprocessing-helper branch 4 times, most recently from 6fbdf4b to f227120 Compare July 16, 2026 09:25
@NguyenHoangSon96
NguyenHoangSon96 force-pushed the refactor/multiprocessing-helper branch 2 times, most recently from 706407e to 53ac912 Compare July 16, 2026 11:55
@NguyenHoangSon96
NguyenHoangSon96 marked this pull request as ready for review July 16, 2026 12:10
Copilot AI review requested due to automatic review settings July 16, 2026 12:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the write stack by removing the generated OpenAPI-style service/client layer and moving write request construction and HTTP dispatch directly into WriteApi, while also updating tests and documentation to reflect the new behavior and configuration surface.

Changes:

  • Refactors WriteApi to perform request preparation + HTTP calls directly via a new low-level RestClient (removing ApiClient/WriteService/Configuration).
  • Updates InfluxDBClient3 initialization and tests to use the new header/rest client approach and to validate required constructor keys.
  • Refactors and adds integration coverage for MultiprocessingWriter start-method behavior and direct WriteApi usage.

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
CHANGELOG.md Documents breaking changes and new construction patterns (example needs a small syntax fix).
influxdb_client_3/init.py Reworks InfluxDBClient3 construction to create RestClient + WriteApi directly and enforce required keys.
influxdb_client_3/write_client/init.py Removes deprecated exports and re-exports updated WriteApi/types.
influxdb_client_3/write_client/_sync/api_client.py Removes legacy generated ApiClient.
influxdb_client_3/write_client/_sync/rest.py Removes legacy generated REST client implementation.
influxdb_client_3/write_client/_sync/rest_client.py Adds new RestClient for low-level HTTP handling + logging/redaction.
influxdb_client_3/write_client/client/init.py Removes re-export of removed WriteService.
influxdb_client_3/write_client/client/_base.py Removes legacy base client/write-api helpers tied to generated stack.
influxdb_client_3/write_client/client/influxdb_client.py Removes legacy v2 InfluxDBClient.
influxdb_client_3/write_client/client/logging_handler.py Removes InfluxLoggingHandler implementation.
influxdb_client_3/write_client/client/util/helpers.py Updates exception import path.
influxdb_client_3/write_client/client/util/multiprocessing_helper.py Refactors MultiprocessingWriter to create WriteApi directly and set default start method.
influxdb_client_3/write_client/client/write/init.py Removes legacy re-export of removed WriteService.
influxdb_client_3/write_client/client/write_api.py Major refactor: WriteApi becomes standalone, builds write requests and calls RestClient directly, retains batching pipeline.
influxdb_client_3/write_client/configuration.py Removes legacy generated configuration object.
influxdb_client_3/write_client/service/init.py Removes legacy service package init exporting WriteService.
influxdb_client_3/write_client/service/_base_service.py Removes legacy service base class.
influxdb_client_3/write_client/service/write_service.py Removes legacy generated WriteService.
influxdb_client_3/write_client/write_exceptions.py Removes _BaseRESTClient logger helper (moved to RestClient).
tests/test_influxdb_client_3.py Updates unit tests for new client/write header/rest behavior and required params.
tests/test_influxdb_client_3_integration.py Adds integration tests for direct WriteApi usage, multiprocessing helper, and async call path.
tests/test_polars.py Updates write tests to mock WriteApi call sites instead of WriteService.
tests/test_query.py Fixes typo in client init parameter (database).
tests/test_write_api.py Refactors tests from generated service/client usage to InfluxDBClient3/WriteApi usage.
tests/test_write_local_server.py Updates exception import path.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread influxdb_client_3/write_client/client/util/multiprocessing_helper.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

Comment thread influxdb_client_3/write_client/client/util/multiprocessing_helper.py Outdated
Comment thread influxdb_client_3/__init__.py
@NguyenHoangSon96
NguyenHoangSon96 force-pushed the refactor/multiprocessing-helper branch 4 times, most recently from f6ef07c to ab75516 Compare July 16, 2026 15:09
@NguyenHoangSon96

Copy link
Copy Markdown
Contributor Author

@bednar @karel-rehor @alespour
Hi guys,
I made some quite big changes to our MultiprocessingWriter class.
Please take a look. I'm not very familiar with multiprocessing in Python.

@NguyenHoangSon96
NguyenHoangSon96 force-pushed the refactor/multiprocessing-helper branch 5 times, most recently from 24b67a3 to aed0973 Compare July 22, 2026 07:32

@karel-rehor karel-rehor left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good so far.

I especially like the use of get_context() for each instance. It's cleaner, more flexible and more stable than what I suggested last week.

A couple of notes.

  1. need to update CHANGELOG.md

  2. The codecov report shows that important parts of the run and terminate methods are not covered by tests.

I think this is partially a false report. I suspect codecov is not tracking code executed in newly spawned processes. When running tests in the debugger, I see the processes enter this area of code. It's probably necessary to configure codecov to follow multiprocess executions.

e.g. in pyproject.toml (+AI+ suggestion 1)

[tool.coverage.run]
concurrency = ["multiprocessing"]
parallel = true
source = ["your_package_name"]

or perhaps in .coveragerc (+AI+ suggestion 2)

[run]
concurrency = multiprocessing
parallel = true
source = your_package_name

It was suggested in a PR for influxdb-client-php that CircleCI has a codecov orb that simplifies working with codecoverage reports. It might help resolve this issue.

@NguyenHoangSon96
NguyenHoangSon96 marked this pull request as draft July 22, 2026 14:09
@NguyenHoangSon96
NguyenHoangSon96 force-pushed the refactor/multiprocessing-helper branch from 62dc5e2 to 3f65c70 Compare July 22, 2026 14:56
@NguyenHoangSon96
NguyenHoangSon96 marked this pull request as ready for review July 22, 2026 14:59
@NguyenHoangSon96
NguyenHoangSon96 force-pushed the refactor/multiprocessing-helper branch 7 times, most recently from 932dd18 to 59fc69c Compare July 22, 2026 17:32
@NguyenHoangSon96

Copy link
Copy Markdown
Contributor Author

This looks good so far.

I especially like the use of get_context() for each instance. It's cleaner, more flexible and more stable than what I suggested last week.

A couple of notes.

  1. need to update CHANGELOG.md
  2. The codecov report shows that important parts of the run and terminate methods are not covered by tests.

I think this is partially a false report. I suspect codecov is not tracking code executed in newly spawned processes. When running tests in the debugger, I see the processes enter this area of code. It's probably necessary to configure codecov to follow multiprocess executions.

e.g. in pyproject.toml (+AI+ suggestion 1)

[tool.coverage.run]
concurrency = ["multiprocessing"]
parallel = true
source = ["your_package_name"]

or perhaps in .coveragerc (+AI+ suggestion 2)

[run]
concurrency = multiprocessing
parallel = true
source = your_package_name

It was suggested in a PR for influxdb-client-php that CircleCI has a codecov orb that simplifies working with codecoverage reports. It might help resolve this issue.

Hi Karel

  • CHANGELOG.md has been updated.
  • Your suggestions have been added to pyproject.

@NguyenHoangSon96

Copy link
Copy Markdown
Contributor Author

Please stop your reviewing, the codecov failure appears today.
It did not fail yesterday.
I will try to fix it.

@NguyenHoangSon96
NguyenHoangSon96 force-pushed the refactor/multiprocessing-helper branch from 92f747a to 6f6cb89 Compare July 23, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants