Skip to content

Use SDTM as input - #1318

Draft
Gero1999 wants to merge 34 commits into
mainfrom
sdtm-input
Draft

Use SDTM as input#1318
Gero1999 wants to merge 34 commits into
mainfrom
sdtm-input

Conversation

@Gero1999

@Gero1999 Gero1999 commented May 22, 2026

Copy link
Copy Markdown
Collaborator

Issue

Closes #624, Closes #1086

Description

Adds SDTM (PC + EX + DM) as an alternative input format alongside the existing ADNCA path, both as R package functions and in the Shiny app.

Phase 1 — R package functions (R/sdtm_to_pknca.R)

Core conversion pipeline from SDTM domain data to PKNCAdata objects:

  • sdtm_to_PKNCAdata(pc, ex, dm, metabolites) — top-level entry point. Accepts PC, EX, and optional DM data.frames, returns a PKNCAdata object ready for NCA computation.
  • pc_to_PKNCAconc(pc, ex, dm) — converts PC domain to PKNCAconc. Derives timing variables (AFRLT, ARRLT, NFRLT, NRRLT), dose assignment (DOSNOA, ATPTREF), and merges DM demographics.
  • ex_to_PKNCAdose(ex) — converts EX domain to PKNCAdose. Parses ISO 8601 durations, maps CDISC routes to PKNCA route values.
  • Helper functions: parse_iso8601_duration(), route_cdisc_to_pknca(), std_dtc_to_rdate(), .check_required_cols(), .prepare_dose_table(), .assign_doses_to_conc().

Phase 2 — Example datasets

  • pc_example, ex_example, dm_example — SDTM-format datasets derived from adnca_example via data-raw/sdtm_example.R.
  • adnca_example updated to fix time-rounding issues that caused inconsistencies between SDTM and ADNCA paths.
  • metadata_nca_variables extended with SDTM, PC, EX, DM domain rows for the mapping UI.

Phase 3 — Shiny app integration

  • Upload step (data_upload.R): ADNCA/SDTM radio toggle. SDTM mode shows 3 file inputs (PC required, EX required, subject-level optional multi-file). Default SDTM example data loaded when no files uploaded.
  • Mapping step (data_mapping.R): SDTM-specific mapping panel with sections (SDTM General, PC Concentrations, EX Dosing, Supplemental Variables). All SDTM input IDs prefixed with sdtm_ to avoid collisions with ADNCA inputs sharing the same variable names.
  • NCA pipeline (tab_data.R): SDTM path calls sdtm_to_PKNCAdata() at the mapping step, then feeds the resulting PKNCAdata object into the existing NCA computation pipeline.
  • Export (zip-utils.R): SDTM mode exports PC/EX/DM as separate RDS files and uses a dedicated script template (script_template_sdtm.R) that applies column mapping before calling sdtm_to_PKNCAdata().
  • Settings (nca_setup.R): Settings YAML records input_mode and sdtm_metabolites.

Tests

  • 60+ unit tests for the R conversion functions (test-sdtm_to_pknca.R).
  • SDTM example dataset validation tests (test-sdtm_example_data.R).
  • Round-trip NCA equivalence tests comparing SDTM and ADNCA paths produce consistent results.

Review guide

This PR is large. To make review manageable, the changes are split into 3 focused review-view PRs. Use these for code review — they show only the relevant files for each phase:

Phase Review PR What to review
1. R functions #1328 Conversion logic, timing derivations, tests
2. Example data #1329 Dataset generation, adnca_example fixes, metadata
3. Shiny app #1330 Upload/mapping UI, export, settings

⚠️ The review-view PRs are not meant to be merged — they exist only to provide focused diffs. This PR (#1318) is the one to merge.

Definition of Done

  • SDTM conversion functions work standalone via R script
  • Shiny app supports SDTM upload, mapping, NCA computation, and export
  • Settings save/restore works for SDTM mode
  • SDTM script template applies column mapping
  • Example SDTM datasets ship with the package
  • NEWS entry added
  • Package version incremented

How to test

1. R package path (code: #1328)

library(aNCA)
pknca_obj <- sdtm_to_PKNCAdata(pc_example, ex_example, dm_example)
str(pknca_obj)  # should be a PKNCAdata object
nrow(pknca_obj$conc$data)  # concentration rows
nrow(pknca_obj$dose$data)  # dose rows

2. Shiny app — default data (code: #1330)

  1. Launch aNCA::run_app()
  2. Select "SDTM" radio button
  3. Click Submit without uploading files (uses example data)
  4. Verify NCA results appear

3. Shiny app — custom data (code: #1330)

  1. Upload PC and EX CSV files in SDTM format
  2. Optionally upload a DM file
  3. Adjust mapping if column names differ from standard
  4. Run NCA

4. Export (code: #1330)

  1. After running NCA in SDTM mode, download ZIP
  2. Verify it contains input_pc.rds, input_ex.rds, input_dm.rds
  3. Verify script_template_sdtm.R has column mapping applied

5. Settings round-trip (code: #1330)

  1. Run NCA in SDTM mode
  2. Download settings YAML — verify it contains input_mode: sdtm
  3. Switch to ADNCA mode, upload the settings YAML
  4. Verify it switches back to SDTM mode

Contributor checklist

  • Code passes lintr checks
  • Code passes all unit tests
  • New logic covered by unit tests
  • New logic is documented
  • App or package changes are reflected in NEWS
  • Package version is incremented
  • R script works with the new implementation (if applicable)
  • Settings upload works with the new implementation (if applicable)
  • If any .scss change was done, run data-raw/compile_css.R
  • If a package dependency was added/changed, run data-raw/test_suggests_hidden.R

Notes to reviewer

Expected SDTM input data

The app expects standard CDISC SDTM domains. Here are the columns used per domain:

PC — Pharmacokinetic Concentrations (required upload)

Column Required Description
STUDYID Study identifier
USUBJID Subject identifier
PCTEST Analyte name (maps to PARAM)
PCSTRESN Numeric concentration result (maps to AVAL)
PCSTRESU Concentration unit (maps to AVALU)
PCDTC Collection datetime (ISO 8601, e.g. 2025-01-15T08:30:00)
PCSPEC Specimen type (e.g. PLASMA, URINE)
PCRFTDTC Reference datetime for the most recent dose
PCELTM Planned elapsed time from most recent dose (ISO 8601 duration, e.g. PT2H)
VOLUME Urine volume (for urinary PK)
VOLUMEU Volume unit

EX — Exposure / Dosing (required upload)

Column Required Description
STUDYID Study identifier
USUBJID Subject identifier
EXTRT Treatment name (maps to DOSETRT)
EXDOSE Dose amount (maps to DOSEA)
EXDOSU Dose unit
EXROUTE Route of administration (e.g. INTRAVENOUS, ORAL)
EXSTDTC Dosing start datetime (ISO 8601)
EXENDTC Dosing end datetime — used to derive dose duration if EXDUR is absent
EXDUR Dose duration (ISO 8601 duration, e.g. PT1H) — takes priority over EXENDTC
EXELTM Planned elapsed time from first dose (ISO 8601 duration)

DM — Demographics (optional upload, labeled "Subject-level data" in the app)

Column Required Description
USUBJID Subject identifier (join key)
STUDYID Study identifier (join key if present)
RFXSTDTC Date/time of first study treatment — used as reference time if PCRFTDTC is absent
AGE Subject age
AGEU Age unit
SEX Sex
RACE Race
ARM Planned treatment arm
ACTARM Actual treatment arm

The app auto-maps columns by name. If column names differ from the standard, users can remap them in the mapping step. The mapping UI also supports WTBL/WTBLU (baseline weight) and custom grouping variables.

Code review

For manual code review, use the phase-specific PRs linked in the Review guide above — each shows only the relevant subset of files. This PR contains the full integrated diff.

Gero1999 and others added 10 commits May 18, 2026 18:08
…estthat-check

feat: automate data-raw rebuild with helper script and CI staleness check
…x_example, pc_example, test-multianalyte-ADNCA.csv)
- Fix RRLTU from "hour" to "Hours" in pc_to_PKNCAconc to match ADNCA
  convention and produce identical units tables between paths.

- Pass metabolites parameter in round-trip tests so METABFL is populated.

- Add PKNCAdata equivalence tests comparing SDTM and ADNCA objects:
  units table, formulas, column mappings, all shared conc/dose columns,
  dose row count, DOSEA, std_route, ADOSEDUR, DOSNOA, demographics.

- Add pre-dose normalization tests: both paths assign pre-dose samples
  to different reference doses (SDTM: previous, ADNCA: upcoming). Tests
  normalize to the same convention and verify ARRLT, NFRLT, DOSNOA, and
  ADOSEDUR all match after normalization.

- Add CONCDUR test for serum samples (both paths produce 0).

- Add SDTM example data consistency tests: inter-sample intervals,
  dose amounts, dose durations, inter-dose intervals.

- Fix inter-dose interval test to use tapply on ATPTREF (collapse
  infusion start/end into one dose time per period) and relax tolerance
  to 0.005h for rounding in AFRLT values.
@Gero1999 Gero1999 changed the title Sdtm input Use SDTM as input May 22, 2026
Gero1999 and others added 18 commits May 22, 2026 14:53
Add alternative data input mode for SDTM domain files (PC, EX, DM)
alongside the existing ADNCA upload workflow.

Upload: ADNCA/SDTM radio toggle with separate file inputs for PC
(required), EX (required), and subject-level data (optional, merged
by STUDYID+USUBJID).

Mapping: conditional UI with PC Concentrations, EX Dosing, and
Supplemental Variables sections. Standard SDTM column names auto-
detect. On submit, calls sdtm_to_PKNCAdata() directly.

Settings: exports/imports input_mode and sdtm_metabolites so SDTM
sessions can be restored.

Script export: auto-selects script_template_sdtm.R which loads
domain files separately and calls sdtm_to_PKNCAdata().

Metadata: adds PC, EX, DM domain rows to metadata_nca_variables
with mapping tooltips, sections, and alternatives.

Co-authored-by: Ona <no-reply@ona.com>
Change DM domain rows (AGE, AGEU, SEX, RACE, RFXSTDTC) from
Core=Exp/Req to Core=Perm so export_cdisc() does not try to
create these columns in ADNCA/PP output.

Add CSV exports of pc_example, ex_example, dm_example to
tests/testthat/data/ for SDTM upload testing. Update
sdtm_example.R to regenerate them.

Update man/get_settings_code.Rd to reflect template_path=NULL
default.

Co-authored-by: Ona <no-reply@ona.com>
Replace conditionalPanel with shinyjs::hidden/show/hide for reliable
panel toggling between ADNCA and SDTM mapping modes.

Prefix all SDTM widget input IDs with 'sdtm_' (e.g. select_sdtm_PCTEST)
to avoid collisions with ADNCA inputs that share variable names
(PCSPEC, VOLUME, VOLUMEU, WTBL, WTBLU, Metabolites, Grouping_Variables).

Add .update_sdtm_selectize_inputs() that strips the sdtm_ prefix
before looking up variable names in SDTM_MAPPING_INFO for auto-
detection. All 18 standard SDTM columns auto-detect correctly.

Add .apply_sdtm_column_rename() for renaming non-standard columns
back to expected SDTM names before calling sdtm_to_PKNCAdata().

Co-authored-by: Ona <no-reply@ona.com>
Root cause of freeze: PKNCAdata accessor used $data.conc$data
instead of $conc$data, returning NULL and silently stalling the
reactive chain. Fixed in both data_mapping.R and tab_data.R.

Add SDTM General section to metadata_nca_variables with STUDYID
and USUBJID as mappable variables shared across all domains.
Remove duplicate STUDYID/USUBJID rows from PC, EX, DM domains.

Add diagnostic logging throughout the SDTM pipeline:
- Upload: log domain sizes after successful load
- Mapping: log column rename steps, sdtm_to_PKNCAdata call with
  row counts, and successful PKNCAdata creation
- Tab data: log PKNCAdata retrieval and filter application
- Errors: show user-facing notification with SDTM context

Co-authored-by: Ona <no-reply@ona.com>
The || operator in .apply_sdtm_column_rename failed when
Grouping_Variables mapped to a multi-value vector (e.g. 5 columns).
selected == "" produced a length-5 logical vector which || cannot
handle. Fix: check length(selected) != 1 before any scalar
comparisons.

Co-authored-by: Ona <no-reply@ona.com>
Downstream modules (TLG listings, exploration plots) reference
TRT01A which is an ADNCA column name. The SDTM path produces
DOSETRT instead. Add TRT01A as a copy of DOSETRT so all
downstream modules work without modification.

Co-authored-by: Ona <no-reply@ona.com>
Co-authored-by: Ona <no-reply@ona.com>
Co-authored-by: Ona <no-reply@ona.com>
test-sdtm_to_pknca: update expected RRLTU from 'hour' to 'Hours'
to match the fix from a previous commit.

test-sdtm_example_data: tapply returns an array, diff(POSIXct)
returns numeric. Use as.numeric() on tapply output so expect_equal
compares values instead of failing on dim mismatch.

Co-authored-by: Ona <no-reply@ona.com>
- Rename Tmax/Cmax to t_max/c_max in data-raw/adnca_example.R
- Fix hanging indent in data_mapping.R (lines 278, 387)
- Remove trailing blank line in data_mapping.R
- Extract .populate_adnca_inputs, .populate_sdtm_inputs,
  .get_adnca_metabolite_selection to reduce data_mapping_server
  cyclomatic complexity from 24 to ~13
- Extract .read_sdtm_domain, .read_subject_level_files,
  .build_sdtm_result to reduce .process_sdtm_uploads
  cyclomatic complexity from 21 to ~4
- Add script_template_sdtm.R to .lintr exclusions
- Split test-sdtm_example_data.R into 4 top-level describe blocks
  to reduce cyclomatic complexity from 18 to ~8 per block
- Fix hanging indent in test-sdtm_to_pknca.R (line 318)

Co-authored-by: Ona <no-reply@ona.com>
@Gero1999
Gero1999 marked this pull request as ready for review May 27, 2026 13:20
@Gero1999
Gero1999 marked this pull request as draft May 27, 2026 13:45
@Gero1999
Gero1999 marked this pull request as ready for review May 27, 2026 13:56
@Gero1999
Gero1999 requested a review from KOBANAK May 28, 2026 07:45
- parse_iso8601_duration now supports full ISO 8601: PnYnMnDTnHnMnS
  and PnW formats (days, weeks, months, years), not just PTnHnMnS
- Empty strings and whitespace-only values return NA instead of
  failing the regex match
- EXDUR: when column exists but is all empty/NA, falls back to
  EXENDTC - EXSTDTC, then to 0. Per-row fallback for mixed data.
- EXELTM/PCELTM: same empty-column detection before attempting parse
- PCELTM: per-row fallback to ARRLT for rows where value is empty

Co-authored-by: Ona <no-reply@ona.com>
@Gero1999
Gero1999 marked this pull request as draft June 2, 2026 08:45
Gero1999 and others added 5 commits June 2, 2026 08:45
.prepare_dose_table() now emits a warning when dose duration is
defaulted to 0 hours due to missing EXDUR and EXENDTC. The Shiny
app surfaces this as a notification via withCallingHandlers.

Co-authored-by: Ona <no-reply@ona.com>
When PCREASND is present in PC data, non-empty values are used as
NCA exclusion reasons (e.g. 'Sample hemolyzed'). Falls back to
PCSTAT (e.g. 'NOT DONE') when PCREASND is absent. This lets PKNCA
skip samples that were not collected/analysed, preventing errors
from missing time values on those rows.

Co-authored-by: Ona <no-reply@ona.com>
Add PCREASND to the PC domain metadata so users can map it in the
SDTM mapping UI. When present, non-empty values are used as
nca_exclude reasons (same role as NCAwXRS in the ADNCA path),
letting PKNCA skip samples that were not done.

Co-authored-by: Ona <no-reply@ona.com>
PKNCA (CRAN) validates time values before checking the exclude
column, so excluded rows with NA AFRLT cause an assertion error.
Workaround: remove those rows and warn the user.

TODO: Remove this filter once PKNCA honours exclude before
validating time.

Co-authored-by: Ona <no-reply@ona.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enhancement: Consider allowing new inputs for the App Enhancement: Create example_PC and example_EX datasets from example_ADNCA

1 participant