Masking Several Datasets: run2

A common job: your study data are several CDISC ADaM datasets (ADSL, ADEX, ADPC, ADLB, ...), and you want an LLM to combine them into one NONMEM-style master dataset without seeing any real identifier. run cannot do that: it masks one dataset at a time, with different codes in each, so the LLM could not join them. run2 masks the datasets together.

1. What run2 does

The four example datasets used on this page (download them to follow along): csvexample-adam-adsl.csv (12 subjects, 17 columns), csvexample-adam-adex.csv (24 dosing records), csvexample-adam-adpc.csv (168 concentrations) and csvexample-adam-adlb.csv (72 laboratory values).

2. Giving run2 its datasets

Three ways; all give the same result.

data-obfuscator run2 --input adam/adsl.csv --input adam/adex.csv --input adam/adpc.csv --input adam/adlb.csv --output ./masked2 --level 3
data-obfuscator run2 --input adam --output ./masked2                    # every .csv .tsv .txt .dat .tab file in the folder
data-obfuscator run2 --config options-run2.txt                          # input = ... once per dataset

An options file: txtoptions-run2.txt (tested on the example data). The folder form reads the files in alphabetical order.

3. A complete run2

data-obfuscator run2 --input adam/adsl.csv --input adam/adex.csv --input adam/adpc.csv --input adam/adlb.csv \
  --output ./masked2 --level 3 --seed 7 --scale-columns AVAL,WEIGHTBL

Read the log from the top:

1. run2: 4 dataset(s), level 3, seed 7, then one read line per dataset, with rows and columns.
2. The ADaM hints: columns that run's own detection would have left alone are recognised by their ADaM names.
3. date shift (one value for all 4 dataset(s), intervals preserved): 3 days.
4. The scaling: one factor per column name: AGE is multiplied by 11.16, WEIGHTBL divided by 6.39, AVAL divided by 8.79, in every dataset that has the column.
5. The shared code tables: one per kind, with the number of distinct values and the columns they cover (24 subject values from USUBJID and SUBJID together).
6. The files written.

Two columns of the example are not recognised by name and so are not scaled: CRCLBL (baseline creatinine clearance) stays as it was unless you add it with --scale-columns CRCLBL. As in run, check what was passed through.

4. The shared code tables, in practice

The same subject has the same code in every file:

ABC-PK-2024-0142-1001 is SUBJ001 in ADSL, ADEX, ADPC and ADLB. The treatment label Drug 50 mg is ARM_A wherever it occurs (in ARM and in TRT01P); the numeric TRT01PN is recoded as well, because it identifies a treatment too.

5. Dates: one whole-day shift

run2 draws the shift as a whole number of days (here 3). A date-only column (ADT, TRTSDT) and a date-time column (ADTM, EXSTDTC) therefore move by exactly the same amount, and the intervals between datasets (time since first dose, visit windows) remain real.

If you insist on a fractional value (--date-shift-days 23.6) the program warns that date-only columns will move by 23 days and date-time columns by 23.6, so an interval between a date-only and a date-time column will be off by 0.60 day. Use a whole number. The default window is 1–100, as in run; set your own with --date-shift-days 30-60.

6. ADaM names are recognised

On top of run's own role detection, run2 recognises ADaM and SDTM naming: names ending in DT, DTM, DTC, DTTM, DTMC or DTS are dates, names starting with TRT are treatments, and INVID and INVNAM are users. Only columns that run's detection leaves alone, and that you did not name yourself, are taken. --adam-hints false switches this off; the role options (--date-columns, --subject-column, ...) work as in run.

7. What run2 writes

Downloads (level 3 example, seed 7): csvadsl, csvadex, csvadpc, csvadlb, txtllm2.txt, txtsummary2.txt.

keymap2.txt

The key has the same rules as keymap.txt (tab-separated, # comments, private), organised for several datasets:

Download: txtexample-run2-keymap2.txt.

llm2.txt

The file lists every dataset, its columns and which of them are masked and how, followed by two prompts:

1. Build the master dataset: combine the datasets into one NONMEM-ready dataset, with rules that keep it reversible: treat identifiers as opaque codes, keep the ADaM column names, derive the numeric ID from the digits of the masked USUBJID (SUBJ007 gives 7), compute TIME from differences of dates, do not transform scaled columns in place, and say which source column each DV row came from.
2. Check the master dataset against the sources: every subject present, record counts reconcile, times in order, valid EVID/MDV/CMT/DVID combinations, and so on.

It ends with the command that restores the result (next page).

8. How run2 differs from run

• Levels 1 to 3 only. Level 4 re-sorts rows and gives every value its own factor, which could not be traced through a dataset an LLM has merged and re-ordered. The program refuses with a clear message.
• No --keep: the datasets have different columns. Use --drop (it applies to every dataset that has the column): --drop BRTHDT,COMMENT leaves ADSL with 15 columns.
• No renaming: the LLM has to see the ADaM names.
• One scaling factor per column name, not per dataset.

data-obfuscator run2 --input adam/adsl.csv --input adam/adex.csv --input adam/adpc.csv --input adam/adlb.csv --output ./x --level 4
[X] ERROR: level 4 is not available in run2: it re-sorts rows and gives every value its own random factor, which could not be traced back
    through a dataset that an LLM has merged and re-ordered. Use level 1-3

9. Options of run2

run2 takes the options of run where they apply:

Next

Restoring a run2 Master Dataset shows what happens after the LLM has built the master dataset: how reverse puts the real values back.