HyperPKPD version 0.0.0.12

Restoring a run2 Master Dataset

After run2, the LLM builds a master dataset from the masked files. This page shows how reverse puts the real values back into that dataset: the rows are new and in any order, some columns have new names, and the LLM may have made mistakes. The same reverse command is used; it recognises the key (keymap2.txt) by itself.

1. What the LLM hands back

Following the prompts of llm2.txt, the LLM combines ADSL, ADEX, ADPC and ADLB into one NONMEM-style dataset: one row per dose or observation, with ID, DATE, EVID, AMT, DV, DVID, WT, ... and the masked identifiers it kept.

Which source column each master column comes from, and the first rows of master_masked.csv
Figure 2.15.1. The master dataset the LLM builds from the masked files.

Notice that the master dataset has new rows (264 of them, none in any source file), new columns (ID, EVID, DVID), and columns with new names (DV is AVAL of ADPC and ADLB; WT is WEIGHTBL of ADSL). Nothing can be matched by row. So reverse restores every column on its own, from the key.

Download the example: csvexample-master_masked.csv.

2. How each column is restored

Column of the master dataset How it is restored
kept its ADaM name (USUBJID, STUDYID, ARM, SEX, ...) the code is looked up in the shared table. A column with another name is also recognised when at least 80 % of its values are codes of one table
a date column (ADT, DATE, ...) the shift is taken off; a date or date-time in an unnamed column is found by its look. --date-columns names more
a scaled column that kept its name the exact original number
renamed or merged by the LLM (DV from AVAL, WT from WEIGHTBL, ID from the digits of USUBJID) tell reverse with --map COMBINED=SOURCE
anything the LLM derived (a log of DV, a unit conversion, a new category) cannot be recognised: left as it is

3. --map: where did this column come from?

Anatomy of --map DV=AVAL:DVID=1 and four example rules
Figure 2.15.2. The --map option: COMBINED=SOURCE, optionally with a condition.
--map COMBINED=SOURCE
--map COMBINED=SOURCE:WHENCOL=VALUE
  • COMBINED is the column in your master dataset; SOURCE is the column in the key (the original ADaM name).
  • :WHENCOL=VALUE is optional: use it when one master column takes values from sources that were scaled differently. For example, if DV holds concentrations (PCSTRESN) where DVID = 1 and laboratory values (LBSTRESN) where DVID = 2: --map DV=PCSTRESN:DVID=1 --map DV=LBSTRESN:DVID=2.
  • ID=USUBJID is special: a number built from the digits of the masked code (SUBJ007 becomes 7) is accepted, and the real USUBJID is written.
  • The option can be repeated.

The three rules needed for the example:

Rule Meaning
--map ID=USUBJID ID is the subject number built from USUBJID
--map WT=WEIGHTBL WT is the scaled WEIGHTBL of ADSL
--map DV=AVAL DV is AVAL, scaled with one factor (the same in ADPC and ADLB)

4. First try: no --map

data-obfuscator reverse --input master_masked.csv --keymap keymap2.txt --output ./restored
Terminal: reverse without --map; the warning says that ID matches the digits of the subject codes and suggests --map ID=USUBJID
Figure 2.15.3. Without --map the program still restores what it recognises, and tells you what to add.

The program recognised the key (keymap2.txt: level 3, four datasets, seven identifier tables, three scaled columns, six date columns), restored what kept its name (USUBJID, ARM, the date column) and warned: column ID holds whole numbers that match the digits of the masked subject codes (12 of 12 values); if it is a subject number built from USUBJID, restore it with --map ID=USUBJID. It also lists the columns of the key that are not in the master dataset, as a hint at what could have been renamed.

5. With --map

data-obfuscator reverse --input master_masked.csv --keymap keymap2.txt --output ./restored \
  --map ID=USUBJID --map WT=WEIGHTBL --map DV=AVAL --strict
Terminal: reverse with three --map rules and --strict
Figure 2.15.4. With --map and --strict: no warning, every master column restored.

No warnings, and --strict (stop on anything uncertain) did not stop: every value could be traced. The files written are restored.csv, reverse_report.txt and reverse.log, as for run.

The result

Four rows of the master dataset before and after reverse
Figure 2.15.5. master_masked.csv becomes the real master dataset.

In the figure, ID 1 became the real subject identifier ABC-PK-2024-0142-1001 (the --map ID=USUBJID rule), the date is back three days (2024-03-16T08:00:00 to 2024-03-13T08:00:00), DV 0.29966054 became 2.6335 (the original concentration), the weight 12.937247 became 82.7, and ARM_A became Drug 50 mg.

The report

reverse_report.txt lists, column by column, how each was restored, and what cannot be:

Columns of the dataset
  ID               identifier (subject) via --map  restored 264 value(s)
  USUBJID          identifier (subject)  restored 264 value(s)
  DATE             date found by its values  restored 264 value(s)
  EVID             unchanged
  AMT              unchanged
  DV               scaled (AVAL) via --map  restored 240 value(s)
  DVID             unchanged
  WT               scaled (WEIGHTBL) via --map  restored 264 value(s)
  ARM              identifier (treatment)  restored 264 value(s)

Cannot be restored (removed when masking)
  adsl.csv: COMMENT - free text redacted (.)

Note DV restored 240 value(s): the 24 dosing rows have no DV (.), which is not a number to restore. Download the files: csvexample-master_restored.csv and txtexample-reverse_report-run2.txt.

6. Does it match the truth?

The same master dataset was also built from the original (unmasked) ADaM files, and the restored dataset was compared with it value by value:

Table: every column of the restored master dataset equals the master built from the originals, 264 of 264
Figure 2.15.6. Restored versus a master built from the originals.

All nine columns are equal in all 264 records. That holds because the LLM kept the values as they were; values it derived (a log of DV, a new category) cannot be recognised and stay as they are.

7. When the LLM changed or invented something

If the master dataset holds a code that is not in the key, reverse leaves it unchanged and says so; with --strict it stops:

Without --strict a warning and a restored file; with --strict an error and no file
Figure 2.15.7. A subject code (SUBJ999) that does not exist in the key.
[WARN] column USUBJID: 1 value(s) are not codes of the masked datasets (for example 'SUBJ999') and were left unchanged - the LLM changed or invented them

Look into such rows: they may be hallucinated records, or a code the LLM typed wrongly.

8. Choosing the steps

--only and --skip take recoding, dates and scaling for a run2 key (there is no renaming step):

data-obfuscator reverse --input master_masked.csv --keymap keymap2.txt --map ID=USUBJID --only recoding,dates --output ./restored_ids

This restores identifiers and dates but leaves the scaled numbers (DV, WT) masked.

9. Making the restore work: advice for the prompt

llm2.txt already asks the LLM for what reverse needs. If you write your own prompt, ask for the same:

  1. Keep the ADaM column names for the masked identifier columns (USUBJID, STUDYID, ARM, ...) in the master dataset.
  2. Derive the numeric ID from the digits of the masked USUBJID (SUBJ007 gives 7); do not renumber 1 to N.
  3. Do not transform scaled columns in place (AVAL, WEIGHTBL, AGE): put derived values in new columns.
  4. Say where DV came from (a DVID or SRC column), so a --map ... :DVID=1 rule can be written when sources were scaled differently.
  5. Keep dates in ISO format (YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS) and under their original column name.

10. What cannot be restored

  • Free text that was redacted and columns that were dropped (keymap2.txt section 5 lists them).
  • Values the LLM derived or transformed: a log of DV, a unit conversion, a recoded SEX.
  • A master column that mixes several scaled source columns without a --map ...:WHENCOL=VALUE rule.
  • Invented codes: reported, left as they are.

keymap_columns.txt is not used by run2 (there is no level 4). Treat the restored master dataset, like any restored file, as real data: do not share it, nor reverse_report.txt.

11. Quick reference

Option What it does for a run2 key
--keymap FILE keymap2.txt (found next to the input when there is no keymap.txt)
--map COMBINED=SOURCE[:WHENCOL=VALUE] where a renamed or merged column came from; repeatable
--date-columns COLS extra date columns of the master dataset
--only STEPS, --skip STEPS recoding, dates, scaling
--strict stop on values the key does not know
--output DIR, --overwrite as for run

Where to go next

You have now seen every command. The About the Project section says which version this documentation belongs to and who wrote the program; README.md and CONFIGS.md in the project folder are the complete reference.