Dates and Free Text

Two things that identify a person without being an identifier: when things happened and what people typed. From level 2 on, the program shifts every date by one constant and redacts free text. This page shows how, with the options that control both.

1. The date shift

One shift is drawn for the whole dataset and added to every date column, so the gaps between dates are preserved: visit intervals, time on study and the order of events survive; the calendar does not.

Which columns are dates is decided from the name (DATE, DATETIME, names ending DTC or _DT, DOB, VISITDT, ...) or by --date-columns. Level 1 does not shift dates and warns if it finds a date column.

Choosing the shift: --date-shift-days

The first subject's date, 2024-03-11, becomes 2024-04-30 (default, seed 2026, shift 50.8 days), 2024-04-03 (--date-shift-days 23.6) and 2024-11-16 (--date-shift-days 200-300, shift 250.3 days).

Choosing a window. The default window of 1–100 days is small. When the real calendar period matters (before sharing outside your organisation), widen it: --date-shift-days 365-1460 moves dates by one to four years. A negative window is written -30--5.

The exact shift is written in keymap.txt (section 3):

date_shift_days	23.6	DATE
date_shift_whole_days	23	DATE

Never share it; summary.txt and config.used.txt state it too.

2. Formats the program understands

Not every date looks alike, and a fractional shift (23.6 days) cannot be applied to a text that has no time of day. The rule: values with a time of day move by the exact shift; date-only values move by the whole days (floor).

A cell that cannot be read as a date is left unchanged, and the program says so, because that cell still shows the real value:

[WARN] date column DATE: 1 value(s) are not in a recognised date format and were left UNSHIFTED (first at data row 7)
       - they still show the real date; fix or drop them before sharing

Fix or remove such cells before sharing. You can reproduce all of the above with a small file that has one date of each kind; the program was run on such a file for this page.

3. Free text

Free text is where a name, a phone number or a site slips out ("Nurse comment: late sample, pager 555-0142"). Columns named like free text (COMMENT, NOTE, EMAIL, PHONE, NAME, INITIALS, SUBJINIT, ADDRESS, ...) are treated as such, and you can add more with --freetext-columns.

data-obfuscator run --input pk_study.csv --output ./masked_nt --level 2 --seed 2026 --drop-freetext

This writes 19 columns instead of 22 (the three free-text columns are gone). To treat a column the program does not recognise as free text, name it: --freetext-columns COLUMN1,COLUMN2. A free-text column named in --keep is kept.

Neither redacted nor dropped text can be restored: the original text is not stored in the key. After a restore the column is . (or absent) and the report names it: these source columns cannot be restored because their values were never stored: COMMENT, EMAIL, SUBJINIT.

4. Practical advice

• Look at column_report.txt: any text column that is passed through may hold identifying text; name it with --freetext-columns or drop it.
• A clock time such as 08:15 is not treated as text.
• When in doubt, --drop-freetext is stronger than redaction: the column is not in the file at all.
• If you need to keep the real calendar for analysis, share at level 1 inside the team only; at level 2 the shift is the price of leaving the building.

Next

Scaling Covariates covers level 3 and 4: multiplying and dividing numbers by hidden factors.