Microdata

Microdata commands turn source-level census records into the smaller working tables that SynthPopCan can fit, train, inspect, and validate. We use them when we need to move from a local census file to either:

• an IPF seed CSV, where each row can receive a fitted weight; or

• a tree-training CSV, where selected columns become targets and conditioning variables for a model.

The command group is deliberately cautious. It inspects structure, checks household-level assumptions, and writes derived CSVs without printing private rows by default. That matters because census microdata can be restricted, sensitive, or simply easy to misread. A derived file is not neutral: the chosen columns, level, geography, and row weights all become part of the research argument. If you export only age group and sex, the seed can only carry those relationships forward — household income, language, immigration status, and dwelling type will be absent from any generated output. That is a choice, and it should be a deliberate one.

Concept

Public aggregate tables tell us counts by category. Microdata gives us example records with relationships among fields. Synthetic-population workflows often need both: microdata gives row structure, while controls or model conditions give the target population context.

For IPF, the microdata export creates seed rows. IPF can only reweight those rows, so the seed has to contain the columns used in the controls. If a control uses AGEGRP and SEX, the seed must contain AGEGRP and SEX.

For tree models, the microdata export creates training rows. Target columns are the fields the model will generate; conditioning columns are the fields we use to choose an appropriate distribution. The tree documentation explains the model risks in more detail, especially low support, high purity, and overfit geography-specific models.

SynthPopCan currently exposes two adapter formats in the CLI:

• statcan-2016-hierarchical for the Statistics Canada 2016 hierarchical PUMF shape used by the current local workflow; and

• fixture-v1 for small test and demonstration files.

The 2016 hierarchical PUMF is a person-row file with household and family identifiers such as HH_ID, EF_ID, CF_ID, and PP_ID. Household-level exports are derived from that person-row file and are only valid when selected household columns are constant within each household.

Getting Started

Always inspect the file first. The inspect command reports row and household counts, detected identifier and weight columns, available geography columns, and the full column list — without printing any source rows. Use this output to confirm the file loaded correctly and to identify which columns are available before choosing what to export.

synthpopcan microdata inspect \
  tests/fixtures/workflows/linked_tree/hierarchical.csv \
  --input-format statcan-2016-hierarchical

For an IPF seed (person level): export the columns your controls use. IPF can only reweight rows that exist in the seed, so the seed must contain every column referenced in any control margin. The weight column is included automatically.

synthpopcan microdata export-seed \
  tests/fixtures/workflows/linked_tree/hierarchical.csv \
  --input-format statcan-2016-hierarchical \
  --columns AGEGRP,SEX \
  --out seed.csv

For an IPF seed (household level): household columns in a person-row file should be constant within each household — tenure is the same for all people in a household, but age is not. Run check-seed first to confirm this holds for the columns you want. If any column varies within a household, a household-level export would silently discard that variation.

synthpopcan microdata check-seed \
  tests/fixtures/workflows/linked_tree/hierarchical.csv \
  --input-format statcan-2016-hierarchical \
  --level household \
  --columns TENUR

synthpopcan microdata export-seed \
  tests/fixtures/workflows/linked_tree/hierarchical.csv \
  --input-format statcan-2016-hierarchical \
  --level household \
  --columns TENUR \
  --out household-seed.csv

For a tree model (training export): the column choices here are a modelling decision, not just a technical step. suggest-tree-columns gives you a starting point organized by block (household core, person demographics, etc.) based on what the adapter considers safe and coherent at each level. Review the suggestions before accepting them — a block that is sensible at the provincial level may be too detailed for a small geography.

synthpopcan microdata suggest-tree-columns \
  tests/fixtures/workflows/linked_tree/hierarchical.csv \
  --input-format statcan-2016-hierarchical

Export a training view only after deciding which fields belong in the model. Target columns are what the model will generate; conditioning columns are what it conditions generation on.

synthpopcan microdata export-training \
  tests/fixtures/workflows/linked_tree/hierarchical.csv \
  --input-format statcan-2016-hierarchical \
  --level person \
  --target-columns AGEGRP,SEX \
  --conditioning-columns TENUR,household_size \
  --out person-training.csv

Subcommands

microdata inspect

Inspects a microdata file without printing source rows. Reports row count, household count, detected identifier columns (HH_ID, PP_ID, etc.), the weight column, available geography columns, and the full column inventory. Use this before any export to confirm the file is loaded correctly and to identify which columns are available for seed or training export.

synthpopcan microdata inspect hierarchical.csv \
  --input-format statcan-2016-hierarchical \
  --format table

Important options:

• --input-format: source adapter, currently statcan-2016-hierarchical or fixture-v1.

• --level: required for fixture-v1, where the fixture must be declared as household or person.

• --weight-column, --geo-columns, and --id-columns: fixture-oriented options used when a small local file needs explicit metadata.

• --format json|table: choose machine-readable or reader-facing output.

microdata check-seed

Checks whether selected household columns can be safely exported as seed rows from a statcan-2016-hierarchical file.

synthpopcan microdata check-seed hierarchical.csv \
  --input-format statcan-2016-hierarchical \
  --level household \
  --columns TENUR

Use this before household-level IPF. The check reports whether selected columns and WEIGHT are constant within HH_ID. If they are not constant, a household seed would collapse person-row differences into one household row and should not be exported without another modelling decision.

microdata export-seed

Exports selected microdata columns as an IPF seed CSV. The weight column is always included. Only columns named in --columns appear in the output — any column your controls reference must be listed here, or IPF will fail with a missing-column error.

synthpopcan microdata export-seed hierarchical.csv \
  --input-format statcan-2016-hierarchical \
  --columns AGEGRP,SEX \
  --out seed.csv

For statcan-2016-hierarchical, person-level export is the default. Use --level household when you intentionally want one row per household. Run check-seed first to confirm the selected columns are constant within each household — if they are not, the export will collapse variation silently.

synthpopcan microdata export-seed hierarchical.csv \
  --input-format statcan-2016-hierarchical \
  --level household \
  --columns TENUR \
  --out household-seed.csv

For fixture-v1, --level is required because the fixture file does not carry the same adapter assumptions as the StatCan hierarchical source.

microdata suggest-tree-columns

Suggests broad household and person column blocks for tree modelling.

synthpopcan microdata suggest-tree-columns hierarchical.csv \
  --input-format statcan-2016-hierarchical

Treat this as a starting point, not as an automatic model design. A suggested block can still be too detailed for a small geography or unsuitable for a particular humanities question. See Tree Models for the longer discussion of support, purity, forests, and bad models.

For broad linked tree models, tree train-linked also accepts --household-block all and --person-block all. These combine the source adapter’s currently supported household or person blocks. They do not mean “every raw column”: identifier, weight, replicate-weight, geography, and person-varying columns are excluded from the household side unless a supported source adapter derives a household-level version.

microdata tree-geography-feasibility

Reports how many person and household rows each geography value has, and flags which geographies are likely viable, borderline, or too sparse for the chosen column blocks at publishable quality thresholds. Use this before committing to a geography in tree train-linked — a geography with too few rows will produce a model with low support and high purity that fails the audit step.

The output groups geographies into likely viable, borderline (may need a reduced column profile or private-only use), and too sparse (should not be trained at publishable thresholds with the chosen blocks). A borderline geography is not automatically excluded; it may still work with a reduced target profile or a broader category scheme.

synthpopcan microdata tree-geography-feasibility hierarchical.csv \
  --input-format statcan-2016-hierarchical \
  --geo-column PR \
  --household-block household_core \
  --person-block person_demographics

Important options:

• --likely-person-rows and --likely-household-rows: thresholds for geographies that look plausibly supported.

• --borderline-person-rows and --borderline-household-rows: thresholds for geographies that may need reduced targets or private-only use.

• --min-support and --max-purity: release-oriented model-risk thresholds aligned with the tree audit commands.

microdata export-training

Exports selected microdata columns as tree-training rows.

synthpopcan microdata export-training hierarchical.csv \
  --input-format statcan-2016-hierarchical \
  --level household \
  --target-columns household_size,TENUR \
  --conditioning-columns PR \
  --out household-training.csv

The --target-columns are generated by the model. The --conditioning-columns shape which target distributions are used. If a conditioning column is too detailed, the model may have very small groups; if it is too broad, the generated rows may miss patterns that matter for the research question.

Troubleshooting

The command says the input format is unsupported: check the adapter name. The current CLI accepts statcan-2016-hierarchical for the main census microdata workflow and fixture-v1 for small local fixtures.

Household seed export fails: run microdata check-seed and look for columns that vary within HH_ID. We should either remove those columns, derive a household-level summary explicitly, or stay with person-level seed rows.

IPF later reports missing columns: re-run microdata export-seed with the columns used by the controls. IPF cannot fit a control dimension that is absent from the seed CSV.

Tree training looks sparse: run microdata tree-geography-feasibility, reduce the target profile, combine categories, or train for a larger geography. The tree page discusses why low support and high purity are warning signs.

The output should be used in automation: pass --format json on inspection and check commands. CSV exports remain ordinary CSV files.

Further Reading

• See IPF for how exported seed rows are fitted to controls.

• See Tree Models for how exported training rows become flat or linked tree models.

• See Field Primer for the broader discussion of microdata, disclosure-risk framing, and synthetic-population methods.