Advanced Library Use

The Python library is for people who want to use SynthPopCan inside notebooks, scripts, research pipelines, or teaching materials. The command line remains the friendliest surface for one-off work, but the library provides a small beginner API for common workflows and lower-level modules for research code that needs more control.

This section does not repeat the methodological discussion from the command line chapters. If we are new to a modelling approach, start with the corresponding command-line page first:

  • IPF explains calibration, impossible controls, non-convergence, and the interpretive limits of a successful fit.

  • Controls explains normalized control tables and category mappings.

  • Statistics Canada Sources explains WDS and Census Profile source discovery.

  • Tree Models explains household/person generation, tree and forest concepts, support, purity, and model quality.

  • Validate explains what validation reports do and do not prove.

Start with Getting Started With the Beginner API unless we already know we need lower-level objects such as IPFMargin, ControlTable, or FrequencyTreeModel. This page is the advanced guide to the full library surface.

Import Style

For longer research code, import from the module that owns a concept:

from pathlib import Path

from synthpopcan.controls import read_control_table
from synthpopcan.ipf import expand_records, fit_ipf

The module imports are usually clearer in longer research code because they show whether a step belongs to controls, IPF, microdata, tree modelling, or validation.

Use synthpopcan.api or import synthpopcan as spc for the beginner-friendly workflow functions. Use modules such as synthpopcan.ipf, synthpopcan.controls, and synthpopcan.tree when we need lower-level objects or advanced options.

Controls

Controls are the public library representation of target totals. A ControlTable contains one or more ControlMargin objects; each margin contains ControlCell objects with category labels and counts. Use these objects when we want to inspect or transform controls before fitting.

from pathlib import Path

from synthpopcan.controls import read_control_table, write_control_table

controls = read_control_table(Path("controls.csv"))

for margin in controls.margins:
    print(margin.name, margin.dimensions, len(margin.cells))

ipf_margins = controls.to_ipf_margins()
write_control_table(Path("normalized-controls.csv"), controls)

For source-specific control preparation, the library exposes the same adapters used by the command line:

from pathlib import Path

from synthpopcan.controls import (
    build_wds_category_mapping_template,
    inspect_wds_zip,
    read_wds_control_table,
)

inspection = inspect_wds_zip(Path("table.zip"))
print(inspection["dimension_candidates"])

template = build_wds_category_mapping_template(
    Path("table.zip"),
    dimensions=("Age group", "Gender"),
    preset="canonical",
)

controls = read_wds_control_table(
    Path("table.zip"),
    dimensions=("Age group", "Gender"),
    count_column="VALUE",
    margin_name="age_sex",
    category_mapping=template,
)

The category names in controls must match the category names in the seed records that will be fitted. The Controls chapter has more detail on category mappings and common normalization problems.

IPF

The IPF library surface is deliberately small. IPFMargin describes the target cells, fit_ipf calibrates seed-record weights, and IPFResult stores the records, weights, convergence flag, iteration count, and maximum residual.

from pathlib import Path
import csv

from synthpopcan.controls import read_control_table
from synthpopcan.ipf import expand_records, fit_ipf, validate_margin_coverage

with Path("seed.csv").open(newline="") as handle:
    seed_records = list(csv.DictReader(handle))

controls = read_control_table(Path("controls.csv"))
margins = controls.to_ipf_margins()

validate_margin_coverage(seed_records, margins)
fit = fit_ipf(
    seed_records,
    margins,
    weight_field="WEIGHT",
    max_iterations=250,
    tolerance=0.01,
)

print(fit.converged, fit.iterations, fit.max_abs_error)

expanded_rows = expand_records(seed_records, fit.weights)

If validate_margin_coverage or fit_ipf raises a ValueError about a target cell with no seed records, the controls cannot be represented by the seed sample as given. The IPF chapter discusses this as a modelling problem, not just a software error.

Microdata

Microdata helpers load supported seed samples and export columns for IPF or tree training. The library does not hide that these are source-specific adapters: different Statistics Canada files encode households, people, geography, and weights differently.

from pathlib import Path

from synthpopcan.microdata import (
    export_seed_rows,
    read_statcan_2016_hierarchical_seed_sample,
)

sample = read_statcan_2016_hierarchical_seed_sample(Path("hierarchical.csv"))
print(sample.as_summary())

seed_rows, manifest = export_seed_rows(
    sample,
    columns=("AGEGRP", "SEX", "PR"),
)
print(manifest["rows_written"], manifest["columns"])

For tree modelling, first ask the source adapter which column blocks are available, then resolve the blocks into explicit target and conditioning columns. This example keeps the exported rows in memory so we can inspect the design before writing files:

from synthpopcan.microdata import (
    export_training_rows,
    resolve_tree_column_block_pair,
    suggest_tree_column_blocks,
)

suggestions = suggest_tree_column_blocks(sample)
print([block["name"] for block in suggestions["blocks"]])

(
    household_targets,
    household_conditions,
    person_targets,
    person_conditions,
    design_report,
) = resolve_tree_column_block_pair(
    sample,
    household_block="household_core",
    person_block="person_demographics",
)

person_training, person_manifest = export_training_rows(
    sample,
    level="person",
    target_columns=person_targets,
    conditioning_columns=person_conditions,
)

Statistics Canada Sources

The statcan module is for discovery and download automation. It wraps Statistics Canada WDS endpoints and supported 2016 Census Profile bulk files with small Python functions that return plain dataclasses, dictionaries, and paths.

from pathlib import Path

from synthpopcan.statcan import (
    fetch_wds_metadata,
    fetch_wds_table,
    search_wds_tables,
    summarize_wds_metadata,
)

matches = search_wds_tables("age sex population", limit=5)
for match in matches:
    print(match.product_id, match.title_en)

metadata = fetch_wds_metadata(matches[0].product_id)
summary = summarize_wds_metadata(metadata)
print(summary["ipf_suitability"])

zip_path = fetch_wds_table(matches[0].product_id, Path("data/raw/statcan/wds"))

Live source functions depend on Statistics Canada service availability and may raise network or source-format errors. In reproducible research scripts, store the downloaded source files and provenance manifests rather than relying on live downloads during every run.

Tree Models

The tree library exposes two model families: FrequencyTreeModel, which stores conditional aggregate outcomes, and CartTreeModel, which stores a serialized scikit-learn CART classifier. The command-line Tree Models chapter discusses the methodological risks; the library API gives us the objects needed to train, audit, serialize, and generate from those models.

When we are ready to train from a file, the training CSV should contain the same target and conditioning columns chosen in the microdata step above. The command-line microdata export-training examples show one way to write that CSV.

from pathlib import Path

from synthpopcan.tree import (
    audit_tree_model,
    generate_tree_rows,
    read_tree_training_sample,
    train_frequency_model,
    write_tree_model,
)

sample = read_tree_training_sample(
    Path("person-training.csv"),
    level="person",
    target_columns=("AGEGRP", "SEX"),
    conditioning_columns=("PR", "household_size"),
    weight_column="WEIGHT",
)

model = train_frequency_model(sample, random_seed=42, min_support=10)
audit = audit_tree_model(model, min_support=50, max_purity=0.95)
print(audit["passed"], audit["summary"])

rows = generate_tree_rows(
    model,
    rows=1000,
    conditions={"PR": "24", "household_size": "2"},
    random_seed=42,
)

write_tree_model(Path("person-model.json"), model)

Linked household/person generation uses one household model and one person model. The shared conditioning columns on the person model must be available in generated household rows.

from synthpopcan.tree import generate_linked_population, validate_linked_population

# household_model and person_model can be read from JSON with read_tree_model()
# or created earlier in the script with train_frequency_model().
households, persons = generate_linked_population(
    household_model,
    person_model,
    households=500,
    household_conditions={"PR": "24"},
    random_seed=42,
)

link_report = validate_linked_population(households, persons)
print(link_report["passed"])

Maintainer Package Workflow Script

Training, reviewing, and bundling a model package from restricted microdata is not part of the beginner API. It is still useful to keep that workflow in Python when the goal is a reproducible release artifact rather than a one-off interactive command.

The repository includes scripts/build_all_model_packages.py, which builds every province and PUMF-coded CMA package in a single run:

uv run python scripts/build_all_model_packages.py

Pass --only to build a subset:

uv run python scripts/build_all_model_packages.py --only quebec-2016 ontario-2016

That script uses library modules directly to:

  • read the local 2016 hierarchical PUMF once, then filter per geography;

  • resolve all currently supported household and person column blocks;

  • train linked conditional-frequency household and person models;

  • audit the private working models and block on any release-blocking issues;

  • write publishable-candidate model copies and release manifests;

  • write a linked package JSON under data/private/model-release-assets/ ready for upload as a GitHub Release asset.

Ordinary installs never run this script — they fetch packages on demand via synthpopcan models fetch.

The full script is at scripts/build_all_model_packages.py in the source checkout.

Validation

Validation functions return JSON-serializable dictionaries so they can be saved, printed, tested, or rendered in notebooks. They are checks on a particular artifact, not a claim that a synthetic population is substantively correct.

from synthpopcan.validation import build_control_validation_report

report = build_control_validation_report(
    controls,
    fit.records,
    fit.weights,
    tolerance=0.01,
    artifact_kind="weights",
)

if not report["passed"]:
    for issue in report["issues"][:5]:
        print(issue["message"])

Tree-output validation compares generated distributions with a training view:

from synthpopcan.validation import build_tree_output_validation_report

report = build_tree_output_validation_report(
    training_rows=person_training,
    generated_rows=rows,
    target_columns=person_targets,
    conditioning_columns=person_conditions,
    weight_field="WEIGHT",
    tolerance=0.05,
)

See Validate for the interpretive caveats that should accompany validation reports in research notes.

API Reference

The API Reference is generated from docstrings with Sphinx autodoc. It is the place to look for signatures, return types, and member-level notes after we understand the workflow-level concepts above.