CSPro2csv

CSPro2csv is a lightweight converter for multidimensional tabulations produced with CSPro. It transforms CSPro tabulation files (.tbw) into structured CSV datasets that can be loaded into a database or reused by downstream dissemination tools.

The tool was developed in the context of the KNBS-Istat cooperation project on the dissemination of Kenya 2019 Population and Housing Census data. Its design is intentionally modest: it does not replace full statistical dissemination platforms, but fills a specific gap between CSPro tabulations and machine-readable data.

Repository Contents

• src/cspro2csv/: reusable Python module and command line interface.
• cspro2csv.ipynb: short notebook example using the module.
• data/CensusMainShortForm.dcf: CSPro dictionary used as metadata source.
• data/population_by_ict_age_and_sex.tbw: example tabulation by territory, ICT age group, and sex.
• data/population_by_sex_and_marital_status.tbw: example tabulation by territory, sex, and marital status.
• tests/: parser and end-to-end conversion tests.

Installation

The converter uses only the Python standard library. Python 3.10 or newer is recommended.

For regular use from a cloned repository:

python3 -m pip install -e .

For a one-off run without installation, set PYTHONPATH=src before invoking the module.

Usage

Convert the ICT age group by sex example:

python3 -m cspro2csv \
  --table data/population_by_ict_age_and_sex.tbw \
  --dictionary data/CensusMainShortForm.dcf \
  --output output/population_by_ict_age_and_sex.csv

Convert the sex by marital status example:

python3 -m cspro2csv \
  --table data/population_by_sex_and_marital_status.tbw \
  --dictionary data/CensusMainShortForm.dcf \
  --output output/population_by_sex_and_marital_status.csv

Without installation:

PYTHONPATH=src python3 -m cspro2csv \
  --table data/population_by_ict_age_and_sex.tbw \
  --dictionary data/CensusMainShortForm.dcf \
  --output output/population_by_ict_age_and_sex.csv

Output Schema

The generated CSV preserves the structure used by the original project pipeline:

• id: sequential row identifier.
• one column for each CSPro dimension, containing the dimension code.
• VALUES: tabulated cell value.
• DESC_<dimension>: human-readable label for each dimension code.

For example, the ICT age group by sex table produces columns such as SYSTEM_TOTAL, P12_VS4, P11_VS1, VALUES, DESC_SYSTEM_TOTAL, DESC_P12_VS4, and DESC_P11_VS1.

Methodological Note

CSPro2csv follows a metadata-driven approach: the CSPro dictionary is treated as the source of code lists and labels, while the .tbw file provides the multidimensional table structure and cell values. This keeps the transformation close to workflows already used by national statistical offices and reduces manual re-keying of metadata.

In the broader dissemination pipeline, CSPro2csv is the conversion step between CSPro tabulation and the Data Dissemination Database (DDB). Once loaded into the DDB, the data can be exposed through APIs and explored in the Open Data Browser (ODB).

Tests

Run the test suite from the repository root:

PYTHONPATH=src python3 -m unittest discover -s tests

The tests check parsing of the two example tabulations, extraction of known value sets from the CSPro dictionary, and end-to-end CSV conversion.

License

This project is distributed under the GNU General Public License v3.0. See LICENSE for the full license text.