COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives

COCOLogic-V2 is an object-centric dataset for visual inductive reasoning on real-world images. Built on MSCOCO, it frames reasoning as a multilabel classification task over 10 compositional first-order-logic rules (object presence/absence, counting, and count comparisons). Samples of each rule are divided into different positive variants, as well as types of near-boundary (NB) negatives, and the typically easy far-from-boundary (FB) negatives. These annotations enable automated insights for model accountability: they reveal whether a model has actually learned a logical rule or is merely exploiting coarse statistical shortcuts.

This repository contains the dataset splits and all code used to train and evaluate the models reported in the accompanying paper.

---

The COCOLogic-V2 Dataset

Motivation

COCOLogic-V2 extends the original COCOLogic dataset in two ways:

• Multilabel reframing. Each image is annotated for all 10 rules simultaneously. This removes the single-label shortcuts and class imbalance of the original (where, e.g., a single object could give away a class) and isolates the logical difficulty of each rule.
• Broader logical scope. Rules cover a subset of first-order logic including propositional logic, object counting, counting comparisons, and object absence.

Most importantly, a new sampling procedure produces truly hard negatives (NB samples) that sit right next to each rule's decision boundary, alongside labelled positive variants — so model behaviour can be diagnosed automatically, without manual inspection.

The 10 rules

#	Name	Logical rule
1	Signal and Ride	traffic light ∧ one of {bicycle, bus, train}
2	Double Serving	Exactly two categories of {bottle, cup, pizza}
3	Herd Alone	At least two objects of the same category of {cow, elephant, sheep} ∧ no person
4	Either Dog or Car	Either dog or car
5	Three of a Kind	Exactly three bowl ∨ exactly three cup
6	Car Majority	More car than truck ∧ at least one of each
7	Empty Seat	(couch ∨ chair) ∧ no person
8	Single Mode Traffic	Exactly one category of {bicycle, motorcycle, car, bus}
9	Personal Transport	person ∧ (either bicycle or car)
10	Surf Trip	Exactly as many person as surfboard ∧ at least one of each

Rules are grounded in object co-occurrences over the 91 MSCOCO categories (constants.py: NUM_RULES = 10, NUM_CATEGORIES = 91).

Diagnostic sample taxonomy: variants, NB, FB

Every sample is categorized per rule into one of three groups:

• Positive variants — the distinct ways a rule can be satisfied. Each rule is converted to disjunctive normal form (DNF); each disjunct is one positive variant. The full positive set is the union of all variants.
• Near-boundary (NB) negatives — hard negatives derived from a DNF disjunct by flipping a single literal (or, for counting rules, by changing the required object count). These sit just outside the rule and are the key probe of true rule understanding.
• Far-from-boundary (FB) negatives — easy negatives drawn from the remaining MSCOCO images, kept so the overall distribution stays close to MSCOCO's.

Worked example — "Double Serving" (exactly two of {bottle, cup, pizza}):

• Positive variants: bottle ∧ cup ∧ ¬pizza, bottle ∧ pizza ∧ ¬cup, cup ∧ pizza ∧ ¬bottle.
• NB types: the three single-category cases (bottle ∧ ¬cup ∧ ¬pizza, …) plus the all-three case (bottle ∧ cup ∧ pizza).

The complete enumeration of every positive variant and NB type for all 10 rules is given in Appendix A.1 of the paper.

Splits and sizes

Version	Train	Test	Source split	Notes
COCOLogic-V2 (full)	25,000	3,500	MSCOCO train / val	Sampled to balance positive variants, NB types, and FB negatives.
COCOLogic-V2-FS (few-shot)	239	368	curated	24 train samples per rule (8 positive + 16 negative); 40 test per rule (20 + 20).

Sampling (full version). For each split: (1) up to 1,000 images per positive variant are sampled (with cross-rule duplicates removed where possible); (2) each NB type is filled to at least 500 samples where data allows; (3) FB samples are added from the remaining MSCOCO images to keep the distribution close to MSCOCO's. Test images are drawn from the MSCOCO validation split.

Few-shot version. COCOLogic-V2-FS assumes a working perception module — images are manually curated so that relevant objects are clearly visible and labels are correct. It is intended for few-shot / in-context rule learning, not for training perception from scratch. The samples per rule specified in the "rules" part of the JSON for the few-shot version.

The splits ship as JSON in cocologicv2_data/. Each image entry carries its 10-dim labels, the 91-dim COCO categories counts, per-rule rule_variants, and boundary_type annotations (positive variant / NB type / FB

---

Installation

The code is PyTorch-based.

pip install -r requirements.txt

Key dependencies: torch, torchvision, open_clip_torch (CLIP concept encoder), torch-explain (Concept Embedding + Concept Reasoning layers for DCR), scikit-learn (metrics), and wandb (experiment tracking).

A ready-made container is provided under .docker/ and .devcontainer/ (base image pytorch/pytorch:2.7.0-cuda12.8-cudnn9).

Data prerequisites:

• Dataset splits are bundled in cocologicv2_data/
• MSCOCO images must be downloaded from https://cocodataset.org/#home and are expected to be at datasets/coco/images (overridable via --base_path)
• Experiment tracking uses Weights & Biases (project COCOLogic-V2). Run wandb login, or set WANDB_MODE=offline to disable uploads.

---

Data & preprocessing artifacts

Most models do not consume raw images directly; they rely on precomputed concept artifacts in preprocessings/:

Artifact	Produced by	Used by
coco_pos_weights*.json	generate_class_weights.py	BCEWithLogitsLoss positive weights for the 91 categories (concept encoder, DCR concepts)
coco_rule_weights*.json	generate_class_weights.py	per-rule positive weights for the 10-rule task loss
od_*_category_counts.pt	generate_detector_counts.py	Mask-RCNN detection counts → detector_* models
ocb_*_concepts.pt	generate_ocb_concepts.py	per-object CLIP concepts → ocb_* models
concept_vocabs/coco_categories.json	—	text concept vocabulary for CLIP-CBM and OCB
model_weights/concept_encoder.pt	--train_concept_encoder run	frozen backbone for supervised CBMs
model_weights/dcr_seed42.pt	a --save_dcr_path DCR run	inspecting learned DCR rules

The *_fewshot* variants are the equivalents for COCOLogic-V2-FS. All bundled artifacts can be regenerated with the preprocessings/generate_*.py scripts.

---

Models

All models are selected with --model_type. They follow an encoder → predictor structure (except the end-to-end ResNet-50 black box), and differ in how concepts are obtained and how they are reasoned over.

--model_type	Paper name	Encoder → Predictor	Input consumed
resnet	ResNet-50 (black box)	ResNet-50 → linear rule head	raw image
linear	Oracle + Linear	ground-truth COCO counts → linear	91-dim counts
mlp	Oracle + MLP	ground-truth COCO counts → MLP	91-dim counts
detector_linear	MaskRCNN + Linear	detector counts → linear	detection counts
detector_mlp	MaskRCNN + MLP	detector counts → MLP	detection counts
cbm_sup_linear	Supervised + Linear	supervised ResNet-50 concept encoder → linear	raw image
cbm_sup_mlp	Supervised + MLP	supervised ResNet-50 concept encoder → MLP	raw image
cbm_clip_linear	CLIP + Linear	frozen CLIP concept scores → linear	raw image
cbm_clip_mlp	CLIP + MLP	frozen CLIP concept scores → MLP	raw image
ocb_linear	OCB + Linear	aggregated per-object concepts → linear	precomputed object concepts
ocb_mlp	OCB + MLP	aggregated per-object concepts → MLP	precomputed object concepts
dcr	DCR	ResNet-50 → Concept Embedding → Concept Reasoning Layer	raw image

Notes:

• CLIP / OCB use the MSCOCO category names (concept_vocabs/coco_categories.json) as the text concept vocabulary, unmodified.
• OCB aggregates per-object concept vectors via --aggregation {sum, max, sum_count, concat}, optionally capping the number of objects with --num_objects_training.
• DCR uses a Concept Embedding Model encoder (also a pretrained ResNet-50) jointly trained with a logic-based Concept Reasoning Layer; loss = concept loss + --dcr_task_weight × task loss, with Gödel or product logic (--dcr_logic).

model_factory.py (build_model, get_model_inputs) constructs each model and routes the correct input from a batch.

---

Training

train.py dispatches to one of three modes:

Trigger	Mode	What it does
--train_concept_encoder	Concept encoder	Trains a ResNet-50 to predict the 91 COCO categories (multilabel). Saves to --concept_encoder_path.
(default)	Joint multi-rule	Trains one model to predict all 10 rules. 10% of train is held out for validation + early stopping; final test eval at the end.
--fewshot	Per-rule LOOCV	On COCOLogic-V2-FS, trains one binary classifier per rule (10 models × 24 samples). Hyperparameters chosen by leave-one-out cross-validation.

--fewshot automatically switches all paths (JSONs, detector counts, OCB concepts, weights) to their *_fewshot* versions.

Examples

# 1. Train the supervised concept encoder (needed by cbm_sup_* models)
python train.py --train_concept_encoder --epochs 20 --lr 1e-4 --seed 42

# 2. Joint multi-rule training (full dataset)
python train.py --model_type resnet          --lr 1e-5 --epochs 10  --seed 42 --run_name eval
python train.py --model_type linear          --lr 1e-3 --epochs 40  --seed 42 --run_name eval
python train.py --model_type cbm_sup_linear  --lr 1e-3 --epochs 60  --seed 42 --run_name eval
python train.py --model_type cbm_clip_linear --lr 1e-3 --epochs 60  --seed 42 --run_name eval
python train.py --model_type ocb_linear      --lr 1e-3 --epochs 100 --aggregation concat \
                --num_objects_training 10    --seed 42 --run_name eval
python train.py --model_type dcr             --lr 1e-4 --epochs 40  --seed 42 --run_name eval \
                --save_dcr_path model_weights/dcr_seed42.pt

# 3. Few-shot, per-rule training
python train.py --model_type linear --lr 1e-2 --epochs 20 --seed 42 --fewshot --run_name per_rule_eval
python train.py --model_type dcr    --lr 1e-4 --epochs 20 --seed 42 --fewshot --run_name per_rule_eval

To reproduce the paper's full sweeps (5 seeds × all models), use the scripts:

bash scripts/v2_train.sh           # full dataset, joint multi-rule
bash scripts/v2_per_rule_train.sh  # few-shot, per-rule

Recommended hyperparameters

Best learning rate / epochs per model (from scripts/Hyperparamters.md). All models train with early stopping (--patience, default 5) and BCEWithLogitsLoss with class weights.

Full dataset (joint multi-rule):

Model	lr	epochs	extra
Oracle Linear (linear)	1e-3	40
Oracle MLP (mlp)	1e-3	40
ResNet (resnet)	1e-5	10
Detector Linear	1e-3	40
Detector MLP	1e-4	60
Sup. CBM Linear	1e-3	60
Sup. CBM MLP	1e-3	60
CLIP Linear	1e-3	60
CLIP MLP	1e-3	60
OCB Linear	1e-3	100	--aggregation concat --num_objects_training 10
OCB MLP	1e-3	40	--aggregation concat --num_objects_training 3
DCR	1e-4	40

Few-shot per-rule:

Model	lr	epochs	extra
ResNet	1e-4	40
Linear	1e-2	20
MLP	1e-4	60
Detector Linear	1e-2	40
Detector MLP	1e-4	60
Sup. CBM Linear	1e-3	80
Sup. CBM MLP	1e-4	80
CLIP Linear	1e-2	20
CLIP MLP	1e-2	80
OCB Linear	1e-2	60	--aggregation concat --num_objects_training 3
OCB MLP	1e-2	60	--aggregation concat --num_objects_training 3
DCR	1e-4	20

Key CLI flags

Flag	Default	Purpose
--model_type	linear	Which model to train (see table above)
--lr, --epochs, --batch_size	1e-3, 40, 32	Core training hyperparameters
--patience	5	Early-stopping patience (0 disables)
--seed	42	Random seed
--base_path	datasets/coco/images	Root of the MSCOCO images
--concept_encoder_path	model_weights/concept_encoder.pt	Frozen encoder for supervised CBMs
--clip_model	ViT-B-32	OpenCLIP model for CLIP-CBM
--aggregation, --num_objects_training, --max_objects	sum, None, 10	OCB aggregation settings
--cem_embedding_size, --cem_hidden_dim, --dcr_logic, --dcr_task_weight	8, 128, godel, 0.5	DCR / Concept Embedding settings
--save_dcr_path	None	Save the trained DCR model for rule inspection
--run_name	baselines	W&B run name

---

Evaluation & metrics

Metrics are computed per rule and then averaged across rules (evaluation.py):

• B-Acc — balanced accuracy over all samples (the overall summary).
• NB-type accuracy — accuracy on the near-boundary negatives, bucketed by NB type.
• Positive-variant accuracy — accuracy on positive samples, bucketed by variant.

NB and positive-variant accuracy are the informative numbers: unlike raw B-Acc, they are not inflated by easy FB negatives and show directly whether a rule was actually learned. The relevant functions are evaluate (validation), evaluate_cocologic_detailed (test, joint multi-rule), and evaluate_cocologic_per_rule (test, per-rule).

Key finding of the paper: concept-based models and the black-box baseline reach high overall B-Acc but perform at or near random on NB samples. In other words, they learn a coarse separation between positives and easy FB negatives based on the mere presence of relevant objects, rather than the underlying logical rule. NB accuracy degrades further as the concept representation gets noisier (detector > supervised encoder > CLIP). An Oracle MLP achieves near-perfect performance, confirming the dataset is solvable in principle. In the few-shot setting, CBM predictors are unstable with only 24 samples per rule, while in-context VLMs and program synthesis fare better — but reliably identifying which objects are present and relevant remains the main bottleneck.

---

Analysis

Post-training utilities live in analysis/:

• export_results.py, export_per_rule_results.py — pull and aggregate run results from W&B.
• plot_per_rule.py — per-rule B-Acc / variant / NB performance plots.
• plot_rule_examples.py — visualize dataset examples for each rule.
• count_dataset_stats.py — dataset statistics.
• inspect_dcr_rules.ipynb — inspect the logical rules extracted by a trained DCR model.

---

VLM baselines: in-context learning & program synthesis (vlp/)

Two VLM-based methods live in vlp/, both evaluated per-rule on COCOLogic-V2:

• ICL baseline (vlp/icl_baseline.py) — shows a VLM the positive/negative training images, asks it to induce a natural-language rule, then applies that rule to the held-out test images.
• VLP (vlp/main.py) — neuro-symbolic program synthesis: the VLM discovers objects/properties, grounds them into a DSL, and a search over a PCFG finds a logical program that separates positives from negatives.

Prerequisites:

• Run from the vlp/ directory.
• pip install -r requirements.txt (includes the VLM/search deps: anthropic, openai, google-generativeai, transformers, ray, …).
• API models read their key from an environment variable — ANTHROPIC_API_KEY, OPENAI_API_KEY, or GOOGLE_API_KEY. Local models (e.g. gemma-4-31B-it) need a GPU.
• MSCOCO images must be at ../../datasets/coco/images/{train2017,val2017}/ (relative to vlp/).

Run the ICL baseline

cd vlp
python icl_baseline.py --dataset cocologicv2 --model gemini-3.1-pro-preview \
    --seed 0 --max_imgs 16 --think

Results → vlp/results/icl/<dataset>/direct_results_*.json.

Run VLP (program synthesis)

cd vlp
python main.py --dataset cocologicv2 --model gemma-4-31B-it --search_timeout 300 \
    --n_objects 10 --n_properties 10 --n_actions 0 --max_program_depth 8 \
    --seed 0 --variable_distribution naive_weighted --max_imgs 16

Use --dataset cocologicv2-gt for the GT-object ablation (objects come from COCO annotations instead of the VLM). Results → vlp/results/vlp/<dataset>/discovered_programs_*.json.

Reproduce the full sweeps (5 seeds)

bash scripts/icl.sh 0    # ICL baseline   (arg = CUDA device id)
bash scripts/vlp.sh 0    # VLP synthesis  (arg = CUDA device id)

Key flags

Flag	Default	Applies to	Meaning
--dataset	cocologicv2	both	cocologicv2 or cocologicv2-gt (GT-object ablation)
--model	gemma-4-31B-it (VLP) / gemini-3.1-pro-preview (ICL)	both	VLM backend: gemma (local GPU), gemini / claude / gpt (API)
--max_imgs	6	both	images per task
--seed	0	both	random seed
--no_sampling	off	both	greedy VLM decoding (instead of sampling)
--think	off	ICL	ask the VLM to reason step-by-step before answering
--search_timeout	60	VLP	program-search budget per task (seconds)
--max_program_depth	5	VLP	maximum DSL program depth
--n_objects / --n_properties / --n_actions	10 / 10 / 5	VLP	number of symbols the VLM discovers (--n_actions 0 disables actions)
--variable_distribution	naive_weighted	VLP	PCFG variable prior: uniform, naive_weighted

---

Citation

@inproceedings{steinmann2026cocologicv2,
  title     = {COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives},
  author    = {Steinmann, David and W{\"u}st, Antonia and Kersting, Kristian and Stammer, Wolfgang},
  year      = {2026},
}