What if we could find treatments for the 95% of rare diseases that have none — by letting an AI scientist run experiments around the clock?
There are ~10,000 rare diseases. 95% have zero FDA-approved treatment. 300-400 million people affected worldwide. 30% of affected children die before age 5. Yet only 2.63% of AI drug repurposing research even targets them.
We pointed an autonomous AI agent at this problem and let it run for 24 hours.
It ran 15 experiments. Discovered 3 breakthroughs. Correctly reverted 6 failures. Improved drug-disease prediction by +168%. All with zero human intervention.
This is a Karpathy-style autoresearch loop: Claude Code (Opus 4.6) autonomously reads past results, forms hypotheses, edits code, trains on a GPU, evaluates, keeps improvements, reverts failures — every 30 minutes, unattended.
The agent started from raw TxGNN (Nature Medicine 2024) embedding scores on PrimeKG and autonomously discovered multi-relation scoring, disease similarity transfer, and ensemble methods — improving per-disease AUPRC from 0.284 to 0.761 across 278 held-out diseases.
Per-Disease AUPRC Progression
───────────────────────────────────────────────────────────────────
#0 passthrough ███████░░░░░░░░░░░░░░░░░░░ 0.284 baseline
#1 multirel_logreg ███████████████░░░░░░░░░░░ 0.614 +116% ← used all 50 relations
#2 +geometry ████████████████░░░░░░░░░░ 0.620 +1.1%
#3 +disease_transfer ██████████████████░░░░░░░░ 0.698 +12.6% ← collaborative filtering
#4 +drug_transfer ██████████████████░░░░░░░░ 0.715 +2.3%
#5 lightgbm ███████████████████░░░░░░░ 0.756 +5.8% ← nonlinear model
#6 rank_normalize ███████████████████░░░░░░░ 0.752 -0.5% ✗ reverted
#7 tuned_lgbm ███████████████████░░░░░░░ 0.750 -0.8% ✗ reverted
#8 drug_degree ██████████████████░░░░░░░░ 0.700 -7.4% ✗ degree bias!
#9 ensemble ███████████████████░░░░░░░ 0.758 +0.3%
#10 transfer_count ███████████████████░░░░░░░ 0.754 -0.5% ✗ reverted
#11 pca_subtraction ██████████████████░░░░░░░░ 0.749 -1.2% ✗ reverted
#12 ensemble_70_30 ███████████████████░░░░░░░ 0.760 +0.2%
#13 k_neighbors_20 ███████████████████░░░░░░░ 0.761 +0.1% ← BEST
#14 neg_ratio_20 ███████████████████░░░░░░░ 0.759 -0.2% ✗ reverted
The agent discovered three major breakthroughs (runs #1, #3, #5) and correctly reverted six failed experiments — all without human guidance.
| # | Method | AUPRC | Median | Delta vs Baseline | Notes |
|---|---|---|---|---|---|
| 0 | v0_passthrough | 0.2838 | — | — | TxGNN DistMult raw scores (starting point) |
| 1 | v1_multirel_logreg | 0.6135 | 0.6805 | +0.3297 (+116.1%) | 50 relation scores + LogReg + StandardScaler |
| 2 | v2_multirel_geom | 0.6202 | 0.6887 | +0.3364 (+118.5%) | +4 geometry features (cosine, L2, norms) |
| 3 | v3_disease_transfer | 0.6984 | 0.8194 | +0.4146 (+146.1%) | +2 disease similarity transfer features (K=10) |
| 4 | v4_drug_transfer_2hop | 0.7146 | 0.8228 | +0.4308 (+151.8%) | +2 drug transfer features (2-hop via disease similarity) |
| 5 | v5_lightgbm | 0.7559 | 0.8705 | +0.4721 (+166.4%) | LightGBM replaces LogReg on same 58 features |
| 6 | v6_rank_normalize | 0.7520 | 0.8620 | +0.4682 (+165.0%) | Per-disease rank normalization. Lost magnitude info. Reverted |
| 7 | v7_tuned_lgbm | 0.7502 | 0.8830 | +0.4663 (+164.3%) | n_estimators=500, lr=0.03. Mean worse, median better. Reverted |
| 8 | v8_drug_degree | 0.6995 | — | +0.4157 (+146.5%) | +drug indication degree. Degree bias overfitting (-7.4%). Reverted |
| 9 | v9_ensemble | 0.7580 | 0.8765 | +0.4742 (+167.1%) | LightGBM + LogReg ensemble (50/50 avg probabilities) |
| 10 | v10_transfer_count | 0.7540 | 0.8700 | +0.4702 (+165.7%) | +transfer_count feature. Redundant signal. Reverted |
| 11 | v11_pca_subtraction | 0.7492 | 0.8540 | +0.4654 (+164.0%) | +15 PCA embedding features. Diluted tree splits. Reverted |
| 12 | v12_ensemble_70_30 | 0.7596 | 0.8767 | +0.4758 (+167.6%) | 70/30 LightGBM/LogReg weighting |
| 13 | v13_k_neighbors_20 | 0.7605 | 0.8862 | +0.4767 (+168.0%) | K=20 neighbors. +1.08% median. BEST |
| 14 | v14_neg_ratio_20 | 0.7590 | 0.8764 | +0.4752 (+167.4%) | neg_ratio 10→20. Worse on both mean and median. Reverted |
The agent started with a single DistMult relation score per drug-disease pair (AUPRC 0.284). Through systematic experimentation, it autonomously discovered:
| Discovery | Run | Impact | What the Agent Did |
|---|---|---|---|
| Multi-relation scoring | #1 | +116% | Used all 50 PrimeKG relation types instead of 1. The reverse indication relation (idx 15) had weight +4.57, nearly 2x the forward direction — DistMult captures different structure in each direction. Contraindication got negative weight (-0.54), which is biologically correct. |
| Disease similarity transfer | #3 | +12.6% | Collaborative filtering: for each test disease, find K most similar training diseases by embedding cosine similarity, transfer their known drug associations as features. Median AUPRC jumped from 0.69 to 0.82 — over 80% of diseases now above 0.5. |
| Nonlinear ensemble | #5, #9, #12 | +8.4% | LightGBM replaced LogReg (+5.8%), then ensemble with LogReg (+0.3%), then weight tuning 70/30 (+0.2%). LightGBM uses molecular function, exposure, and cellular component relations that LogReg completely ignored. |
The agent also learned what doesn't work and documented why:
- Rank normalization loses magnitude information that tree models use natively
- Drug degree features cause popularity bias — model overfits to well-studied drugs, catastrophic for cold-start diseases
- PCA embedding features dilute LightGBM's split budget away from stronger features
- Feature aggregation over existing signals is redundant when the model already has raw ingredients
Input: TxGNN pretrained embeddings (7,957 drugs × 17,080 diseases × 512-dim)
│
├─ 50 DistMult relation scores (indication, contraindication, off-label, drug-protein, ...)
├─ 4 geometry features (cosine similarity, L2 distance, embedding norms)
├─ 2 disease similarity transfer features (K=20 nearest training diseases)
└─ 2 drug similarity transfer features (2-hop via disease similarity)
│
└─ 58 features total
│
├─ LightGBM (70%) ─── nonlinear feature interactions
└─ LogReg (30%) ────── linear regularizer
│
└─ Weighted probability average → Per-disease drug ranking
| Model | Per-Disease AUPRC | Eval Protocol | Source |
|---|---|---|---|
| Ours (v13) | 0.761 | All diseases, all 7,957 drugs, transductive* | This project |
| BioPathNet | 0.556 | 5 disease areas, filtered candidates | Nature Biomed Eng 2025 |
| TxGNN | 0.437 | 5 disease areas, filtered candidates | Nature Medicine 2024 |
| Ours (baseline) | 0.284 | All diseases, all 7,957 drugs, transductive* | Starting point |
*Important: The reranker uses TxGNN embeddings pretrained on the full PrimeKG graph, while published methods train from scratch per disease-area split. This gives our baseline a significant advantage. The +168% improvement over the TxGNN embedding baseline is real, but the absolute value should not be read as beating published methods.
┌───────────────────────────────────────────────────────────────────┐
│ AUTORESEARCH LOOP (every 30 min) │
│ │
│ ┌────────────┐ ┌─────────────┐ ┌───────────────────────┐ │
│ │ Read state │───▶│ Form │───▶│ Edit train.py │ │
│ │ + results │ │ hypothesis │ │ (one change) │ │
│ └────────────┘ └─────────────┘ └──────────┬────────────┘ │
│ ▲ │ │
│ │ ▼ │
│ ┌────────────┐ ┌─────────────┐ ┌───────────────────────┐ │
│ │ Update │◀───│ Evaluate │◀───│ Train on GPU │ │
│ │ state + │ │ per-disease │ │ (RunPod A5000) │ │
│ │ README │ │ AUPRC │ │ │ │
│ └────────────┘ └─────────────┘ └───────────────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Improved? → git commit & push │ │
│ │ Worse? → git revert train.py & push │ │
│ └────────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────────┘
| File | Rule | Purpose |
|---|---|---|
prepare.py |
Immutable | Load embeddings, compute scores, create splits, evaluate — the agent cannot game metrics |
train.py |
Mutable | Scoring logic — the agent improves this (all 15 experiments are in git history) |
program.md |
Read-only | Experiment ideas and research directions |
The agent also reads RESEARCH.md (comprehensive survey of SOTA methods) and experiments/state.md (cross-run memory with observations, patterns, and failed approaches) before each run.
# 1. Clone
git clone https://github.com/monk1337/Bio-Autoresearch.git
cd Bio-Autoresearch
# 2. Set up secrets
cp .env.example .env
# Edit .env: add WANDB_API_KEY and RunPod SSH details
# 3. Set up GPU server (clones repo, installs deps, downloads data)
chmod +x scripts/setup_gpu.sh
./scripts/setup_gpu.sh
# 4. Run the autoresearch loop
chmod +x run_autoresearch.sh
./run_autoresearch.sh- Tests SSH connection to RunPod GPU server
- Clones the repo on the GPU (
/workspace/Bio-Autoresearch) - Installs Python dependencies (
numpy,pandas,scikit-learn,lightgbm) - Runs
scripts/setup.shon GPU (downloads PrimeKG ~936MB, TxGNN checkpoint ~200MB, extracts embeddings) - Verifies pipeline end-to-end on GPU
- Embeddings: Pretrained TxGNN (transductive — trained on full PrimeKG including test diseases)
- Split: Cold-start disease-level holdout (30% diseases held out, zero drug-disease edges in reranker training)
- Metric: Per-disease macro-averaged AUPRC
- For each of 278 test diseases: rank all 7,957 drugs, compute average precision
- Macro-average across diseases (each disease weighted equally)
- Note: Per-disease AUPRC (~0.28 baseline) is far lower than pooled AUPRC (>0.93). We use per-disease evaluation throughout as it better reflects real-world utility for individual diseases
The pipeline uses pretrained TxGNN assets (~200MB) and PrimeKG (~936MB). scripts/setup.sh downloads everything automatically on the GPU.
- 7,957 drugs with 512-dim embeddings from TxGNN
- 17,080 diseases with 512-dim embeddings from TxGNN
- 50 relation weight vectors (all PrimeKG edge types: indication, contraindication, off-label, drug-protein, disease-protein, phenotype, pathway, etc.)
- PrimeKG knowledge graph (~5M edges, 19 relation types)
- Cold-start evaluation split: 278 test diseases held out entirely
Bio-Autoresearch/
├── prepare.py # Immutable orchestrator (agent cannot modify)
├── train.py # Mutable scoring logic (15 versions in git history)
├── program.md # Experiment ideas
├── RESEARCH.md # Deep research report (SOTA, methods, references)
├── run_autoresearch.sh # Outer loop (invokes Claude Code every 30 min)
├── AUTORESEARCH_PROMPT.md # Full agent instructions
├── config.yaml # Configuration
├── pyproject.toml # Python dependencies
├── .env.example # Environment variable template
├── data/
│ ├── scores/ # TxGNN pretrained assets (.npy embeddings)
│ ├── prepared/ # Generated splits and scores
│ └── kg.csv # PrimeKG knowledge graph
├── experiments/
│ ├── state.md # Cross-run agent memory (observations + patterns)
│ └── results.tsv # All experiment metrics
├── scripts/
│ ├── setup.sh # Full automated setup (run on GPU)
│ ├── setup_gpu.sh # One-command GPU server setup (run locally)
│ ├── extract_txgnn_embeddings.py
│ └── parse_stream.py # Claude Code output parser
└── logs/ # Run logs (full agent traces)
- Local machine (macOS): Runs
run_autoresearch.shwhich invokes Claude Code every 30 min - GPU server (RunPod A5000): Runs
python prepare.pyvia SSH — trains and evaluates - GitHub: All state, results, and code synced via git push/pull after every run
- TxGNN embedding extraction (7,957 drugs, 17,080 diseases, 512-dim)
- Cold-start disease-level evaluation split (278 test diseases)
- Per-disease AUPRC evaluation (honest metric, not pooled)
- Multi-relation DistMult scoring (all 50 PrimeKG edge types)
- Disease similarity transfer (collaborative filtering, K=20)
- Drug similarity transfer (2-hop via disease similarity)
- LightGBM + LogReg ensemble (70/30 weighted)
- 15 autonomous experiments, best AUPRC = 0.761 (+168%)
- BioMedKG multimodal embeddings (BioBERT + MolFormer + ProteinBERT, pretrained on HuggingFace)
- DRKG TransE embeddings (different KG, complementary signal)
- Graph neighborhood features from PrimeKG (shared protein targets, metapath counts)
- Feature selection (prune bottom 20 features by importance)
- Retrain TxGNN from scratch per disease-area split (remove transductive advantage)
- Compare on identical splits as BioPathNet and TxGNN published numbers
- Run on all 9 TxGNN disease areas (cardiovascular, mental health, etc.)
- Explorer agent (KG-based candidate proposal)
- Proponent agent (mechanistic evidence chains: Drug→Target→Pathway→Disease)
- Skeptic agent (contraindication check, safety signals, popularity bias detection)
- Evidence strength classification (Known / Strongly Supported / Likely / Novel)
- RepoDB retrospective validation (6,677 approved + 4,123 failed drug-indication pairs)
- Time-split validation (train pre-2020, test post-2020)
- Known repurposing recovery test (Baricitinib→COVID, Sildenafil→PAH, etc.)
- Open-source release with reproducibility guide
- TxGNN — Zero-shot drug repurposing foundation model. Nature Medicine 2024 | Code
- BioPathNet — Path-based GNN for drug repurposing. Nature Biomed Eng 2025 | Code
- PrimeKG — Precision medicine knowledge graph. Nature Sci Data 2023
- Consilience Ensemble — 7-method ensemble achieves MRR=0.9792 vs individual peak 0.2557. PLOS Comp Bio 2024
- RareAgent — Multi-agent LLM for rare disease repurposing. arXiv 2510.05764
- Karpathy's Autoresearch — The three-file contract pattern. Tweet
If you use this work in your research, please cite:
@software{pal2026bioautoresearch,
title = {Bio-Autoresearch: Karpathy-Style Autoresearch for Rare Disease Drug Discovery},
author = {Pal, Ankit},
year = {2026},
url = {https://github.com/monk1337/Bio-Autoresearch},
note = {Autonomous AI experimentation for drug repurposing on PrimeKG using TxGNN embeddings}
}This project is for research purposes only. All predictions are computational hypotheses, not medical advice.
MIT