PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models

Qwen Large Model Application Team, Alibaba

📖 Overview

Safety alignment of LLMs typically depends on high-quality supervision data, such as safe demonstrations or preference pairs. However, in real-world deployment, emerging safety requirements are often specified as natural-language policies, while corresponding supervision data may be costly, delayed, or unavailable.

PolicyAlign is a simple yet effective framework for directly aligning LLMs with safety policies. Given a safety policy, PolicyAlign:

1. Synthesizes policy-violating instructions to construct targeted training data
2. Performs on-policy self-distillation to internalize policy-guided behavior
3. Applies Policy-Sensitive Filtering to select instructions where the policy induces the largest behavioral shift, improving training stability and data efficiency

PolicyAlign consistently improves safety while maintaining low over-refusal and preserving general capabilities. It also generalizes across multiple safety domains:

Domain	System Prompt	Description
LLM Safety	system_prompts/llm_safety.txt	General AI safety covering hate speech, violence, privacy, misinformation, etc.
Medical Safety	system_prompts/medical_safety.txt	Medical ethics, patient privacy, evidence-based practice
Legal Safety	system_prompts/law_safety.txt	Legal compliance and ethical legal practice
Financial Safety	system_prompts/finance_safety.txt	Financial regulations and responsible advisory

---

🔧 Requirements

conda create -n policyalign python=3.10
conda activate policyalign
pip install vllm==0.8.5
pip install -e ./verl --no-deps 
pip install torchdata textarena trl==0.28.0 rouge-score
pip install codetiming hydra-core peft pybind11 pylatexenc tensordict==0.6.2 wandb ray
pip install datasets==4.0.0
pip install transformers==4.52.3
pip install setuptools==68.2.2

---

🚀 Getting Started

1. Launch the Guard Model

Start the Qwen3Guard-Gen-8B safety review service:

bash guard_server.sh

2. Run Training

bash scripts/sys_consolidate.sh \
  --model Qwen2.5-7B-Instruct \
  --exp_name align-qwen7b \
  --nnodes 1 \
  --rollout_n 1 \
  --kl_loss_type full \
  --kl_topk 256 \
  --actor_lr 5e-6 \
  --max_response_length 1024 \
  --experience_max_length 1024 \
  --system_prompt_type safety \
  --exp_path system_prompts/llm_safety.txt \
  --total_training_steps 100 \
  --save_freq 10 \
  --teacher_mode ema \
  --teacher_ema_decay 0.99

3. Run Evaluation

Safety benchmarks (StrongREJECT, Advbench, Wildjailbreak, Fortress):

bash scripts/sys_eval.sh \
  --model Qwen2.5-7B-Instruct \
  --exp_name eval-qwen7b \
  --nnodes 1 \
  --ckpt 0 \
  --prompt_version v4 \
  --use_bsl true \
  --eval_prepend_experience false \
  --exp_path system_prompts/llm_safety.txt \
  --max_response_length 1024 \
  --experience_max_length 2048 \
  --system_prompt_type safety \
  --system_prompt_version v1

XSTest:

python data/xstest_eval.py --file_path /path/xstest_eval

General capabilities (MMLU-Pro, GPQA-Diamond, MATH500):

cd general_eval
python simple_evals.py \
  --model Qwen2.5-7B-Instruct \
  --model_path /path/model \
  --eval_mode mmlu_pro  # or gpqa / math

---

📁 Project Structure

PolicyAlign/
├── scripts/                    # Training & evaluation scripts
│   ├── sys_consolidate.sh      #   PPO alignment training
│   └── sys_eval.sh             #   Safety & capability evaluation
├── system_prompts/             # Domain-specific safety policies
│   ├── llm_safety.txt          #   General LLM safety rules
│   ├── medical_safety.txt      #   Medical domain policies
│   ├── law_safety.txt          #   Legal domain policies
│   └── finance_safety.txt      #   Financial domain policies
├── general_eval/               # General capability evaluation
│   ├── simple_evals.py         #   Evaluation entry point
│   ├── mmlu_pro_eval.py        #   MMLU-Pro benchmark
│   ├── gpqa_eval.py            #   GPQA-Diamond benchmark
│   └── math_eval.py            #   MATH-500 benchmark
├── tools/                      # Utility scripts
│   ├── prepare_data.py         #   Data preparation
│   ├── merge_model2hf.py       #   Checkpoint merging
│   └── make_exp_list.py        #   Experiment management
├── verl/                       # Core RL framework (modified verl)
├── guard_server.sh             # Guard model service launcher
└── README.md                   # This file

---

🙏 Acknowledgements

This codebase is built upon verl and OPCD. We thank all teams for their excellent open-source contributions.

---

📜 Citation

If you find our work useful, please consider citing:

@article{wu2026policyalign,
  title={PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models},
  author={Wu, Chang and Fang, Junfeng and Jiang, Houcheng and Tang, Kai and Cheng, Pengyu and Jiang, Xiaoxi and Jiang, Guanjun and Wang, Xiang},
  journal={arXiv preprint arXiv:2606.25442},
  year={2026}
}