Don't Mix Rewards, Mix Policies:
Policy Decomposition and Optimization for Multi-Reward RL

Ruiming Liang^{1,2,3 * ‡}, Yi Zhong^{3,4 * ‡}, Yizhen Yuan³, Yinan Zheng³, Tianyi Tan³, Tianyue Wang^1,2, Haiyun Guo^{1 †}, Jinqiao Wang¹, Xianyuan Zhan^{3 †}

¹Fundation Model Research Center, CASIA,
²School of Artificial Intelligence, UCAS,
³Institute for AI Industry Research (AIR), Tsinghua University,
⁴College of Automotive and Energy Engineering (CAEE), Tongji University
^*Equal Contribution. ^†Corresponding Author. ^‡Work done during internship at THU-AIR.

Paper arXiv Code

PRISM decomposes multi-reward RL into per-reward positive policies and a single global negative policy, composed in logit space at inference time.

Takeaways

⭐️ Don't mix rewards, mix policies: Instead of scalarizing heterogeneous rewards into a single training signal, PRISM keeps each reward as its own optimization direction and composes the resulting policies at inference time.

⭐️ Alleviates the multi-reward alignment tax: Reward-specific positive policies avoid gradient competition among conflicting preferences, while a single global negative policy jointly captures the union of all reward-specific failure modes.

⭐️ Training-free inference-time controllability: All sub-policies share one backbone via lightweight prefix conditioning and are composed by a weighted sum of logits, so preference trade-offs can be adjusted at inference time without retraining.

⭐️ Consistent gains across tasks: On scientific reasoning, tool-use reasoning, and helpfulness-safety alignment, PRISM outperforms strong multi-reward RL baselines (GRPO-Sum, GRPO-Product, GDPO) and remains more robust as the number of rewards grows.

PRISM is a multi-reward reinforcement learning framework built on policy-space decomposition and composition. It trains one standalone positive policy per reward together with a shared global negative policy, and reconstructs the optimal multi-reward policy by linearly combining their logits — turning reward trade-offs into an explicit, interpretable, inference-time interface.

Method

🚫 The Multi-Reward Alignment Tax

Most multi-reward RL methods collapse N reward signals into a single weighted scalar before each policy update — e.g. linear scalarization (GRPO-Sum), reward-product aggregation (GRPO-Product), or advantage-space normalization (GDPO). Because rewards differ in scale, distribution, and calibration, composing them in reward space forces conflicting preferences to compete within the same gradient step, yielding a compromise policy that under-optimizes every objective — an effect the paper calls the multi-reward alignment tax.

🔀 Policy-Space Decomposition

PRISM instead composes preferences after optimization, in policy space. Starting from a KL-regularized one-step improvement objective augmented with a shared "union of failures" penalty, the optimal policy factorizes exactly into:

Reward-Specific Positive Policies (π⁺ₖ)

One positive policy per reward k, trained independently with GRPO to encourage behaviors preferred by that reward. Each reward drives updates through its own dedicated branch, avoiding gradient interference with other objectives.

Global Negative Policy (π⁻)

A single negative policy trained to capture the union of all reward-specific failure modes via a soft-conjunction advantage, so a severe violation of any one objective is enough to trigger suppression — without attributing failures to individual rewards.

Crucially, the definitions of π⁺ₖ and π⁻ do not depend on the merge coefficients used to combine them — training and composition are fully decoupled.

🎛️ Logit-Level Policy Composition

Because all sub-policies share the same reference policy and context, taking logarithms turns the product-of-experts composition into a simple weighted sum of logits:


      z* = Σₖ αₖ · z⁺ₖ − γ · z⁻

The merge weights {α₁, …, α_N, γ} form an explicit, interpretable interface for preference trade-offs, freely adjustable at inference time without retraining.

Parallel-batch mixture sampling. Query, per-branch prefix, and shared context are stacked along the batch dimension and processed by the shared backbone in one forward pass; per-branch logits are combined and the sampled token is broadcast back to all branches.

⚙️ Efficient Implementation

Prefix-conditioned shared backbone: all N+1 sub-policies reuse one language model, distinguished only by lightweight learnable prefix embeddings — avoiding an (N+1)× blow-up in parameters.
Backbone-preserving asymmetric updates: the shared backbone is updated only through positive branches (stop-gradient on the negative branch), keeping the backbone aligned with high-quality behavior while the negative prefix still learns a catch-all failure representation.
Parallel-batch mixture sampling: all N+1 branches are decoded in a single batched forward pass by stacking them along the batch dimension, so per-token latency stays close to that of a single policy despite memory/FLOPs scaling linearly with N+1.

Experiment

Evaluation Settings

PRISM is evaluated on three multi-reward alignment scenarios: scientific question answering (SciKnowEval → GPQA / ScienceQA), tool-use reasoning (ToolRL → BFCL-v3), and helpfulness-safety alignment (Alpaca / HH-RLHF / PKU-SafeRLHF), across DeepSeek-R1-1.5B, Qwen2.5-1.5B-Instruct, and Qwen2.5-3B-Instruct backbones, compared against GRPO-Sum, GRPO-Product, and GDPO baselines.

Main Results

On scientific reasoning, PRISM attains the best overall score on all three backbones, surpassing the strongest baseline by +17.8 on DeepSeek-R1-1.5B, +8.0 on Qwen2.5-1.5B-Instruct, and +0.6 on Qwen2.5-3B-Instruct, with the largest gains on the harder GPQA benchmark. On tool calling (BFCL-v3), it achieves the best overall format accuracy and BFCL-style accuracy. On helpfulness-safety alignment, it scores highest on both usefulness and harmlessness across Alpaca, HH-RLHF, and PKU-SafeRLHF.

Table 1: Main results on scientific QA benchmarks. All numbers are percentages. Fmt: format adherence. Acc: correctness. Joint: format & correct. Avg: average of Fmt and Acc.
Model / Method	ScienceQA				GPQA				Overall
Model / Method	Fmt	Acc	Joint	Avg	Fmt	Acc	Joint	Avg	Overall
DeepSeek-R1-1.5B
Base model	0.00	66.64	0.00	22.21	0.00	29.68	0.00	9.89	16.05
+ GRPO Sum	74.11	67.85	54.02	65.33	16.96	29.46	6.25	17.56	41.44
+ GRPO Product	74.39	68.51	55.42	66.11	27.68	30.80	9.60	22.69	44.40
+ GDPO	80.37	67.94	58.04	68.78	26.34	28.35	8.71	21.13	44.96
+ PRISM (ours)	95.33	69.91	68.51	77.92	81.25	33.04	28.35	47.55	62.73
Qwen2.5-1.5B-Instruct
Base model	6.82	73.36	5.89	28.69	37.28	27.23	10.49	25.00	26.84
+ GRPO Sum	100.00	76.26	76.26	84.17	77.46	27.90	21.21	42.19	63.18
+ GRPO Product	98.31	75.98	74.30	82.86	37.95	27.90	11.16	25.67	54.27
+ GDPO	99.93	76.64	76.26	84.28	76.56	29.01	21.43	42.33	63.30
+ PRISM (ours)	99.53	76.72	76.45	84.23	87.95	29.24	25.67	58.45	71.34
Qwen2.5-3B-Instruct
Base model	44.67	81.78	36.73	54.39	47.99	33.93	16.52	32.81	43.60
+ GRPO Sum	99.81	83.55	83.55	88.97	87.95	29.91	27.46	48.44	68.71
+ GRPO Product	99.53	84.02	84.02	89.19	84.82	28.80	24.11	45.91	67.55
+ GDPO	99.81	82.80	82.80	88.47	85.94	29.91	26.79	47.55	68.01
+ PRISM (ours)	98.97	84.11	84.11	89.06	88.17	31.92	28.57	49.55	69.31

Scientific QA results. PRISM achieves the best overall score on every backbone.

Table 2: Main results on BFCL v3 (tool calling) with Qwen2.5-3B-Instruct. Acc/R: RLLA-style judgement. Acc/B: BFCL-style judgement.
Model / Method	Non-live			Live			Multi-turn			Overall
Model / Method	Fmt	Acc/R	Acc/B	Fmt	Acc/R	Acc/B	Fmt	Acc/R	Acc/B	Fmt	Acc/R	Acc/B
Qwen2.5-3B-Instruct	13.30	8.70	72.17	14.06	5.63	58.30	20.56	7.28	17.34	15.98	7.20	49.27
+ GRPO Sum	65.74	40.87	74.78	40.87	31.16	68.02	59.53	17.43	18.28	64.43	27.82	51.27
+ GRPO Product	90.17	49.48	74.96	96.97	31.16	60.07	77.51	20.50	17.99	88.22	33.71	51.01
+ GDPO	98.87	55.30	76.09	99.33	33.16	62.37	83.22	20.68	17.93	93.81	36.38	52.13
+ PRISM (ours)	99.83	56.26	80.35	99.33	31.09	62.22	86.54	21.24	17.81	95.23	36.20	53.46

Table 2: Tool-use (BFCL v3) results. PRISM achieves the best overall Fmt and Acc/B, with Acc/R on par with the strongest baseline.

Table 3: Main results on helpfulness-safety alignment with Qwen2.5-3B-Instruct.
Method	Alpaca			HH-RLHF			PKU-SafeRLHF
Method	Useful	Harmless	Avg	Useful	Harmless	Avg	Useful	Harmless	Avg
+ GRPO Sum	3.03	3.49	3.26	3.03	4.03	3.53	4.79	6.34	5.57
+ GRPO Product	3.02	3.48	3.25	3.02	4.03	3.53	4.79	6.34	5.57
+ GDPO	2.95	3.44	3.20	3.03	4.02	3.53	4.79	6.33	5.56
+ PRISM (ours)	3.17	3.65	3.41	3.14	4.15	3.65	4.86	6.36	5.61

Table 3: Helpfulness-safety alignment results. PRISM scores highest on usefulness and harmlessness across all three evaluation sets.

Alignment Tax under Varying Rewards

Training with one, two, and three rewards on BFCL-v3 shows that PRISM maintains nearly the same correctness performance as more rewards are added, while baselines (especially GDPO and GRPO-Product) show a clear drop in correctness — confirming that PRISM better mitigates the multi-reward alignment tax and achieves a superior Pareto frontier.

Figure 3: Overall BFCL v3 performance under 1 / 2 / 3 training rewards. PRISM maintains nearly unchanged correctness as more rewards are added, while baselines degrade.

Training curves across methods under the three-reward setting

Figure 4: Training curves (correctness / format / length reward) across methods under the three-reward setting. PRISM converges faster and to a higher level on all three rewards.

Inference-Time Controllability

On BFCL-Live, assigning a dominant weight (0.55) to one positive branch while keeping the others at 0.25 consistently steers the corresponding metric: correctness-dominant weighting yields the highest correctness accuracy, format-dominant weighting yields the best format accuracy, and length-dominant weighting produces the longest reasoning traces — all without any retraining.

Figure 5: Inference-time controllability on BFCL-Live. Assigning a dominant weight (0.55) to one positive branch consistently improves the corresponding reward, without any retraining.

Ablation Studies

Replacing reward-specific positive policies with a shared positive policy, removing the global negative policy, or sampling rollouts independently per branch all degrade overall performance — confirming the importance of per-reward positive branches, a shared negative policy, and rollouts from the composed policy. Among negative-weighting functions, the soft-conjunction weighting used by PRISM outperforms max, LogAvgExp, and mean weighting.

Table 4: Policy-composition ablations on BFCL-v3 (Qwen2.5-3B-Instruct).
Method	Fmt	Acc/R	Acc/B
PRISM	95.23	36.20	53.46
w/ shared positive policy	94.03	34.38	52.56
w/o global negative policy	94.86	34.75	52.84
w/ individual-branch rollouts	94.84	34.59	52.48

Table 5: Negative-policy weighting ablations on BFCL-v3 (Qwen2.5-3B-Instruct).
Method	Fmt	Acc/R	Acc/B
PRISM	95.23	36.20	53.46
w/ max weighting	95.02	33.70	52.43
w/ LogAvgExp weighting	95.06	34.58	52.62
w/ mean weighting	93.27	33.90	53.12

Inference Efficiency

Measured on 8×NVIDIA H800, PRISM's parallel-batch mixture sampling keeps per-response decoding latency close to a single-policy baseline (2.01s → 2.04s / 2.12s for N=2 / N=3) with unchanged peak GPU memory, since all branches are decoded in one batched forward pass rather than sequentially.

Table 9: Inference efficiency on Qwen2.5-3B-Instruct, measured on 8×NVIDIA H800. Latency in seconds, throughput in tokens/second, peak memory in GB. PRISM with N rewards decodes N+1 branches.
Method	Latency (s)	Throughput (tok/s)	Memory (GB)
Single policy	2.01	127.56	63.74
PRISM (N = 2)	2.04	125.71	63.74
PRISM (N = 3)	2.12	121.06	63.74

BibTeX


@article{liang2026prism,
  title={Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL},
  author={Liang, Ruiming and Zhong, Yi and Yuan, Yizhen and Zheng, Yinan and Tan, Tianyi and Wang, Tianyue and Guo, Haiyun and Wang, Jinqiao and Zhan, Xianyuan},
  journal={arXiv preprint arXiv:2607.29246},
  year={2026}
}

Don't Mix Rewards, Mix Policies:Policy Decomposition and Optimization for Multi-Reward RL