A Constitution-Grid Instrument for Data-Efficient RL Alignment (C-Guard)

1 · Introduction

Many RL alignment problems train two objectives that pull against each other (Bai et al., 2022b;a). A safety guard is a good instance, because safety guard classification is in high demand: on OpenRouter, classification is the largest single task, 9.0% of all spend and 6.9% of all tokens (OpenRouter, 2026). A guard reads every prompt an LLM receives and must catch real harm without refusing benign requests. Our 4B baseline flags How do I kill someone in Call of Duty? as unsafe because it only sees the trigger word kill, and it blocks 22.4% of benign prompts (Röttger et al., 2024). The opposite failure means a harmful request dressed as fiction or roleplay walks past the same benchmark (Han et al., 2024).

The standard fix is more boundary data, but volume without a target is wasteful (Diddee & Ippolito, 2024). We added 187 rows to our weakest topic and its accuracy gain is 0. Worse, the data spending is one-sided if XSTest is the only reference test set. It grades over-refusal and stays nearly flat on the other axis, so training could drift to one side while paying invisible cost in missed attacks. The question is how to aim every data move by a measurement, and how to see both objectives at once. We build a constitution grid that instruments coverage: every new row lands where the board shows learning headroom, which is what makes the RL training data-efficient.

Framing guard data generation as playing moves on a constitution grid, we write a constitution, one policy per harm topic, and cross its topics with the ways a user can ask (Figure 1). C-LIM, a per-cell learnability score adapted from learning-impact measurement (Li et al., 2025) and computed on unseen data rows, reads the board and decides each cell's move: prune a mastered cell, densify a still-learning one, amend a cell whose rule is wrong, expand the board with a new topic. Every generation feeds towards RL training with GRPO (Shao et al., 2024) and every gain is measured on the trained model.

Our contributions are:

1. Constitutional grid drives data coverage: read cell learnability, route each cell to a move, train with RL. Aimed generation lifts the flagged region's learning impact 0.733 to 0.80.
2. Two measurements: C-LIM flags dead-weight data before any budget is spent. A two-channel read reveals the drift tax: over-refusal improves 22.4% to 12.8% while adversarial under-refusal silently worsens 0.27 to 0.33.
3. Executable gates on moves. The gate rejected an amendment that over-reached and a topic that helped itself but hurt the rest of the board.

No prior guard combines constitution policy data, per-cell probe aiming, a live attack channel, and RL (Table 1). The closest neighbor is Calibrated Reasoning (Garg et al., 2025), which trains a reasoning model with RL but calibrates a verifier at inference rather than the training data.

Table 1. Where C-Guard sits, ∼ marks a partial mechanism. GuardReasoner aims by hard-sample mining and tunes with DPO (Rafailov et al., 2023). HaloGuard's attack side is static augmentation. Constitutional Classifiers red-team once. Calibrated Reasoning trains a reasoning model with RL but calibrates a verifier at inference rather than the training data. C-Guard aims per-cell and re-measures attacks every checkpoint.

2 · Method

The constitution grid has an automatic loop, as it takes a checkpoint in and puts directed training rows out. We read the board, make the move, train the model, and iterate (Figure 2). There are four moves: prune a mastered cell, densify a still-learning one, amend a cell whose rule is wrong, expand the board with a new topic.

2.1 · Read then act

The constitution is one policy per harm topic (Bai et al., 2022a; Sharma et al., 2025; Sangameswaran et al., 2026), and each clause draws one line between safe and unsafe. Its topics crossed with the ways a user can ask span the grid. The generator writes a safe twin and an unsafe twin at each clause boundary (Table 2). A twin pair shares the trigger word and flips only the intent, so the guard must learn the boundary rather than the word.

Table 2. Example twin pairs, one per topic.

To read a cell on learnability, we sample the model 8 times on unseen rows at each checkpoint, then score the pass rate against the labels, where every label is fixed by construction and traces to the clause. C-LIM scores the trajectory: for cell \(c\) with mean pass rate \(r_c^k\) at checkpoint \(k\) and field mean \(\bar{r}^k\),

\[ s_c = 1 - \frac{\sum_k \left(r_c^k - \bar{r}^k\right)^2}{\sum_k \left(1 - \bar{r}^k\right)^2}. \]

A cell that tracks the field scores near 1. A cell that stays flat below a rising field scores a large negative, which is dead weight. Unlike LIMR (Li et al., 2025), which scores training samples to select a subset, C-LIM scores a grid cell on rows the model never trains on, so the score diagnoses a region of the board. Two moves follow the score directly (Figure 3): (1) Prune a mastered cell (score near 1): generate nothing more, keep its rows as a retention set. (2) Densify a still-learning cell (rising score): add more rows under the same rule.

2.2 · Attack then amend

Every cell is read from both directions. The benign channel checks that safe look-alikes pass, and the attack channel checks that disguised unsafe prompts in the same style are blocked (Han et al., 2024), revealing the coverage gaps the benign channel cannot see. When the attack channel shows a rule is wrong, the amend move edits the constitution text (Figure 4). An amendment must survive a precedent-flip regression that re-judges settled rows, and it ships as paired data, the attack catch and its safe twin, so a sharper boundary does not refuse the legitimate request. XSTest (Röttger et al., 2024) stays outside the loop as a read-only reference, which lets Section 3 measure the drift tax on data the model never trained on.

2.3 · Grow the board

Two moves grow the board (Figure 5). A new topic must pass a benefit gate: it learns, the global score does not regress, and no existing topic drops. Section 3 shows one rejection from this gate and one from the amendment gate. (1) Subdivide handles a stuck cell: when more rows stop helping, decompose the cell along a new local axis. (2) Expand adds a new topic row when a coverage gap has no topic to live in, the only move that adds a row.

2.4 · RL training

We start from Nemotron-Content-Safety-Reasoning-4B (Sreedhar et al., 2025), a safety guard that reasons before it outputs a safe or unsafe label and is trained with SFT only. The goal is to add RL on top of that SFT base to make it reason better, following RSafe and GuardReasoner (Zheng et al., 2025; Liu et al., 2025). It is the right SFT base for two reasons. Its reasoning makes rollouts vary, so RL has a gradient that a verdict-only classifier would not give. And its errors are one-sided over-refusal, the side with headroom to fix. The recipe is deliberately plain: vanilla GRPO (Shao et al., 2024) with a rule reward,

\[ \pi^* = \arg\max_{\pi}\ \mathbb{E}_{x\sim D,\,c\sim\pi}\big[R(c,x)\big], \qquad R = \mathbb{1}[\text{answer matches label}] - 0.2\cdot\mathbb{1}[\text{format invalid}]. \]

The label is read only from the answer slot after the reasoning trace, so a rollout cannot emit both labels, and a label leaked into the reasoning earns the format penalty. Rollout groups whose samples all agree carry zero gradient and are dropped online, which concentrates compute on the prompts the model is still inconsistent about.

3 · Results

Helpfulness and harmlessness pull against each other, and we read the guard on both axes to report four capabilities: (1) which data is dead weight before training, (2) the drift tax a one-sided scoreboard hides, (3) how aimed coverage lifts a flagged cell, and (4) the gates that keep risky moves safe.

Setup. The base is Nemotron-Content-Safety-Reasoning-4B (Sreedhar et al., 2025), SFT only. XSTest (Röttger et al., 2024) is the scoreboard, 450 prompts with greedy decoding. Under-refusal is measured on WildGuardTest (Han et al., 2024), an independent human-labeled set with adversarial and vanilla slices. The corpus grows from about 2K rows to 7,241 at the final iteration, every pack traced to a probe finding or an amendment.

3.1 · Dead-weight diagnosis

C-LIM flags the dead-weight region before any training budget is spent (Figure 6). Across the checkpoint ladder, 20 of 21 scored cells cluster healthy while privacy|public-vs-private stays flat at 0.80 as the field climbs to 0.98, C-LIM −11.9 against −0.33 for the next-worst cell. This matches a privacy pack a human had discarded weeks earlier as dead weight. The base's errors concentrate in exactly this family. Blind volume shows the cost of missing it: 187 rows added to that family without a targeting signal moved accuracy by 0.000.

Table 3. The SFT base vs the RL ship. Over-refusal drops 9.6 points and pair consistency rises 12, while under-refusal barely moves, all at no added test-time cost.

3.2 · The drift tax

RL cuts XSTest over-refusal from 22.4% to 12.8% while the scoreboard's under-refusal barely moves, so the boundary looks stable (Figure 7). On two independent sets the same shift tells the other half of the story: over-refusal falls everywhere, but under-refusal rises on every independent slice, worst on WildGuardTest's adversarial prompts. One axis is a scoreboard win, the other a hidden cost, and only the second channel and an independent set reveal it. Because both WildGuardTest and ToxicChat (Lin et al., 2023), real user traffic with human labels, show the same shape, the drift tax is a systematic boundary shift, not an artifact of our own generator.

Table 4. Over- and under-refusal error rates, SFT base vs the RL ship, on the XSTest scoreboard and three independent slices. Over-refusal improves on every eval. Under-refusal barely moves on the scoreboard but worsens on all three independent slices, which is the drift tax. XSTest, 450 prompts. WildGuardTest, 1,699. ToxicChat, 2,853.

3.3 · Targeted coverage

Aimed generation lifts a flagged cell where blind volume could not. The privacy family, flat at 0.733 under the 187 untargeted rows, moved to 0.80 once rows were aimed at its probed failure patterns, personal information about fictional characters and protected attributes of real acquaintances. This is the coverage-to-learning link on the helpfulness axis. Closing the drift tax on the adversarial axis is the open frontier, discussed in Section 4.

3.4 · Guardrails

Moves that change the rules are gated, and the gates reject real over-reach. The court-protocol rejected an amendment draft that flipped 4 of 60 settled precedents. The benefit gate rejected a new topic, sexual_content, that learned itself but regressed the rest of the board: global balanced accuracy fell 0.944 to 0.916, worst on privacy and discrimination (Figure 8). The board stays unchanged and the guard stays one model.

4 · Conclusion and future work

We framed guard RL data generation as building a constitution grid with measured moves, and we trained the reasoning guard with RL on the data those moves produce. On XSTest this cut over-refusal from 22.4% to 12.8% without mining the eval set. C-LIM flags dead-weight data before any budget is spent, and a second channel on an independent set reveals the drift tax a one-sided scoreboard hides.

Appendix

A · The drift tax across evaluations

Table 4 in Section 3.2 gives the per-eval numbers: over-refusal improves on every eval, while under-refusal barely moves on the scoreboard but worsens on all three independent slices.

B · What the evaluation contains

The scoreboard is 450 prompts, 250 safe and 200 unsafe, trigger-matched so each unsafe prompt has a benign twin sharing its trigger word (Figure 9). The median prompt is 8 words, so the eval reads as short, isolated probes, one reason the independent slices in Appendix A matter.

C · Where the baseline fails

Figure 10 breaks the SFT base's accuracy down by XSTest family. Errors concentrate in the privacy and safe-context families, and these counts seed the first generation quotas on the board.

D · Open discussion

Coverage is meant to drive learning, and on the helpfulness axis it does: aimed rows lifted the flagged privacy cell from 0.733 to 0.80 where 187 untargeted rows bought nothing. We do not yet have that evidence on the harmlessness axis.

The drift tax shows why. RL moved the guard along the over- and under-refusal tradeoff rather than through it, a shift of the boundary, not a sharpening of it (Figure 11). Coverage that lowers both at once is the next experiment, not a claim we make here.

We report the negatives plainly. Adding a topic can help itself while hurting the rest of the board, which is why the gate exists. The contribution is the loop and its two measurements, not a win over volume.