Two policy gradients, one blind spot

This distinction appears across reinforcement learning. PPO and GRPO use the score-function route [1], [2], [3]. SAC and differentiable simulation use pathwise derivatives, although they obtain those derivatives from different places [4], [5]. The names will make more sense after we solve one small robot problem. First, here is the whole setup in one picture: the input stays fixed, but a stochastic policy can sample different actions.

Start with one noisy robot action

Imagine that the robot repeats the same reach many times. The camera image, instruction, cup, and all but one joint stay fixed. The only number we record is where the gripper closes along the horizontal axis. Call this position $a$, measured in centimetres from the cup’s centre.

We model the action as

Here $\mu_\theta$ is the mean position chosen by policy parameters $\theta$, $\sigma$ is the amount of exploration noise, and $\epsilon$ is one draw from a standard normal distribution. In plain language: the policy aims at $\mu_\theta$, but each attempt lands a little to the left or right.

The robot receives one point if the gripper closes inside the success interval $G = [\ell,r]$, and zero otherwise:

The symbol $\mathbf{1}\{\cdot\}$ is an indicator: it equals $1$ when the condition is true and $0$ when it is false. Throughout the post, $\sigma = 3$ cm and $G = [-2,2]$ cm. This reward has no gentle slope: crossing either edge changes the reward instantly from $0$ to $1$, or from $1$ to $0$.

The second figure deliberately tells the same story twice. The robot drawings show the physical consequence; the strips below them show the mathematical abstraction. The next diagram adds the policy’s full action distribution on top of that line.

A policy update moves probability

Before deriving either gradient estimator, we need a precise description of what learning changes. Let $\lambda$ run continuously from $0$ to $1$ as the policy moves from its old parameters to its new ones: $\theta(0)=\theta_{\text{old}}$ and $\theta(1)=\theta_{\text{new}}$. This is not the robot’s physical time $t$. The index $t$ moves the robot through one rollout; $\lambda$ moves the policy through one update.#In the one-action example, we can simply set $\lambda=\mu$. Increasing $\lambda$ then means sliding the Gaussian policy to the right.

A rollout $\tau=(s_0,a_0,\ldots,s_T)$ is one complete sequence of states and actions. Let $\rho_\lambda(\tau)$ be the probability density that the current policy and environment assign to that rollout, and let $R(\tau)$ be its total reward. The objective is the expected reward:

Equation (1) says: consider every possible rollout, multiply its reward by its probability, and add the results. The integration variable $d\tau$ means that the integral ranges over all possible rollouts. For a fixed task, the scoring rule $R$ stays the same. The policy update changes $\rho_\lambda$, so it changes which rollouts are likely.

We can describe this change as a flow of probability. Let $v_\lambda(\tau)$ be a velocity field: it tells us the direction and speed at which probability near rollout $\tau$ moves as $\lambda$ increases. Conservation of probability gives the continuity equation:

The first term, $\partial_\lambda\rho_\lambda$, is the rate at which the density changes at a fixed location. The second term measures net probability flowing out of that location: $\rho_\lambda v_\lambda$ is the probability flux, $\nabla_\tau\!\cdot$ is its divergence, and the minus sign says that net outflow lowers the local density. In short, density changes because probability flows in or out; probability is not created or destroyed.

The one-dimensional robot example makes (2) easy to check. Set $\lambda=\mu$ and shift the Gaussian mean to the right. Every sampled action moves right at unit speed, so $v=1$. The equation becomes $\partial_\mu\rho_\mu(a)=-\partial_a\rho_\mu(a)$: changing the mean has the same effect as translating the whole density in the opposite coordinate direction.

Probability in a region falls not because probability was destroyed, but because it went somewhere else.

View one: keep the outcomes fixed

The first view keeps each possible rollout $\tau$ fixed and asks how its probability changes. Differentiate (1) with respect to $\lambda$. Because the task reward $R$ does not depend on the policy update, the derivative acts only on $\rho_\lambda$:

The last expression is the score-function, or likelihood-ratio, gradient. The reward $R(\tau)$ says how good the sampled rollout was. The score $\partial_\lambda\log\rho_\lambda(\tau)$ says how quickly the update changes that rollout’s relative probability. Their product is the rollout’s contribution to the gradient, and the expectation averages that contribution over sampled rollouts.

Why can this estimator work with a black-box environment? The probability of a rollout factors into the initial-state distribution $p_0$, the policy $\pi_\theta$, and the environment dynamics $P$:

The product on the left assigns a probability to the whole rollout. After taking the logarithm, that product becomes a sum. If the environment does not itself depend on $\theta$, the derivatives of $p_0$ and $P$ are zero, so only derivatives of the policy remain on the right. We need sampled states, actions, and rewards, plus $\nabla_\theta\log\pi_\theta$ from the policy network. We do not need derivatives of the environment or reward. This is the key idea behind REINFORCE [7] and the policy gradient theorem [8].

This route is tolerant of black boxes, but it is not assumption-free. Moving the derivative through the integral must be valid, and the set of possible rollouts—the support of $\rho_\theta$—must behave regularly. If the support itself changes shape with $\theta$, the usual formula needs extra care [9].

View two: keep the randomness fixed

The second view matches an outcome under the old policy with an outcome under the new one. Write $\epsilon\sim q$ for a random draw from a fixed noise distribution, and write $\tau=T_\lambda(\epsilon)$ for the rollout produced by passing that draw through the policy and environment. In the robot example, this is simply $a_\lambda=\mu_{\theta(\lambda)}+\sigma\epsilon$.

The important step is to give the old and new policies the same $\epsilon$. Think of $\epsilon$ as a numbered lottery ticket. If both policies receive the same ticket, we can say how that particular sampled outcome moved. If they receive independent tickets, we can compare two distributions, but we cannot match one old sample to one new sample.

With shared noise, $\partial T_\lambda(\epsilon)/\partial\lambda$ is the velocity of the sampled rollout. If the reward is differentiable along this path, the chain rule gives the pathwise gradient:

The vector $\nabla_\tau R$ is the local reward slope: it points toward nearby rollouts with higher reward. The vector $\partial T_\lambda/\partial\lambda$ says how the sampled rollout moves as the policy changes. Their dot product measures how quickly the reward changes along that motion, and the expectation averages over noise draws.

For a multi-step rollout, we can write $a_t=g_\theta(s_t,\epsilon_t)$ for the policy and $s_{t+1}=f(s_t,a_t,\xi_t)$ for the environment. Here $g_\theta$ maps a state and policy noise to an action; $f$ maps the current state, action, and environment noise $\xi_t$ to the next state. Once every $\epsilon_t$ and $\xi_t$ is held fixed, the rollout becomes an ordinary computation graph, $\theta\rightarrow a_0\rightarrow s_1\rightarrow\cdots\rightarrow R$, and backpropagation can differentiate it. Stochastic computation graphs formalize how score-function and pathwise estimators can be mixed at different random nodes [10].

Integration by parts connects the two views

The two formulas look unrelated: (3) differentiates probability, while (4) differentiates reward. The continuity equation shows that they describe the same flow. Substitute $\partial_\lambda\rho_\lambda=-\nabla_\tau\cdot(\rho_\lambda v_\lambda)$ into the score-function derivation, then use integration by parts to move the derivative from the probability flux onto the reward:

The left expression is the score-function view: keep locations fixed and watch their probability weights change. The middle expression replaces that weight change with probability flow. The right expression is the pathwise view: follow the moving probability and measure the reward slope along its path. Integration by parts is the operation that moves the derivative from $\rho$ to $R$.

This transport interpretation of pathwise derivatives is developed by Jankowiak and Obermeyer [6]. Parmas and Sugiyama give a unified treatment of likelihood-ratio and reparameterization gradients as estimators of the same probability movement [11].

Three caveats are important. First, equal expectations do not imply equal finite-sample variance. Second, the velocity field is not unique, so different pathwise estimators can have the same mean. Third, claim 1 assumes that the ordinary reward gradient $\nabla_\tau R$ exists. Our hard $0/1$ reward breaks that last assumption.

Where common algorithms fit

We can now sort familiar algorithms by one practical question: what derivative can the algorithm obtain outside the policy itself? The “view” column follows from that requirement.

The actor-critic row is easy to misunderstand. DDPG and SAC do not differentiate the real environment. They differentiate a learned critic $Q(s,a)$ with respect to the action. The smoothness requirement therefore falls on a neural network the algorithm controls, not directly on the world’s reward or dynamics. By contrast, methods that backpropagate through a learned dynamics model or a differentiable simulator need a longer differentiable path. Evolution strategies and random search sit outside both columns: they estimate changes in performance without forming $\nabla_\theta\log\pi_\theta$ at all [16], [17].

One boundary is enough to fool autodiff

Return to the robot’s actual $0/1$ reward and set the update coordinate $\lambda$ equal to the policy mean $\mu$. The objective $J(\mu)$ is now simply the probability that a Gaussian action lands between $\ell$ and $r$. Let $\Phi$ denote the standard normal cumulative distribution function and $\phi$ its probability density. Then both the objective and its exact derivative have closed forms:

The formula for $J$ subtracts the Gaussian probability to the left of $\ell$ from the probability to the left of $r$, leaving exactly the probability inside the success interval. The derivative has an equally concrete meaning: $\phi((\ell-\mu)/\sigma)/\sigma$ is probability density entering through the left edge as the policy moves right, while $\phi((r-\mu)/\sigma)/\sigma$ is density leaving through the right edge. Their difference is inflow minus outflow.

The same conclusion follows directly from the continuity equation. Integrating (2) over $G=[\ell,r]$ gives

The two terms are the probability flux through the interval’s left and right edges. In our example $v=1$, so the gradient is $\rho_\mu(\ell)-\rho_\mu(r)$, matching the formula above and figure 4.#This boundary is not the boundary term at infinity in claim 1. It appears because we integrate over the finite success interval $G$.

The score-function view has no problem with this boundary because it never differentiates the reward. For one sampled action, its gradient estimate is

The reward $R(a)$ says whether the sample succeeded. The factor $(a-\mu)/\sigma^2$ is the derivative of the Gaussian log-probability with respect to its mean. Averaging these products gives an unbiased estimate: over repeated sample sets, its mean equals the exact gradient.

Now apply ordinary autodiff through the same sampled action. The chain rule produces

The action moves one-for-one with $\mu$, which explains $\partial a/\partial\mu=1$. But the indicator reward is flat everywhere except at its two edges, so ordinary autodiff sees $\partial R/\partial a=0$ for every sample that does not land exactly on an edge. A continuous distribution hits either exact edge with probability zero. Therefore the estimator is not merely noisy: it returns exactly zero for every practical sample set, at every $\mu$.

Why is the true gradient nonzero? Because success improves when probability crosses the boundary, not because reward has a local slope inside either flat region. In figure 6, all particles shift by the same amount. The change in success count comes entirely from particles crossing into or out of $G$. Pointwise autodiff examines the neighborhood around each sampled particle, so it misses a contribution concentrated exactly at the edges.

In generalized-function notation, the missing derivative is

where the Dirac delta $\delta$ represents a unit contribution concentrated at one point. Substituting this generalized derivative into (4) with $v=1$ recovers $\rho_\mu(\ell)-\rho_\mu(r)$. So the pathwise identity is not mathematically wrong. The problem is that ordinary autodiff differentiates the program’s pointwise operations and does not automatically recover this boundary contribution.

For example, when $\mu=-3$ cm, the robot succeeds about $32\%$ of the time and the exact gradient is $0.09264$. The score-function Monte Carlo mean is $0.09208$, close to the exact answer. Naive pathwise autodiff still reports $0$, while the smoothed reward reports $0.08558$—a useful direction, but a biased value for the original objective.

A common workaround is to replace each hard step with a smooth sigmoid:

Here $\varsigma(x)=1/(1+e^{-x})$ is the logistic sigmoid, and $\beta$ controls how sharp the two softened edges are. This surrogate reward has nonzero slopes near $\ell$ and $r$, so autodiff can see a pathwise signal. But it is a different objective. At $\beta=2$, the resulting gradient is about $8\%$ too small at $\mu=-3$ and about $19\%$ too large at $\mu=-9$; the bias even changes sign. Increasing $\beta$ makes the surrogate closer to the hard reward, but it also concentrates the useful gradient into narrower regions near the edges, which tends to increase variance.

The exact objective, exact derivative, score-function estimate, and variance calculation fit in a few lines of dependency-free Python. The code below makes the numerical claims reproducible.¹

import math

SIGMA, L, R = 3.0, -2.0, 2.0

def phi(x): return math.exp(-0.5 * x * x) / math.sqrt(2.0 * math.pi)
def Phi(x): return 0.5 * (1.0 + math.erf(x / math.sqrt(2.0)))

def J(mu):
    """Success probability in closed form."""
    return Phi((R - mu) / SIGMA) - Phi((L - mu) / SIGMA)

def dJ(mu):
    """Its exact derivative: density at the left edge minus density at the right."""
    return (phi((L - mu) / SIGMA) - phi((R - mu) / SIGMA)) / SIGMA

def score(mu, n, rng):
    """Unbiased at every mu; needs only sampled successes and failures."""
    xs = [mu + SIGMA * rng.gauss(0.0, 1.0) for _ in range(n)]
    return sum((1.0 if L <= a <= R else 0.0) * (a - mu) / SIGMA**2 for a in xs) / n

def score_sd(mu):
    """Exact standard deviation of one score sample, via int u^2 phi(u) du."""
    ul, ur = (L - mu) / SIGMA, (R - mu) / SIGMA
    m2 = (ul * phi(ul) - ur * phi(ur) + Phi(ur) - Phi(ul)) / SIGMA**2
    return math.sqrt(m2 - dJ(mu) ** 2)

# The naive pathwise estimator is a function that sums zeros, so it is not written here:
# dR/da is 0 wherever it exists, and the mean of n zeros is 0 for every n.

What each view costs

The score-function estimator can see a hard boundary, but rare successes make it noisy. If one sample has standard deviation $s$ and the true gradient has magnitude $|g|$, then the number of independent samples needed for a standard error equal to $10\%$ of $|g|$ is approximately

This equation comes from the usual standard error $s/\sqrt{n}$. As the policy mean moves away from the cup, success becomes rare, $|g|$ becomes small relative to the estimator’s noise, and the required sample count grows rapidly:

The two views therefore fail in opposite ways on the same task:

• The score-function view sees the correct boundary flux without differentiating the reward, but it needs many samples when successes are rare.
• The pathwise view often provides a low-variance local signal when the computation is smooth, but ordinary autodiff sees no signal at all from this hard boundary.

This does not contradict claim 1. The theorem equates the ideal expectations under its regularity assumptions; it does not promise that every finite-sample implementation is useful, or that ordinary autodiff will construct the generalized derivative of a discontinuity.

Read table 1 with that tradeoff in mind. PPO and GRPO can learn from a $0/1$ reward, but may pay heavily in samples. A differentiable simulator given the same raw indicator reward can return zero because the gradient it computes contains no boundary term.#One practical compromise is to use an accurate simulator for the forward pass and a smoothed surrogate for the backward pass [19].

The main lesson is simple: a policy gradient measures how probability moves between outcomes. You can estimate that movement by watching probability weights change or by following samples through the computation. Hard reward boundaries make the choice visible: the true gradient lives in probability crossing the boundary, and an estimator is useful only if it can see that crossing.