Score Matching Without the Normalizer

The missing constant

A probability model can tell us which observations look plausible without giving us its density in a form that is easy to normalize.

That distinction matters. Suppose a model assigns an unnormalized score

q(x;\theta) = \exp\{-E(x;\theta)\}.

The corresponding density is

p(x;\theta) = \frac{q(x;\theta)}{Z(\theta)}, \qquad Z(\theta)=\int q(x;\theta)\,dx.

The energy $E$ is easy to evaluate. The integral $Z(\theta)$ may be impossible to compute in a high-dimensional space. Maximum likelihood appears to require it anyway.

Hyvärinen’s 2005 paper proposes a different target for continuous-valued data: match the slope of the model’s log-density to the slope of the data distribution. The slope is called the score.

Compare local gradients, not absolute density heights. A constant multiplier disappears when we take a derivative.

This article follows the paper’s central theorem, then makes its three demonstrations tangible: a one-dimensional score field, the integration-by-parts objective, and the independent-component analysis example.

Why the normalizer gets in the way

Given observations $x^{(1)},\ldots,x^{(T)}$ , maximum likelihood maximizes

\frac{1}{T}\sum_{t=1}^{T}\log p(x^{(t)};\theta) = \frac{1}{T}\sum_{t=1}^{T}\log q(x^{(t)};\theta)-\log Z(\theta).

The first term is a sample average. The second term is an integral over every possible value of $x$ .

For a one-dimensional toy model, we can still draw the shape and its local slope. Move the model parameters in the panel below. The curve’s shape changes locally, while its normalization would change globally.

The normalizer is global; the score is local

Adjust the model centre and precision. The score field depends on the derivative of log q, not on the unknown constant that would make q integrate to one.

The paper’s proposal is not to approximate $Z$ . It is to choose an objective whose derivatives never contain $Z$ in the first place.

What this method assumes

Score matching in this paper is defined for smooth densities over $\mathbb{R}^n$ . The boundary terms used in the proof must vanish, and the relevant expectations must be finite. Discrete variables and nonsmooth boundaries need different treatment.

Match the score, not the density

Let the data density be $p_x(x)$ and define its score

\psi_x(x)=\nabla_x\log p_x(x).

For the model, write

\psi(x;\theta)=\nabla_x\log p(x;\theta).

Because $p=q/Z$ and $Z$ does not depend on $x$ ,

\psi(x;\theta)=\nabla_x\log q(x;\theta).

The unknown normalizer has vanished. Hyvärinen defines the population objective as the expected squared distance between these vector fields:

J(\theta)=\frac12\,\mathbb{E}_{p_x} \left[\|\psi(x;\theta)-\psi_x(x)\|^2\right].

At the correct model, the two arrows agree almost everywhere and $J=0$ . We do not observe $\psi_x$ directly, though. The next step removes it without estimating a density.

In one dimension, the score is just a slope. Positive means “move right toward greater density”; negative means “move left.” In many dimensions, it is a vector pointing uphill in log-density.

A score field is a set of local instructions

Hover or tap the field. The arrows show how a model would move a sample using only derivatives of log q.

The integration-by-parts trick

Expand the squared distance and ignore the term that depends only on the data:

J(\theta)=\mathbb{E}_{p_x}\left[ \frac12\|\psi(x;\theta)\|^2 -\psi_x(x)^\top\psi(x;\theta) \right]+\text{const.}

The cross-term seems to require the unknown data score. But for each coordinate, partial integration turns it into a derivative of the model score:

-\mathbb{E}_{p_x}[\psi_{x,i}(x)\,\psi_i(x;\theta)] = \mathbb{E}_{p_x}\left[\frac{\partial}{\partial x_i}\psi_i(x;\theta)\right].

Therefore the usable objective is

J(\theta)=\mathbb{E}_{p_x}\left[ \sum_{i=1}^{n}\frac{\partial\psi_i(x;\theta)}{\partial x_i} +\frac12\sum_{i=1}^{n}\psi_i(x;\theta)^2 \right]+\text{const.}

The data density appears only through an expectation, which becomes a sample average. No data-score estimator. No partition-function integral.

Two terms replace one impossible term

For a Gaussian-shaped unnormalized model, tune the precision. The curvature term and squared-score term trade off until the population objective is minimized at the data precision.

The identity is exact under the paper’s regularity conditions. It is not a Monte Carlo approximation to the normalizer.

From Gaussians to independent components

The paper first checks score matching on a multivariate Gaussian. For

q(x;M,\mu)=\exp\left[-\frac12(x-\mu)^\top M(x-\mu)\right],

the model score is $-M(x-\mu)$ . Minimizing the sample objective recovers the sample mean and inverse sample covariance, exactly as maximum likelihood does in this special case.

Gaussian estimation from local slopes

Draw more observations and watch the score-matching estimates settle. In the Gaussian case, the result agrees with maximum likelihood for every finite sample.

The more revealing example is independent component analysis (ICA). A linear mixture hides independent sources:

x=As, \qquad s_1,\ldots,s_n\text{ independent}.

The estimator searches for rows $w_k^\top$ whose projections behave like independent components. With a logistic component model, the score uses a saturating nonlinearity such as $g(u)=\tanh(u)$ .

Rotate the mixture back into its sources

Choose an unmixing angle. The score-matching objective is lowest when the projections recover the two independent source axes.

Why this example matters

The normalization constant is known in the basic ICA setup, so score matching is not needed merely to make the example computable. It is a controlled demonstration that the objective can recover the generative structure and remain useful when the component score is slightly misspecified.

A practical recipe

For a smooth unnormalized model $q(x;\theta)$ :

Differentiate $\log q$ with respect to the data coordinates to get $\psi_i(x;\theta)$ .
Differentiate each score coordinate once more with respect to its matching coordinate.
Evaluate $\widetilde J(\theta)=\frac1T\sum_{t=1}^{T}\left[\sum_i \partial_i\psi_i(x^{(t)};\theta)+\frac12\|\psi(x^{(t)};\theta)\|^2\right].$
Minimize the sample objective, remembering that optimization can still have local minima.

The implementation is derivative bookkeeping

The hard part moves from integrating q over all of space to computing first and second derivatives of log q at observed samples.

The paper proves local consistency under its assumptions: as the sample size grows, the empirical objective approaches the population objective, and the true parameter is the unique global minimizer when the model is identifiable.

This does not mean every optimizer finds the right basin, nor that every modern energy-based model satisfies the assumptions automatically. It means the objective itself is statistically well-founded in the setting analyzed by the paper.

The takeaway

Score matching changes the question.

Instead of asking, “How much probability mass does this model assign after normalization?”, it asks, “Does the local geometry of this model point in the same direction as the data distribution?”

That change removes the partition function from the estimator. A difficult global integral becomes a sample average of local derivatives:

\boxed{ \widetilde J(\theta)=\frac1T\sum_t \left(\operatorname{div}\psi(x^{(t)};\theta)+\frac12\|\psi(x^{(t)};\theta)\|^2\right) }

Hyvärinen’s result is compact, but its lesson is broad: when absolute values are hard to calibrate, gradients can still carry enough information to learn the shape.

Read the source

Aapo Hyvärinen. “Estimation of Non-Normalized Statistical Models by Score Matching.” Journal of Machine Learning Research, 6(24):695–709, 2005. Original PDF · JMLR entry

Score Matching
Without the Normalizer

Author

Affiliation

Published

PDF