Random perceptrons: free energy, correlated patterns, and microscopic jamming

← All digests

Family 222 · Probability and statistical mechanics
Importance 5/5 · Provisional editorial rating
Sources consulted 2026-10-11

1. The question

A perceptron chooses a vector of weights and tests its projections onto a collection of input vectors. Can it satisfy many random requirements at once? If violations are allowed but penalized, how many good choices remain? Near the point where satisfying every requirement becomes impossible, do the nearly binding constraints follow universal power laws?

These questions link a basic learning model to spin glasses and to jamming: the loss of freedom that occurs when too many constraints compete for too few degrees of freedom. The connection concerns the geometry of a high-dimensional optimization problem, rather than the behavior of a particular training algorithm.

Family 222 contains four manuscripts. Two claim general limiting free-energy formulas for independent Gaussian patterns, with either binary or spherical weights. A third treats spherical weights with correlated, rotationally invariant patterns. The fourth claims microscopic gap and force exponents at the jamming transition of a particular negative-margin spherical model. The last result requires additional arguments beyond the bounded-potential free-energy theorems.

2. Background and essential definitions

Patterns, weights, and pressure

Let $N$ be the number of weights and $M=\lfloor\alpha N\rfloor$ the number of patterns, where the density $\alpha>0$ stays fixed as $N$ grows. For independent standard Gaussian vectors $g^a\in\mathbb R^N$, define the field

$$b_a(x)=\frac{g^a\cdot x}{\sqrt N}.$$

The Ising model restricts $x$ to $\{-1,1\}^N$; the spherical model imposes $\|x\|^2=N$. In either case, $\mu_N$ denotes the uniform probability measure on the allowed weights. For a real log-potential $f$, set

$$Z_N=\int \exp\!\left(\sum_{a=1}^M f(b_a(x))\right)d\mu_N(x), \qquad p_N=\frac1N\log Z_N.$$

The pressure $p_N$ balances rewards against the scarcity of configurations earning them. Taking $f=\beta\varphi$ introduces inverse temperature $\beta$. The physical free-energy density is then $-p_N/\beta$. Because the prior has total mass one, zero potential gives zero pressure; replacing the Ising prior by counting measure adds $\log 2$.

Two independently sampled configurations have overlap $R(x,x')=x\cdot x'/N$. A single typical overlap gives a replica-symmetric description. A hierarchy of overlaps describes clusters, subclusters, and successively closer configurations. The corresponding variational formulas optimize over a whole nondecreasing function, not just one scalar.

Hard constraints and jamming

A hard requirement assigns weight zero to a forbidden configuration. Its log-potential is therefore $-\infty$, which is outside a theorem for bounded real $f$.

For the jamming manuscript, write $y=x/\sqrt N$, so $\|y\|=1$, and let $A$ have independent standard Gaussian entries initially. Its constraints are

$$h_a(y)=1+(Ay)_a\geq0.$$

Thus the required projection is at least $-1$: the margin is negative. A positive gap $h_a$ measures spare room; zero marks contact with a constraint. The energy

$$H_N(y)=\frac12\sum_{a=1}^M(-h_a(y))_+^2, \qquad z_+=\max\{z,0\},$$

penalizes violations. The model is satisfiable, or SAT, if some unit vector has zero energy; otherwise it is UNSAT. Negative margin and the spherical restriction produce a nonconvex feasible-set problem.

3. Development history and previous partial results

Gardner introduced the statistical-mechanical storage-volume problem, and Gardner and Derrida studied both spherical and binary weights. Their work made the number and organization of feasible classifiers central objects. Krauth and Mézard subsequently developed the binary storage-capacity prediction. Gardner’s original paper provides the starting point for this history.

Rigorous Ising formulas first covered restricted regimes, including Talagrand’s small-density and high-temperature results. Bolthausen, Nakajima, Sun, and Xu developed a small-density approach using approximate message passing and conditional moment estimates. M1 presents its removal of density and temperature restrictions for bounded log-potentials as the main advance over this line. Separate hard-constraint capacity results do not automatically evaluate every positive-temperature pressure, or vice versa.

For spherical models, Shcherbina and Tirozzi established a Gardner formula in a nonnegative-margin hard-constraint regime. The more general hierarchical finite-temperature expression also predates these manuscripts: Györgyi and Reimann developed continuous replica symmetry breaking, and Montanari and Zhou supplied a related replica derivation alongside rigorous results about algorithmically attainable projections. M4 claims a rigorous evaluation of the general bounded-potential pressure, rather than the invention of the physical functional.

The jamming story grew from the theory of high-dimensional hard spheres. Franz and Parisi proposed the negative spherical perceptron as a minimal model sharing their critical behavior. Franz, Parisi, Sevelev, Urbani, and Zamponi analyzed the SAT–UNSAT transition and its scaling predictions. The predicted exponents and their relation were therefore established targets. M2 claims to derive them with the necessary existence, limiting, and profile-selection arguments.

4. What the new manuscripts claim

Independent Gaussian patterns: binary and spherical weights

M1 claims convergence of the Ising pressure, both in expectation and probability, for every bounded Borel real log-potential, every fixed positive density, and every fixed positive temperature. Borel measurability allows discontinuities. There is no symmetry, concavity, or smallness assumption on the potential.

Its formula has the structure

$$\lim_{N\to\infty}\mathbb E p_N =\inf_q\{\alpha\mathcal V_f(q)+\mathcal S_I(q)\}.$$

Here $q:(0,1)\to[0,1]$ is a nondecreasing overlap quantile. The pattern term $\mathcal V_f$ is obtained by successively averaging Gaussian fields along the hierarchy; $\mathcal S_I$ is the associated binary-spin entropy term, defined through a dual optimization of one-spin $\log\cosh$ recursions. The coefficient $\alpha$ counts patterns per weight. The two contributions must use the same hierarchy.

M4 gives the spherical analogue for every bounded continuous potential, again at arbitrary fixed density and temperature. Its formula can be described using a nondecreasing function $m:[0,1]\to[0,1]$. Define

$$D_m(t)=\int_t^1m(s)\,ds, \qquad \mathcal S(m)=\frac12\int_0^1 \left(\frac1{D_m(t)}-\frac1{1-t}\right)dt.$$

An infinite value excludes an inadmissibly expensive hierarchy. The pattern term has a stochastic-control description:

$$\mathcal V_f(m)=\sup_v\mathbb E\left[ f\!\left(B_1+\int_0^1m(t)v_t\,dt\right) -\frac12\int_0^1m(t)v_t^2\,dt\right].$$

Here $B$ is standard Brownian motion, and $v_t$ may depend on information observed up to time $t$, with finite expected cost. One chooses a drift to improve the terminal reward while paying a quadratic penalty. The claimed limiting pressure is $\inf_m\{\alpha\mathcal V_f(m)+\mathcal S(m)\}$. The variable $t$ indexes the hierarchy; it is not the physical evolution time of a learning process.

Correlated rotationally invariant patterns

M3 replaces the independent-pattern matrix by an $M\times N$ matrix whose distribution is unchanged by orthogonal rotations on either side. Its singular-vector bases are uniformly random, while its singular values can have a prescribed limiting distribution. In this normalization the field is $Ax$, with the usual independent Gaussian comparison having entries of size $N^{-1/2}$.

For bounded continuous $f$, a deterministic compact limiting singular-value distribution, and the stated spectral control, M3 claims a deterministic pressure limit in expectation and probability. The main theorem excludes singular-value outliers away from the limiting support; an appendix weakens this to control by the interval between its extreme points, with a further allowance for a structural input null space.

The formula optimizes how much squared input norm lies in each singular-value band, together with a hierarchical angular contribution. It permits spectral atoms, gaps, and rank deficiency. Thus the spectrum of correlations enters the answer explicitly; replacing the matrix by an independent Gaussian one would generally change the model.

Microscopic jamming at margin minus one

M2 claims a finite positive critical density $\alpha_c$: SAT probability tends to one for fixed $\alpha<\alpha_c$, and to zero for fixed $\alpha>\alpha_c$. It further claims the same threshold and critical conclusions when each matrix entry has distribution

$$\tfrac12\mathcal N(0,1-\varepsilon)+ \tfrac12\mathcal N(0,1+\varepsilon),$$

for sufficiently small fixed $\varepsilon\geq0$. This is a particular variance-one non-Gaussian family.

Its microscopic statements use an ordered limit: first $N\to\infty$, then inverse temperature $\beta\to\infty$, then $\alpha\downarrow\alpha_c$ from the UNSAT side. Define $G_J(u)$ as the limiting fraction of constraints with positive gap at most $u$, excluding the contact atom by first imposing a positive lower cutoff and subsequently sending that cutoff to zero.

For violated constraints, normalize the magnitude $-h_a$ by its average over those constraints. The resulting forces have mean one. Their limiting cumulative distribution is $F_J(s)$. M2 claims

$$G_J(u)=u^{1-\gamma+o(1)},\qquad F_J(s)=s^{1+\theta+o(1)},\qquad \gamma=\frac1{2+\theta},$$

as $u,s\downarrow0$, with

$$0.4126930<\gamma<0.4126934, \qquad 0.4231063<\theta<0.4231088.$$

These are cumulative-law statements; differentiating them into density asymptotics would require additional justification. The complete gap law has a contact atom of mass $1/\alpha_c$. Since there are approximately $\alpha_c N$ constraints, this expresses one contact per degree of freedom, or isostaticity. The positive-gap function itself need not have total mass one.

5. Why the advance matters

Importance: 5/5. The family addresses both a central thermodynamic quantity across broad classes of perceptrons and the detailed critical geometry of a major jamming model.

The general pressure formulas would replace regime-specific calculations with a common framework for arbitrary bounded rewards. The rotationally invariant extension would show how correlated data change that framework. The jamming claim reaches further than locating a phase boundary: it describes the abundance of almost-binding constraints and unusually weak forces at that boundary.

The connection is mathematically valuable because free energy is a global average, while small-gap and small-force tails resolve a delicate microscopic population. Neither level of information automatically supplies the other. These statements would connect the hierarchical thermodynamics to its most singular geometric predictions.

The rating concerns the mathematical claims as stated. It does not assert that the formulas yield efficient training algorithms or directly predict the behavior of a finite neural network.

6. The main idea

For the independent-pattern pressure, one first enriches the model with auxiliary Gaussian fields that encode candidate overlap hierarchies. Gaussian averaging proceeds level by level, retaining shared randomness for configurations in the same branch. Small perturbations organize overlap identities without changing the limiting pressure.

The upper and lower bounds have different jobs. A variational comparison controls pressure from above despite the nonlinear pattern interaction. A cavity calculation removes and restores a small number of variables or patterns, expressing the pressure increment through the limiting overlap structure. Matching those increments identifies the entropy and pattern terms. Gaussian patterns alone are insufficient to invoke the ordinary spin-glass formula: after applying a nonlinear $f$, the Hamiltonian is generally no longer a Gaussian process.

For rotationally invariant matrices, M3 first separates radii from angles. Putting an atypical fraction of the input norm into a spectral band has an entropy cost. Conditional on those radii, random orthogonal subspaces govern the angular problem. Optimizing both parts retains information that a single scalar variance would lose.

The jamming argument needs its own control of the unbounded quadratic penalty. It also must turn small energy into exact feasibility: an energy density tending to zero does not by itself produce a vector satisfying every constraint. M2 describes a repair argument using a small surplus of constraints before removing a fraction of them.

At criticality, its variational description supplies coupled gap and force scales. A selected limiting profile determines both exponents, while conservation and normalization link them. The manuscript reports a finite numerical enclosure for the resulting spectral problem, yielding the displayed intervals. This is an attributed component of the source’s claimed result, not a separate numerical check here.

7. Scope, limitations, and open questions

The main free-energy theorems use fixed finite temperature and fixed density. A bounded soft potential cannot simply be replaced by a hard indicator or an infinite penalty without controlling another limit. M2 supplies separate analysis for its specific negative-margin quadratic model; it is not a consequence of substituting that potential into M4’s bounded theorem.

M1 also discusses integrable unbounded potentials and random-sign pattern universality under additional hypotheses. For the latter, bounded potentials must have a discontinuity set of Gaussian measure zero; arbitrary bounded Borel functions do not all qualify. M3 likewise states extensions beyond continuous bounded potentials with their own spectral and integrability conditions.

The jamming theorem fixes margin $-1$, approaches from the UNSAT side, and uses the stated order of limits. Its small Gaussian-mixture universality does not cover every independent-entry distribution. It does not state SAT probability at the critical density itself, nor give a universal finite-size scaling window.

Further questions include broader disorder universality, other margins, convergence rates, and the relation between equilibrium predictions and what feasible algorithms actually find. Finite-dimensional particle packings remain a different model; the perceptron’s connection to them is a motivating universality picture, not an identification of their theorems.

8. Sources and further reading

For the origins of the storage problem, begin with Gardner’s paper linked above. Györgyi–Reimann and Montanari–Zhou explain the earlier hierarchical and algorithmic viewpoints. Franz–Parisi and the subsequent SciPost paper develop the physical jamming predictions that M2 targets.

In the source list below, M1 concerns binary weights, M4 independent Gaussian spherical weights, M3 correlated rotationally invariant matrices, and M2 the microscopic jamming transition. Their assumptions should be read separately when applying a particular conclusion.

Source manuscripts

• M1. OpenAI, The free energy of the Ising random perceptron (September 24, 2026).
• M2. OpenAI, Microscopic jamming in the negative spherical perceptron (September 24, 2026).
• M3. OpenAI, The spherical perceptron with bi-orthogonally invariant disorder (September 24, 2026).
• M4. OpenAI, The free energy of the spherical random perceptron (September 24, 2026).

Sources consulted on 11 October 2026 at repository revision fd4aeeb. Download the manuscript citations.