变分法
函数\(y(x)\)对于任意给定的输入变量\(x\)，给出输出值\(y\)；类似地，定义关于函数的函数\(F[y]\)，亦称泛函，给定函数\(y\)，输出值为\(F\)。熵\(\text{H}[x]\)也是泛函的一种，它定义在概率密度函数\(p(x)\)上，可等价记为\(\text{H}[p]\)。
泛函中变分法类似于函数中求极值点，即寻求某个最大化或最小化泛函\(F[y]\)的函数\(y(x)\)。利用变分法可证明两点之间的最短路径为直线以及最大熵分布为高斯分布。
对于多元函数\(y(\mathbf{x})=y(x_1,...,x_D)\)，其泰勒展开为
\[y(\mathbf{x}+\boldsymbol\epsilon)=y(\mathbf{x})+\boldsymbol\epsilon^\text{T}\frac{\partial y}{\partial \mathbf{x}}+O(\Vert\boldsymbol\epsilon\Vert^2).\] 对于某个泛函\(F[y]\)，考虑\(y(x)\)上的微小改变\(\epsilon\eta(x)\)，其中\(\eta(x)\)为任意函数，对比上式，将\(\mathbf{x}\)展开到无限维，从而\[F[y(x)+\epsilon\eta(x)]=F[y(x)]+\int \frac{\delta F}{\delta y(x)}\epsilon\eta(x){\rm{d}}x+O(\epsilon^2),\] 其中\(\delta F/\delta y(x)\)为泛函梯度。为了使\(F[y]\)取得极值，上式一阶条件为
\[\begin{aligned}\lim_{\epsilon\rightarrow 0}\frac{ \int \frac{\delta F}{\delta y(x)}\epsilon\eta(x){\rm{d}}x}{\epsilon}&=\int \frac{\delta F}{\delta y(x)}\eta(x){\rm{d}}x \\&=0, \end{aligned}\] 注意到\(\eta(x)\)选取的任意性，可得泛函梯度需要处处为零，即\[\frac{\delta F}{\delta y(x)}\equiv 0.\]考虑以下实例，泛函\(F[y]\)定义如下
\[F[y]=\int G(y(x),y'(x),x)\text{ d}x,\] 并且假定\(y(x)\)在积分边界上值为常数，从而在积分边界上\(\eta(x)=0\)。 考虑\(y(x)\)上的变分\[\begin{aligned} F[y(x)+\epsilon\eta(x)]&= \int G\left(y(x)+\epsilon \eta(x),y'(x)+\epsilon\eta'(x),x\right)\text{ d} x\\&= \int\left\{G\left(y(x),y'(x),x\right)+\frac{\partial G}{\partial y}\epsilon\eta(x)+\frac{\partial G}{\partial y'}\epsilon\eta'(x)\right\}\text{ d}x+O(\epsilon^2)\\&= F[y(x)]+\epsilon\int\left\{\frac{\partial G}{\partial y}\eta(x)+\frac{\partial G}{\partial y'}\eta'(x)\right\}\text{ d}x+O(\epsilon^2) \end{aligned}\] 注意到\[\begin{aligned} \int \frac{\partial G}{\partial y'}\eta'(x)\text{ d}x&=\int \frac{\partial G}{\partial y'}\text{ d}\eta(x)\\&= \eta(x)\frac{\partial G}{\partial y'}\Big\vert-\int\eta(x)\frac{\text{d}}{\text{d}x}\left(\frac{\partial G}{\partial y'}\right)\text{ d}x\\&= -\int\eta(x)\frac{\text{d}}{\text{d}x}\left(\frac{\partial G}{\partial y'}\right)\text{ d}x \end{aligned}\] 带入上式，整理得\[\begin{aligned} F[y(x)+\epsilon\eta(x)]&=F[y(x)]+\epsilon\int\left\{\frac{\partial G}{\partial y}-\frac{\text{d}}{\text{d}x}\left(\frac{\partial G}{\partial y'}\right)\right\}\eta(x)\text{ d}x+O(\epsilon^2), \end{aligned}\] 令泛函梯度为零\[\frac{\partial G}{\partial y}-\frac{\text{d}}{\text{d}x}\left(\frac{\partial G}{\partial y'}\right)=0,\] 余下步骤可用微分方程求解。