falkon.hopt
Objectives
- class falkon.hopt.objectives.objectives.HyperoptObjective(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None, pen_transform: Transform | None)
Nystrom Complexity Regularization
- class falkon.hopt.objectives.NystromCompReg(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)
Stochastic Nystrom Computational Regularization
- class falkon.hopt.objectives.StochasticNystromCompReg(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, flk_opt: FalkonOptions, flk_maxiter: int = 10, num_trace_est: int = 20, centers_transform: Transform | None = None, pen_transform: Transform | None = None)
Complexity Regularization
- class falkon.hopt.objectives.CompReg(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)
Generalized Cross Validation
- class falkon.hopt.objectives.GCV(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)
GCV objective is
\[\dfrac{\dfrac{1}{n} \lVert (I - \widetilde{K}_\lambda \widetilde{K}) Y \rVert^2} {\frac{1}{n} \mathrm{Tr}(I - \widetilde{K}_\lambda \widetilde{K}) }\]We must compute the two terms denoted as the numerator and the denominator. Using the usual names for matrix variable substitutions (taken from gpflow code), we have that the numerator can be computed as
\[\dfrac{1}{n} \lVert (I - A^\top \mathrm{LB}^{-\top} \mathrm{LB}^{-1} A) Y \rVert^2\]We compute the terms inside the norm first, from right to left using matrix-vector multiplications and triangular solves. Finally we compute the norm.
The denominator is far less efficient to compute, since it requires working with m*n matrices. It can be expressed in terms of the same matrices as above:
\[\Big( \frac{1}{n} (\mathrm{Tr}(I) - \lVert \mathrm{LB}^{-1}A \rVert_F^2 ) \Big)^2\]
Hold Out Cross Validation
- class falkon.hopt.objectives.HoldOut(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, val_pct: float, per_iter_split: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)
Leave One Out Cross Validation
- class falkon.hopt.objectives.LOOCV(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)
LOOCV objective is to minimize the PRESS error:
\[\min \sum_{i=1}^n \big(e_{(i)}\big)_i^2\]whose components are calculated as
\[\big(e_{(i)}\big)_i = (y_i - f(\alpha_{(i)})_i) = \dfrac{y_i - f(\alpha)_i}{1 - S_{ii}}\]where
\[S = K_{nm} (\lambda K_{mm} + K_{nm}^\top K_{nm})^{-1} K_{nm}^\top\]So we first need to compute the N-KRR solution \(\alpha\) and then the diagonal of matrix \(S\). For simplicity we use a direct solver the N-KRR problem, although Falkon could easily be substituted in.
SGPR
- class falkon.hopt.objectives.SGPR(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)