falkon.hopt

Objectives

class falkon.hopt.objectives.objectives.HyperoptObjective(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None, pen_transform: Transform | None)

Nystrom Complexity Regularization

class falkon.hopt.objectives.NystromCompReg(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)

Stochastic Nystrom Computational Regularization

class falkon.hopt.objectives.StochasticNystromCompReg(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, flk_opt: FalkonOptions, flk_maxiter: int = 10, num_trace_est: int = 20, centers_transform: Transform | None = None, pen_transform: Transform | None = None)

Complexity Regularization

class falkon.hopt.objectives.CompReg(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)

Generalized Cross Validation

class falkon.hopt.objectives.GCV(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)

GCV objective is

\[\dfrac{\dfrac{1}{n} \lVert (I - \widetilde{K}_\lambda \widetilde{K}) Y \rVert^2} {\frac{1}{n} \mathrm{Tr}(I - \widetilde{K}_\lambda \widetilde{K}) }\]

We must compute the two terms denoted as the numerator and the denominator. Using the usual names for matrix variable substitutions (taken from gpflow code), we have that the numerator can be computed as

\[\dfrac{1}{n} \lVert (I - A^\top \mathrm{LB}^{-\top} \mathrm{LB}^{-1} A) Y \rVert^2\]

We compute the terms inside the norm first, from right to left using matrix-vector multiplications and triangular solves. Finally we compute the norm.

The denominator is far less efficient to compute, since it requires working with m*n matrices. It can be expressed in terms of the same matrices as above:

\[\Big( \frac{1}{n} (\mathrm{Tr}(I) - \lVert \mathrm{LB}^{-1}A \rVert_F^2 ) \Big)^2\]

Hold Out Cross Validation

class falkon.hopt.objectives.HoldOut(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, val_pct: float, per_iter_split: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)

Leave One Out Cross Validation

class falkon.hopt.objectives.LOOCV(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)

LOOCV objective is to minimize the PRESS error:

\[\min \sum_{i=1}^n \big(e_{(i)}\big)_i^2\]

whose components are calculated as

\[\big(e_{(i)}\big)_i = (y_i - f(\alpha_{(i)})_i) = \dfrac{y_i - f(\alpha)_i}{1 - S_{ii}}\]

where

\[S = K_{nm} (\lambda K_{mm} + K_{nm}^\top K_{nm})^{-1} K_{nm}^\top\]

So we first need to compute the N-KRR solution \(\alpha\) and then the diagonal of matrix \(S\). For simplicity we use a direct solver the N-KRR problem, although Falkon could easily be substituted in.

SGPR

class falkon.hopt.objectives.SGPR(kernel: DiffKernel, centers_init: Tensor, penalty_init: Tensor, opt_centers: bool, opt_penalty: bool, centers_transform: Transform | None = None, pen_transform: Transform | None = None)