The BetaML.GMM Module

GMM module

Generative (Gaussian) Mixed Model learners (supervised/unsupervised)

Provides clustering and regressors using (Generative) Gaussiam Mixture Model (probabilistic).

Collaborative filtering / missing values imputation / reccomendation systems based on GMM is available in the Imputation module.

The module provides the following models. Use ?[model] to access their documentation:

• GaussianMixtureClusterer: soft-clustering using GMM
• GaussianMixtureRegressor2: regressor using GMM as back-end (first algorithm)
• GaussianMixtureRegressor2: regressor using GMM as back-end (second algorithm)

All the algorithms works with arbitrary mixture distribution, altought only {Spherical|Diagonal|Full} Gaussian mixtures has been implemented. User defined mixtures can be used defining a struct as subtype of AbstractMixture and implementing for that mixture the following functions:

• init_mixtures!(mixtures, X; minimum_variance, minimum_covariance, initialisation_strategy)
• lpdf(m,x,mask) (for the e-step)
• update_parameters!(mixtures, X, pₙₖ; minimum_variance, minimum_covariance) (the m-step)
• npar(mixtures::Array{T,1}) (for the BIC/AIC computation)

All the GMM-based algorithms works only with numerical data, but accepts also Missing one.

The GaussianMixtureClusterer algorithm reports the BIC and the AIC in its info(model), but some metrics of the clustered output are also available, for example the silhouette score.

DiagonalGaussian(
    μ::Union{Nothing, Array{T, 1}}
) -> DiagonalGaussian
DiagonalGaussian(
    μ::Union{Nothing, Array{T, 1}},
    σ²::Union{Nothing, Array{T, 1}}
) -> DiagonalGaussian

DiagonalGaussian(μ,σ²) - Gaussian mixture with mean μ and variances σ² (and fixed zero covariances)

FullGaussian(μ::Union{Nothing, Array{T, 1}}) -> FullGaussian
FullGaussian(
    μ::Union{Nothing, Array{T, 1}},
    σ²::Union{Nothing, Array{T, 2}}
) -> FullGaussian

FullGaussian(μ,σ²) - Gaussian mixture with mean μ and variance/covariance matrix σ²

mutable struct GaussianMixtureClusterer <: BetaMLUnsupervisedModel

Assign class probabilities to records (i.e. soft clustering) assuming a probabilistic generative model of observed data using mixtures.

For the parameters see ?GaussianMixture_hp and ?BML_options.

Notes:

• Data must be numerical
• Mixtures can be user defined: see the ?GMM module documentation for a discussion on provided vs custom mixtures.
• Online fitting (re-fitting with new data) is supported by setting the old learned mixtrures as the starting values
• The model is fitted using an Expectation-Minimisation (EM) algorithm that supports Missing data and is implemented in the log-domain for better numerical accuracy with many dimensions

Example:

julia> using BetaML

julia> X = [1.1 10.1; 0.9 9.8; 10.0 1.1; 12.1 0.8; 0.8 9.8];

julia> mod = GaussianMixtureClusterer(n_classes=2)
GaussianMixtureClusterer - A Generative Mixture Model (unfitted)

julia> prob_belong_classes = fit!(mod,X)
Iter. 1:        Var. of the post  2.15612140465882        Log-likelihood -29.06452054772657
5×2 Matrix{Float64}:
 1.0  0.0
 1.0  0.0
 0.0  1.0
 0.0  1.0
 1.0  0.0

julia> new_probs = fit!(mod,[11 0.9])
Iter. 1:        Var. of the post  1.0     Log-likelihood -1.3312256125240092
1×2 Matrix{Float64}:
 0.0  1.0

julia> info(mod)
Dict{String, Any} with 6 entries:
  "xndims"         => 2
  "error"          => [1.0, 0.0, 0.0]
  "AIC"            => 15.7843
  "fitted_records" => 6
  "lL"             => 1.10786
  "BIC"            => -2.21571

julia> parameters(mod)
BetaML.GMM.GMMCluster_lp (a BetaMLLearnableParametersSet struct)
- mixtures: DiagonalGaussian{Float64}[DiagonalGaussian{Float64}([0.9333333333333332, 9.9], [0.05, 0.05]), DiagonalGaussian{Float64}([11.05, 0.9500000000000001], [0.05, 0.05])]
- initial_probmixtures: [0.0, 1.0]

mutable struct GaussianMixtureRegressor <: BetaMLUnsupervisedModel

A multi-dimensional, missing data friendly non-linear regressor based on Generative (Gaussian) Mixture Model.

The training data is used to fit a probabilistic model with latent mixtures (Gaussian distributions with different covariances are already implemented) and then predictions of new data is obtained by fitting the new data to the mixtures.

For hyperparameters see GaussianMixture_hp and BML_options.

Thsi strategy (GaussianMixtureRegressor) works by training the EM algorithm on a combined (hcat) matrix of X and Y. At predict time, the new data is first fitted to the learned mixtures using the e-step part of the EM algorithm (and using missing values for the dimensions belonging to Y) to obtain the probabilistic assignment of each record to the various mixtures. Then these probabilities are multiplied to the mixture averages for the Y dimensions to obtain the predicted value(s) for each record.

Example:

julia> using BetaML

julia> X = [1.1 10.1; 0.9 9.8; 10.0 1.1; 12.1 0.8; 0.8 9.8];

julia> Y = X[:,1] .* 2 - X[:,2]
5-element Vector{Float64}:
 -7.8999999999999995
 -8.0
 18.9
 23.4
 -8.200000000000001

julia> mod = GaussianMixtureRegressor(n_classes=2)
GaussianMixtureRegressor - A regressor based on Generative Mixture Model (unfitted)

julia> ŷ = fit!(mod,X,Y)
Iter. 1:        Var. of the post  2.2191120060614065      Log-likelihood -47.70971887023561
5×1 Matrix{Float64}:
 -8.033333333333333
 -8.033333333333333
 21.15
 21.15
 -8.033333333333333

julia> new_probs = predict(mod,[11 0.9])
1×1 Matrix{Float64}:
 21.15

julia> info(mod)
Dict{String, Any} with 6 entries:
  "xndims"         => 3
  "error"          => [2.21911, 0.0260833, 3.19141e-39, 0.0]
  "AIC"            => 60.0684
  "fitted_records" => 5
  "lL"             => -17.0342
  "BIC"            => 54.9911

julia> parameters(mod)
BetaML.GMM.GMMCluster_lp (a BetaMLLearnableParametersSet struct)
- mixtures: DiagonalGaussian{Float64}[DiagonalGaussian{Float64}([0.9333333333333332, 9.9, -8.033333333333333], [1.1024999999999996, 0.05, 5.0625]), DiagonalGaussian{Float64}([11.05, 0.9500000000000001, 21.15], [1.1024999999999996, 0.05, 5.0625])]
- initial_probmixtures: [0.6, 0.4]

mutable struct GaussianMixtureRegressor2 <: BetaMLUnsupervisedModel

A multi-dimensional, missing data friendly non-linear regressor based on Generative (Gaussian) Mixture Model (strategy "1").

The training data is used to fit a probabilistic model with latent mixtures (Gaussian distributions with different covariances are already implemented) and then predictions of new data is obtained by fitting the new data to the mixtures.

For hyperparameters see GaussianMixture_hp and BML_options.

This strategy (GaussianMixtureRegressor2) works by fitting the EM algorithm on the feature matrix X. Once the data has been probabilistically assigned to the various classes, a mean value of fitting values Y is computed for each cluster (using the probabilities as weigths). At predict time, the new data is first fitted to the learned mixtures using the e-step part of the EM algorithm to obtain the probabilistic assignment of each record to the various mixtures. Then these probabilities are multiplied to the mixture averages for the Y dimensions learned at training time to obtain the predicted value(s) for each record.

Notes:

• Predicted values are always a matrix, even when a single variable is predicted (use dropdims(ŷ,dims=2) to get a single vector).

Example:

julia> using BetaML

julia> X = [1.1 10.1; 0.9 9.8; 10.0 1.1; 12.1 0.8; 0.8 9.8];

julia> Y = X[:,1] .* 2 - X[:,2]
5-element Vector{Float64}:
 -7.8999999999999995
 -8.0
 18.9
 23.4
 -8.200000000000001

julia> mod = GaussianMixtureRegressor2(n_classes=2)
GaussianMixtureRegressor2 - A regressor based on Generative Mixture Model (unfitted)

julia> ŷ = fit!(mod,X,Y)
Iter. 1:        Var. of the post  2.15612140465882        Log-likelihood -29.06452054772657
5×1 Matrix{Float64}:
 -8.033333333333333
 -8.033333333333333
 21.15
 21.15
 -8.033333333333333

julia> new_probs = predict(mod,[11 0.9])
1×1 Matrix{Float64}:
 21.15

julia> info(mod)
Dict{String, Any} with 6 entries:
  "xndims"         => 2
  "error"          => [2.15612, 0.118848, 4.19495e-7, 0.0, 0.0]
  "AIC"            => 32.7605
  "fitted_records" => 5
  "lL"             => -7.38023
  "BIC"            => 29.2454

mutable struct GaussianMixture_hp <: BetaMLHyperParametersSet

Hyperparameters for GMM clusters and other GMM-based algorithms

Parameters:

• n_classes: Number of mixtures (latent classes) to consider [def: 3]

• initial_probmixtures: Initial probabilities of the categorical distribution (n_classes x 1) [default: []]

• mixtures: An array (of length n_classes) of the mixtures to employ (see the ?GMM module). Each mixture object can be provided with or without its parameters (e.g. mean and variance for the gaussian ones). Fully qualified mixtures are useful only if the initialisation_strategy parameter is set to "gived". This parameter can also be given symply in term of a type. In this case it is automatically extended to a vector of n_classes mixtures of the specified type. Note that mixing of different mixture types is not currently supported and that currently implemented mixtures are SphericalGaussian, DiagonalGaussian and FullGaussian. [def: DiagonalGaussian]

• tol: Tolerance to stop the algorithm [default: 10^(-6)]

• minimum_variance: Minimum variance for the mixtures [default: 0.05]

• minimum_covariance: Minimum covariance for the mixtures with full covariance matrix [default: 0]. This should be set different than minimum_variance.

• initialisation_strategy: The computation method of the vector of the initial mixtures. One of the following:

  • "grid": using a grid approach
  • "given": using the mixture provided in the fully qualified mixtures parameter
  • "kmeans": use first kmeans (itself initialised with a "grid" strategy) to set the initial mixture centers [default]

  Note that currently "random" and "shuffle" initialisations are not supported in gmm-based algorithms.

• maximum_iterations: Maximum number of iterations [def: 5000]

• tunemethod: The method - and its parameters - to employ for hyperparameters autotuning. See SuccessiveHalvingSearch for the default method (suitable for the GMM-based regressors) To implement automatic hyperparameter tuning during the (first) fit! call simply set autotune=true and eventually change the default tunemethod options (including the parameter ranges, the resources to employ and the loss function to adopt).

SphericalGaussian(
    μ::Union{Nothing, Array{T, 1}}
) -> SphericalGaussian
SphericalGaussian(
    μ::Union{Nothing, Array{T, 1}},
    σ²::Union{Nothing, T} where T
) -> SphericalGaussian

SphericalGaussian(μ,σ²) - Spherical Gaussian mixture with mean μ and (single) variance σ²

init_mixtures!(mixtures::Array{T,1}, X; minimum_variance=0.25, minimum_covariance=0.0, initialisation_strategy="grid",rng=Random.GLOBAL_RNG)

The parameter initialisation_strategy can be grid, kmeans or given:

• grid: Uniformly cover the space observed by the data
• kmeans: Use the kmeans algorithm. If the data contains missing values, a first run of predictMissing is done under init=grid to impute the missing values just to allow the kmeans algorithm. Then the em algorithm is used with the output of kmean as init values.
• given: Leave the provided set of initial mixtures

lpdf(m::DiagonalGaussian,x,mask) - Log PDF of the mixture given the observation x

lpdf(m::FullGaussian,x,mask) - Log PDF of the mixture given the observation x

lpdf(m::SphericalGaussian,x,mask) - Log PDF of the mixture given the observation x