The BetaML.Nn Module

BetaML.Nn module

Implement the functionality required to define an artificial Neural Network, train it with data, forecast data and assess its performance.

Common types of layers and optimisation algorithms are already provided, but you can define your own by subclassing respectively the AbstractLayer and OptimisationAlgorithm abstract types.

The module provides the following types or functions. Use ?[type or function] to access their full signature and detailed documentation:

Model definition:

• DenseLayer: Classical feed-forward layer with user-defined activation function
• DenseNoBiasLayer: Classical layer without the bias parameter
• VectorFunctionLayer: Layer whose activation function runs over the ensemble of its nodes rather than on each one individually. No learnable weights on input, optional learnable weights as parameters of the activation function.
• ScalarFunctionLayer: Layer whose activation function runs over each node individually, like a classic DenseLayer, but with no learnable weights on input and optional learnable weights as parameters of the activation function.
• ReplicatorLayer: Alias for a ScalarFunctionLayer with no learnable parameters and identity as activation function
• ReshaperLayer: Reshape the output of a layer (or the input data) to the shape needed for the next one
• PoolingLayer: In the middle between VectorFunctionLayer and ScalarFunctionLayer, it applies a function to the set of nodes defined in a sliding kernel. Weightless.
• ConvLayer: A generic N+1 (channels) dimensional convolutional layer
• GroupedLayer: To stack several layers into a single layer, e.g. for multi-branches networks
• NeuralNetworkEstimator: Build the chained network and define a cost function

Each layer can use a default activation function, one of the functions provided in the Utils module (relu, tanh, softmax,...) or one provided by you. BetaML will try to recognise if it is a "known" function for which it sets the exact derivatives, otherwise you can normally provide the layer with it. If the derivative of the activation function is not provided (either manually or automatically), AD will be used and training may be slower, although this difference tends to vanish with bigger datasets.

You can alternatively implement your own layer by defining a new type as subtype of the abstract type AbstractLayer. Each user-implemented layer must define the following methods:

• A suitable constructor
• forward(layer,x)
• backward(layer,x,next_gradient)
• get_params(layer)
• get_gradient(layer,x,next_gradient)
• set_params!(layer,w)
• size(layer)

The first three functions work on a single record. Training and prediction internally work on whole batches of records, calling the batched versions forward_batch(layer,X), backward_batch(layer,X,next_gradient) and get_gradient_batch(layer,X,next_gradient), whose default implementations simply loop over the records of the batch. Implement the batched versions instead of (or in addition to) the single-record ones to speed up your layer, as all the default layers do.

Running on GPUs

Neural networks can be trained and used on GPUs of any vendor (NVIDIA, AMD, Apple, Intel). Just load the package of your GPU vendor (e.g. using CUDA, using AMDGPU, using Metal or using oneAPI) and either:

• pass your data already on the GPU (e.g. fit!(m,cu(X),cu(Y))): with the default option device=:auto the network is trained where the data is, and the results are returned there;
• or keep your data on the CPU and use the option device=:gpu of NeuralNetworkEstimator: each batch is moved to the GPU for the computation, while the results are returned on the CPU.

On GPUs, unless the data is already there with a different floating point type, the computations are performed in Float32. To run your own layers on GPUs, their forward_batch, backward_batch and get_gradient_batch functions must use array-level operations (and not loops over the individual elements of the arrays) and BetaML must be able to rebuild the layer with its arrays moved to the GPU. This is automatic if the layer has a constructor taking all its fields in order (like the default one), otherwise define Adapt.adapt_structure(to,layer::MyLayer) (see the Adapt.jl package).

Most default layers run entirely on GPUs. The exceptions, that still work with data on GPUs but compute record by record on the CPU, are:

• VectorFunctionLayer with functions other than softmax, unless the function can work on a whole batch and the layer is created with batched=true;
• PoolingLayer with functions other than maximum and mean.

In particular, for pooling on GPUs use PoolingLayer with f=maximum or f=mean (e.g. PoolingLayer(nₗ,k,1,stride=1,padding=0,f=mean) is the equivalent of pool1d(x,k,f=mean)), or a VectorFunctionLayer with batched=true and a pooling function working on whole batches with array slices (see the documentation of VectorFunctionLayer).

Model fitting:

• fit!(nn,X,Y): fitting function
• fitting_info(nn): Default callback function during fitting
• SGD: The classical optimisation algorithm
• ADAM: A faster moment-based optimisation algorithm

To define your own optimisation algorithm define a subtype of OptimisationAlgorithm and implement the function single_update!(θ,▽;opt_alg) and optionally init_optalg!(⋅) specific for it.

Model predictions and assessment:

• predict(nn) or predict(nn,X): Return the output given the data

While high-level functions operating on the dataset expect it to be in the standard format (nrecords × ndimensions matrices) it is customary to represent the chain of a neural network as a flow of column vectors, so all low-level operations (operating on a single datapoint) expect both the input and the output as a column vector.

ADAM(;η, λ, β₁, β₂, ϵ)

The ADAM algorithm, an adaptive moment estimation optimiser.

Fields:

• η: Learning rate (stepsize, α in the paper), as a function of the current epoch [def: t -> 0.001 (i.e. fixed)]
• λ: Multiplicative constant to the learning rate [def: 1]
• β₁: Exponential decay rate for the first moment estimate [range: ∈ [0,1], def: 0.9]
• β₂: Exponential decay rate for the second moment estimate [range: ∈ [0,1], def: 0.999]
• ϵ: Epsilon value to avoid division by zero [def: 10^-8]

struct ConvLayer{ND, NDPLUS1, NDPLUS2, TF<:Function, TDF<:Union{Nothing, Function}, WET<:Number, WT<:AbstractArray{WET<:Number, NDPLUS2}, BT<:AbstractArray{WET<:Number, 1}} <: AbstractLayer

A generic N+1 (channels) dimensional convolutional layer

EXPERIMENTAL: Still too slow for practical applications

This convolutional layer has two constructors, one with the form ConvLayer(input_size,kernel_size,nchannels_in,nchannels_out), and an alternative one as ConvLayer(input_size_with_channel,kernel_size,nchannels_out). If the input is a vector, use a ReshaperLayer in front.

Fields:

• input_size::StaticArraysCore.SVector{NDPLUS1, Int64} where NDPLUS1: Input size (including nchannel_in as last dimension)

• output_size::StaticArraysCore.SVector{NDPLUS1, Int64} where NDPLUS1: Output size (including nchannel_out as last dimension)

• weight::AbstractArray{WET, NDPLUS2} where {NDPLUS2, WET<:Number}: Weight tensor (aka "filter" or "kernel") with respect to the input from previous layer or data (kernelsize array augmented by the nchannelsin and nchannels_out dimensions)

• usebias::Bool: Whether to use (and learn) a bias weight [def: true]

• bias::AbstractVector{WET} where WET<:Number: Bias (nchannels_out array)

• padding_start::StaticArraysCore.SVector{ND, Int64} where ND: Padding (initial)

• padding_end::StaticArraysCore.SVector{ND, Int64} where ND: Padding (ending)

• stride::StaticArraysCore.SVector{ND, Int64} where ND: Stride

• ndims::Int64: Number of dimensions (excluding input and output channels)

• f::Function: Activation function

• df::Union{Nothing, Function}: Derivative of the activation function

ConvLayer(
    input_size,
    kernel_size,
    nchannels_in,
    nchannels_out;
    stride,
    rng,
    padding,
    kernel_eltype,
    kernel_init,
    usebias,
    bias_init,
    f,
    df
) -> ConvLayer{_A, _B, _C, typeof(identity), _D, Float64, _E, _F} where {_A, _B, _C, _D<:Union{Nothing, Function}, _E<:AbstractArray{Float64, _C}, _F<:AbstractVector{Float64}}

Instantiate a new nD-dimensional, possibly multichannel ConvolutionalLayer

The input data is either a column vector (in which case it is reshaped) or an array of input_size augmented by the n_channels dimension, the output size depends on the input_size, kernel_size, padding and striding but always has nchannels_out as its last dimension.

Positional arguments:

• input_size: Shape of the input layer (integer for 1D convolution, tuple otherwise). Do not consider the channels number here.
• kernel_size: Size of the kernel (aka filter or learnable weights) (integer for 1D or hypercube kernels or nD-sized tuple for asymmetric kernels). Do not consider the channels number here.
• nchannels_in: Number of channels in input
• nchannels_out: Number of channels in output

Keyword arguments:

• stride: "Steps" to move the convolution with across the various tensor dimensions [def: ones]
• padding: Integer or 2-element tuple of tuples of the starting and ending padding across the various dimensions [def: nothing, i.e. set the padding required to keep the same dimensions in output (with stride==1)]
• f: Activation function [def: identity]
• df: Derivative of the activation function [default: try to match a known function, AD otherwise. Use nothing to force AD]
• kernel_eltype: Kernel eltype [def: Float64]
• kernel_init: Initial weights with respect to the input [default: Xavier initialisation]. If explicitly provided, it should be a multidimensional array of kernel_size augmented by nchannels_in and nchannels_out dimensions
• bias_init: Initial weights with respect to the bias [default: Xavier initialisation]. If given it should be a nchannels_out vector of scalars.
• rng: Random Number Generator (see FIXEDSEED) [default: Random.GLOBAL_RNG]

Notes:

• Xavier initialization is sampled from a Uniform distribution between ⨦ sqrt(6/(prod(input_size)*nchannels_in))
• to retrieve the output size of the layer, use size(ConvLayer[2]). The output size on each dimension d (except the last one that is given by nchannels_out) is given by the following formula (ceiled): output_size[d] = 1 + (input_size[d]+2*padding[d]-kernel_size[d])/stride[d]
• with strides higher than 1, the automatic padding is set to keep outsize = insize/stride

ConvLayer(
    input_size_with_channel,
    kernel_size,
    nchannels_out;
    stride,
    rng,
    padding,
    kernel_eltype,
    kernel_init,
    usebias,
    bias_init,
    f,
    df
) -> Union{ConvLayer{_A, _B, _C, typeof(identity), _D, _E, _F, _G} where {_E<:Number, _F<:AbstractArray{_E, _C}, _G<:AbstractVector{_E}}, ConvLayer{_A, _B, _C, typeof(identity), _D, WET, _F} where {WET<:Number, _F<:AbstractArray{WET, _C}}} where {_A, _B, _C, _D<:Union{Nothing, Function}}

Alternative constructor for a ConvLayer where the number of channels in input is specified as a further dimension in the input size instead of as a separate parameter, so as to use size(previous_layer)[2] if one wishes.

For arguments and default values see the documentation of the main constructor.

struct DenseLayer{TF<:Function, TDF<:Union{Nothing, Function}, WET<:Number, WT<:AbstractArray{WET<:Number, 2}, BT<:AbstractArray{WET<:Number, 1}} <: AbstractLayer

Representation of a layer in the network

Fields:

• w: Weights matrix with respect to the input from previous layer or data (n x n pr. layer)
• wb: Biases (n)
• f: Activation function
• df: Derivative of the activation function

DenseLayer(
    nₗ,
    n;
    rng,
    w_eltype,
    w,
    wb,
    f,
    df
) -> DenseLayer{typeof(identity), _A, Float64, _B, _C} where {_A<:Union{Nothing, Function}, _B<:AbstractMatrix{Float64}, _C<:AbstractVector{Float64}}

Instantiate a new DenseLayer

Positional arguments:

• nₗ: Number of nodes of the previous layer
• n: Number of nodes

Keyword arguments:

• w_eltype: Eltype of the weights [def: Float64]
• w: Initial weights with respect to input [default: Xavier initialisation, dims = (n,nₗ)]
• wb: Initial weights with respect to bias [default: Xavier initialisation, dims = (n)]
• f: Activation function [def: identity]
• df: Derivative of the activation function [default: try to match with well-known derivatives, resort to AD if f is unknown]
• rng: Random Number Generator (see FIXEDSEED) [default: Random.GLOBAL_RNG]

Notes:

• Xavier initialization = rand(Uniform(-sqrt(6)/sqrt(nₗ+n),sqrt(6)/sqrt(nₗ+n))
• Specify df=nothing to explicitly use AD

struct DenseNoBiasLayer{TF<:Function, TDF<:Union{Nothing, Function}, WET<:Number, WT<:AbstractArray{WET<:Number, 2}} <: AbstractLayer

Representation of a layer without bias in the network

Fields:

• w: Weights matrix with respect to the input from previous layer or data (n x n pr. layer)
• f: Activation function
• df: Derivative of the activation function

DenseNoBiasLayer(
    nₗ,
    n;
    rng,
    w_eltype,
    w,
    f,
    df
) -> DenseNoBiasLayer{typeof(identity), _A, Float64, _B} where {_A<:Union{Nothing, Function}, _B<:AbstractMatrix{Float64}}

Instantiate a new DenseNoBiasLayer

Positional arguments:

• nₗ: Number of nodes of the previous layer
• n: Number of nodes

Keyword arguments:

• w_eltype: Eltype of the weights [def: Float64]
• w: Initial weights with respect to input [default: Xavier initialisation, dims = (n,nₗ)]
• f: Activation function [def: identity]
• df: Derivative of the activation function [default: try to match with well-known derivatives, resort to AD if f is unknown]
• rng: Random Number Generator (see FIXEDSEED) [default: Random.GLOBAL_RNG]

Notes:

• Xavier initialization = rand(Uniform(-sqrt(6)/sqrt(nₗ+n),sqrt(6)/sqrt(nₗ+n))

struct GroupedLayer <: AbstractLayer

Representation of a "group" of layers, each of which operates on different inputs (features), acting as a single layer in the network.

Fields:

• layers: The individual layers that compose this grouped layer

GroupedLayer(layers) -> GroupedLayer

Instantiate a new GroupedLayer, a layer made up of several other layers stacked together in order to cover all the data dimensions but without connecting all the inputs to all the outputs like a single DenseLayer would do.

Positional arguments:

• layers: The individual layers that compose this grouped layer

Notes:

• can be used to create composable neural networks with multiple branches
• tested only with 1 dimensional layers. For convolutional networks use ReshaperLayers before and/or after.

Learnable(data)

Structure representing the learnable parameters of a layer or its gradient.

The learnable parameters of a layer are given in the form of a N-tuple of arrays of any dimension (e.g. we can have a layer with the first parameter being a matrix, and the second one being a vector). The arrays can be stored on any device (CPU or GPU). We wrap the tuple in its own structure a bit for some efficiency gain, but above all to define standard mathematical operations on the gradients without doing "type piracy" with respect to Base tuples.

Operations with a scalar convert the scalar to the element type of the arrays, so that e.g. Float32 parameters stay Float32.

**`

mutable struct NeuralNetworkE_hp <: BetaMLHyperParametersSet

`**

Hyperparameters for the Feedforward neural network model

Parameters:

• layers: Array of layer objects [def: nothing, i.e. basic network]. See subtypes(BetaML.AbstractLayer) for supported layers

• loss: Loss (cost) function [def: squared_cost] It must always assume y and ŷ as (n x d) matrices, possibly using dropdims inside, and return the loss summed over the n records. Write it with array-level operations (e.g. broadcasting) for it to work also on GPUs.

• dloss: Derivative of the loss function [def: dsquared_cost if loss==squared_cost, nothing otherwise, i.e. use the derivative of the squared cost or autodiff]

• epochs: Number of epochs, i.e. passages through the whole training sample [def: 200]

• batch_size: Size of each individual batch [def: 16]

• opt_alg: The optimisation algorithm to update the gradient at each batch [def: ADAM()]

• shuffle: Whether to randomly shuffle the data at each iteration (epoch) [def: true]

• tunemethod: The method - and its parameters - to employ for hyperparameters autotuning. See SuccessiveHalvingSearch for the default method. To implement automatic hyperparameter tuning during the (first) fit! call simply set autotune=true and optionally change the default tunemethod options (including the parameter ranges, the resources to employ and the loss function to adopt).

To know the available layers, type subtypes(AbstractLayer) and then type ?LayerName for information on how to use each layer.

NeuralNetworkE_options

A struct defining the options used by the Feedforward neural network model

Parameters:

• cache: Cache the results of the fitting stage, so as to allow predict(mod) [default: true]. Set it to false to save memory for large data.

• descr: An optional title and/or description for this model

• verbosity: The verbosity level to be used in training or prediction (see Verbosity) [default: STD]

• cb: A callback function to provide information during training [def: fitting_info]

• autotune: Option for hyper-parameters autotuning [def: false, i.e. no autotuning performed]. If activated, autotuning is performed on the first fit!() call. Control auto-tuning through the option tunemethod (see the model hyper-parameters)

• onfail: The strategy to apply in case of failure during the fitting (loss not decreasing) [def: "tryagain"]. The default is to try again with a new random initialization of the weights and the batch ("tryagain"). Other valid values are "continue", "stop" and "error". The first ignores the issue, the second one stops the training (but without generating an error) and the last one stops the fitting and generates an error.

• fail_epoch_ref: The epoch to use as initial comparison to check for the onfail strategy [def: 2]

• fail_epoch: The epoch at which to try the onfail strategy [def: 5]

• fail_threshold: The (relative) threshold to use for the onfail strategy [def: 0.01]. If the loss doesn't decrease by at least this value, the onfail strategy is applied. Note that this is computed in relation to the loss on the second epoch, so it is not an absolute value.

• fail_attempts: The maximum number of attempts to use for the tryagain strategy [def: 10]

• rng: Random Number Generator (see FIXEDSEED) [default: Random.GLOBAL_RNG]

• device: The device where to train the network and compute the predictions [def: :auto]. Options are:

  • :auto: use the device where the training data is (e.g. the GPU if X is a GPU array);
  • :cpu: use the CPU;
  • :gpu: use the GPU of the backend package loaded (e.g. using CUDA, using AMDGPU, using Metal or using oneAPI), moving each batch of data there if it is on the CPU;
  • a device object of the MLDataDevices.jl package (e.g. CUDADevice()), to choose a specific GPU backend.

  Predictions are always returned on the device of the input data. On GPUs the network parameters are converted to Float32 (unless the data is already on the GPU with a different floating point type). Models saved with model_save while on a GPU have their parameters saved on the CPU. With device=:auto, they are moved back to the same kind of GPU when used for predictions, if its package is loaded (otherwise they stay on the CPU).

NeuralNetworkEstimator

A "feedforward" (but also multi-branch) neural network (supervised).

For the parameters see NeuralNetworkE_hp and for the training options NeuralNetworkE_options (we have a few more options for this specific estimator, in particular the device where to run the computation, e.g. :cpu or :gpu).

Notes:

• data must be numerical
• the label can be a n-records vector or a n-records by n-dimensions matrix, but the result is always a matrix.
  • For one-dimension regressions drop the unnecessary dimension with dropdims(ŷ,dims=2)
  • For classification tasks the columns should normally be interpreted as the probabilities for each category

Examples:

• Classification...

julia> using BetaML

julia> X = [1.8 2.5; 0.5 20.5; 0.6 18; 0.7 22.8; 0.4 31; 1.7 3.7];

julia> y = ["a","b","b","b","b","a"];

julia> ohmod = OneHotEncoder()
A OneHotEncoder BetaMLModel (unfitted)

julia> y_oh  = fit!(ohmod,y)
6×2 Matrix{Bool}:
 1  0
 0  1
 0  1
 0  1
 0  1
 1  0

julia> layers = [DenseLayer(2,6),DenseLayer(6,2),VectorFunctionLayer(2,f=softmax)];

julia> m      = NeuralNetworkEstimator(layers=layers,opt_alg=ADAM(),epochs=300,verbosity=LOW)
NeuralNetworkEstimator - A Feed-forward neural network (unfitted)

julia> ŷ_prob = fit!(m,X,y_oh)
***
*** Training  for 300 epochs with algorithm ADAM.
Training..       avg ϵ on (Epoch 1 Batch 1):     0.4116936481380642
Training of 300 epochs completed. Final epoch error: 0.44308719831108734.
6×2 Matrix{Float64}:
 0.853198    0.146802
 0.0513715   0.948629
 0.0894273   0.910573
 0.0367079   0.963292
 0.00548038  0.99452
 0.808334    0.191666

julia> ŷ      = inverse_predict(ohmod,ŷ_prob)
6-element Vector{String}:
 "a"
 "b"
 "b"
 "b"
 "b"
 "a"

• Regression...

julia> using BetaML

julia> X = [1.8 2.5; 0.5 20.5; 0.6 18; 0.7 22.8; 0.4 31; 1.7 3.7];

julia> y = 2 .* X[:,1] .- X[:,2] .+ 3;

julia> layers = [DenseLayer(2,6),DenseLayer(6,6),DenseLayer(6,1)];

julia> m      = NeuralNetworkEstimator(layers=layers,opt_alg=ADAM(),epochs=3000,verbosity=LOW)
NeuralNetworkEstimator - A Feed-forward neural network (unfitted)

julia> ŷ      = fit!(m,X,y);
***
*** Training  for 3000 epochs with algorithm ADAM.
Training..       avg ϵ on (Epoch 1 Batch 1):     33.30063874270561
Training of 3000 epochs completed. Final epoch error: 34.61265465430473.

julia> hcat(y,ŷ)
6×2 Matrix{Float64}:
   4.1    4.11015
 -16.5  -16.5329
 -13.8  -13.8381
 -18.4  -18.3876
 -27.2  -27.1667
   2.7    2.70542

struct PoolingLayer{ND, NDPLUS1, NDPLUS2, TF<:Function, TDF<:Union{Nothing, Function}, WET<:Number} <: AbstractLayer

Representation of a pooling layer in the network (weightless)

EXPERIMENTAL: Still too slow for practical applications

In the middle between VectorFunctionLayer and ScalarFunctionLayer, it applies a function to the set of nodes defined in a sliding kernel.

Fields:

• input_size::StaticArraysCore.SVector{NDPLUS1, Int64} where NDPLUS1: Input size (including nchannel_in as last dimension)

• output_size::StaticArraysCore.SVector{NDPLUS1, Int64} where NDPLUS1: Output size (including nchannel_out as last dimension)

• kernel_size::StaticArraysCore.SVector{NDPLUS2, Int64} where NDPLUS2: kernelsize augmented by the nchannelsin and nchannels_out dimensions

• padding_start::StaticArraysCore.SVector{ND, Int64} where ND: Padding (initial)

• padding_end::StaticArraysCore.SVector{ND, Int64} where ND: Padding (ending)

• stride::StaticArraysCore.SVector{ND, Int64} where ND: Stride

• ndims::Int64: Number of dimensions (excluding input and output channels)

• f::Function: Activation function

• df::Union{Nothing, Function}: Derivative of the activation function

PoolingLayer(
    input_size,
    kernel_size,
    nchannels_in;
    stride,
    kernel_eltype,
    padding,
    f,
    df
) -> PoolingLayer{_A, _B, _C, typeof(maximum), _D, Float64} where {_A, _B, _C, _D<:Union{Nothing, Function}}

Instantiate a new nD-dimensional, possibly multichannel PoolingLayer

The input data is either a column vector (in which case it is reshaped) or an array of input_size augmented by the n_channels dimension, the output size depends on the input_size, kernel_size, padding and striding but always has nchannels_out as its last dimension.

Positional arguments:

• input_size: Shape of the input layer (integer for 1D convolution, tuple otherwise). Do not consider the channels number here.
• kernel_size: Size of the kernel (aka filter) (integer for 1D or hypercube kernels or nD-sized tuple for asymmetric kernels). Do not consider the channels number here.
• nchannels_in: Number of channels in input

Keyword arguments:

• kernel_eltype: Kernel eltype [def: Float64]
• stride: "Steps" to move the convolution with across the various tensor dimensions [def: kernel_size, i.e. each X contributes to a single y]
• padding: Integer or 2-element tuple of tuples of the starting and ending padding across the various dimensions [def: nothing, i.e. set the padding required to keep outsize = insize / stride ]
• f: Activation function. It should have a vector as input and produce a scalar as output [def: maximum]
• df: Derivative (gradient) of the activation function for the various inputs. [default: nothing (i.e. use AD)]
• kernel_eltype: Eltype of the output when the input is not made of floating point numbers [def: Float64]

Notes:

• to retrieve the output size of the layer, use size(PoolLayer[2]). The output size on each dimension d (except the last one that is given by nchannels_out) is given by the following formula (ceiled): output_size[d] = 1 + (input_size[d]+2*padding[d]-kernel_size[d])/stride[d]
• on GPUs only f=maximum and f=mean are supported (their derivatives are computed directly within the GPU kernel), while other functions work only on the CPU. For other pooling functions on GPUs, use a VectorFunctionLayer with batched=true (see its documentation)
• a 1-dimensional pooling of nₗ nodes over windows of k nodes moving one node at a time, as done by pool1d, is obtained with PoolingLayer(nₗ,k,1,stride=1,padding=0,f=f). The output has a further singleton "channel" dimension (size(layer) == ((nₗ,1),(nₗ-k+1,1))), that the following dense layers simply ignore
• differently from a ConvLayer, the pooling always applies on a single channel level, so that the output always has the same number of channels as the input. If you want to reduce the channels number either use a ConvLayer with the desired number of channels in output or use a ReshaperLayer to add a 1-element further dimension that will be treated as "channel" and choose the desired stride for the last pooling dimension (the one that was originally the channel dimension)

PoolingLayer(
    input_size_with_channel,
    kernel_size;
    stride,
    padding,
    f,
    kernel_eltype,
    df
) -> PoolingLayer{_A, _B, _C, typeof(maximum)} where {_A, _B, _C}

Alternative constructor for a PoolingLayer where the number of channels in input is specified as a further dimension in the input size instead of as a separate parameter, so as to use size(previous_layer)[2] if one wishes.

For arguments and default values see the documentation of the main constructor.

ReshaperLayer(input_size) -> ReshaperLayer
ReshaperLayer(input_size, output_size) -> ReshaperLayer

Instantiate a new ReshaperLayer

Positional arguments:

• input_size: Shape of the input layer (tuple).
• output_size: Shape of the output layer (tuple) [def: prod([input_size...]), i.e. reshape to a vector of appropriate length].

struct ReshaperLayer{NDIN, NDOUT} <: AbstractLayer

Representation of a "reshaper" (weightless) layer in the network

Reshape the output of a layer (or the input data) to the shape needed for the next one.

Fields:

• input_size::StaticArraysCore.SVector{NDIN, Int64} where NDIN: Input size

• output_size::StaticArraysCore.SVector{NDOUT, Int64} where NDOUT: Output size

SGD(;η=t -> 1/(1+t), λ=2)

Stochastic Gradient Descent algorithm

Fields:

• η: Learning rate, as a function of the current epoch [def: t -> 1/(1+t)]
• λ: Multiplicative constant to the learning rate [def: 2]

struct ScalarFunctionLayer{N, TF<:Function, TDFX<:Union{Nothing, Function}, TDFW<:Union{Nothing, Function}, WET<:Number, WT<:AbstractArray{WET<:Number, N}} <: AbstractLayer

Representation of a ScalarFunction layer in the network. ScalarFunctionLayer applies the activation function directly to the output of the previous layer (i.e., without passing through a weight matrix), node by node, optionally using a learnable weight as second argument: yᵢ = f(xᵢ) or yᵢ = f(xᵢ,wᵢ). The learnable weights can be either one per node, or a single one shared by all nodes. Differently from VectorFunctionLayer, the function is applied scalarwise to each node.

The number of nodes in input must be set to the same as in the previous layer

Fields:

• w: Weights (parameter) vector passed as second argument to the activation function (if not empty), either with one element per node or a single element
• n: Number of nodes in output (≡ number of nodes in input )
• f: Activation function (scalar)
• dfx: Derivative of the activation function with respect to the layer input of the node (x)
• dfw: Derivative of the activation function with respect to the learnable weight of the node (w)

Notes:

• The output size of this layer is the same as that of the previous layer.

ScalarFunctionLayer(
    nₗ;
    rng,
    wsize,
    w_eltype,
    w,
    f,
    dfx,
    dfw
) -> ScalarFunctionLayer{_A, typeof(softmax), _B, Nothing, Float64, _C} where {_A, _B<:Union{Nothing, Function}, _C<:AbstractArray{Float64, _A}}

Instantiate a new ScalarFunctionLayer

Positional arguments:

• nₗ: Number of nodes (must be the same as in the previous layer)

Keyword arguments:

• wsize: The size of the learnable weights, either [nₗ] (one weight per node) or [1] (a single weight shared by all nodes) [def: empty array, i.e. no learnable weights]
• w_eltype: Eltype of the weights [def: Float64]
• w: Initial weights [default: Xavier initialisation, dims = wsize]
• f: Activation function. Without learnable weights it is called as f(xᵢ), with learnable weights as f(xᵢ,wᵢ) [def: softmax]
• dfx: Derivative of the activation function with respect to the data: dfx(xᵢ) or dfx(xᵢ,wᵢ) [default: try to match with well-known derivatives, resort to AD if f is unknown]
• dfw: Derivative of the activation function with respect to the learnable weight: dfw(xᵢ,wᵢ) [default: nothing (i.e. use AD)]
• rng: Random Number Generator (see FIXEDSEED) [default: Random.GLOBAL_RNG]

Notes:

• The derivatives, if provided, are scalar functions (of the scalar input and, if any, of the scalar weight of the node)
• With a single shared weight, wᵢ is the same for all nodes
• Xavier initialization = rand(Uniform(-sqrt(6)/sqrt(sum(wsize...)),sqrt(6)/sqrt(sum(wsize...))))

struct VectorFunctionLayer{N, TF<:Function, TDFX<:Union{Nothing, Function}, TDFW<:Union{Nothing, Function}, WET<:Number, WT<:AbstractArray{WET<:Number, N}} <: AbstractLayer

Representation of a VectorFunction layer in the network. Vector function layer expects a vector activation function, i.e. a function taking the whole output of the previous layer as input rather than working on a single node as "normal" activation functions would do. Useful for example with the SoftMax function in classification or with the pool1D function to implement a "pool" layer in 1 dimension. By default it is weightless, i.e. it doesn't apply any transformation to the output coming from the previous layer except the activation function. However, by passing the parameter wsize (a tuple or array - tested only 1D) you can pass the learnable parameter to the activation function too. It is your responsibility to be sure the activation function accepts only X or also this learnable array (as second argument). The number of nodes in input must be set to the same as in the previous layer (and if you are using this for classification, to the number of classes, i.e. the previous layer must be set equal to the number of classes in the predictions).

Fields:

• w: Weights (parameter) array passed as second argument to the activation function (if not empty)
• nₗ: Number of nodes in input (i.e. length of previous layer)
• n: Number of nodes in output (automatically inferred in the constructor)
• f: Activation function (vector)
• dfx: Derivative of the (vector) activation function with respect to the layer inputs (x)
• dfw: Derivative of the (vector) activation function with respect to the optional learnable weights (w)

Notes:

• The output size of this layer is given by the size of the output function,

which is not necessarily the same as that of the previous layer.

VectorFunctionLayer(
    nₗ;
    rng,
    wsize,
    w_eltype,
    w,
    f,
    dfx,
    dfw,
    batched,
    dummyDataToTestOutputSize
) -> VectorFunctionLayer{_A, typeof(softmax), _B, Nothing, Float64, _C} where {_A, _B<:Union{Nothing, Function}, _C<:AbstractArray{Float64, _A}}

Instantiate a new VectorFunctionLayer

Positional arguments:

• nₗ: Number of nodes (must be the same as in the previous layer)

Keyword arguments:

• wsize: A tuple or array specifying the size (number of elements) of the learnable parameter [def: empty array]
• w_eltype: Eltype of the weights [def: Float64]
• w: Initial weights with respect to input [default: Xavier initialisation, dims = (nₗ,n)]
• f: Activation function [def: softmax]
• dfx: Derivative of the activation function with respect to the data

[default: try to match with well-known derivatives, resort to AD if f is unknown]

• dfw: Derivative of the activation function with respect to the learnable parameter [default: nothing (i.e. use AD)]
• batched: Whether f can be applied directly to a whole batch of records, i.e. to a (nₗ × n records) matrix, returning the (n × n records) matrix of the outputs of each record (e.g. f = X -> X ./ sum(X,dims=1)) [def: false, i.e. f is applied record by record]
• dummyDataToTestOutputSize: Dummy data to test the output size [def:

ones(nₗ)]

• rng: Random Number Generator (see FIXEDSEED) [default: Random.GLOBAL_RNG]

Notes:

• If the derivative is provided, it should return the gradient as a (n,n) matrix (i.e. the Jacobian) and it is computed record by record. Otherwise the derivatives are computed by AD (with Zygote), on the whole batch if batched=true
• With batched=true (and no derivative provided) the layer is computed with array-level operations on the whole batch, so it is much faster and it can run on GPUs (if f is written with array-level operations, e.g. broadcasting, sums and array slices, and not with loops over the individual elements). Otherwise the computation is performed record by record on the CPU (except for softmax, that is always computed on the whole batch)
• Pooling functions with a sliding window can be written for whole batches using array slices, e.g. a root-mean-square pooling over windows of k nodes: VectorFunctionLayer(nₗ,f=X -> sqrt.(sum(X[i:end-k+i,:].^2 for i in 1:k) ./ k),batched=true). For maximum and mean pooling use instead PoolingLayer
• To avoid recomputing the activation function just to determine its output size, we compute the output size once here in the layer constructor by calling the activation function with dummyDataToTestOutputSize. Feel free to change it if it doesn't match with the activation function you are setting
• Xavier initialization = rand(Uniform(-sqrt(6)/sqrt(sum(wsize...)),sqrt(6)/sqrt(sum(wsize...))))

size(layer)

Get the size of the layer in terms of (size in input, size in output) - both as tuples

Notes:

• You need to use import Base.size before defining this function for your layer

size(
    layer::ConvLayer
) -> Tuple{Tuple{Vararg{Int64}}, Tuple{Vararg{Int64}}}

Get the dimensions of the layer in terms of (dimensions in input, dimensions in output) including channels as last dimension

size(
    layer::PoolingLayer{ND, NDPLUS1, NDPLUS2}
) -> Tuple{Tuple{Vararg{Int64}}, Tuple{Vararg{Int64}}}

Get the dimensions of the layer in terms of (dimensions in input, dimensions in output) including channels as last dimension

ReplicatorLayer(
    n
) -> ScalarFunctionLayer{_A, typeof(identity), _B, Nothing, Float64, _C} where {_A, _B<:Union{Nothing, Function}, _C<:AbstractArray{Float64, _A}}

Create a weightless layer whose output is equal to the input.

Fields:

• n: Number of nodes in output (≡ number of nodes in input )

Notes:

• The output size of this layer is the same as that of the previous layer.
• This is just an alias for a ScalarFunctionLayer with no weights and identity function.

backward(layer,x,next_gradient)

Compute backpropagation for this layer with respect to its inputs

Parameters:

• layer: Worker layer
• x: Input to the layer
• next_gradient: Derivative of the overall loss with respect to the input of the next layer (output of this layer)

Return:

• The evaluated gradient of the loss with respect to this layer's inputs

Notes:

• For layers implementing backward_batch this is computed by calling it on a batch with a single record

backward_batch(layer,X,next_gradient)

Compute backpropagation for this layer with respect to its inputs for a whole batch of records

Parameters:

• layer: Worker layer
• X: Input to the layer, with the records along the last dimension
• next_gradient: Derivative of the overall loss with respect to the input of the next layer (output of this layer), with the records along the last dimension

Return:

• The evaluated gradient of the loss with respect to this layer's inputs, with the records along the last dimension

Notes:

• The default implementation calls backward(layer,x,next_gradient) for each record. Implement this function directly with array-level operations to speed up your layer.

fittinginfo(nn,xbatch,ybatch,x,y;n,nbatches,epochs,epochsran,verbosity,nepoch,n_batch)

Default callback function to display information during training, depending on the verbosity level

Parameters:

• nn: Worker network
• xbatch: Batch input to the network (batch_size,din)
• ybatch: Batch label input (batch_size,dout)
• x: Full input to the network (n_records,din)
• y: Full label input (n_records,dout)
• n: Size of the full training set
• n_batches : Number of batches per epoch
• epochs: Number of epochs defined for the training
• epochs_ran: Number of epochs already run in previous training sessions
• verbosity: Verbosity level defined for the training (NONE,LOW,STD,HIGH,FULL)
• n_epoch: Counter of the current epoch
• n_batch: Counter of the current batch

#Notes:

• Reporting of the error (loss of the network) is expensive. Use verbosity=NONE for better performance

forward(layer,x)

Predict the output of the layer given the input

Parameters:

• layer: Worker layer
• x: Input to the layer

Return:

• An Array{T,1} of the prediction (even for a scalar)

Notes:

• For layers implementing forward_batch this is computed by calling it on a batch with a single record

forward_batch(layer,X)

Predict the output of the layer for a whole batch of records

Parameters:

• layer: Worker layer
• X: Input to the layer, with the records along the last dimension (i.e. an array whose size is the input size of the layer augmented by the number of records)

Return:

• The output of the layer, again with the records along the last dimension

Notes:

• The default implementation calls forward(layer,x) for each record. Implement this function directly with array-level operations to speed up your layer.

forward_batch(layer::ConvLayer, X) -> Any

Compute forward pass of a ConvLayer for a batch of records (on the last dimension)

forward_batch(layer::PoolingLayer{ND}, X) -> AbstractArray

Compute forward pass of a PoolingLayer for a batch of records (on the last dimension)

get_gradient(layer,x,next_gradient)

Compute backpropagation for this layer with respect to the layer weights

Parameters:

• layer: Worker layer
• x: Input to the layer
• next_gradient: Derivative of the overall loss with respect to the input of the next layer (output of this layer)

Return:

• The evaluated gradient of the loss with respect to this layer's trainable parameters as a tuple of matrices. It is up to you to decide how to organise this tuple, as long as you are consistent with the get_params() and set_params() functions. Note that starting from BetaML 0.2.2 this tuple needs to be wrapped in its Learnable type.

Notes:

• For layers implementing get_gradient_batch this is computed by calling it on a batch with a single record

get_gradient(nn,x,y)

Low level function that retrieves the current gradient of the weights (i.e. derivative of the cost with respect to the weights) for a single record. Unexported in BetaML >= v0.9

Parameters:

• nn: Worker network
• x: Input to the network (d,1)
• y: Label input (d,1)

#Notes:

• The output is a vector of tuples of each layer's input weights and bias weights

get_gradient_batch(layer,X,next_gradient)

Compute backpropagation for this layer with respect to the layer weights for a whole batch of records

Parameters:

• layer: Worker layer
• X: Input to the layer, with the records along the last dimension
• next_gradient: Derivative of the overall loss with respect to the input of the next layer (output of this layer), with the records along the last dimension

Return:

• The sum over the records of the batch of the gradients of the loss with respect to this layer's trainable parameters, wrapped in a Learnable object

Notes:

• The default implementation sums get_gradient(layer,x,next_gradient) over the records. Implement this function directly with array-level operations to speed up your layer.

get_params(layer)

Get the current value of the layer's trainable parameters

Parameters:

• layer: Worker layer

Return:

• The current value of the layer's trainable parameters as a tuple of matrices. It is up to you to decide how to organise this tuple, as long as you are consistent with the get_gradient() and set_params() functions. Note that starting from BetaML 0.2.2 this tuple needs to be wrapped in its Learnable type.

get_params(nn)

Retrieve current weights

Parameters:

• nn: Worker network

Notes:

• The output is a vector of tuples of each layer's input weights and bias weights

init_optalg!(opt_alg::ADAM;θ,batch_size,x,y,rng)

Initialize the ADAM algorithm with the parameters m and v as zeros and check parameter bounds

initoptalg!(optalg;θ,batch_size,x,y)

Initialize the optimisation algorithm

Parameters:

• opt_alg: The Optimisation algorithm to use
• θ: Current parameters
• batch_size: The size of the batch
• x: The training (input) data
• y: The training "labels" to match
• rng: Random Number Generator (see FIXEDSEED) [default: Random.GLOBAL_RNG]

Notes:

• Only a few optimizers need this function and consequently override it. By default it does nothing, so if you want to write your own optimizer and don't need to initialise it, you don't have to override this method

preprocess!(layer::AbstractLayer)

Preprocess the layer with information known at layer creation (i.e. no data info used)

This function is used for some layers to cache some computation that doesn't require the data and it is called at the beginning of fit!.

Notes:

• as it doesn't depend on data, it is not reset by reset!

set_params!(layer,w)

Set the trainable parameters of the layer with the given values

Parameters:

• layer: Worker layer
• w: The new parameters to set (Learnable)

Notes:

• The format of the tuple wrapped by Learnable must be consistent with those of the get_params() and get_gradient() functions.

set_params!(nn,w)

Update weights of the network

Parameters:

• nn: Worker network
• w: The new weights to set

singleupdate!(θ,▽;nepoch,nbatch,nbatches,xbatch,ybatch,opt_alg)

Perform the parameters update based on the average batch gradient.

Parameters:

• θ: Current parameters
• ▽: Average gradient of the batch
• n_epoch: Count of current epoch
• n_batch: Count of current batch
• n_batches: Number of batches per epoch
• xbatch: Data associated to the current batch
• ybatch: Labels associated to the current batch
• opt_alg: The Optimisation algorithm to use for the update

Notes:

• This function is overridden so that each optimisation algorithm implements its

own version

• Most parameters are not used by any optimisation algorithm. They are provided

to support the largest possible class of optimisation algorithms

• Some optimisation algorithms may change their internal structure in this function