BetaML Benchmarks

This benchmark page allows us to quickly check for regressions across versions. As it is run and compiled using GitHub actions, and these may be powered by different computational resources, timing results are normalized using SystemBenchmark.

This page also provides a basic, far-from-exhaustive, comparison with other leading Julia libraries for the same model. Note that while we did try to match hyperparameters to the BetaML default ones, differences still occour and may in part explain the different model's performances and scores.

This file is intended just for benchmarking, not much as a tutorial, and it doesn't employ a full ML workflow, just the minimum preprocessing such that the algorithms work.

Benchmark setup

Threads.nthreads()

avg_factor_to_ref

1.7470931883779202

Regression

A simple regression over 500 points with y = x₁²-x₂+x₃²

bm_regression

6 rows × 8 columns

	name	time	memory	allocs	mre_train	std_train	mre_test	std_test
	String	Float64	Int64	Int64	Float64	Float64	Float64	Float64
1	DT	7.96821e6	8302672	55606	0.000723725	2.76557e-5	0.0052213	0.000643395
2	RF	6.28088e7	115798000	997719	0.00123145	3.35691e-5	0.00309481	0.000272821
3	NN	1.08781e9	1048887408	11227028	0.00117329	5.52884e-5	0.00119524	0.000245931
4	DT (DecisionTrees.jl)	1.04506e5	73888	505	0.00648375	0.000214495	0.00878351	0.00135441
5	RF (DecisionTrees.jl)	1.36738e6	2757968	14201	0.00269667	0.000108721	0.00385966	0.000828025
6	NN (Flux.jl)	1.2886e8	215321680	1178001	0.00334591	9.90508e-5	0.0033654	9.90508e-5

Classification

A dicotomic diagnostic breast cancer classification based on the Wisconsin Breast Cancer Database.

bm_classification

9 rows × 8 columns

	name	time	memory	allocs	acc_train	std_train	acc_test	std_test
	String	Float64	Int64	Int64	Float64	Float64	Float64	Float64
1	DT	3.92619e8	210752976	1300623	0.999414	0.00131826	0.912155	0.0247153
2	RF	6.50396e8	694666128	4120496	0.99961	0.00123286	0.961341	0.0215558
3	NN	2.5655e9	10846554912	15040744	0.999414	0.000943449	0.97193	0.0205928
4	Perc	7.94392e7	352294496	620910	0.990041	0.00578215	0.957895	0.0236824
5	KPerc	1.15248e9	1447524096	11995132	1.0	0.0	0.940351	0.0332871
6	Peg	1.65821e8	606676416	1378009	0.940832	0.0185063	0.938596	0.0312195
7	DT (DT.jl)	3.60508e6	56960	242	1.0	0.0	0.926284	0.0458556
8	RF (DT.jl)	4.88405e6	6057328	6707	1.0	0.0	0.961341	0.0258813
9	NN (Flux.jl)	2.57455e8	572928080	1408801	1.0	0.0	0.968421	0.0161217

Clustering

Cluster benchmarks using the iris dataset. Note that all algorithms use only the features, the label is used only to asses the cluster accuracy score.

bm_clustering

3 rows × 8 columns

	name	time	memory	allocs	acc_mean	acc_std	sil_mean	sil_std
	String	Float64	Int64	Int64	Float64	Float64	Float64	Float64
1	KMeans	523509.0	518128	4873	0.891111	0.0085889	0.552261	0.00939593
2	KMedoids	1.06986e7	15012368	134300	0.902222	0.0163038	0.541025	0.0140424
3	GMM	1.16944e7	20867520	199983	0.914815	0.0188853	0.543741	0.014114

Missing imputation

TODO :-)