Running SparkMLib with native LAPACK and BLAS

When getting started with the Spark maschine learning library MLib you probably saw this warning in your console:

WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK

It gets thrown by “netlib-java”, a wrapper for the low level linear algebra libraries. Netlib-java provides an interface to those native libraries and a pure jvm-fallback.

Installing BLAS and LAPACK

To get rid of this warning i first installed BLAS and LAPACK through my system’s package-manager.
If you are on Ubuntu or Debian you can run:

sudo apt-get install libatlas3-base libopenblas-base

I’m on Archlinux so i ran

yaourt -S openblas-lapack

Unfortunately that does not solve the problem. The warning stays.

Link your java or scala code to the native library

The netlib-java README file is not very helpful here. So i started looking through the maven repository and found that only netlib-java-core is a dependency of spark-mllib. To link to the native libraries you need to netlib-native_system-… for your system. The easiest way to do that is adding the netlib-all package to your project dependencies.

Maven:

com.github.fommil.netlib
all
1.1.2
pom

SBT:

// https://mvnrepository.com/artifact/com.github.fommil.netlib/all
libraryDependencies += "com.github.fommil.netlib" % "all" % "1.1.2" pomOnly()

This will add the netlib-native packages for all supported operating systems and architectures. Netlib will automatically go through all available versions and select the best one.
As a result if you now rebuild your project the warnings should disappear.

Adding netlib-native only for a specific os and arch

If you know your project will only be used on the specific os and architecture you might not want to add netlib-native for all the other ones.
For example to add only add natives for linux x86_64 add the following to your projects dependencies:

Maven:

net.sourceforge.f2j
arpack_combined_all
0.1
com.github.fommil.netlib
netlib-native_ref-linux-x86_64
1.1
natives
com.github.fommil.netlib
netlib-native_system-linux-x86_64
1.1
natives

SBT:

libraryDependencies += "net.sourceforge.f2j" % "arpack_combined_all" % "0.1"<br> libraryDependencies += "com.github.fommil.netlib" % "netlib-native_ref-linux-x86_64" % "1.1" classifier "natives"<br> libraryDependencies += "com.github.fommil.netlib" % "netlib-native_system-linux-x86_64" % "1.1" classifier "natives"

Where netlib-native_ref is a provided reference implementation. And netlib-native_system is the one we installed earlier.
I hope this works for you.