Running SparkMLib with native LAPACK and BLAS
When getting started with the Spark maschine learning library MLib you probably saw this warning in your console:
WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK
It gets thrown by “netlib-java”, a wrapper for the low level linear algebra libraries. Netlib-java provides an interface to those native libraries and a pure jvm-fallback.
Installing BLAS and LAPACK
To get rid of this warning i first installed BLAS and LAPACK through my system’s package-manager.
If you are on Ubuntu or Debian you can run:
sudo apt-get install libatlas3-base libopenblas-base
I’m on Archlinux so i ran
yaourt -S openblas-lapack
Unfortunately that does not solve the problem. The warning stays.
Link your java or scala code to the native library
The netlib-java README file is not very helpful here. So i started looking through the maven repository and found that only netlib-java-core is a dependency of spark-mllib. To link to the native libraries you need to netlib-native_system-… for your system. The easiest way to do that is adding the netlib-all package to your project dependencies.
libraryDependencies += "com.github.fommil.netlib" % "all" % "1.1.2" pomOnly()
This will add the netlib-native packages for all supported operating systems and architectures. Netlib will automatically go through all available versions and select the best one.
As a result if you now rebuild your project the warnings should disappear.
Adding netlib-native only for a specific os and arch
If you know your project will only be used on the specific os and architecture you might not want to add netlib-native for all the other ones.
For example to add only add natives for linux x86_64 add the following to your projects dependencies:
libraryDependencies += "net.sourceforge.f2j" % "arpack_combined_all" % "0.1"
libraryDependencies += "com.github.fommil.netlib" % "netlib-native_ref-linux-x86_64" % "1.1" classifier "natives"
libraryDependencies += "com.github.fommil.netlib" % "netlib-native_system-linux-x86_64" % "1.1" classifier "natives"
Where netlib-native_ref is a provided reference implementation. And netlib-native_system is the one we installed earlier.
I hope this works for you.