Окружающая среда
**Платформа ОС, дистрибутив и версия: Redhat 7.5 ALT ppc64le Версия Python (дополнительно): python 3.6 от DriverlessAI Версия CUDA/cuDNN: CUDA 9.2 cuDNN 7.1 Модель графического процессора (дополнительно): Модель ЦП V100: POWER9 Доступно ОЗУ: 512 ГБ Версия R: 3.4.1 Версия Tensorflow: 1.8.0 (собрана из исходного кода)
Я пытаюсь использовать h2o.deepwater, включенный в DriverlessAI, в среде python и R вместо веб-интерфейса DAI. Кроме того, я хотел бы использовать tensorflow в качестве бэкэнда.
Для этого я установил переменные среды для использования python из DriverlessAI.
$ export PATH=/opt/h2oai/dai/python/bin:$PATH
$ export LD_LIBRARY_PATH=/opt/h2oai/dai/python/lib:/opt/h2oai/dai/lib:$LD_LIBRARY_PATH
$ export PYTHONPATH=/opt/h2oai/dai/cuda-9.2/lib/python3.6/site-packages
Это отлично работает с h2o.deeplearning.
gpu_xgb <- h2o.deeplearning(x = c("TemperatureCelcius","ExhaustVacuumHg","AmbientPressureMillibar","RelativeHumidity"),
y = "HourlyEnergyOutputMW",
training_frame = train
)
Однако h2o.deepwater выдает ошибку или
«Невозможно инициализировать собственный бэкэнд глубокого обучения: бэкэнд не найден. Невозможно построить модель Deep Water».
Ниже приведено сообщение об ошибке, связанное с запуском h2o.deepwater в R с бэкэндом tensorflow.
$ cat t4.R
'# Package Load
library(reticulate)
use_python("/opt/h2oai/dai/python/bin/python")
library(Metrics)
library(h2o)
h2o.init(max_mem_size = "500g")
'# Data Load
df <- read.csv('/data/rpjt/R_script/user/yslee/powerplant_output.csv')
'# Randomly sample 80% of the rows for the training set
set.seed(1)
train_idx <- sample(1:nrow(df), 0.8*nrow(df))
'# h2o Dataset
train <- df[train_idx,]
test <- df[-train_idx,]
train <- as.h2o(train,col.types=c("string"))
test <- as.h2o(test,col.types=c("string"))
'# h2o.deepwater model
gpu_dl <- h2o.deepwater(x = c("TemperatureCelcius","ExhaustVacuumHg","AmbientPressureMillibar","RelativeHumidity"),
y = "HourlyEnergyOutputMW",
training_frame = train,
backend = "tensorflow",
hidden = 10,
standardize =T,
activation = "Tanh",
seed = 1234)
h2o.performance(gpu_dl, newdata = test)
$ Rscript t4.R
...
R is connected to the H2O cluster:
H2O cluster uptime: 16 minutes 30 seconds
H2O cluster timezone: Asia/Seoul
H2O data parsing timezone: UTC
H2O cluster version: 3.20.0.2
H2O cluster version age: 1 month and 22 days
H2O cluster name: dai
H2O cluster total nodes: 1
H2O cluster total memory: 227.37 GB
H2O cluster total cores: 128
H2O cluster allowed cores: 128
H2O cluster healthy: TRUE
H2O Connection ip: localhost
H2O Connection port: 54321
H2O Connection proxy: NA
H2O Internal Security: FALSE
H2O API Extensions: Algos, MLI, MLI-Driver, AutoML, Core V3, Core V4
R Version: R version 3.4.1 (2017-06-30)
|======================================================================| 100%
|======================================================================| 100%
| | 0%
java.lang.RuntimeException: Unable to initialize the native Deep Learning backend: No backend found. Cannot build a Deep Water model.
java.lang.RuntimeException: Unable to initialize the native Deep Learning backend: No backend found. Cannot build a Deep Water model.
at hex.deepwater.DeepWaterModelInfo.setupNativeBackend(DeepWaterModelInfo.java:267)
at hex.deepwater.DeepWaterModelInfo.(DeepWaterModelInfo.java:214)
at hex.deepwater.DeepWaterModel.(DeepWaterModel.java:227)
at hex.deepwater.DeepWater$DeepWaterDriver.buildModel(DeepWater.java:131)
at hex.deepwater.DeepWater$DeepWaterDriver.computeImpl(DeepWater.java:118)
at hex.ModelBuilder$Driver.compute2(ModelBuilder.java:214)
at hex.deepwater.DeepWater$DeepWaterDriver.compute2(DeepWater.java:111)
at water.H2O$H2OCountedCompleter.compute(H2O.java:1260)
at jsr166y.CountedCompleter.exec(CountedCompleter.java:468)
at jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:263)
at jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:974)
at jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1477)
at jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
Error: java.lang.RuntimeException: Unable to initialize the native Deep Learning backend: No backend found. Cannot build a Deep Water model.
Execution halted
install.packages("tensorflow") и library(tensorflow) отлично работали в R,
$ ls -l /usr/local/lib64/R/library/tensorflow
total 12
-rw-rw-r-- 1 root root 2456 Aug 7 17:45 DESCRIPTION
drwxrwxr-x 5 root root 112 Aug 7 17:45 examples
drwxrwxr-x 2 root root 125 Aug 7 17:45 help
drwxrwxr-x 2 root root 39 Aug 7 17:45 html
-rw-rw-r-- 1 root root 1095 Aug 7 17:45 INDEX
drwxrwxr-x 2 root root 113 Aug 7 17:45 Meta
-rw-rw-r-- 1 root root 2713 Aug 7 17:45 NAMESPACE
drwxrwxr-x 2 root root 84 Aug 7 17:45 R
Также в питоне установлен тензорный поток от DriverlessAI.
$ which python
/opt/h2oai/dai/python/bin/python
$ pip list | grep tensorflow
tensorflow 1.8.0
