How to Install R on an HPC: A Comedy in T̶w̶o̶ -- NO -- THREE Acts (a.k.a. 'The Longest Day')

TL;DR: Just look at the Gist.

Summary:

  • Act I, in which I try and fail.
  • Act II, in which I think I succeed but actually failed without knowing it till I tried to use it.
  • Act III, in which I return to my beginning, ponder the universe, dive deep into the depths of the abyss, and come back with the magic bean that makes everything work.

Act I: Build from Source (Estimated Run Time, Including Trial-and-Error Figuring Things Out, Running through the configure-make Dance a Million Times Before Hitting Walls and Needing to Start Again: The Better Part of An Afternoon When You Really Were Planning to Just Getting The Actual Analyses Done)

Let us start things off slow. No need to go stampeding for the final bits. Let us just begin with a gentle dependency dance here:

cd ~/src
wget --no-check-certificate https://zlib.net/zlib-1.2.11.tar.gz
tar xf zlib-1.2.11.tar.gz
cd zlib-1.2.11.tar.gz
configure --prefix=$HOME/Environment/local
make && make install

cd ~/src
wget http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz
tar xf bzip2-1.0.6.tar.gz
cd bzip2-1.0.6
sed -i 's/^CFLAGS=-Wall/CFLAGS=-fPIC -Wall/' Makefile
make && make install PREFIX=$HOME/Environment/local

cd ~/src
wget --no-check-certificate https://tukaani.org/xz/xz-5.2.3.tar.gz
tar xf xz-5.2.3.tar.gz
cd xz-5.2.3
./configure --prefix=$HOME/Environment/local
make && make install

cd ~/src
wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.41.tar.bz2
tar xf pcre-8.41.tar.bz2
cd pcre-8.41
./configure --enable-utf8 --prefix=$HOME/Environment/local
make && make install

cd ~/src
wget http://git.savannah.gnu.org/cgit/readline.git/snapshot/readline-master.tar.gz
tar xf readline-master.tar.gz
cd readline-master
./configure --prefix=$HOME/Environment/local
make && make install

cd ~/src
wget https://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.15.tar.gz
tar xf libiconv-1.15.tar.gz
cd libiconv-1.15
./configure --prefix=$HOME/Environment/local
make && make install

And then something to point the R configure to the right place (some of these are actually taken care off in my “~/.bashrc”, but for some of those the R build still fails to find things if I do not explicitly export them like this here.)

export PATH=$HOME/Environment/local/bin:$PATH
export LD_LIBRARY_PATH=$HOME/Environment/local/lib:$LD_LIBRARY_PATH
export ZLIB_HOME=$HOME/Environment/local/
export PKG_CONFIG_PATH="$PKG_CONFIG_PATH:$HOME/Environment/local/lib/pkgconfig"
export CFLAGS="-I$HOME/Environment/local/include"
export CPPFLAGS="-I$HOME/Environment/local/include"
export LDFLAGS="-L$HOME/Environment/local/lib"
libtool --finish $HOME/Environment/local/lib

Before trying to get down to business:

cd ~/src
export RVER=3.5.0
wget https://cran.r-project.org/src/base/R-3/R-$RVER.tar.gz
tar xf R-$RVER.tar.gz
cd R-$RVER
./configure --prefix=$HOME/Environment/local --enable-R-shlib
make && make install

Yay?

Nope.

.
.
.
platform.c:(.text+0x781a): undefined reference to `u_getVersion_54'
platform.c:(.text+0x7833): undefined reference to `u_versionToString_54'
util.o: In function `resetICUcollator':
util.c:(.text+0x4e50): undefined reference to `ucol_close_54'
util.o: In function `do_ICUset':
util.c:(.text+0x4fab): undefined reference to `ucol_close_54'
util.c:(.text+0x501f): undefined reference to `uloc_setDefault_54'
util.c:(.text+0x5034): undefined reference to `uloc_setDefault_54'
util.c:(.text+0x5065): undefined reference to `ucol_open_54'
util.c:(.text+0x519c): undefined reference to `ucol_setStrength_54'
util.c:(.text+0x51cf): undefined reference to `ucol_setAttribute_54'
util.o: In function `do_ICUget':
util.c:(.text+0x52c7): undefined reference to `ucol_getLocaleByType_54'
util.o: In function `Rf_Scollate':
util.c:(.text+0x5373): undefined reference to `uloc_setDefault_54'
util.c:(.text+0x53a8): undefined reference to `ucol_open_54'
util.c:(.text+0x54be): undefined reference to `uiter_setUTF8_54'
util.c:(.text+0x54d7): undefined reference to `uiter_setUTF8_54'
util.c:(.text+0x5505): undefined reference to `ucol_strcollIter_54'
collect2: error: ld returned 1 exit status
make[3]: *** [R.bin] Error 1
make[3]: Leaving directory `/.../src/R-3.5.0/src/main'
make[2]: *** [R] Error 2
make[2]: Leaving directory `/.../src/R-3.5.0/src/main'
make[1]: *** [R] Error 1
make[1]: Leaving directory `/.../src/R-3.5.0/src'
make: *** [R] Error 1

OK.

That’s it – Imma getting out of this hole.

Act II: Use Conda (Run time: approximately 4 mins; User time: approximately 8 seconds)

$ conda create -n my_py36 --clone=/panfs/pfs.local/software/install/anaconda/4.3.11/envs/py36
$ source activate my_py36
$ conda install r

… aaaaaand done:

$ R

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
.
.
.

To install packages:

$ conda install r r-ape
$ conda install r r-testthat

Act III: Go Back to Building From Source

2018-05-02 18:52 UPDATE: Turns out that if the package you want is not in the Anacando repository, and you do not have admin rights on the cluster (which, if you did, you probably would not need to go through any of this as R is available from the repo’s of most distributions), then you might be SOL. Because the only way to get a package not in the Anaconda repo’s is to build it, and to build it you need write access to the root conda.

So, that sucks.

So, back to trying to build and install R from source from scratch. The stumper was that the error I was seeing was simply not turning up in any of my Google searches. Furthermore, the missing names (e.g., ucol_close_54) were not showing up even in the R source code, though the stem parts of the name were (e.g., ucol_close); the “\_54” suffix was presumably the result of some sort of compiler macro name-munging. I started trawling through various code repos, and finally stumbled on ICU. This seemed to be it: providing the missing names, albeit without the numerical suffix decoration. My initial attempt at installing it and trying (for the ba-gazillionth time) to build R failed, and again I was about to give up (I came up with a desperate scheme to install a vagrant box and then having access to R would be a “sudo ” away …). But then I considered that I had pulled the latest version, which was 55.x, and what if — just what if — I needed version 54.x instead, just like the munged-name extensions suggested??!! So, downloaded, built and installed version 54.1:

$ cd ~/src
$ wget http://download.icu-project.org/files/icu4c/54.1/icu4c-54_1-src.tgz
$ tar xf icu4c-54_1-src.tgz
$ cd icu/source
$ ./configure --prefix=$HOME/Environment/local
$ make && make install

Then back to the R directory to try and build and …

$ R

R version 3.5.0 (2018-04-23) -- "Joy in Playing"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
.
.
.

IT WORKED!!!!!

Wow.

What a ride!

Afterword

In R’s defense (just in case there was any criticism implied or read in the above), just last week I did manage to install R 3.4 successfully from source on a 10-year old machine in about 20 minutes, so I suspect the problem here is something with R 3.5 or this particular cluster environment.

Share