HPC Slurm: batchtools works; future.batchtools command not found

[cbuglino]

Hello All,

I'm having an issue with future.batchtools that I've been unable to resolve: batchtools::submitJobs fails when called from future.batchtools, but not when called from batchtools. I believe it may be a problem with how future.batchtools handles the config file, but I'm not sure.

Running with batchtools works as expected. R loads my batchtools temp file and configuration file.

> Project '/scratch/Renaldi/TEST' loaded. [renv 0.15.5]
> library(batchtools)
> reg = makeRegistry("test")
Sourcing configuration file '/scratch/Renaldi/TEST/batchtools.conf.R' ...
Created registry in '/scratch/Renaldi/TEST/test' using cluster functions 'Slurm'

> piApprox = function(n) {
  nums = matrix(runif(2 * n), ncol = 2)
  d = sqrt(nums[, 1]^2 + nums[, 2]^2)
  4 * mean(d <= 1)
}

> batchMap(fun = piApprox, n = rep(1e5, 10))
Adding 10 jobs ...

> submitJobs(resources = list(walltime = 3600, memory = 1024, ntasks = 1, ncpus = 1))
Submitting 10 jobs in 10 chunks using cluster functions 'Slurm' ...

> getJobStatus()
    job.id           submitted             started                done error
 1:      1 2023-02-24 15:37:34 2023-02-24 15:37:59 2023-02-24 15:37:59  <NA>
 2:      2 2023-02-24 15:37:35 2023-02-24 15:37:59 2023-02-24 15:37:59  <NA>
 3:      3 2023-02-24 15:37:35 2023-02-24 15:37:59 2023-02-24 15:37:59  <NA>
 4:      4 2023-02-24 15:37:36 2023-02-24 15:37:59 2023-02-24 15:37:59  <NA>
 5:      5 2023-02-24 15:37:37 2023-02-24 15:37:59 2023-02-24 15:37:59  <NA>
 6:      6 2023-02-24 15:37:37 2023-02-24 15:37:59 2023-02-24 15:37:59  <NA>
 7:      7 2023-02-24 15:37:38 2023-02-24 15:37:59 2023-02-24 15:37:59  <NA>
 8:      8 2023-02-24 15:37:38 2023-02-24 15:37:59 2023-02-24 15:37:59  <NA>
 9:      9 2023-02-24 15:37:39 2023-02-24 15:37:59 2023-02-24 15:37:59  <NA>
10:     10 2023-02-24 15:37:39 2023-02-24 15:37:59 2023-02-24 15:37:59  <NA>

> sapply(1:10, loadResult)
 [1] 3.14096 3.14524 3.14800 3.15160 3.14620 3.13772 3.14400 3.13456 3.14068
[10] 3.13800

Running using future.batchtools fails to submit jobs.

Project '/scratch/Renaldi/TEST' loaded. [renv 0.15.5]
 
> library(future)
> library(future.batchtools)
Loading required package: parallelly
> piApprox = function(n) {
  nums = matrix(runif(2 * n), ncol = 2)
  d = sqrt(nums[, 1]^2 + nums[, 2]^2)
  4 * mean(d <= 1)
}

> plan(tweak(
    batchtools_slurm,
    resources = list(
      walltime = 60,
      ntasks = 1,
      ncpus = 1,
      memory = 1024
    )
  ))
> results <- list()
> for(i in 1:10){results[[i]] <- future::future(piApprox(1e5))}
Error: Failed to submit BatchtoolsSlurmFuture (<none>). The reason was: Listing of jobs failed (exit code 127);
cmd: 'squeue --user=$USER --states=R,S,CG --noheader --format=%i -r'
output:
command not found
TROUBLESHOOTING INFORMATION:
batchtools::submitJobs() was called with the following 'resources' argument:
List of 4
 $ walltime: num 60
 $ ntasks  : num 1
 $ ncpus   : num 1
 $ memory  : num 1024

batchtools.slurm.tmpl:

#!/bin/bash

## Job Resource Interface Definition
##
## ntasks [integer(1)]:       Number of required tasks,
##                            Set larger than 1 if you want to further parallelize
##                            with MPI within your job.
## ncpus [integer(1)]:        Number of required cpus per task,
##                            Set larger than 1 if you want to further parallelize
##                            with multicore/parallel within each task.
## walltime [integer(1)]:     Walltime for this job, in seconds.
##                            Must be at least 60 seconds for Slurm to work properly.
## memory   [integer(1)]:     Memory in megabytes for each cpu.
##                            Must be at least 100 (when I tried lower values my
##                            jobs did not start at all).
##
## Default resources can be set in your .batchtools.conf.R by defining the variable
## 'default.resources' as a named list.

<%
# relative paths are not handled well by Slurm
log.file = fs::path_expand(log.file)
-%>


#SBATCH --job-name=<%= job.name %>
#SBATCH --output=<%= log.file %>
#SBATCH --error=<%= log.file %>
#SBATCH --time=<%= ceiling(resources$walltime / 60) %>
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=<%= resources$ncpus %>
#SBATCH --mem-per-cpu=<%= resources$memory %>
<%= if (!is.null(resources$partition)) sprintf(paste0("#SBATCH --partition='", resources$partition, "'")) %>
<%= if (array.jobs) sprintf("#SBATCH --array=1-%i", nrow(jobs)) else "" %>

## Initialize work environment like
## source /etc/profile
## module add ...
module purge
module load r/gcc/4.2.0

## Export value of DEBUGME environemnt var to slave
export DEBUGME=<%= Sys.getenv("DEBUGME") %>

<%= sprintf("export OMP_NUM_THREADS=%i", resources$omp.threads) -%>
<%= sprintf("export OPENBLAS_NUM_THREADS=%i", resources$blas.threads) -%>
<%= sprintf("export MKL_NUM_THREADS=%i", resources$blas.threads) -%>

## Run R:
## we merge R output with stdout from SLURM, which gets then logged via --output option
Rscript -e 'batchtools::doJobCollection("<%= uri %>")'

batchtools.conf.R
Note: The HPC login node is named "log-3"

cluster.functions = makeClusterFunctionsSlurm(nodename = "log-3")

[HenrikBengtsson]

The development version of future.batchtools implements a patch for a batchtools bug that has caused issues for some Slurm users when launching jobs (https://future.batchtools.futureverse.org/news/index.html). It should make it more stable. Install it as:

remotes::install_github("futureverse/future.batchtools", ref="develop")

Looking at your example, I think the built-in job-script template may handle it, using something like:

library(future)
plan(future.batchtools::batchtools_slurm, resources = list(
  modules = c("r/gcc/4.2.0"),
  envs = c(DEBUGME = Sys.getenv("DEBUGME")),
  time = "01:00:00",    # walltime=60 => 60s
  "cpus-per-task" = 1,
  "mem-per-cpu" = "1024M"  
  ntasks = 1
))

f <- future(sessionInfo())
info <- value(f)
print(info)

resources$omp.threads and resources$blas.threads were not set, so I skipped OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, and MKL_NUM_THREADS.

cluster.functions = makeClusterFunctionsSlurm(nodename = "log-3")

If you really need this, please let me know.