Beast2 fails with EigenDecompositionCube

[aaron-wes]

Hi

I'm currently running into an issue with Beast 2.7.5 on a linux cluster. When run, it fails quickly with a java runtime error, ID'ing this as the problematic frame:

C [libhmsbeagle-cpu-sse.so.31.0.0+0x21dc1] beagle::cpu::EigenDecompositionCube<double, 2>::updateTransitionMatrices(int, int const*, int const*, int const*, double const*, double const*, double**, int)+0x231

This seems to be related to the beagle library but I am not sure how to go about troubleshooting, as I'm not sure if its just a versioning issue (I built the xml with BEAUti 2.7.7 but am running Beast 2.7.5, as it is already installed on the cluster), an issue with priors (see warning below), an issue with the beagle install. Here is what returns when I call -beagle_info:

0 : CPU
    Flags: PRECISION_SINGLE PRECISION_DOUBLE COMPUTATION_SYNCH EIGEN_REAL EIGEN_COMPLEX SCALING_MANUAL SCALING_AUTO SCALING_ALWAYS SCALERS_RAW SCALERS_LOG VECTOR_SSE VECTOR_NONE THREADING_CPP THREADING_NONE PROCESSOR_CPU FRAMEWORK_CPU

I have also been getting a warning in stderr:

WARNING: the dimension of the parameter is -1 at the start of the run.
         The operator groupSizesDelta.t:nextclade_1 has no effect (if this does not change).

From what I can tell, the warning seems to be related to the Bayesian Coalescent Skyline element of the model (I've kept default parameters for both Pop and GroupSize, as well as defaults for dimensions). Once again, I'm not sure if this is related to the beagle issue or something independent just because I built the XML using a slightly newer version of BEAUti. For background, I'm running a GTR + I + g substitution model, an ORC with tip dates, and largely default priors.

Any guidance greatly appreciated and happy to provide any other information that may be useful!

[rbouckaert]

The XML for v2.7.7 should be compatible with v2.7.5, though new features may have been added. For the models you mention, I don't think it matters which version you run it in.

It looks more like an issue with BEAGLE and BEAST interacting. The fact that -beagle_info produces sensible output is a good sign. What happens when you run a small example like beast2/examples/testHKY.xml. If that behaves OK, how about beast2/examples/testGTR.xml?

[aaron-wes]

Looks like both throw the same error - though no warnings in stderr.

[rbouckaert]

Both should run without problem. Can you post the complete error message -- perhaps there are more clues in there?

[aaron-wes]

Sure, here's the full stderr from the testHKY file:

File: test_hky.xml seed: 1741016680900 threads: 1
Loading package BEAST.app v2.7.5 BEAST.base v2.7.5 BEASTLabs v2.0.2 ORC v1.2.0    
  Ignoring ambiguities in tree likelihood.
  Ignoring character uncertainty in tree likelihood.
  With 69 unique site patterns.
  Using rescaling scheme : dynamic
/uufs/chpc.utah.edu/sys/installdir/r8/beast/2.7.5/bin/beast: line 81: 3004864 Aborted                 (core dumped) "$JAVA" -Dlauncher.wait.for.exit=true -Xss256m -Xmx8g -Djava.library.path="$LD_LIBRARY_PATH" -Duser.language=en -cp "$BEAST_LIB/launcher.jar" beast.pkgmgmt.launcher.BeastLauncher $*

Here's the log file output:

 A fatal error has been detected by the Java Runtime Environment:

  SIGILL (0x4) at pc=0x000014a0ed2fcdc1, pid=3004864, tid=3004865

 JRE version: OpenJDK Runtime Environment Zulu17.34+19-CA (17.0.3+7) (build 17.0.3+7-LTS)
 Java VM: OpenJDK 64-Bit Server VM Zulu17.34+19-CA (17.0.3+7-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
 Problematic frame:
 C  [libhmsbeagle-cpu-sse.so.31.0.0+0x21dc1]  beagle::cpu::EigenDecompositionCube<double, 2>::updateTransitionMatrices(int, int const*, int const*, int const*, double const*, double const*, double**, int)+0x231

 Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /scratch/general/vast/u6064643/14098122/core.3004864)

 An error report file with more information is saved as:
 /scratch/general/vast/u6064643/14098122/hs_err_pid3004864.log

 If you would like to submit a bug report, please visit:
   http://www.azul.com/support/
 The crash happened outside the Java Virtual Machine in native code.
 See problematic frame for where to report the bug.

And I've attached the full hs-err file as well.

hs_err_pid3004864.log

[rbouckaert]

Thanks for all that! Looking at the logs, the trace of BEAST calls goes to the point of exponentiating matrices in BEAGLE, but then crashes. I have not found any other clue to what is going wrong.
Currently, it is picking up BEAGLE from /uufs/chpc.utah.edu/sys/installdir/beagle-lib/3.1.2/lib while the default is /usr/local/lib. Is this intentional?
Would it be possible to reinstall BEAGLE, or install in /usr/local/lib?

[aaron-wes]

I've been using a university HPC with an install of BEAST and BEAGLE that are managed as part of their module system. I'll look around in their documentation to see if it's feasible to install BEAGLE in another location. I've also tried installing BEAST in a mamba environment but it seems like the current build on bioconda is a bit older (2.6.7) and had an issue with versioning of BEAST.base as a result (error from mamba version below).

Loading package BEAST v2.6.7 Loading package BEASTLabs v1.9.7 Loading package FastRelaxedClockLogNormal v1.1.1 Loading package ORC v1.0.3 Loading package BEAST v2.6.7 The following package is required, but is not available: BEAST.base v2.7.5 See http://beast2.org/managing-packages/ for details on how to install packages.

I'll see if there's an easy way to re-install beagle somewhere that BEAST will be able to find it though if that's likely to cause an issue.

[rbouckaert]

Hope it works out. Be aware that XML for BEAST v2.7 is not compatible with XML for BEAST v2.6, so if you want to use the older version (you mentioned 2.6.7) you need to set up that analysis again in BEAUti for v2.6.