cellpose error in comseg run

[PietroD]

Description of the bug

Trying to run Comseg with cellpose prior, I get this error:

executor > local (25)
[5d/6af09c] NFC…PA:SOPA:TO_SPATIALDATA (1) | 1 of 1, cached: 1 ✔
[6a/094fee] NFC…SOPA:SOPA:EXPLORER_RAW (1) | 1 of 1, cached: 1 ✔
[37/4a57e2] NFC…OPA:MAKE_IMAGE_PATCHES (1) | 1 of 1, cached: 1 ✔
[19/a9a8f6] NFC…SEGMENTATION_CELLPOSE (27) | 0 of 294
[- ] NFC…:CELLPOSE:RESOLVE_CELLPOSE -
[- ] NFC…PA:MAKE_TRANSCRIPT_PATCHES -
[- ] NFC…:PATCH_SEGMENTATION_COMSEG -
[- ] NFC…SOPA:COMSEG:RESOLVE_COMSEG -
[- ] NFCORE_SOPA:SOPA:AGGREGATE -
[- ] NFCORE_SOPA:SOPA:EXPLORER -
[- ] NFCORE_SOPA:SOPA:REPORT -
[- ] NFCORE_SOPA:SOPA:PUBLISH -
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/sopa] Pipeline completed with errors-
WARN: Failed to render execution report -- see the log file for details
WARN: Failed to render execution timeline -- see the log file for details
ERROR ~ Error executing process > 'NFCORE_SOPA:SOPA:CELLPOSE:PATCH_SEGMENTATION_CELLPOSE (21)'

Caused by:
Process NFCORE_SOPA:SOPA:CELLPOSE:PATCH_SEGMENTATION_CELLPOSE (21) terminated with an error exit status (1)

Command executed:

sopa segmentation cellpose human_brain_1k.zarr --patch-index 20 --diameter 60 --channels 'DAPI' --model-type 'nuclei' --flow-threshold 2 --cellprob-threshold -6 --min-area 2000

Command exit status:
1

Command output:

Welcome to CellposeSAM, cellpose v
cellpose version: │4.0.6
platform: linux
python version: ──────3.10.18
torch version: rue 2.8.0+cu128! The neural network component of
CPSAM is much larger than in previous versions and CPU excution is slow.
We encourage users to use GPU/MPS if available.

Command error:
│ 1176 │ │ except FileNotFoundError: │
│ 1177 │ │ │ if not parents or self.parent == self: │
│ 1178 │ │ │ │ raise │
│ ❱ 1179 │ │ │ self.parent.mkdir(parents=True, exist_ok=True) │
│ 1180 │ │ │ self.mkdir(mode, parents=False, exist_ok=exist_ok) │
│ 1181 │ │ except OSError: │
│ 1182 │ │ │ # Cannot rely on checking for EEXIST, since the operating │
│ │
│ ╭─────────────────────────── locals ────────────────────────────╮ │
│ │ exist_ok = True │ │
│ │ mode = 511 │ │
│ │ parents = True │ │
│ │ self = PosixPath('/home/delfino.pietro/.cellpose/models') │ │
│ ╰───────────────────────────────────────────────────────────────╯ │
│ │
│ /usr/local/lib/python3.10/pathlib.py:1179 in mkdir │
│ │
│ 1176 │ │ except FileNotFoundError: │
│ 1177 │ │ │ if not parents or self.parent == self: │
│ 1178 │ │ │ │ raise │
│ ❱ 1179 │ │ │ self.parent.mkdir(parents=True, exist_ok=True) │
│ 1180 │ │ │ self.mkdir(mode, parents=False, exist_ok=exist_ok) │
│ 1181 │ │ except OSError: │
│ 1182 │ │ │ # Cannot rely on checking for EEXIST, since the operating │
│ │
│ ╭──────────────────────── locals ────────────────────────╮ │
│ │ exist_ok = True │ │
│ │ mode = 511 │ │
│ │ parents = True │ │
│ │ self = PosixPath('/home/delfino.pietro/.cellpose') │ │
│ ╰────────────────────────────────────────────────────────╯ │
│ │
│ /usr/local/lib/python3.10/pathlib.py:1175 in mkdir │
│ │
│ 1172 │ │ Create a new directory at this given path. │
│ 1173 │ │ """ │
│ 1174 │ │ try: │
│ ❱ 1175 │ │ │ self._accessor.mkdir(self, mode) │
│ 1176 │ │ except FileNotFoundError: │
│ 1177 │ │ │ if not parents or self.parent == self: │
│ 1178 │ │ │ │ raise │
│ │
│ ╭─────────────────── locals ───────────────────╮ │
│ │ exist_ok = True │ │
│ │ mode = 511 │ │
│ │ parents = True │ │
│ │ self = PosixPath('/home/delfino.pietro') │ │
│ ╰──────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
OSError: [Errno 30] Read-only file system: '/home/delfino.pietro'

Work dir:
/beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/code/work/cd/1a0ba7f41ae46cf53aad9065328568

Container:
/beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/code/sopa/workflow/.nextflow/singularity/docker.io-quentinblampey-sopa-latest-cellpose.img

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

-- Check '/beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/code/sopa/workflow/.nextflow/nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: https://nf-co.re/docs/usage/troubleshooting

-- Check '/beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/code/sopa/workflow/.nextflow/nextflow.log' file for details

Command used and terminal output

#!/bin/bash
#SBATCH --job-name=nextflow_run
#SBATCH --partition=workq
#SBATCH --cpus-per-task=24
#SBATCH --mem=128G
#SBATCH --mail-type=END
#SBATCH --mail-user=delfino.pietro@hsr.it
#SBATCH --output=/beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/code/sopa/workflow/logs/nextflow_%j.out
#SBATCH --error=/beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/code/sopa/workflow/logs/nextflow_%j.err

source ~/.bashrc
eval "$(conda shell.bash hook)"
conda activate nf-env  # Cambia con il tuo env

export NXF_HOME=/beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/code/sopa/workflow/.nextflow
export NXF_CACHE_DIR=$NXF_HOME/cache
export NXF_SINGULARITY_CACHEDIR=$NXF_HOME/singularity
export NXF_LOG_FILE=$NXF_HOME/nextflow.log

export CELLPOSE_HOME=/beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/cellpose
mkdir -p $CELLPOSE_HOME

export NXF_SINGULARITY_OPTS="--env CELLPOSE_HOME=$CELLPOSE_HOME --home $CELLPOSE_HOME:/home/delfino.pietro"

nextflow run nf-core/sopa \
   -r dev \
   -profile singularity \
   --input /beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/code/sopa/workflow/samplesheet_nextflow/samplesheet.csv \
   -params-file /beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/code/sopa/workflow/config/merscope/comseg_cellpose.yaml \
   --outdir /beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/results/merscope/human_brain_1k/nextflow_test \
   -c /beegfs/scratch/ric.iannacone/ric.iannacone/prj/Spatial/benchmark_ist/code/nextflow.config \
   -resume \
   -with-report ./report.html \
   -with-trace ./trace.txt \
   -with-timeline ./timeline.html \
   -with-dag ./dag.html

Relevant files

No response

System information

Nextflow version: 25.04.6
Hardware: HPC
Executor: slurm
Container engine: Singularity
OS: CentOS Linux
Version of nf-core/sopa: 1.0.0

[quentinblampey]

Hi @PietroD,

Thanks for reporting! The nextflow version of sopa is still not fully stable, and cellpose is indeed not working on singularity yet (sorry, you're not lucky, this is exactly what you're trying to run 😅)

It's a permission issue that should be resolved by passing the right container options to singularity, I'll work on this soon.

Meanwhile, you can try to run solely comseg, or other tools like proseg. You might still experience issues (as nf-core/sopa is still in dev mode), but it may work, contrary to cellpose+singularity, which is known not to work.

[quentinblampey]

Hello @PietroD, it may be solved now.

One comment, though: the current cellpose image is based on the latest cellpose version (v4), which is slow if not using a GPU. I'm assuming that by default we would prefer using cellpose on CPU only, so having an image with cellpose v3 may be preferred.
Do you have an opinion on this? Were you planning to use GPUs or not?

[PietroD]

Thanks Quentin,
Well if I had to decide, I believe that I would use the GPU unless it comes with headache in setting things up.

[quentinblampey]

Okay thanks for your feedback @PietroD, then I'll keep the image with cellpose 4 by default, and I may add another image with cellpose v3 in the future in case some users want it