TypeError: annotate_integrated() got multiple values for argument 'ref_data'

[zhangbaifeng]

Code:
blueprint_ref = celldex.fetch_reference("blueprint_encode", "2024-02-26", realize_assays=True)
immune_cell_ref = celldex.fetch_reference("hpca", "2024-02-26", realize_assays=True)

import singlecellexperiment as sce
mat = adata.X.T
features = [str(x) for x in adata.var_names]

Reference datasets (blueprint_ref and hpca_ref)

ref_data = [
blueprint_ref,
hpca_ref
]
ref_labels = [
blueprint_ref.get_column_data().column("label.main"),
hpca_ref.get_column_data().column("label.main")
]

Correct the call to annotate_integrated:

single_results, integrated = singler.annotate_integrated(mat,features,ref_data=ref_data,ref_labels=ref_labels,num_threads=24)

Access results

integrated.column("best_label")

Error message:

TypeError Traceback (most recent call last)
Cell In[26], line 32
25 ref_labels = [
26 blueprint_ref.get_column_data().column("label.main"),
27 hpca_ref.get_column_data().column("label.main")
28 ]
31 # Correct the call to annotate_integrated:
---> 32 single_results, integrated = singler.annotate_integrated(mat,features,ref_data=ref_data,ref_labels=ref_labels,num_threads=24)
33 # Access results
34 integrated.column("best_label")

TypeError: annotate_integrated() got multiple values for argument 'ref_data'

[LTLA]

I assume you're referring to the README, in which case this is fixed in 068c47c.

[zhangbaifeng]

thank you for your prompt reply. It seems that dolomite_matrix is not suitable. Since the server is offline, I downloaded the reference on my personal computer, saved it as a SummarizedExperiment using dolomite_base, and then imported it to the server, but it doesn't seem to work either.

[LTLA]

If you set realize_assays=True, there should never be any ReloadedArraySeed class; the assay matrices in the SCEs created by fetch_reference() should be regular NumPy arrays.

import celldex
blueprint_ref = celldex.fetch_reference("blueprint_encode", "2024-02-26", realize_assays=True)
type(blueprint_ref.get_assay(0))
## <class 'numpy.ndarray'>

immune_cell_ref = celldex.fetch_reference("hpca", "2024-02-26", realize_assays=True)
type(immune_cell_ref.get_assay(0))
## <class 'numpy.ndarray'>

All of this works fine for me:

# Make up some data.
import numpy
mat = numpy.random.rand(10000, 1000)
features = blueprint_ref.get_row_names()[:10000]

# Run it as described in the README.
import singler
single_results, integrated = singler.annotate_integrated(
    mat,
    ref_data = [
        blueprint_ref,
        immune_cell_ref
    ],
    ref_labels = [
        blueprint_ref.get_column_data().column("label.main"),
        immune_cell_ref.get_column_data().column("label.main")
    ],
    test_features = features,
    num_threads = 6
)

Running on Python 3.10, singler 0.4.1, celldex 0.3.0.

[zhangbaifeng]

Thank you for your reply. I used fetch reference and set realize_assays = True. However, since my server is offline, I had to export the fetched reference using dolomite_base and then import it again on the server. Perhaps this process converted the reference matrix into a dolomite matrix. Do you have a better method to save the fetched reference and reload it in Python in a format suitable for Singler input?

here is my code:

import celldex
import singler
blueprint_ref = celldex.fetch_reference("blueprint_encode","2024-02-26",realize_assays=True)
hpca_ref = celldex.fetch_reference("hpca","2024-02-26",realize_assays=True)

save

from dolomite_base import save_object
import dolomite_se
save_object(blueprint_ref,'/mnt/raid6/baifeng/blueprint_ref')
save_object(hpca_ref,'/mnt/raid6/baifeng/hpca_ref')

#load
from dolomite_base import read_object
blueprint_ref = read_object('/mnt/raid6/baifeng/blueprint_ref')

[LTLA]

The ReloadedArray is just a DelayedArray so you can do

import celldex

# Using realize_assays=False to get a ReloadedArray.
blueprint_ref = celldex.fetch_reference("blueprint_encode","2024-02-26",realize_assays=False)

# Force it to be a dense array.
import delayedarray
forced = delayedarray.to_dense_array(blueprint_ref.assay("logcounts"))

# Then just reassign it.
blueprint_ref = blueprint_ref.set_assay("logcounts", forced)

[zhangbaifeng]

Thank you. That works but:

RuntimeError Traceback (most recent call last)
Cell In[62], line 39
32 ref_labels = [
33 blueprint_ref.get_column_data().column("label.main"),
34 hpca_ref.get_column_data().column("label.main")
35 ]
38 # Correct the call to annotate_integrated:
---> 39 single_results, integrated = singler.annotate_integrated(mat,ref_data,ref_labels,features,num_threads=24)
40 # Access results
41 integrated.column("best_label")

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/site-packages/singler/annotate_integrated.py:182, in annotate_integrated(test_data, ref_data, ref_labels, test_features, ref_features, test_assay_type, ref_assay_type, test_check_missing, ref_check_missing, train_single_args, classify_single_args, train_integrated_args, classify_integrated_args, num_threads)
171 for r in range(nrefs):
172 curbuilt = train_single(
173 ref_data=all_ref_data[r],
174 ref_labels=all_ref_labels[r],
(...)
179 num_threads=num_threads,
180 )
--> 182 res = classify_single(
183 test_data,
184 ref_prebuilt=curbuilt,
185 **classify_single_args,
186 num_threads=num_threads,
187 )
189 all_built.append(curbuilt)
190 all_results.append(res)

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/site-packages/singler/classify_single.py:69, in classify_single(test_data, ref_prebuilt, assay_type, quantile, use_fine_tune, fine_tune_threshold, num_threads)
66 if isinstance(test_data, summarizedexperiment.SummarizedExperiment):
67 test_data = test_data.assay(assay_type)
---> 69 test_ptr = mattress.initialize(test_data)
71 best, raw_scores, delta = lib.classify_single(
72 test_ptr.ptr,
73 ref_prebuilt._ptr,
(...)
77 num_threads
78 )
80 all_labels = ref_prebuilt.labels

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/functools.py:888, in singledispatch..wrapper(*args, **kw)
884 if not args:
885 raise TypeError(f'{funcname} requires at least '
886 '1 positional argument')
--> 888 return dispatch(args[0].class)(*args, **kw)

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/site-packages/mattress/initialize.py:82, in _initialize_delayed_array(x)
80 @initialize.register
81 def _initialize_delayed_array(x: delayedarray.DelayedArray) -> InitializedMatrix:
---> 82 return initialize(x.seed)

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/functools.py:888, in singledispatch..wrapper(*args, **kw)
884 if not args:
885 raise TypeError(f'{funcname} requires at least '
886 '1 positional argument')
--> 888 return dispatch(args[0].class)(*args, **kw)

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/site-packages/mattress/initialize.py:127, in _initialize_delayed_subset(x)
125 @initialize.register
126 def _initialize_delayed_subset(x: delayedarray.Subset) -> InitializedMatrix:
--> 127 components = initialize(x.seed)
128 for dim in range(2):
129 current = x.subset[dim]

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/functools.py:888, in singledispatch..wrapper(*args, **kw)
884 if not args:
885 raise TypeError(f'{funcname} requires at least '
886 '1 positional argument')
--> 888 return dispatch(args[0].class)(*args, **kw)

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/site-packages/mattress/initialize.py:82, in _initialize_delayed_array(x)
80 @initialize.register
81 def _initialize_delayed_array(x: delayedarray.DelayedArray) -> InitializedMatrix:
---> 82 return initialize(x.seed)

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/functools.py:888, in singledispatch..wrapper(*args, **kw)
884 if not args:
885 raise TypeError(f'{funcname} requires at least '
886 '1 positional argument')
--> 888 return dispatch(args[0].class)(*args, **kw)

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/site-packages/mattress/initialize.py:127, in _initialize_delayed_subset(x)
125 @initialize.register
126 def _initialize_delayed_subset(x: delayedarray.Subset) -> InitializedMatrix:
--> 127 components = initialize(x.seed)
128 for dim in range(2):
129 current = x.subset[dim]

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/functools.py:888, in singledispatch..wrapper(*args, **kw)
884 if not args:
885 raise TypeError(f'{funcname} requires at least '
886 '1 positional argument')
--> 888 return dispatch(args[0].class)(*args, **kw)

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/site-packages/mattress/initialize.py:77, in _initialize_sparse_csc_matrix(x)
75 @initialize.register
76 def _initialize_sparse_csc_matrix(x: scipy.sparse.csc_matrix) -> InitializedMatrix:
---> 77 return _initialize_sparse_csc_array(x)

File /rsrch8/home/genomic_med/bzhang11/.conda/envs/spatial/lib/python3.9/site-packages/mattress/initialize.py:72, in _initialize_sparse_csc_array(x)
70 itmp = _contiguify(x.indices)
71 indtmp = x.indptr.astype(numpy.uint64, copy=False, order="A")
---> 72 return InitializedMatrix(lib.initialize_compressed_sparse_matrix(x.shape[0], x.shape[1], dtmp, itmp, indtmp, False))

RuntimeError: 'indices' should be strictly increasing within each column

[LTLA]

Pretty much as the error message says; your test data is (nominally) a compressed sparse column matrix but the indices are not in the expected order for the CSC format.

I can reproduce your error by manually mutating the sparse matrix:

import celldex
blueprint_ref = celldex.fetch_reference("blueprint_encode", "2024-02-26", realize_assays=True)
immune_cell_ref = celldex.fetch_reference("hpca", "2024-02-26", realize_assays=True)

import scipy
mat = scipy.sparse.rand(10000, 100).tocsc()
features = blueprint_ref.get_row_names()[:10000]

# Manually changing the contents of the matrix so that the expected ordering is broken:
mat.indices[0] = 100

import singler
single_results, integrated = singler.annotate_integrated(
    mat,
    ref_data = [
        blueprint_ref,
        immune_cell_ref
    ],
    ref_labels = [
        blueprint_ref.get_column_data().column("label.main"),
        immune_cell_ref.get_column_data().column("label.main")
    ],
    test_features = features,
    num_threads = 6
)
## Traceback (most recent call last):
##   File "<stdin>", line 1, in <module>
##   File "/home/luna/.local/lib/python3.10/site-packages/singler-0.0.0-py3.10-linux-x86_64.egg/singler/annotate_integrated.py", line 182, in annotate_integrated
##     res = classify_single(
##   File "/home/luna/.local/lib/python3.10/site-packages/singler-0.0.0-py3.10-linux-x86_64.egg/singler/classify_single.py", line 69, in classify_single
##     test_ptr = mattress.initialize(test_data)
##   File "/usr/lib/python3.10/functools.py", line 889, in wrapper
##     return dispatch(args[0].__class__)(*args, **kw)
##   File "/home/luna/.local/lib/python3.10/site-packages/mattress/initialize.py", line 82, in _initialize_delayed_array
##     return initialize(x.seed)
##   File "/usr/lib/python3.10/functools.py", line 889, in wrapper
##     return dispatch(args[0].__class__)(*args, **kw)
##   File "/home/luna/.local/lib/python3.10/site-packages/mattress/initialize.py", line 127, in _initialize_delayed_subset
##     components = initialize(x.seed)
##   File "/usr/lib/python3.10/functools.py", line 889, in wrapper
##     return dispatch(args[0].__class__)(*args, **kw)
##   File "/home/luna/.local/lib/python3.10/site-packages/mattress/initialize.py", line 77, in _initialize_sparse_csc_matrix
##     return _initialize_sparse_csc_array(x)
##   File "/home/luna/.local/lib/python3.10/site-packages/mattress/initialize.py", line 72, in _initialize_sparse_csc_array
##     return InitializedMatrix(lib.initialize_compressed_sparse_matrix(x.shape[0], x.shape[1], dtmp, itmp, indtmp, False))
## RuntimeError: 'indices' should be strictly increasing within each column

Outside of these mutations, I don't know how you managed to obtain a matrix that would trigger this error. Consider providing a minimal reproducible example.

[etfrer-yi]

I got the exact same issue and was able to solve it.

The problem lies within a discrepancy between the documentation at https://pypi.org/project/singler/ and the actual code itself.

The documentation indicates that you should provide your data as

...
single_results, integrated = singler.annotate_integrated(
    mat,
    features,
    ref_data = [
        blueprint_ref,
        immune_cell_ref
    ],
    ref_labels = [
        blueprint_ref.get_column_data().column("label.main"),
        immune_cell_ref.get_column_data().column("label.main")
    ],
    num_threads = 6
)

Notice how features is being passed as a second argument.

In the actual code itself, however, we see at

singler-py/src/singler/annotate_integrated.py

Line 15 in d32f6da

ref_data: Sequence,

that the second argument is NOT features but rather ref_data.

To solve the issue, pass to singler.annotate_integrated test_features=features as opposed to passing in features as the second argument.

[LTLA]

Yes, see #33 (comment).