imread() and imwrite() freeze after soft reset

[sfe-SparkFro]

Replication code (Edit: probably works with imwrite() too, after discovering root problem):

import cv2
img = cv2.imread("valve.png")

With the image valve.png (source: Wikipedia, downscaled to 320x240)

Running the code above after a hard reset works fine. Can even run it multiple times without trouble.

After a soft reset, running the code again freezes after calling cv2.imread("valve.png")

[sfe-SparkFro]

Poking around with a debugger, slight change in behavior with a3f6cd5

Before a3f6cd5, cv2_imgcodecs_imread() would get all the way to the end, then crash upon de-allocating img because the allocator was not correctly set. Partial backtrace:

Thread 1 "rp2350.dap.core0" received signal SIGTRAP, Trace/breakpoint trap.
HardFault_Handler () at /home/dryw/micropython-opencv/micropython/lib/pico-sdk/src/rp2_common/pico_crt0/crt0.S:170
170     decl_isr_bkpt isr_hardfault
(gdb) bt
#0  HardFault_Handler ()
    at /home/dryw/micropython-opencv/micropython/lib/pico-sdk/src/rp2_common/pico_crt0/crt0.S:170
#1  <signal handler called>
#2  0x101fd4c2 in cv::Mat::deallocate (this=<optimized out>)
    at /home/dryw/micropython-opencv/src/opencv/modules/core/src/matrix.cpp:739
#3  0x101fd5bc in cv::Mat::release (this=0x20081d88)
    at /home/dryw/micropython-opencv/src/opencv/modules/core/src/matrix.cpp:550
#4  cv::Mat::~Mat (this=this@entry=0x20081d88, __in_chrg=<optimized out>)
    at /home/dryw/micropython-opencv/src/opencv/modules/core/src/matrix.cpp:479
#5  0x10075094 in cv2_imgcodecs_imread (n_args=<optimized out>, 
    pos_args=<optimized out>, kw_args=<optimized out>)
    at /home/dryw/micropython-opencv/src/imgcodecs.cpp:77
#6  0x100115ee in fun_builtin_var_call (args=0x20081edc, n_kw=0, n_args=1, 
    self_in=0x1030d7e8 <cv2_imgcodecs_imread_obj>)
    at /home/dryw/micropython-opencv/micropython/py/objfun.c:118
#7  fun_builtin_var_call (self_in=0x1030d7e8 <cv2_imgcodecs_imread_obj>, n_args=1, 
    n_kw=0, args=0x20081edc)
    at /home/dryw/micropython-opencv/micropython/py/objfun.c:102
#8  0x1001bf0e in mp_execute_bytecode (code_state=code_state@entry=0x20081ec0, 
    inject_exc=inject_exc@entry=0x0)

After a3f6cd5, imdecode() freezes when attempting to determine the image decoder for the file. Backtrace:

Thread 1 "rp2350.dap.core0" received signal SIGTRAP, Trace/breakpoint trap.
HardFault_Handler () at /home/dryw/micropython-opencv/micropython/lib/pico-sdk/src/rp2_common/pico_crt0/crt0.S:170
170     decl_isr_bkpt isr_hardfault
(gdb) bt
#0  HardFault_Handler ()
    at /home/dryw/micropython-opencv/micropython/lib/pico-sdk/src/rp2_common/pico_crt0/crt0.S:170
#1  <signal handler called>
#2  0x1010e25c in cv::findDecoder (buf=...)
    at /home/dryw/micropython-opencv/src/opencv/modules/imgcodecs/src/loadsave.cpp:299
#3  0x1010e3f6 in cv::imdecode_ (buf=..., flags=flags@entry=1, mat=...)
    at /home/dryw/micropython-opencv/src/opencv/modules/imgcodecs/src/loadsave.cpp:1000
#4  0x1010f186 in cv::imdecode (_buf=..., flags=flags@entry=1)
    at /home/dryw/micropython-opencv/src/opencv/modules/imgcodecs/src/loadsave.cpp:1125
#5  0x10071e2c in cv2_imgcodecs_imread (n_args=<optimized out>, 
    pos_args=<optimized out>, kw_args=<optimized out>)
    at /home/dryw/micropython-opencv/src/opencv/modules/core/include/opencv2/core/mat.inl.hpp:92
#6  0x100115ee in fun_builtin_var_call (args=0x20081edc, n_kw=0, n_args=1, 
    self_in=0x1030d828 <cv2_imgcodecs_imread_obj>)
    at /home/dryw/micropython-opencv/micropython/py/objfun.c:118
#7  fun_builtin_var_call (self_in=0x1030d828 <cv2_imgcodecs_imread_obj>, n_args=1, 
    n_kw=0, args=0x20081edc)
    at /home/dryw/micropython-opencv/micropython/py/objfun.c:102
#8  0x1001bf0e in mp_execute_bytecode (code_state=code_state@entry=0x20081ec0, inject_exc=inject_exc@entry=0x0) at /home/dryw/micropython-opencv/micropython/py/vm.c:1042
#9  0x1001170e in fun_bc_call (self_in=0x2003acd0, n_args=<optimized out>, n_kw=<optimized out>, args=0x0) at /home/dryw/micropython-opencv/micropython/py/objfun.c:295
#10 0x100316de in parse_compile_execute (source=source@entry=0x20081f88, input_kind=<optimized out>, exec_flags=exec_flags@entry=22) at /home/dryw/micropython-opencv/micropython/shared/runtime/pyexec.c:119
#11 0x100319a6 in pyexec_friendly_repl () at /home/dryw/micropython-opencv/micropython/shared/runtime/pyexec.c:671
#12 0x10033ffe in main (argc=<optimized out>, argv=<optimized out>) at /home/dryw/micropython-opencv/micropython/ports/rp2/main.c:236

[sfe-SparkFro]

Possibly an issue with StdMatAllocator(): https://github.com/opencv/opencv/blob/7ca36d9c508180e6db2a95f2c6cfaaea3750c10a/modules/core/src/matrix.cpp#L196-L199

It uses OpenCV's lazy init, which only creates the object when first needed, then returns a pointer on all subsequent calls. I suspect there are some places where OpenCV internally creates Mat objects (such as here in imdecode()) that use the StdMatAllocator, and the memory location of StdMatAllocator gets overwritten after a soft reset when the garbage collector is re-initialized, similar to the root problem of #13. If so, might be able to fix with a similar approach to a2bf9f2.

Or better yet, might be able to use a similar solution to call setDefaultAllocator(&GetNumpyAllocator()) at startup so all Mat objects use the NumPy allocator.

[sfe-SparkFro]

Looks like the problem is here: https://github.com/opencv/opencv/blob/85b45656fa99a82d34ad9ee129fd03ef82efa448/modules/imgcodecs/src/loadsave.cpp#L244-L249

g_codecs stores all the encoders and decoders available in OpenCV, and the struct only gets constructed one time upon first use, then a pointer is returned on subsequent calls. It's allocated on the GC heap, so after a soft reset, the GC is reinitialized and the memory location has garbage data.

Before a soft reset, the first decoder is clearly for Bmp:

After a soft reset, the memory location has apparently garbage data:

I'm guessing we can do a similar thing to a2bf9f2

[sfe-SparkFro]

50f29e5 is an improvement, but now it freezes for a different reason. Actually, the same reason, but this time with StdMatAllocator. Backtrace:

Thread 1 "rp2350.dap.core0" received signal SIGTRAP, Trace/breakpoint trap.
HardFault_Handler () at /home/dryw/micropython-opencv/micropython/lib/pico-sdk/src/rp2_common/pico_crt0/crt0.S:170
170     decl_isr_bkpt isr_hardfault
(gdb) bt
#0  HardFault_Handler ()
    at /home/dryw/micropython-opencv/micropython/lib/pico-sdk/src/rp2_common/pico_crt0/crt0.S:170
#1  <signal handler called>
#2  0x02f90000 in ?? ()
#3  0x101fdb1a in cv::Mat::create (this=0x20081dc0, d=d@entry=2, 
    _sizes=<optimized out>, _sizes@entry=0x20081ad0, _type=16)
    at /home/dryw/micropython-opencv/src/opencv/modules/core/src/matrix.cpp:699
#4  0x101fdc74 in cv::Mat::create (this=this@entry=0x20081dc0, _rows=_rows@entry=240, 
    _cols=_cols@entry=320, _type=<optimized out>)
    at /home/dryw/micropython-opencv/src/opencv/modules/core/src/matrix.cpp:533
#5  0x1010e62e in cv::imdecode_ (buf=..., flags=flags@entry=1, mat=...)
    at /home/dryw/micropython-opencv/src/opencv/modules/imgcodecs/src/loadsave.cpp:1075
#6  0x1010f1d6 in cv::imdecode (_buf=..., flags=flags@entry=1)
    at /home/dryw/micropython-opencv/src/opencv/modules/imgcodecs/src/loadsave.cpp:1125
#7  0x10071e44 in cv2_imgcodecs_imread (n_args=<optimized out>, 
    pos_args=<optimized out>, kw_args=<optimized out>)
    at /home/dryw/micropython-opencv/src/opencv/modules/core/include/opencv2/core/mat.inl.hpp:92
#8  0x100115ee in fun_builtin_var_call (args=0x20081edc, n_kw=0, n_args=1, 
    self_in=0x1030d8ac <cv2_imgcodecs_imread_obj>)
    at /home/dryw/micropython-opencv/micropython/py/objfun.c:118
#9  fun_builtin_var_call (self_in=0x1030d8ac <cv2_imgcodecs_imread_obj>, n_args=1, n_kw=0, args=0x20081edc) at /home/dryw/micropython-opencv/micropython/py/objfun.c:102
#10 0x1001bf0e in mp_execute_bytecode (code_state=code_state@entry=0x20081ec0, inject_exc=inject_exc@entry=0x0) at /home/dryw/micropython-opencv/micropython/py/vm.c:1042
#11 0x1001170e in fun_bc_call (self_in=0x20047fd0, n_args=<optimized out>, n_kw=<optimized out>, args=0x0) at /home/dryw/micropython-opencv/micropython/py/objfun.c:295
#12 0x100316de in parse_compile_execute (source=source@entry=0x20081f88, input_kind=<optimized out>, exec_flags=exec_flags@entry=22) at /home/dryw/micropython-opencv/micropython/shared/runtime/pyexec.c:119
#13 0x100319a6 in pyexec_friendly_repl () at /home/dryw/micropython-opencv/micropython/shared/runtime/pyexec.c:671
#14 0x10033ffe in main (argc=<optimized out>, argv=<optimized out>) at /home/dryw/micropython-opencv/micropython/ports/rp2/main.c:236

imdecode() creates a couple internal Mat objects, which end up using StdMatAllocator instead of the NumPy allocator. StdMatAllocator gets created on the first call to imread(), then after a soft reset, the memory location gets overwritten.

Before a soft reset:

(gdb) p g_matAllocator 
$25 = (cv::MatAllocator *) 0x20047ff8
(gdb) p *g_matAllocator 
$26 = {_vptr.MatAllocator = 0x2000cae4 <vtable for cv::StdMatAllocator+8>}

After a soft reset:

(gdb) print g_matAllocator 
$27 = (cv::MatAllocator *) 0x20047ff8
(gdb) print *g_matAllocator 
$28 = {_vptr.MatAllocator = 0x614d6474}

Instead of ensuring StdMatAllocator is on the C heap, I think it would actually be better to call Mat::setDefaultAllocator(&GetNumpyAllocator()) so the NumPy allocator is always called for all Mat objects, ensuring this same problem can't come up again elsewhere. Also means we don't need to explicitly set Mat allocators anywhere else.

[sfe-SparkFro]

Fixed by 52a290e

[sfe-SparkFro]

Update in 37756bb

Setting the NumPy allocator as the default allocator caused problems with Canny(), which internally creates a temporary 64-bit float Mat. which ulab does not support. It's not a problem to leave StdMatAllocator as the default, because __wrap_malloc() ensures all memory gets allocated on the GC heap, and ensures all OpenCV data types are supported, at least for internal operations.