`spin lldb -- python -mspin test` doesn't work anymore

[seberg]

I think this used to work, but I suspect that python -mspin test now runs in a new process so that the debugger/lldb doesn't attach to you.

I am not sure that this can be changed easily, FWIW, python lldb -c ..., etc. all work fine (I just like to run a test that fails if it does).

EDIT: This is not new in 0.13, if it changed (I think it did), it changed a while ago.

[stefanv]

@seberg Could you give me an example of what you'd like to see happen here?

[seberg]

The way spin spawns the actual spawns the test process, lldb doesn't wrap it. Not sure if there is a clear solution though (maybe I need to tell lldb to wrap the child process or just launch the tests more explicitly).

Basically, add a test like:

def test_crash():
    import numpy as np
    from numpy.lib.stride_tricks import as_strided
    a = as_strided(np.arange(10), (10_000_000,), (8,))
    a[...] = 3

which segfaults. If you just run it with spin lldb -- python -mspin test -t ... you should find that it's not running inside lldb and doesn't stop for debugging. (Even disabling the fault handler with PYTEST_ADDOPTS="-p no:faulthandler").

Maybe there is a very simple work-around. I think last time I around I made a minimal example and used spin lldb -c "exec(open('test.py').read())" (to both run in the right process and to be sure to find the right numpy install).

[stefanv]

I looked into this a bit, but I don't know lldb well enough to proceed.

I tried -O settings set target.process.stop-on-exec false, I tried to change spin test to replace the current process, etc. and I could not get lldb to see the traceback.

With the first option I get the traceback, but trying something like bt it says the process has aborted.

Are you able to help me explore the solutions here?

[seberg]

Happy to poke this a bit, but not sure what to actually try. I can't find a good hint on how one might tell lldb to follow the new process better.
I.e. settings set target.process.stop-on-exec false or target.process.follow-fork-mode (parent|child) has no effect? (This thread makes me think maybe it just can't work on MacOS, but I am not sure it is the same problem.)

One thing to help if you poke more might be running with:

PYTEST_ADDOPTS=" -p no:faulthandler"

That will disable the Python traceback from the faulthandler. But I don't recall it should change behavior, but better safe than sorry.

I tried to change spin test to replace the current process, etc. and I could not get lldb to see the traceback.

That was the solution I was hoping might work relatively easily. If it doesn't than maybe it just isn't to be (until someone more knowledgeable teaches us a new trick...).

[stefanv]

It's possible that some of those solutions work; it's hard for me to tell, since I don't know what "successful" behavior is. Do you have an example I can look at that does not involve spin, that shows how a traceback should present?

[seberg]

Ah sorry, the "success" is that you get a segfault and can do a bt at that point.

To see it, put the "test" into a file, test.py. NOTE: I increased the fake array size here, without that it wasn't reliably and would crash very cryptically!

def test_crash():
    import numpy as np
    from numpy.lib.stride_tricks import as_strided
    a = as_strided(np.arange(10), (1_000_000_000,), (8,))
    a[1_000_000_000-1] = 3

test_crash()

then run:

spin lldb -c "exec(open('test.py').read())"

and of course then r to actually run. It should crash at least almost always with a traceback, something like:\

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x6001dd495138)
    frame #0: 0x0000000100de3540 _multiarray_umath.cpython-311-darwin.so`LONG_copyswap [inlined] _basic_copy(dst=0x00006001dd495138, src=0x000000016fdfc710, elsize=8) at arraytypes.c.src:1931:5 [opt]
   1928	    if (src == NULL) {
   1929	        return;
   1930	    }
-> 1931	    memcpy(dst, src, elsize);
   1932	}
   1933	
   1934	
Target 0: (python3.11) stopped.

And bt should will sho something obvious. (If bt does anything at all, things are probably fine!)

[seberg]

OTOH, I suppose there is a decent chance the bug is in mypy, and I guess then we would need to sanitize mypy to get anywhere...

[ngoldbaum]

We chatted about this on the NumPy triage meeting today and I shared that spin lldb does still work for me if I do something like:

spin lldb -- $(pyenv which python) $(pyenv which pytest) "build-install/usr/lib/python3.13/site-packages/numpy/_core/tests/test_stringdtype.py"

For this to work, you need to pass the absolute path to the python intepreter, the pytest script, and the relative path to the build-install version of the test you want to debug. In particular it doesn't work with e.g. python -m pytest (at least for NumPy) because that has slightly different test discovery semantics that are broken.

Maybe spin could help make this less arcane in the UI?

[stefanv]

Ouch, yes, we certainly should!

It would be easy enough to launch the correct python and pytest, but not sure how we'd deal with the build-dir. I presume this doesn't work?

spin lldb -- $(pyenv which python) $(pyenv which pytest) numpy/_core/tests/test_stringdtype.py

If it did, we could add, e.g., spin lldb -t numpy/_core_tests_test_stringdtype.py.

[ngoldbaum]

Nope, for whatever reason, that hits an error:

Details

± spin lldb -- $(pyenv which python) $(pyenv which pytest) numpy/_core/tests/test_stringdtype.py
python(94040,0x20c141f00) malloc: nano zone abandoned due to inability to reserve vm space.
Invoking `build` prior to invoking lldb:
$ /Users/goldbaum/.pyenv/versions/3.13-dev-asan/bin/python vendored-meson/meson/meson.py compile -C build
python(94094,0x20c141f00) malloc: nano zone abandoned due to inability to reserve vm space.
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /Users/goldbaum/.pyenv/versions/3.13-dev-asan/bin/ninja -C /Users/goldbaum/Documents/numpy/build
ninja: Entering directory `/Users/goldbaum/Documents/numpy/build'
[1/1] Generating numpy/generate-version with a custom command
Saving version to numpy/version.py
$ /Users/goldbaum/.pyenv/versions/3.13-dev-asan/bin/python vendored-meson/meson/meson.py install --only-changed -C build --destdir ../build-install
$ export PYTHONPATH="/Users/goldbaum/Documents/numpy/build-install/usr/lib/python3.13/site-packages"
$ lldb -O 'settings set target.process.follow-fork-mode child' -- /Users/goldbaum/.pyenv/versions/3.13-dev-asan/bin/python /Users/goldbaum/.pyenv/versions/3.13-dev-asan/bin/pytest numpy/_core/tests/test_stringdtype.py
(lldb) settings set target.process.follow-fork-mode child
(lldb) target create "/Users/goldbaum/.pyenv/versions/3.13-dev-asan/bin/python"
Current executable set to '/Users/goldbaum/.pyenv/versions/3.13-dev-asan/bin/python' (arm64).
(lldb) settings set -- target.run-args  "/Users/goldbaum/.pyenv/versions/3.13-dev-asan/bin/pytest" "numpy/_core/tests/test_stringdtype.py"
(lldb) r
Process 94122 launched: '/Users/goldbaum/.pyenv/versions/3.13-dev-asan/bin/python' (arm64)
python(94122,0x20c141f00) malloc: nano zone abandoned due to inability to reserve vm space.
ImportError while loading conftest '/Users/goldbaum/Documents/numpy/numpy/conftest.py'.
numpy/__init__.py:93: in <module>
    from . import version
E   ImportError: cannot import name 'version' from partially initialized module 'numpy' (most likely due to a circular import) (/Users/goldbaum/Documents/numpy/numpy/__init__.py)
Process 94122 exited with status = 4 (0x00000004)

[stefanv]

Right, so we'll need the same workaround we use for our regular test command to avoid importing from the wrong path.

In this case, I wonder if perhaps we add a flag to the test command, so that we wrap lldb around it?

[stefanv]

It's very much a half-baked PR, but take a look and let's see if we can iterate on that until it's working: #281