jamulus 3.12.1 headless crash with out-of-bounds vector access

[gpasset]

usr/lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_vector.h:1263: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](size_type) [with _Tp = CVector<unsigned char>; _Alloc = std::allocator<CVector<unsigned char> >; reference = CVector<unsigned char>&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.

Jamulus headless runs fine for a while but seems to crash at unknown moments. Other jamulus instances on the same server seem free of crashes.

Reproducing is semi difficult as its unclear as to when this happens. Jamulus is started as:

/usr/local/bin/Jamulus --nogui --server --serverbindip 1.2.3.4 --port 22424 --directoryaddress jazz.jamulus.io:22324 --serverinfo MyServer;MyLocation;XX --multithreading --numchannels 12

Jamulus runs on Gentoo Linux using the hardened profile. ( default/linux/amd64/23.0/hardened (stable) )
Kernel 6.18.35-gentoo-r1 #1 SMP PREEMPT_DYNAMIC Wed Jun 10 09:23:11 CEST 2026 x86_64 Intel(R) Xeon(R) CPU E3-1230L v3 @ 1.80GHz GenuineIntel GNU/Linux

Jamulus 3.12.1 has been downloaded from official github and compiled from source using only CONFIG+=headless serveronly.

ldd /usr/local/bin/Jamulus 
        linux-vdso.so.1 (0x00007fea70e20000)
        libQt6Network.so.6 => /usr/lib64/libQt6Network.so.6 (0x00007fea7080a000)
        libQt6Xml.so.6 => /usr/lib64/libQt6Xml.so.6 (0x00007fea70dee000)
        libQt6Core.so.6 => /usr/lib64/libQt6Core.so.6 (0x00007fea6fe00000)
        libstdc++.so.6 => /usr/lib/gcc/x86_64-pc-linux-gnu/15/libstdc++.so.6 (0x00007fea6fa00000)
        libm.so.6 => /usr/lib64/libm.so.6 (0x00007fea70cd0000)
        libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/15/libgcc_s.so.1 (0x00007fea70ca1000)
        libc.so.6 => /usr/lib64/libc.so.6 (0x00007fea6f814000)
        libzstd.so.1 => /usr/lib64/libzstd.so.1 (0x00007fea6fd36000)
        libz.so.1 => /usr/lib64/libz.so.1 (0x00007fea70c86000)
        libcrypto.so.3 => /usr/lib64/libcrypto.so.3 (0x00007fea6f200000)
        libglib-2.0.so.0 => /usr/lib64/libglib-2.0.so.0 (0x00007fea6f0ac000)
        libdouble-conversion.so.3 => /usr/lib64/libdouble-conversion.so.3 (0x00007fea70c6c000)
        libb2.so.1 => /usr/lib64/libb2.so.1 (0x00007fea70c63000)
        libpcre2-16.so.0 => /usr/lib64/libpcre2-16.so.0 (0x00007fea6fcd3000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fea70e22000)
        libpcre2-8.so.0 => /usr/lib64/libpcre2-8.so.0 (0x00007fea6f7a6000)
        libgomp.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/15/libgomp.so.1 (0x00007fea6f750000)

Not sure if relevant..

gcc-v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/15/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-pc-linux-gnu
Configured with: /var/tmp/portage/sys-devel/gcc-15.2.1_p20260214/work/gcc-15-20260214/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/15 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/15/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/15 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/15/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/15/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15 --disable-silent-rules --disable-dependency-tracking --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/15/python --enable-languages=c,c++ --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --disable-nls --disable-libunwind-exceptions --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo Hardened 15.2.1_p20260214 p5' --with-gcc-major-version-only --enable-libstdcxx-time --enable-lto --disable-libstdcxx-pch --enable-linker-build-id --disable-libada --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --with-multilib-list=m32,m64 --disable-fixed-point --enable-targets=all --enable-offload-defaulted --enable-offload-targets=nvptx-none --enable-libgomp --disable-libssp --enable-cet --disable-systemtap --disable-valgrind-annotations --disable-vtable-verify --disable-libvtv --with-zstd --without-isl --enable-default-pie --enable-host-pie --enable-host-bind-now --enable-default-ssp --disable-fixincludes --with-gxx-libcxx-include-dir=/usr/include/c++/v1 --with-build-config=bootstrap-cet
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 15.2.1 20260214 (Gentoo Hardened 15.2.1_p20260214 p5) 

Current suspect is that the crash occurs when all numchannels (12) are occupied or close to full, but there is no clear indication that it is.

Any help or pointers into a direction of possible resolution would be more than welcome.
cheers,
guid0

[dingodoppelt]

That's very strange. I've never had a server crash.
I'm running my servers on Ubuntu 24.04 with Qt5.

usr/lib/gcc/x86_64-pc-linux-gnu/15/include/g++-v15/bits/stl_vector.h:1263: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator [with _Tp = CVector; _Alloc = std::allocator<CVector >; reference = CVector&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.

This looks like it could be an actual Jamulus issue but I wonder if your gentoo build (I've never used gentoo) setup has something to do with this.
Could you please check if _GLIBCXX_ASSERTIONS are enabled in your build? I enabled assertions but couldn't reproduce the crash so far.

[gpasset]

Seems enabled as in release/moc_predefs.h there is: #define _GLIBCXX_ASSERTIONS 1

I figure most people still build against qt5 hence i included the ldd output to indicate i've alreay switched to Qt6.

[dingodoppelt]

Seems enabled as in release/moc_predefs.h there is: #define _GLIBCXX_ASSERTIONS 1

I figure most people still build against qt5 hence i included the ldd output to indicate i've alreay switched to Qt6.

Could you please rebuild with CXXFLAGS="-U_GLIBCXX_ASSERTIONS" and check if the crashes still occur? I still can't reproduce it with assertions enabled though

[gpasset]

Done and thanks for the quick followups. May take some creative beating by clients for it to crash again as precise environment/parameters for crash behavior are still unclear.

[pljones]

I run all my Linux servers on the latest Qt (have done for years -- we only build on Github with legacy Qt for legacy reasons).
a) I've not known one of the Directories to crash (1 client limit)
b) I've not known my Public Server to crash (but that's got a 50 client limit that's never met)
c) I've not known my private Servers to crash (but they've got 1+ limits and generally only one client - me - and aren't compiled headless, though they run headless)

So, GCC:

gcc version 15.2.1 20260214 (Gentoo Hardened 15.2.1_p20260214 p5)

Which exact Qt6?

I'm using

gcc version 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1)

and Qt 6.11.1.

[gpasset]

Qt 6.11.1.

dev-qt/qtbase-6.11.1
dev-qt/qttools-6.11.1

compile/install method:

make distclean
qmake6 "CONFIG+=headless serveronly" QMAKE_CXXFLAGS+="-U_GLIBCXX_ASSERTIONS"
make
make install

Gentoo GCC useflags-configuration:
sys-devel/gcc-15.2.1_p20260214:15::gentoo USE="cet (cxx) (default-stack-clash-protection) (default-znow) hardened (multilib) openmp (pie) sanitize (ssp) zstd -ada -cobol (-custom-cflags) -d -debug -doc (-fixed-point) -fortran -go -graphite (-ieee-long-double) -jit -libgdiagnostics (-libssp) -lto -modula2 -nls -objc -objc++ -objc-gc (-pch) -pgo -rust -systemtap -test (-time64) -valgrind -vanilla -vtv"

[dingodoppelt]

Gentoo GCC useflags-configuration:
sys-devel/gcc-15.2.1_p20260214:15::gentoo USE="cet (cxx) (default-stack-clash-protection) (default-znow) hardened (multilib) openmp (pie) sanitize (ssp) zstd -ada -cobol (-custom-cflags) -d -debug -doc (-fixed-point) -fortran -go -graphite (-ieee-long-double) -jit -libgdiagnostics (-libssp) -lto -modula2 -nls -objc -objc++ -objc-gc (-pch) -pgo -rust -systemtap -test (-time64) -valgrind -vanilla -vtv"

I don't know gentoo but could you please post the output of strings Jamulus | grep glibcxx_assert? Just to be sure :)

[gpasset]

No output when I run: /usr/x86_64-pc-linux-gnu/binutils-bin/2.46.0/strings /usr/local/bin/Jamulus | grep -i glibcxx_assert

[dingodoppelt]

Server crash reported but found this:

# grep -r "CVector<unsigned char>"   
src/client.h:    CVector<unsigned char> vecCeltData;

Makes me wonder if this is related to the infrequent "unclean exits" by the client 🤔

[softins]

I've been talking to ChatGPT about this. It's actually a vector of CVectors that is being accessed out of range, and typedefs are resolved before reporting. So what we're looking for is CVector<CVector<unsigned char>>, or CVector<CVector<uint8_t>> (CVector is derived from std::vector).

We have a few of the latter:

• vecvecbyCodedData in server.*
• vecvecMemory and vecvecTempMemory in buffer.*.

So we need to audit those for possible out-of-range indexing of the outer vector.

[softins]

I have set up a 3.12.1 server in Jazz called "softins debug server", compiled with assertions and debug, and with capturing of coredumps enabled. It has multi-threading enabled (only 2 cpus though) and max clients of 5.