You are not logged in.

#1 2024-05-14 16:43:29

jordz
Member
Registered: 2006-02-01
Posts: 250

Glibc segfaults on bhyve vm

I need to downgrade to glibc-2.39-1, because newer versions fail to generate my locale/segfault.
Other apps like vim or neovim also fail to launch.
The strange thing is that I only have this issue on my freebsd bhyve vm (Ryzen 7600), other machines where I run bare metal don't have this issue.

upgrade of glibc:

jordy@archvm ~ % sudo pacman -Syu
[sudo] password for jordy: 
:: Synchronizing package databases...
 core is up to date
 extra is up to date
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (1) glibc-2.39+r52+gf8e4623421-1

Total Installed Size:  47.25 MiB
Net Upgrade Size:      -0.02 MiB

:: Proceed with installation? [Y/n] 
(1/1) checking keys in keyring                      [##########################] 100%
(1/1) checking package integrity                    [##########################] 100%
(1/1) loading package files                         [##########################] 100%
(1/1) checking for file conflicts                   [##########################] 100%
(1/1) checking available disk space                 [##########################] 100%
:: Processing package changes...
(1/1) upgrading glibc                               [##########################] 100%
Generating locales...
  en_US.UTF-8.../usr/bin/locale-gen: line 41: 350344 Segmentation fault      (core dumped) localedef -i $input -c -f $charset -A /usr/share/locale/locale.alias $locale
:: Running post-transaction hooks...
(1/3) Reloading system manager configuration...
(2/3) Arming ConditionNeedsUpdate...
(3/3) Updating the info directory file...
jordy@archvm ~ % vim
realloc(): invalid next size







Vim: Caught deadly signal ABRT
Vim: Finished.
134 jordy@archvm ~ % nvim
134 jordy@archvm ~ %

Offline

#2 2024-05-14 17:04:08

loqs
Member
Registered: 2014-03-06
Posts: 17,719

Re: Glibc segfaults on bhyve vm

Please post the backtrace with debug symbols from a  coredump produce by both localedef and vim.

Offline

#3 2024-05-14 17:28:15

jordz
Member
Registered: 2006-02-01
Posts: 250

Re: Glibc segfaults on bhyve vm

I have a feeling I'm not debugging it right.
I downgrade to the older glibc and check coredumpctl + enable debuginfod


localedef:

jordy@archvm ~ % coredumpctl debug 1320  
           PID: 1320 (localedef)
           UID: 1000 (jordy)
           GID: 1000 (jordy)
        Signal: 11 (SEGV)
     Timestamp: Tue 2024-05-14 19:09:31 CEST (14min ago)
  Command Line: localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
    Executable: /usr/bin/localedef
 Control Group: /user.slice/user-1000.slice/session-1.scope
          Unit: session-1.scope
         Slice: user-1000.slice
       Session: 1
     Owner UID: 1000 (jordy)
       Boot ID: 3a04dee9f36d4e28984ccaa3a2504136
    Machine ID: 1be263a307b845cdb90380e4aee3222e
      Hostname: archvm
       Storage: /var/lib/systemd/coredump/core.localedef.1000.3a04dee9f36d4e28984ccaa3a2504136.1320.1715706571000000.zst (present)
  Size on Disk: 17.3M
       Message: Process 1320 (localedef) of user 1000 dumped core.
                
                Stack trace of thread 1320:
                #0  0x0000000000000000 n/a (n/a + 0x0)
                #1  0x000059c35d4e176d n/a (localedef + 0x1a76d)
                #2  0x000059c35d4fbdd3 n/a (localedef + 0x34dd3)
                #3  0x000059c35d4ca571 n/a (localedef + 0x3571)
                #4  0x000078074d838c88 n/a (libc.so.6 + 0x25c88)
                #5  0x000078074d838d4c __libc_start_main (libc.so.6 + 0x25d4c)
                #6  0x000059c35d4ca8f5 n/a (localedef + 0x38f5)
                ELF object binary architecture: AMD x86-64

GNU gdb (GDB) 14.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/localedef...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.archlinux.org>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading separate debug info for /usr/bin/localedef
(No debugging symbols found in /usr/bin/localedef)                                   
[New LWP 1320]
Core was generated by `localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000000000 in ?? ()
(gdb) thread apply all backtrace full

Thread 1 (LWP 1320):
#0  0x0000000000000000 in ?? ()
No symbol table info available.
#1  0x000059c35d4df487 in ?? ()
No symbol table info available.
#2  0x000059c363e56110 in ?? ()
No symbol table info available.
#3  0x000059c363e56160 in ?? ()
No symbol table info available.
#4  0x000059c363e561b0 in ?? ()
No symbol table info available.
#5  0x000059c363e56200 in ?? ()
No symbol table info available.
#6  0x000059c363e56250 in ?? ()
No symbol table info available.
#7  0x000059c363e562a0 in ?? ()
No symbol table info available.
#8  0x000059c363e562f0 in ?? ()
No symbol table info available.
#9  0x000059c363e56340 in ?? ()
No symbol table info available.
#10 0x000059c363e56390 in ?? ()
No symbol table info available.
#11 0x000059c363e563e0 in ?? ()
No symbol table info available.
#12 0x000059c363e56430 in ?? ()
No symbol table info available.
#13 0x000059c363e56480 in ?? ()
No symbol table info available.
#14 0x000059c363e564d0 in ?? ()
No symbol table info available.
#15 0x000059c363e56520 in ?? ()
No symbol table info available.
#16 0x000059c363e56580 in ?? ()
No symbol table info available.
#17 0x000059c363e565e0 in ?? ()
No symbol table info available.
--Type <RET> for more, q to quit, c to continue without paging--
#18 0x000059c363e56630 in ?? ()
No symbol table info available.
#19 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)

vim

jordy@archvm ~ % coredumpctl debug 350439
           PID: 350439 (vim)
           UID: 1000 (jordy)
           GID: 1000 (jordy)
        Signal: 6 (ABRT)
     Timestamp: Tue 2024-05-14 18:36:24 CEST (46min ago)
  Command Line: vim
    Executable: /usr/bin/vim
 Control Group: /user.slice/user-1000.slice/session-13.scope
          Unit: session-13.scope
         Slice: user-1000.slice
       Session: 13
     Owner UID: 1000 (jordy)
       Boot ID: e79cc7ef153d4d6396609b1b8e49fe09
    Machine ID: 1be263a307b845cdb90380e4aee3222e
      Hostname: archvm
       Storage: /var/lib/systemd/coredump/core.vim.1000.e79cc7ef153d4d6396609b1b8e49fe09.350439.1715704584000000.zst (present)
  Size on Disk: 490.8K
       Message: Process 350439 (vim) of user 1000 dumped core.
                
                Stack trace of thread 350439:
                #0  0x0000791b2107eceb kill (libc.so.6 + 0x3cceb)
                #1  0x0000576c66c0be41 mch_exit (vim + 0x1f5e41)
                #2  0x0000576c66dbdfaa getout (vim + 0x3a7faa)
                #3  0x0000791b2107eae0 n/a (libc.so.6 + 0x3cae0)
                #4  0x0000791b210d6e44 n/a (libc.so.6 + 0x94e44)
                #5  0x0000791b2107ea30 raise (libc.so.6 + 0x3ca30)
                #6  0x0000791b210664c3 abort (libc.so.6 + 0x244c3)
                #7  0x0000791b21067354 n/a (libc.so.6 + 0x25354)
                #8  0x0000791b210e1085 n/a (libc.so.6 + 0x9f085)
                #9  0x0000791b210e5234 n/a (libc.so.6 + 0xa3234)
                #10 0x0000791b210e61e5 realloc (libc.so.6 + 0xa41e5)
                #11 0x0000576c66a85359 ga_grow_inner (vim + 0x6f359)
                #12 0x0000576c66d0e5c7 ex_command (vim + 0x2f85c7)
                #13 0x0000576c66b2e82f do_cmdline (vim + 0x11882f)
                #14 0x0000576c66c74ef1 n/a (vim + 0x25eef1)
                #15 0x0000576c66c738ab do_in_path (vim + 0x25d8ab)
                #16 0x0000576c66dc3601 vim_main2 (vim + 0x3ad601)
                #17 0x0000576c66a83741 main (vim + 0x6d741)
                #18 0x0000791b21067c88 n/a (libc.so.6 + 0x25c88)
                #19 0x0000791b21067d4c __libc_start_main (libc.so.6 + 0x25d4c)
                #20 0x0000576c66a84b15 _start (vim + 0x6eb15)
                ELF object binary architecture: AMD x86-64

GNU gdb (GDB) 14.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/vim...

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.archlinux.org>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading separate debug info for /usr/bin/vim
(No debugging symbols found in /usr/bin/vim)                                         
[New LWP 350439]

warning: .dynamic section for "/usr/lib/libm.so.6" is not at the expected address (wrong library or version mismatch?)
Downloading separate debug info for /usr/lib/libm.so.6
Downloading separate debug info for /usr/lib/libncursesw.so.6                        
Downloading separate debug info for /usr/lib/libacl.so.1                             
Downloading separate debug info for /usr/lib/libgpm.so.2                             
Downloading separate debug info for system-supplied DSO at 0x7ffc40be8000            
                                                                                     
warning: File "/usr/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
--Type <RET> for more, q to quit, c to continue without paging--
	add-auto-load-safe-path /usr/lib/libthread_db.so.1
line to your configuration file "/home/jordy/.config/gdb/gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/home/jordy/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `vim'.
Program terminated with signal SIGABRT, Aborted.
#0  0x0000791b2107eceb in ?? () from /usr/lib/libc.so.6
(gdb) thread apply all backtrace full

Thread 1 (LWP 350439):
#0  0x0000791b2107eceb in ?? () from /usr/lib/libc.so.6
No symbol table info available.
#1  0x0000000000000006 in ?? ()
No symbol table info available.
#2  0x00007ffc40bd2140 in ?? ()
No symbol table info available.
#3  0x00007ffc40bd2000 in ?? ()
No symbol table info available.
#4  0x0000791b2107eae0 in ?? () from /usr/lib/libc.so.6
No symbol table info available.
#5  0x0000791b2103d740 in ?? ()
No symbol table info available.
#6  0x00007ffc40bd2140 in ?? ()
No symbol table info available.
#7  0x0000000000000006 in ?? ()
No symbol table info available.
#8  0x00007ffc40bd2140 in ?? ()
No symbol table info available.
#9  0x00000000000558e7 in ?? ()
No symbol table info available.
#10 0x00000000000558e7 in ?? ()
No symbol table info available.
#11 0x00007ffc40bd2000 in ?? ()
No symbol table info available.
#12 0x00000000000558e7 in ?? ()
No symbol table info available.
#13 0x0000000000000006 in ?? ()
No symbol table info available.
#14 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)

Last edited by jordz (2024-05-14 17:31:41)

Offline

#4 2024-05-20 18:29:07

ht
Member
Registered: 2024-05-20
Posts: 2

Re: Glibc segfaults on bhyve vm

I'm also seeing this problem running under Bhyve. Everything works with glibc 2.39-1 but with 2.39-2 localedef and various other things start crashing.

Getting a backtrace is complicated by the fact that gdb also crashes with the new glibc, but I managed to get this by crashing localedef on one machine (running Bhyve) and debugging the core dump on another (not running Bhyve), both with glibc 2.39+r52+gf8e4623421-1:

Core was generated by `localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000000000 in ?? ()
(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x000056464904e487 in memmove (__dest=<optimized out>, 
    __src=<optimized out>, __len=<optimized out>)
    at ../string/bits/string_fortified.h:36
#2  allocate_arrays (ctype=ctype@entry=0x56464c407560, 
    charmap=<optimized out>, repertoire=<optimized out>)
    at programs/ld-ctype.c:3895
#3  0x000056464905076d in ctype_output (locale=<optimized out>, 
    charmap=<optimized out>, 
    output_path=0x56464aa4e9b0 "/usr/lib/locale/en_US.utf8/")
    at programs/ld-ctype.c:914
#4  0x000056464906add3 in write_all_categories (definitions=0x7ffe84b9fdf0, 
    charmap=charmap@entry=0x56464aa4f060, 
    locname=0x7ffe84ba0f28 "en_US.UTF-8", 
    output_path=output_path@entry=0x56464aa4e9b0 "/usr/lib/locale/en_US.utf8/")
    at programs/locfile.c:336
#5  0x0000564649039571 in main (argc=<optimized out>, argv=0x7ffe84ba00a8)
    at programs/localedef.c:306

And here is the backtrace of gdb 14.2-2 crashing (also with glibc 2.39+r52+gf8e4623421-1):

Core was generated by `gdb localedef /localedef.core'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __pthread_kill_implementation (threadid=<optimized out>, 
    signo=signo@entry=11, no_tid=no_tid@entry=0) at pthread_kill.c:44
Downloading source file /usr/src/debug/glibc/glibc/nptl/pthread_kill.c
44            return INTERNAL_SYSCALL_ERROR_P (ret) ? INTERNAL_SYSCALL_ERRNO (ret) : 0;
[Current thread is 1 (Thread 0x7f1410eaa580 (LWP 1281))]
(gdb) bt
#0  __pthread_kill_implementation (threadid=<optimized out>, 
    signo=signo@entry=11, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007f1413de2eb3 in __pthread_kill_internal (threadid=<optimized out>, 
    signo=11) at pthread_kill.c:78
#2  0x00007f1413d8aa30 in __GI_raise (sig=11) at ../sysdeps/posix/raise.c:26
#3  0x000055ca2c5b548f in ?? ()
#4  0x000055ca2c5b56dc in ?? ()
#5  <signal handler called>
#6  PyUnicode_GET_LENGTH (op=0x1300000000)
    at ./Include/cpython/unicodeobject.h:282
#7  unsafe_latin_compare (v=0x7f140fd4b5a0, w=0x1300000000, ms=0x7fff349ed9a0)
    at Objects/listobject.c:2133
#8  0x00007f141467c132 in binarysort (start=0x7f140fdd80c8, hi=0x7f140fdd8138, 
    lo=..., ms=0x7fff349ed9a0) at Objects/listobject.c:1257
#9  list_sort_impl (self=self@entry=0x7f140fdd1200, keyfunc=<optimized out>, 
    keyfunc@entry=0x0, reverse=reverse@entry=0) at Objects/listobject.c:2443
#10 0x00007f141472f34f in PyList_Sort (v=0x7f140fdd1200)
    at Objects/listobject.c:2520
#11 list_builtin_module_names () at ./Python/sysmodule.c:2438
#12 _PySys_InitCore (sysdict=0x7f140fdd0240, 
    tstate=0x7f1414b20ae8 <_PyRuntime+459656>) at ./Python/sysmodule.c:3302
#13 _PySys_Create (sysmod_p=<synthetic pointer>, 
    tstate=0x7f1414b20ae8 <_PyRuntime+459656>) at ./Python/sysmodule.c:3604
#14 pycore_interp_init (tstate=tstate@entry=0x7f1414b20ae8 <_PyRuntime+459656>)
    at Python/pylifecycle.c:856
#15 0x00007f1414728d76 in pyinit_config (runtime=0x7f1414ab0760 <_PyRuntime>, 
    config=0x7fff349eec50, tstate_p=<optimized out>)
    at Python/pylifecycle.c:897
#16 pyinit_core (src_config=src_config@entry=0x7fff349eef10, 
    tstate_p=tstate_p@entry=0x7fff349eee58, 
    runtime=0x7f1414ab0760 <_PyRuntime>) at Python/pylifecycle.c:1060
#17 0x00007f1414728604 in Py_InitializeFromConfig (config=0x7fff349eef10)
    at Python/pylifecycle.c:1282
#18 0x000055ca2c787664 in ?? ()
#19 0x000055ca2c9d7401 in ?? ()
#20 0x000055ca2c3ee560 in ?? ()
#21 0x00007f1413d73c88 in __libc_start_call_main (
    main=main@entry=0x55ca2c3ee500, argc=argc@entry=3, 
    argv=argv@entry=0x7fff349ef618)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#22 0x00007f1413d73d4c in __libc_start_main_impl (main=0x55ca2c3ee500, argc=3, 
--Type <RET> for more, q to quit, c to continue without paging--c
    argv=0x7fff349ef618, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7fff349ef608)
    at ../csu/libc-start.c:360
#23 0x000055ca2c3f6b05 in ?? ()

And since that seems to be crashing in Python, here's python 3.12.3-1 crashing immediately when running "python3":

Core was generated by `python3'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  PyUnicode_GET_LENGTH (op=0x0) at ./Include/cpython/unicodeobject.h:282
Downloading source file /usr/src/debug/python/Python-3.12.3/./Include/cpython/unicodeobject.h
282         return _PyASCIIObject_CAST(op)->length;                             
(gdb) bt
#0  PyUnicode_GET_LENGTH (op=0x0) at ./Include/cpython/unicodeobject.h:282
#1  unsafe_latin_compare (v=0x7f53a991f5a0, w=0x0, ms=0x7ffdc6f82800)
    at Objects/listobject.c:2133
#2  0x00007f53a9ecc132 in binarysort (start=0x7f53a99442c8, hi=0x7f53a9944328, 
    lo=..., ms=0x7ffdc6f82800) at Objects/listobject.c:1257
#3  list_sort_impl (self=self@entry=0x7f53a99a5200, keyfunc=<optimized out>, 
    keyfunc@entry=0x0, reverse=reverse@entry=0) at Objects/listobject.c:2443
#4  0x00007f53a9f7f34f in PyList_Sort (v=0x7f53a99a5200)
    at Objects/listobject.c:2520
#5  list_builtin_module_names () at ./Python/sysmodule.c:2438
#6  _PySys_InitCore (sysdict=0x7f53a99a4240, 
    tstate=0x7f53aa370ae8 <_PyRuntime+459656>) at ./Python/sysmodule.c:3302
#7  _PySys_Create (sysmod_p=<synthetic pointer>, 
    tstate=0x7f53aa370ae8 <_PyRuntime+459656>) at ./Python/sysmodule.c:3604
#8  pycore_interp_init (tstate=tstate@entry=0x7f53aa370ae8 <_PyRuntime+459656>)
    at Python/pylifecycle.c:856
#9  0x00007f53a9f78d76 in pyinit_config (runtime=0x7f53aa300760 <_PyRuntime>, 
    config=0x7ffdc6f83ab0, tstate_p=<optimized out>)
    at Python/pylifecycle.c:897
#10 pyinit_core (src_config=src_config@entry=0x7ffdc6f83d70, 
    tstate_p=tstate_p@entry=0x7ffdc6f83cb8, 
    runtime=0x7f53aa300760 <_PyRuntime>) at Python/pylifecycle.c:1060
#11 0x00007f53a9f78604 in Py_InitializeFromConfig (config=0x7ffdc6f83d70)
    at Python/pylifecycle.c:1282
#12 0x00007f53a9f78361 in pymain_init (args=0x7ffdc6f83fc0)
    at Modules/main.c:67
#13 0x00007f53a9f77420 in pymain_main (args=<optimized out>)
    at Modules/main.c:730
#14 0x00007f53a9f76fab in Py_BytesMain (argc=<optimized out>, 
    argv=<optimized out>) at Modules/main.c:763
#15 0x00007f53a9b87c88 in __libc_start_call_main (
    main=main@entry=0x55ba82094120 <main>, argc=argc@entry=1, 
    argv=argv@entry=0x7ffdc6f84118)
    at ../sysdeps/nptl/libc_start_call_main.h:58
#16 0x00007f53a9b87d4c in __libc_start_main_impl (main=0x55ba82094120 <main>, 
    argc=1, argv=0x7ffdc6f84118, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7ffdc6f84108)
    at ../csu/libc-start.c:360
#17 0x000055ba82094045 in _start ()

And here's vim 9.1.0408-1 printing some error messages before crashing:

[root@hostname /]# vim
malloc(): invalid size (unsorted)
malloc(): invalid size (unsorted)
Vim: Double signal, exiting
malloc(): invalid size (unsorted)
Aborted (core dumped)
Core was generated by `vim'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f6d21986ceb in __GI_kill ()
    at ../sysdeps/unix/syscall-template.S:120
Downloading source file /usr/src/debug/glibc/glibc/signal/../sysdeps/unix/syscall-template.S
120     T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)                  
(gdb) bt
#0  0x00007f6d21986ceb in __GI_kill ()
    at ../sysdeps/unix/syscall-template.S:120
#1  0x000055da1b74e69c in ?? ()
#2  <signal handler called>
#3  __pthread_kill_implementation (threadid=<optimized out>, 
    signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#4  0x00007f6d219deeb3 in __pthread_kill_internal (threadid=<optimized out>, 
    signo=6) at pthread_kill.c:78
#5  0x00007f6d21986a30 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/posix/raise.c:26
#6  0x00007f6d2196e4c3 in __GI_abort () at abort.c:79
#7  0x00007f6d2196f354 in __libc_message_impl (
    fmt=fmt@entry=0x7f6d21af82ea "%s\n") at ../sysdeps/posix/libc_fatal.c:132
#8  0x00007f6d219e9085 in malloc_printerr (
    str=str@entry=0x7f6d21afb828 "malloc(): invalid size (unsorted)")
    at malloc.c:5772
#9  0x00007f6d219ec554 in _int_malloc (
    av=av@entry=0x7f6d21b2cac0 <main_arena>, bytes=bytes@entry=168)
    at malloc.c:4078
#10 0x00007f6d219ee831 in __libc_calloc (n=n@entry=1, 
    elem_size=elem_size@entry=168) at malloc.c:3754
#11 0x00007f6d21b7f320 in tparm_setup (term=0x55da1bf9a8c0, 
    string=0x55da1b9409b9 "\033[%i%p1%d;%p2%dH", result=0x7ffe722155f0)
    at ../ncurses/./tinfo/lib_tparm.c:655
#12 0x00007f6d21b88f2d in _nc_tiparm (expected=2, 
    string=0x55da1b9409b9 "\033[%i%p1%d;%p2%dH")
    at ../ncurses/./tinfo/lib_tparm.c:1331
#13 0x00007f6d21b846c1 in tgoto (string=0x55da1b9409b9 "\033[%i%p1%d;%p2%dH", 
    x=<optimized out>, y=60) at ../ncurses/./tinfo/lib_tgoto.c:210
#14 0x000055da1b7af96f in windgoto ()
#15 0x000055da1b90227e in getout ()
#16 0x000055da1b74e6e3 in ?? ()
#17 <signal handler called>
#18 __pthread_kill_implementation (threadid=<optimized out>, 
    signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#19 0x00007f6d219deeb3 in __pthread_kill_internal (threadid=<optimized out>, 
    signo=6) at pthread_kill.c:78
#20 0x00007f6d21986a30 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/posix/raise.c:26
#21 0x00007f6d2196e4c3 in __GI_abort () at abort.c:79
#22 0x00007f6d2196f354 in __libc_message_impl (
    fmt=fmt@entry=0x7f6d21af82ea "%s\n") at ../sysdeps/posix/libc_fatal.c:132
#23 0x00007f6d219e9085 in malloc_printerr (
    str=str@entry=0x7f6d21afb828 "malloc(): invalid size (unsorted)")
    at malloc.c:5772
#24 0x00007f6d219ec554 in _int_malloc (
    av=av@entry=0x7f6d21b2cac0 <main_arena>, bytes=bytes@entry=168)
    at malloc.c:4078
#25 0x00007f6d219ee831 in __libc_calloc (n=n@entry=1, 
    elem_size=elem_size@entry=168) at malloc.c:3754
#26 0x00007f6d21b7f320 in tparm_setup (term=0x55da1bf9a8c0, 
    string=0x55da1b9409b9 "\033[%i%p1%d;%p2%dH", result=0x7ffe72216250)
    at ../ncurses/./tinfo/lib_tparm.c:655
#27 0x00007f6d21b88f2d in _nc_tiparm (expected=2, 
    string=0x55da1b9409b9 "\033[%i%p1%d;%p2%dH")
    at ../ncurses/./tinfo/lib_tparm.c:1331
#28 0x00007f6d21b846c1 in tgoto (string=0x55da1b9409b9 "\033[%i%p1%d;%p2%dH", 
    x=<optimized out>, y=60) at ../ncurses/./tinfo/lib_tgoto.c:210
#29 0x000055da1b7af96f in windgoto ()
#30 0x000055da1b70baea in preserve_exit ()
--Type <RET> for more, q to quit, c to continue without paging--c
#31 <signal handler called>
#32 __pthread_kill_implementation (threadid=<optimized out>, 
    signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#33 0x00007f6d219deeb3 in __pthread_kill_internal (threadid=<optimized out>, 
    signo=6) at pthread_kill.c:78
#34 0x00007f6d21986a30 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/posix/raise.c:26
#35 0x00007f6d2196e4c3 in __GI_abort () at abort.c:79
#36 0x00007f6d2196f354 in __libc_message_impl (
    fmt=fmt@entry=0x7f6d21af82ea "%s\n") at ../sysdeps/posix/libc_fatal.c:132
#37 0x00007f6d219e9085 in malloc_printerr (
    str=str@entry=0x7f6d21afb828 "malloc(): invalid size (unsorted)")
    at malloc.c:5772
#38 0x00007f6d219ec554 in _int_malloc (
    av=av@entry=0x7f6d21b2cac0 <main_arena>, bytes=106) at malloc.c:4078
#39 0x00007f6d219ed7f2 in __GI___libc_malloc (bytes=<optimized out>)
    at malloc.c:3328
#40 0x000055da1b5caf3e in alloc ()
#41 0x000055da1b73e52a in do_set ()
#42 0x000055da1b67282f in do_cmdline ()
#43 0x000055da1b7b8ef1 in ?? ()
#44 0x000055da1b7b78ab in do_in_path ()
#45 0x000055da1b7b7ba8 in ?? ()
#46 0x000055da1b67282f in do_cmdline ()
#47 0x000055da1b7b8ef1 in ?? ()
#48 0x000055da1b5c82a5 in main ()

Last edited by ht (2024-05-20 19:35:36)

Offline

#5 2024-05-20 23:47:16

ht
Member
Registered: 2024-05-20
Posts: 2

Re: Glibc segfaults on bhyve vm

I narrowed it down to commit aa4249266e9906c4bc833e4847f4d8feef59504f in glibc.

With

_commit=aa4249266e9906c4bc833e4847f4d8feef59504f

in the PKGBUILD, I get crashes left and right. But with the previous commit,

_commit=5a461f2949ded98d8211939f84988bc464c7b4fe

everything is fine. Reverting that specific commit (on top of glibc 2.39+r52+gf8e4623421) also seems to work OK.

The commit in question has something to do with memmove on AMD, which seems to make sense since memmove shows up in the first backtrace and we're both using AMD processors on the affected machines:

x86: Fix Zen3/Zen4 ERMS selection (BZ 30994)

The REP MOVSB usage on memcpy/memmove does not show much performance
improvement on Zen3/Zen4 cores compared to the vectorized loops.  Also,
as from BZ 30994, if the source is aligned and the destination is not
the performance can be 20x slower.

The performance difference is noticeable with small buffer sizes, closer
to the lower bounds limits when memcpy/memmove starts to use ERMS.  The
performance of REP MOVSB is similar to vectorized instruction on the
size limit (the L2 cache).  Also, there is no drawback to multiple cores
sharing the cache.

Checked on x86_64-linux-gnu on Zen3.
Reviewed-by: H.J. Lu <hjl.tools@gmail.com>
(cherry picked from commit 0c0d39fe4aeb0f69b26e76337c5dfd5530d5d44e)

Offline

#6 2024-05-20 23:52:31

loqs
Member
Registered: 2014-03-06
Posts: 17,719

Re: Glibc segfaults on bhyve vm

Please report the issue on Arch's gitlab instance or to upstream glibc on the sourceware bugzilla or raise it on the issue that commit was fixing https://sourceware.org/bugzilla/show_bug.cgi?id=30994.

Last edited by loqs (2024-05-20 23:54:42)

Offline

Board footer

Powered by FluxBB