You are not logged in.
Hey there, my homeserver runs a mariadb, and during the night this mariadb failed.
journalctl gives me the following output:
Jul 20 11:06:26 Hoerspiel systemd[1]: Starting MariaDB 11.0.2 database server...
Jul 20 11:06:26 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:26 0 [Note] Starting MariaDB 11.0.2-MariaDB source revision 0005f2f06c8e1aea4915887decad67885108a929 as process 548742
Jul 20 11:06:26 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:26 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
Jul 20 11:06:26 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:26 0 [Note] InnoDB: Number of transaction pools: 1
Jul 20 11:06:26 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:26 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
Jul 20 11:06:26 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:26 0 [Note] InnoDB: Using liburing
Jul 20 11:06:26 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:26 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
Jul 20 11:06:26 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:26 0 [Note] InnoDB: Completed initialization of buffer pool
Jul 20 11:06:26 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:26 0 [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=7766548647014
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 1 row operations to undo
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: Trx id counter is 2803501968
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: Starting final batch to recover 1024 pages from redo log.
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: Last binlog file './mysql-bin.000007', position 261163103
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Warning] InnoDB: Cannot change innodb_undo_tablespaces=3 because previous shutdown was not with innodb_fast_shutdown=0
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: 128 rollback segments are active.
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: log sequence number 7766576280726; transaction id 2803501969
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] Plugin 'wsrep-provider' is disabled.
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 230720 11:06:27 [ERROR] mysqld got signal 11 ;
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: This could be because you hit a bug. It is also possible that this binary
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: or one of the libraries it was linked against is corrupt, improperly built,
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: or misconfigured. This error can also be caused by malfunctioning hardware.
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: We will try our best to scrape up some info that will hopefully help
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: diagnose the problem, but since we have already crashed,
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: something is definitely wrong and this may fail.
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Server version: 11.0.2-MariaDB source revision: 0005f2f06c8e1aea4915887decad67885108a929
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: key_buffer_size=134217728
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: read_buffer_size=131072
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: max_used_connections=0
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: max_threads=153
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: thread_count=1
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: It is possible that mysqld could use up to
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468023 K bytes of memory
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Hope that's ok; if not, decrease some variables in the equation.
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Thread pointer: 0x0
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Attempting backtrace. You can use the following information to find out
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: where mysqld died. If you see no messages after this, something went
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: terribly wrong...
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: stack_bottom = 0x0 thread_stack 0x49000
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] Server socket created on IP: '0.0.0.0'.
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: 2023-07-20 11:06:27 0 [Note] Server socket created on IP: '::'.
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(my_print_stacktrace+0x31)[0x5639c0dd74c1]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(handle_fatal_signal+0x48a)[0x5639c09b94da]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/lib/libc.so.6(+0x39ab0)[0x7f88f9362ab0]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0xce003b)[0x5639c0c8603b]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0xce1855)[0x5639c0c87855]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0xd699d5)[0x5639c0d0f9d5]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0x5cb9c1)[0x5639c05719c1]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0xd532f2)[0x5639c0cf92f2]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0xd0a0f9)[0x5639c0cb00f9]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0xdc52f1)[0x5639c0d6b2f1]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0xd2c271)[0x5639c0cd2271]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0xce41a6)[0x5639c0c8a1a6]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0x5c5c12)[0x5639c056bc12]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0x5c5f55)[0x5639c056bf55]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(+0x5c60c9)[0x5639c056c0c9]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(_ZN5tpool10task_group7executeEPNS_4taskE+0x97)[0x5639c0d6d967]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/bin/mariadbd(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x7e)[0x5639c0d65e9e]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: bits/unique_ptr.h:99(std::default_delete<std::thread::_State>::operator()(std::thread::_State*) const)[0x7f88f96e1943]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/lib/libc.so.6(+0x8744b)[0x7f88f93b044b]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: /usr/lib/libc.so.6(+0x10ae40)[0x7f88f9433e40]
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: information that should help you find out what is causing the crash.
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Writing a core file...
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Working directory at /var/lib/mysql
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Resource Limits:
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Limit Soft Limit Hard Limit Units
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max cpu time unlimited unlimited seconds
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max file size unlimited unlimited bytes
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max data size unlimited unlimited bytes
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max stack size 8388608 unlimited bytes
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max core file size unlimited unlimited bytes
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max resident set unlimited unlimited bytes
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max processes 95826 95826 processes
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max open files 32768 32768 files
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max locked memory 524288 524288 bytes
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max address space unlimited unlimited bytes
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max file locks unlimited unlimited locks
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max pending signals 95826 95826 signals
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max msgqueue size 819200 819200 bytes
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max nice priority 0 0
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max realtime priority 0 0
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Max realtime timeout unlimited unlimited us
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h
Jul 20 11:06:27 Hoerspiel mariadbd[548742]: Kernel version: Linux version 6.4.3-arch1-2 (linux@archlinux) (gcc (GCC) 13.1.1 20230714, GNU ld (GNU Binutils) 2.40.0) #1 SMP PREEMPT_DYNAMIC Sat, 15 Jul 2023 19:25:49 +0000
Jul 20 11:06:28 Hoerspiel systemd[1]: mariadb.service: Main process exited, code=dumped, status=11/SEGV
Jul 20 11:06:28 Hoerspiel systemd[1]: mariadb.service: Failed with result 'core-dump'.
Jul 20 11:06:28 Hoerspiel systemd[1]: Failed to start MariaDB 11.0.2 database server.As far as I understand, the DB tries to repair itself but get stuck somewhere.
I have daily backups of the DB, but there are some unique data that have not been backuped.
Is there anything I could do to "force" the repair?
Last edited by produnis (2023-07-21 15:14:04)
Offline
In
/etc/my.cnfI added line
innodb_force_recovery = 3After that I was able to start mariadb and do a backup of the unique data of the day. After that, I wiped mariadb, reinstalled and read in the backup.
After that, I removed the innodb_force_recovery = 3 entry.
Now everything works.
Offline