You are not logged in.
Hi, this is my first post here so let it be my introduction. I'm a fairly new arch user, but I've been using Linux (other distros) for a long time. Skip to the bold text below to get to the point
First some introductory info.
Most recently I've used Ubuntu and then Guix, but I wanted to give Arch a try too so I installed the system, I configured it with btrfs root and a btrfs /home (including automatic snapshots triggered by pacman update and scheduled snapshots of home with snapper). I use gnome desktop and nVidia drivers. In general everything works pretty well.
My cpu is AMD ryzen 7 3700x with 32gb of ddr 4 ram and a 64GB swap partition on an nvme drive. The root and home are btrfs on the same nvme drive, there is an extra tmp nvme device that uses ext4 on a separate nvme drive. I also have a couple of 4TB spinning disk's in a mirror as my big file storage. (this will be relevant later)
Everything, but one very significant issue I discovered. Specifically, I often run AI models I download from various places, this time I've been using a 7.1B parameter model called bloom. It is too big to fit on my 8GB rtx card so I run it on the CPU.
This model requires 4*7.1B+some of RAM(definitely more than 32GB), but I only have 32GB. So I've been using an offload folder on the nvme tmp drive and a swap partition on another nvme drive (same as root +home).
This has run fine on my ubuntu 20.04 with Linux kernel 5.13 (and a custom kernel 5.15 with same config) as well as when I booted Ubuntu live cd with 5.15 kernel. Same on Guix with default (nonguix) Linux kernel 5.19. However, it gets stuck on Arch Linux on the current linux (5.19) and linux-lts (5.15) as well as when run from Arch live cd. Additionally I took the kernel config from Ubuntu and I compiled current arch Linux kernel with it. It runs fine on the resulting custom kernel on arch. All tests were done on two same venv's one with python 3. 8 one with 3.10
This leads me to believe the cause is some kernel configuration option. I run a diff and I tried to compare both kernel configs, but they differ so much it difficult to say which option causes it. I have few candidates on my mind, but first let's describe the issue in more detail.
When the model inference runs fine it takes almost exactly 3minutes to load the model from the HDD mirror into ram+swap. It allocates all of ram and few GB of swap (up to ~6GB). Then itt takes 80s~100s to run and it spits its response. During those ~80s there is a brief burst of IO activity at the beginning on the swap drive (about 100k iops 450MB/s lasting ~2s) then disk's do nothing and the rest of inference runs from RAM (cpu utilisation in top goes to 700%).at the end all memory is freed.
When it fails (on Arch with default kernel config) it takes same 3minutes to load the model into RAM+swap, but it seems to allocate more of swap (~10GB) then instead of a brief burst of high IO activity it 'settles' into a very long time when the swap drive sees about 20k IOPS and 150~200MB/s and it stays like that. I once have it 20 min, but it still was stuck the same way. When interrupted in this state it always stops during running of a forward inference pass.
As mentioned all tests were run on exact same virtual environments of python 3. 8 and 3.10(both were tested on all kernels). Prior to using venv I also used conda environment deployed with the same specification (same versions of libraries etc). I've tried replacing my btrfs /home with an ext4 home (just in case). I tried running from a live cd replacing my rootfs.
All this leads me to believe there is some kernel option that causes it, but there are too many to try them one by one. Therefore I'm starting this thread. Perhaps someone who knows the difference between the default and arch kernel config very well reads it and has an idea that it is this option or that. I'm willing to try them to see.
Running large AI models is an increasing use case for a Linux desktop so IMO it is important a distro should support it out of the box. This is why I'm publishing this rather than just running my custom kernel and forgetting about it.
Offline
Buy more RAM, swap is not extra ram
https://ugjka.net
paru > yay | webcord > discord
pacman -S spotify-launcher
mount /dev/disk/by-...
Offline
All this leads me to believe there is some kernel option that causes it, but there are too many to try them one by one.
Indeed there are.
Let's get some data:
cat /proc/vmstat
Please post the full output twice 30s apart on vanilla kernel during an inference pass.
Last edited by sabroad (2022-09-29 11:20:50)
--
saint_abroad
Offline
Also diff'ing
sudo sysctl -a | grep vm
between kernels might provide some insight.
Offline
Thank you for the replies. I can't reboot to the stock kernel until my PC finishes what its doing so I'll run the collection of data late in the evening (CET) or tomorrow.
I wasn't sure if just running those commands 30s apart would catch what's needed so I wrote a script that will execute both commands every second and save the values they give to a CSV file (timestamped). So hopefully this will be much better.
Offline
Well, I run both data captures and I can clearly see that on the custom kernel it works it seems to get to few GB of swap use, then it does its thing and it finishes. With stock kernel however, it appears to diverge at around 2.5~3min mark. It continuously grows swap size larger and larger. I run it for 7 minutes this time while collecting data. There are 223 data points per second (per kernel) so my attempts to plot the differences were not very successful (I did plot them, but with so much data the plots were a mess). I hope this will be of use.
I can't figure out how to attach files on this forum so I'm submitting them via pastebin and file.io - I post both, because pastebin cuts the files at 300s mark (not much happens after, but for the sake of completeness I kept the stuck capture on until 7 min mark).
This is the stock Arch kernel (fails): https://pastebin.com/Uude2vfe https://file.io/gPb1GuosvWE7
This is the custom kernel (works): https://pastebin.com/HSWNpiju https://file.io/ricYUv1fSeb1
Both were captures on a freshly booted system, no X11, no other software running other than iostat, top, screen and the python inference. The CSV files are created by running cat /proc/vmstat and sudo sysctl -A | grep vm, then taking the resulting data and putting it in one row of a CSV file, repeat every 1 second.
Additionally my custom kernel (that works) doesn't contain all of ubuntu's settings. Basically I went through the diff between Arch and ubuntu defaults and I applied desktop preempt configuration as well as anything that based by name could have anything to do with memory or swap.
This is my custom kernel config: https://pastebin.com/ZJien8re https://file.io/ricYUv1fSeb1
I'll continue looking at this data too, but so far I just found symptoms of an issue rather than their cause...
Offline
The vm config won't change but seems equal between kernels and the config uses CRLF… notepad?
Anyway, kernel config diff:
--- config_arch.txt 2022-10-01 09:52:19.014278897 +0200
+++ config_ubuntu.txt 2022-10-01 09:51:57.974197432 +0200
@@ -14,11 +14,7 @@ CONFIG_LLD_VERSION=0
CONFIG_CC_CAN_LINK=y
CONFIG_CC_CAN_LINK_STATIC=y
CONFIG_CC_HAS_ASM_GOTO=y
-CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y
-CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y
CONFIG_CC_HAS_ASM_INLINE=y
-CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y
-CONFIG_PAHOLE_VERSION=123
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_TABLE_SORT=y
CONFIG_THREAD_INFO_IN_TASK=y
@@ -28,7 +24,6 @@ CONFIG_THREAD_INFO_IN_TASK=y
#
CONFIG_INIT_ENV_ARG_LIMIT=32
# CONFIG_COMPILE_TEST is not set
-# CONFIG_WERROR is not set
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_BUILD_SALT=""
@@ -50,7 +45,6 @@ CONFIG_DEFAULT_INIT=""
CONFIG_DEFAULT_HOSTNAME="archlinux"
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
-CONFIG_SYSVIPC_COMPAT=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_WATCH_QUEUE=y
@@ -68,11 +62,9 @@ CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_GENERIC_IRQ_MIGRATION=y
-CONFIG_GENERIC_IRQ_INJECTION=y
CONFIG_HARDIRQS_SW_RESEND=y
CONFIG_GENERIC_IRQ_CHIP=y
CONFIG_IRQ_DOMAIN=y
-CONFIG_IRQ_SIM=y
CONFIG_IRQ_DOMAIN_HIERARCHY=y
CONFIG_GENERIC_MSI_IRQ=y
CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
@@ -101,13 +93,10 @@ CONFIG_POSIX_CPU_TIMERS_TASK_WORK=y
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
-# CONFIG_NO_HZ_IDLE is not set
-CONFIG_NO_HZ_FULL=y
-CONFIG_CONTEXT_TRACKING=y
-# CONFIG_CONTEXT_TRACKING_FORCE is not set
+CONFIG_NO_HZ_IDLE=y
+# CONFIG_NO_HZ_FULL is not set
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
-CONFIG_CLOCKSOURCE_WATCHDOG_MAX_SKEW_US=100
# end of Timers subsystem
CONFIG_BPF=y
@@ -123,19 +112,13 @@ CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_BPF_UNPRIV_DEFAULT_OFF=y
CONFIG_USERMODE_DRIVER=y
-CONFIG_BPF_PRELOAD=y
-CONFIG_BPF_PRELOAD_UMD=m
+#CONFIG_BPF_PRELOAD is not set
CONFIG_BPF_LSM=y
# end of BPF subsystem
-CONFIG_PREEMPT_BUILD=y
# CONFIG_PREEMPT_NONE is not set
-# CONFIG_PREEMPT_VOLUNTARY is not set
-CONFIG_PREEMPT=y
-CONFIG_PREEMPT_COUNT=y
-CONFIG_PREEMPTION=y
-CONFIG_PREEMPT_DYNAMIC=y
-CONFIG_SCHED_CORE=y
+CONFIG_PREEMPT_VOLUNTARY=y
+#CONFIG_PREEMPT is not set
#
# CPU/Task time and stats accounting
@@ -143,7 +126,6 @@ CONFIG_SCHED_CORE=y
CONFIG_VIRT_CPU_ACCOUNTING=y
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_IRQ_TIME_ACCOUNTING=y
-CONFIG_HAVE_SCHED_AVG_IRQ=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
@@ -160,36 +142,22 @@ CONFIG_CPU_ISOLATION=y
# RCU Subsystem
#
CONFIG_TREE_RCU=y
-CONFIG_PREEMPT_RCU=y
-CONFIG_RCU_EXPERT=y
+# CONFIG_RCU_EXPERT is not set
CONFIG_SRCU=y
CONFIG_TREE_SRCU=y
CONFIG_TASKS_RCU_GENERIC=y
-# CONFIG_FORCE_TASKS_RCU is not set
-CONFIG_TASKS_RCU=y
-# CONFIG_FORCE_TASKS_RUDE_RCU is not set
CONFIG_TASKS_RUDE_RCU=y
-# CONFIG_FORCE_TASKS_TRACE_RCU is not set
CONFIG_TASKS_TRACE_RCU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_NEED_SEGCBLIST=y
-CONFIG_RCU_FANOUT=64
-CONFIG_RCU_FANOUT_LEAF=16
-CONFIG_RCU_BOOST=y
-CONFIG_RCU_BOOST_DELAY=500
-# CONFIG_RCU_EXP_KTHREAD is not set
-CONFIG_RCU_NOCB_CPU=y
-# CONFIG_TASKS_TRACE_RCU_READ_MB is not set
# end of RCU Subsystem
CONFIG_BUILD_BIN2C=y
CONFIG_IKCONFIG=y
-CONFIG_IKCONFIG_PROC=y
CONFIG_IKHEADERS=m
CONFIG_LOG_BUF_SHIFT=17
CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13
-CONFIG_PRINTK_INDEX=y
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
#
@@ -202,9 +170,6 @@ CONFIG_UCLAMP_BUCKETS_COUNT=5
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y
CONFIG_CC_HAS_INT128=y
-CONFIG_CC_IMPLICIT_FALLTHROUGH="-Wimplicit-fallthrough=5"
-CONFIG_GCC12_NO_ARRAY_BOUNDS=y
-CONFIG_CC_NO_ARRAY_BOUNDS=y
CONFIG_ARCH_SUPPORTS_INT128=y
CONFIG_NUMA_BALANCING=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
@@ -238,7 +203,6 @@ CONFIG_UTS_NS=y
CONFIG_TIME_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
-CONFIG_USER_NS_UNPRIVILEGED=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_CHECKPOINT_RESTORE=y
@@ -255,8 +219,6 @@ CONFIG_RD_LZO=y
CONFIG_RD_LZ4=y
CONFIG_RD_ZSTD=y
CONFIG_BOOT_CONFIG=y
-# CONFIG_BOOT_CONFIG_EMBED is not set
-CONFIG_INITRAMFS_PRESERVE_MTIME=y
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_LD_ORPHAN_WARN=y
@@ -272,6 +234,7 @@ CONFIG_SYSFS_SYSCALL=y
CONFIG_FHANDLE=y
CONFIG_POSIX_TIMERS=y
CONFIG_PRINTK=y
+CONFIG_PRINTK_NMI=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
@@ -286,14 +249,18 @@ CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_IO_URING=y
CONFIG_ADVISE_SYSCALLS=y
+CONFIG_HAVE_ARCH_USERFAULTFD_WP=y
+CONFIG_HAVE_ARCH_USERFAULTFD_MINOR=y
CONFIG_MEMBARRIER=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
+CONFIG_USERFAULTFD=y
CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE=y
CONFIG_KCMP=y
CONFIG_RSEQ=y
+# CONFIG_DEBUG_RSEQ is not set
# CONFIG_EMBEDDED is not set
CONFIG_HAVE_PERF_EVENTS=y
CONFIG_GUEST_PERF_EVENTS=y
@@ -305,6 +272,17 @@ CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
# end of Kernel Performance Events And Counters
+CONFIG_VM_EVENT_COUNTERS=y
+CONFIG_SLUB_DEBUG=y
+# CONFIG_COMPAT_BRK is not set
+# CONFIG_SLAB is not set
+CONFIG_SLUB=y
+# CONFIG_SLOB is not set
+CONFIG_SLAB_MERGE_DEFAULT=y
+CONFIG_SLAB_FREELIST_RANDOM=y
+CONFIG_SLAB_FREELIST_HARDENED=y
+CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
+CONFIG_SLUB_CPU_PARTIAL=y
CONFIG_SYSTEM_DATA_VERIFICATION=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
@@ -328,9 +306,14 @@ CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
+CONFIG_ARCH_HAS_FILTER_PGPROT=y
+CONFIG_HAVE_SETUP_PER_CPU_AREA=y
+CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
+CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
-CONFIG_ARCH_NR_GPIO=1024
CONFIG_ARCH_SUSPEND_POSSIBLE=y
+CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
+CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_X86_64_SMP=y
@@ -343,13 +326,20 @@ CONFIG_CC_HAS_SANE_STACKPROTECTOR=y
#
# Processor type and features
#
+CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_FEATURE_NAMES=y
CONFIG_X86_X2APIC=y
CONFIG_X86_MPPARSE=y
# CONFIG_GOLDFISH is not set
+CONFIG_RETPOLINE=y
CONFIG_X86_CPU_RESCTRL=y
-# CONFIG_X86_EXTENDED_PLATFORM is not set
+CONFIG_X86_EXTENDED_PLATFORM=y
+CONFIG_X86_NUMACHIP=y
+# CONFIG_X86_VSMP is not set
+CONFIG_X86_UV=y
+# CONFIG_X86_GOLDFISH is not set
+# CONFIG_X86_INTEL_MID is not set
CONFIG_X86_INTEL_LPSS=y
CONFIG_X86_AMD_PLATFORM_DEVICE=y
CONFIG_IOSF_MBI=y
@@ -373,7 +363,6 @@ CONFIG_XEN_PVHVM_GUEST=y
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
CONFIG_XEN_PVH=y
-CONFIG_XEN_DOM0=y
CONFIG_KVM_GUEST=y
CONFIG_ARCH_CPUIDLE_HALTPOLL=y
CONFIG_PVH=y
@@ -381,7 +370,6 @@ CONFIG_PARAVIRT_TIME_ACCOUNTING=y
CONFIG_PARAVIRT_CLOCK=y
CONFIG_JAILHOUSE_GUEST=y
CONFIG_ACRN_GUEST=y
-CONFIG_INTEL_TDX_GUEST=y
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
@@ -396,6 +384,7 @@ CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_IA32_FEAT_CTL=y
CONFIG_X86_VMX_FEATURE_NAMES=y
+CONFIG_PROCESSOR_SELECT=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_HYGON=y
@@ -404,14 +393,13 @@ CONFIG_CPU_SUP_ZHAOXIN=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
-# CONFIG_GART_IOMMU is not set
+CONFIG_GART_IOMMU=y
CONFIG_BOOT_VESA_SUPPORT=y
-# CONFIG_MAXSMP is not set
-CONFIG_NR_CPUS_RANGE_BEGIN=2
-CONFIG_NR_CPUS_RANGE_END=512
-CONFIG_NR_CPUS_DEFAULT=64
-CONFIG_NR_CPUS=320
-CONFIG_SCHED_CLUSTER=y
+CONFIG_MAXSMP=y
+CONFIG_NR_CPUS_RANGE_BEGIN=8192
+CONFIG_NR_CPUS_RANGE_END=8192
+CONFIG_NR_CPUS_DEFAULT=8192
+CONFIG_NR_CPUS=8192
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_SCHED_MC_PRIO=y
@@ -440,6 +428,7 @@ CONFIG_X86_16BIT=y
CONFIG_X86_ESPFIX64=y
CONFIG_X86_VSYSCALL_EMULATION=y
CONFIG_X86_IOPL_IOPERM=y
+CONFIG_I8K=m
CONFIG_MICROCODE=y
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
@@ -448,8 +437,7 @@ CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_X86_5LEVEL=y
CONFIG_X86_DIRECT_GBPAGES=y
-CONFIG_X86_CPA_STATISTICS=y
-CONFIG_X86_MEM_ENCRYPT=y
+# CONFIG_X86_CPA_STATISTICS is not set
CONFIG_AMD_MEM_ENCRYPT=y
# CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT is not set
CONFIG_NUMA=y
@@ -459,13 +447,15 @@ CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SHIFT=5
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
-# CONFIG_ARCH_MEMORY_PROBE is not set
+CONFIG_ARCH_SELECT_MEMORY_MODEL=y
+CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_X86_PMEM_LEGACY_DEVICE=y
CONFIG_X86_PMEM_LEGACY=m
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
+CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
@@ -473,9 +463,8 @@ CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAU
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_ARCH_RANDOM=y
+CONFIG_X86_SMAP=y
CONFIG_X86_UMIP=y
-CONFIG_CC_HAS_IBT=y
-CONFIG_X86_KERNEL_IBT=y
CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=y
# CONFIG_X86_INTEL_TSX_MODE_OFF is not set
# CONFIG_X86_INTEL_TSX_MODE_ON is not set
@@ -485,8 +474,8 @@ CONFIG_EFI=y
CONFIG_EFI_STUB=y
CONFIG_EFI_MIXED=y
# CONFIG_HZ_100 is not set
-# CONFIG_HZ_250 is not set
-CONFIG_HZ_300=y
+CONFIG_HZ_250=y
+# CONFIG_HZ_300 uis not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=300
CONFIG_SCHED_HRTICK=y
@@ -510,27 +499,18 @@ CONFIG_HOTPLUG_CPU=y
# CONFIG_BOOTPARAM_HOTPLUG_CPU0 is not set
# CONFIG_DEBUG_HOTPLUG_CPU0 is not set
# CONFIG_COMPAT_VDSO is not set
+# CONFIG_LEGACY_VSYSCALL_EMULATE is not set
CONFIG_LEGACY_VSYSCALL_XONLY=y
# CONFIG_LEGACY_VSYSCALL_NONE is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_MODIFY_LDT_SYSCALL=y
-# CONFIG_STRICT_SIGALTSTACK_SIZE is not set
CONFIG_HAVE_LIVEPATCH=y
# CONFIG_LIVEPATCH is not set
# end of Processor type and features
-CONFIG_CC_HAS_SLS=y
-CONFIG_CC_HAS_RETURN_THUNK=y
-CONFIG_SPECULATION_MITIGATIONS=y
-CONFIG_PAGE_TABLE_ISOLATION=y
-CONFIG_RETPOLINE=y
-CONFIG_RETHUNK=y
-CONFIG_CPU_UNRET_ENTRY=y
-CONFIG_CPU_IBPB_ENTRY=y
-CONFIG_CPU_IBRS_ENTRY=y
-CONFIG_SLS=y
CONFIG_ARCH_HAS_ADD_PAGES=y
CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=y
+CONFIG_USE_PERCPU_NUMA_NODE_ID=y
#
# Power management and ACPI options
@@ -538,6 +518,7 @@ CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=
CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
+# CONFIG_SUSPEND_SKIP_SYNC is not set
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_HIBERNATION_SNAPSHOT_DEV=y
@@ -551,6 +532,7 @@ CONFIG_PM_DEBUG=y
# CONFIG_PM_ADVANCED_DEBUG is not set
# CONFIG_PM_TEST_SUSPEND is not set
CONFIG_PM_SLEEP_DEBUG=y
+# CONFIG_DPM_WATCHDOG is not set
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_PM_CLK=y
@@ -588,6 +570,7 @@ CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_PLATFORM_PROFILE=m
+CONFIG_ACPI_CUSTOM_DSDT_FILE=""
CONFIG_ARCH_HAS_ACPI_TABLE_UPGRADE=y
CONFIG_ACPI_TABLE_UPGRADE=y
CONFIG_ACPI_DEBUG=y
@@ -599,6 +582,7 @@ CONFIG_ACPI_SBS=m
CONFIG_ACPI_HED=y
CONFIG_ACPI_CUSTOM_METHOD=m
CONFIG_ACPI_BGRT=y
+# CONFIG_ACPI_REDUCED_HARDWARE_ONLY is not set
CONFIG_ACPI_NFIT=m
# CONFIG_NFIT_SECURITY_DEBUG is not set
CONFIG_ACPI_NUMA=y
@@ -654,9 +638,9 @@ CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
# CPU frequency scaling drivers
#
CONFIG_X86_INTEL_PSTATE=y
-CONFIG_X86_PCC_CPUFREQ=m
+CONFIG_X86_PCC_CPUFREQ=y
CONFIG_X86_AMD_PSTATE=m
-CONFIG_X86_ACPI_CPUFREQ=m
+CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_ACPI_CPUFREQ_CPB=y
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_AMD_FREQ_SENSITIVITY=m
@@ -724,6 +708,7 @@ CONFIG_KVM_XFER_TO_GUEST_WORK=y
CONFIG_HAVE_KVM_PM_NOTIFIER=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
+CONFIG_KVM_WERROR=y
CONFIG_KVM_INTEL=m
CONFIG_X86_SGX_KVM=y
CONFIG_KVM_AMD=m
@@ -752,18 +737,14 @@ CONFIG_UPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_KRETPROBES=y
-CONFIG_KRETPROBE_ON_RETHOOK=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
-CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=y
CONFIG_HAVE_FUNCTION_ERROR_INJECTION=y
CONFIG_HAVE_NMI=y
-CONFIG_TRACE_IRQFLAGS_SUPPORT=y
-CONFIG_TRACE_IRQFLAGS_NMI_SUPPORT=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_CONTIGUOUS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
@@ -772,7 +753,6 @@ CONFIG_ARCH_HAS_SET_MEMORY=y
CONFIG_ARCH_HAS_SET_DIRECT_MAP=y
CONFIG_HAVE_ARCH_THREAD_STRUCT_WHITELIST=y
CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y
-CONFIG_ARCH_WANTS_NO_INSTR=y
CONFIG_HAVE_ASM_MODVERSIONS=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_RSEQ=y
@@ -788,7 +768,6 @@ CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE=y
CONFIG_MMU_GATHER_TABLE_FREE=y
CONFIG_MMU_GATHER_RCU_TABLE_FREE=y
-CONFIG_MMU_GATHER_MERGE_VMAS=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
@@ -817,7 +796,6 @@ CONFIG_HAVE_MOVE_PMD=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=y
CONFIG_HAVE_ARCH_HUGE_VMAP=y
-CONFIG_HAVE_ARCH_HUGE_VMALLOC=y
CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
@@ -831,13 +809,6 @@ CONFIG_ARCH_MMAP_RND_BITS=28
CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS=y
CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES=y
-CONFIG_PAGE_SIZE_LESS_THAN_64KB=y
-CONFIG_PAGE_SIZE_LESS_THAN_256KB=y
-CONFIG_HAVE_OBJTOOL=y
-CONFIG_HAVE_JUMP_LABEL_HACK=y
-CONFIG_HAVE_NOINSTR_HACK=y
-CONFIG_HAVE_NOINSTR_VALIDATION=y
-CONFIG_HAVE_UACCESS_VALIDATION=y
CONFIG_HAVE_STACK_VALIDATION=y
CONFIG_HAVE_RELIABLE_STACKTRACE=y
CONFIG_ISA_BUS_API=y
@@ -847,7 +818,6 @@ CONFIG_COMPAT_32BIT_TIME=y
CONFIG_HAVE_ARCH_VMAP_STACK=y
CONFIG_VMAP_STACK=y
CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET=y
-CONFIG_RANDOMIZE_KSTACK_OFFSET=y
CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT=y
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
CONFIG_STRICT_KERNEL_RWX=y
@@ -861,14 +831,9 @@ CONFIG_ARCH_HAS_CC_PLATFORM=y
CONFIG_HAVE_STATIC_CALL=y
CONFIG_HAVE_STATIC_CALL_INLINE=y
CONFIG_HAVE_PREEMPT_DYNAMIC=y
-CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y
CONFIG_ARCH_WANT_LD_ORPHAN_WARN=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
-CONFIG_ARCH_SUPPORTS_PAGE_TABLE_CHECK=y
CONFIG_ARCH_HAS_ELFCORE_COMPAT=y
-CONFIG_ARCH_HAS_PARANOID_L1D_FLUSH=y
-CONFIG_DYNAMIC_SIGFRAME=y
-CONFIG_HAVE_ARCH_NODE_DEV_GROUP=y
#
# GCOV-based kernel profiling
@@ -878,8 +843,6 @@ CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
# end of GCOV-based kernel profiling
CONFIG_HAVE_GCC_PLUGINS=y
-CONFIG_GCC_PLUGINS=y
-# CONFIG_GCC_PLUGIN_LATENT_ENTROPY is not set
# end of General architecture-dependent options
CONFIG_RT_MUTEXES=y
@@ -907,23 +870,22 @@ CONFIG_MODULE_SIG_HASH="sha512"
CONFIG_MODULE_COMPRESS_ZSTD=y
CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS=y
CONFIG_MODPROBE_PATH="/sbin/modprobe"
+# CONFIG_TRIM_UNUSED_KSYMS is not set
CONFIG_MODULES_TREE_LOOKUP=y
CONFIG_BLOCK=y
-CONFIG_BLOCK_LEGACY_AUTOLOAD=y
CONFIG_BLK_RQ_ALLOC_TIME=y
+CONFIG_BLK_SCSI_REQUEST=y
CONFIG_BLK_CGROUP_RWSTAT=y
-CONFIG_BLK_DEV_BSG_COMMON=y
-CONFIG_BLK_ICQ=y
+CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_BSGLIB=y
CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_BLK_DEV_INTEGRITY_T10=y
CONFIG_BLK_DEV_ZONED=y
CONFIG_BLK_DEV_THROTTLING=y
-CONFIG_BLK_DEV_THROTTLING_LOW=y
+# CONFIG_BLK_DEV_THROTTLING_LOW is not set
+CONFIG_BLK_CMDLINE_PARSER=y
CONFIG_BLK_WBT=y
-CONFIG_BLK_WBT_MQ=y
-CONFIG_BLK_CGROUP_IOLATENCY=y
-CONFIG_BLK_CGROUP_FC_APPID=y
+# CONFIG_BLK_CGROUP_IOLATENCY is not set
CONFIG_BLK_CGROUP_IOCOST=y
CONFIG_BLK_CGROUP_IOPRIO=y
CONFIG_BLK_DEBUG_FS=y
@@ -980,6 +942,10 @@ CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
CONFIG_ASN1=y
CONFIG_UNINLINE_SPIN_UNLOCK=y
+CONFIG_INLINE_READ_UNLOCK=y
+CONFIG_INLINE_READ_UNLOCK_IRQ=y
+CONFIG_INLINE_WRITE_UNLOCK=y
+CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_RWSEM_SPIN_ON_OWNER=y
@@ -1008,53 +974,22 @@ CONFIG_COREDUMP=y
#
# Memory Management options
#
-CONFIG_ZPOOL=y
-CONFIG_SWAP=y
-CONFIG_ZSWAP=y
-CONFIG_ZSWAP_DEFAULT_ON=y
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO is not set
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set
-CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4=y
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set
-CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lz4"
-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD is not set
-CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD=y
-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
-CONFIG_ZSWAP_ZPOOL_DEFAULT="z3fold"
-CONFIG_ZBUD=y
-CONFIG_Z3FOLD=y
-CONFIG_ZSMALLOC=y
-# CONFIG_ZSMALLOC_STAT is not set
-
-#
-# SLAB allocator options
-#
-# CONFIG_SLAB is not set
-CONFIG_SLUB=y
-CONFIG_SLAB_MERGE_DEFAULT=y
-CONFIG_SLAB_FREELIST_RANDOM=y
-CONFIG_SLAB_FREELIST_HARDENED=y
-# CONFIG_SLUB_STATS is not set
-CONFIG_SLUB_CPU_PARTIAL=y
-# end of SLAB allocator options
-
-CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
-# CONFIG_COMPAT_BRK is not set
+CONFIG_SELECT_MEMORY_MODEL=y
+CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
+CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_FAST_GUP=y
CONFIG_NUMA_KEEP_MEMINFO=y
CONFIG_MEMORY_ISOLATION=y
-CONFIG_EXCLUSIVE_SYSTEM_RAM=y
CONFIG_HAVE_BOOTMEM_INFO_NODE=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
-CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_MEMORY_HOTPLUG=y
+CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
+CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_MHP_MEMMAP_ON_MEMORY=y
CONFIG_SPLIT_PTLOCK_CPUS=4
@@ -1064,7 +999,6 @@ CONFIG_BALLOON_COMPACTION=y
CONFIG_COMPACTION=y
CONFIG_PAGE_REPORTING=y
CONFIG_MIGRATION=y
-CONFIG_DEVICE_MIGRATION=y
CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=y
CONFIG_ARCH_ENABLE_THP_MIGRATION=y
CONFIG_CONTIG_ALLOC=y
@@ -1076,66 +1010,54 @@ CONFIG_DEFAULT_MMAP_MIN_ADDR=65536
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_MEMORY_FAILURE=y
CONFIG_HWPOISON_INJECT=m
-CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
-CONFIG_ARCH_WANTS_THP_SWAP=y
CONFIG_TRANSPARENT_HUGEPAGE=y
-CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
-# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
+# CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is not set
+CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y
+CONFIG_ARCH_WANTS_THP_SWAP=y
CONFIG_THP_SWAP=y
-CONFIG_READ_ONLY_THP_FOR_FS=y
-CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
-CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
-CONFIG_USE_PERCPU_NUMA_NODE_ID=y
-CONFIG_HAVE_SETUP_PER_CPU_AREA=y
+CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
-CONFIG_CMA=y
-# CONFIG_CMA_DEBUG is not set
-CONFIG_CMA_DEBUGFS=y
-CONFIG_CMA_SYSFS=y
-CONFIG_CMA_AREAS=7
+# CONFIG_CMA is not set
CONFIG_MEM_SOFT_DIRTY=y
+CONFIG_ZSWAP=y
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo"
+CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
+CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud"
+# CONFIG_ZSWAP_DEFAULT_ON is not set
+CONFIG_ZPOOL=y
+CONFIG_ZBUD=y
+CONFIG_Z3FOLD=m
+CONFIG_ZSMALLOC=y
+# CONFIG_ZSMALLOC_STAT is not set
CONFIG_GENERIC_EARLY_IOREMAP=y
# CONFIG_DEFERRED_STRUCT_PAGE_INIT is not set
-CONFIG_PAGE_IDLE_FLAG=y
CONFIG_IDLE_PAGE_TRACKING=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
-CONFIG_ARCH_HAS_CURRENT_STACK_POINTER=y
-CONFIG_ARCH_HAS_VM_GET_PAGE_PROT=y
CONFIG_ARCH_HAS_PTE_DEVMAP=y
-CONFIG_ZONE_DMA=y
-CONFIG_ZONE_DMA32=y
CONFIG_ZONE_DEVICE=y
+CONFIG_DEV_PAGEMAP_OPS=y
CONFIG_HMM_MIRROR=y
CONFIG_DEVICE_PRIVATE=y
CONFIG_VMAP_PFN=y
CONFIG_ARCH_USES_HIGH_VMA_FLAGS=y
CONFIG_ARCH_HAS_PKEYS=y
-CONFIG_VM_EVENT_COUNTERS=y
# CONFIG_PERCPU_STATS is not set
# CONFIG_GUP_TEST is not set
+# CONFIG_READ_ONLY_THP_FOR_FS is not set
CONFIG_ARCH_HAS_PTE_SPECIAL=y
CONFIG_MAPPING_DIRTY_HELPERS=y
-CONFIG_SECRETMEM=y
-CONFIG_ANON_VMA_NAME=y
-CONFIG_USERFAULTFD=y
-CONFIG_HAVE_ARCH_USERFAULTFD_WP=y
-CONFIG_HAVE_ARCH_USERFAULTFD_MINOR=y
-CONFIG_PTE_MARKER=y
-CONFIG_PTE_MARKER_UFFD_WP=y
-
-#
-# Data Access Monitoring
-#
-CONFIG_DAMON=y
-CONFIG_DAMON_VADDR=y
-CONFIG_DAMON_PADDR=y
-CONFIG_DAMON_SYSFS=y
-CONFIG_DAMON_DBGFS=y
-CONFIG_DAMON_RECLAIM=y
-# end of Data Access Monitoring
# end of Memory Management options
CONFIG_NET=y
+CONFIG_WANT_COMPAT_NETLINK_MESSAGES=y
CONFIG_COMPAT_NETLINK_MESSAGES=y
CONFIG_NET_INGRESS=y
CONFIG_NET_EGRESS=y
Edit: possibly very relevant
-CONFIG_ZSWAP_DEFAULT_ON=y
+# CONFIG_ZSWAP_DEFAULT_ON is not set
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO is not set
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y
-CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4=y
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set
-CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lz4"
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo"
-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD is not set
+CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y
-CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD=y
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set
-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
-CONFIG_ZSWAP_ZPOOL_DEFAULT="z3fold"
+CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud"
Edit #2
-CONFIG_SWAP=y
+CONFIG_VM_EVENT_COUNTERS=y
-CONFIG_HAVE_ARCH_HUGE_VMALLOC=y
-CONFIG_ARCH_HAS_VM_GET_PAGE_PROT=y
-CONFIG_VM_EVENT_COUNTERS=y
-CONFIG_ANON_VMA_NAME=y
Last edited by seth (2022-10-01 08:04:39)
Offline
The vm config won't change but seems equal between kernels and the config uses CRLF… notepad?
Anyway, kernel config diff:--- config_arch.txt 2022-10-01 09:52:19.014278897 +0200 +++ config_ubuntu.txt 2022-10-01 09:51:57.974197432 +0200 @@ -14,11 +14,7 @@ CONFIG_LLD_VERSION=0 CONFIG_CC_CAN_LINK=y CONFIG_CC_CAN_LINK_STATIC=y CONFIG_CC_HAS_ASM_GOTO=y -CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y -CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y CONFIG_CC_HAS_ASM_INLINE=y -CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y -CONFIG_PAHOLE_VERSION=123 CONFIG_IRQ_WORK=y CONFIG_BUILDTIME_TABLE_SORT=y CONFIG_THREAD_INFO_IN_TASK=y @@ -28,7 +24,6 @@ CONFIG_THREAD_INFO_IN_TASK=y # CONFIG_INIT_ENV_ARG_LIMIT=32 # CONFIG_COMPILE_TEST is not set -# CONFIG_WERROR is not set CONFIG_LOCALVERSION="" CONFIG_LOCALVERSION_AUTO=y CONFIG_BUILD_SALT="" @@ -50,7 +45,6 @@ CONFIG_DEFAULT_INIT="" CONFIG_DEFAULT_HOSTNAME="archlinux" CONFIG_SYSVIPC=y CONFIG_SYSVIPC_SYSCTL=y -CONFIG_SYSVIPC_COMPAT=y CONFIG_POSIX_MQUEUE=y CONFIG_POSIX_MQUEUE_SYSCTL=y CONFIG_WATCH_QUEUE=y @@ -68,11 +62,9 @@ CONFIG_GENERIC_IRQ_SHOW=y CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK=y CONFIG_GENERIC_PENDING_IRQ=y CONFIG_GENERIC_IRQ_MIGRATION=y -CONFIG_GENERIC_IRQ_INJECTION=y CONFIG_HARDIRQS_SW_RESEND=y CONFIG_GENERIC_IRQ_CHIP=y CONFIG_IRQ_DOMAIN=y -CONFIG_IRQ_SIM=y CONFIG_IRQ_DOMAIN_HIERARCHY=y CONFIG_GENERIC_MSI_IRQ=y CONFIG_GENERIC_MSI_IRQ_DOMAIN=y @@ -101,13 +93,10 @@ CONFIG_POSIX_CPU_TIMERS_TASK_WORK=y CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ_COMMON=y # CONFIG_HZ_PERIODIC is not set -# CONFIG_NO_HZ_IDLE is not set -CONFIG_NO_HZ_FULL=y -CONFIG_CONTEXT_TRACKING=y -# CONFIG_CONTEXT_TRACKING_FORCE is not set +CONFIG_NO_HZ_IDLE=y +# CONFIG_NO_HZ_FULL is not set CONFIG_NO_HZ=y CONFIG_HIGH_RES_TIMERS=y -CONFIG_CLOCKSOURCE_WATCHDOG_MAX_SKEW_US=100 # end of Timers subsystem CONFIG_BPF=y @@ -123,19 +112,13 @@ CONFIG_BPF_JIT_ALWAYS_ON=y CONFIG_BPF_JIT_DEFAULT_ON=y CONFIG_BPF_UNPRIV_DEFAULT_OFF=y CONFIG_USERMODE_DRIVER=y -CONFIG_BPF_PRELOAD=y -CONFIG_BPF_PRELOAD_UMD=m +#CONFIG_BPF_PRELOAD is not set CONFIG_BPF_LSM=y # end of BPF subsystem -CONFIG_PREEMPT_BUILD=y # CONFIG_PREEMPT_NONE is not set -# CONFIG_PREEMPT_VOLUNTARY is not set -CONFIG_PREEMPT=y -CONFIG_PREEMPT_COUNT=y -CONFIG_PREEMPTION=y -CONFIG_PREEMPT_DYNAMIC=y -CONFIG_SCHED_CORE=y +CONFIG_PREEMPT_VOLUNTARY=y +#CONFIG_PREEMPT is not set # # CPU/Task time and stats accounting @@ -143,7 +126,6 @@ CONFIG_SCHED_CORE=y CONFIG_VIRT_CPU_ACCOUNTING=y CONFIG_VIRT_CPU_ACCOUNTING_GEN=y CONFIG_IRQ_TIME_ACCOUNTING=y -CONFIG_HAVE_SCHED_AVG_IRQ=y CONFIG_BSD_PROCESS_ACCT=y CONFIG_BSD_PROCESS_ACCT_V3=y CONFIG_TASKSTATS=y @@ -160,36 +142,22 @@ CONFIG_CPU_ISOLATION=y # RCU Subsystem # CONFIG_TREE_RCU=y -CONFIG_PREEMPT_RCU=y -CONFIG_RCU_EXPERT=y +# CONFIG_RCU_EXPERT is not set CONFIG_SRCU=y CONFIG_TREE_SRCU=y CONFIG_TASKS_RCU_GENERIC=y -# CONFIG_FORCE_TASKS_RCU is not set -CONFIG_TASKS_RCU=y -# CONFIG_FORCE_TASKS_RUDE_RCU is not set CONFIG_TASKS_RUDE_RCU=y -# CONFIG_FORCE_TASKS_TRACE_RCU is not set CONFIG_TASKS_TRACE_RCU=y CONFIG_RCU_STALL_COMMON=y CONFIG_RCU_NEED_SEGCBLIST=y -CONFIG_RCU_FANOUT=64 -CONFIG_RCU_FANOUT_LEAF=16 -CONFIG_RCU_BOOST=y -CONFIG_RCU_BOOST_DELAY=500 -# CONFIG_RCU_EXP_KTHREAD is not set -CONFIG_RCU_NOCB_CPU=y -# CONFIG_TASKS_TRACE_RCU_READ_MB is not set # end of RCU Subsystem CONFIG_BUILD_BIN2C=y CONFIG_IKCONFIG=y -CONFIG_IKCONFIG_PROC=y CONFIG_IKHEADERS=m CONFIG_LOG_BUF_SHIFT=17 CONFIG_LOG_CPU_MAX_BUF_SHIFT=12 CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13 -CONFIG_PRINTK_INDEX=y CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y # @@ -202,9 +170,6 @@ CONFIG_UCLAMP_BUCKETS_COUNT=5 CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y CONFIG_CC_HAS_INT128=y -CONFIG_CC_IMPLICIT_FALLTHROUGH="-Wimplicit-fallthrough=5" -CONFIG_GCC12_NO_ARRAY_BOUNDS=y -CONFIG_CC_NO_ARRAY_BOUNDS=y CONFIG_ARCH_SUPPORTS_INT128=y CONFIG_NUMA_BALANCING=y CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y @@ -238,7 +203,6 @@ CONFIG_UTS_NS=y CONFIG_TIME_NS=y CONFIG_IPC_NS=y CONFIG_USER_NS=y -CONFIG_USER_NS_UNPRIVILEGED=y CONFIG_PID_NS=y CONFIG_NET_NS=y CONFIG_CHECKPOINT_RESTORE=y @@ -255,8 +219,6 @@ CONFIG_RD_LZO=y CONFIG_RD_LZ4=y CONFIG_RD_ZSTD=y CONFIG_BOOT_CONFIG=y -# CONFIG_BOOT_CONFIG_EMBED is not set -CONFIG_INITRAMFS_PRESERVE_MTIME=y CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y # CONFIG_CC_OPTIMIZE_FOR_SIZE is not set CONFIG_LD_ORPHAN_WARN=y @@ -272,6 +234,7 @@ CONFIG_SYSFS_SYSCALL=y CONFIG_FHANDLE=y CONFIG_POSIX_TIMERS=y CONFIG_PRINTK=y +CONFIG_PRINTK_NMI=y CONFIG_BUG=y CONFIG_ELF_CORE=y CONFIG_PCSPKR_PLATFORM=y @@ -286,14 +249,18 @@ CONFIG_SHMEM=y CONFIG_AIO=y CONFIG_IO_URING=y CONFIG_ADVISE_SYSCALLS=y +CONFIG_HAVE_ARCH_USERFAULTFD_WP=y +CONFIG_HAVE_ARCH_USERFAULTFD_MINOR=y CONFIG_MEMBARRIER=y CONFIG_KALLSYMS=y CONFIG_KALLSYMS_ALL=y CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y CONFIG_KALLSYMS_BASE_RELATIVE=y +CONFIG_USERFAULTFD=y CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE=y CONFIG_KCMP=y CONFIG_RSEQ=y +# CONFIG_DEBUG_RSEQ is not set # CONFIG_EMBEDDED is not set CONFIG_HAVE_PERF_EVENTS=y CONFIG_GUEST_PERF_EVENTS=y @@ -305,6 +272,17 @@ CONFIG_PERF_EVENTS=y # CONFIG_DEBUG_PERF_USE_VMALLOC is not set # end of Kernel Performance Events And Counters +CONFIG_VM_EVENT_COUNTERS=y +CONFIG_SLUB_DEBUG=y +# CONFIG_COMPAT_BRK is not set +# CONFIG_SLAB is not set +CONFIG_SLUB=y +# CONFIG_SLOB is not set +CONFIG_SLAB_MERGE_DEFAULT=y +CONFIG_SLAB_FREELIST_RANDOM=y +CONFIG_SLAB_FREELIST_HARDENED=y +CONFIG_SHUFFLE_PAGE_ALLOCATOR=y +CONFIG_SLUB_CPU_PARTIAL=y CONFIG_SYSTEM_DATA_VERIFICATION=y CONFIG_PROFILING=y CONFIG_TRACEPOINTS=y @@ -328,9 +306,14 @@ CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_ARCH_HAS_CPU_RELAX=y +CONFIG_ARCH_HAS_FILTER_PGPROT=y +CONFIG_HAVE_SETUP_PER_CPU_AREA=y +CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y +CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y CONFIG_ARCH_HIBERNATION_POSSIBLE=y -CONFIG_ARCH_NR_GPIO=1024 CONFIG_ARCH_SUSPEND_POSSIBLE=y +CONFIG_ARCH_WANT_GENERAL_HUGETLB=y +CONFIG_ZONE_DMA32=y CONFIG_AUDIT_ARCH=y CONFIG_HAVE_INTEL_TXT=y CONFIG_X86_64_SMP=y @@ -343,13 +326,20 @@ CONFIG_CC_HAS_SANE_STACKPROTECTOR=y # # Processor type and features # +CONFIG_ZONE_DMA=y CONFIG_SMP=y CONFIG_X86_FEATURE_NAMES=y CONFIG_X86_X2APIC=y CONFIG_X86_MPPARSE=y # CONFIG_GOLDFISH is not set +CONFIG_RETPOLINE=y CONFIG_X86_CPU_RESCTRL=y -# CONFIG_X86_EXTENDED_PLATFORM is not set +CONFIG_X86_EXTENDED_PLATFORM=y +CONFIG_X86_NUMACHIP=y +# CONFIG_X86_VSMP is not set +CONFIG_X86_UV=y +# CONFIG_X86_GOLDFISH is not set +# CONFIG_X86_INTEL_MID is not set CONFIG_X86_INTEL_LPSS=y CONFIG_X86_AMD_PLATFORM_DEVICE=y CONFIG_IOSF_MBI=y @@ -373,7 +363,6 @@ CONFIG_XEN_PVHVM_GUEST=y CONFIG_XEN_SAVE_RESTORE=y # CONFIG_XEN_DEBUG_FS is not set CONFIG_XEN_PVH=y -CONFIG_XEN_DOM0=y CONFIG_KVM_GUEST=y CONFIG_ARCH_CPUIDLE_HALTPOLL=y CONFIG_PVH=y @@ -381,7 +370,6 @@ CONFIG_PARAVIRT_TIME_ACCOUNTING=y CONFIG_PARAVIRT_CLOCK=y CONFIG_JAILHOUSE_GUEST=y CONFIG_ACRN_GUEST=y -CONFIG_INTEL_TDX_GUEST=y # CONFIG_MK8 is not set # CONFIG_MPSC is not set # CONFIG_MCORE2 is not set @@ -396,6 +384,7 @@ CONFIG_X86_MINIMUM_CPU_FAMILY=64 CONFIG_X86_DEBUGCTLMSR=y CONFIG_IA32_FEAT_CTL=y CONFIG_X86_VMX_FEATURE_NAMES=y +CONFIG_PROCESSOR_SELECT=y CONFIG_CPU_SUP_INTEL=y CONFIG_CPU_SUP_AMD=y CONFIG_CPU_SUP_HYGON=y @@ -404,14 +393,13 @@ CONFIG_CPU_SUP_ZHAOXIN=y CONFIG_HPET_TIMER=y CONFIG_HPET_EMULATE_RTC=y CONFIG_DMI=y -# CONFIG_GART_IOMMU is not set +CONFIG_GART_IOMMU=y CONFIG_BOOT_VESA_SUPPORT=y -# CONFIG_MAXSMP is not set -CONFIG_NR_CPUS_RANGE_BEGIN=2 -CONFIG_NR_CPUS_RANGE_END=512 -CONFIG_NR_CPUS_DEFAULT=64 -CONFIG_NR_CPUS=320 -CONFIG_SCHED_CLUSTER=y +CONFIG_MAXSMP=y +CONFIG_NR_CPUS_RANGE_BEGIN=8192 +CONFIG_NR_CPUS_RANGE_END=8192 +CONFIG_NR_CPUS_DEFAULT=8192 +CONFIG_NR_CPUS=8192 CONFIG_SCHED_SMT=y CONFIG_SCHED_MC=y CONFIG_SCHED_MC_PRIO=y @@ -440,6 +428,7 @@ CONFIG_X86_16BIT=y CONFIG_X86_ESPFIX64=y CONFIG_X86_VSYSCALL_EMULATION=y CONFIG_X86_IOPL_IOPERM=y +CONFIG_I8K=m CONFIG_MICROCODE=y CONFIG_MICROCODE_INTEL=y CONFIG_MICROCODE_AMD=y @@ -448,8 +437,7 @@ CONFIG_X86_MSR=y CONFIG_X86_CPUID=y CONFIG_X86_5LEVEL=y CONFIG_X86_DIRECT_GBPAGES=y -CONFIG_X86_CPA_STATISTICS=y -CONFIG_X86_MEM_ENCRYPT=y +# CONFIG_X86_CPA_STATISTICS is not set CONFIG_AMD_MEM_ENCRYPT=y # CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT is not set CONFIG_NUMA=y @@ -459,13 +447,15 @@ CONFIG_X86_64_ACPI_NUMA=y CONFIG_NODES_SHIFT=5 CONFIG_ARCH_SPARSEMEM_ENABLE=y CONFIG_ARCH_SPARSEMEM_DEFAULT=y -# CONFIG_ARCH_MEMORY_PROBE is not set +CONFIG_ARCH_SELECT_MEMORY_MODEL=y +CONFIG_ARCH_MEMORY_PROBE=y CONFIG_ARCH_PROC_KCORE_TEXT=y CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000 CONFIG_X86_PMEM_LEGACY_DEVICE=y CONFIG_X86_PMEM_LEGACY=m CONFIG_X86_CHECK_BIOS_CORRUPTION=y CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y +CONFIG_X86_RESERVE_LOW=64 CONFIG_MTRR=y CONFIG_MTRR_SANITIZER=y CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1 @@ -473,9 +463,8 @@ CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAU CONFIG_X86_PAT=y CONFIG_ARCH_USES_PG_UNCACHED=y CONFIG_ARCH_RANDOM=y +CONFIG_X86_SMAP=y CONFIG_X86_UMIP=y -CONFIG_CC_HAS_IBT=y -CONFIG_X86_KERNEL_IBT=y CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=y # CONFIG_X86_INTEL_TSX_MODE_OFF is not set # CONFIG_X86_INTEL_TSX_MODE_ON is not set @@ -485,8 +474,8 @@ CONFIG_EFI=y CONFIG_EFI_STUB=y CONFIG_EFI_MIXED=y # CONFIG_HZ_100 is not set -# CONFIG_HZ_250 is not set -CONFIG_HZ_300=y +CONFIG_HZ_250=y +# CONFIG_HZ_300 uis not set # CONFIG_HZ_1000 is not set CONFIG_HZ=300 CONFIG_SCHED_HRTICK=y @@ -510,27 +499,18 @@ CONFIG_HOTPLUG_CPU=y # CONFIG_BOOTPARAM_HOTPLUG_CPU0 is not set # CONFIG_DEBUG_HOTPLUG_CPU0 is not set # CONFIG_COMPAT_VDSO is not set +# CONFIG_LEGACY_VSYSCALL_EMULATE is not set CONFIG_LEGACY_VSYSCALL_XONLY=y # CONFIG_LEGACY_VSYSCALL_NONE is not set # CONFIG_CMDLINE_BOOL is not set CONFIG_MODIFY_LDT_SYSCALL=y -# CONFIG_STRICT_SIGALTSTACK_SIZE is not set CONFIG_HAVE_LIVEPATCH=y # CONFIG_LIVEPATCH is not set # end of Processor type and features -CONFIG_CC_HAS_SLS=y -CONFIG_CC_HAS_RETURN_THUNK=y -CONFIG_SPECULATION_MITIGATIONS=y -CONFIG_PAGE_TABLE_ISOLATION=y -CONFIG_RETPOLINE=y -CONFIG_RETHUNK=y -CONFIG_CPU_UNRET_ENTRY=y -CONFIG_CPU_IBPB_ENTRY=y -CONFIG_CPU_IBRS_ENTRY=y -CONFIG_SLS=y CONFIG_ARCH_HAS_ADD_PAGES=y CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=y +CONFIG_USE_PERCPU_NUMA_NODE_ID=y # # Power management and ACPI options @@ -538,6 +518,7 @@ CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE= CONFIG_ARCH_HIBERNATION_HEADER=y CONFIG_SUSPEND=y CONFIG_SUSPEND_FREEZER=y +# CONFIG_SUSPEND_SKIP_SYNC is not set CONFIG_HIBERNATE_CALLBACKS=y CONFIG_HIBERNATION=y CONFIG_HIBERNATION_SNAPSHOT_DEV=y @@ -551,6 +532,7 @@ CONFIG_PM_DEBUG=y # CONFIG_PM_ADVANCED_DEBUG is not set # CONFIG_PM_TEST_SUSPEND is not set CONFIG_PM_SLEEP_DEBUG=y +# CONFIG_DPM_WATCHDOG is not set CONFIG_PM_TRACE=y CONFIG_PM_TRACE_RTC=y CONFIG_PM_CLK=y @@ -588,6 +570,7 @@ CONFIG_ACPI_HOTPLUG_CPU=y CONFIG_ACPI_PROCESSOR_AGGREGATOR=m CONFIG_ACPI_THERMAL=y CONFIG_ACPI_PLATFORM_PROFILE=m +CONFIG_ACPI_CUSTOM_DSDT_FILE="" CONFIG_ARCH_HAS_ACPI_TABLE_UPGRADE=y CONFIG_ACPI_TABLE_UPGRADE=y CONFIG_ACPI_DEBUG=y @@ -599,6 +582,7 @@ CONFIG_ACPI_SBS=m CONFIG_ACPI_HED=y CONFIG_ACPI_CUSTOM_METHOD=m CONFIG_ACPI_BGRT=y +# CONFIG_ACPI_REDUCED_HARDWARE_ONLY is not set CONFIG_ACPI_NFIT=m # CONFIG_NFIT_SECURITY_DEBUG is not set CONFIG_ACPI_NUMA=y @@ -654,9 +638,9 @@ CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y # CPU frequency scaling drivers # CONFIG_X86_INTEL_PSTATE=y -CONFIG_X86_PCC_CPUFREQ=m +CONFIG_X86_PCC_CPUFREQ=y CONFIG_X86_AMD_PSTATE=m -CONFIG_X86_ACPI_CPUFREQ=m +CONFIG_X86_ACPI_CPUFREQ=y CONFIG_X86_ACPI_CPUFREQ_CPB=y CONFIG_X86_POWERNOW_K8=m CONFIG_X86_AMD_FREQ_SENSITIVITY=m @@ -724,6 +708,7 @@ CONFIG_KVM_XFER_TO_GUEST_WORK=y CONFIG_HAVE_KVM_PM_NOTIFIER=y CONFIG_VIRTUALIZATION=y CONFIG_KVM=m +CONFIG_KVM_WERROR=y CONFIG_KVM_INTEL=m CONFIG_X86_SGX_KVM=y CONFIG_KVM_AMD=m @@ -752,18 +737,14 @@ CONFIG_UPROBES=y CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y CONFIG_ARCH_USE_BUILTIN_BSWAP=y CONFIG_KRETPROBES=y -CONFIG_KRETPROBE_ON_RETHOOK=y CONFIG_USER_RETURN_NOTIFIER=y CONFIG_HAVE_IOREMAP_PROT=y CONFIG_HAVE_KPROBES=y CONFIG_HAVE_KRETPROBES=y CONFIG_HAVE_OPTPROBES=y CONFIG_HAVE_KPROBES_ON_FTRACE=y -CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=y CONFIG_HAVE_FUNCTION_ERROR_INJECTION=y CONFIG_HAVE_NMI=y -CONFIG_TRACE_IRQFLAGS_SUPPORT=y -CONFIG_TRACE_IRQFLAGS_NMI_SUPPORT=y CONFIG_HAVE_ARCH_TRACEHOOK=y CONFIG_HAVE_DMA_CONTIGUOUS=y CONFIG_GENERIC_SMP_IDLE_THREAD=y @@ -772,7 +753,6 @@ CONFIG_ARCH_HAS_SET_MEMORY=y CONFIG_ARCH_HAS_SET_DIRECT_MAP=y CONFIG_HAVE_ARCH_THREAD_STRUCT_WHITELIST=y CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y -CONFIG_ARCH_WANTS_NO_INSTR=y CONFIG_HAVE_ASM_MODVERSIONS=y CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y CONFIG_HAVE_RSEQ=y @@ -788,7 +768,6 @@ CONFIG_HAVE_ARCH_JUMP_LABEL=y CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE=y CONFIG_MMU_GATHER_TABLE_FREE=y CONFIG_MMU_GATHER_RCU_TABLE_FREE=y -CONFIG_MMU_GATHER_MERGE_VMAS=y CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y CONFIG_HAVE_CMPXCHG_LOCAL=y @@ -817,7 +796,6 @@ CONFIG_HAVE_MOVE_PMD=y CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=y CONFIG_HAVE_ARCH_HUGE_VMAP=y -CONFIG_HAVE_ARCH_HUGE_VMALLOC=y CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y CONFIG_HAVE_ARCH_SOFT_DIRTY=y CONFIG_HAVE_MOD_ARCH_SPECIFIC=y @@ -831,13 +809,6 @@ CONFIG_ARCH_MMAP_RND_BITS=28 CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS=y CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8 CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES=y -CONFIG_PAGE_SIZE_LESS_THAN_64KB=y -CONFIG_PAGE_SIZE_LESS_THAN_256KB=y -CONFIG_HAVE_OBJTOOL=y -CONFIG_HAVE_JUMP_LABEL_HACK=y -CONFIG_HAVE_NOINSTR_HACK=y -CONFIG_HAVE_NOINSTR_VALIDATION=y -CONFIG_HAVE_UACCESS_VALIDATION=y CONFIG_HAVE_STACK_VALIDATION=y CONFIG_HAVE_RELIABLE_STACKTRACE=y CONFIG_ISA_BUS_API=y @@ -847,7 +818,6 @@ CONFIG_COMPAT_32BIT_TIME=y CONFIG_HAVE_ARCH_VMAP_STACK=y CONFIG_VMAP_STACK=y CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET=y -CONFIG_RANDOMIZE_KSTACK_OFFSET=y CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT=y CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y CONFIG_STRICT_KERNEL_RWX=y @@ -861,14 +831,9 @@ CONFIG_ARCH_HAS_CC_PLATFORM=y CONFIG_HAVE_STATIC_CALL=y CONFIG_HAVE_STATIC_CALL_INLINE=y CONFIG_HAVE_PREEMPT_DYNAMIC=y -CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y CONFIG_ARCH_WANT_LD_ORPHAN_WARN=y CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y -CONFIG_ARCH_SUPPORTS_PAGE_TABLE_CHECK=y CONFIG_ARCH_HAS_ELFCORE_COMPAT=y -CONFIG_ARCH_HAS_PARANOID_L1D_FLUSH=y -CONFIG_DYNAMIC_SIGFRAME=y -CONFIG_HAVE_ARCH_NODE_DEV_GROUP=y # # GCOV-based kernel profiling @@ -878,8 +843,6 @@ CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y # end of GCOV-based kernel profiling CONFIG_HAVE_GCC_PLUGINS=y -CONFIG_GCC_PLUGINS=y -# CONFIG_GCC_PLUGIN_LATENT_ENTROPY is not set # end of General architecture-dependent options CONFIG_RT_MUTEXES=y @@ -907,23 +870,22 @@ CONFIG_MODULE_SIG_HASH="sha512" CONFIG_MODULE_COMPRESS_ZSTD=y CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS=y CONFIG_MODPROBE_PATH="/sbin/modprobe" +# CONFIG_TRIM_UNUSED_KSYMS is not set CONFIG_MODULES_TREE_LOOKUP=y CONFIG_BLOCK=y -CONFIG_BLOCK_LEGACY_AUTOLOAD=y CONFIG_BLK_RQ_ALLOC_TIME=y +CONFIG_BLK_SCSI_REQUEST=y CONFIG_BLK_CGROUP_RWSTAT=y -CONFIG_BLK_DEV_BSG_COMMON=y -CONFIG_BLK_ICQ=y +CONFIG_BLK_DEV_BSG=y CONFIG_BLK_DEV_BSGLIB=y CONFIG_BLK_DEV_INTEGRITY=y CONFIG_BLK_DEV_INTEGRITY_T10=y CONFIG_BLK_DEV_ZONED=y CONFIG_BLK_DEV_THROTTLING=y -CONFIG_BLK_DEV_THROTTLING_LOW=y +# CONFIG_BLK_DEV_THROTTLING_LOW is not set +CONFIG_BLK_CMDLINE_PARSER=y CONFIG_BLK_WBT=y -CONFIG_BLK_WBT_MQ=y -CONFIG_BLK_CGROUP_IOLATENCY=y -CONFIG_BLK_CGROUP_FC_APPID=y +# CONFIG_BLK_CGROUP_IOLATENCY is not set CONFIG_BLK_CGROUP_IOCOST=y CONFIG_BLK_CGROUP_IOPRIO=y CONFIG_BLK_DEBUG_FS=y @@ -980,6 +942,10 @@ CONFIG_PREEMPT_NOTIFIERS=y CONFIG_PADATA=y CONFIG_ASN1=y CONFIG_UNINLINE_SPIN_UNLOCK=y +CONFIG_INLINE_READ_UNLOCK=y +CONFIG_INLINE_READ_UNLOCK_IRQ=y +CONFIG_INLINE_WRITE_UNLOCK=y +CONFIG_INLINE_WRITE_UNLOCK_IRQ=y CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y CONFIG_MUTEX_SPIN_ON_OWNER=y CONFIG_RWSEM_SPIN_ON_OWNER=y @@ -1008,53 +974,22 @@ CONFIG_COREDUMP=y # # Memory Management options # -CONFIG_ZPOOL=y -CONFIG_SWAP=y -CONFIG_ZSWAP=y -CONFIG_ZSWAP_DEFAULT_ON=y -# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set -# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO is not set -# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set -CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4=y -# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set -# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set -CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lz4" -# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD is not set -CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD=y -# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set -CONFIG_ZSWAP_ZPOOL_DEFAULT="z3fold" -CONFIG_ZBUD=y -CONFIG_Z3FOLD=y -CONFIG_ZSMALLOC=y -# CONFIG_ZSMALLOC_STAT is not set - -# -# SLAB allocator options -# -# CONFIG_SLAB is not set -CONFIG_SLUB=y -CONFIG_SLAB_MERGE_DEFAULT=y -CONFIG_SLAB_FREELIST_RANDOM=y -CONFIG_SLAB_FREELIST_HARDENED=y -# CONFIG_SLUB_STATS is not set -CONFIG_SLUB_CPU_PARTIAL=y -# end of SLAB allocator options - -CONFIG_SHUFFLE_PAGE_ALLOCATOR=y -# CONFIG_COMPAT_BRK is not set +CONFIG_SELECT_MEMORY_MODEL=y +CONFIG_SPARSEMEM_MANUAL=y CONFIG_SPARSEMEM=y +CONFIG_NEED_MULTIPLE_NODES=y CONFIG_SPARSEMEM_EXTREME=y CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y CONFIG_SPARSEMEM_VMEMMAP=y CONFIG_HAVE_FAST_GUP=y CONFIG_NUMA_KEEP_MEMINFO=y CONFIG_MEMORY_ISOLATION=y -CONFIG_EXCLUSIVE_SYSTEM_RAM=y CONFIG_HAVE_BOOTMEM_INFO_NODE=y CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y -CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y CONFIG_MEMORY_HOTPLUG=y +CONFIG_MEMORY_HOTPLUG_SPARSE=y CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y +CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y CONFIG_MEMORY_HOTREMOVE=y CONFIG_MHP_MEMMAP_ON_MEMORY=y CONFIG_SPLIT_PTLOCK_CPUS=4 @@ -1064,7 +999,6 @@ CONFIG_BALLOON_COMPACTION=y CONFIG_COMPACTION=y CONFIG_PAGE_REPORTING=y CONFIG_MIGRATION=y -CONFIG_DEVICE_MIGRATION=y CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=y CONFIG_ARCH_ENABLE_THP_MIGRATION=y CONFIG_CONTIG_ALLOC=y @@ -1076,66 +1010,54 @@ CONFIG_DEFAULT_MMAP_MIN_ADDR=65536 CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y CONFIG_MEMORY_FAILURE=y CONFIG_HWPOISON_INJECT=m -CONFIG_ARCH_WANT_GENERAL_HUGETLB=y -CONFIG_ARCH_WANTS_THP_SWAP=y CONFIG_TRANSPARENT_HUGEPAGE=y -CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y -# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set +# CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is not set +CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y +CONFIG_ARCH_WANTS_THP_SWAP=y CONFIG_THP_SWAP=y -CONFIG_READ_ONLY_THP_FOR_FS=y -CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y -CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y -CONFIG_USE_PERCPU_NUMA_NODE_ID=y -CONFIG_HAVE_SETUP_PER_CPU_AREA=y +CONFIG_CLEANCACHE=y CONFIG_FRONTSWAP=y -CONFIG_CMA=y -# CONFIG_CMA_DEBUG is not set -CONFIG_CMA_DEBUGFS=y -CONFIG_CMA_SYSFS=y -CONFIG_CMA_AREAS=7 +# CONFIG_CMA is not set CONFIG_MEM_SOFT_DIRTY=y +CONFIG_ZSWAP=y +# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set +CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y +# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set +# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set +# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set +# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set +CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo" +CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y +# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set +# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set +CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud" +# CONFIG_ZSWAP_DEFAULT_ON is not set +CONFIG_ZPOOL=y +CONFIG_ZBUD=y +CONFIG_Z3FOLD=m +CONFIG_ZSMALLOC=y +# CONFIG_ZSMALLOC_STAT is not set CONFIG_GENERIC_EARLY_IOREMAP=y # CONFIG_DEFERRED_STRUCT_PAGE_INIT is not set -CONFIG_PAGE_IDLE_FLAG=y CONFIG_IDLE_PAGE_TRACKING=y CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y -CONFIG_ARCH_HAS_CURRENT_STACK_POINTER=y -CONFIG_ARCH_HAS_VM_GET_PAGE_PROT=y CONFIG_ARCH_HAS_PTE_DEVMAP=y -CONFIG_ZONE_DMA=y -CONFIG_ZONE_DMA32=y CONFIG_ZONE_DEVICE=y +CONFIG_DEV_PAGEMAP_OPS=y CONFIG_HMM_MIRROR=y CONFIG_DEVICE_PRIVATE=y CONFIG_VMAP_PFN=y CONFIG_ARCH_USES_HIGH_VMA_FLAGS=y CONFIG_ARCH_HAS_PKEYS=y -CONFIG_VM_EVENT_COUNTERS=y # CONFIG_PERCPU_STATS is not set # CONFIG_GUP_TEST is not set +# CONFIG_READ_ONLY_THP_FOR_FS is not set CONFIG_ARCH_HAS_PTE_SPECIAL=y CONFIG_MAPPING_DIRTY_HELPERS=y -CONFIG_SECRETMEM=y -CONFIG_ANON_VMA_NAME=y -CONFIG_USERFAULTFD=y -CONFIG_HAVE_ARCH_USERFAULTFD_WP=y -CONFIG_HAVE_ARCH_USERFAULTFD_MINOR=y -CONFIG_PTE_MARKER=y -CONFIG_PTE_MARKER_UFFD_WP=y - -# -# Data Access Monitoring -# -CONFIG_DAMON=y -CONFIG_DAMON_VADDR=y -CONFIG_DAMON_PADDR=y -CONFIG_DAMON_SYSFS=y -CONFIG_DAMON_DBGFS=y -CONFIG_DAMON_RECLAIM=y -# end of Data Access Monitoring # end of Memory Management options CONFIG_NET=y +CONFIG_WANT_COMPAT_NETLINK_MESSAGES=y CONFIG_COMPAT_NETLINK_MESSAGES=y CONFIG_NET_INGRESS=y CONFIG_NET_EGRESS=y
Thanks, no, not notepad :-) emacs. I haven't even noticed I have CRLF on. However, the above normal diff is quite verbose so I pasted a linux's scripts/diffconfig output towards the end of this post. It will hopefully let more eyeballs see it.
Edit: possibly very relevant
-CONFIG_ZSWAP_DEFAULT_ON=y +# CONFIG_ZSWAP_DEFAULT_ON is not set -# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO is not set +CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y -CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4=y +# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set -CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lz4" +CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo" -# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD is not set +CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y -CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD=y +# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set -# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set +# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set -CONFIG_ZSWAP_ZPOOL_DEFAULT="z3fold" +CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud"
Yes, this is very interesting, thank you. I just re-checked and indeed in stock Arch config CONFIG_ZSWAP_DEFAULT_ON is set to yes, while my config disables it. This might be it. It seems zswap can be enabled/disabled at runtime too, so I will be able to check this without having to reboot into the stock kernel. I can enable zswap on my custom kernel at runtime and see if it degrades the same way.
Edit #2
-CONFIG_SWAP=y +CONFIG_VM_EVENT_COUNTERS=y -CONFIG_HAVE_ARCH_HUGE_VMALLOC=y -CONFIG_ARCH_HAS_VM_GET_PAGE_PROT=y -CONFIG_VM_EVENT_COUNTERS=y -CONFIG_ANON_VMA_NAME=y
CONFIG_SWAP is on in my running kernel (otherwise there would be no swap at all). This is because I run make oldconfig with the file I posted and although it is missing 'CONFIG_SWAP=y' it also doesn't have the '#CONFIG_SWAP is not set' line in which case make oldconfig sets the value to default which is on.
This time I wen to the kernel build folder and I got the .config file. I then run scripts/diffconfig on both files so the below shows the difference from Arch stock to custom in more succint form:
-ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP y
-BPF_PRELOAD_UMD m
-CMA_ALIGNMENT 8
-CMA_AREAS 7
-CMA_DEBUG n
-CMA_DEBUGFS y
-CMA_SIZE_MBYTES 0
-CMA_SIZE_SEL_MAX n
-CMA_SIZE_SEL_MBYTES y
-CMA_SIZE_SEL_MIN n
-CMA_SIZE_SEL_PERCENTAGE n
-CMA_SYSFS y
-DAMON_DBGFS y
-DAMON_PADDR y
-DAMON_RECLAIM y
-DAMON_SYSFS y
-DAMON_VADDR y
-DEBUG_KMAP_LOCAL_FORCE_MAP n
-DMABUF_HEAPS_CMA y
-DMA_CMA y
-DMA_PERNUMA_CMA n
-FORCE_TASKS_RCU n
-FORCE_TASKS_RUDE_RCU n
-FORCE_TASKS_TRACE_RCU n
-RCU_BOOST y
-RCU_BOOST_DELAY 500
-RCU_EXP_KTHREAD n
-RCU_FANOUT 64
-RCU_FANOUT_LEAF 16
-RCU_NOCB_CPU y
-TASKS_TRACE_RCU_READ_MB n
-USERMODE_DRIVER y
ANON_VMA_NAME y -> n
ARCH_MEMORY_PROBE n -> y
BLK_CGROUP_FC_APPID y -> n
BLK_CGROUP_IOLATENCY y -> n
BLK_DEV_THROTTLING_LOW y -> n
BPF_PRELOAD y -> n
CMA y -> n
CONTEXT_TRACKING_FORCE n -> y
CRYPTO_LZO m -> y
DAMON y -> n
GART_IOMMU n -> y
HZ 300 -> 250
HZ_250 n -> y
HZ_300 y -> n
IKCONFIG_PROC y -> n
INTEL_TDX_GUEST y -> n
MAXSMP n -> y
NODES_SHIFT 5 -> 10
NO_HZ_FULL y -> n
NO_HZ_IDLE n -> y
NR_CPUS 320 -> 8192
NR_CPUS_DEFAULT 64 -> 8192
NR_CPUS_RANGE_BEGIN 2 -> 8192
NR_CPUS_RANGE_END 512 -> 8192
PREEMPT y -> n
PREEMPT_VOLUNTARY n -> y
PRINTK_INDEX y -> n
RCU_EXPERT y -> n
READ_ONLY_THP_FOR_FS y -> n
SCHED_CORE y -> n
SLS y -> n
TRANSPARENT_HUGEPAGE_ALWAYS y -> n
TRANSPARENT_HUGEPAGE_MADVISE n -> y
X86_ACPI_CPUFREQ m -> y
X86_CPA_STATISTICS y -> n
X86_EXTENDED_PLATFORM n -> y
X86_KERNEL_IBT y -> n
X86_PCC_CPUFREQ m -> y
Z3FOLD y -> m
ZSWAP_COMPRESSOR_DEFAULT "lz4" -> "lzo"
ZSWAP_COMPRESSOR_DEFAULT_LZ4 y -> n
ZSWAP_COMPRESSOR_DEFAULT_LZO n -> y
ZSWAP_DEFAULT_ON y -> n
ZSWAP_ZPOOL_DEFAULT "z3fold" -> "zbud"
ZSWAP_ZPOOL_DEFAULT_Z3FOLD y -> n
ZSWAP_ZPOOL_DEFAULT_ZBUD n -> y
+CPUMASK_OFFSTACK y
+IOMMU_DEBUG n
+IOMMU_HELPER y
+SGI_GRU n
+SGI_XP n
+TICK_CPU_ACCOUNTING n
+UV_MMTIMER m
+UV_SYSFS n
+X86_GOLDFISH n
+X86_INTEL_MID n
+X86_NUMACHIP y
+X86_UV y
+X86_VSMP n
Offline
This is the stock Arch kernel (fails): https://pastebin.com/Uude2vfe https://file.io/gPb1GuosvWE7
This is the custom kernel (works): https://pastebin.com/HSWNpiju https://file.io/ricYUv1fSeb1
Cute, but there must be something wrong with your processing of
cat /proc/vmstat
since they're both essentially identical (highly unlikely given the difference in aforementioned swap usage):
$ sdiff -s <(cut -d, -f1,3- stock\ kernel\ csv.txt ) custom\ kernel\ csv.txt
timestamp,nr_zone_inactive_anon,nr_zone_active_anon,nr_z | nr_free_pages,nr_zone_inactive_anon,nr_zone_active_anon,nr_zo
308,3722,922,57376,12499,0,50,0,0,0,0,29233906,0,0,2475,29233 / 308,3722,922,57376,12499,0,50,0,0,0,0,29233906,0,0,2475,29233
Just the raw output 30s apart should do.
Last edited by sabroad (2022-10-01 23:30:02)
--
saint_abroad
Offline
7000k wrote:This is the stock Arch kernel (fails): https://pastebin.com/Uude2vfe https://file.io/gPb1GuosvWE7
This is the custom kernel (works): https://pastebin.com/HSWNpiju https://file.io/ricYUv1fSeb1Cute, but there must be something wrong with your processing of
cat /proc/vmstat
since they're both essentially identical (highly unlikely given the difference in aforementioned swap usage):
Thank you.
However, I messed up the files before upload. I cut one column from the "custom kernel" file. Also I used one set of column names for both files(copy/paste) while as they came out from my script they were different. No doubt due to different kernel configs.
My mistake. Nevermind. Here are corrected files(again timestamps are replaced with row numbers, also vm.lowmem_reserve_ratio is reported as a set of 5 numbers, this was split into 5 columns manually to make comparisons easier):
Stock kernel (broken): https://drive.google.com/file/d/1omVM8R … sp=sharing
Custom kernel(working): https://drive.google.com/file/d/1pcEGlN … sp=sharing
This is on google drive so it should remain online.
$ sdiff -s <(cut -d, -f1,3- stock\ kernel\ csv.txt ) custom\ kernel\ csv.text timestamp,nr_zone_inactive_anon,nr_zone_active_anon,nr_z | nr_free_pages,nr_zone_inactive_anon,nr_zone_active_anon,nr_zo 308,3722,922,57376,12499,0,50,0,0,0,0,29233906,0,0,2475,29233 / 308,3722,922,57376,12499,0,50,0,0,0,0,29233906,0,0,2475,29233
Just the raw output 30s apart should do.
From which moment? IMO it is completely pointless to just grab two snapshots 30s apart on two dynamic processes we don't really know at which point they diverge. This is why I saved the data that was mentioned previously every second for the duration of the process. Also, the reason is that I can't really run it too often because it requires a reboot of my main PC.
The best is to load the files into Libre Calc, Excel or similar then one can select data from 30s apart by selecting a row, and another one 30 rows further. Columns captured differ due to difference in kernel config.
However, for you I've extracted 4 rows (two from each kernel run 30s apart). Those were taken at 180s from process start and then at 210s mark. I selected to skip first 3 minutes because that's how long loading of the model into memory took.
This is custom kernel(works)
timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin pgpgout pswpin pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree pgactivate pgdeactivate pglazyfree pgfault pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode
180 1941284 6101460 1754 10574 7294 0 33 0 0 0 0 27477729 0 0 2475 27477729 0 6101460 1754 10574 7294 0 20183 25558 346 0 59096 452279 4829242 433438 3157732 135 9998 2221 6102420 4326 32922 33 0 0 359 0 0 0 0 0 2872836 41 29848 2902645 0 0 0 0 5448 14242 14513 0 378605 189071 26394549 11610564 452368 2872836 1025 2699131 24813240 0 0 0 357 36326 0 0 2609918 0 30715851 2616392 4460740 0 14971905 84650 0 4990531 53544 11111093 3224881 0 0 17704401 10988213 0 15077092 13615522 2846440 11489534 0 0 62557 4431 2331 73 2410 2843630 0 0 0 0 0 0 0 0 1208170 26 0 0 0 1461470 3973865 2435329 0 0 0 1333 248286 994284 0 0 6033 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 368204 367106 0 0 0 0 121 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0
210 990735 6992475 30158 37868 10811 0 46 0 0 0 0 28901242 0 0 2475 28901242 0 6992475 30158 37868 10811 0 20179 25626 0 0 58636 1277506 4853361 461842 3160455 135 12396 2221 7021408 9497 886594 46 0 0 359 0 0 0 0 0 2872836 41 31268 2904052 0 0 0 0 5520 14454 837555 0 194702 97232 29814053 11616212 1277506 2872836 1025 2699131 26236751 0 0 0 357 36326 0 0 2609918 0 31546120 2616964 4460740 0 16368153 916648 0 4990531 55287 11111093 3224881 0 0 17704401 10988213 0 15077092 13615522 2846440 11489534 0 0 62557 4431 2331 73 2410 2843654 0 0 0 0 0 0 0 0 1564842 26 0 0 0 2141187 6714068 3147752 0 0 0 1333 248286 994284 0 0 6033 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 370178 368905 0 0 0 0 121 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0
This is stock kernel(broken):
timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin pgpgout pswpin pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree pgactivate pgdeactivate pglazyfree pgfault pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail cma_alloc_success cma_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode
180 62704 1742969 11388 3484972 2761940 0 40 0 0 0 0 14500716 0 0 2448 14500716 0 1742969 11388 3484972 2761940 0 29966 25737 0 0 13905 148 6389469 99 4831303 2 10 0 1753986 43350 6247420 40 0 0 383 0 0 0 0 3340 1485104 0 16019 1501080 0 0 0 0 5040 6922 98 0 1249077 623776 27659569 5942960 104 1469719 1025 1233176 13300887 0 0 0 0 0 0 0 0 0 16357181 87154 2147272 0 578266 416 0 2152098 35944 8765311 0 0 0 12493291 0 0 5166041 7327250 1485067 7280244 0 0 8192 0 837 243 1084 1469682 0 0 0 0 0 0 0 0 5680 0 0 0 0 28430 98365 11878 0 0 0 43 28430 98365 0 0 0 0 6154 0 0 0 0 0 0 12763 21 0 40 0 0 0 0 0 0 0 3 2834 0 24 0 0 0 0 2831 0 0 0 0 110 67 0 0 44 15385 92 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0
210 66830 6203412 109528 5688 3380 0 25 0 0 0 0 24360205 0 0 2448 24360205 0 6203643 109528 5716 3380 0 30746 51385 0 0 58880 3452052 6935356 98482 5371579 2 2289 2112 6296023 2395 2121284 25 0 0 320 0 0 0 0 3575 6209290 53 17639 6226920 0 0 0 0 5168 18780 2111856 0 2051 1024 42579105 18307704 3450961 4559265 1025 2152313 22241749 0 0 0 167 12745 0 0 971591 0 26408120 117043 5471949 0 8735187 578255 0 6417191 37831 19446349 2161862 0 0 33081116 4143427 0 23058616 14165927 7548510 14059701 0 0 43448 1420 3682 302 3996 4558831 0 0 0 0 0 0 0 0 51095 22 0 0 0 195362 441148 107781 0 0 0 2078 195362 441148 0 0 0 0 6154 0 0 0 0 0 0 18768 7890 0 40 0 0 0 0 0 0 0 3 8604 0 40 0 0 0 0 8601 0 0 0 0 2874398 2874123 0 0 1256 1650044 92 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0
This is again but with columns that don't change removed:
Custom kernel(working):
timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_write_pending numa_hit numa_interleave numa_local nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_file nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_dirtied nr_written nr_kernel_stack nr_page_table_pages nr_swapcached nr_dirty_threshold nr_dirty_background_threshold pgpgin pgpgout pswpin pswpout pgalloc_dma32 pgalloc_normal pgfree pgactivate pgfault pgmajfault pgreuse pgrotated pgmigrate_success compact_migrate_scanned compact_free_scanned compact_isolated compact_daemon_free_scanned swap_ra swap_ra_hit
180 1941284 6101460 1754 10574 7294 33 27477729 2475 27477729 6101460 1754 10574 7294 20183 25558 346 59096 452279 4829242 433438 3157732 9998 6102420 4326 32922 33 29848 2902645 5448 14242 14513 378605 189071 26394549 11610564 452368 2872836 2699131 24813240 30715851 2616392 14971905 84650 53544 2843630 1208170 1461470 3973865 2435329 994284 368204 367106
210 990735 6992475 30158 37868 10811 46 28901242 2475 28901242 6992475 30158 37868 10811 20179 25626 0 58636 1277506 4853361 461842 3160455 12396 7021408 9497 886594 46 31268 2904052 5520 14454 837555 194702 97232 29814053 11616212 1277506 2872836 2699131 26236751 31546120 2616964 16368153 916648 55287 2843654 1564842 2141187 6714068 3147752 994284 370178 368905
Stock(broken): - it has many more columns, because many more columns changed during those 30 seconds -
timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_write_pending numa_hit numa_interleave numa_local nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_shmem nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_kernel_stack nr_page_table_pages nr_swapcached nr_dirty_threshold nr_dirty_background_threshold pgpgin pgpgout pswpin pswpout pgalloc_dma32 pgalloc_normal allocstall_normal allocstall_movable pgskip_normal pgfree pgactivate pgdeactivate pgfault pgmajfault pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgscan_kswapd pgscan_direct pgscan_anon pgscan_file pgsteal_anon pgsteal_file slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated pgmigrate_success pgmigrate_fail compact_migrate_scanned compact_free_scanned compact_isolated compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned unevictable_pgs_culled thp_fault_alloc thp_fault_fallback thp_split_pmd thp_scan_exceed_swap_pte thp_swpout swap_ra swap_ra_hit zswpin zswpout
180 62704 1742969 11388 3484972 2761940 40 14500716 2448 14500716 1742969 11388 3484972 2761940 29966 25737 0 13905 148 6389469 99 4831303 10 0 1753986 43350 6247420 40 383 3340 1485104 0 16019 1501080 5040 6922 98 1249077 623776 27659569 5942960 104 1469719 1233176 13300887 0 0 0 16357181 87154 2147272 578266 416 2152098 35944 8765311 0 12493291 0 5166041 7327250 1485067 7280244 8192 0 837 243 1084 1469682 5680 0 28430 98365 11878 43 28430 98365 6154 12763 21 2834 24 2831 110 67 44 15385
210 66830 6203412 109528 5688 3380 25 24360205 2448 24360205 6203643 109528 5716 3380 30746 51385 0 58880 3452052 6935356 98482 5371579 2289 2112 6296023 2395 2121284 25 320 3575 6209290 53 17639 6226920 5168 18780 2111856 2051 1024 42579105 18307704 3450961 4559265 2152313 22241749 167 12745 971591 26408120 117043 5471949 8735187 578255 6417191 37831 19446349 2161862 33081116 4143427 23058616 14165927 7548510 14059701 43448 1420 3682 302 3996 4558831 51095 22 195362 441148 107781 2078 195362 441148 6154 18768 7890 8604 40 8601 2874398 2874123 1256 1650044
I hope this helps to get to the bottom of it. I suppose increasingly more people will want to run inference of large models on their CPUs and not all of those models will fit in the RAM. Therefore we will see more users being hit by this and it will be beneficial to arch to find out which default setting could be changed to avoid this issue.
Offline
Maybe this is of help to you https://unix.stackexchange.com/question … -available
https://ugjka.net
paru > yay | webcord > discord
pacman -S spotify-launcher
mount /dev/disk/by-...
Offline
Stock kernel (broken): https://drive.google.com/file/d/1omVM8R … sp=sharing
Custom kernel(working): https://drive.google.com/file/d/1pcEGlN … sp=sharing
Unfortunately, Google requires login (or tracking cookies) to download.
This is stock kernel(broken):
timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin pgpgout pswpin pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree pgactivate pgdeactivate pglazyfree pgfault pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail cma_alloc_success cma_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode 180 62704 1742969 11388 3484972 2761940 0 40 0 0 0 0 14500716 0 0 2448 14500716 0 1742969 11388 3484972 2761940 0 29966 25737 0 0 13905 148 6389469 99 4831303 2 10 0 1753986 43350 6247420 40 0 0 383 0 0 0 0 3340 1485104 0 16019 1501080 0 0 0 0 5040 6922 98 0 1249077 623776 27659569 5942960 104 1469719 1025 1233176 13300887 0 0 0 0 0 0 0 0 0 16357181 87154 2147272 0 578266 416 0 2152098 35944 8765311 0 0 0 12493291 0 0 5166041 7327250 1485067 7280244 0 0 8192 0 837 243 1084 1469682 0 0 0 0 0 0 0 0 5680 0 0 0 0 28430 98365 11878 0 0 0 43 28430 98365 0 0 0 0 6154 0 0 0 0 0 0 12763 21 0 40 0 0 0 0 0 0 0 3 2834 0 24 0 0 0 0 2831 0 0 0 0 110 67 0 0 44 15385 92 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0 210 66830 6203412 109528 5688 3380 0 25 0 0 0 0 24360205 0 0 2448 24360205 0 6203643 109528 5716 3380 0 30746 51385 0 0 58880 3452052 6935356 98482 5371579 2 2289 2112 6296023 2395 2121284 25 0 0 320 0 0 0 0 3575 6209290 53 17639 6226920 0 0 0 0 5168 18780 2111856 0 2051 1024 42579105 18307704 3450961 4559265 1025 2152313 22241749 0 0 0 167 12745 0 0 971591 0 26408120 117043 5471949 0 8735187 578255 0 6417191 37831 19446349 2161862 0 0 33081116 4143427 0 23058616 14165927 7548510 14059701 0 0 43448 1420 3682 302 3996 4558831 0 0 0 0 0 0 0 0 51095 22 0 0 0 195362 441148 107781 0 0 0 2078 195362 441148 0 0 0 0 6154 0 0 0 0 0 0 18768 7890 0 40 0 0 0 0 0 0 0 3 8604 0 40 0 0 0 0 8601 0 0 0 0 2874398 2874123 0 0 1256 1650044 92 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0
Looks like the system is under severe memory pressure, with writes/allocations whilst scanning pages to swap/free/compact:
nr_vmscan_write 6209290
This is custom kernel(works)
timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin pgpgout pswpin pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree pgactivate pgdeactivate pglazyfree pgfault pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode 180 1941284 6101460 1754 10574 7294 0 33 0 0 0 0 27477729 0 0 2475 27477729 0 6101460 1754 10574 7294 0 20183 25558 346 0 59096 452279 4829242 433438 3157732 135 9998 2221 6102420 4326 32922 33 0 0 359 0 0 0 0 0 2872836 41 29848 2902645 0 0 0 0 5448 14242 14513 0 378605 189071 26394549 11610564 452368 2872836 1025 2699131 24813240 0 0 0 357 36326 0 0 2609918 0 30715851 2616392 4460740 0 14971905 84650 0 4990531 53544 11111093 3224881 0 0 17704401 10988213 0 15077092 13615522 2846440 11489534 0 0 62557 4431 2331 73 2410 2843630 0 0 0 0 0 0 0 0 1208170 26 0 0 0 1461470 3973865 2435329 0 0 0 1333 248286 994284 0 0 6033 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 368204 367106 0 0 0 0 121 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0 210 990735 6992475 30158 37868 10811 0 46 0 0 0 0 28901242 0 0 2475 28901242 0 6992475 30158 37868 10811 0 20179 25626 0 0 58636 1277506 4853361 461842 3160455 135 12396 2221 7021408 9497 886594 46 0 0 359 0 0 0 0 0 2872836 41 31268 2904052 0 0 0 0 5520 14454 837555 0 194702 97232 29814053 11616212 1277506 2872836 1025 2699131 26236751 0 0 0 357 36326 0 0 2609918 0 31546120 2616964 4460740 0 16368153 916648 0 4990531 55287 11111093 3224881 0 0 17704401 10988213 0 15077092 13615522 2846440 11489534 0 0 62557 4431 2331 73 2410 2843654 0 0 0 0 0 0 0 0 1564842 26 0 0 0 2141187 6714068 3147752 0 0 0 1333 248286 994284 0 0 6033 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 370178 368905 0 0 0 0 121 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0
With the custom config, the system is under memory pressure, but less so:
nr_vmscan_write 2872836
Let's narrow the largest differences:
vmstat vanilla custom
compact_free_scanned 441148 6714068 # custom config is freeing >10x more pages for contiguous allocation
compact_isolated 107781 3147752
compact_migrate_scanned 195362 2141187 # custom config is migrating >10x more pages for contiguous allocation
nr_anon_transparent_hugepages 3575 0
nr_dirty_background_threshold 1024 97232
nr_dirty_threshold 2051 194702
nr_free_pages 66830 990735
pgactivate 117043 2616964 # custom config is reactivating pages that have gone inactive
pgmigrate_success 51095 1564842
swap_ra_hit 2874123 368905
thp_collapse_alloc 40 0
thp_deferred_split_page 3 0
thp_fault_alloc 18768 0 # custom config is not allocating transparent huge pages
thp_fault_fallback 7890 0
thp_scan_exceed_swap_pte 40 0
thp_split_pmd 8604 0
thp_swpout 8601 0
workingset_restore_anon 2 135
zswpin 1256 0
zswpout 1650044 0
With this, we can conclude:
* specific workload allocates memory in chunks (contiguous allocation);
* memory pressure is freeing sparse pages (causing fragmentation);
* specific workload does not access memory contiguously (allowing sparse pages to deactivate);
* using THP prevents sparse pages being freed/migrated effectively;
Memory fragmentation can be confirmed with kcompactd cpu-usage and examining the buddyinfo allocator
cat /proc/buddyinfo
increasingly more people will want to run inference of large models on their CPUs and not all of those models will fit in the RAM. Therefore we will see more users being hit by this and it will be beneficial to arch to find out which default setting could be changed to avoid this issue.
Disable THP with:
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
THP is great for latency (+10%) [1] but terrible for fragmentation [2] where memory pressure is high.
[1] https://alexandrnikitin.github.io/blog/ … ce-impact/
[2] https://www.pingcap.com/blog/transparen … databases/
Last edited by sabroad (2022-10-14 16:53:36)
--
saint_abroad
Offline
7000k wrote:Stock kernel (broken): https://drive.google.com/file/d/1omVM8R … sp=sharing
Custom kernel(working): https://drive.google.com/file/d/1pcEGlN … sp=sharingUnfortunately, Google requires login (or tracking cookies) to download.
Thanks for the reply.
In a sense all cookies are tracking cookies.
Personally I don't get what is the technical problem with allowing momentary tracking cookies in a private browsing session. One can open a private browsing session in any modern browser and cookies set in that session do not persist outside it.
However, if you(or anyone else) know of any alternative one can upload 350k text file and share it without cookies please do let me know.
7000k wrote:This is stock kernel(broken):
timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin pgpgout pswpin pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree pgactivate pgdeactivate pglazyfree pgfault pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail cma_alloc_success cma_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode 180 62704 1742969 11388 3484972 2761940 0 40 0 0 0 0 14500716 0 0 2448 14500716 0 1742969 11388 3484972 2761940 0 29966 25737 0 0 13905 148 6389469 99 4831303 2 10 0 1753986 43350 6247420 40 0 0 383 0 0 0 0 3340 1485104 0 16019 1501080 0 0 0 0 5040 6922 98 0 1249077 623776 27659569 5942960 104 1469719 1025 1233176 13300887 0 0 0 0 0 0 0 0 0 16357181 87154 2147272 0 578266 416 0 2152098 35944 8765311 0 0 0 12493291 0 0 5166041 7327250 1485067 7280244 0 0 8192 0 837 243 1084 1469682 0 0 0 0 0 0 0 0 5680 0 0 0 0 28430 98365 11878 0 0 0 43 28430 98365 0 0 0 0 6154 0 0 0 0 0 0 12763 21 0 40 0 0 0 0 0 0 0 3 2834 0 24 0 0 0 0 2831 0 0 0 0 110 67 0 0 44 15385 92 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0 210 66830 6203412 109528 5688 3380 0 25 0 0 0 0 24360205 0 0 2448 24360205 0 6203643 109528 5716 3380 0 30746 51385 0 0 58880 3452052 6935356 98482 5371579 2 2289 2112 6296023 2395 2121284 25 0 0 320 0 0 0 0 3575 6209290 53 17639 6226920 0 0 0 0 5168 18780 2111856 0 2051 1024 42579105 18307704 3450961 4559265 1025 2152313 22241749 0 0 0 167 12745 0 0 971591 0 26408120 117043 5471949 0 8735187 578255 0 6417191 37831 19446349 2161862 0 0 33081116 4143427 0 23058616 14165927 7548510 14059701 0 0 43448 1420 3682 302 3996 4558831 0 0 0 0 0 0 0 0 51095 22 0 0 0 195362 441148 107781 0 0 0 2078 195362 441148 0 0 0 0 6154 0 0 0 0 0 0 18768 7890 0 40 0 0 0 0 0 0 0 3 8604 0 40 0 0 0 0 8601 0 0 0 0 2874398 2874123 0 0 1256 1650044 92 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0
Looks like the system is under severe memory pressure, with writes/allocations whilst scanning pages to swap/free/compact:
nr_vmscan_write 6209290
7000k wrote:This is custom kernel(works)
timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin pgpgout pswpin pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree pgactivate pgdeactivate pglazyfree pgfault pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode 180 1941284 6101460 1754 10574 7294 0 33 0 0 0 0 27477729 0 0 2475 27477729 0 6101460 1754 10574 7294 0 20183 25558 346 0 59096 452279 4829242 433438 3157732 135 9998 2221 6102420 4326 32922 33 0 0 359 0 0 0 0 0 2872836 41 29848 2902645 0 0 0 0 5448 14242 14513 0 378605 189071 26394549 11610564 452368 2872836 1025 2699131 24813240 0 0 0 357 36326 0 0 2609918 0 30715851 2616392 4460740 0 14971905 84650 0 4990531 53544 11111093 3224881 0 0 17704401 10988213 0 15077092 13615522 2846440 11489534 0 0 62557 4431 2331 73 2410 2843630 0 0 0 0 0 0 0 0 1208170 26 0 0 0 1461470 3973865 2435329 0 0 0 1333 248286 994284 0 0 6033 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 368204 367106 0 0 0 0 121 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0 210 990735 6992475 30158 37868 10811 0 46 0 0 0 0 28901242 0 0 2475 28901242 0 6992475 30158 37868 10811 0 20179 25626 0 0 58636 1277506 4853361 461842 3160455 135 12396 2221 7021408 9497 886594 46 0 0 359 0 0 0 0 0 2872836 41 31268 2904052 0 0 0 0 5520 14454 837555 0 194702 97232 29814053 11616212 1277506 2872836 1025 2699131 26236751 0 0 0 357 36326 0 0 2609918 0 31546120 2616964 4460740 0 16368153 916648 0 4990531 55287 11111093 3224881 0 0 17704401 10988213 0 15077092 13615522 2846440 11489534 0 0 62557 4431 2331 73 2410 2843654 0 0 0 0 0 0 0 0 1564842 26 0 0 0 2141187 6714068 3147752 0 0 0 1333 248286 994284 0 0 6033 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 370178 368905 0 0 0 0 121 0 0 8192 1 20 0 10 0 3000 20 500 43200 500 0 0 0 0 256 256 32 0 0 65530 0 1 67584 5 1 65536 28 8 0 0 0 1 Node 1 0 0 0 50 3 5 0 0 1 60 1 131072 100 15000 10 0
With the custom config, the system is under memory pressure, but less so:
nr_vmscan_write 2872836
Let's narrow the largest differences:
vmstat vanilla custom compact_free_scanned 441148 6714068 # custom config is freeing >10x more pages for contiguous allocation compact_isolated 107781 3147752 compact_migrate_scanned 195362 2141187 # custom config is migrating >10x more pages for contiguous allocation nr_anon_transparent_hugepages 3575 0 nr_dirty_background_threshold 1024 97232 nr_dirty_threshold 2051 194702 nr_free_pages 66830 990735 pgactivate 117043 2616964 # custom config is reactivating pages that have gone inactive pgmigrate_success 51095 1564842 swap_ra_hit 2874123 368905 thp_collapse_alloc 40 0 thp_deferred_split_page 3 0 thp_fault_alloc 18768 0 # custom config is not allocating transparent huge pages thp_fault_fallback 7890 0 thp_scan_exceed_swap_pte 40 0 thp_split_pmd 8604 0 thp_swpout 8601 0 workingset_restore_anon 2 135 zswpin 1256 0 zswpout 1650044 0
With this, we can conclude:
* specific workload allocates memory in chunks (contiguous allocation);
* memory pressure is freeing sparse pages (causing fragmentation);
* specific workload does not access memory contiguously (allowing sparse pages to deactivate);
* using THP prevents sparse pages being freed/migrated effectively;Memory fragmentation can be confirmed with kcompactd cpu-usage and examining the buddyinfo allocator
cat /proc/buddyinfo
7000k wrote:increasingly more people will want to run inference of large models on their CPUs and not all of those models will fit in the RAM. Therefore we will see more users being hit by this and it will be beneficial to arch to find out which default setting could be changed to avoid this issue.
Disable THP with:
echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag
THP is great for latency (+10%) [1] but terrible for fragmentation [2] where memory pressure is high.
I'll give that a try. In the meantime new kernel came out. I'm just in a process of updating so I hope to test the default config on it too.
Offline
However, if you(or anyone else) know of any alternative one can upload 350k text file and share it without cookies please do let me know.
1st link below.
quota is around 1MB or so, so not a problem.
Offline
Unfortunately there is no difference on kernel 6.0.2.
It turns out it is ZSWAP. Disabling it at runtime makes the process succeed rather than get stuck. However when transparent_hugepage is on it is close to 40% slower than on my custom (based on ubuntu) kernel config. Just disabling transparent_hugepage after ZSWAP was off by writing to /proc files helped a little, but what really helped was to disable both in a specific sequence:
echo 0 > /sys/module/zswap/parameters/enabled
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
swapoff --all
sleep 1
swapon --all
Following this the inference is actually a little faster on 6.0.2 with stock arch settings(up to 10% - the majority of the speedup comes from faster loading from disk it appears).
This raises a question of having those settings as on by default. I'm glad ubuntu had them off as the discrepancy between the results helped in discovering it is a config setting issue.
However, if you(or anyone else) know of any alternative one can upload 350k text file and share it without cookies please do let me know.
1st link below.
quota is around 1MB or so, so not a problem.
The first link is pastebin right? If so, then it doesn't work for very long text lines. I tried before. There is a limit of some sort that stops one from pasting very long lines.
Offline
The first link is pastebin right?
Instead of guessing you could just look, but oh, well:
cat file | curl -F 'file=@-' 0x0.st
Do you get better behavior w/ different compressor, max_pool or accept_threshold?
https://wiki.archlinux.org/title/Zswap# … parameters
Offline
The first link is pastebin right?
Instead of guessing you could just look, but oh, well:
Seriously? Do you think I just guessed it without checking? Very funny :-) Also I did mention another limitation of pastebin in my very first email in this thread. Anyway.
cat file | curl -F 'file=@-' 0x0.st
Thanks. Bookmarked. Any idea how long files actually stay there? There is a "smart" equation there, which in absence of definition of its variables is as useful as the ascii drawing that seems to go to infinity below 1mb of file size :-) Still I plan to use it to test it myself.
Do you get better behavior w/ different compressor, max_pool or accept_threshold?
https://wiki.archlinux.org/title/Zswap# … parameters
No idea, at this point I disabled it altogether. TBH I see little value in this feature in the age of fast nvme storage and plenty of RAM. If I'm running a workload that consumes all of my RAM and more it usually constrains the CPU too. I'd rather it uses my 100k iops low latency nvme storage than attempt to save few extra % of RAM by taxing the cpu even more.
I was surprised this feature (zswap_default_on) is enabled in Arch as I never had it on by default on other distros I use so I did some research (looking for some discussion where it was decided to enable it, benchmark results etc. ) and all I found is this fairly recent thread on the Debian mailing list where they discuss a request to enable it in Debian: https://bugs.debian.org/cgi-bin/bugrepo … bug=970639 The thread is quite enlightenimg in general, but I failed to find any such discussion for Arch.
It appears the feature was considered experimental when it was added in kernel 3. Then around Linux kernel 5.7 the note about its experimental nature was removed despite there being very little published info about its cost/benefit in various configurations and for various workloads (often quoted page here https://linuxreviews.org/Zswap hilariously doesn't contain a comparison for the default compression type, doesn't mention storage type in the test system nor it tries few storage device types).
So for me the best is to disable it. In absence of comprehensive benchmarks and in light of the fact the feature has (documented) potential to severely impact performance in negative way I would prefer to not have it on by default in Arch. Now that I know it is what caused my problem in the first place I might log a bug at arch bug tracker. Perhaps this will make a larger conversion about it happen.
Offline
Seriously? Do you think I just guessed it without checking?
Yes.
Note: An acceptable pastebin service does not require enabling JavaScript for viewing, does not display adverts, manipulate the pasted content or require a login. pastebin.com is blocked for some people because of malware found on the site and has a history of annoying issues (requires JavaScript, displays adverts, inserts CRLF line-endings and displaying CAPTCHAs at random). Do not use it.
Anyway.
Any idea how long files actually stay there?
I might log a bug at arch bug tracker. Perhaps this will make a larger conversion about it happen.
The needs of the many… but there should certainly be some notes in the wiki since you've found at least one access pattern where zswap is detrimental.
Offline
Seriously? Do you think I just guessed it without checking?
Yes.
Hillarious
The linked wiki wrote:Note: An acceptable pastebin service does not require enabling JavaScript for viewing, does not display adverts, manipulate the pasted content or require a login. pastebin.com is blocked for some people because of malware found on the site and has a history of annoying issues (requires JavaScript, displays adverts, inserts CRLF line-endings and displaying CAPTCHAs at random). Do not use it.
I'm not sure how this is relevant. Pastebin is a generic term (not just referring to pastebin.com). I've tried to post the same thing using paste.debian.net and I hit a line length,and a paste length limits a lot sooner there too.
Any idea how long files actually stay there?
I'll quote from my previous post in case it was missed:
There is a "smart" equation there, which in absence of definition of its variables is as useful as the ascii drawing that seems to go to infinity below 1mb of file size :-)
And then to clarify further.
For the equation to have any sense values for min_age max_age as well as max size have to be known.
I might log a bug at arch bug tracker. Perhaps this will make a larger conversion about it happen.
The needs of the many… but there should certainly be some notes in the wiki since you've found at least one access pattern where zswap is detrimental.
I doubt very much zswap is beneficial "for the many" and is detrimental in some edge cases. In absence of evidence either way we simply don't know.
Also what I encountered it is not the only access pattern where zswap is detrimental. In the article people often point to as evidence of zswap benefits, in addition to 3 positive ones there are shown 3 scenarios when it is detrimental. This is the site.
https://linuxreviews.org/Zswap Also, Interestingly, the default compression mechanism wasn't even tested.
I think it is the other way around. Zswap can be beneficial in some edge cases (very slow storage, just a little bit of ram constrain), but in the majority of cases in fairly modern systems it is probably not. I'm open to be convinced otherwise if there is some evidence I don't know about.
Offline
seth wrote:7000k wrote:Any idea how long files actually stay there?
I'll quote from my previous post in case it was missed:
There is a "smart" equation there, which in absence of definition of its variables is as useful as the ascii drawing that seems to go to infinity below 1mb of file size :-)And then to clarify further.
For the equation to have any sense values for min_age max_age as well as max size have to be known.
File URLs are valid for at least 30 days and up to a year (see below).
Maximum file size: 512.0 MiB
CLI Paste | How To Ask Questions
Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L
Offline
Zswap can be beneficial in some edge cases (very slow storage, just a little bit of ram constrain), but in the majority of cases in fairly modern systems it is probably not.
There are a *lot* of 24/7 server workloads out there with fast SSDs and heavy memory requirements. Given the tradeoff between burning CPUs (10 year lifespan) and burning SSDs (1 month lifespan at 500MB/s) many would prefer the former as the safer default.
--
saint_abroad
Offline
7000k wrote:Zswap can be beneficial in some edge cases (very slow storage, just a little bit of ram constrain), but in the majority of cases in fairly modern systems it is probably not.
There are a *lot* of 24/7 server workloads out there with fast SSDs and heavy memory requirements. Given the tradeoff between burning CPUs (10 year lifespan) and burning SSDs (1 month lifespan at 500MB/s) many would prefer the former as the safer default.
Wouldn't that disfavour making zswap default? Or am I not following the logic here correctly?
CLI Paste | How To Ask Questions
Arch Linux | x86_64 | GPT | EFI boot | refind | stub loader | systemd | LVM2 on LUKS
Lenovo x270 | Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz | Intel Wireless 8265/8275 | US keyboard w/ Euro | 512G NVMe INTEL SSDPEKKF512G7L
Offline