Weird issue with arch default kernel config and swap memory.

7000k · 2022-09-29 09:26:35

Hi, this is my first post here so let it be my introduction. I'm a fairly new arch user, but I've been using Linux (other distros) for a long time. Skip to the bold text below to get to the point

First some introductory info.
Most recently I've used Ubuntu and then Guix, but I wanted to give Arch a try too so I installed the system, I configured it with btrfs root and a btrfs /home (including automatic snapshots triggered by pacman update and scheduled snapshots of home with snapper). I use gnome desktop and nVidia drivers. In general everything works pretty well.

My cpu is AMD ryzen 7 3700x with 32gb of ddr 4 ram and a 64GB swap partition on an nvme drive. The root and home are btrfs on the same nvme drive, there is an extra tmp nvme device that uses ext4 on a separate nvme drive. I also have a couple of 4TB spinning disk's in a mirror as my big file storage. (this will be relevant later)

Everything, but one very significant issue I discovered. Specifically, I often run AI models I download from various places, this time I've been using a 7.1B parameter model called bloom. It is too big to fit on my 8GB rtx card so I run it on the CPU.

This model requires 4*7.1B+some of RAM(definitely more than 32GB), but I only have 32GB. So I've been using an offload folder on the nvme tmp drive and a swap partition on another nvme drive (same as root +home).

This has run fine on my ubuntu 20.04 with Linux kernel 5.13 (and a custom kernel 5.15 with same config) as well as when I booted Ubuntu live cd with 5.15 kernel. Same on Guix with default (nonguix) Linux kernel 5.19. However, it gets stuck on Arch Linux on the current linux (5.19) and linux-lts (5.15) as well as when run from Arch live cd. Additionally I took the kernel config from Ubuntu and I compiled current arch Linux kernel with it. It runs fine on the resulting custom kernel on arch. All tests were done on two same venv's one with python 3. 8 one with 3.10

This leads me to believe the cause is some kernel configuration option. I run a diff and I tried to compare both kernel configs, but they differ so much it difficult to say which option causes it. I have few candidates on my mind, but first let's describe the issue in more detail.

When the model inference runs fine it takes almost exactly 3minutes to load the model from the HDD mirror into ram+swap. It allocates all of ram and few GB of swap (up to ~6GB). Then itt takes 80s~100s to run and it spits its response. During those ~80s there is a brief burst of IO activity at the beginning on the swap drive (about 100k iops 450MB/s lasting ~2s) then disk's do nothing and the rest of inference runs from RAM (cpu utilisation in top goes to 700%).at the end all memory is freed.

When it fails (on Arch with default kernel config) it takes same 3minutes to load the model into RAM+swap, but it seems to allocate more of swap (~10GB) then instead of a brief burst of high IO activity it 'settles' into a very long time when the swap drive sees about 20k IOPS and 150~200MB/s and it stays like that. I once have it 20 min, but it still was stuck the same way. When interrupted in this state it always stops during running of a forward inference pass.

As mentioned all tests were run on exact same virtual environments of python 3. 8 and 3.10(both were tested on all kernels). Prior to using venv I also used conda environment deployed with the same specification (same versions of libraries etc). I've tried replacing my btrfs /home with an ext4 home (just in case). I tried running from a live cd replacing my rootfs.

All this leads me to believe there is some kernel option that causes it, but there are too many to try them one by one. Therefore I'm starting this thread. Perhaps someone who knows the difference between the default and arch kernel config very well reads it and has an idea that it is this option or that. I'm willing to try them to see.

Running large AI models is an increasing use case for a Linux desktop so IMO it is important a distro should support it out of the box. This is why I'm publishing this rather than just running my custom kernel and forgetting about it.

ugjka · 2022-09-29 09:32:42

Buy more RAM, swap is not extra ram

sabroad · 2022-09-29 11:18:49

7000k wrote:

All this leads me to believe there is some kernel option that causes it, but there are too many to try them one by one.

Indeed there are.

Let's get some data:

cat /proc/vmstat

Please post the full output twice 30s apart on vanilla kernel during an inference pass.

Last edited by sabroad (2022-09-29 11:20:50)

seth · 2022-09-29 14:40:39

Also diff'ing

sudo sysctl -a | grep vm

between kernels might provide some insight.

7000k · 2022-09-30 16:13:08

Thank you for the replies. I can't reboot to the stock kernel until my PC finishes what its doing so I'll run the collection of data late in the evening (CET) or tomorrow.

I wasn't sure if just running those commands 30s apart would catch what's needed so I wrote a script that will execute both commands every second and save the values they give to a CSV file (timestamped). So hopefully this will be much better.

7000k · 2022-09-30 22:03:36

Well, I run both data captures and I can clearly see that on the custom kernel it works it seems to get to few GB of swap use, then it does its thing and it finishes. With stock kernel however, it appears to diverge at around 2.5~3min mark. It continuously grows swap size larger and larger. I run it for 7 minutes this time while collecting data. There are 223 data points per second (per kernel) so my attempts to plot the differences were not very successful (I did plot them, but with so much data the plots were a mess). I hope this will be of use.

I can't figure out how to attach files on this forum so I'm submitting them via pastebin and file.io - I post both, because pastebin cuts the files at 300s mark (not much happens after, but for the sake of completeness I kept the stuck capture on until 7 min mark).
This is the stock Arch kernel (fails): https://pastebin.com/Uude2vfe https://file.io/gPb1GuosvWE7
This is the custom kernel (works): https://pastebin.com/HSWNpiju https://file.io/ricYUv1fSeb1

Both were captures on a freshly booted system, no X11, no other software running other than iostat, top, screen and the python inference. The CSV files are created by running cat /proc/vmstat and sudo sysctl -A | grep vm, then taking the resulting data and putting it in one row of a CSV file, repeat every 1 second.

Additionally my custom kernel (that works) doesn't contain all of ubuntu's settings. Basically I went through the diff between Arch and ubuntu defaults and I applied desktop preempt configuration as well as anything that based by name could have anything to do with memory or swap.

This is my custom kernel config: https://pastebin.com/ZJien8re https://file.io/ricYUv1fSeb1

I'll continue looking at this data too, but so far I just found symptoms of an issue rather than their cause...

seth · 2022-10-01 07:54:48

The vm config won't change but seems equal between kernels and the config uses CRLF… notepad?
Anyway, kernel config diff:

--- config_arch.txt	2022-10-01 09:52:19.014278897 +0200
+++ config_ubuntu.txt	2022-10-01 09:51:57.974197432 +0200
@@ -14,11 +14,7 @@ CONFIG_LLD_VERSION=0
 CONFIG_CC_CAN_LINK=y
 CONFIG_CC_CAN_LINK_STATIC=y
 CONFIG_CC_HAS_ASM_GOTO=y
-CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y
-CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y
 CONFIG_CC_HAS_ASM_INLINE=y
-CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y
-CONFIG_PAHOLE_VERSION=123
 CONFIG_IRQ_WORK=y
 CONFIG_BUILDTIME_TABLE_SORT=y
 CONFIG_THREAD_INFO_IN_TASK=y
@@ -28,7 +24,6 @@ CONFIG_THREAD_INFO_IN_TASK=y
 #
 CONFIG_INIT_ENV_ARG_LIMIT=32
 # CONFIG_COMPILE_TEST is not set
-# CONFIG_WERROR is not set
 CONFIG_LOCALVERSION=""
 CONFIG_LOCALVERSION_AUTO=y
 CONFIG_BUILD_SALT=""
@@ -50,7 +45,6 @@ CONFIG_DEFAULT_INIT=""
 CONFIG_DEFAULT_HOSTNAME="archlinux"
 CONFIG_SYSVIPC=y
 CONFIG_SYSVIPC_SYSCTL=y
-CONFIG_SYSVIPC_COMPAT=y
 CONFIG_POSIX_MQUEUE=y
 CONFIG_POSIX_MQUEUE_SYSCTL=y
 CONFIG_WATCH_QUEUE=y
@@ -68,11 +62,9 @@ CONFIG_GENERIC_IRQ_SHOW=y
 CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK=y
 CONFIG_GENERIC_PENDING_IRQ=y
 CONFIG_GENERIC_IRQ_MIGRATION=y
-CONFIG_GENERIC_IRQ_INJECTION=y
 CONFIG_HARDIRQS_SW_RESEND=y
 CONFIG_GENERIC_IRQ_CHIP=y
 CONFIG_IRQ_DOMAIN=y
-CONFIG_IRQ_SIM=y
 CONFIG_IRQ_DOMAIN_HIERARCHY=y
 CONFIG_GENERIC_MSI_IRQ=y
 CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
@@ -101,13 +93,10 @@ CONFIG_POSIX_CPU_TIMERS_TASK_WORK=y
 CONFIG_TICK_ONESHOT=y
 CONFIG_NO_HZ_COMMON=y
 # CONFIG_HZ_PERIODIC is not set
-# CONFIG_NO_HZ_IDLE is not set
-CONFIG_NO_HZ_FULL=y
-CONFIG_CONTEXT_TRACKING=y
-# CONFIG_CONTEXT_TRACKING_FORCE is not set
+CONFIG_NO_HZ_IDLE=y
+# CONFIG_NO_HZ_FULL is not set
 CONFIG_NO_HZ=y
 CONFIG_HIGH_RES_TIMERS=y
-CONFIG_CLOCKSOURCE_WATCHDOG_MAX_SKEW_US=100
 # end of Timers subsystem
 
 CONFIG_BPF=y
@@ -123,19 +112,13 @@ CONFIG_BPF_JIT_ALWAYS_ON=y
 CONFIG_BPF_JIT_DEFAULT_ON=y
 CONFIG_BPF_UNPRIV_DEFAULT_OFF=y
 CONFIG_USERMODE_DRIVER=y
-CONFIG_BPF_PRELOAD=y
-CONFIG_BPF_PRELOAD_UMD=m
+#CONFIG_BPF_PRELOAD is not set
 CONFIG_BPF_LSM=y
 # end of BPF subsystem
 
-CONFIG_PREEMPT_BUILD=y
 # CONFIG_PREEMPT_NONE is not set
-# CONFIG_PREEMPT_VOLUNTARY is not set
-CONFIG_PREEMPT=y
-CONFIG_PREEMPT_COUNT=y
-CONFIG_PREEMPTION=y
-CONFIG_PREEMPT_DYNAMIC=y
-CONFIG_SCHED_CORE=y
+CONFIG_PREEMPT_VOLUNTARY=y
+#CONFIG_PREEMPT is not set
 
 #
 # CPU/Task time and stats accounting
@@ -143,7 +126,6 @@ CONFIG_SCHED_CORE=y
 CONFIG_VIRT_CPU_ACCOUNTING=y
 CONFIG_VIRT_CPU_ACCOUNTING_GEN=y
 CONFIG_IRQ_TIME_ACCOUNTING=y
-CONFIG_HAVE_SCHED_AVG_IRQ=y
 CONFIG_BSD_PROCESS_ACCT=y
 CONFIG_BSD_PROCESS_ACCT_V3=y
 CONFIG_TASKSTATS=y
@@ -160,36 +142,22 @@ CONFIG_CPU_ISOLATION=y
 # RCU Subsystem
 #
 CONFIG_TREE_RCU=y
-CONFIG_PREEMPT_RCU=y
-CONFIG_RCU_EXPERT=y
+# CONFIG_RCU_EXPERT is not set
 CONFIG_SRCU=y
 CONFIG_TREE_SRCU=y
 CONFIG_TASKS_RCU_GENERIC=y
-# CONFIG_FORCE_TASKS_RCU is not set
-CONFIG_TASKS_RCU=y
-# CONFIG_FORCE_TASKS_RUDE_RCU is not set
 CONFIG_TASKS_RUDE_RCU=y
-# CONFIG_FORCE_TASKS_TRACE_RCU is not set
 CONFIG_TASKS_TRACE_RCU=y
 CONFIG_RCU_STALL_COMMON=y
 CONFIG_RCU_NEED_SEGCBLIST=y
-CONFIG_RCU_FANOUT=64
-CONFIG_RCU_FANOUT_LEAF=16
-CONFIG_RCU_BOOST=y
-CONFIG_RCU_BOOST_DELAY=500
-# CONFIG_RCU_EXP_KTHREAD is not set
-CONFIG_RCU_NOCB_CPU=y
-# CONFIG_TASKS_TRACE_RCU_READ_MB is not set
 # end of RCU Subsystem
 
 CONFIG_BUILD_BIN2C=y
 CONFIG_IKCONFIG=y
-CONFIG_IKCONFIG_PROC=y
 CONFIG_IKHEADERS=m
 CONFIG_LOG_BUF_SHIFT=17
 CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
 CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13
-CONFIG_PRINTK_INDEX=y
 CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
 
 #
@@ -202,9 +170,6 @@ CONFIG_UCLAMP_BUCKETS_COUNT=5
 CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
 CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y
 CONFIG_CC_HAS_INT128=y
-CONFIG_CC_IMPLICIT_FALLTHROUGH="-Wimplicit-fallthrough=5"
-CONFIG_GCC12_NO_ARRAY_BOUNDS=y
-CONFIG_CC_NO_ARRAY_BOUNDS=y
 CONFIG_ARCH_SUPPORTS_INT128=y
 CONFIG_NUMA_BALANCING=y
 CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
@@ -238,7 +203,6 @@ CONFIG_UTS_NS=y
 CONFIG_TIME_NS=y
 CONFIG_IPC_NS=y
 CONFIG_USER_NS=y
-CONFIG_USER_NS_UNPRIVILEGED=y
 CONFIG_PID_NS=y
 CONFIG_NET_NS=y
 CONFIG_CHECKPOINT_RESTORE=y
@@ -255,8 +219,6 @@ CONFIG_RD_LZO=y
 CONFIG_RD_LZ4=y
 CONFIG_RD_ZSTD=y
 CONFIG_BOOT_CONFIG=y
-# CONFIG_BOOT_CONFIG_EMBED is not set
-CONFIG_INITRAMFS_PRESERVE_MTIME=y
 CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
 # CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
 CONFIG_LD_ORPHAN_WARN=y
@@ -272,6 +234,7 @@ CONFIG_SYSFS_SYSCALL=y
 CONFIG_FHANDLE=y
 CONFIG_POSIX_TIMERS=y
 CONFIG_PRINTK=y
+CONFIG_PRINTK_NMI=y
 CONFIG_BUG=y
 CONFIG_ELF_CORE=y
 CONFIG_PCSPKR_PLATFORM=y
@@ -286,14 +249,18 @@ CONFIG_SHMEM=y
 CONFIG_AIO=y
 CONFIG_IO_URING=y
 CONFIG_ADVISE_SYSCALLS=y
+CONFIG_HAVE_ARCH_USERFAULTFD_WP=y
+CONFIG_HAVE_ARCH_USERFAULTFD_MINOR=y
 CONFIG_MEMBARRIER=y
 CONFIG_KALLSYMS=y
 CONFIG_KALLSYMS_ALL=y
 CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
 CONFIG_KALLSYMS_BASE_RELATIVE=y
+CONFIG_USERFAULTFD=y
 CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE=y
 CONFIG_KCMP=y
 CONFIG_RSEQ=y
+# CONFIG_DEBUG_RSEQ is not set
 # CONFIG_EMBEDDED is not set
 CONFIG_HAVE_PERF_EVENTS=y
 CONFIG_GUEST_PERF_EVENTS=y
@@ -305,6 +272,17 @@ CONFIG_PERF_EVENTS=y
 # CONFIG_DEBUG_PERF_USE_VMALLOC is not set
 # end of Kernel Performance Events And Counters
 
+CONFIG_VM_EVENT_COUNTERS=y
+CONFIG_SLUB_DEBUG=y
+# CONFIG_COMPAT_BRK is not set
+# CONFIG_SLAB is not set
+CONFIG_SLUB=y
+# CONFIG_SLOB is not set
+CONFIG_SLAB_MERGE_DEFAULT=y
+CONFIG_SLAB_FREELIST_RANDOM=y
+CONFIG_SLAB_FREELIST_HARDENED=y
+CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
+CONFIG_SLUB_CPU_PARTIAL=y
 CONFIG_SYSTEM_DATA_VERIFICATION=y
 CONFIG_PROFILING=y
 CONFIG_TRACEPOINTS=y
@@ -328,9 +306,14 @@ CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
 CONFIG_ARCH_MAY_HAVE_PC_FDC=y
 CONFIG_GENERIC_CALIBRATE_DELAY=y
 CONFIG_ARCH_HAS_CPU_RELAX=y
+CONFIG_ARCH_HAS_FILTER_PGPROT=y
+CONFIG_HAVE_SETUP_PER_CPU_AREA=y
+CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
+CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
 CONFIG_ARCH_HIBERNATION_POSSIBLE=y
-CONFIG_ARCH_NR_GPIO=1024
 CONFIG_ARCH_SUSPEND_POSSIBLE=y
+CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
+CONFIG_ZONE_DMA32=y
 CONFIG_AUDIT_ARCH=y
 CONFIG_HAVE_INTEL_TXT=y
 CONFIG_X86_64_SMP=y
@@ -343,13 +326,20 @@ CONFIG_CC_HAS_SANE_STACKPROTECTOR=y
 #
 # Processor type and features
 #
+CONFIG_ZONE_DMA=y
 CONFIG_SMP=y
 CONFIG_X86_FEATURE_NAMES=y
 CONFIG_X86_X2APIC=y
 CONFIG_X86_MPPARSE=y
 # CONFIG_GOLDFISH is not set
+CONFIG_RETPOLINE=y
 CONFIG_X86_CPU_RESCTRL=y
-# CONFIG_X86_EXTENDED_PLATFORM is not set
+CONFIG_X86_EXTENDED_PLATFORM=y
+CONFIG_X86_NUMACHIP=y
+# CONFIG_X86_VSMP is not set
+CONFIG_X86_UV=y
+# CONFIG_X86_GOLDFISH is not set
+# CONFIG_X86_INTEL_MID is not set
 CONFIG_X86_INTEL_LPSS=y
 CONFIG_X86_AMD_PLATFORM_DEVICE=y
 CONFIG_IOSF_MBI=y
@@ -373,7 +363,6 @@ CONFIG_XEN_PVHVM_GUEST=y
 CONFIG_XEN_SAVE_RESTORE=y
 # CONFIG_XEN_DEBUG_FS is not set
 CONFIG_XEN_PVH=y
-CONFIG_XEN_DOM0=y
 CONFIG_KVM_GUEST=y
 CONFIG_ARCH_CPUIDLE_HALTPOLL=y
 CONFIG_PVH=y
@@ -381,7 +370,6 @@ CONFIG_PARAVIRT_TIME_ACCOUNTING=y
 CONFIG_PARAVIRT_CLOCK=y
 CONFIG_JAILHOUSE_GUEST=y
 CONFIG_ACRN_GUEST=y
-CONFIG_INTEL_TDX_GUEST=y
 # CONFIG_MK8 is not set
 # CONFIG_MPSC is not set
 # CONFIG_MCORE2 is not set
@@ -396,6 +384,7 @@ CONFIG_X86_MINIMUM_CPU_FAMILY=64
 CONFIG_X86_DEBUGCTLMSR=y
 CONFIG_IA32_FEAT_CTL=y
 CONFIG_X86_VMX_FEATURE_NAMES=y
+CONFIG_PROCESSOR_SELECT=y
 CONFIG_CPU_SUP_INTEL=y
 CONFIG_CPU_SUP_AMD=y
 CONFIG_CPU_SUP_HYGON=y
@@ -404,14 +393,13 @@ CONFIG_CPU_SUP_ZHAOXIN=y
 CONFIG_HPET_TIMER=y
 CONFIG_HPET_EMULATE_RTC=y
 CONFIG_DMI=y
-# CONFIG_GART_IOMMU is not set
+CONFIG_GART_IOMMU=y
 CONFIG_BOOT_VESA_SUPPORT=y
-# CONFIG_MAXSMP is not set
-CONFIG_NR_CPUS_RANGE_BEGIN=2
-CONFIG_NR_CPUS_RANGE_END=512
-CONFIG_NR_CPUS_DEFAULT=64
-CONFIG_NR_CPUS=320
-CONFIG_SCHED_CLUSTER=y
+CONFIG_MAXSMP=y
+CONFIG_NR_CPUS_RANGE_BEGIN=8192
+CONFIG_NR_CPUS_RANGE_END=8192
+CONFIG_NR_CPUS_DEFAULT=8192
+CONFIG_NR_CPUS=8192
 CONFIG_SCHED_SMT=y
 CONFIG_SCHED_MC=y
 CONFIG_SCHED_MC_PRIO=y
@@ -440,6 +428,7 @@ CONFIG_X86_16BIT=y
 CONFIG_X86_ESPFIX64=y
 CONFIG_X86_VSYSCALL_EMULATION=y
 CONFIG_X86_IOPL_IOPERM=y
+CONFIG_I8K=m
 CONFIG_MICROCODE=y
 CONFIG_MICROCODE_INTEL=y
 CONFIG_MICROCODE_AMD=y
@@ -448,8 +437,7 @@ CONFIG_X86_MSR=y
 CONFIG_X86_CPUID=y
 CONFIG_X86_5LEVEL=y
 CONFIG_X86_DIRECT_GBPAGES=y
-CONFIG_X86_CPA_STATISTICS=y
-CONFIG_X86_MEM_ENCRYPT=y
+# CONFIG_X86_CPA_STATISTICS is not set
 CONFIG_AMD_MEM_ENCRYPT=y
 # CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT is not set
 CONFIG_NUMA=y
@@ -459,13 +447,15 @@ CONFIG_X86_64_ACPI_NUMA=y
 CONFIG_NODES_SHIFT=5
 CONFIG_ARCH_SPARSEMEM_ENABLE=y
 CONFIG_ARCH_SPARSEMEM_DEFAULT=y
-# CONFIG_ARCH_MEMORY_PROBE is not set
+CONFIG_ARCH_SELECT_MEMORY_MODEL=y
+CONFIG_ARCH_MEMORY_PROBE=y
 CONFIG_ARCH_PROC_KCORE_TEXT=y
 CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
 CONFIG_X86_PMEM_LEGACY_DEVICE=y
 CONFIG_X86_PMEM_LEGACY=m
 CONFIG_X86_CHECK_BIOS_CORRUPTION=y
 CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
+CONFIG_X86_RESERVE_LOW=64
 CONFIG_MTRR=y
 CONFIG_MTRR_SANITIZER=y
 CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
@@ -473,9 +463,8 @@ CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAU
 CONFIG_X86_PAT=y
 CONFIG_ARCH_USES_PG_UNCACHED=y
 CONFIG_ARCH_RANDOM=y
+CONFIG_X86_SMAP=y
 CONFIG_X86_UMIP=y
-CONFIG_CC_HAS_IBT=y
-CONFIG_X86_KERNEL_IBT=y
 CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=y
 # CONFIG_X86_INTEL_TSX_MODE_OFF is not set
 # CONFIG_X86_INTEL_TSX_MODE_ON is not set
@@ -485,8 +474,8 @@ CONFIG_EFI=y
 CONFIG_EFI_STUB=y
 CONFIG_EFI_MIXED=y
 # CONFIG_HZ_100 is not set
-# CONFIG_HZ_250 is not set
-CONFIG_HZ_300=y
+CONFIG_HZ_250=y
+# CONFIG_HZ_300 uis not set
 # CONFIG_HZ_1000 is not set
 CONFIG_HZ=300
 CONFIG_SCHED_HRTICK=y
@@ -510,27 +499,18 @@ CONFIG_HOTPLUG_CPU=y
 # CONFIG_BOOTPARAM_HOTPLUG_CPU0 is not set
 # CONFIG_DEBUG_HOTPLUG_CPU0 is not set
 # CONFIG_COMPAT_VDSO is not set
+# CONFIG_LEGACY_VSYSCALL_EMULATE is not set
 CONFIG_LEGACY_VSYSCALL_XONLY=y
 # CONFIG_LEGACY_VSYSCALL_NONE is not set
 # CONFIG_CMDLINE_BOOL is not set
 CONFIG_MODIFY_LDT_SYSCALL=y
-# CONFIG_STRICT_SIGALTSTACK_SIZE is not set
 CONFIG_HAVE_LIVEPATCH=y
 # CONFIG_LIVEPATCH is not set
 # end of Processor type and features
 
-CONFIG_CC_HAS_SLS=y
-CONFIG_CC_HAS_RETURN_THUNK=y
-CONFIG_SPECULATION_MITIGATIONS=y
-CONFIG_PAGE_TABLE_ISOLATION=y
-CONFIG_RETPOLINE=y
-CONFIG_RETHUNK=y
-CONFIG_CPU_UNRET_ENTRY=y
-CONFIG_CPU_IBPB_ENTRY=y
-CONFIG_CPU_IBRS_ENTRY=y
-CONFIG_SLS=y
 CONFIG_ARCH_HAS_ADD_PAGES=y
 CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=y
+CONFIG_USE_PERCPU_NUMA_NODE_ID=y
 
 #
 # Power management and ACPI options
@@ -538,6 +518,7 @@ CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=
 CONFIG_ARCH_HIBERNATION_HEADER=y
 CONFIG_SUSPEND=y
 CONFIG_SUSPEND_FREEZER=y
+# CONFIG_SUSPEND_SKIP_SYNC is not set
 CONFIG_HIBERNATE_CALLBACKS=y
 CONFIG_HIBERNATION=y
 CONFIG_HIBERNATION_SNAPSHOT_DEV=y
@@ -551,6 +532,7 @@ CONFIG_PM_DEBUG=y
 # CONFIG_PM_ADVANCED_DEBUG is not set
 # CONFIG_PM_TEST_SUSPEND is not set
 CONFIG_PM_SLEEP_DEBUG=y
+# CONFIG_DPM_WATCHDOG is not set
 CONFIG_PM_TRACE=y
 CONFIG_PM_TRACE_RTC=y
 CONFIG_PM_CLK=y
@@ -588,6 +570,7 @@ CONFIG_ACPI_HOTPLUG_CPU=y
 CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
 CONFIG_ACPI_THERMAL=y
 CONFIG_ACPI_PLATFORM_PROFILE=m
+CONFIG_ACPI_CUSTOM_DSDT_FILE=""
 CONFIG_ARCH_HAS_ACPI_TABLE_UPGRADE=y
 CONFIG_ACPI_TABLE_UPGRADE=y
 CONFIG_ACPI_DEBUG=y
@@ -599,6 +582,7 @@ CONFIG_ACPI_SBS=m
 CONFIG_ACPI_HED=y
 CONFIG_ACPI_CUSTOM_METHOD=m
 CONFIG_ACPI_BGRT=y
+# CONFIG_ACPI_REDUCED_HARDWARE_ONLY is not set
 CONFIG_ACPI_NFIT=m
 # CONFIG_NFIT_SECURITY_DEBUG is not set
 CONFIG_ACPI_NUMA=y
@@ -654,9 +638,9 @@ CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
 # CPU frequency scaling drivers
 #
 CONFIG_X86_INTEL_PSTATE=y
-CONFIG_X86_PCC_CPUFREQ=m
+CONFIG_X86_PCC_CPUFREQ=y
 CONFIG_X86_AMD_PSTATE=m
-CONFIG_X86_ACPI_CPUFREQ=m
+CONFIG_X86_ACPI_CPUFREQ=y
 CONFIG_X86_ACPI_CPUFREQ_CPB=y
 CONFIG_X86_POWERNOW_K8=m
 CONFIG_X86_AMD_FREQ_SENSITIVITY=m
@@ -724,6 +708,7 @@ CONFIG_KVM_XFER_TO_GUEST_WORK=y
 CONFIG_HAVE_KVM_PM_NOTIFIER=y
 CONFIG_VIRTUALIZATION=y
 CONFIG_KVM=m
+CONFIG_KVM_WERROR=y
 CONFIG_KVM_INTEL=m
 CONFIG_X86_SGX_KVM=y
 CONFIG_KVM_AMD=m
@@ -752,18 +737,14 @@ CONFIG_UPROBES=y
 CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
 CONFIG_ARCH_USE_BUILTIN_BSWAP=y
 CONFIG_KRETPROBES=y
-CONFIG_KRETPROBE_ON_RETHOOK=y
 CONFIG_USER_RETURN_NOTIFIER=y
 CONFIG_HAVE_IOREMAP_PROT=y
 CONFIG_HAVE_KPROBES=y
 CONFIG_HAVE_KRETPROBES=y
 CONFIG_HAVE_OPTPROBES=y
 CONFIG_HAVE_KPROBES_ON_FTRACE=y
-CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=y
 CONFIG_HAVE_FUNCTION_ERROR_INJECTION=y
 CONFIG_HAVE_NMI=y
-CONFIG_TRACE_IRQFLAGS_SUPPORT=y
-CONFIG_TRACE_IRQFLAGS_NMI_SUPPORT=y
 CONFIG_HAVE_ARCH_TRACEHOOK=y
 CONFIG_HAVE_DMA_CONTIGUOUS=y
 CONFIG_GENERIC_SMP_IDLE_THREAD=y
@@ -772,7 +753,6 @@ CONFIG_ARCH_HAS_SET_MEMORY=y
 CONFIG_ARCH_HAS_SET_DIRECT_MAP=y
 CONFIG_HAVE_ARCH_THREAD_STRUCT_WHITELIST=y
 CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y
-CONFIG_ARCH_WANTS_NO_INSTR=y
 CONFIG_HAVE_ASM_MODVERSIONS=y
 CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
 CONFIG_HAVE_RSEQ=y
@@ -788,7 +768,6 @@ CONFIG_HAVE_ARCH_JUMP_LABEL=y
 CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE=y
 CONFIG_MMU_GATHER_TABLE_FREE=y
 CONFIG_MMU_GATHER_RCU_TABLE_FREE=y
-CONFIG_MMU_GATHER_MERGE_VMAS=y
 CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
 CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
 CONFIG_HAVE_CMPXCHG_LOCAL=y
@@ -817,7 +796,6 @@ CONFIG_HAVE_MOVE_PMD=y
 CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
 CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=y
 CONFIG_HAVE_ARCH_HUGE_VMAP=y
-CONFIG_HAVE_ARCH_HUGE_VMALLOC=y
 CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
 CONFIG_HAVE_ARCH_SOFT_DIRTY=y
 CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
@@ -831,13 +809,6 @@ CONFIG_ARCH_MMAP_RND_BITS=28
 CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS=y
 CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
 CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES=y
-CONFIG_PAGE_SIZE_LESS_THAN_64KB=y
-CONFIG_PAGE_SIZE_LESS_THAN_256KB=y
-CONFIG_HAVE_OBJTOOL=y
-CONFIG_HAVE_JUMP_LABEL_HACK=y
-CONFIG_HAVE_NOINSTR_HACK=y
-CONFIG_HAVE_NOINSTR_VALIDATION=y
-CONFIG_HAVE_UACCESS_VALIDATION=y
 CONFIG_HAVE_STACK_VALIDATION=y
 CONFIG_HAVE_RELIABLE_STACKTRACE=y
 CONFIG_ISA_BUS_API=y
@@ -847,7 +818,6 @@ CONFIG_COMPAT_32BIT_TIME=y
 CONFIG_HAVE_ARCH_VMAP_STACK=y
 CONFIG_VMAP_STACK=y
 CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET=y
-CONFIG_RANDOMIZE_KSTACK_OFFSET=y
 CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT=y
 CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
 CONFIG_STRICT_KERNEL_RWX=y
@@ -861,14 +831,9 @@ CONFIG_ARCH_HAS_CC_PLATFORM=y
 CONFIG_HAVE_STATIC_CALL=y
 CONFIG_HAVE_STATIC_CALL_INLINE=y
 CONFIG_HAVE_PREEMPT_DYNAMIC=y
-CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y
 CONFIG_ARCH_WANT_LD_ORPHAN_WARN=y
 CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
-CONFIG_ARCH_SUPPORTS_PAGE_TABLE_CHECK=y
 CONFIG_ARCH_HAS_ELFCORE_COMPAT=y
-CONFIG_ARCH_HAS_PARANOID_L1D_FLUSH=y
-CONFIG_DYNAMIC_SIGFRAME=y
-CONFIG_HAVE_ARCH_NODE_DEV_GROUP=y
 
 #
 # GCOV-based kernel profiling
@@ -878,8 +843,6 @@ CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
 # end of GCOV-based kernel profiling
 
 CONFIG_HAVE_GCC_PLUGINS=y
-CONFIG_GCC_PLUGINS=y
-# CONFIG_GCC_PLUGIN_LATENT_ENTROPY is not set
 # end of General architecture-dependent options
 
 CONFIG_RT_MUTEXES=y
@@ -907,23 +870,22 @@ CONFIG_MODULE_SIG_HASH="sha512"
 CONFIG_MODULE_COMPRESS_ZSTD=y
 CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS=y
 CONFIG_MODPROBE_PATH="/sbin/modprobe"
+# CONFIG_TRIM_UNUSED_KSYMS is not set
 CONFIG_MODULES_TREE_LOOKUP=y
 CONFIG_BLOCK=y
-CONFIG_BLOCK_LEGACY_AUTOLOAD=y
 CONFIG_BLK_RQ_ALLOC_TIME=y
+CONFIG_BLK_SCSI_REQUEST=y
 CONFIG_BLK_CGROUP_RWSTAT=y
-CONFIG_BLK_DEV_BSG_COMMON=y
-CONFIG_BLK_ICQ=y
+CONFIG_BLK_DEV_BSG=y
 CONFIG_BLK_DEV_BSGLIB=y
 CONFIG_BLK_DEV_INTEGRITY=y
 CONFIG_BLK_DEV_INTEGRITY_T10=y
 CONFIG_BLK_DEV_ZONED=y
 CONFIG_BLK_DEV_THROTTLING=y
-CONFIG_BLK_DEV_THROTTLING_LOW=y
+# CONFIG_BLK_DEV_THROTTLING_LOW is not set
+CONFIG_BLK_CMDLINE_PARSER=y
 CONFIG_BLK_WBT=y
-CONFIG_BLK_WBT_MQ=y
-CONFIG_BLK_CGROUP_IOLATENCY=y
-CONFIG_BLK_CGROUP_FC_APPID=y
+# CONFIG_BLK_CGROUP_IOLATENCY is not set
 CONFIG_BLK_CGROUP_IOCOST=y
 CONFIG_BLK_CGROUP_IOPRIO=y
 CONFIG_BLK_DEBUG_FS=y
@@ -980,6 +942,10 @@ CONFIG_PREEMPT_NOTIFIERS=y
 CONFIG_PADATA=y
 CONFIG_ASN1=y
 CONFIG_UNINLINE_SPIN_UNLOCK=y
+CONFIG_INLINE_READ_UNLOCK=y
+CONFIG_INLINE_READ_UNLOCK_IRQ=y
+CONFIG_INLINE_WRITE_UNLOCK=y
+CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
 CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
 CONFIG_MUTEX_SPIN_ON_OWNER=y
 CONFIG_RWSEM_SPIN_ON_OWNER=y
@@ -1008,53 +974,22 @@ CONFIG_COREDUMP=y
 #
 # Memory Management options
 #
-CONFIG_ZPOOL=y
-CONFIG_SWAP=y
-CONFIG_ZSWAP=y
-CONFIG_ZSWAP_DEFAULT_ON=y
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO is not set
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set
-CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4=y
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set
-CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lz4"
-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD is not set
-CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD=y
-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
-CONFIG_ZSWAP_ZPOOL_DEFAULT="z3fold"
-CONFIG_ZBUD=y
-CONFIG_Z3FOLD=y
-CONFIG_ZSMALLOC=y
-# CONFIG_ZSMALLOC_STAT is not set
-
-#
-# SLAB allocator options
-#
-# CONFIG_SLAB is not set
-CONFIG_SLUB=y
-CONFIG_SLAB_MERGE_DEFAULT=y
-CONFIG_SLAB_FREELIST_RANDOM=y
-CONFIG_SLAB_FREELIST_HARDENED=y
-# CONFIG_SLUB_STATS is not set
-CONFIG_SLUB_CPU_PARTIAL=y
-# end of SLAB allocator options
-
-CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
-# CONFIG_COMPAT_BRK is not set
+CONFIG_SELECT_MEMORY_MODEL=y
+CONFIG_SPARSEMEM_MANUAL=y
 CONFIG_SPARSEMEM=y
+CONFIG_NEED_MULTIPLE_NODES=y
 CONFIG_SPARSEMEM_EXTREME=y
 CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
 CONFIG_SPARSEMEM_VMEMMAP=y
 CONFIG_HAVE_FAST_GUP=y
 CONFIG_NUMA_KEEP_MEMINFO=y
 CONFIG_MEMORY_ISOLATION=y
-CONFIG_EXCLUSIVE_SYSTEM_RAM=y
 CONFIG_HAVE_BOOTMEM_INFO_NODE=y
 CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
-CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
 CONFIG_MEMORY_HOTPLUG=y
+CONFIG_MEMORY_HOTPLUG_SPARSE=y
 CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
+CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
 CONFIG_MEMORY_HOTREMOVE=y
 CONFIG_MHP_MEMMAP_ON_MEMORY=y
 CONFIG_SPLIT_PTLOCK_CPUS=4
@@ -1064,7 +999,6 @@ CONFIG_BALLOON_COMPACTION=y
 CONFIG_COMPACTION=y
 CONFIG_PAGE_REPORTING=y
 CONFIG_MIGRATION=y
-CONFIG_DEVICE_MIGRATION=y
 CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=y
 CONFIG_ARCH_ENABLE_THP_MIGRATION=y
 CONFIG_CONTIG_ALLOC=y
@@ -1076,66 +1010,54 @@ CONFIG_DEFAULT_MMAP_MIN_ADDR=65536
 CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
 CONFIG_MEMORY_FAILURE=y
 CONFIG_HWPOISON_INJECT=m
-CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
-CONFIG_ARCH_WANTS_THP_SWAP=y
 CONFIG_TRANSPARENT_HUGEPAGE=y
-CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
-# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
+# CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is not set
+CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y
+CONFIG_ARCH_WANTS_THP_SWAP=y
 CONFIG_THP_SWAP=y
-CONFIG_READ_ONLY_THP_FOR_FS=y
-CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
-CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
-CONFIG_USE_PERCPU_NUMA_NODE_ID=y
-CONFIG_HAVE_SETUP_PER_CPU_AREA=y
+CONFIG_CLEANCACHE=y
 CONFIG_FRONTSWAP=y
-CONFIG_CMA=y
-# CONFIG_CMA_DEBUG is not set
-CONFIG_CMA_DEBUGFS=y
-CONFIG_CMA_SYSFS=y
-CONFIG_CMA_AREAS=7
+# CONFIG_CMA is not set
 CONFIG_MEM_SOFT_DIRTY=y
+CONFIG_ZSWAP=y
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo"
+CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
+CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud"
+# CONFIG_ZSWAP_DEFAULT_ON is not set
+CONFIG_ZPOOL=y
+CONFIG_ZBUD=y
+CONFIG_Z3FOLD=m
+CONFIG_ZSMALLOC=y
+# CONFIG_ZSMALLOC_STAT is not set
 CONFIG_GENERIC_EARLY_IOREMAP=y
 # CONFIG_DEFERRED_STRUCT_PAGE_INIT is not set
-CONFIG_PAGE_IDLE_FLAG=y
 CONFIG_IDLE_PAGE_TRACKING=y
 CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
-CONFIG_ARCH_HAS_CURRENT_STACK_POINTER=y
-CONFIG_ARCH_HAS_VM_GET_PAGE_PROT=y
 CONFIG_ARCH_HAS_PTE_DEVMAP=y
-CONFIG_ZONE_DMA=y
-CONFIG_ZONE_DMA32=y
 CONFIG_ZONE_DEVICE=y
+CONFIG_DEV_PAGEMAP_OPS=y
 CONFIG_HMM_MIRROR=y
 CONFIG_DEVICE_PRIVATE=y
 CONFIG_VMAP_PFN=y
 CONFIG_ARCH_USES_HIGH_VMA_FLAGS=y
 CONFIG_ARCH_HAS_PKEYS=y
-CONFIG_VM_EVENT_COUNTERS=y
 # CONFIG_PERCPU_STATS is not set
 # CONFIG_GUP_TEST is not set
+# CONFIG_READ_ONLY_THP_FOR_FS is not set
 CONFIG_ARCH_HAS_PTE_SPECIAL=y
 CONFIG_MAPPING_DIRTY_HELPERS=y
-CONFIG_SECRETMEM=y
-CONFIG_ANON_VMA_NAME=y
-CONFIG_USERFAULTFD=y
-CONFIG_HAVE_ARCH_USERFAULTFD_WP=y
-CONFIG_HAVE_ARCH_USERFAULTFD_MINOR=y
-CONFIG_PTE_MARKER=y
-CONFIG_PTE_MARKER_UFFD_WP=y
-
-#
-# Data Access Monitoring
-#
-CONFIG_DAMON=y
-CONFIG_DAMON_VADDR=y
-CONFIG_DAMON_PADDR=y
-CONFIG_DAMON_SYSFS=y
-CONFIG_DAMON_DBGFS=y
-CONFIG_DAMON_RECLAIM=y
-# end of Data Access Monitoring
 # end of Memory Management options
 
 CONFIG_NET=y
+CONFIG_WANT_COMPAT_NETLINK_MESSAGES=y
 CONFIG_COMPAT_NETLINK_MESSAGES=y
 CONFIG_NET_INGRESS=y
 CONFIG_NET_EGRESS=y

Edit: possibly very relevant

-CONFIG_ZSWAP_DEFAULT_ON=y
+# CONFIG_ZSWAP_DEFAULT_ON is not set

-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO is not set
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y

-CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4=y
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set

-CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lz4"
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo"

-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD is not set
+CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y

-CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD=y
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set

-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set

-CONFIG_ZSWAP_ZPOOL_DEFAULT="z3fold"
+CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud"

Edit #2

-CONFIG_SWAP=y

+CONFIG_VM_EVENT_COUNTERS=y
-CONFIG_HAVE_ARCH_HUGE_VMALLOC=y
-CONFIG_ARCH_HAS_VM_GET_PAGE_PROT=y
-CONFIG_VM_EVENT_COUNTERS=y
-CONFIG_ANON_VMA_NAME=y

Last edited by seth (2022-10-01 08:04:39)

7000k · 2022-10-01 09:10:24

seth wrote:

The vm config won't change but seems equal between kernels and the config uses CRLF… notepad?
Anyway, kernel config diff:

--- config_arch.txt	2022-10-01 09:52:19.014278897 +0200
+++ config_ubuntu.txt	2022-10-01 09:51:57.974197432 +0200
@@ -14,11 +14,7 @@ CONFIG_LLD_VERSION=0
 CONFIG_CC_CAN_LINK=y
 CONFIG_CC_CAN_LINK_STATIC=y
 CONFIG_CC_HAS_ASM_GOTO=y
-CONFIG_CC_HAS_ASM_GOTO_OUTPUT=y
-CONFIG_CC_HAS_ASM_GOTO_TIED_OUTPUT=y
 CONFIG_CC_HAS_ASM_INLINE=y
-CONFIG_CC_HAS_NO_PROFILE_FN_ATTR=y
-CONFIG_PAHOLE_VERSION=123
 CONFIG_IRQ_WORK=y
 CONFIG_BUILDTIME_TABLE_SORT=y
 CONFIG_THREAD_INFO_IN_TASK=y
@@ -28,7 +24,6 @@ CONFIG_THREAD_INFO_IN_TASK=y
 #
 CONFIG_INIT_ENV_ARG_LIMIT=32
 # CONFIG_COMPILE_TEST is not set
-# CONFIG_WERROR is not set
 CONFIG_LOCALVERSION=""
 CONFIG_LOCALVERSION_AUTO=y
 CONFIG_BUILD_SALT=""
@@ -50,7 +45,6 @@ CONFIG_DEFAULT_INIT=""
 CONFIG_DEFAULT_HOSTNAME="archlinux"
 CONFIG_SYSVIPC=y
 CONFIG_SYSVIPC_SYSCTL=y
-CONFIG_SYSVIPC_COMPAT=y
 CONFIG_POSIX_MQUEUE=y
 CONFIG_POSIX_MQUEUE_SYSCTL=y
 CONFIG_WATCH_QUEUE=y
@@ -68,11 +62,9 @@ CONFIG_GENERIC_IRQ_SHOW=y
 CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK=y
 CONFIG_GENERIC_PENDING_IRQ=y
 CONFIG_GENERIC_IRQ_MIGRATION=y
-CONFIG_GENERIC_IRQ_INJECTION=y
 CONFIG_HARDIRQS_SW_RESEND=y
 CONFIG_GENERIC_IRQ_CHIP=y
 CONFIG_IRQ_DOMAIN=y
-CONFIG_IRQ_SIM=y
 CONFIG_IRQ_DOMAIN_HIERARCHY=y
 CONFIG_GENERIC_MSI_IRQ=y
 CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
@@ -101,13 +93,10 @@ CONFIG_POSIX_CPU_TIMERS_TASK_WORK=y
 CONFIG_TICK_ONESHOT=y
 CONFIG_NO_HZ_COMMON=y
 # CONFIG_HZ_PERIODIC is not set
-# CONFIG_NO_HZ_IDLE is not set
-CONFIG_NO_HZ_FULL=y
-CONFIG_CONTEXT_TRACKING=y
-# CONFIG_CONTEXT_TRACKING_FORCE is not set
+CONFIG_NO_HZ_IDLE=y
+# CONFIG_NO_HZ_FULL is not set
 CONFIG_NO_HZ=y
 CONFIG_HIGH_RES_TIMERS=y
-CONFIG_CLOCKSOURCE_WATCHDOG_MAX_SKEW_US=100
 # end of Timers subsystem
 
 CONFIG_BPF=y
@@ -123,19 +112,13 @@ CONFIG_BPF_JIT_ALWAYS_ON=y
 CONFIG_BPF_JIT_DEFAULT_ON=y
 CONFIG_BPF_UNPRIV_DEFAULT_OFF=y
 CONFIG_USERMODE_DRIVER=y
-CONFIG_BPF_PRELOAD=y
-CONFIG_BPF_PRELOAD_UMD=m
+#CONFIG_BPF_PRELOAD is not set
 CONFIG_BPF_LSM=y
 # end of BPF subsystem
 
-CONFIG_PREEMPT_BUILD=y
 # CONFIG_PREEMPT_NONE is not set
-# CONFIG_PREEMPT_VOLUNTARY is not set
-CONFIG_PREEMPT=y
-CONFIG_PREEMPT_COUNT=y
-CONFIG_PREEMPTION=y
-CONFIG_PREEMPT_DYNAMIC=y
-CONFIG_SCHED_CORE=y
+CONFIG_PREEMPT_VOLUNTARY=y
+#CONFIG_PREEMPT is not set
 
 #
 # CPU/Task time and stats accounting
@@ -143,7 +126,6 @@ CONFIG_SCHED_CORE=y
 CONFIG_VIRT_CPU_ACCOUNTING=y
 CONFIG_VIRT_CPU_ACCOUNTING_GEN=y
 CONFIG_IRQ_TIME_ACCOUNTING=y
-CONFIG_HAVE_SCHED_AVG_IRQ=y
 CONFIG_BSD_PROCESS_ACCT=y
 CONFIG_BSD_PROCESS_ACCT_V3=y
 CONFIG_TASKSTATS=y
@@ -160,36 +142,22 @@ CONFIG_CPU_ISOLATION=y
 # RCU Subsystem
 #
 CONFIG_TREE_RCU=y
-CONFIG_PREEMPT_RCU=y
-CONFIG_RCU_EXPERT=y
+# CONFIG_RCU_EXPERT is not set
 CONFIG_SRCU=y
 CONFIG_TREE_SRCU=y
 CONFIG_TASKS_RCU_GENERIC=y
-# CONFIG_FORCE_TASKS_RCU is not set
-CONFIG_TASKS_RCU=y
-# CONFIG_FORCE_TASKS_RUDE_RCU is not set
 CONFIG_TASKS_RUDE_RCU=y
-# CONFIG_FORCE_TASKS_TRACE_RCU is not set
 CONFIG_TASKS_TRACE_RCU=y
 CONFIG_RCU_STALL_COMMON=y
 CONFIG_RCU_NEED_SEGCBLIST=y
-CONFIG_RCU_FANOUT=64
-CONFIG_RCU_FANOUT_LEAF=16
-CONFIG_RCU_BOOST=y
-CONFIG_RCU_BOOST_DELAY=500
-# CONFIG_RCU_EXP_KTHREAD is not set
-CONFIG_RCU_NOCB_CPU=y
-# CONFIG_TASKS_TRACE_RCU_READ_MB is not set
 # end of RCU Subsystem
 
 CONFIG_BUILD_BIN2C=y
 CONFIG_IKCONFIG=y
-CONFIG_IKCONFIG_PROC=y
 CONFIG_IKHEADERS=m
 CONFIG_LOG_BUF_SHIFT=17
 CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
 CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13
-CONFIG_PRINTK_INDEX=y
 CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
 
 #
@@ -202,9 +170,6 @@ CONFIG_UCLAMP_BUCKETS_COUNT=5
 CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
 CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y
 CONFIG_CC_HAS_INT128=y
-CONFIG_CC_IMPLICIT_FALLTHROUGH="-Wimplicit-fallthrough=5"
-CONFIG_GCC12_NO_ARRAY_BOUNDS=y
-CONFIG_CC_NO_ARRAY_BOUNDS=y
 CONFIG_ARCH_SUPPORTS_INT128=y
 CONFIG_NUMA_BALANCING=y
 CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
@@ -238,7 +203,6 @@ CONFIG_UTS_NS=y
 CONFIG_TIME_NS=y
 CONFIG_IPC_NS=y
 CONFIG_USER_NS=y
-CONFIG_USER_NS_UNPRIVILEGED=y
 CONFIG_PID_NS=y
 CONFIG_NET_NS=y
 CONFIG_CHECKPOINT_RESTORE=y
@@ -255,8 +219,6 @@ CONFIG_RD_LZO=y
 CONFIG_RD_LZ4=y
 CONFIG_RD_ZSTD=y
 CONFIG_BOOT_CONFIG=y
-# CONFIG_BOOT_CONFIG_EMBED is not set
-CONFIG_INITRAMFS_PRESERVE_MTIME=y
 CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
 # CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
 CONFIG_LD_ORPHAN_WARN=y
@@ -272,6 +234,7 @@ CONFIG_SYSFS_SYSCALL=y
 CONFIG_FHANDLE=y
 CONFIG_POSIX_TIMERS=y
 CONFIG_PRINTK=y
+CONFIG_PRINTK_NMI=y
 CONFIG_BUG=y
 CONFIG_ELF_CORE=y
 CONFIG_PCSPKR_PLATFORM=y
@@ -286,14 +249,18 @@ CONFIG_SHMEM=y
 CONFIG_AIO=y
 CONFIG_IO_URING=y
 CONFIG_ADVISE_SYSCALLS=y
+CONFIG_HAVE_ARCH_USERFAULTFD_WP=y
+CONFIG_HAVE_ARCH_USERFAULTFD_MINOR=y
 CONFIG_MEMBARRIER=y
 CONFIG_KALLSYMS=y
 CONFIG_KALLSYMS_ALL=y
 CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
 CONFIG_KALLSYMS_BASE_RELATIVE=y
+CONFIG_USERFAULTFD=y
 CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE=y
 CONFIG_KCMP=y
 CONFIG_RSEQ=y
+# CONFIG_DEBUG_RSEQ is not set
 # CONFIG_EMBEDDED is not set
 CONFIG_HAVE_PERF_EVENTS=y
 CONFIG_GUEST_PERF_EVENTS=y
@@ -305,6 +272,17 @@ CONFIG_PERF_EVENTS=y
 # CONFIG_DEBUG_PERF_USE_VMALLOC is not set
 # end of Kernel Performance Events And Counters
 
+CONFIG_VM_EVENT_COUNTERS=y
+CONFIG_SLUB_DEBUG=y
+# CONFIG_COMPAT_BRK is not set
+# CONFIG_SLAB is not set
+CONFIG_SLUB=y
+# CONFIG_SLOB is not set
+CONFIG_SLAB_MERGE_DEFAULT=y
+CONFIG_SLAB_FREELIST_RANDOM=y
+CONFIG_SLAB_FREELIST_HARDENED=y
+CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
+CONFIG_SLUB_CPU_PARTIAL=y
 CONFIG_SYSTEM_DATA_VERIFICATION=y
 CONFIG_PROFILING=y
 CONFIG_TRACEPOINTS=y
@@ -328,9 +306,14 @@ CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
 CONFIG_ARCH_MAY_HAVE_PC_FDC=y
 CONFIG_GENERIC_CALIBRATE_DELAY=y
 CONFIG_ARCH_HAS_CPU_RELAX=y
+CONFIG_ARCH_HAS_FILTER_PGPROT=y
+CONFIG_HAVE_SETUP_PER_CPU_AREA=y
+CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
+CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
 CONFIG_ARCH_HIBERNATION_POSSIBLE=y
-CONFIG_ARCH_NR_GPIO=1024
 CONFIG_ARCH_SUSPEND_POSSIBLE=y
+CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
+CONFIG_ZONE_DMA32=y
 CONFIG_AUDIT_ARCH=y
 CONFIG_HAVE_INTEL_TXT=y
 CONFIG_X86_64_SMP=y
@@ -343,13 +326,20 @@ CONFIG_CC_HAS_SANE_STACKPROTECTOR=y
 #
 # Processor type and features
 #
+CONFIG_ZONE_DMA=y
 CONFIG_SMP=y
 CONFIG_X86_FEATURE_NAMES=y
 CONFIG_X86_X2APIC=y
 CONFIG_X86_MPPARSE=y
 # CONFIG_GOLDFISH is not set
+CONFIG_RETPOLINE=y
 CONFIG_X86_CPU_RESCTRL=y
-# CONFIG_X86_EXTENDED_PLATFORM is not set
+CONFIG_X86_EXTENDED_PLATFORM=y
+CONFIG_X86_NUMACHIP=y
+# CONFIG_X86_VSMP is not set
+CONFIG_X86_UV=y
+# CONFIG_X86_GOLDFISH is not set
+# CONFIG_X86_INTEL_MID is not set
 CONFIG_X86_INTEL_LPSS=y
 CONFIG_X86_AMD_PLATFORM_DEVICE=y
 CONFIG_IOSF_MBI=y
@@ -373,7 +363,6 @@ CONFIG_XEN_PVHVM_GUEST=y
 CONFIG_XEN_SAVE_RESTORE=y
 # CONFIG_XEN_DEBUG_FS is not set
 CONFIG_XEN_PVH=y
-CONFIG_XEN_DOM0=y
 CONFIG_KVM_GUEST=y
 CONFIG_ARCH_CPUIDLE_HALTPOLL=y
 CONFIG_PVH=y
@@ -381,7 +370,6 @@ CONFIG_PARAVIRT_TIME_ACCOUNTING=y
 CONFIG_PARAVIRT_CLOCK=y
 CONFIG_JAILHOUSE_GUEST=y
 CONFIG_ACRN_GUEST=y
-CONFIG_INTEL_TDX_GUEST=y
 # CONFIG_MK8 is not set
 # CONFIG_MPSC is not set
 # CONFIG_MCORE2 is not set
@@ -396,6 +384,7 @@ CONFIG_X86_MINIMUM_CPU_FAMILY=64
 CONFIG_X86_DEBUGCTLMSR=y
 CONFIG_IA32_FEAT_CTL=y
 CONFIG_X86_VMX_FEATURE_NAMES=y
+CONFIG_PROCESSOR_SELECT=y
 CONFIG_CPU_SUP_INTEL=y
 CONFIG_CPU_SUP_AMD=y
 CONFIG_CPU_SUP_HYGON=y
@@ -404,14 +393,13 @@ CONFIG_CPU_SUP_ZHAOXIN=y
 CONFIG_HPET_TIMER=y
 CONFIG_HPET_EMULATE_RTC=y
 CONFIG_DMI=y
-# CONFIG_GART_IOMMU is not set
+CONFIG_GART_IOMMU=y
 CONFIG_BOOT_VESA_SUPPORT=y
-# CONFIG_MAXSMP is not set
-CONFIG_NR_CPUS_RANGE_BEGIN=2
-CONFIG_NR_CPUS_RANGE_END=512
-CONFIG_NR_CPUS_DEFAULT=64
-CONFIG_NR_CPUS=320
-CONFIG_SCHED_CLUSTER=y
+CONFIG_MAXSMP=y
+CONFIG_NR_CPUS_RANGE_BEGIN=8192
+CONFIG_NR_CPUS_RANGE_END=8192
+CONFIG_NR_CPUS_DEFAULT=8192
+CONFIG_NR_CPUS=8192
 CONFIG_SCHED_SMT=y
 CONFIG_SCHED_MC=y
 CONFIG_SCHED_MC_PRIO=y
@@ -440,6 +428,7 @@ CONFIG_X86_16BIT=y
 CONFIG_X86_ESPFIX64=y
 CONFIG_X86_VSYSCALL_EMULATION=y
 CONFIG_X86_IOPL_IOPERM=y
+CONFIG_I8K=m
 CONFIG_MICROCODE=y
 CONFIG_MICROCODE_INTEL=y
 CONFIG_MICROCODE_AMD=y
@@ -448,8 +437,7 @@ CONFIG_X86_MSR=y
 CONFIG_X86_CPUID=y
 CONFIG_X86_5LEVEL=y
 CONFIG_X86_DIRECT_GBPAGES=y
-CONFIG_X86_CPA_STATISTICS=y
-CONFIG_X86_MEM_ENCRYPT=y
+# CONFIG_X86_CPA_STATISTICS is not set
 CONFIG_AMD_MEM_ENCRYPT=y
 # CONFIG_AMD_MEM_ENCRYPT_ACTIVE_BY_DEFAULT is not set
 CONFIG_NUMA=y
@@ -459,13 +447,15 @@ CONFIG_X86_64_ACPI_NUMA=y
 CONFIG_NODES_SHIFT=5
 CONFIG_ARCH_SPARSEMEM_ENABLE=y
 CONFIG_ARCH_SPARSEMEM_DEFAULT=y
-# CONFIG_ARCH_MEMORY_PROBE is not set
+CONFIG_ARCH_SELECT_MEMORY_MODEL=y
+CONFIG_ARCH_MEMORY_PROBE=y
 CONFIG_ARCH_PROC_KCORE_TEXT=y
 CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
 CONFIG_X86_PMEM_LEGACY_DEVICE=y
 CONFIG_X86_PMEM_LEGACY=m
 CONFIG_X86_CHECK_BIOS_CORRUPTION=y
 CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
+CONFIG_X86_RESERVE_LOW=64
 CONFIG_MTRR=y
 CONFIG_MTRR_SANITIZER=y
 CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
@@ -473,9 +463,8 @@ CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAU
 CONFIG_X86_PAT=y
 CONFIG_ARCH_USES_PG_UNCACHED=y
 CONFIG_ARCH_RANDOM=y
+CONFIG_X86_SMAP=y
 CONFIG_X86_UMIP=y
-CONFIG_CC_HAS_IBT=y
-CONFIG_X86_KERNEL_IBT=y
 CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=y
 # CONFIG_X86_INTEL_TSX_MODE_OFF is not set
 # CONFIG_X86_INTEL_TSX_MODE_ON is not set
@@ -485,8 +474,8 @@ CONFIG_EFI=y
 CONFIG_EFI_STUB=y
 CONFIG_EFI_MIXED=y
 # CONFIG_HZ_100 is not set
-# CONFIG_HZ_250 is not set
-CONFIG_HZ_300=y
+CONFIG_HZ_250=y
+# CONFIG_HZ_300 uis not set
 # CONFIG_HZ_1000 is not set
 CONFIG_HZ=300
 CONFIG_SCHED_HRTICK=y
@@ -510,27 +499,18 @@ CONFIG_HOTPLUG_CPU=y
 # CONFIG_BOOTPARAM_HOTPLUG_CPU0 is not set
 # CONFIG_DEBUG_HOTPLUG_CPU0 is not set
 # CONFIG_COMPAT_VDSO is not set
+# CONFIG_LEGACY_VSYSCALL_EMULATE is not set
 CONFIG_LEGACY_VSYSCALL_XONLY=y
 # CONFIG_LEGACY_VSYSCALL_NONE is not set
 # CONFIG_CMDLINE_BOOL is not set
 CONFIG_MODIFY_LDT_SYSCALL=y
-# CONFIG_STRICT_SIGALTSTACK_SIZE is not set
 CONFIG_HAVE_LIVEPATCH=y
 # CONFIG_LIVEPATCH is not set
 # end of Processor type and features
 
-CONFIG_CC_HAS_SLS=y
-CONFIG_CC_HAS_RETURN_THUNK=y
-CONFIG_SPECULATION_MITIGATIONS=y
-CONFIG_PAGE_TABLE_ISOLATION=y
-CONFIG_RETPOLINE=y
-CONFIG_RETHUNK=y
-CONFIG_CPU_UNRET_ENTRY=y
-CONFIG_CPU_IBPB_ENTRY=y
-CONFIG_CPU_IBRS_ENTRY=y
-CONFIG_SLS=y
 CONFIG_ARCH_HAS_ADD_PAGES=y
 CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=y
+CONFIG_USE_PERCPU_NUMA_NODE_ID=y
 
 #
 # Power management and ACPI options
@@ -538,6 +518,7 @@ CONFIG_ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE=
 CONFIG_ARCH_HIBERNATION_HEADER=y
 CONFIG_SUSPEND=y
 CONFIG_SUSPEND_FREEZER=y
+# CONFIG_SUSPEND_SKIP_SYNC is not set
 CONFIG_HIBERNATE_CALLBACKS=y
 CONFIG_HIBERNATION=y
 CONFIG_HIBERNATION_SNAPSHOT_DEV=y
@@ -551,6 +532,7 @@ CONFIG_PM_DEBUG=y
 # CONFIG_PM_ADVANCED_DEBUG is not set
 # CONFIG_PM_TEST_SUSPEND is not set
 CONFIG_PM_SLEEP_DEBUG=y
+# CONFIG_DPM_WATCHDOG is not set
 CONFIG_PM_TRACE=y
 CONFIG_PM_TRACE_RTC=y
 CONFIG_PM_CLK=y
@@ -588,6 +570,7 @@ CONFIG_ACPI_HOTPLUG_CPU=y
 CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
 CONFIG_ACPI_THERMAL=y
 CONFIG_ACPI_PLATFORM_PROFILE=m
+CONFIG_ACPI_CUSTOM_DSDT_FILE=""
 CONFIG_ARCH_HAS_ACPI_TABLE_UPGRADE=y
 CONFIG_ACPI_TABLE_UPGRADE=y
 CONFIG_ACPI_DEBUG=y
@@ -599,6 +582,7 @@ CONFIG_ACPI_SBS=m
 CONFIG_ACPI_HED=y
 CONFIG_ACPI_CUSTOM_METHOD=m
 CONFIG_ACPI_BGRT=y
+# CONFIG_ACPI_REDUCED_HARDWARE_ONLY is not set
 CONFIG_ACPI_NFIT=m
 # CONFIG_NFIT_SECURITY_DEBUG is not set
 CONFIG_ACPI_NUMA=y
@@ -654,9 +638,9 @@ CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
 # CPU frequency scaling drivers
 #
 CONFIG_X86_INTEL_PSTATE=y
-CONFIG_X86_PCC_CPUFREQ=m
+CONFIG_X86_PCC_CPUFREQ=y
 CONFIG_X86_AMD_PSTATE=m
-CONFIG_X86_ACPI_CPUFREQ=m
+CONFIG_X86_ACPI_CPUFREQ=y
 CONFIG_X86_ACPI_CPUFREQ_CPB=y
 CONFIG_X86_POWERNOW_K8=m
 CONFIG_X86_AMD_FREQ_SENSITIVITY=m
@@ -724,6 +708,7 @@ CONFIG_KVM_XFER_TO_GUEST_WORK=y
 CONFIG_HAVE_KVM_PM_NOTIFIER=y
 CONFIG_VIRTUALIZATION=y
 CONFIG_KVM=m
+CONFIG_KVM_WERROR=y
 CONFIG_KVM_INTEL=m
 CONFIG_X86_SGX_KVM=y
 CONFIG_KVM_AMD=m
@@ -752,18 +737,14 @@ CONFIG_UPROBES=y
 CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
 CONFIG_ARCH_USE_BUILTIN_BSWAP=y
 CONFIG_KRETPROBES=y
-CONFIG_KRETPROBE_ON_RETHOOK=y
 CONFIG_USER_RETURN_NOTIFIER=y
 CONFIG_HAVE_IOREMAP_PROT=y
 CONFIG_HAVE_KPROBES=y
 CONFIG_HAVE_KRETPROBES=y
 CONFIG_HAVE_OPTPROBES=y
 CONFIG_HAVE_KPROBES_ON_FTRACE=y
-CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE=y
 CONFIG_HAVE_FUNCTION_ERROR_INJECTION=y
 CONFIG_HAVE_NMI=y
-CONFIG_TRACE_IRQFLAGS_SUPPORT=y
-CONFIG_TRACE_IRQFLAGS_NMI_SUPPORT=y
 CONFIG_HAVE_ARCH_TRACEHOOK=y
 CONFIG_HAVE_DMA_CONTIGUOUS=y
 CONFIG_GENERIC_SMP_IDLE_THREAD=y
@@ -772,7 +753,6 @@ CONFIG_ARCH_HAS_SET_MEMORY=y
 CONFIG_ARCH_HAS_SET_DIRECT_MAP=y
 CONFIG_HAVE_ARCH_THREAD_STRUCT_WHITELIST=y
 CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y
-CONFIG_ARCH_WANTS_NO_INSTR=y
 CONFIG_HAVE_ASM_MODVERSIONS=y
 CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
 CONFIG_HAVE_RSEQ=y
@@ -788,7 +768,6 @@ CONFIG_HAVE_ARCH_JUMP_LABEL=y
 CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE=y
 CONFIG_MMU_GATHER_TABLE_FREE=y
 CONFIG_MMU_GATHER_RCU_TABLE_FREE=y
-CONFIG_MMU_GATHER_MERGE_VMAS=y
 CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
 CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
 CONFIG_HAVE_CMPXCHG_LOCAL=y
@@ -817,7 +796,6 @@ CONFIG_HAVE_MOVE_PMD=y
 CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
 CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=y
 CONFIG_HAVE_ARCH_HUGE_VMAP=y
-CONFIG_HAVE_ARCH_HUGE_VMALLOC=y
 CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
 CONFIG_HAVE_ARCH_SOFT_DIRTY=y
 CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
@@ -831,13 +809,6 @@ CONFIG_ARCH_MMAP_RND_BITS=28
 CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS=y
 CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
 CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES=y
-CONFIG_PAGE_SIZE_LESS_THAN_64KB=y
-CONFIG_PAGE_SIZE_LESS_THAN_256KB=y
-CONFIG_HAVE_OBJTOOL=y
-CONFIG_HAVE_JUMP_LABEL_HACK=y
-CONFIG_HAVE_NOINSTR_HACK=y
-CONFIG_HAVE_NOINSTR_VALIDATION=y
-CONFIG_HAVE_UACCESS_VALIDATION=y
 CONFIG_HAVE_STACK_VALIDATION=y
 CONFIG_HAVE_RELIABLE_STACKTRACE=y
 CONFIG_ISA_BUS_API=y
@@ -847,7 +818,6 @@ CONFIG_COMPAT_32BIT_TIME=y
 CONFIG_HAVE_ARCH_VMAP_STACK=y
 CONFIG_VMAP_STACK=y
 CONFIG_HAVE_ARCH_RANDOMIZE_KSTACK_OFFSET=y
-CONFIG_RANDOMIZE_KSTACK_OFFSET=y
 CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT=y
 CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
 CONFIG_STRICT_KERNEL_RWX=y
@@ -861,14 +831,9 @@ CONFIG_ARCH_HAS_CC_PLATFORM=y
 CONFIG_HAVE_STATIC_CALL=y
 CONFIG_HAVE_STATIC_CALL_INLINE=y
 CONFIG_HAVE_PREEMPT_DYNAMIC=y
-CONFIG_HAVE_PREEMPT_DYNAMIC_CALL=y
 CONFIG_ARCH_WANT_LD_ORPHAN_WARN=y
 CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
-CONFIG_ARCH_SUPPORTS_PAGE_TABLE_CHECK=y
 CONFIG_ARCH_HAS_ELFCORE_COMPAT=y
-CONFIG_ARCH_HAS_PARANOID_L1D_FLUSH=y
-CONFIG_DYNAMIC_SIGFRAME=y
-CONFIG_HAVE_ARCH_NODE_DEV_GROUP=y
 
 #
 # GCOV-based kernel profiling
@@ -878,8 +843,6 @@ CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
 # end of GCOV-based kernel profiling
 
 CONFIG_HAVE_GCC_PLUGINS=y
-CONFIG_GCC_PLUGINS=y
-# CONFIG_GCC_PLUGIN_LATENT_ENTROPY is not set
 # end of General architecture-dependent options
 
 CONFIG_RT_MUTEXES=y
@@ -907,23 +870,22 @@ CONFIG_MODULE_SIG_HASH="sha512"
 CONFIG_MODULE_COMPRESS_ZSTD=y
 CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS=y
 CONFIG_MODPROBE_PATH="/sbin/modprobe"
+# CONFIG_TRIM_UNUSED_KSYMS is not set
 CONFIG_MODULES_TREE_LOOKUP=y
 CONFIG_BLOCK=y
-CONFIG_BLOCK_LEGACY_AUTOLOAD=y
 CONFIG_BLK_RQ_ALLOC_TIME=y
+CONFIG_BLK_SCSI_REQUEST=y
 CONFIG_BLK_CGROUP_RWSTAT=y
-CONFIG_BLK_DEV_BSG_COMMON=y
-CONFIG_BLK_ICQ=y
+CONFIG_BLK_DEV_BSG=y
 CONFIG_BLK_DEV_BSGLIB=y
 CONFIG_BLK_DEV_INTEGRITY=y
 CONFIG_BLK_DEV_INTEGRITY_T10=y
 CONFIG_BLK_DEV_ZONED=y
 CONFIG_BLK_DEV_THROTTLING=y
-CONFIG_BLK_DEV_THROTTLING_LOW=y
+# CONFIG_BLK_DEV_THROTTLING_LOW is not set
+CONFIG_BLK_CMDLINE_PARSER=y
 CONFIG_BLK_WBT=y
-CONFIG_BLK_WBT_MQ=y
-CONFIG_BLK_CGROUP_IOLATENCY=y
-CONFIG_BLK_CGROUP_FC_APPID=y
+# CONFIG_BLK_CGROUP_IOLATENCY is not set
 CONFIG_BLK_CGROUP_IOCOST=y
 CONFIG_BLK_CGROUP_IOPRIO=y
 CONFIG_BLK_DEBUG_FS=y
@@ -980,6 +942,10 @@ CONFIG_PREEMPT_NOTIFIERS=y
 CONFIG_PADATA=y
 CONFIG_ASN1=y
 CONFIG_UNINLINE_SPIN_UNLOCK=y
+CONFIG_INLINE_READ_UNLOCK=y
+CONFIG_INLINE_READ_UNLOCK_IRQ=y
+CONFIG_INLINE_WRITE_UNLOCK=y
+CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
 CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
 CONFIG_MUTEX_SPIN_ON_OWNER=y
 CONFIG_RWSEM_SPIN_ON_OWNER=y
@@ -1008,53 +974,22 @@ CONFIG_COREDUMP=y
 #
 # Memory Management options
 #
-CONFIG_ZPOOL=y
-CONFIG_SWAP=y
-CONFIG_ZSWAP=y
-CONFIG_ZSWAP_DEFAULT_ON=y
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO is not set
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set
-CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4=y
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set
-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set
-CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lz4"
-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD is not set
-CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD=y
-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
-CONFIG_ZSWAP_ZPOOL_DEFAULT="z3fold"
-CONFIG_ZBUD=y
-CONFIG_Z3FOLD=y
-CONFIG_ZSMALLOC=y
-# CONFIG_ZSMALLOC_STAT is not set
-
-#
-# SLAB allocator options
-#
-# CONFIG_SLAB is not set
-CONFIG_SLUB=y
-CONFIG_SLAB_MERGE_DEFAULT=y
-CONFIG_SLAB_FREELIST_RANDOM=y
-CONFIG_SLAB_FREELIST_HARDENED=y
-# CONFIG_SLUB_STATS is not set
-CONFIG_SLUB_CPU_PARTIAL=y
-# end of SLAB allocator options
-
-CONFIG_SHUFFLE_PAGE_ALLOCATOR=y
-# CONFIG_COMPAT_BRK is not set
+CONFIG_SELECT_MEMORY_MODEL=y
+CONFIG_SPARSEMEM_MANUAL=y
 CONFIG_SPARSEMEM=y
+CONFIG_NEED_MULTIPLE_NODES=y
 CONFIG_SPARSEMEM_EXTREME=y
 CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
 CONFIG_SPARSEMEM_VMEMMAP=y
 CONFIG_HAVE_FAST_GUP=y
 CONFIG_NUMA_KEEP_MEMINFO=y
 CONFIG_MEMORY_ISOLATION=y
-CONFIG_EXCLUSIVE_SYSTEM_RAM=y
 CONFIG_HAVE_BOOTMEM_INFO_NODE=y
 CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
-CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
 CONFIG_MEMORY_HOTPLUG=y
+CONFIG_MEMORY_HOTPLUG_SPARSE=y
 CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
+CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
 CONFIG_MEMORY_HOTREMOVE=y
 CONFIG_MHP_MEMMAP_ON_MEMORY=y
 CONFIG_SPLIT_PTLOCK_CPUS=4
@@ -1064,7 +999,6 @@ CONFIG_BALLOON_COMPACTION=y
 CONFIG_COMPACTION=y
 CONFIG_PAGE_REPORTING=y
 CONFIG_MIGRATION=y
-CONFIG_DEVICE_MIGRATION=y
 CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=y
 CONFIG_ARCH_ENABLE_THP_MIGRATION=y
 CONFIG_CONTIG_ALLOC=y
@@ -1076,66 +1010,54 @@ CONFIG_DEFAULT_MMAP_MIN_ADDR=65536
 CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
 CONFIG_MEMORY_FAILURE=y
 CONFIG_HWPOISON_INJECT=m
-CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
-CONFIG_ARCH_WANTS_THP_SWAP=y
 CONFIG_TRANSPARENT_HUGEPAGE=y
-CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
-# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
+# CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is not set
+CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y
+CONFIG_ARCH_WANTS_THP_SWAP=y
 CONFIG_THP_SWAP=y
-CONFIG_READ_ONLY_THP_FOR_FS=y
-CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
-CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
-CONFIG_USE_PERCPU_NUMA_NODE_ID=y
-CONFIG_HAVE_SETUP_PER_CPU_AREA=y
+CONFIG_CLEANCACHE=y
 CONFIG_FRONTSWAP=y
-CONFIG_CMA=y
-# CONFIG_CMA_DEBUG is not set
-CONFIG_CMA_DEBUGFS=y
-CONFIG_CMA_SYSFS=y
-CONFIG_CMA_AREAS=7
+# CONFIG_CMA is not set
 CONFIG_MEM_SOFT_DIRTY=y
+CONFIG_ZSWAP=y
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_DEFLATE is not set
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_842 is not set
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4HC is not set
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_ZSTD is not set
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo"
+CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
+CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud"
+# CONFIG_ZSWAP_DEFAULT_ON is not set
+CONFIG_ZPOOL=y
+CONFIG_ZBUD=y
+CONFIG_Z3FOLD=m
+CONFIG_ZSMALLOC=y
+# CONFIG_ZSMALLOC_STAT is not set
 CONFIG_GENERIC_EARLY_IOREMAP=y
 # CONFIG_DEFERRED_STRUCT_PAGE_INIT is not set
-CONFIG_PAGE_IDLE_FLAG=y
 CONFIG_IDLE_PAGE_TRACKING=y
 CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
-CONFIG_ARCH_HAS_CURRENT_STACK_POINTER=y
-CONFIG_ARCH_HAS_VM_GET_PAGE_PROT=y
 CONFIG_ARCH_HAS_PTE_DEVMAP=y
-CONFIG_ZONE_DMA=y
-CONFIG_ZONE_DMA32=y
 CONFIG_ZONE_DEVICE=y
+CONFIG_DEV_PAGEMAP_OPS=y
 CONFIG_HMM_MIRROR=y
 CONFIG_DEVICE_PRIVATE=y
 CONFIG_VMAP_PFN=y
 CONFIG_ARCH_USES_HIGH_VMA_FLAGS=y
 CONFIG_ARCH_HAS_PKEYS=y
-CONFIG_VM_EVENT_COUNTERS=y
 # CONFIG_PERCPU_STATS is not set
 # CONFIG_GUP_TEST is not set
+# CONFIG_READ_ONLY_THP_FOR_FS is not set
 CONFIG_ARCH_HAS_PTE_SPECIAL=y
 CONFIG_MAPPING_DIRTY_HELPERS=y
-CONFIG_SECRETMEM=y
-CONFIG_ANON_VMA_NAME=y
-CONFIG_USERFAULTFD=y
-CONFIG_HAVE_ARCH_USERFAULTFD_WP=y
-CONFIG_HAVE_ARCH_USERFAULTFD_MINOR=y
-CONFIG_PTE_MARKER=y
-CONFIG_PTE_MARKER_UFFD_WP=y
-
-#
-# Data Access Monitoring
-#
-CONFIG_DAMON=y
-CONFIG_DAMON_VADDR=y
-CONFIG_DAMON_PADDR=y
-CONFIG_DAMON_SYSFS=y
-CONFIG_DAMON_DBGFS=y
-CONFIG_DAMON_RECLAIM=y
-# end of Data Access Monitoring
 # end of Memory Management options
 
 CONFIG_NET=y
+CONFIG_WANT_COMPAT_NETLINK_MESSAGES=y
 CONFIG_COMPAT_NETLINK_MESSAGES=y
 CONFIG_NET_INGRESS=y
 CONFIG_NET_EGRESS=y

Thanks, no, not notepad :-) emacs. I haven't even noticed I have CRLF on. However, the above normal diff is quite verbose so I pasted a linux's scripts/diffconfig output towards the end of this post. It will hopefully let more eyeballs see it.

Edit: possibly very relevant

-CONFIG_ZSWAP_DEFAULT_ON=y
+# CONFIG_ZSWAP_DEFAULT_ON is not set

-# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO is not set
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZO=y

-CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4=y
+# CONFIG_ZSWAP_COMPRESSOR_DEFAULT_LZ4 is not set

-CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lz4"
+CONFIG_ZSWAP_COMPRESSOR_DEFAULT="lzo"

-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD is not set
+CONFIG_ZSWAP_ZPOOL_DEFAULT_ZBUD=y

-CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD=y
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_Z3FOLD is not set

-# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set
+# CONFIG_ZSWAP_ZPOOL_DEFAULT_ZSMALLOC is not set

-CONFIG_ZSWAP_ZPOOL_DEFAULT="z3fold"
+CONFIG_ZSWAP_ZPOOL_DEFAULT="zbud"

Yes, this is very interesting, thank you. I just re-checked and indeed in stock Arch config CONFIG_ZSWAP_DEFAULT_ON is set to yes, while my config disables it. This might be it. It seems zswap can be enabled/disabled at runtime too, so I will be able to check this without having to reboot into the stock kernel. I can enable zswap on my custom kernel at runtime and see if it degrades the same way.

Edit #2

-CONFIG_SWAP=y

+CONFIG_VM_EVENT_COUNTERS=y
-CONFIG_HAVE_ARCH_HUGE_VMALLOC=y
-CONFIG_ARCH_HAS_VM_GET_PAGE_PROT=y
-CONFIG_VM_EVENT_COUNTERS=y
-CONFIG_ANON_VMA_NAME=y

CONFIG_SWAP is on in my running kernel (otherwise there would be no swap at all). This is because I run make oldconfig with the file I posted and although it is missing 'CONFIG_SWAP=y' it also doesn't have the '#CONFIG_SWAP is not set' line in which case make oldconfig sets the value to default which is on.

This time I wen to the kernel build folder and I got the .config file. I then run scripts/diffconfig on both files so the below shows the difference from Arch stock to custom in more succint form:

-ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP y
-BPF_PRELOAD_UMD m
-CMA_ALIGNMENT 8
-CMA_AREAS 7
-CMA_DEBUG n
-CMA_DEBUGFS y
-CMA_SIZE_MBYTES 0
-CMA_SIZE_SEL_MAX n
-CMA_SIZE_SEL_MBYTES y
-CMA_SIZE_SEL_MIN n
-CMA_SIZE_SEL_PERCENTAGE n
-CMA_SYSFS y
-DAMON_DBGFS y
-DAMON_PADDR y
-DAMON_RECLAIM y
-DAMON_SYSFS y
-DAMON_VADDR y
-DEBUG_KMAP_LOCAL_FORCE_MAP n
-DMABUF_HEAPS_CMA y
-DMA_CMA y
-DMA_PERNUMA_CMA n
-FORCE_TASKS_RCU n
-FORCE_TASKS_RUDE_RCU n
-FORCE_TASKS_TRACE_RCU n
-RCU_BOOST y
-RCU_BOOST_DELAY 500
-RCU_EXP_KTHREAD n
-RCU_FANOUT 64
-RCU_FANOUT_LEAF 16
-RCU_NOCB_CPU y
-TASKS_TRACE_RCU_READ_MB n
-USERMODE_DRIVER y
 ANON_VMA_NAME y -> n
 ARCH_MEMORY_PROBE n -> y
 BLK_CGROUP_FC_APPID y -> n
 BLK_CGROUP_IOLATENCY y -> n
 BLK_DEV_THROTTLING_LOW y -> n
 BPF_PRELOAD y -> n
 CMA y -> n
 CONTEXT_TRACKING_FORCE n -> y
 CRYPTO_LZO m -> y
 DAMON y -> n
 GART_IOMMU n -> y
 HZ 300 -> 250
 HZ_250 n -> y
 HZ_300 y -> n
 IKCONFIG_PROC y -> n
 INTEL_TDX_GUEST y -> n
 MAXSMP n -> y
 NODES_SHIFT 5 -> 10
 NO_HZ_FULL y -> n
 NO_HZ_IDLE n -> y
 NR_CPUS 320 -> 8192
 NR_CPUS_DEFAULT 64 -> 8192
 NR_CPUS_RANGE_BEGIN 2 -> 8192
 NR_CPUS_RANGE_END 512 -> 8192
 PREEMPT y -> n
 PREEMPT_VOLUNTARY n -> y
 PRINTK_INDEX y -> n
 RCU_EXPERT y -> n
 READ_ONLY_THP_FOR_FS y -> n
 SCHED_CORE y -> n
 SLS y -> n
 TRANSPARENT_HUGEPAGE_ALWAYS y -> n
 TRANSPARENT_HUGEPAGE_MADVISE n -> y
 X86_ACPI_CPUFREQ m -> y
 X86_CPA_STATISTICS y -> n
 X86_EXTENDED_PLATFORM n -> y
 X86_KERNEL_IBT y -> n
 X86_PCC_CPUFREQ m -> y
 Z3FOLD y -> m
 ZSWAP_COMPRESSOR_DEFAULT "lz4" -> "lzo"
 ZSWAP_COMPRESSOR_DEFAULT_LZ4 y -> n
 ZSWAP_COMPRESSOR_DEFAULT_LZO n -> y
 ZSWAP_DEFAULT_ON y -> n
 ZSWAP_ZPOOL_DEFAULT "z3fold" -> "zbud"
 ZSWAP_ZPOOL_DEFAULT_Z3FOLD y -> n
 ZSWAP_ZPOOL_DEFAULT_ZBUD n -> y
+CPUMASK_OFFSTACK y
+IOMMU_DEBUG n
+IOMMU_HELPER y
+SGI_GRU n
+SGI_XP n
+TICK_CPU_ACCOUNTING n
+UV_MMTIMER m
+UV_SYSFS n
+X86_GOLDFISH n
+X86_INTEL_MID n
+X86_NUMACHIP y
+X86_UV y
+X86_VSMP n

sabroad · 2022-10-01 23:27:46

7000k wrote:

This is the stock Arch kernel (fails): https://pastebin.com/Uude2vfe https://file.io/gPb1GuosvWE7
This is the custom kernel (works): https://pastebin.com/HSWNpiju https://file.io/ricYUv1fSeb1

Cute, but there must be something wrong with your processing of

cat /proc/vmstat

since they're both essentially identical (highly unlikely given the difference in aforementioned swap usage):

$ sdiff -s <(cut -d, -f1,3- stock\ kernel\ csv.txt ) custom\ kernel\ csv.txt
timestamp,nr_zone_inactive_anon,nr_zone_active_anon,nr_z	|	nr_free_pages,nr_zone_inactive_anon,nr_zone_active_anon,nr_zo
308,3722,922,57376,12499,0,50,0,0,0,0,29233906,0,0,2475,29233 /	308,3722,922,57376,12499,0,50,0,0,0,0,29233906,0,0,2475,29233

Just the raw output 30s apart should do.

Last edited by sabroad (2022-10-01 23:30:02)

7000k · 2022-10-11 12:13:11

sabroad wrote:

7000k wrote:
This is the stock Arch kernel (fails): https://pastebin.com/Uude2vfe https://file.io/gPb1GuosvWE7
This is the custom kernel (works): https://pastebin.com/HSWNpiju https://file.io/ricYUv1fSeb1
Cute, but there must be something wrong with your processing of
cat /proc/vmstat
since they're both essentially identical (highly unlikely given the difference in aforementioned swap usage):

Thank you.

However, I messed up the files before upload. I cut one column from the "custom kernel" file. Also I used one set of column names for both files(copy/paste) while as they came out from my script they were different. No doubt due to different kernel configs.

My mistake. Nevermind. Here are corrected files(again timestamps are replaced with row numbers, also vm.lowmem_reserve_ratio is reported as a set of 5 numbers, this was split into 5 columns manually to make comparisons easier):

Stock kernel (broken): https://drive.google.com/file/d/1omVM8R … sp=sharing
Custom kernel(working): https://drive.google.com/file/d/1pcEGlN … sp=sharing

This is on google drive so it should remain online.

$ sdiff -s <(cut -d, -f1,3- stock\ kernel\ csv.txt ) custom\ kernel\ csv.text
timestamp,nr_zone_inactive_anon,nr_zone_active_anon,nr_z	|	nr_free_pages,nr_zone_inactive_anon,nr_zone_active_anon,nr_zo
308,3722,922,57376,12499,0,50,0,0,0,0,29233906,0,0,2475,29233 /	308,3722,922,57376,12499,0,50,0,0,0,0,29233906,0,0,2475,29233

Just the raw output 30s apart should do.

From which moment? IMO it is completely pointless to just grab two snapshots 30s apart on two dynamic processes we don't really know at which point they diverge. This is why I saved the data that was mentioned previously every second for the duration of the process. Also, the reason is that I can't really run it too often because it requires a reboot of my main PC.

The best is to load the files into Libre Calc, Excel or similar then one can select data from 30s apart by selecting a row, and another one 30 rows further. Columns captured differ due to difference in kernel config.

However, for you I've extracted 4 rows (two from each kernel run 30s apart). Those were taken at 180s from process start and then at 210s mark. I selected to skip first 3 minutes because that's how long loading of the model into memory took.

This is custom kernel(works)

timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin   pgpgout  pswpin  pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree   pgactivate pgdeactivate pglazyfree pgfault  pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode 
180       1941284       6101460               1754                10574                 7294                0                   33                    0        0         0          0           27477729 0         0            2475            27477729   0          6101460          1754           10574            7294           0              20183               25558                 346              0                59096            452279                  4829242                 433438                   3157732                  135                     9998                    2221                   6102420       4326      32922         33       0            0                 359      0                  0                  0                 0                 0                             2872836         41                          29848      2902645    0                    0                          0                    0                    5448            14242               14513         0                 378605             189071                        26394549 11610564 452368  2872836 1025        2699131       24813240       0               0              0                357               36326              0          0            2609918       0              30715851 2616392    4460740      0          14971905 84650      0           4990531  53544   11111093       3224881        0               0               17704401      10988213      0                      15077092    13615522    2846440      11489534     0                   0            62557         4431              2331                         73                            2410       2843630   0              0         0        0                0                     0                0                      0                   1208170           26             0                     0                  0                   1461470                 3973865              2435329          0             0            0               1333                248286                         994284                      0                        0                     6033                   0                       0                       0                       0                         0                       0                        0               0                  0                         0                  0                         0              0                 0                        0               0              0                     0                       0             0                        0                        0                         0             0                   0                          0          0                   0               0               0               368204  367106      0              0       0      0       121                      0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0                    
210       990735        6992475               30158               37868                 10811               0                   46                    0        0         0          0           28901242 0         0            2475            28901242   0          6992475          30158          37868            10811          0              20179               25626                 0                0                58636            1277506                 4853361                 461842                   3160455                  135                     12396                   2221                   7021408       9497      886594        46       0            0                 359      0                  0                  0                 0                 0                             2872836         41                          31268      2904052    0                    0                          0                    0                    5520            14454               837555        0                 194702             97232                         29814053 11616212 1277506 2872836 1025        2699131       26236751       0               0              0                357               36326              0          0            2609918       0              31546120 2616964    4460740      0          16368153 916648     0           4990531  55287   11111093       3224881        0               0               17704401      10988213      0                      15077092    13615522    2846440      11489534     0                   0            62557         4431              2331                         73                            2410       2843654   0              0         0        0                0                     0                0                      0                   1564842           26             0                     0                  0                   2141187                 6714068              3147752          0             0            0               1333                248286                         994284                      0                        0                     6033                   0                       0                       0                       0                         0                       0                        0               0                  0                         0                  0                         0              0                 0                        0               0              0                     0                       0             0                        0                        0                         0             0                   0                          0          0                   0               0               0               370178  368905      0              0       0      0       121                      0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0

This is stock kernel(broken):

timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin   pgpgout  pswpin  pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree   pgactivate pgdeactivate pglazyfree pgfault pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail cma_alloc_success cma_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode 
180       62704         1742969               11388               3484972               2761940             0                   40                    0        0         0          0           14500716 0         0            2448            14500716   0          1742969          11388          3484972          2761940        0              29966               25737                 0                0                13905            148                     6389469                 99                       4831303                  2                       10                      0                      1753986       43350     6247420       40       0            0                 383      0                  0                  0                 0                 3340                          1485104         0                           16019      1501080    0                    0                          0                    0                    5040            6922                98            0                 1249077            623776                        27659569 5942960  104     1469719 1025        1233176       13300887       0               0              0                0                 0                  0          0            0             0              16357181 87154      2147272      0          578266  416        0           2152098  35944   8765311        0              0               0               12493291      0             0                      5166041     7327250     1485067      7280244      0                   0            8192          0                 837                          243                           1084       1469682   0              0         0        0                0                     0                0                      0                   5680              0              0                     0                  0                   28430                   98365                11878            0             0            0               43                  28430                          98365                       0                        0                     0                 0              6154                   0                       0                       0                       0                         0                       0                        12763           21                 0                         40                 0                         0              0                 0                        0               0              0                     3                       2834          0                        24                       0                         0             0                   0                          2831       0                   0               0               0               110     67          0              0       44     15385   92                       0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0                    
210       66830         6203412               109528              5688                  3380                0                   25                    0        0         0          0           24360205 0         0            2448            24360205   0          6203643          109528         5716             3380           0              30746               51385                 0                0                58880            3452052                 6935356                 98482                    5371579                  2                       2289                    2112                   6296023       2395      2121284       25       0            0                 320      0                  0                  0                 0                 3575                          6209290         53                          17639      6226920    0                    0                          0                    0                    5168            18780               2111856       0                 2051               1024                          42579105 18307704 3450961 4559265 1025        2152313       22241749       0               0              0                167               12745              0          0            971591        0              26408120 117043     5471949      0          8735187 578255     0           6417191  37831   19446349       2161862        0               0               33081116      4143427       0                      23058616    14165927    7548510      14059701     0                   0            43448         1420              3682                         302                           3996       4558831   0              0         0        0                0                     0                0                      0                   51095             22             0                     0                  0                   195362                  441148               107781           0             0            0               2078                195362                         441148                      0                        0                     0                 0              6154                   0                       0                       0                       0                         0                       0                        18768           7890               0                         40                 0                         0              0                 0                        0               0              0                     3                       8604          0                        40                       0                         0             0                   0                          8601       0                   0               0               0               2874398 2874123     0              0       1256   1650044 92                       0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0

This is again but with columns that don't change removed:

Custom kernel(working):

timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_write_pending numa_hit numa_interleave numa_local nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_file nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_dirtied nr_written nr_kernel_stack nr_page_table_pages nr_swapcached nr_dirty_threshold nr_dirty_background_threshold pgpgin   pgpgout  pswpin  pswpout pgalloc_dma32 pgalloc_normal pgfree   pgactivate pgfault  pgmajfault pgreuse pgrotated pgmigrate_success compact_migrate_scanned compact_free_scanned compact_isolated compact_daemon_free_scanned swap_ra swap_ra_hit 
180       1941284       6101460               1754                10574                 7294                33                    27477729 2475            27477729   6101460          1754           10574            7294           20183               25558                 346              59096            452279                  4829242                 433438                   3157732                  9998                    6102420       4326      32922         33       29848      2902645    5448            14242               14513         378605             189071                        26394549 11610564 452368  2872836 2699131       24813240       30715851 2616392    14971905 84650      53544   2843630   1208170           1461470                 3973865              2435329          994284                      368204  367106      
210       990735        6992475               30158               37868                 10811               46                    28901242 2475            28901242   6992475          30158          37868            10811          20179               25626                 0                58636            1277506                 4853361                 461842                   3160455                  12396                   7021408       9497      886594        46       31268      2904052    5520            14454               837555        194702             97232                         29814053 11616212 1277506 2872836 2699131       26236751       31546120 2616964    16368153 916648     55287   2843654   1564842           2141187                 6714068              3147752          994284                      370178  368905

Stock(broken): - it has many more columns, because many more columns changed during those 30 seconds -

timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_write_pending numa_hit numa_interleave numa_local nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_shmem nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_kernel_stack nr_page_table_pages nr_swapcached nr_dirty_threshold nr_dirty_background_threshold pgpgin   pgpgout  pswpin  pswpout pgalloc_dma32 pgalloc_normal allocstall_normal allocstall_movable pgskip_normal pgfree   pgactivate pgdeactivate pgfault pgmajfault pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgscan_kswapd pgscan_direct pgscan_anon pgscan_file pgsteal_anon pgsteal_file slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated pgmigrate_success pgmigrate_fail compact_migrate_scanned compact_free_scanned compact_isolated compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned unevictable_pgs_culled thp_fault_alloc thp_fault_fallback thp_split_pmd thp_scan_exceed_swap_pte thp_swpout swap_ra swap_ra_hit zswpin zswpout 
180       62704         1742969               11388               3484972               2761940             40                    14500716 2448            14500716   1742969          11388          3484972          2761940        29966               25737                 0                13905            148                     6389469                 99                       4831303                  10                      0                      1753986       43350     6247420       40       383      3340                          1485104         0                           16019      1501080    5040            6922                98            1249077            623776                        27659569 5942960  104     1469719 1233176       13300887       0                 0                  0             16357181 87154      2147272      578266  416        2152098  35944   8765311        0              12493291      0             5166041     7327250     1485067      7280244      8192          0                 837                          243                           1084       1469682   5680              0              28430                   98365                11878            43                  28430                          98365                       6154                   12763           21                 2834          24                       2831       110     67          44     15385   
210       66830         6203412               109528              5688                  3380                25                    24360205 2448            24360205   6203643          109528         5716             3380           30746               51385                 0                58880            3452052                 6935356                 98482                    5371579                  2289                    2112                   6296023       2395      2121284       25       320      3575                          6209290         53                          17639      6226920    5168            18780               2111856       2051               1024                          42579105 18307704 3450961 4559265 2152313       22241749       167               12745              971591        26408120 117043     5471949      8735187 578255     6417191  37831   19446349       2161862        33081116      4143427       23058616    14165927    7548510      14059701     43448         1420              3682                         302                           3996       4558831   51095             22             195362                  441148               107781           2078                195362                         441148                      6154                   18768           7890               8604          40                       8601       2874398 2874123     1256   1650044

I hope this helps to get to the bottom of it. I suppose increasingly more people will want to run inference of large models on their CPUs and not all of those models will fit in the RAM. Therefore we will see more users being hit by this and it will be beneficial to arch to find out which default setting could be changed to avoid this issue.

ugjka · 2022-10-11 12:24:21

Maybe this is of help to you https://unix.stackexchange.com/question … -available

sabroad · 2022-10-14 16:52:14

7000k wrote:

Stock kernel (broken): https://drive.google.com/file/d/1omVM8R … sp=sharing
Custom kernel(working): https://drive.google.com/file/d/1pcEGlN … sp=sharing

Unfortunately, Google requires login (or tracking cookies) to download.

7000k wrote:

This is stock kernel(broken):

timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin   pgpgout  pswpin  pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree   pgactivate pgdeactivate pglazyfree pgfault pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail cma_alloc_success cma_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode 
180       62704         1742969               11388               3484972               2761940             0                   40                    0        0         0          0           14500716 0         0            2448            14500716   0          1742969          11388          3484972          2761940        0              29966               25737                 0                0                13905            148                     6389469                 99                       4831303                  2                       10                      0                      1753986       43350     6247420       40       0            0                 383      0                  0                  0                 0                 3340                          1485104         0                           16019      1501080    0                    0                          0                    0                    5040            6922                98            0                 1249077            623776                        27659569 5942960  104     1469719 1025        1233176       13300887       0               0              0                0                 0                  0          0            0             0              16357181 87154      2147272      0          578266  416        0           2152098  35944   8765311        0              0               0               12493291      0             0                      5166041     7327250     1485067      7280244      0                   0            8192          0                 837                          243                           1084       1469682   0              0         0        0                0                     0                0                      0                   5680              0              0                     0                  0                   28430                   98365                11878            0             0            0               43                  28430                          98365                       0                        0                     0                 0              6154                   0                       0                       0                       0                         0                       0                        12763           21                 0                         40                 0                         0              0                 0                        0               0              0                     3                       2834          0                        24                       0                         0             0                   0                          2831       0                   0               0               0               110     67          0              0       44     15385   92                       0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0                    
210       66830         6203412               109528              5688                  3380                0                   25                    0        0         0          0           24360205 0         0            2448            24360205   0          6203643          109528         5716             3380           0              30746               51385                 0                0                58880            3452052                 6935356                 98482                    5371579                  2                       2289                    2112                   6296023       2395      2121284       25       0            0                 320      0                  0                  0                 0                 3575                          6209290         53                          17639      6226920    0                    0                          0                    0                    5168            18780               2111856       0                 2051               1024                          42579105 18307704 3450961 4559265 1025        2152313       22241749       0               0              0                167               12745              0          0            971591        0              26408120 117043     5471949      0          8735187 578255     0           6417191  37831   19446349       2161862        0               0               33081116      4143427       0                      23058616    14165927    7548510      14059701     0                   0            43448         1420              3682                         302                           3996       4558831   0              0         0        0                0                     0                0                      0                   51095             22             0                     0                  0                   195362                  441148               107781           0             0            0               2078                195362                         441148                      0                        0                     0                 0              6154                   0                       0                       0                       0                         0                       0                        18768           7890               0                         40                 0                         0              0                 0                        0               0              0                     3                       8604          0                        40                       0                         0             0                   0                          8601       0                   0               0               0               2874398 2874123     0              0       1256   1650044 92                       0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0

Looks like the system is under severe memory pressure, with writes/allocations whilst scanning pages to swap/free/compact:

nr_vmscan_write	6209290

7000k wrote:

This is custom kernel(works)

timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin   pgpgout  pswpin  pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree   pgactivate pgdeactivate pglazyfree pgfault  pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode 
180       1941284       6101460               1754                10574                 7294                0                   33                    0        0         0          0           27477729 0         0            2475            27477729   0          6101460          1754           10574            7294           0              20183               25558                 346              0                59096            452279                  4829242                 433438                   3157732                  135                     9998                    2221                   6102420       4326      32922         33       0            0                 359      0                  0                  0                 0                 0                             2872836         41                          29848      2902645    0                    0                          0                    0                    5448            14242               14513         0                 378605             189071                        26394549 11610564 452368  2872836 1025        2699131       24813240       0               0              0                357               36326              0          0            2609918       0              30715851 2616392    4460740      0          14971905 84650      0           4990531  53544   11111093       3224881        0               0               17704401      10988213      0                      15077092    13615522    2846440      11489534     0                   0            62557         4431              2331                         73                            2410       2843630   0              0         0        0                0                     0                0                      0                   1208170           26             0                     0                  0                   1461470                 3973865              2435329          0             0            0               1333                248286                         994284                      0                        0                     6033                   0                       0                       0                       0                         0                       0                        0               0                  0                         0                  0                         0              0                 0                        0               0              0                     0                       0             0                        0                        0                         0             0                   0                          0          0                   0               0               0               368204  367106      0              0       0      0       121                      0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0                    
210       990735        6992475               30158               37868                 10811               0                   46                    0        0         0          0           28901242 0         0            2475            28901242   0          6992475          30158          37868            10811          0              20179               25626                 0                0                58636            1277506                 4853361                 461842                   3160455                  135                     12396                   2221                   7021408       9497      886594        46       0            0                 359      0                  0                  0                 0                 0                             2872836         41                          31268      2904052    0                    0                          0                    0                    5520            14454               837555        0                 194702             97232                         29814053 11616212 1277506 2872836 1025        2699131       26236751       0               0              0                357               36326              0          0            2609918       0              31546120 2616964    4460740      0          16368153 916648     0           4990531  55287   11111093       3224881        0               0               17704401      10988213      0                      15077092    13615522    2846440      11489534     0                   0            62557         4431              2331                         73                            2410       2843654   0              0         0        0                0                     0                0                      0                   1564842           26             0                     0                  0                   2141187                 6714068              3147752          0             0            0               1333                248286                         994284                      0                        0                     6033                   0                       0                       0                       0                         0                       0                        0               0                  0                         0                  0                         0              0                 0                        0               0              0                     0                       0             0                        0                        0                         0             0                   0                          0          0                   0               0               0               370178  368905      0              0       0      0       121                      0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0

With the custom config, the system is under memory pressure, but less so:

nr_vmscan_write	2872836

Let's narrow the largest differences:

vmstat	vanilla	custom
compact_free_scanned	441148	6714068	# custom config is freeing >10x more pages for contiguous allocation
compact_isolated	107781	3147752
compact_migrate_scanned	195362	2141187	# custom config is migrating >10x more pages for contiguous allocation 
nr_anon_transparent_hugepages	3575	0
nr_dirty_background_threshold	1024	97232
nr_dirty_threshold	2051	194702
nr_free_pages	66830	990735
pgactivate	117043	2616964	# custom config is reactivating pages that have gone inactive
pgmigrate_success	51095	1564842
swap_ra_hit	2874123	368905
thp_collapse_alloc	40	0
thp_deferred_split_page	3	0
thp_fault_alloc	18768	0	# custom config is not allocating transparent huge pages
thp_fault_fallback	7890	0
thp_scan_exceed_swap_pte	40	0
thp_split_pmd	8604	0
thp_swpout	8601	0
workingset_restore_anon	2	135
zswpin	1256	0
zswpout	1650044	0

With this, we can conclude:
* specific workload allocates memory in chunks (contiguous allocation);
* memory pressure is freeing sparse pages (causing fragmentation);
* specific workload does not access memory contiguously (allowing sparse pages to deactivate);
* using THP prevents sparse pages being freed/migrated effectively;

Memory fragmentation can be confirmed with kcompactd cpu-usage and examining the buddyinfo allocator

cat /proc/buddyinfo

7000k wrote:

increasingly more people will want to run inference of large models on their CPUs and not all of those models will fit in the RAM. Therefore we will see more users being hit by this and it will be beneficial to arch to find out which default setting could be changed to avoid this issue.

Disable THP with:

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

THP is great for latency (+10%) [1] but terrible for fragmentation [2] where memory pressure is high.

[1] https://alexandrnikitin.github.io/blog/ … ce-impact/
[2] https://www.pingcap.com/blog/transparen … databases/

Last edited by sabroad (2022-10-14 16:53:36)

7000k · 2022-10-20 22:00:11

sabroad wrote:

7000k wrote:
Stock kernel (broken): https://drive.google.com/file/d/1omVM8R … sp=sharing
Custom kernel(working): https://drive.google.com/file/d/1pcEGlN … sp=sharing
Unfortunately, Google requires login (or tracking cookies) to download.

Thanks for the reply.

In a sense all cookies are tracking cookies.
Personally I don't get what is the technical problem with allowing momentary tracking cookies in a private browsing session. One can open a private browsing session in any modern browser and cookies set in that session do not persist outside it.

However, if you(or anyone else) know of any alternative one can upload 350k text file and share it without cookies please do let me know.

7000k wrote:

This is stock kernel(broken):

timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin   pgpgout  pswpin  pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree   pgactivate pgdeactivate pglazyfree pgfault pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail cma_alloc_success cma_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode 
180       62704         1742969               11388               3484972               2761940             0                   40                    0        0         0          0           14500716 0         0            2448            14500716   0          1742969          11388          3484972          2761940        0              29966               25737                 0                0                13905            148                     6389469                 99                       4831303                  2                       10                      0                      1753986       43350     6247420       40       0            0                 383      0                  0                  0                 0                 3340                          1485104         0                           16019      1501080    0                    0                          0                    0                    5040            6922                98            0                 1249077            623776                        27659569 5942960  104     1469719 1025        1233176       13300887       0               0              0                0                 0                  0          0            0             0              16357181 87154      2147272      0          578266  416        0           2152098  35944   8765311        0              0               0               12493291      0             0                      5166041     7327250     1485067      7280244      0                   0            8192          0                 837                          243                           1084       1469682   0              0         0        0                0                     0                0                      0                   5680              0              0                     0                  0                   28430                   98365                11878            0             0            0               43                  28430                          98365                       0                        0                     0                 0              6154                   0                       0                       0                       0                         0                       0                        12763           21                 0                         40                 0                         0              0                 0                        0               0              0                     3                       2834          0                        24                       0                         0             0                   0                          2831       0                   0               0               0               110     67          0              0       44     15385   92                       0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0                    
210       66830         6203412               109528              5688                  3380                0                   25                    0        0         0          0           24360205 0         0            2448            24360205   0          6203643          109528         5716             3380           0              30746               51385                 0                0                58880            3452052                 6935356                 98482                    5371579                  2                       2289                    2112                   6296023       2395      2121284       25       0            0                 320      0                  0                  0                 0                 3575                          6209290         53                          17639      6226920    0                    0                          0                    0                    5168            18780               2111856       0                 2051               1024                          42579105 18307704 3450961 4559265 1025        2152313       22241749       0               0              0                167               12745              0          0            971591        0              26408120 117043     5471949      0          8735187 578255     0           6417191  37831   19446349       2161862        0               0               33081116      4143427       0                      23058616    14165927    7548510      14059701     0                   0            43448         1420              3682                         302                           3996       4558831   0              0         0        0                0                     0                0                      0                   51095             22             0                     0                  0                   195362                  441148               107781           0             0            0               2078                195362                         441148                      0                        0                     0                 0              6154                   0                       0                       0                       0                         0                       0                        18768           7890               0                         40                 0                         0              0                 0                        0               0              0                     3                       8604          0                        40                       0                         0             0                   0                          8601       0                   0               0               0               2874398 2874123     0              0       1256   1650044 92                       0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0

Looks like the system is under severe memory pressure, with writes/allocations whilst scanning pages to swap/free/compact:

nr_vmscan_write	6209290

7000k wrote:

This is custom kernel(works)

timestamp nr_free_pages nr_zone_inactive_anon nr_zone_active_anon nr_zone_inactive_file nr_zone_active_file nr_zone_unevictable nr_zone_write_pending nr_mlock nr_bounce nr_zspages nr_free_cma numa_hit numa_miss numa_foreign numa_interleave numa_local numa_other nr_inactive_anon nr_active_anon nr_inactive_file nr_active_file nr_unevictable nr_slab_reclaimable nr_slab_unreclaimable nr_isolated_anon nr_isolated_file workingset_nodes workingset_refault_anon workingset_refault_file workingset_activate_anon workingset_activate_file workingset_restore_anon workingset_restore_file workingset_nodereclaim nr_anon_pages nr_mapped nr_file_pages nr_dirty nr_writeback nr_writeback_temp nr_shmem nr_shmem_hugepages nr_shmem_pmdmapped nr_file_hugepages nr_file_pmdmapped nr_anon_transparent_hugepages nr_vmscan_write nr_vmscan_immediate_reclaim nr_dirtied nr_written nr_throttled_written nr_kernel_misc_reclaimable nr_foll_pin_acquired nr_foll_pin_released nr_kernel_stack nr_page_table_pages nr_swapcached pgpromote_success nr_dirty_threshold nr_dirty_background_threshold pgpgin   pgpgout  pswpin  pswpout pgalloc_dma pgalloc_dma32 pgalloc_normal pgalloc_movable allocstall_dma allocstall_dma32 allocstall_normal allocstall_movable pgskip_dma pgskip_dma32 pgskip_normal pgskip_movable pgfree   pgactivate pgdeactivate pglazyfree pgfault  pgmajfault pglazyfreed pgrefill pgreuse pgsteal_kswapd pgsteal_direct pgdemote_kswapd pgdemote_direct pgscan_kswapd pgscan_direct pgscan_direct_throttle pgscan_anon pgscan_file pgsteal_anon pgsteal_file zone_reclaim_failed pginodesteal slabs_scanned kswapd_inodesteal kswapd_low_wmark_hit_quickly kswapd_high_wmark_hit_quickly pageoutrun pgrotated drop_pagecache drop_slab oom_kill numa_pte_updates numa_huge_pte_updates numa_hint_faults numa_hint_faults_local numa_pages_migrated pgmigrate_success pgmigrate_fail thp_migration_success thp_migration_fail thp_migration_split compact_migrate_scanned compact_free_scanned compact_isolated compact_stall compact_fail compact_success compact_daemon_wake compact_daemon_migrate_scanned compact_daemon_free_scanned htlb_buddy_alloc_success htlb_buddy_alloc_fail unevictable_pgs_culled unevictable_pgs_scanned unevictable_pgs_rescued unevictable_pgs_mlocked unevictable_pgs_munlocked unevictable_pgs_cleared unevictable_pgs_stranded thp_fault_alloc thp_fault_fallback thp_fault_fallback_charge thp_collapse_alloc thp_collapse_alloc_failed thp_file_alloc thp_file_fallback thp_file_fallback_charge thp_file_mapped thp_split_page thp_split_page_failed thp_deferred_split_page thp_split_pmd thp_scan_exceed_none_pte thp_scan_exceed_swap_pte thp_scan_exceed_share_pte thp_split_pud thp_zero_page_alloc thp_zero_page_alloc_failed thp_swpout thp_swpout_fallback balloon_inflate balloon_deflate balloon_migrate swap_ra swap_ra_hit ksm_swpin_copy cow_ksm zswpin zswpout direct_map_level2_splits direct_map_level3_splits nr_unstable vm.admin_reserve_kbytes vm.compact_unevictable_allowed vm.compaction_proactiveness vm.dirty_background_bytes vm.dirty_background_ratio vm.dirty_bytes vm.dirty_expire_centisecs vm.dirty_ratio vm.dirty_writeback_centisecs vm.dirtytime_expire_seconds vm.extfrag_threshold vm.hugetlb_optimize_vmemmap vm.hugetlb_shm_group vm.laptop_mode vm.legacy_va_layout vm.lowmem_reserve_ratio 1 vm.lowmem_reserve_ratio 2 vm.lowmem_reserve_ratio 3 vm.lowmem_reserve_ratio 4 vm.lowmem_reserve_ratio 5 vm.max_map_count vm.memory_failure_early_kill vm.memory_failure_recovery vm.min_free_kbytes vm.min_slab_ratio vm.min_unmapped_ratio vm.mmap_min_addr vm.mmap_rnd_bits vm.mmap_rnd_compat_bits vm.nr_hugepages vm.nr_hugepages_mempolicy vm.nr_overcommit_hugepages vm.numa_stat vm.numa_zonelist_order vm.oom_dump_tasks vm.oom_kill_allocating_task vm.overcommit_kbytes vm.overcommit_memory vm.overcommit_ratio vm.page-cluster vm.page_lock_unfairness vm.panic_on_oom vm.percpu_pagelist_high_fraction vm.stat_interval vm.swappiness vm.unprivileged_userfaultfd vm.user_reserve_kbytes vm.vfs_cache_pressure vm.watermark_boost_factor vm.watermark_scale_factor vm.zone_reclaim_mode 
180       1941284       6101460               1754                10574                 7294                0                   33                    0        0         0          0           27477729 0         0            2475            27477729   0          6101460          1754           10574            7294           0              20183               25558                 346              0                59096            452279                  4829242                 433438                   3157732                  135                     9998                    2221                   6102420       4326      32922         33       0            0                 359      0                  0                  0                 0                 0                             2872836         41                          29848      2902645    0                    0                          0                    0                    5448            14242               14513         0                 378605             189071                        26394549 11610564 452368  2872836 1025        2699131       24813240       0               0              0                357               36326              0          0            2609918       0              30715851 2616392    4460740      0          14971905 84650      0           4990531  53544   11111093       3224881        0               0               17704401      10988213      0                      15077092    13615522    2846440      11489534     0                   0            62557         4431              2331                         73                            2410       2843630   0              0         0        0                0                     0                0                      0                   1208170           26             0                     0                  0                   1461470                 3973865              2435329          0             0            0               1333                248286                         994284                      0                        0                     6033                   0                       0                       0                       0                         0                       0                        0               0                  0                         0                  0                         0              0                 0                        0               0              0                     0                       0             0                        0                        0                         0             0                   0                          0          0                   0               0               0               368204  367106      0              0       0      0       121                      0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0                    
210       990735        6992475               30158               37868                 10811               0                   46                    0        0         0          0           28901242 0         0            2475            28901242   0          6992475          30158          37868            10811          0              20179               25626                 0                0                58636            1277506                 4853361                 461842                   3160455                  135                     12396                   2221                   7021408       9497      886594        46       0            0                 359      0                  0                  0                 0                 0                             2872836         41                          31268      2904052    0                    0                          0                    0                    5520            14454               837555        0                 194702             97232                         29814053 11616212 1277506 2872836 1025        2699131       26236751       0               0              0                357               36326              0          0            2609918       0              31546120 2616964    4460740      0          16368153 916648     0           4990531  55287   11111093       3224881        0               0               17704401      10988213      0                      15077092    13615522    2846440      11489534     0                   0            62557         4431              2331                         73                            2410       2843654   0              0         0        0                0                     0                0                      0                   1564842           26             0                     0                  0                   2141187                 6714068              3147752          0             0            0               1333                248286                         994284                      0                        0                     6033                   0                       0                       0                       0                         0                       0                        0               0                  0                         0                  0                         0              0                 0                        0               0              0                     0                       0             0                        0                        0                         0             0                   0                          0          0                   0               0               0               370178  368905      0              0       0      0       121                      0                        0           8192                    1                              20                          0                         10                        0              3000                      20             500                          43200                       500                  0                           0                    0              0                   256                       256                       32                        0                         0                         65530            0                            1                          67584              5                 1                     65536            28               8                       0               0                         0                          1            Node                   1                 0                           0                    0                    50                  3               5                       0               0                                1                60            1                           131072                 100                   15000                     10                        0

With the custom config, the system is under memory pressure, but less so:

nr_vmscan_write	2872836

Let's narrow the largest differences:

vmstat	vanilla	custom
compact_free_scanned	441148	6714068	# custom config is freeing >10x more pages for contiguous allocation
compact_isolated	107781	3147752
compact_migrate_scanned	195362	2141187	# custom config is migrating >10x more pages for contiguous allocation 
nr_anon_transparent_hugepages	3575	0
nr_dirty_background_threshold	1024	97232
nr_dirty_threshold	2051	194702
nr_free_pages	66830	990735
pgactivate	117043	2616964	# custom config is reactivating pages that have gone inactive
pgmigrate_success	51095	1564842
swap_ra_hit	2874123	368905
thp_collapse_alloc	40	0
thp_deferred_split_page	3	0
thp_fault_alloc	18768	0	# custom config is not allocating transparent huge pages
thp_fault_fallback	7890	0
thp_scan_exceed_swap_pte	40	0
thp_split_pmd	8604	0
thp_swpout	8601	0
workingset_restore_anon	2	135
zswpin	1256	0
zswpout	1650044	0

With this, we can conclude:
* specific workload allocates memory in chunks (contiguous allocation);
* memory pressure is freeing sparse pages (causing fragmentation);
* specific workload does not access memory contiguously (allowing sparse pages to deactivate);
* using THP prevents sparse pages being freed/migrated effectively;

Memory fragmentation can be confirmed with kcompactd cpu-usage and examining the buddyinfo allocator

cat /proc/buddyinfo

7000k wrote:

increasingly more people will want to run inference of large models on their CPUs and not all of those models will fit in the RAM. Therefore we will see more users being hit by this and it will be beneficial to arch to find out which default setting could be changed to avoid this issue.

Disable THP with:

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

THP is great for latency (+10%) [1] but terrible for fragmentation [2] where memory pressure is high.

I'll give that a try. In the meantime new kernel came out. I'm just in a process of updating so I hope to test the default config on it too.

seth · 2022-10-20 22:16:58

However, if you(or anyone else) know of any alternative one can upload 350k text file and share it without cookies please do let me know.

1st link below.
quota is around 1MB or so, so not a problem.

7000k · 2022-10-20 23:46:14

Unfortunately there is no difference on kernel 6.0.2.

It turns out it is ZSWAP. Disabling it at runtime makes the process succeed rather than get stuck. However when transparent_hugepage is on it is close to 40% slower than on my custom (based on ubuntu) kernel config. Just disabling transparent_hugepage after ZSWAP was off by writing to /proc files helped a little, but what really helped was to disable both in a specific sequence:

echo 0 > /sys/module/zswap/parameters/enabled
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
swapoff --all
sleep 1
swapon --all

Following this the inference is actually a little faster on 6.0.2 with stock arch settings(up to 10% - the majority of the speedup comes from faster loading from disk it appears).

This raises a question of having those settings as on by default. I'm glad ubuntu had them off as the discrepancy between the results helped in discovering it is a config setting issue.

seth wrote:

However, if you(or anyone else) know of any alternative one can upload 350k text file and share it without cookies please do let me know.
1st link below.
quota is around 1MB or so, so not a problem.

The first link is pastebin right? If so, then it doesn't work for very long text lines. I tried before. There is a limit of some sort that stops one from pasting very long lines.

seth · 2022-10-21 07:57:48

The first link is pastebin right?

Instead of guessing you could just look, but oh, well:

cat file | curl -F 'file=@-' 0x0.st

Do you get better behavior w/ different compressor, max_pool or accept_threshold?
https://wiki.archlinux.org/title/Zswap# … parameters

7000k · 2022-10-21 23:42:53

seth wrote:

The first link is pastebin right?
Instead of guessing you could just look, but oh, well:

Seriously? Do you think I just guessed it without checking? Very funny :-) Also I did mention another limitation of pastebin in my very first email in this thread. Anyway.

cat file | curl -F 'file=@-' 0x0.st

Thanks. Bookmarked. Any idea how long files actually stay there? There is a "smart" equation there, which in absence of definition of its variables is as useful as the ascii drawing that seems to go to infinity below 1mb of file size :-) Still I plan to use it to test it myself.

Do you get better behavior w/ different compressor, max_pool or accept_threshold?
https://wiki.archlinux.org/title/Zswap# … parameters

No idea, at this point I disabled it altogether. TBH I see little value in this feature in the age of fast nvme storage and plenty of RAM. If I'm running a workload that consumes all of my RAM and more it usually constrains the CPU too. I'd rather it uses my 100k iops low latency nvme storage than attempt to save few extra % of RAM by taxing the cpu even more.

I was surprised this feature (zswap_default_on) is enabled in Arch as I never had it on by default on other distros I use so I did some research (looking for some discussion where it was decided to enable it, benchmark results etc. ) and all I found is this fairly recent thread on the Debian mailing list where they discuss a request to enable it in Debian: https://bugs.debian.org/cgi-bin/bugrepo … bug=970639 The thread is quite enlightenimg in general, but I failed to find any such discussion for Arch.

It appears the feature was considered experimental when it was added in kernel 3. Then around Linux kernel 5.7 the note about its experimental nature was removed despite there being very little published info about its cost/benefit in various configurations and for various workloads (often quoted page here https://linuxreviews.org/Zswap hilariously doesn't contain a comparison for the default compression type, doesn't mention storage type in the test system nor it tries few storage device types).

So for me the best is to disable it. In absence of comprehensive benchmarks and in light of the fact the feature has (documented) potential to severely impact performance in negative way I would prefer to not have it on by default in Arch. Now that I know it is what caused my problem in the first place I might log a bug at arch bug tracker. Perhaps this will make a larger conversion about it happen.

seth · 2022-10-22 06:54:41

Seriously? Do you think I just guessed it without checking?

Yes.

The linked wiki wrote:

Note: An acceptable pastebin service does not require enabling JavaScript for viewing, does not display adverts, manipulate the pasted content or require a login. pastebin.com is blocked for some people because of malware found on the site and has a history of annoying issues (requires JavaScript, displays adverts, inserts CRLF line-endings and displaying CAPTCHAs at random). Do not use it.

Anyway.

Any idea how long files actually stay there?

https://0x0.st/

I might log a bug at arch bug tracker. Perhaps this will make a larger conversion about it happen.

The needs of the many… but there should certainly be some notes in the wiki since you've found at least one access pattern where zswap is detrimental.

7000k · 2022-10-24 01:03:06

seth wrote:

Seriously? Do you think I just guessed it without checking?
Yes.

Hillarious

The linked wiki wrote:
Note: An acceptable pastebin service does not require enabling JavaScript for viewing, does not display adverts, manipulate the pasted content or require a login. pastebin.com is blocked for some people because of malware found on the site and has a history of annoying issues (requires JavaScript, displays adverts, inserts CRLF line-endings and displaying CAPTCHAs at random). Do not use it.

I'm not sure how this is relevant. Pastebin is a generic term (not just referring to pastebin.com). I've tried to post the same thing using paste.debian.net and I hit a line length,and a paste length limits a lot sooner there too.

Any idea how long files actually stay there?
https://0x0.st/

I'll quote from my previous post in case it was missed:
There is a "smart" equation there, which in absence of definition of its variables is as useful as the ascii drawing that seems to go to infinity below 1mb of file size :-)

And then to clarify further.
For the equation to have any sense values for min_age max_age as well as max size have to be known.

I might log a bug at arch bug tracker. Perhaps this will make a larger conversion about it happen.
The needs of the many… but there should certainly be some notes in the wiki since you've found at least one access pattern where zswap is detrimental.

I doubt very much zswap is beneficial "for the many" and is detrimental in some edge cases. In absence of evidence either way we simply don't know.

Also what I encountered it is not the only access pattern where zswap is detrimental. In the article people often point to as evidence of zswap benefits, in addition to 3 positive ones there are shown 3 scenarios when it is detrimental. This is the site.
https://linuxreviews.org/Zswap Also, Interestingly, the default compression mechanism wasn't even tested.

I think it is the other way around. Zswap can be beneficial in some edge cases (very slow storage, just a little bit of ram constrain), but in the majority of cases in fairly modern systems it is probably not. I'm open to be convinced otherwise if there is some evidence I don't know about.

cfr · 2022-10-24 03:51:00

7000k wrote:

seth wrote:
7000k wrote:
Any idea how long files actually stay there?
https://0x0.st/

I'll quote from my previous post in case it was missed:
There is a "smart" equation there, which in absence of definition of its variables is as useful as the ascii drawing that seems to go to infinity below 1mb of file size :-)
And then to clarify further.
For the equation to have any sense values for min_age max_age as well as max size have to be known.

https://0x0.st/ wrote:

File URLs are valid for at least 30 days and up to a year (see below).
Maximum file size: 512.0 MiB

sabroad · 2022-10-26 08:48:03

7000k wrote:

Zswap can be beneficial in some edge cases (very slow storage, just a little bit of ram constrain), but in the majority of cases in fairly modern systems it is probably not.

There are a *lot* of 24/7 server workloads out there with fast SSDs and heavy memory requirements. Given the tradeoff between burning CPUs (10 year lifespan) and burning SSDs (1 month lifespan at 500MB/s) many would prefer the former as the safer default.

cfr · 2022-12-19 14:53:04

sabroad wrote:

7000k wrote:
Zswap can be beneficial in some edge cases (very slow storage, just a little bit of ram constrain), but in the majority of cases in fairly modern systems it is probably not.
There are a *lot* of 24/7 server workloads out there with fast SSDs and heavy memory requirements. Given the tradeoff between burning CPUs (10 year lifespan) and burning SSDs (1 month lifespan at 500MB/s) many would prefer the former as the safer default.

Wouldn't that disfavour making zswap default? Or am I not following the logic here correctly?

Arch Linux

#1 2022-09-29 09:26:35

Weird issue with arch default kernel config and swap memory.

#2 2022-09-29 09:32:42

Re: Weird issue with arch default kernel config and swap memory.

#3 2022-09-29 11:18:49

Re: Weird issue with arch default kernel config and swap memory.

#4 2022-09-29 14:40:39

Re: Weird issue with arch default kernel config and swap memory.

#5 2022-09-30 16:13:08

Re: Weird issue with arch default kernel config and swap memory.

#6 2022-09-30 22:03:36

Re: Weird issue with arch default kernel config and swap memory.

#7 2022-10-01 07:54:48

Re: Weird issue with arch default kernel config and swap memory.

#8 2022-10-01 09:10:24

Re: Weird issue with arch default kernel config and swap memory.

#9 2022-10-01 23:27:46

Re: Weird issue with arch default kernel config and swap memory.

#10 2022-10-11 12:13:11

Re: Weird issue with arch default kernel config and swap memory.

#11 2022-10-11 12:24:21

Re: Weird issue with arch default kernel config and swap memory.

#12 2022-10-14 16:52:14

Re: Weird issue with arch default kernel config and swap memory.

#13 2022-10-20 22:00:11

Re: Weird issue with arch default kernel config and swap memory.

#14 2022-10-20 22:16:58

Re: Weird issue with arch default kernel config and swap memory.

#15 2022-10-20 23:46:14

Re: Weird issue with arch default kernel config and swap memory.

#16 2022-10-21 07:57:48

Re: Weird issue with arch default kernel config and swap memory.

#17 2022-10-21 23:42:53

Re: Weird issue with arch default kernel config and swap memory.

#18 2022-10-22 06:54:41

Re: Weird issue with arch default kernel config and swap memory.

#19 2022-10-24 01:03:06

Re: Weird issue with arch default kernel config and swap memory.

#20 2022-10-24 03:51:00

Re: Weird issue with arch default kernel config and swap memory.

#21 2022-10-26 08:48:03

Re: Weird issue with arch default kernel config and swap memory.

#22 2022-12-19 14:53:04

Re: Weird issue with arch default kernel config and swap memory.

Board footer