AS5402T memory leak

ducamps · 2024-03-31 09:59:58

Hi I use archlinux on my AS5402T and I have a memory leak memory use is always growing but I don't see any application who use it.
somebody have already meet this type of issue on this hardware?
or if someone know how to find which component is involve it will appreciate

Last edited by ducamps (2024-03-31 10:01:51)

ducamps · 2024-04-07 16:21:12

nobdy could give me some guidance to do a diagnostic?

loqs · 2024-04-07 16:32:29

ducamps wrote:

Hi I use archlinux on my AS5402T and I have a memory leak memory use is always growing but I don't see any application who use it.

How have you concluded the memory is being leaked rather than unused memory being used by the kernel for caching?

jl2 · 2024-04-07 18:06:33

This looks more like the opposite of a memory leak? I mean, your memory usage is constantly decreasing...

unused memory being used by the kernel for caching?

next time you have a memory leak:
* 'cat /proc/meminfo | curl -F 'file=@-' 0x0.st' post the output
* 'echo 3 | sudo tee /proc/sys/vm/drop_caches' and look if the memory usage drops.

ducamps · 2024-04-07 20:24:08

@loqs just compare with top command

@jl2
Output: http://0x0.st/XifD.txt
Already try drop cache without effect

seth · 2024-04-07 20:49:04

There're roughly 2GB unaccounted for

Google says "AS5402T" is a case,

lspci

Likely https://bbs.archlinux.org/viewtopic.php?id=285160 ?

ducamps · 2024-04-08 06:33:52

indeed this iis a case I have installed on Nimbustor 2 Gen2

lspci
00:00.0 Host bridge: Intel Corporation Device 4e24
00:02.0 VGA compatible controller: Intel Corporation JasperLake [UHD Graphics] (rev 01)
00:04.0 Signal processing controller: Intel Corporation Dynamic Tuning service
00:05.0 Multimedia controller: Intel Corporation JasperLake IPU
00:08.0 System peripheral: Intel Corporation Device 4e11
00:14.0 USB controller: Intel Corporation Jasper Lake USB 3.1 xHCI Host Controller (rev 01)
00:14.2 RAM memory: Intel Corporation Jasper Lake Shared SRAM (rev 01)
00:15.0 Serial bus controller: Intel Corporation Jasper Lake Serial IO I2C Host Controller #0 (rev 01)
00:15.2 Serial bus controller: Intel Corporation Jasper Lake Serial IO I2C Host Controller #2 (rev 01)
00:16.0 Communication controller: Intel Corporation Management Engine Interface (rev 01)
00:17.0 SATA controller: Intel Corporation Jasper Lake SATA AHCI Controller (rev 01)
00:19.0 Serial bus controller: Intel Corporation Jasper Lake Serial IO I2C Host Controller #4 (rev 01)
00:19.1 Serial bus controller: Intel Corporation Jasper Lake Serial IO I2C Host Controller #5 (rev 01)
00:1c.0 PCI bridge: Intel Corporation Jasper Lake PCIe Root Port #1 (rev 01)
00:1c.2 PCI bridge: Intel Corporation Device 4dba (rev 01)
00:1c.3 PCI bridge: Intel Corporation Device 4dbb (rev 01)
00:1c.5 PCI bridge: Intel Corporation Device 4dbd (rev 01)
00:1e.0 Communication controller: Intel Corporation Jasper Lake Serial IO UART Controller #0 (rev 01)
00:1e.3 Serial bus controller: Intel Corporation Jasper Lake Serial IO SPI Controller #1 (rev 01)
00:1f.0 ISA bridge: Intel Corporation Jasper Lake eSPI Controller (rev 01)
00:1f.3 Audio device: Intel Corporation Jasper Lake HD Audio (rev 01)
00:1f.4 SMBus: Intel Corporation Jasper Lake SMBus (rev 01)
00:1f.5 Serial bus controller: Intel Corporation Jasper Lake SPI Controller (rev 01)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal]

Last edited by ducamps (2024-04-08 06:56:12)

seth · 2024-04-08 06:54:57

Please use [code][/code] tags, not "quote" tags. Edit your post in this regard.

SO not AMD but an intel JasperLake
Does the unattributed RAM return if you
1. isolate the rescue.target
2. from there allocate a lot of RAM

head -c 5G    /dev/zero | tail

eg. takes 5GB, disable the swap, and grab more RAM than is available but less than what is available + what you're missing.
/proc/meminfo is authorative itr. You can just ballpark a number inbetween - you only want to know whether this gets you the difference returned.

Please post your complete system journal for the boot:

sudo journalctl -b | curl -F 'file=@-' 0x0.st

(amd logs GTT and GART but this would be the first intel case I've seen, so idk what to look for specifically)

ducamps · 2024-04-08 07:50:31

Just perform isolante rescue mode and

head -c 2G    /dev/zero | tail

After that I free 400mo memory so not every missing memory

Link to boot logs:
Http://0x0.st/XiJw.txt

Last edited by ducamps (2024-04-08 07:51:48)

seth · 2024-04-08 08:29:21

After that I free 400mo memory so not every missing memory

seth wrote:

disable the swap, and grab more RAM than is available but less than what is available + what you're missing

If you had 1.6GB available, you would have allocated that and then reclaimed 400MB from GTT.
Try to allocate more RAM, but make sure you're not just getting it from dropping the GUI.
In doubt post the meminfo before and after the reclamation attempts.

ducamps · 2024-04-08 09:18:51

I have reboot my NAS this morning so ram is come back to a normal level (1.16G)
I will perform again the test in some day when I will get again 2,5 or 3 Giga
thanks for your help

ducamps · 2024-04-12 08:29:49

After that I free 400mo memory so not every missing memory

sorry I think this 400mo was free when switching in isolate mode

I have try again the manipulation this morning but sadly I have freeze my system with the head command and need to reboot (head -c 3G /dev/zero|tail)

I have capture meminfo file before and after reboot if this could help

before :

MemTotal:        3765600 kB
MemFree:          441588 kB
MemAvailable:    1558876 kB
Buffers:              40 kB
Cached:          1264344 kB
SwapCached:            0 kB
Active:           941120 kB
Inactive:         744716 kB
Active(anon):     262984 kB
Inactive(anon):   159944 kB
Active(file):     678136 kB
Inactive(file):   584772 kB
Unevictable:          88 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        421500 kB
Mapped:            66692 kB
Shmem:              1460 kB
KReclaimable:     125220 kB
Slab:             223556 kB
SReclaimable:     125220 kB
SUnreclaim:        98336 kB
KernelStack:        5504 kB
PageTables:         7280 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1882800 kB
Committed_AS:    2195096 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       39016 kB
VmallocChunk:          0 kB
Percpu:             2576 kB
HardwareCorrupted:     0 kB
AnonHugePages:    139264 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:    129024 kB
FilePmdMapped:      2048 kB
CmaTotal:              0 kB
CmaFree:               0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      378968 kB
DirectMap2M:     3567616 kB

after reboot

MemTotal:        3765604 kB
MemFree:          568144 kB
MemAvailable:    2401228 kB
Buffers:            5996 kB
Cached:          2024792 kB
SwapCached:            0 kB
Active:          1739932 kB
Inactive:        1176820 kB
Active(anon):     869772 kB
Inactive(anon):    18232 kB
Active(file):     870160 kB
Inactive(file):  1158588 kB
Unevictable:          88 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:        4194300 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:              2424 kB
Writeback:             0 kB
AnonPages:        854028 kB
Mapped:           592956 kB
Shmem:              2040 kB
KReclaimable:      61996 kB
Slab:             143308 kB
SReclaimable:      61996 kB
SUnreclaim:        81312 kB
KernelStack:        9552 kB
PageTables:        15048 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6077100 kB
Committed_AS:    4394268 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       41872 kB
VmallocChunk:          0 kB
Percpu:             2560 kB
HardwareCorrupted:     0 kB
AnonHugePages:    368640 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:    172032 kB
FilePmdMapped:     10240 kB
CmaTotal:              0 kB
CmaFree:               0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      159832 kB
DirectMap2M:     3786752 kB

seth · 2024-04-12 08:54:00

The system has <4GB usable RAM
You had ~1.4GB unaccounted for and ~1.5GB available (ie. free + droppable file caches) so allocating 3GB (esp. w/o swap) got you OOM.
This is all very narrow, next time look at MemAvailable, that's the minimum you've to allocate (as it'll mostly come from there first)

The gap you can seek to reclaim is "MemTotal - (MemFree + Active + Inactive + Slab)"

awk '/MemTotal:/{v=$2}/MemFree:|Active:|Inactive:|Slab:/{v-=$2} END {printf v/1024 " MB\n"}' /proc/meminfo

will get you that.
The amount you try to reclaim needs to be <= MemAvailable + The unaccounted RAM, ie. 2.0-2.5GB would still have been fine in the previous case

ducamps · 2024-04-17 11:15:56

Hi

meminfo before

MemTotal:        3765604 kB
MemFree:          219124 kB
MemAvailable:     562800 kB
Buffers:              40 kB
Cached:           533516 kB
SwapCached:            0 kB
Active:           580692 kB
Inactive:         331532 kB
Active(anon):     196848 kB
Inactive(anon):   183332 kB
Active(file):     383844 kB
Inactive(file):   148200 kB
Unevictable:          88 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        378756 kB
Mapped:            47412 kB
Shmem:              1512 kB
KReclaimable:      39724 kB
Slab:             121504 kB
SReclaimable:      39724 kB
SUnreclaim:        81780 kB
KernelStack:        5728 kB
PageTables:         7132 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1882800 kB
Committed_AS:    2232876 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       39236 kB
VmallocChunk:          0 kB
Percpu:             2608 kB
HardwareCorrupted:     0 kB
AnonHugePages:     75776 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:     71680 kB
FilePmdMapped:      4096 kB
CmaTotal:              0 kB
CmaFree:               0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      225368 kB
DirectMap2M:     3721216 kB

command

awk '/MemTotal:/{v=$2}/MemFree:|Active:|Inactive:|Slab:/{v-=$2} END {printf v/1024 " MB\n"}' /proc/meminfo

get 2500MB

add I have try to reclaim 1GB

and Get again an OOM today

meminfo after reboot

MemTotal:        3765600 kB
MemFree:         1607616 kB
MemAvailable:    2728140 kB
Buffers:            5996 kB
Cached:          1310116 kB
SwapCached:            0 kB
Active:          1575180 kB
Inactive:         366308 kB
Active(anon):     627048 kB
Inactive(anon):        0 kB
Active(file):     948132 kB
Inactive(file):   366308 kB
Unevictable:          88 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:        4194300 kB
Zswap:                 0 kB
Zswapped:              0 kB
Dirty:              1012 kB
Writeback:             0 kB
AnonPages:        605768 kB
Mapped:           569688 kB
Shmem:              1672 kB
KReclaimable:      28996 kB
Slab:              99676 kB
SReclaimable:      28996 kB
SUnreclaim:        70680 kB
KernelStack:        7280 kB
PageTables:        11616 kB
SecPageTables:         0 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     6077100 kB
Committed_AS:    2863716 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       39456 kB
VmallocChunk:          0 kB
Percpu:             2512 kB
HardwareCorrupted:     0 kB
AnonHugePages:    278528 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:     47104 kB
FilePmdMapped:      4096 kB
CmaTotal:              0 kB
CmaFree:               0 kB
Unaccepted:            0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      165976 kB
DirectMap2M:     3780608 kB

seth · 2024-04-17 14:53:39

Have you tried to isolate the rescue.target before the reclaim?
Try to add

transparent_hugepage=never

to the https://wiki.archlinux.org/title/Kernel_parameters

ducamps · 2024-04-17 15:28:15

indeed I have switched in rescue mode with

systemctl isolate rescue.target

I will try to add this to my kernel parameters

ducamps · 2024-04-18 17:57:40

memory continue to grow same with "transparent_hugepage=never" in kernel parameter

seth · 2024-04-18 20:29:42

W/o the ability to reclaim the RAM the kernel will actually leak that RAM, but apparently not through the THP
Does this also happen w/ the LTS kernel?
Does it happen w/o nomad?

ducamps · 2024-04-22 19:53:13

Hi

since update to kernel 6.8.7 memory leak seem stopped.
I confirm in some days

ducamps · 2024-04-27 07:54:00

I confirm ram is stable now.
same if we didn't found root cause, thanks for your help

Arch Linux

#1 2024-03-31 09:59:58

AS5402T memory leak

#2 2024-04-07 16:21:12

Re: AS5402T memory leak

#3 2024-04-07 16:32:29

Re: AS5402T memory leak

#4 2024-04-07 18:06:33

Re: AS5402T memory leak

#5 2024-04-07 20:24:08

Re: AS5402T memory leak

#6 2024-04-07 20:49:04

Re: AS5402T memory leak

#7 2024-04-08 06:33:52

Re: AS5402T memory leak

#8 2024-04-08 06:54:57

Re: AS5402T memory leak

#9 2024-04-08 07:50:31

Re: AS5402T memory leak

#10 2024-04-08 08:29:21

Re: AS5402T memory leak

#11 2024-04-08 09:18:51

Re: AS5402T memory leak

#12 2024-04-12 08:29:49

Re: AS5402T memory leak

#13 2024-04-12 08:54:00

Re: AS5402T memory leak

#14 2024-04-17 11:15:56

Re: AS5402T memory leak

#15 2024-04-17 14:53:39

Re: AS5402T memory leak

#16 2024-04-17 15:28:15

Re: AS5402T memory leak

#17 2024-04-18 17:57:40

Re: AS5402T memory leak

#18 2024-04-18 20:29:42

Re: AS5402T memory leak

#19 2024-04-22 19:53:13

Re: AS5402T memory leak

#20 2024-04-27 07:54:00

Re: AS5402T memory leak

Board footer