So what's the best course of action regarding how to handle the below issues? kernel, distros??
I work on a multithreaded photogrammetry geometry and image processor and image mosaicking software. The applications are set up as desktop software, 64 bit only. Everything is c++ with some c libraries.
The current problem solution is on boot for the system to execute the following:
echo "never" > /sys/kernel/mm/transparent_hugepage/defrag
Application: heavy weight processing threads. Typically one or two threads per cpu. Systems are typically (but not limited to) core i7 or opterons with 16GB ram or higher.
The symptoms: When application starts and the threads spin up, the system may become unresponsive. All cpus become busy inside kernel space. "perf top" shows all cpus busy inside a spinlock. Application debugger shows all cpus busy waiting on heap access. The system stays busy for possibly 10s of minutes and may or may not "break loose" In particular this has been observed during multithreaded loading of multiple different jpeg, png or jpeg compressed tifs.
My unprofessional diagnosis is that the in kernel memory defragging is competing with the application thread heap management, the spinlock is causing the high cpu usage.
I've noticed this on and off the past 6 months but really looked into it after a customer ran into this problem testing migration from windows over to linux with our software.
NOTE: this is a desktop application, although a notice may be given to end users to set this up properly, demo versions will soon be available for desktop evaluation.
I diagnosed this on arch linux but a customer reported this problem on ubuntu, redhat, suse and centos.