You are not logged in.

#1 2016-11-20 08:20:53

y0g1
Member
Registered: 2008-09-25
Posts: 13

[SOLVED][C] Help needed with pthreads and segfault

Hello,

I need help with an example app that use pthreads and segfaults from time to time.
Can you help me to find the reason why it segfaults?

the source is:

#include <unistd.h>
#include <stdio.h>
#include <pthread.h>

pthread_mutex_t lock;

#define QSIZE 65108

size_t active = QSIZE;

pthread_t ret[QSIZE];

void *threadapp(void *o)
{
        pthread_mutex_lock(&lock);
        active--;
        pthread_mutex_unlock(&lock);
        pthread_exit(NULL);
}

int threadsrun(void)
{
        size_t i;
        for(i=0; i<QSIZE; ++i)  {
                int status;
                do      {
                        status = pthread_create(&ret[i], NULL, threadapp, NULL);
                        if(status != 0) {
                                perror("create");
                        }
                }
                while(status != 0);
                pthread_detach(ret[i]);
        }
        return 0;
}

int main(int argc, char *argv[])
{
        pthread_mutex_init(&lock, NULL);
        threadsrun();
        for(;;) {
                size_t act;
                pthread_mutex_lock(&lock);
                act = active;
                pthread_mutex_unlock(&lock);
                if(act == 0)    {
                        break;
                }
                usleep(100);
        }
        printf("Done\n");
        return 0;
}

I compile it with:
gcc multitest.c -o multitest -lpthread
I changed linking option to -pthread, but it didn't help.
I use shell loop for testing:
while ./multitest ; do sleep 0; done
The loop ends with first segfault.

Last edited by y0g1 (2016-11-25 08:42:21)

Offline

#2 2016-11-21 09:09:12

Michail
Member
From: Moscow, Russia
Registered: 2016-06-23
Posts: 34
Website

Re: [SOLVED][C] Help needed with pthreads and segfault

I'm not the most experienced person when it comes to multithreading, so take my words with a grain of salt. I think that the problem is your inner loop here:

y0g1 wrote:
        for(i=0; i<QSIZE; ++i)  {
                int status;
                do      {
                        status = pthread_create(&ret[i], NULL, threadapp, NULL);
                        if(status != 0) {
                                perror("create");
                        }
                }
                while(status != 0);
                pthread_detach(ret[i]);
        }

There are two things wrong with it: firstly, it creates the threads using the same variable, so you can manipulate only the last created thread and lose ability to manipulate previously created ones. Second, you detach the thread regardless of status. Imagine that your inner loop fails on the first iteration. Then

ret[i]

is filled with garbage, and you try to detach that. That's what can cause segfaults, I think.


> Don't get impatient, you cannot solve problems by coding, you can only solve them by thinking.

Offline

#3 2016-11-21 10:44:36

y0g1
Member
Registered: 2008-09-25
Posts: 13

Re: [SOLVED][C] Help needed with pthreads and segfault

Michail wrote:

I'm not the most experienced person when it comes to multithreading, so take my words with a grain of salt. I think that the problem is your inner loop here:

y0g1 wrote:
        for(i=0; i<QSIZE; ++i)  {
                int status;
                do      {
                        status = pthread_create(&ret[i], NULL, threadapp, NULL);
                        if(status != 0) {
                                perror("create");
                        }
                }
                while(status != 0);
                pthread_detach(ret[i]);
        }

There are two things wrong with it: firstly, it creates the threads using the same variable, so you can manipulate only the last created thread and lose ability to manipulate previously created ones. Second, you detach the thread regardless of status. Imagine that your inner loop fails on the first iteration. Then

ret[i]

is filled with garbage, and you try to detach that. That's what can cause segfaults, I think.

thanks for answer,
as you can see - I check the status of pthread_create(). When it fails - I try create thread again. I do it in loop until it success. (while status != 0)
when the thread is created I detach it to release resources with pthread_exit() without pthread_join().
It mostly work ok, but when I change system resources during test (I turn firefox on and some other aps) then it fails quick.
When I let the test program running in continues loop alone, then it takes more time to report segfault.
Maybe there are some resources problem, but then pthread_create shoud report error EAGAIN (and mostly it does).
I tested this program on archlinux with glibc and alpinelinux with musl-libc.
In both cases test segfaults (libpthread in Archlinux and ld-musl in Alpinelinux).

Offline

#4 2016-11-21 10:54:04

WorMzy
Forum Moderator
From: Scotland
Registered: 2010-06-16
Posts: 11,783
Website

Re: [SOLVED][C] Help needed with pthreads and segfault


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#5 2016-11-21 13:50:13

y0g1
Member
Registered: 2008-09-25
Posts: 13

Re: [SOLVED][C] Help needed with pthreads and segfault

When I test coredump with gdb I get:

[New LWP 28277]
[New LWP 28163]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `./multitest'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f8dad98d5a6 in pthread_detach () from /usr/lib/libpthread.so.0
[Current thread is 1 (Thread 0x7f8daddc0700 (LWP 27890))]
(gdb) 
(gdb) bt
#0  0x00007f8dad98d5a6 in pthread_detach () from /usr/lib/libpthread.so.0
#1  0x00000000004008b4 in threadsrun () at multitest.c:29
#2  0x0000000000400916 in main (argc=1, argv=0x7ffd54ad1d68) at multitest.c:41

Then I modified the program to use pthread_attr_setdetachstate() with PTHREAD_CREATE_DETACHED

int threadsrun(void)
{
        size_t i;
        for(i=0; i<QSIZE; ++i)  {
                int status;
                do      {
                        pthread_attr_t attr;
                        if(pthread_attr_init(&attr) != 0)       {
                                perror("attr init");
                                return 1;
                        }
                        pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
                        status = pthread_create(&ret[i], &attr, threadapp, NULL);
                        if(pthread_attr_destroy(&attr) != 0)    {
                                perror("attr destroy");
                                return 1;
                        }
                }
                while(status != 0);
        }
        return 0;
}

after that modification I got coredump more frequent. GDB log:

[New LWP 24363]
[New LWP 24256]
[New LWP 24254]
Cannot access memory at address 0x7fdc2054b0e8
Cannot access memory at address 0x7fdc2054b0e0
Failed to read a valid object file image from memory.
Core was generated by `./multitest'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fdc20111c1e in ?? ()
[Current thread is 1 (LWP 30172)]
(gdb) bt
#0  0x00007fdc20111c1e in ?? ()
Backtrace stopped: Cannot access memory at address 0x7fff2c6b3170
(gdb) 

Offline

#6 2016-11-21 16:03:05

Michail
Member
From: Moscow, Russia
Registered: 2016-06-23
Posts: 34
Website

Re: [SOLVED][C] Help needed with pthreads and segfault

Sorry for the earlier reply, apparently my brain doesn't work very well in the morning. I've tried running the program with different QSIZEs, it seems that lower ones result in less frequent segfaults, but higher ones don't really make them noticeably more frequent. Out of curiosity, though: why 65108?


> Don't get impatient, you cannot solve problems by coding, you can only solve them by thinking.

Offline

#7 2016-11-21 20:11:36

y0g1
Member
Registered: 2008-09-25
Posts: 13

Re: [SOLVED][C] Help needed with pthreads and segfault

Michail wrote:

Sorry for the earlier reply, apparently my brain doesn't work very well in the morning. I've tried running the program with different QSIZEs, it seems that lower ones result in less frequent segfaults, but higher ones don't really make them noticeably more frequent. Out of curiosity, though: why 65108?

I get this error in program with 255108 threads. Then I begin to test with this example and try to change the QSIZE. As you note - the QSIZE is not so important.
Sometimes program segfaults, sometimes not. Why? What is wrong with it.
When I try to run it with gdb - theres is no segfaults.
When I replace pthread_detach() with pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED) - the program segfaults each time I run it
(outside gdb).

Offline

#8 2016-11-22 12:10:45

y0g1
Member
Registered: 2008-09-25
Posts: 13

Re: [SOLVED][C] Help needed with pthreads and segfault

I modified test case with additional mutex and lock(). It does nothing but program runs without segfault.
current version is:

#include <unistd.h>
#include <stdio.h>
#include <pthread.h>

#define QSIZE 200000
pthread_mutex_t lock;
pthread_mutex_t pclock;
int active = QSIZE;

void *threadapp(void *o)
{
        pthread_mutex_lock(&pclock);
        pthread_mutex_unlock(&pclock);
        pthread_mutex_lock(&lock);
        active--;
        pthread_mutex_unlock(&lock);
        pthread_exit(NULL);
}

pthread_attr_t attr;

int threadsrun(void)
{
        int i;
        static int max = 0;
        for(i=0; i<QSIZE; ++i)  {
                int status;
                do      {
                        pthread_t ret;
                        pthread_mutex_lock(&pclock);
                        status = pthread_create(&ret, &attr, threadapp, NULL);
                        pthread_mutex_unlock(&pclock);
                }
                while(status != 0);
        }
        return 0;
}

int main(int argc, char *argv[])
{
        pthread_mutex_init(&pclock, NULL);
        pthread_mutex_init(&lock, NULL);
        if(pthread_attr_init(&attr) != 0)       {
                perror("attr init");
                return 1;
        }
        pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
        printf("Start");
        fflush(stdout);
        if(threadsrun() == 1)   {
                return 1;
        }
        for(;;) {
                pthread_mutex_lock(&lock);
                if(active == 0) {
                        pthread_mutex_unlock(&lock);
                        break;
                }
                pthread_mutex_unlock(&lock);
        }
        if(pthread_attr_destroy(&attr) != 0)    {
                perror("attr destroy");
                return 1;
        }
        return 0;
}

Mutex pclock at the beginning of the thread and during ptrhead_create / pthread_detach is needed to not segfault.
Is it kernel bug, or my program bug?
The problem exists when the thread lives for a short period of time.
Help.

Offline

#9 2016-11-24 19:02:29

y0g1
Member
Registered: 2008-09-25
Posts: 13

Re: [SOLVED][C] Help needed with pthreads and segfault

I found on another forrum that it may be glibc bug:
https://sourceware.org/bugzilla/show_bug.cgi?id=20116

Offline

#10 2016-12-11 19:10:07

BenderRodriguez
Member
Registered: 2010-07-05
Posts: 16
Website

Re: [SOLVED][C] Help needed with pthreads and segfault

A bit late to the party but I think the main problem is memory. Each created thread reserves some memory to hold it's stack. On 32 Bit OS the default should be 2MB whereas on 64 Bits it should be 8MB (See ulimit -s).
At each point in time this program runs there are more threads created (from main) while other threads exit. Depending on the actual scheduling you might see different behavior.
All in all, this program requires up to 508 GB of RAM :-)

Last edited by BenderRodriguez (2016-12-11 19:10:39)

Offline

#11 2017-02-15 07:15:07

y0g1
Member
Registered: 2008-09-25
Posts: 13

Re: [SOLVED][C] Help needed with pthreads and segfault

BenderRodriguez wrote:

A bit late to the party but I think the main problem is memory. Each created thread reserves some memory to hold it's stack. On 32 Bit OS the default should be 2MB whereas on 64 Bits it should be 8MB (See ulimit -s).
At each point in time this program runs there are more threads created (from main) while other threads exit. Depending on the actual scheduling you might see different behavior.
All in all, this program requires up to 508 GB of RAM :-)

I checked the same program with glibc patch and there is no more segfaults.
The same problem I got with akonadi - it mostly segfaults without glibc patch:
https://sourceware.org/bugzilla/show_bug.cgi?id=20116

Offline

Board footer

Powered by FluxBB