You are not logged in.

#1 2013-12-18 07:15:03

rezahousseini
Member
Registered: 2013-09-06
Posts: 17

What to do with buffer overflow?

I experienced this twice already and I'm not sure this can be called a memory leak.

The first one was with Octave when I solved a big set of huge ODE's. The memory got slowly filled up until it reached its limit and everything got really slow.

The second one was a C++ program writing a lot of stuff to std output. It ran a while and the memory was filling up. htop said that tmux was consuming the biggest part of my buffer. I had to restart the system to free the memory, closing tmux didn't help.

Is this normal behaviour or is there a missing mechanism to prevent such things?

Cheers
Reza

Offline

#2 2013-12-18 08:26:47

brebs
Member
Registered: 2007-04-03
Posts: 3,742

Re: What to do with buffer overflow?

You mean, "out of memory".

Offline

#3 2013-12-18 08:43:53

rezahousseini
Member
Registered: 2013-09-06
Posts: 17

Re: What to do with buffer overflow?

yes, sorry of course an out of memory problem

Offline

#4 2013-12-18 08:49:54

WorMzy
Administrator
From: Scotland
Registered: 2010-06-16
Posts: 13,723
Website

Re: What to do with buffer overflow?

Valgrind can help you find memory bugs in your code. It's up to you to fix them though. If you think you've found a problem in somebody else's program, file a bug report and help to squash it.


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#5 2013-12-18 09:37:41

jakobcreutzfeldt
Member
Registered: 2011-05-12
Posts: 1,042

Re: What to do with buffer overflow?

You need to start thinking like a computer user from the 1970s. Sure, for every day tasks, RAM can be thought of as being overabundant. But if you are doing any sort of complex computations, you need to immediately consider your RAM usage. They were solving complex math with computers in the 70s but they had to be extremely aware of their resource usage and had to effectively divide and conquer to solve the problems.

Is there a way you can optimize solving your ODEs? Rather than just handing them off to Octave, you may have to manually do a lot of the solving (calculating the Jacobians, etc), storing things to disk and reading them as necessary, etc. Don't assume that you can just store everything in memory. Treat RAM as a convenient place to rapidly do a few computations while the rest of your data sits in relatively cheaper disk space.

As for your tmux problem, there's almost certainly a setting for it in which you can limit the scrollback buffer size. Alternatively, redirect the output to a file, assuming you have sufficient disk space.

Offline

#6 2013-12-18 10:04:24

rezahousseini
Member
Registered: 2013-09-06
Posts: 17

Re: What to do with buffer overflow?

@jakobcreutzfeldt
Thanks for your tips! But I was wondering about a system wide approach. Musn't there be something take control over the processes before the system gets untradeable?

Offline

#7 2013-12-18 10:23:18

WorMzy
Administrator
From: Scotland
Registered: 2010-06-16
Posts: 13,723
Website

Re: What to do with buffer overflow?

That would be the OOM killer.


Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#8 2013-12-18 11:31:05

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,546
Website

Re: What to do with buffer overflow?

And the OOM killer is much more proactive if you disable swap.  This is one of the main reasons I don't generally use swap.  I can always activate it if I know I'll need it, but under normal conditions I know I never get anywhere near using all my ram - so if the ram is full something has gone very wrong.  Rather that grinding to a slow crawl for tens of minutes or more as swap also fills up and gets swapped in and out of ram repeatedly, the OOM killer will go into action as soon as the ram is full.

However, disabling swap for this purpose assumes that something has actually gone wrong to use all the ram.  If you are doing calculations that really will need all the ram, then the only approach is to simplify the process as jakobcreutzfeldt suggests.


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Online

#9 2013-12-18 15:31:47

rezahousseini
Member
Registered: 2013-09-06
Posts: 17

Re: What to do with buffer overflow?

How then my OOM killer didn't react and killed the memory consuming processes? Do I have to enable it somewhere?

Offline

#10 2013-12-18 16:45:31

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,546
Website

Re: What to do with buffer overflow?

You haven't given us any information with which to answer that question.  Is your memory actually *full*?  Do you have swap enabled?  If so the answer is in my last post.


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Online

#11 2013-12-19 07:13:28

rezahousseini
Member
Registered: 2013-09-06
Posts: 17

Re: What to do with buffer overflow?

When I use

free -m

I get

             total       used       free     shared    buffers     cached
Mem:          7869       2917       4951        149        132        729
-/+ buffers/cache:       2055       5813
Swap:            0          0          0

So I'm not using swap at all but the OOM killer didn't react proactive at all (I had to force shutdown by pressing the power off button).

Last edited by rezahousseini (2013-12-19 13:07:58)

Offline

#12 2013-12-19 07:16:29

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,426
Website

Re: What to do with buffer overflow?

Please use code tags when pasting to the boards; it makes the output readable
https://wiki.archlinux.org/index.php/Fo … s_and_Code


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#13 2013-12-19 07:17:22

HalosGhost
Forum Fellow
From: Twin Cities, MN
Registered: 2012-06-22
Posts: 2,097
Website

Re: What to do with buffer overflow?

rezahousseini wrote:

When I use free -m I get

<snip>

So I'm not using swap at all but the OOM killer didn't react proactive at all (I had to force shutdown by pressing the power off button).

Please, please use code tags for any terminal output or code snippets. Trying to read something that was originally formatted for a monospaced environment in anything but verbatim is awful.

Edit: beaten by jwr (it wasn't the first time; nor—I believe—shall it be the last)

All the best,

-HG

Last edited by HalosGhost (2013-12-19 07:17:56)

Offline

#14 2013-12-19 13:48:05

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 30,546
Website

Re: What to do with buffer overflow?

Your not even remotely close to being out of memory - so the out-of-memory killer will of course not do anything.

I think you need to go back to the start and provide us with information about what is actually happening, not about your assumptions.  It seems the only symptom you've described is that the system slows or is unresponsive, is this correct?  You've had various contributors in this thread now chasing their tails for 13 posts because you started out with the claim that you were out of memory (and a title that is even more inaccurate).

Please describe the actual syptoms, and provide real diagnostic information.  For a start (but only a start) run htop when you think this will happen, and see if anything stands out.


"UNIX is simple and coherent" - Dennis Ritchie; "GNU's Not Unix" - Richard Stallman

Online

Board footer

Powered by FluxBB