You are not logged in.

#1 2017-01-04 05:52:50

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Diagnosing a possible fork bomb (zsh and/or vim)

Today while I was doing some work my vim and tmux sessions stopped responding, followed quickly by being unable to switch tags in Awesome and a general halt to everything (even mouse cursor movement).

After a few reboots it seems this primarily happens when I'm in vim/tmux. I'm currently working on a kivy project, but starting that has not been a prequisite to the sort of system halt I've been getting.

The only error message I've been able to catch is 'fork failed: resource temporarily available'. Obviously by the time that happens it's too late for me to get additional diagnostics on what's actually going on. Googling resulted in quite a few zsh-related (oh-my-zsh specifically) hits from years back, but no conclusions as far as I could tell beyond disabling seemingly unrelated plugins.

This vim/tmux combo is my go-to and I'm using that environment probably 70% of the time on my machine daily, so this is really quite a new problem. My last system update was 1st January, and I've certainly done lots of coding since then, so it's unlikely to be package based.

Out of paranoia I'm running `ps aux | wc -l` every few minutes currently but as I'm using the web rather than coding it doesn't seem like the problem is recurring.

In summary:-
1. Possible fork bomb, seems related to the use of vim/tmux
2. How do I diagnose this and get to the bottom of it?

A mostly complete list of my vim plugins:- gundo, jedi-vim, neocomplete, pep8, todo.txt, beancount, fugitive, latex, obsession, pathogen, slime, textobj, tmux-navigator, unimpaired, virtualenv, YouCompleteMe

For tmux, I'm using continuum and resurrect (installed using tpm).

Additional information from my journalctl, from the various reboots I seem to be getting usb device descriptor errors (possibly power related), even though I do not have any USB devices plugged in to my laptop. Also, there are multiple occurences of the following error just before the reboot

INFO: rcu_preempt detected stalls on CPUs/tasks:
        Tasks blocked on level-0 rcu_node (CPUs 0-7): P0
        (detected by 4, t=18002 jiffies, g=2190, c=2189, q=141733)

However these seem related to the shutdown I'm trying to initiate rather than the cause of the hanging itself.

Last edited by ngoonee (2017-01-04 06:04:32)


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#2 2017-01-04 06:00:50

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Diagnosing a possible fork bomb (zsh and/or vim)

Is nothing printed to the journal when this happens?

I have no issues with zsh/vim/tmux; but I don't use any plugins.


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#3 2017-01-04 06:05:24

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: Diagnosing a possible fork bomb (zsh and/or vim)

jasonwryan wrote:

Is nothing printed to the journal when this happens?

I have no issues with zsh/vim/tmux; but I don't use any plugins.

Nothing that stands out to me. Edited in the only errors I could find, but it's not consistently appearing for each reboot so far.


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#4 2017-01-04 06:54:03

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: Diagnosing a possible fork bomb (zsh and/or vim)


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#5 2017-01-04 06:59:15

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: Diagnosing a possible fork bomb (zsh and/or vim)

Just read that. Timing seems off though, as I'm using [testing] and that poster has been having this since mid/early December. I AM using tlp (well, 'using' in the sense that I never actually found the time to properly tune it) and he's using zsh as well....


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

#6 2017-01-04 10:16:42

R00KIE
Forum Fellow
From: Between a computer and a chair
Registered: 2008-09-14
Posts: 4,734

Re: Diagnosing a possible fork bomb (zsh and/or vim)

This suggestion is not a debugging tip but it could help, have you thought of limiting the number of processes you can have open?

You could probably limit nproc and sigpending to something like 512 or 1024 and if you are dealing with a fork bomb it might help keep the machine running, and you'll probably be able to notice what stopped working or what complains or have a look at the logs. Mind you that I've tested this only with the simple bash fork bomb you can find everywhere as an example, it might not be easily feasible to stop more complex bork bombs.


R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

Offline

#7 2017-01-04 12:39:43

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,444
Website

Re: Diagnosing a possible fork bomb (zsh and/or vim)

I might also first suspect something more mundane than a fork bomb.  A combination of an infinite looping script and a broken pipe could lead to similar symptoms.  Tinkering with status bar like scripts is a common source of such issues (loops getting information from various sources and piping it to lemonbar, dwm, etc).

Watching `ps` output seems like a good diagnostic step.  My first check would be that when the problem starts, does the ps output show a huge number of processes being started, or just a few run-away processes.

Of course, also divide and conquer: remove/disable/move add-ons, plug-ins, and rc files to get as vanilla a setup as possible to narrow down whether one of the configs either causes or opens up the vulnerability to the problem.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#8 2017-01-05 05:35:26

ngoonee
Forum Fellow
From: Between Thailand and Singapore
Registered: 2009-03-17
Posts: 7,354

Re: Diagnosing a possible fork bomb (zsh and/or vim)

Haven't had time to return to coding (hence not seen any symptoms recently), but yes Trilby the more i think about it the more i think it may just be some rogue script. Certainly doubt anything malicious.

R00KIE as I understand it those limits are per user, and I've modified my ulimit to a high number because it was required for Matlab (I think). A per-process limit (based on cgroups maybe, since we're all systemd) would be ideal, but I don't think that's currently possible?


Allan-Volunteer on the (topic being discussed) mailn lists. You never get the people who matters attention on the forums.
jasonwryan-Installing Arch is a measure of your literacy. Maintaining Arch is a measure of your diligence. Contributing to Arch is a measure of your competence.
Griemak-Bleeding edge, not bleeding flat. Edge denotes falls will occur from time to time. Bring your own parachute.

Offline

Board footer

Powered by FluxBB