[SOLVED] Salt server unable to start either salt-master or salt-minion

ephreal · 2018-09-14 17:39:29

Hey All,

I wanted to start playing around with salt, as centralized management of my servers sounds REALLY nice. Unfortunately, on most of my servers, when trying to start salt-master or salt-minion, I get the following

Sep 14 10:12:53 archlinux systemd[1]: Starting The Salt Minion...
Sep 14 10:12:53 archlinux systemd[1]: Started The Salt Minion.
Sep 14 10:12:53 archlinux salt-minion[14862]: Usage: salt-minion [options]
Sep 14 10:12:53 archlinux salt-minion[14862]: salt-minion: error: Failed to load configuration: Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)
Sep 14 10:12:53 archlinux systemd[1]: salt-minion.service: Main process exited, code=exited, status=64/USAGE
Sep 14 10:12:53 archlinux systemd[1]: salt-minion.service: Failed with result 'exit-code'.

After some testing, it turns out that the machines this happens on are the ones that have updated today, specifically, within the the last few hours (2018-09-14 16:00-17:30 UTC). I had a friend who had updated yesterday afternoon try install salt and run the minion/master services, and both worked normally. She tried updating arch, then stopping and starting the services, and immediately received the same error message.

I've just started diagnosing this and haven't scoured google to the fullest extent yet, but I figured I'd toss this here in case someone is having the same issue. If anyone has any steps to try isolate the problem, please let me know. It's very possible that I'll need to roll back an updated package with a cached package, but I'd like to first try diagnose this.

The steps I've taken so far follow.

Fully update the archlinux system
Restart said system
Attempt to start services
Test services on non-fully updated machine

# salt-master -l all
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: archlinux
Usage: salt-master [options]

salt-master: error: Failed to load configuration: Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)

$ sudo journalctl -xe

--SNIP--

Sep 14 10:35:19 archlinux sudo[15494]: pam_unix(sudo:session): session opened for user root by (uid=0)
Sep 14 10:35:19 archlinux systemd[1]: Starting The Salt Master Server...
-- Subject: Unit salt-master.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit salt-master.service has begun starting up.
Sep 14 10:35:19 archlinux salt-master[15498]: Usage: salt-master [options]
Sep 14 10:35:19 archlinux salt-master[15498]: salt-master: error: Failed to load configuration: Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)
Sep 14 10:35:19 archlinux systemd[1]: salt-master.service: Main process exited, code=exited, status=64/USAGE
Sep 14 10:35:19 archlinux systemd[1]: salt-master.service: Failed with result 'exit-code'.
Sep 14 10:35:19 archlinux systemd[1]: Failed to start The Salt Master Server.
-- Subject: Unit salt-master.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit salt-master.service has failed.

--SNIP--

Last edited by ephreal (2018-09-14 18:56:26)

ephreal · 2018-09-14 18:54:52

Time for an update.

I have moved on from trying to diagnose what the problem is and instead figure out what packages are causing the problem, and I believe I've found the package causing issues.

I grabbed the upgraded package list from my friend's computer, and started downgrading packages that could cause the problem. I am not sure what the root cause is, but openssl-1.1.1-1 is the problem child.

$ sudo pacman -Qi openssl

Name            : openssl
Version         : 1.1.1-1
Description     : The Open Source toolkit for Secure Sockets Layer and Transport Layer Security
Architecture    : x86_64

--SNIP--

$ sudo pacman -U /var/cache/pacman/pkg/openssl-1.1.0.i-1-x86_64.pkg.tar.xz
$ sudo systemctl start salt-master
$ systemctl status salt-master

● salt-master.service - The Salt Master Server
   Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-09-14 11:47:56 MST; 3min 10s ago
     Docs: man:salt-master(1)
           file:///usr/share/doc/salt/html/contents.html
           https://docs.saltstack.com/en/latest/contents.html

--SNIP--

I'll mark this as solved for now. As a general question, should I file this as a bug? If so, where would I file a bug report?

Last edited by ephreal (2018-09-14 20:53:43)

CoolGenie · 2018-09-14 19:19:41

+1, I got a similar error when running salt stack:

salt-ssh -l debug -c _all 'reddconnect-transip-odoo-1' test.ping
...
    raise OSError("Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)")
OSError: Failed to initialize OpenSSL library (OPENSSL_init_crypto failed)

I also confirm that downgrading to 1.1.0 fixes the problem.

Last edited by CoolGenie (2018-09-14 19:20:50)

Slithery · 2018-09-14 19:46:29

This isn't solved, your system is now in a partially updated, unsupported state.

Have you reported this upstream to the salt devs?

ephreal · 2018-09-14 20:48:54

I'm not sure where to report this to (First time really running across a bug of this caliber). I'm also not 100% sure if this is a salt bug or an openssl bug.

EDIT: After looking around for a while, I determined that I should probably report this to the salt team. I have left a bug report on github.

Last edited by ephreal (2018-09-14 22:10:45)

demaio · 2018-09-15 19:16:47

... and this is the link to ephreals bug report: https://github.com/saltstack/salt/issues/49661

EDIT: Somebody has already reported this... you can try the workaround in this bug report: https://bugs.archlinux.org/task/60031

ephreal, perhaps you should link your github issue to the older one?

Last edited by demaio (2018-09-15 19:49:57)

ephreal · 2018-09-16 18:19:45

Thanks demaio, I'm a bit new when it comes to bug reports and such, and the proper etiquette when reporting them isn't quite second nature yet. I'll look into those and make sure they are the same issue, and then learn how to link them if they are.

ephreal · 2018-09-16 18:30:42

I tested the patch, and it does fix the issue. Again, thanks demaio. Despite what I felt was a thorough search, I missed a bug report somewhere. The current fix is the patch, and the long term fix is wait for salt-2018.3.3 to be released. If, for some reason, the patch does not work, you can grab the current rsax931.py from https://raw.githubusercontent.com/salts … rsax931.py and replace your current rsax931.py on your system. Remember to make a backup of it before replacing it just in case though. I believe this can now be considered solved.

Last edited by ephreal (2018-09-16 18:37:59)

demaio · 2018-09-17 10:31:24

salt 2018.3.2-3 is available in [Community] which includes the patch, thanks @demize

ephreal, I also don't have much experience with bug reports but this one was important, so thank you! I felt a bit uncomfortable without my salt-master ;-)

Last edited by demaio (2018-09-17 10:31:55)

Arch Linux

#1 2018-09-14 17:39:29

[SOLVED] Salt server unable to start either salt-master or salt-minion

#2 2018-09-14 18:54:52

Re: [SOLVED] Salt server unable to start either salt-master or salt-minion

#3 2018-09-14 19:19:41

Re: [SOLVED] Salt server unable to start either salt-master or salt-minion

#4 2018-09-14 19:46:29

Re: [SOLVED] Salt server unable to start either salt-master or salt-minion

#5 2018-09-14 20:48:54

Re: [SOLVED] Salt server unable to start either salt-master or salt-minion

#6 2018-09-15 19:16:47

Re: [SOLVED] Salt server unable to start either salt-master or salt-minion

#7 2018-09-16 18:19:45

Re: [SOLVED] Salt server unable to start either salt-master or salt-minion

#8 2018-09-16 18:30:42

Re: [SOLVED] Salt server unable to start either salt-master or salt-minion

#9 2018-09-17 10:31:24

Re: [SOLVED] Salt server unable to start either salt-master or salt-minion

Board footer