[BUG] Salt 3008 PAM auth always returns 401 - __find_pyexe() returns system Python instead of Salt's bundled Python

[ebraheem-abo]

Description

On a fresh installation of Salt 3008, all salt-api
authentication requests via PAM return 401 Unauthorized.

No upgrade was involved. This is a clean first-time
installation of Salt 3008 on Ubuntu 22.04.

The root cause was identified through step-by-step debugging:
__find_pyexe() in salt/auth/pam.py returns the system Python
(/usr/bin/python3.10) instead of Salt's own bundled Python
(/opt/saltstack/salt/bin/python3.14), causing the PAM
authentication subprocess to fail silently with exit code 1.

---

Verified Environment

• Salt Version: 3008.0 (Argon)
• OS: Ubuntu 22.04.5 LTS (jammy) running on WSL2 (Windows 11)
• System Python: /usr/bin/python3.10 (Python 3.10.12)
• Salt bundled Python: /opt/saltstack/salt/bin/python3.14
• Installation type: Fresh install — no upgrade from 3007
• Installation source: packages.broadcom.com (official)

---

Steps to Reproduce

1. Install Salt 3008 fresh on Ubuntu 22.04 via official package:

curl -fsSL https://packages.broadcom.com/artifactory/api/security/keypair/SaltProjectKey/public
| sudo tee /usr/share/keyrings/salt-archive-keyring.pgp

echo "deb [signed-by=/usr/share/keyrings/salt-archive-keyring.pgp arch=amd64]
https://packages.broadcom.com/artifactory/saltproject-deb/ stable main"
| sudo tee /etc/apt/sources.list.d/salt.list

sudo apt update && sudo apt install salt-master salt-api -y

2. Verify installed version:

salt --version

Output: salt 3008.0 (Argon)

3. Configure external_auth with PAM:

/etc/salt/master.d/api.conf

rest_cherrypy:
port: 8000
disable_ssl: true

external_auth:
pam:
saltapi:
- .*
- '@runner'
- '@wheel'

4. Create a system user for authentication:

sudo useradd -M -s /bin/bash saltapi
sudo passwd saltapi

5. Install python-pam in Salt's Python environment:

sudo /opt/saltstack/salt/bin/pip install python-pam

6. Start services:

sudo service salt-master start
sudo service salt-api start

7. Attempt to authenticate:

curl -si http://localhost:8000/login
-H 'Accept: application/json'
-d username=saltapi
-d password=yourpassword
-d eauth=pam

---

Expected Result

HTTP/1.1 200 OK
{
"return": [{
"token": "xxxxxxxxxxxxxxxx",
"user": "saltapi",
"eauth": "pam"
}]
}

---

Actual Result

HTTP/1.1 401 Unauthorized
"Could not authenticate using provided credentials"

Salt master log shows:
[salt.loaded.int.auth.pam][ERROR] Pam auth failed for saltapi:

Note: stdout and stderr are both empty in the log,
making the root cause impossible to identify without
adding custom debug output.

---

Root Cause Investigation

Step 1 — Modified error log in authenticate() in pam.py

Changed:
log.error("Pam auth failed for %s: %s %s",
username, ret.stdout, ret.stderr)

To:
log.error("Pam auth failed for %s: returncode=%s stdout=%s stderr=%s pyexe=%s",
username, ret.returncode, ret.stdout, ret.stderr, str(pyexe))

Step 2 — Debug Evidence

Salt master log immediately revealed:

Pam auth failed for saltapi: returncode=1 stdout= stderr= pyexe=/usr/bin/python3.10

__find_pyexe() is returning:
/usr/bin/python3.10 ← WRONG

Instead of Salt's bundled Python:
/opt/saltstack/salt/bin/python3.14 ← CORRECT

Step 3 — Confirmed the mismatch

Salt 3008 is installed with its own bundled Python at:
/opt/saltstack/salt/bin/python3.14

System Python is at:
/usr/bin/python3.10 (Python 3.10.12)

The system Python 3.10 does NOT have:

• python-pam library
• Salt's internal libraries
• Any dependencies required for PAM auth

When Salt runs the PAM subprocess using python3.10,
it fails with exit code 1, causing 401 on every
authentication attempt.

Step 4 — Verified PAM works correctly with Salt's Python

sudo /opt/saltstack/salt/bin/python3.14 -c
"import pam; p = pam.pam(); print(p.authenticate('saltapi', 'password'))"

Result: True

Step 5 — Verified subprocess works correctly

sudo SALT_PAM_USERNAME=saltapi
SALT_PAM_PASSWORD=yourpassword
SALT_PAM_SERVICE=login
SALT_PAM_ENCODING=utf-8
/opt/saltstack/salt/bin/python3.14
/opt/saltstack/salt/lib/python3.14/site-packages/salt/auth/pam.py

Exit code: 0

This confirms PAM authentication works correctly when
using Salt's own Python. The failure happens ONLY because
__find_pyexe() returns the wrong Python executable.

---

Workaround

Add the following to /etc/salt/master.d/pam.conf:

auth.pam.python: /opt/saltstack/salt/bin/python3.14

This forces __find_pyexe() to use Salt's own Python
instead of the system Python.

---

Suggested Fix

In salt/auth/pam.py, the __find_pyexe() function should
always prioritize Salt's own bundled Python.

Current behavior:
__find_pyexe() → /usr/bin/python3.10

Expected behavior:
__find_pyexe() → /opt/saltstack/salt/bin/python3.14

---

Impact

• Affects ALL fresh installations of Salt 3008 on Ubuntu 22.04
• salt-api PAM authentication is completely broken
• The error log shows no useful information about the root cause
• Empty stdout and stderr make debugging extremely difficult
• Workaround requires deep knowledge of Salt internals

---

Additional Notes

• python-pam must also be manually installed in Salt's Python:
  sudo /opt/saltstack/salt/bin/pip install python-pam
  This is not documented anywhere in the official Salt docs.

• Even after fixing the Python path, additional PAM configuration
  issues exist in WSL2 environments due to forked process context
  limitations with pam_unix.so

---

Environment Details

salt --version

salt 3008.0 (Argon)

lsb_release -a

Ubuntu 22.04.5 LTS (jammy)

python3 --version

Python 3.10.12

/opt/saltstack/salt/bin/python3.14 --version

Python 3.14.x