Winbind in SmartOS, Part II (Running in Base-64)

We left off last time with a very basic (but working) Winbind deployment. In this post, we will focus on getting that same basic Winbind functionality in a base-64 zone. If you want to try polishing first, you can skip this post for now and use your base-32 zone for part 3.

Starting with a base-64 zone, we can start following the steps listed in part 1 of the series. Returning here when things start to go sideways, and we’ll work out the steps necessary for the new architecture.

Following NSS configuration, the “getent” commands still aren’t picking up on Active Directory users or groups, even after adding the /opt/local/lib path to ld.config.

Troubleshooting

Following the troubleshooting steps from post 1 (disable name-service-cache, grep truss for winbind) doesn’t reveal anything crazy: getent finds and loads our “nss_winbind.so.1” link we created. Running truss without grep, however, reveals something interesting:

# truss getent passwd 2>&1 >/dev/null | tail
stat64("/usr/lib/nss_winbind.so.1", 0x08047110) Err#2 ENOENT
stat64("/opt/local/lib/nss_winbind.so.1", 0x08047110) = 0
resolvepath("/opt/local/lib/nss_winbind.so.1", "/opt/local/lib/libnss_winbind.so", 1023) = 32
open("/opt/local/lib/nss_winbind.so.1", O_RDONLY) = 3
mmapobj(3, MMOBJ_INTERPRET, 0xFED60790, 0x0804717C, 0x00000000) Err#48 ENOTSUP
mmap(0x00000000, 4096, PROT_READ, MAP_PRIVATE, 3, 0) = 0xFED50000
munmap(0xFED50000, 4096)                        = 0
close(3)                                        = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/SUNW_OST_SGS.mo", O_RDONLY) Err#2 ENOENT
_exit(0)

An error is produced when getent tries to load nss_winbind (“mmapobj(3, […]) Err#48 ENOTSUP”). Unfortunately, a quick search for that error doesn’t give us anything useful. Time to roll up our sleeves and read the mmapobj documentation:

Errors

The mmapobj() function will fail if:

[…]

ENOTSUP

The current user data model does not match the fd to be interpreted. For example, a 32-bit process that tried to use mmapobj() to interpret a 64-bit object would return ENOTSUP.

The flags argument contains MMOBJ_INTERPRET and the fd argument is a file whose type can not be interpreted.

The ELF header contains an unaligned e_phentsize value.

Are we running a 32-bit process and trying to interpret a 64-bit object?

# which getent
/usr/bin/getent
# file /usr/bin/getent
/usr/bin/getent:        ELF 32-bit LSB executable 80386 Version 1, dynamically linked, not stripped, no debugging information available
# file /opt/local/lib/libnss_winbind.so
/opt/local/lib/libnss_winbind.so:       ELF 64-bit LSB dynamic lib AMD64 Version 1, dynamically linked, not stripped

Sure enough, getent is 32-bit and our libnss_winbind.so is 64. Old Solaris, and by lineage SmartOS, has a mix of 32 and 64-bit processes and commands in a 64-bit environment. Pkgsrc doesn’t support a mix of 32-bit and 64-bit in packages, so the 32-bit commands are out of luck in a 64-bit Samba package obtained through pkgsrc.

Can we get 32-bit packages?

Long story short, we can! We will use pkg_add, with a few special options to convince it to install 32-bit packages in a separate directory (we chose /opt/i386):

  • Set PKG_PATH environment variable to the value specified in /opt/local/etc/pkg_install.conf, substituting i386 for x86_64. Use this command to see what you should use: “sed -n ‘s/x86_64/i386/p’ /opt/local/etc/pkg_install.conf”.
  • “-m i386”, override the architecture otherwise the process will error.
  • “-P /opt/i386 -I”, install the packages to an alternate location and don’t run scripts, preventing the overwriting of any 64-bit packages.

Install the 32-bit samba package (will also install all 32-bit dependencies):

# sed -n 's/x86_64/i386/p' /opt/local/etc/pkg_install.conf
PKG_PATH=http://pkgsrc.joyent.com/packages/SmartOS/2015Q4/i386/All
# env PKG_PATH=http://pkgsrc.joyent.com/packages/SmartOS/2015Q4/i386/All pkg_add -I -m i386 -P /opt/i386 samba
[...]

Since we are now working with a new directory structure, we need to link libnss_winbind.so again:

# ln -s libnss_winbind.so /opt/i386/opt/local/lib/nss_winbind.so.1
# 
Troubleshooting

Let’s try getent again:

# getent passwd
[no domain users]

Using truss, we see that getent isn’t looking in our new i386 folder. We then need to ensure that 32-bit processes use the 32-bit libraries.

Replace /opt/local/lib with the i386 path in ld.config:

# crle -c /var/ld/ld.config -l /lib:/usr/lib:/opt/i386/opt/local/lib -s /lib/secure:/usr/lib/secure
# 

The base-64 library paths are stored in a different ld.config, so we will add /opt/local/lib there so that the 64-bit processes can find the Winbind-related libraries:

# crle -64 -c /var/ld/64/ld.config -l /opt/local/lib -u
# 
Troubleshooting

Running getent again, we still find no domain users. Back to truss, we discover that it is trying to load /opt/local/lib/samba/private/libwinbind-client-samba4.so, a 64-bit library, but failing with another “ENOTSUP” error.

Add /opt/i386/opt/local/lib/samba/private to ld.config:

# crle -c /var/ld/ld.config -l /opt/i386/opt/local/lib/samba/private -u
# 

Testing again with getent, we now have domain users! Restart the ssh and cron services, and re-enable the name-service-cache service in case you disabled it earlier:

# svcadm restart cron
# svcadm restart ssh
# svcadm enable name-service-cache
#

Now we head back to part 1, continuing with the PAM configuration.

We get to the last step, but SSH access doesn’t work as advertised.

Troubleshooting

Checking in the authentication logs at /var/log/authlog, we see this PAM error message:

2016-02-22T22:10:00+00:00 localhost cron[9433]: [ID 705739 auth.error] open_module[0:/etc/pam.conf]: /opt/local/lib/samba/security/pam_winbind.so failed: ld.so.1: cron: fatal: /opt/local/lib/samba/security/pam_winbind.so: wrong ELF class: ELFCLASS64

So the path we added in pam.conf was for the 64-bit pam_winbind module, let’s update that path to work for both 32 and 64-bit processes requiring PAM.

We need to update pam.conf with a link to the base-32 pam_winbind.so, adding “$ISA” for 64-bit process support:

other auth requisite          pam_authtok_get.so.1
other auth required           pam_dhkeys.so.1
other auth required           pam_unix_cred.so.1
other auth sufficient         /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so use_first_pass
other auth sufficient         pam_unix_auth.so.1

other account sufficient      /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so use_first_pass
other account requisite       pam_roles.so.1
other account required        pam_unix_account.so.1

other session required        pam_unix_session.so.1
other session required        /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so

other password required       pam_dhkeys.so.1
other password sufficient     /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so
other password requisite      pam_authtok_get.so.1
other password requisite      pam_authtok_check.so.1
other password required       pam_authtok_store.so.1

A link to the 64-bit module needs to be inserted into a subdirectory of the 32-bit path, since “$ISA” will be replaced with “64” for 64-bit services utilizing PAM:

# mkdir /opt/i386/opt/local/lib/samba/security/64
# ln -s /opt/local/lib/samba/security/pam_winbind.so /opt/i386/opt/local/lib/samba/security/64/
# 

SSH access via domain credentials is now available in our base-64 zone:

$ ssh jeremy.einfeld@10.88.88.148
Password:
Could not chdir to home directory /home/jeremy.einfeld: No such file or directory
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (base-64-lts 15.4.0)
                   `-'  https://docs.joyent.com/images/smartos/base

-bash-4.1$

Our next step is to polish our Winbind deployment, getting it ready for a real environment in part 3.