Winbind in SmartOS, Part II (Running in Base-64)

We left off last time with a very basic (but working) Winbind deployment. In this post, we will focus on getting that same basic Winbind functionality in a base-64 zone. If you want to try polishing first, you can skip this post for now and use your base-32 zone for part 3.

Starting with a base-64 zone, we can start following the steps listed in part 1 of the series. Returning here when things start to go sideways, and we’ll work out the steps necessary for the new architecture.

…

Following NSS configuration, the “getent” commands still aren’t picking up on Active Directory users or groups, even after adding the /opt/local/lib path to ld.config.

Troubleshooting

Following the troubleshooting steps from post 1 (disable name-service-cache, grep truss for winbind) doesn’t reveal anything crazy: getent finds and loads our “nss_winbind.so.1” link we created. Running truss without grep, however, reveals something interesting:

# truss getent passwd 2>&1 >/dev/null | tail
stat64("/usr/lib/nss_winbind.so.1", 0x08047110) Err#2 ENOENT
stat64("/opt/local/lib/nss_winbind.so.1", 0x08047110) = 0
resolvepath("/opt/local/lib/nss_winbind.so.1", "/opt/local/lib/libnss_winbind.so", 1023) = 32
open("/opt/local/lib/nss_winbind.so.1", O_RDONLY) = 3
mmapobj(3, MMOBJ_INTERPRET, 0xFED60790, 0x0804717C, 0x00000000) Err#48 ENOTSUP
mmap(0x00000000, 4096, PROT_READ, MAP_PRIVATE, 3, 0) = 0xFED50000
munmap(0xFED50000, 4096)                        = 0
close(3)                                        = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/SUNW_OST_SGS.mo", O_RDONLY) Err#2 ENOENT
_exit(0)

An error is produced when getent tries to load nss_winbind (“mmapobj(3, […]) Err#48 ENOTSUP”). Unfortunately, a quick search for that error doesn’t give us anything useful. Time to roll up our sleeves and read the mmapobj documentation:

Errors

The mmapobj() function will fail if:
[…]
ENOTSUP

The current user data model does not match the fd to be interpreted. For example, a 32-bit process that tried to use mmapobj() to interpret a 64-bit object would return ENOTSUP.

The flags argument contains MMOBJ_INTERPRET and the fd argument is a file whose type can not be interpreted.

The ELF header contains an unaligned e_phentsize value.

Are we running a 32-bit process and trying to interpret a 64-bit object?

# which getent
/usr/bin/getent
# file /usr/bin/getent
/usr/bin/getent:        ELF 32-bit LSB executable 80386 Version 1, dynamically linked, not stripped, no debugging information available
# file /opt/local/lib/libnss_winbind.so
/opt/local/lib/libnss_winbind.so:       ELF 64-bit LSB dynamic lib AMD64 Version 1, dynamically linked, not stripped

Sure enough, getent is 32-bit and our libnss_winbind.so is 64. Old Solaris, and by lineage SmartOS, has a mix of 32 and 64-bit processes and commands in a 64-bit environment. Pkgsrc doesn’t support a mix of 32-bit and 64-bit in packages, so the 32-bit commands are out of luck in a 64-bit Samba package obtained through pkgsrc.

Can we get 32-bit packages?

Long story short, we can! We will use pkg_add, with a few special options to convince it to install 32-bit packages in a separate directory (we chose /opt/i386):

Set PKG_PATH environment variable to the value specified in /opt/local/etc/pkg_install.conf, substituting i386 for x86_64. Use this command to see what you should use: “sed -n ‘s/x86_64/i386/p’ /opt/local/etc/pkg_install.conf”.
“-m i386”, override the architecture otherwise the process will error.
“-P /opt/i386 -I”, install the packages to an alternate location and don’t run scripts, preventing the overwriting of any 64-bit packages.

Install the 32-bit samba package (will also install all 32-bit dependencies):

# sed -n 's/x86_64/i386/p' /opt/local/etc/pkg_install.conf
PKG_PATH=http://pkgsrc.joyent.com/packages/SmartOS/2015Q4/i386/All
# env PKG_PATH=http://pkgsrc.joyent.com/packages/SmartOS/2015Q4/i386/All pkg_add -I -m i386 -P /opt/i386 samba
[...]

Since we are now working with a new directory structure, we need to link libnss_winbind.so again:

# ln -s libnss_winbind.so /opt/i386/opt/local/lib/nss_winbind.so.1
#

Troubleshooting

Let’s try getent again:

# getent passwd
[no domain users]

Using truss, we see that getent isn’t looking in our new i386 folder. We then need to ensure that 32-bit processes use the 32-bit libraries.

Replace /opt/local/lib with the i386 path in ld.config:

# crle -c /var/ld/ld.config -l /lib:/usr/lib:/opt/i386/opt/local/lib -s /lib/secure:/usr/lib/secure
#

The base-64 library paths are stored in a different ld.config, so we will add /opt/local/lib there so that the 64-bit processes can find the Winbind-related libraries:

# crle -64 -c /var/ld/64/ld.config -l /opt/local/lib -u
#

Troubleshooting

Running getent again, we still find no domain users. Back to truss, we discover that it is trying to load /opt/local/lib/samba/private/libwinbind-client-samba4.so, a 64-bit library, but failing with another “ENOTSUP” error.

Add /opt/i386/opt/local/lib/samba/private to ld.config:

# crle -c /var/ld/ld.config -l /opt/i386/opt/local/lib/samba/private -u
#

Testing again with getent, we now have domain users! Restart the ssh and cron services, and re-enable the name-service-cache service in case you disabled it earlier:

# svcadm restart cron
# svcadm restart ssh
# svcadm enable name-service-cache
#

Now we head back to part 1, continuing with the PAM configuration.

…

We get to the last step, but SSH access doesn’t work as advertised.

Troubleshooting

Checking in the authentication logs at /var/log/authlog, we see this PAM error message:

2016-02-22T22:10:00+00:00 localhost cron[9433]: [ID 705739 auth.error] open_module[0:/etc/pam.conf]: /opt/local/lib/samba/security/pam_winbind.so failed: ld.so.1: cron: fatal: /opt/local/lib/samba/security/pam_winbind.so: wrong ELF class: ELFCLASS64

So the path we added in pam.conf was for the 64-bit pam_winbind module, let’s update that path to work for both 32 and 64-bit processes requiring PAM.

We need to update pam.conf with a link to the base-32 pam_winbind.so, adding “$ISA” for 64-bit process support:

other auth requisite          pam_authtok_get.so.1
other auth required           pam_dhkeys.so.1
other auth required           pam_unix_cred.so.1
other auth sufficient         /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so use_first_pass
other auth sufficient         pam_unix_auth.so.1

other account sufficient      /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so use_first_pass
other account requisite       pam_roles.so.1
other account required        pam_unix_account.so.1

other session required        pam_unix_session.so.1
other session required        /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so

other password required       pam_dhkeys.so.1
other password sufficient     /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so
other password requisite      pam_authtok_get.so.1
other password requisite      pam_authtok_check.so.1
other password required       pam_authtok_store.so.1

A link to the 64-bit module needs to be inserted into a subdirectory of the 32-bit path, since “$ISA” will be replaced with “64” for 64-bit services utilizing PAM:

# mkdir /opt/i386/opt/local/lib/samba/security/64
# ln -s /opt/local/lib/samba/security/pam_winbind.so /opt/i386/opt/local/lib/samba/security/64/
#

SSH access via domain credentials is now available in our base-64 zone:

$ ssh jeremy.einfeld@10.88.88.148
Password:
Could not chdir to home directory /home/jeremy.einfeld: No such file or directory
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (base-64-lts 15.4.0)
                   `-'  https://docs.joyent.com/images/smartos/base

-bash-4.1$

Our next step is to polish our Winbind deployment, getting it ready for a real environment in part 3.