Winbind in SmartOS, Part II (Running in Base-64)

We left off last time with a very basic (but working) Winbind deployment. In this post, we will focus on getting that same basic Winbind functionality in a base-64 zone. If you want to try polishing first, you can skip this post for now and use your base-32 zone for part 3.

Starting with a base-64 zone, we can start following the steps listed in part 1 of the series. Returning here when things start to go sideways, and we’ll work out the steps necessary for the new architecture.

Following NSS configuration, the “getent” commands still aren’t picking up on Active Directory users or groups, even after adding the /opt/local/lib path to ld.config.

Troubleshooting

Following the troubleshooting steps from post 1 (disable name-service-cache, grep truss for winbind) doesn’t reveal anything crazy: getent finds and loads our “nss_winbind.so.1” link we created. Running truss without grep, however, reveals something interesting:

# truss getent passwd 2>&1 >/dev/null | tail
stat64("/usr/lib/nss_winbind.so.1", 0x08047110) Err#2 ENOENT
stat64("/opt/local/lib/nss_winbind.so.1", 0x08047110) = 0
resolvepath("/opt/local/lib/nss_winbind.so.1", "/opt/local/lib/libnss_winbind.so", 1023) = 32
open("/opt/local/lib/nss_winbind.so.1", O_RDONLY) = 3
mmapobj(3, MMOBJ_INTERPRET, 0xFED60790, 0x0804717C, 0x00000000) Err#48 ENOTSUP
mmap(0x00000000, 4096, PROT_READ, MAP_PRIVATE, 3, 0) = 0xFED50000
munmap(0xFED50000, 4096)                        = 0
close(3)                                        = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/SUNW_OST_SGS.mo", O_RDONLY) Err#2 ENOENT
_exit(0)

An error is produced when getent tries to load nss_winbind (“mmapobj(3, […]) Err#48 ENOTSUP”). Unfortunately, a quick search for that error doesn’t give us anything useful. Time to roll up our sleeves and read the mmapobj documentation:

Errors

The mmapobj() function will fail if:

[…]

ENOTSUP

The current user data model does not match the fd to be interpreted. For example, a 32-bit process that tried to use mmapobj() to interpret a 64-bit object would return ENOTSUP.

The flags argument contains MMOBJ_INTERPRET and the fd argument is a file whose type can not be interpreted.

The ELF header contains an unaligned e_phentsize value.

Are we running a 32-bit process and trying to interpret a 64-bit object?

# which getent
/usr/bin/getent
# file /usr/bin/getent
/usr/bin/getent:        ELF 32-bit LSB executable 80386 Version 1, dynamically linked, not stripped, no debugging information available
# file /opt/local/lib/libnss_winbind.so
/opt/local/lib/libnss_winbind.so:       ELF 64-bit LSB dynamic lib AMD64 Version 1, dynamically linked, not stripped

Sure enough, getent is 32-bit and our libnss_winbind.so is 64. Old Solaris, and by lineage SmartOS, has a mix of 32 and 64-bit processes and commands in a 64-bit environment. Pkgsrc doesn’t support a mix of 32-bit and 64-bit in packages, so the 32-bit commands are out of luck in a 64-bit Samba package obtained through pkgsrc.

Can we get 32-bit packages?

Long story short, we can! We will use pkg_add, with a few special options to convince it to install 32-bit packages in a separate directory (we chose /opt/i386):

  • Set PKG_PATH environment variable to the value specified in /opt/local/etc/pkg_install.conf, substituting i386 for x86_64. Use this command to see what you should use: “sed -n ‘s/x86_64/i386/p’ /opt/local/etc/pkg_install.conf”.
  • “-m i386”, override the architecture otherwise the process will error.
  • “-P /opt/i386 -I”, install the packages to an alternate location and don’t run scripts, preventing the overwriting of any 64-bit packages.

Install the 32-bit samba package (will also install all 32-bit dependencies):

# sed -n 's/x86_64/i386/p' /opt/local/etc/pkg_install.conf
PKG_PATH=http://pkgsrc.joyent.com/packages/SmartOS/2015Q4/i386/All
# env PKG_PATH=http://pkgsrc.joyent.com/packages/SmartOS/2015Q4/i386/All pkg_add -I -m i386 -P /opt/i386 samba
[...]

Since we are now working with a new directory structure, we need to link libnss_winbind.so again:

# ln -s libnss_winbind.so /opt/i386/opt/local/lib/nss_winbind.so.1
# 
Troubleshooting

Let’s try getent again:

# getent passwd
[no domain users]

Using truss, we see that getent isn’t looking in our new i386 folder. We then need to ensure that 32-bit processes use the 32-bit libraries.

Replace /opt/local/lib with the i386 path in ld.config:

# crle -c /var/ld/ld.config -l /lib:/usr/lib:/opt/i386/opt/local/lib -s /lib/secure:/usr/lib/secure
# 

The base-64 library paths are stored in a different ld.config, so we will add /opt/local/lib there so that the 64-bit processes can find the Winbind-related libraries:

# crle -64 -c /var/ld/64/ld.config -l /opt/local/lib -u
# 
Troubleshooting

Running getent again, we still find no domain users. Back to truss, we discover that it is trying to load /opt/local/lib/samba/private/libwinbind-client-samba4.so, a 64-bit library, but failing with another “ENOTSUP” error.

Add /opt/i386/opt/local/lib/samba/private to ld.config:

# crle -c /var/ld/ld.config -l /opt/i386/opt/local/lib/samba/private -u
# 

Testing again with getent, we now have domain users! Restart the ssh and cron services, and re-enable the name-service-cache service in case you disabled it earlier:

# svcadm restart cron
# svcadm restart ssh
# svcadm enable name-service-cache
#

Now we head back to part 1, continuing with the PAM configuration.

We get to the last step, but SSH access doesn’t work as advertised.

Troubleshooting

Checking in the authentication logs at /var/log/authlog, we see this PAM error message:

2016-02-22T22:10:00+00:00 localhost cron[9433]: [ID 705739 auth.error] open_module[0:/etc/pam.conf]: /opt/local/lib/samba/security/pam_winbind.so failed: ld.so.1: cron: fatal: /opt/local/lib/samba/security/pam_winbind.so: wrong ELF class: ELFCLASS64

So the path we added in pam.conf was for the 64-bit pam_winbind module, let’s update that path to work for both 32 and 64-bit processes requiring PAM.

We need to update pam.conf with a link to the base-32 pam_winbind.so, adding “$ISA” for 64-bit process support:

other auth requisite          pam_authtok_get.so.1
other auth required           pam_dhkeys.so.1
other auth required           pam_unix_cred.so.1
other auth sufficient         /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so use_first_pass
other auth sufficient         pam_unix_auth.so.1

other account sufficient      /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so use_first_pass
other account requisite       pam_roles.so.1
other account required        pam_unix_account.so.1

other session required        pam_unix_session.so.1
other session required        /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so

other password required       pam_dhkeys.so.1
other password sufficient     /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so
other password requisite      pam_authtok_get.so.1
other password requisite      pam_authtok_check.so.1
other password required       pam_authtok_store.so.1

A link to the 64-bit module needs to be inserted into a subdirectory of the 32-bit path, since “$ISA” will be replaced with “64” for 64-bit services utilizing PAM:

# mkdir /opt/i386/opt/local/lib/samba/security/64
# ln -s /opt/local/lib/samba/security/pam_winbind.so /opt/i386/opt/local/lib/samba/security/64/
# 

SSH access via domain credentials is now available in our base-64 zone:

$ ssh jeremy.einfeld@10.88.88.148
Password:
Could not chdir to home directory /home/jeremy.einfeld: No such file or directory
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (base-64-lts 15.4.0)
                   `-'  https://docs.joyent.com/images/smartos/base

-bash-4.1$

Our next step is to polish our Winbind deployment, getting it ready for a real environment in part 3.

Winbind in SmartOS, Part I (the Basics)

Centralized authentication is a terrific tool for anything more than a handful of servers, and Active Directory is often the go-to for authentication within a datacenter.

After testing several centralized-authentications for suitability in our environment, we settled on Winbind both for its features and the issues presented by the other solutions. A future post may detail the investigation and decision, but the biggest reason for us to use Winbind (of several reasons) was the ability to limit SSH access to servers through Active Directory group membership

Note that these steps apply to local SDC/Triton deployments and the Joyent Public Cloud, in addition to stand-alone SmartOS servers, and no changes should be necessary in Active Directory.

TL;DR: see this setup script for a summarization of steps taken in this series, and this repository for configuration files used. Note that some details from part 3 are not covered in the repository, see the readme for a list.

Troubleshooting Steps

You can ignore these if you just want to get Winbind working, but check them out if you want to see why the prior command didn’t work and how the next commands fix it.

WARNING

Don’t try this in base-64 (yet), there are additional difficulties that are encountered that will be handled in part 2. Use base-32 for now. Additionally, image releases prior to 15.2.0 require additional steps that will not be covered in this series so using 15.2.0+ is recommended.

Configuring Kerberos

Winbind uses a variety of protocols to interact with domain accounts, and the primary one that we need to set up is Kerberos. The configuration is managed in /etc/krb5/krb5.conf, and we can use this minimalistic config to get us started:

[libdefaults]
        default_realm = DOMAIN.NET
        dns_lookup_kdc = true
        default_tkt_enctypes = rc4-hmac des-cbc-crc des-cbc-md5

[realms]
        DOMAIN.NET = {
                kpasswd_protocol = SET_CHANGE
        }

Install the mit-krb5 package, then test Kerberos key initialization:

# pkgin -y install mit-krb5
[...]
# kinit [user]
kinit: Configuration file does not specify default realm when parsing name [user]
#

Something isn’t adding up, we did set default_realm in krb5.conf.

Troubleshooting

Let’s use truss to see where kinit is looking for the config file:

# truss kinit [user] 2>&1 >/dev/null | grep krb5.conf
stat("/opt/local/etc/krb5.conf", 0xFFFFFD7FFFDFF390) Err#2 ENOENT
stat("/etc/krb5.conf", 0xFFFFFD7FFFDFF390) Err#2 ENOENT
#

There’s the issue, the default krb5.conf included by SmartOS isn’t where Kerberos is looking.

Link krb5.conf to /etc/:

# ln -s krb5/krb5.conf /etc/

Test Kerberos again:

# kinit [user]
Password for [user]@DOMAIN.NET:
# klist
Default principal: [user]@DOMAIN.NET

Valid starting     Expires            Service principal
02/18/16 18:31:54  02/19/16 04:32:06  krbtgt/DOMAIN.NET@DOMAIN.NET
        renew until 02/19/16 18:31:54
# 

Success! Kerberos is now ready to start working for Winbind.

Configuring Winbind

Our first step for Winbind itself is to install the necessary package via pkgsrc. Winbind is bundled in the Samba package, so that is the one that we will use here:

# pkgin -y install samba

We then need to configure Winbind. The settings are stored in the Samba config file at /opt/local/etc/samba/smb.conf; here is a bare-bones smb.conf for our “domain.net” access:

[global]
        workgroup = DOMAIN
        realm = DOMAIN.NET
        idmap config * : backend = tdb
        idmap config * : range = 1000 - 9999
        idmap config DOMAIN:backend = rid
        idmap config DOMAIN:range = 10000 - 1073751823
        idmap config DOMAIN:schema_mode = rfc2307
        kerberos method = secrets and keytab
        security = ADS
        winbind enum groups = yes
        winbind enum users = yes
        winbind offline logon = yes
        winbind use default domain = yes
        template homedir = /home/%U
        template shell = /usr/bin/bash

We now domain-bind the instance using the Kerberos ticket we created previously with kinit:

# net join -k
Using short domain name -- DOMAIN
Joined '[HOSTNAME]' to dns domain 'domain.net'
#

Now that we have Winbind configured, we can start the daemon with this command:

# winbindd
#

Don’t worry about starting any of the Samba services, they are outside of the scope of Winbind. Test Winbind with these commands:

# wbinfo -u
[list of domain users]
# wbinfu -g
[list of domain groups]
# wbinfo -i [user]
[passwd entry for user, including full name, id, group id, home directory, and shell]
#

Configuring NSS

User and group data is made available in SmartOS through the Name Service Switch (NSS) facility, and now that Winbind is working we can add it as a source for that information. Update the passwd and group settings in /etc/nsswitch.conf, appending “winbind”:

passwd:     files winbind
group:      files winbind

We can then test to see what the system sees as the users, groups, and credentials with these commands:

# getent passwd
[passwd]
# getent group
[groups]

Winbind sees users and groups, but the system isn’t seeing them through NSS.

Troubleshooting

Let’s run getent through truss to see if it is doing anything with the Winbind NSS module:

# truss getent passwd 2>&1 >/dev/null | grep winbind
#

Nothing. A grep for “nsswitch.conf” also returns nothing, so it appears that getent isn’t even using NSS. Looking at the SMF services, there is a name-service-cache service that is keeping user and group data cached (negating the need for getent to check through NSS). Disabling that service, we try getent again:

# svcadm disable name-service-cache
# truss getent passwd 2>&1 >/dev/null | grep winbind
stat64("/lib/nss_winbind.so.1", 0x080471A0)     Err#2 ENOENT
stat64("/usr/lib/nss_winbind.so.1", 0x080471A0) Err#2 ENOENT
#

Now we’re talking. So where is nss_winbind.so.1, if not there? A few searches later, we find that there is no “nss_winbind.so.1”: the Samba package loads a libnss_winbind.so into /opt/local/lib/. We need to link libnss_winbind into a location that is checked for NSS modules:

# ln -s /opt/local/lib/libnss_winbind.so /lib/nss_winbind.so.1
ln: failed to create symbolic link '/lib/nss_winbind.so.1': Read-only file system
# ln -s /opt/local/lib/libnss_winbind.so /usr/lib/nss_winbind.so.1
ln: failed to create symbolic link '/usr/lib/nss_winbind.so.1': Read-only file system
#

That didn’t exactly work, since /usr and /lib are read-only. With that knowledge, we need to change the search locations by modifying (indirectly) ld.conf. We can see the current configuration by running this command:

# crle

Configuration file [version 4]: /var/ld/ld.config
  Platform:     32-bit LSB 80386
  Default Library Path (ELF):   /lib:/usr/lib
  Trusted Directories (ELF):    /lib/secure:/usr/lib/secure

Command line:
  crle -c /var/ld/ld.config -l /lib:/usr/lib -s /lib/secure:/usr/lib/secure

#

It even gives us the command to use to set it to replicate the current settings! Don’t forget to re-enable the name-service-cache service.

Add /opt/local/lib to ld.config, restart the services that need the new library path, and link libnss_winbind.so as nss_winbind.so.1:

# crle -c /var/ld/ld.config -l /opt/local/lib -u
# svcadm restart ssh
# svcadm restart cron
# ln -s libnss_winbind.so /opt/local/lib/nss_winbind.so.1
# svcadm restart name-service-cache
# getent passwd
[passwd including domain users]
# getent group
[groups including domain groups]

Configuring PAM

Our last step for basic Winbind auth is to configure the Pluggable Authentication Module (PAM) to use the pam_winbind module. We need to look for these sections in /etc/pam.conf and add the module into them (order does matter within each section):

WARNING

Adjusting the PAM configuration is a delicate operation: make any mistakes and authentication will no longer work. Always leave a session open when working with pam.conf, since a mistake won’t affect sessions that are already active. There is an additional safety net available in SmartOS: you can edit pam.conf from the SmartOS global zone via /zones/[uuid]/root/etc/pam.conf.

other auth requisite          pam_authtok_get.so.1
other auth required           pam_dhkeys.so.1
other auth required           pam_unix_cred.so.1
other auth sufficient         /opt/local/lib/samba/security/pam_winbind.so use_first_pass
other auth sufficient         pam_unix_auth.so.1

other account sufficient      /opt/local/lib/samba/security/pam_winbind.so use_first_pass
other account requisite       pam_roles.so.1
other account required        pam_unix_account.so.1

other session required        pam_unix_session.so.1
other session required        /opt/local/lib/samba/security/pam_winbind.so

other password required       pam_dhkeys.so.1
other password sufficient     /opt/local/lib/samba/security/pam_winbind.so
other password requisite      pam_authtok_get.so.1
other password requisite      pam_authtok_check.so.1
other password required       pam_authtok_store.so.1

With that configured, we can finally get SSH access via domain credentials:

$ ssh [user]@[host]
Password:
Could not chdir to home directory /home/[user]: No such file or directory
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (base-32-lts 15.4.0)
                   `-'  https://docs.joyent.com/images/smartos/base

mail: Cannot open file '/var/mail/' for output
-bash-4.1$

Not perfect, but we have Winbind access! Upcoming posts will detail how to get Winbind working in base-64 (part 2) and polishing it up for a real environment (part 3).