Winbind in SmartOS, Part III (Polishing)

You now have a base-32 or base-64 zone running Winbind, what do you need to take it the rest of the way? We will cover a number of topics in this post, pick the ones that are relevant to your environment:

A github repository has been made available for this series, including all relevant configuration files and a setup script that summarizes commands issued (aside from a few points in this post). The script is not intended for production use, but is good for a simple test and for those that read code more readily than blog posts.

Creating the Winbind Service

We have manually started Winbind with the “winbindd” command up to this point, let’s remedy that by creating an SMF service for the Winbind process. First, create a manifest file for the service (we can save it as ~/winbind.xml):

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
  <service name='network/winbind' type='service' version='0'>
    <create_default_instance enabled='true'/>
    <single_instance/>
    <dependency name='fs' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/system/filesystem/local'/>
    </dependency>
    <dependency name='network' grouping='optional_all' restart_on='none' type='service'>
      <service_fmri value='svc:/milestone/network'/>
    </dependency>
    <exec_method name='refresh' type='method' exec=':kill -HUP' timeout_seconds='60'/>
    <exec_method name='start' type='method' exec='/opt/local/sbin/winbindd' timeout_seconds='60'/>
    <exec_method name='stop' type='method' exec=':kill' timeout_seconds='60'/>
    <template>
      <common_name>
        <loctext xml:lang='C'>Winbind</loctext>
      </common_name>
    </template>
  </service>
</service_bundle>

Stop our current winbind process, then import the manifest using svccfg:

# pkill winbind
# svccfg import ~/winbind.xml
# 

We now have Winbind running as an SMF service:

# svcs winbind
STATE          STIME    FMRI
online         19:04:24 svc:/network/winbind:default
# 

Reducing the Authentication Delay

If you are testing Winbind on a larger domain, you may notice that the getent commands and login take a decent amount of patience. We had the Winbind user and group enumeration options enabled for testing purposes, but they are not strictly necessary for the majority of operations (we haven’t yet encountered a case where they were needed). Remove these lines from /opt/local/etc/samba/smb.conf:

        winbind enum groups = yes
        winbind enum users = yes

Restart the winbind service that we just created to apply the new configuration:

# svcadm refresh winbind
# 

Note that you can still use getent to query for particular users or groups by specifying the user or group name in the command, even if the general getent commands will no longer display all domain users or groups.

Creating Home Directories

If you look back at the end result of the previous posts, you will notice this error on login:

Could not chdir to home directory /home/[user]: No such file or directory

We have the home directory configured, but have nothing set up to create individual user directories.

Troubleshooting

Linux has a PAM module called “pam_mkhomedir”, and it seems like a straight-forward solution but is not provided or supported in SmartOS. Compiling the module in Solaris has been done, but we don’t want to support that kind of work through our configuration management solution (puppet) due to the amount of hack required.

Winbind PAM has a “mkhomedir” option, but it has a fatal flaw. Here is what login looks like with the option set:

$ ssh [user]@[host]
Password:
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (base-64-lts 15.4.0)
                   `-'  https://docs.joyent.com/images/smartos/base

-bash-4.1$ ls -al
total 2
drwx------   2 [user] [group]       2 Feb 26 19:19 .
drwxr-xr-x   4 root     root           4 Feb 26 19:19 ..
-bash-4.1$ echo $PATH
/usr/ccs/bin:/usr/bin:/bin:/usr/sbin:/sbin
-bash-4.1$ 

The home directory is created, but it is completely empty and we are left with bash defaults for everything. The PATH variable is especially unhelpful, as we will be missing out on around 400 commands with the default value (including sudo).

The autofs service was originally intended for auto-mounting of network shares, but it can be tweaked for the purposes of auto-generation of user home directories. Using a modified version of Znogger’s auto_homedir script, we will create a home directory for each user (as needed) based on a copy of the /etc/skel directory (very important, it includes good shell defaults including a good PATH value). Save this script to /etc/auto_home (overwrite the file that is there):

#!/usr/bin/bash

HOMEDIRPATH=/home
PHYSICALDIRPATH=/export/home

# The folder being requested is first argument, we assume at
# this point that it is a user
user=$1
group=$(id -gn "$user")
home_dir=$(getent passwd "$user" | cut -d: -f6)
physical_dir="$PHYSICALDIRPATH/$user"

# Only create a directory if it doesn't exist
if [ ! -d "$physical_dir" ]; then
  # Sanity check, ensure that this is actually a user and that
  # his home directory is in the expected location
  if [[ "$home_dir" != $HOMEDIRPATH/* ]]; then
    exit
  fi

  mkdir -p "$PHYSICALDIRPATH"
  # Use the shell defaults in /etc/skel to populate the initial
  # home directory
  cp -r /etc/skel "$physical_dir"
  chown -R "$user":"$group" "$physical_dir"
fi

echo "localhost:$physical_dir"

Make the script executable:

# chmod +x /etc/auto_home
# 

Enable the bind and autofs services (autofs needs bind to be running):

# svcadm enable bind
# svcadm enable autofs
# 

Now try logging in:

$ ssh [user]@[host]
Password:
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (base-64-lts 15.4.0)
                   `-'  https://docs.joyent.com/images/smartos/base

[user@host ~]$ ll -a
total 12
drwxr-xr-x 3 26467 27831  11 Feb 26 19:47 ./
dr-xr-xr-x 2 root  root    2 Feb 26 19:45 ../
-rw-r--r-- 1 26467 27831  76 Feb 26 19:47 .bash_profile
-rw-r--r-- 1 26467 27831 304 Feb 26 19:47 .bashrc
-rw-r--r-- 1 26467 27831 151 Feb 26 19:47 .cshrc
-rw-r--r-- 1 26467 27831  38 Feb 26 19:47 .curlrc
-rw-r--r-- 1 26467 27831 240 Feb 26 19:47 .irbrc
-rw-r--r-- 1 26467 27831  40 Feb 26 19:47 .login
-rw-r--r-- 1 26467 27831 690 Feb 26 19:47 .profile
drwxr-xr-x 2 26467 27831   3 Feb 26 19:47 .ssh/
-rw-r--r-- 1 26467 27831 846 Feb 26 19:47 .vimrc
[user@host ~]$ echo $PATH
/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin
[user@host ~]$ 

Much better!

How did that work?

You can see Znogger’s post for a more detailed explanation of the autofs process. We don’t need step 1 (modifying the /etc/auto_master file) in his instructions, since /etc/auto_home is configured by default in SmartOS to be called for /home. If you wanted to have several auto-mounted sources for /home, or use a different home directory, you will need to store the script in a different location and modify /etc/auto_master or the existing /etc/auto_home file instead.

Limiting Access

For the needs of the Faithlife environment, we want to limit SSH access based on Active Directory group membership. Setting that up in Winbind is straightforward, but let’s lay out some groundwork by moving pam_winbind settings to a separate settings file before it gets too long. Remove “use_first_pass” from pam.conf:

# sed -i 's/ use_first_pass//g' /etc/pam.conf
# 

Then create our initial pam_winbind configuration at /etc/security/pam_winbind.conf with this as the content:

[global]

try_first_pass = yes

We can now add a membership limitation in that file (get the group name or guid from getent groups list, and comma-separate if permitting multiple groups):

# echo 'require_membership_of = [group]' >> /etc/security/pam_winbind.conf
# 

Granting Sudo Access

Winbind access does not allow us to use the more granular features provided by the Solaris RBAC system, so we will need to go to the old admin standby: sudo. Nothing out of the ordinary for Winbind, we just need to add any groups or users into the sudoers file so that they can take administrative actions.

We’ll just append to the sudoers file here, but using the visudo command is recommended when manually editing sudoers. For a group (escape spaces and backslashes with backslashes):

# echo '%[group] ALL=(ALL) ALL' >> /opt/local/etc/sudoers
# 

For a user (also escaped if necessary):

# echo '[user] ALL=(ALL) ALL' >> /opt/local/etc/sudoers
# 

With that change, we now have sudo access:

[user@host ~]$ sudo su -
Password:
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (base-32-lts 15.4.0)
                   `-'  https://docs.joyent.com/images/smartos/base

[root@host ~]# 

Working with More than 16 Group Memberships

If your Active Directory domain users tend to have a significant number of security-group memberships, you may have already encountered this issue when granting sudo access: SmartOS is based on a version of Solaris that (by default) has a group membership limit of 16, and any AD security group memberships beyond 16 are truncated. The limitation does not affect basic SSH access, since the Winbind PAM doesn’t consult NSS for the membership list, but sudo will be blocked if the relevant group was not included in the 16.

The truncation isn’t realistically controllable, and although increasing the group limit is possible it requires grub-level changes (and a reboot) to make that change. That change becomes even more difficult in an SDC or Triton environment, and impossible in the JPC environment.

Our workaround hinges on one predictable aspect of the truncation: the primary user group is never cut. We ensure that the primary AD group for each AD user is the one with the most significant for SSH access.

Eliminating the SSH Key Loophole for Disabled Accounts

Using RSA keys to access SmartOS zones based on domain accounts is possible and a great boost to productivity while working with batches of servers. Unfortunately, use of RSA keys reveals a surprising limitation between PAM and SSH: if an RSA key is used for access, some of the PAM sections are not consulted. As a result, a disabled account will not be denied access by Winbind if an RSA key is utilized that was previously added to ~/.ssh/authorized_keys for that user.

Our workaround for that security hole is fairly simple: we utilized an Active Directory group for “Disabled Users” and block members of that group through the SSH service configuration. Do that by setting the DenyGroups value in sshd_config:

# echo 'DenyGroups "disabled users"' >> /etc/ssh/sshd_config
# svcadm refresh ssh
#

Note that the 16-group-membership limit applies here, so if you leave group memberships applied to the disabled account you should set the “Disabled Users” group as the primary security group.

Updating Passwords

Winbind PAM does support password changes via passwd and during login, but under SmartOS we don’t have that luxury:

[user@host ~]$ passwd
passwd: Changing password for [user]
passwd: Unsupported nsswitch entry for "passwd:". Use "-r repository ".
Unexpected failure. Password file/table unchanged.

Solaris, and by lineage SmartOS, only supports a few configurations for password changes (a hard-coded limitation):

Only five passwd configurations are permitted:

  • passwd: files
  • passwd: files nis
  • passwd: files nisplus
  • passwd: compat
  • passwd: compat passwd_compat: nisplus

There is no real fix for this limitation, but kpasswd (the Kerberos equivalent to passwd command) can be used to change the domain password for a user. The kpasswd command does not work for password changes on login.

Working with Long Hostnames

Since we are joining an Active Directory domain, a familiar limitation comes into play for hosts with longer hostnames:

# net join -k -U [user]
Our netbios name can be at most 15 chars long, "FAIRLY-LONG-HOSTNAME" is 20 chars long
Invalid configuration.  Exiting....
Failed to join domain: The format of the specified computer name is invalid.
ADS join did not work, falling back to RPC...
Our netbios name can be at most 15 chars long, "FAIRLY-LONG-HOSTNAME" is 20 chars long

We can’t go changing Microsoft now, so we need to change the registration name of the host we are trying to join to the network. We can do that via the “netbios name” setting in /opt/local/etc/smb.conf:

# echo '        netbios name = shorter-name' >> /opt/local/etc/samba/smb.conf
# net join -k -U [user]
Enter [user]'s password:
Using short domain name -- LRSCORP
Joined 'SHORTER-NAME' to dns domain 'lrscorp.net'
# 

The kind of truncation that you can do on the hostname depends on your environment, in ours we removed dashes to get all our zone hostnames under the limit.

Allow Normal Users to Query Users and Groups

After testing things out as a normal user, you may notice that the user can’t see his own user or group names:

$ whoami
whoami: cannot find name for user ID 23456
$ id
uid=23456 gid=65432 groups=65432,76543
Troubleshooting

Truss is once again a key to figuring out why things aren’t working. Let’s use our truss-and-grep method from part 1:

$ truss whoami 2>&1 >/dev/null | grep winbind
stat("/lib/64/nss_winbind.so.1", 0xFFFFFD7FFFDFF090) Err#2 ENOENT
stat("/usr/lib/64/nss_winbind.so.1", 0xFFFFFD7FFFDFF090) Err#2 ENOENT
stat("/opt/local/lib/nss_winbind.so.1", 0xFFFFFD7FFFDFF090) = 0
resolvepath("/opt/local/lib/nss_winbind.so.1", "/opt/local/lib/libnss_winbind.so", 1023) = 32
open("/opt/local/lib/nss_winbind.so.1", O_RDONLY) = 4
stat("/opt/local/lib//libwinbind-client-samba4.so", 0xFFFFFD7FFFDFEF60) Err#2 ENOENT
stat("/opt/local/gcc49/x86_64-sun-solaris2.11/lib/amd64/libwinbind-client-samba4.so", 0xFFFFFD7FFFDFEF60) Err#2 ENOENT
stat("/opt/local/gcc49/lib/amd64/libwinbind-client-samba4.so", 0xFFFFFD7FFFDFEF60) Err#2 ENOENT
stat("/opt/local/lib/samba/private/libwinbind-client-samba4.so", 0xFFFFFD7FFFDFEF60) Err#13 EACCES [file_dac_search]
[...]
$

The EACCES error seems fairly self-explanatory, let’s check the permissions on that library:

$ ll -d /opt/local/lib/samba/private/libwinbind-client-samba4.so
ls: cannot access /opt/local/lib/samba/private/libwinbind-client-samba4.so: Permission denied
$ ll -d /opt/local/lib/samba/private
drwx------ 3 root root 104 Mar 10 20:47 /opt/local/lib/samba/private/
$

Allow regular users to read and execute the “private” winbind libraries (from the root user):

# chmod 755 /opt/local/lib/samba/private
# chmod 755 /opt/i386/opt/local/lib/samba/private
#

User info is now available:

$ whoami
[user]
$ id
uid=23456([user]) gid=65432([primary-group]) groups=65432([primary-group]),76543([other-group])

Winbind in SmartOS, Part II (Running in Base-64)

We left off last time with a very basic (but working) Winbind deployment. In this post, we will focus on getting that same basic Winbind functionality in a base-64 zone. If you want to try polishing first, you can skip this post for now and use your base-32 zone for part 3.

Starting with a base-64 zone, we can start following the steps listed in part 1 of the series. Returning here when things start to go sideways, and we’ll work out the steps necessary for the new architecture.

Following NSS configuration, the “getent” commands still aren’t picking up on Active Directory users or groups, even after adding the /opt/local/lib path to ld.config.

Troubleshooting

Following the troubleshooting steps from post 1 (disable name-service-cache, grep truss for winbind) doesn’t reveal anything crazy: getent finds and loads our “nss_winbind.so.1” link we created. Running truss without grep, however, reveals something interesting:

# truss getent passwd 2>&1 >/dev/null | tail
stat64("/usr/lib/nss_winbind.so.1", 0x08047110) Err#2 ENOENT
stat64("/opt/local/lib/nss_winbind.so.1", 0x08047110) = 0
resolvepath("/opt/local/lib/nss_winbind.so.1", "/opt/local/lib/libnss_winbind.so", 1023) = 32
open("/opt/local/lib/nss_winbind.so.1", O_RDONLY) = 3
mmapobj(3, MMOBJ_INTERPRET, 0xFED60790, 0x0804717C, 0x00000000) Err#48 ENOTSUP
mmap(0x00000000, 4096, PROT_READ, MAP_PRIVATE, 3, 0) = 0xFED50000
munmap(0xFED50000, 4096)                        = 0
close(3)                                        = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/SUNW_OST_SGS.mo", O_RDONLY) Err#2 ENOENT
_exit(0)

An error is produced when getent tries to load nss_winbind (“mmapobj(3, […]) Err#48 ENOTSUP”). Unfortunately, a quick search for that error doesn’t give us anything useful. Time to roll up our sleeves and read the mmapobj documentation:

Errors

The mmapobj() function will fail if:

[…]

ENOTSUP

The current user data model does not match the fd to be interpreted. For example, a 32-bit process that tried to use mmapobj() to interpret a 64-bit object would return ENOTSUP.

The flags argument contains MMOBJ_INTERPRET and the fd argument is a file whose type can not be interpreted.

The ELF header contains an unaligned e_phentsize value.

Are we running a 32-bit process and trying to interpret a 64-bit object?

# which getent
/usr/bin/getent
# file /usr/bin/getent
/usr/bin/getent:        ELF 32-bit LSB executable 80386 Version 1, dynamically linked, not stripped, no debugging information available
# file /opt/local/lib/libnss_winbind.so
/opt/local/lib/libnss_winbind.so:       ELF 64-bit LSB dynamic lib AMD64 Version 1, dynamically linked, not stripped

Sure enough, getent is 32-bit and our libnss_winbind.so is 64. Old Solaris, and by lineage SmartOS, has a mix of 32 and 64-bit processes and commands in a 64-bit environment. Pkgsrc doesn’t support a mix of 32-bit and 64-bit in packages, so the 32-bit commands are out of luck in a 64-bit Samba package obtained through pkgsrc.

Can we get 32-bit packages?

Long story short, we can! We will use pkg_add, with a few special options to convince it to install 32-bit packages in a separate directory (we chose /opt/i386):

  • Set PKG_PATH environment variable to the value specified in /opt/local/etc/pkg_install.conf, substituting i386 for x86_64. Use this command to see what you should use: “sed -n ‘s/x86_64/i386/p’ /opt/local/etc/pkg_install.conf”.
  • “-m i386”, override the architecture otherwise the process will error.
  • “-P /opt/i386 -I”, install the packages to an alternate location and don’t run scripts, preventing the overwriting of any 64-bit packages.

Install the 32-bit samba package (will also install all 32-bit dependencies):

# sed -n 's/x86_64/i386/p' /opt/local/etc/pkg_install.conf
PKG_PATH=http://pkgsrc.joyent.com/packages/SmartOS/2015Q4/i386/All
# env PKG_PATH=http://pkgsrc.joyent.com/packages/SmartOS/2015Q4/i386/All pkg_add -I -m i386 -P /opt/i386 samba
[...]

Since we are now working with a new directory structure, we need to link libnss_winbind.so again:

# ln -s libnss_winbind.so /opt/i386/opt/local/lib/nss_winbind.so.1
# 
Troubleshooting

Let’s try getent again:

# getent passwd
[no domain users]

Using truss, we see that getent isn’t looking in our new i386 folder. We then need to ensure that 32-bit processes use the 32-bit libraries.

Replace /opt/local/lib with the i386 path in ld.config:

# crle -c /var/ld/ld.config -l /lib:/usr/lib:/opt/i386/opt/local/lib -s /lib/secure:/usr/lib/secure
# 

The base-64 library paths are stored in a different ld.config, so we will add /opt/local/lib there so that the 64-bit processes can find the Winbind-related libraries:

# crle -64 -c /var/ld/64/ld.config -l /opt/local/lib -u
# 
Troubleshooting

Running getent again, we still find no domain users. Back to truss, we discover that it is trying to load /opt/local/lib/samba/private/libwinbind-client-samba4.so, a 64-bit library, but failing with another “ENOTSUP” error.

Add /opt/i386/opt/local/lib/samba/private to ld.config:

# crle -c /var/ld/ld.config -l /opt/i386/opt/local/lib/samba/private -u
# 

Testing again with getent, we now have domain users! Restart the ssh and cron services, and re-enable the name-service-cache service in case you disabled it earlier:

# svcadm restart cron
# svcadm restart ssh
# svcadm enable name-service-cache
#

Now we head back to part 1, continuing with the PAM configuration.

We get to the last step, but SSH access doesn’t work as advertised.

Troubleshooting

Checking in the authentication logs at /var/log/authlog, we see this PAM error message:

2016-02-22T22:10:00+00:00 localhost cron[9433]: [ID 705739 auth.error] open_module[0:/etc/pam.conf]: /opt/local/lib/samba/security/pam_winbind.so failed: ld.so.1: cron: fatal: /opt/local/lib/samba/security/pam_winbind.so: wrong ELF class: ELFCLASS64

So the path we added in pam.conf was for the 64-bit pam_winbind module, let’s update that path to work for both 32 and 64-bit processes requiring PAM.

We need to update pam.conf with a link to the base-32 pam_winbind.so, adding “$ISA” for 64-bit process support:

other auth requisite          pam_authtok_get.so.1
other auth required           pam_dhkeys.so.1
other auth required           pam_unix_cred.so.1
other auth sufficient         /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so use_first_pass
other auth sufficient         pam_unix_auth.so.1

other account sufficient      /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so use_first_pass
other account requisite       pam_roles.so.1
other account required        pam_unix_account.so.1

other session required        pam_unix_session.so.1
other session required        /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so

other password required       pam_dhkeys.so.1
other password sufficient     /opt/i386/opt/local/lib/samba/security/$ISA/pam_winbind.so
other password requisite      pam_authtok_get.so.1
other password requisite      pam_authtok_check.so.1
other password required       pam_authtok_store.so.1

A link to the 64-bit module needs to be inserted into a subdirectory of the 32-bit path, since “$ISA” will be replaced with “64” for 64-bit services utilizing PAM:

# mkdir /opt/i386/opt/local/lib/samba/security/64
# ln -s /opt/local/lib/samba/security/pam_winbind.so /opt/i386/opt/local/lib/samba/security/64/
# 

SSH access via domain credentials is now available in our base-64 zone:

$ ssh jeremy.einfeld@10.88.88.148
Password:
Could not chdir to home directory /home/jeremy.einfeld: No such file or directory
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (base-64-lts 15.4.0)
                   `-'  https://docs.joyent.com/images/smartos/base

-bash-4.1$

Our next step is to polish our Winbind deployment, getting it ready for a real environment in part 3.

Winbind in SmartOS, Part I (the Basics)

Centralized authentication is a terrific tool for anything more than a handful of servers, and Active Directory is often the go-to for authentication within a datacenter.

After testing several centralized-authentications for suitability in our environment, we settled on Winbind both for its features and the issues presented by the other solutions. A future post may detail the investigation and decision, but the biggest reason for us to use Winbind (of several reasons) was the ability to limit SSH access to servers through Active Directory group membership

Note that these steps apply to local SDC/Triton deployments and the Joyent Public Cloud, in addition to stand-alone SmartOS servers, and no changes should be necessary in Active Directory.

TL;DR: see this setup script for a summarization of steps taken in this series, and this repository for configuration files used. Note that some details from part 3 are not covered in the repository, see the readme for a list.

Troubleshooting Steps

You can ignore these if you just want to get Winbind working, but check them out if you want to see why the prior command didn’t work and how the next commands fix it.

WARNING

Don’t try this in base-64 (yet), there are additional difficulties that are encountered that will be handled in part 2. Use base-32 for now. Additionally, image releases prior to 15.2.0 require additional steps that will not be covered in this series so using 15.2.0+ is recommended.

Configuring Kerberos

Winbind uses a variety of protocols to interact with domain accounts, and the primary one that we need to set up is Kerberos. The configuration is managed in /etc/krb5/krb5.conf, and we can use this minimalistic config to get us started:

[libdefaults]
        default_realm = DOMAIN.NET
        dns_lookup_kdc = true
        default_tkt_enctypes = rc4-hmac des-cbc-crc des-cbc-md5

[realms]
        DOMAIN.NET = {
                kpasswd_protocol = SET_CHANGE
        }

Install the mit-krb5 package, then test Kerberos key initialization:

# pkgin -y install mit-krb5
[...]
# kinit [user]
kinit: Configuration file does not specify default realm when parsing name [user]
#

Something isn’t adding up, we did set default_realm in krb5.conf.

Troubleshooting

Let’s use truss to see where kinit is looking for the config file:

# truss kinit [user] 2>&1 >/dev/null | grep krb5.conf
stat("/opt/local/etc/krb5.conf", 0xFFFFFD7FFFDFF390) Err#2 ENOENT
stat("/etc/krb5.conf", 0xFFFFFD7FFFDFF390) Err#2 ENOENT
#

There’s the issue, the default krb5.conf included by SmartOS isn’t where Kerberos is looking.

Link krb5.conf to /etc/:

# ln -s krb5/krb5.conf /etc/

Test Kerberos again:

# kinit [user]
Password for [user]@DOMAIN.NET:
# klist
Default principal: [user]@DOMAIN.NET

Valid starting     Expires            Service principal
02/18/16 18:31:54  02/19/16 04:32:06  krbtgt/DOMAIN.NET@DOMAIN.NET
        renew until 02/19/16 18:31:54
# 

Success! Kerberos is now ready to start working for Winbind.

Configuring Winbind

Our first step for Winbind itself is to install the necessary package via pkgsrc. Winbind is bundled in the Samba package, so that is the one that we will use here:

# pkgin -y install samba

We then need to configure Winbind. The settings are stored in the Samba config file at /opt/local/etc/samba/smb.conf; here is a bare-bones smb.conf for our “domain.net” access:

[global]
        workgroup = DOMAIN
        realm = DOMAIN.NET
        idmap config * : backend = tdb
        idmap config * : range = 1000 - 9999
        idmap config DOMAIN:backend = rid
        idmap config DOMAIN:range = 10000 - 1073751823
        idmap config DOMAIN:schema_mode = rfc2307
        kerberos method = secrets and keytab
        security = ADS
        winbind enum groups = yes
        winbind enum users = yes
        winbind offline logon = yes
        winbind use default domain = yes
        template homedir = /home/%U
        template shell = /usr/bin/bash

We now domain-bind the instance using the Kerberos ticket we created previously with kinit:

# net join -k
Using short domain name -- DOMAIN
Joined '[HOSTNAME]' to dns domain 'domain.net'
#

Now that we have Winbind configured, we can start the daemon with this command:

# winbindd
#

Don’t worry about starting any of the Samba services, they are outside of the scope of Winbind. Test Winbind with these commands:

# wbinfo -u
[list of domain users]
# wbinfu -g
[list of domain groups]
# wbinfo -i [user]
[passwd entry for user, including full name, id, group id, home directory, and shell]
#

Configuring NSS

User and group data is made available in SmartOS through the Name Service Switch (NSS) facility, and now that Winbind is working we can add it as a source for that information. Update the passwd and group settings in /etc/nsswitch.conf, appending “winbind”:

passwd:     files winbind
group:      files winbind

We can then test to see what the system sees as the users, groups, and credentials with these commands:

# getent passwd
[passwd]
# getent group
[groups]

Winbind sees users and groups, but the system isn’t seeing them through NSS.

Troubleshooting

Let’s run getent through truss to see if it is doing anything with the Winbind NSS module:

# truss getent passwd 2>&1 >/dev/null | grep winbind
#

Nothing. A grep for “nsswitch.conf” also returns nothing, so it appears that getent isn’t even using NSS. Looking at the SMF services, there is a name-service-cache service that is keeping user and group data cached (negating the need for getent to check through NSS). Disabling that service, we try getent again:

# svcadm disable name-service-cache
# truss getent passwd 2>&1 >/dev/null | grep winbind
stat64("/lib/nss_winbind.so.1", 0x080471A0)     Err#2 ENOENT
stat64("/usr/lib/nss_winbind.so.1", 0x080471A0) Err#2 ENOENT
#

Now we’re talking. So where is nss_winbind.so.1, if not there? A few searches later, we find that there is no “nss_winbind.so.1”: the Samba package loads a libnss_winbind.so into /opt/local/lib/. We need to link libnss_winbind into a location that is checked for NSS modules:

# ln -s /opt/local/lib/libnss_winbind.so /lib/nss_winbind.so.1
ln: failed to create symbolic link '/lib/nss_winbind.so.1': Read-only file system
# ln -s /opt/local/lib/libnss_winbind.so /usr/lib/nss_winbind.so.1
ln: failed to create symbolic link '/usr/lib/nss_winbind.so.1': Read-only file system
#

That didn’t exactly work, since /usr and /lib are read-only. With that knowledge, we need to change the search locations by modifying (indirectly) ld.conf. We can see the current configuration by running this command:

# crle

Configuration file [version 4]: /var/ld/ld.config
  Platform:     32-bit LSB 80386
  Default Library Path (ELF):   /lib:/usr/lib
  Trusted Directories (ELF):    /lib/secure:/usr/lib/secure

Command line:
  crle -c /var/ld/ld.config -l /lib:/usr/lib -s /lib/secure:/usr/lib/secure

#

It even gives us the command to use to set it to replicate the current settings! Don’t forget to re-enable the name-service-cache service.

Add /opt/local/lib to ld.config, restart the services that need the new library path, and link libnss_winbind.so as nss_winbind.so.1:

# crle -c /var/ld/ld.config -l /opt/local/lib -u
# svcadm restart ssh
# svcadm restart cron
# ln -s libnss_winbind.so /opt/local/lib/nss_winbind.so.1
# svcadm restart name-service-cache
# getent passwd
[passwd including domain users]
# getent group
[groups including domain groups]

Configuring PAM

Our last step for basic Winbind auth is to configure the Pluggable Authentication Module (PAM) to use the pam_winbind module. We need to look for these sections in /etc/pam.conf and add the module into them (order does matter within each section):

WARNING

Adjusting the PAM configuration is a delicate operation: make any mistakes and authentication will no longer work. Always leave a session open when working with pam.conf, since a mistake won’t affect sessions that are already active. There is an additional safety net available in SmartOS: you can edit pam.conf from the SmartOS global zone via /zones/[uuid]/root/etc/pam.conf.

other auth requisite          pam_authtok_get.so.1
other auth required           pam_dhkeys.so.1
other auth required           pam_unix_cred.so.1
other auth sufficient         /opt/local/lib/samba/security/pam_winbind.so use_first_pass
other auth sufficient         pam_unix_auth.so.1

other account sufficient      /opt/local/lib/samba/security/pam_winbind.so use_first_pass
other account requisite       pam_roles.so.1
other account required        pam_unix_account.so.1

other session required        pam_unix_session.so.1
other session required        /opt/local/lib/samba/security/pam_winbind.so

other password required       pam_dhkeys.so.1
other password sufficient     /opt/local/lib/samba/security/pam_winbind.so
other password requisite      pam_authtok_get.so.1
other password requisite      pam_authtok_check.so.1
other password required       pam_authtok_store.so.1

With that configured, we can finally get SSH access via domain credentials:

$ ssh [user]@[host]
Password:
Could not chdir to home directory /home/[user]: No such file or directory
   __        .                   .
 _|  |_      | .-. .  . .-. :--. |-
|_    _|     ;|   ||  |(.-' |  | |
  |__|   `--'  `-' `;-| `-' '  ' `-'
                   /  ; Instance (base-32-lts 15.4.0)
                   `-'  https://docs.joyent.com/images/smartos/base

mail: Cannot open file '/var/mail/' for output
-bash-4.1$

Not perfect, but we have Winbind access! Upcoming posts will detail how to get Winbind working in base-64 (part 2) and polishing it up for a real environment (part 3).

Faithlife’s sdc-portal

Today we’re pleased to announce that our developer / customer facing portal for Joyent‘s SmartDataCenter 7 has been open sourced.

After transitioning from OpenStack to SDC 7 the only thing we were left wanting was a portal for developers and non-operations staff (SDC already has a portal for Admin and Operations engineers). Luckily, SDC has a fantastic set of APIs that we’ve leveraged to create sdc-portal. The portal started out of necessity because our developers were used to having at least the ability to start, stop and reboot their VMs, but it is growing in to much more. Today we want to give back to the open source community that has helped us immensely and invite others to help us make sdc-portal even better.

The portal is in its infancy, and we’ll be iterating on the documentation and feature set rapidly in the next few weeks. However; we’ve been encouraged by Joyent and a few other organizations to make the code available now due to the high demand and the large set of people that are eager to contribute.

Today the portal supports the following features:

  • OAuth sign-in
  • Integration with SmartDataCenter 7 and Joyent’s public cloud
  • Start, stop and reboot VMs
  • Get the current status and information about VMs

In the next few days and weeks we’ll be adding the following features:

  • Generic authentication provider support
  • VM provisioning
  • SSH key management
  • Things the community comes up with…

We welcome your feedback in our sdc-portal group, in the #smartos IRC channel, or in the form of GitHub issues. We also welcome your pull requests!

sdc-portal lab