ESXi iSCSI initiator and data store

At this point I have an ESXi hypervisor, which is hosting a few important virtual machines from a SSD drive. That is, a Windows 2003 AD server, and Solaris 11 NAS server. NAS server has 4 large drives passed through to it. Solaris 11 organizes these 4 drives into RAID-Z (Stripped, 1 parity disk) ZFS pool. From this pool, NAS server caters to the windows and UNIX clients on the network. I also created an iSCSI target ( or LUN) in the pool, of 500GB.

Now I want to make this LUN available on the ESXi hypervisor itself. ESXi will create a data store on the LUN, where other virtual machines will be stored. Those won't be the machines that are always on, like AD server or NAS server. They will be my lab machines.

In the future I will be able to 'mount' this LUN from another ESXi host, should I need more resources to run these machines. But for now, all iSCSI network traffic will be on the virtual network inside the ESXi host. I want to make the setup simple and inexpensive, so there won't be hardware iSCSI adapters or dedicated network cards or any security between iSCSI initiator (ESXi) and target ( Solaris NAS )

This makes any ESXi networking configuration unnecessary - I will just use the  management network already configured at the installation time.

1. Log in to the vSphere Client, and select a server from the inventory panel.
2. Click the Configuration tab and click Storage Adapters in the Hardware panel.The list of available storage adapters appears.
3. Select the iSCSI initiator to configure and click Properties.
4. Click Configure.
5. To enable the initiator, select Enabled and click OK.

Add static target. I use the IP address of the Solaris 11 NAS virtual machine and the target ID created for me (see the previous post). I use no authentication.

ESXi will offer a rescan, and new LUN shows up

Create new data store on the iSCSI LUN

iSCSI target

Install and enable COMSTAR

# pkg install storage-server

# svcadm enable stmf

Create volume for the target. Size is required.

# zfs create -V 500g tank/iscsi1

Create LUN 

# stmfadm create-lu /dev/zvol/rdsk/tank/iscsi1 

# stmfadm list-lu 
LU Name: 600144F0544C880000004CFF6DFE0001

Publish LUN to all hosts (to make our life easier for now)

# stmfadm add-view 600144F0544C880000004CFF6DFE0001

# stmfadm list-view -l 600144F0544C880000004CFF6DFE0001
View Entry: 0
    Host group   : All
    Target group : All
    LUN          : 0 

Start the service  (ignore the warning)

# svcadm enable -r svc:/network/iscsi/target:default
svcadm: svc:/milestone/network depends on svc:/network/physical, which has multiple instances.

# svcs -l iscsi/target (to test)

Create the target
# itadm create-target
Target iqn.1986-03.com.sun:02:08739864-69d3-cb63-c4be-c1b2b03d83b3 successfully created


# itadm list-target -v (to test)

Creating shares

I already had several shares created for me by Nexenta GUI when I played with it. I had them imported automatically to Solaris machine with their disk pool 'tank' using command

#zpool import -f tank

Now I'm creating an additional file system named 'work' on pool 'tank' manually.

# zfs create -o casesensitivity=mixed -o nbmand=on tank/work

Check if it worked:

#df

...

tank/work 4332676508 44 4332676465 1% /volumes/tank/work

Share it to the Windows clients

# zfs set sharesmb=on tank/work

I could see share 'tank_work' from my Windows machine. But I liked name 'Work' better.

# zfs set sharesmb=name=Work tank/work

Now, it would be nice to write something to this share. One way to grant myself all rights is the old chown command:

# chown sergey /volumes/tank/work/

At this point I could see share Work on my Solaris NAS server and I could create files and folders under it.

Configuring Solaris for NAS filer role

 Installation and Networking
Install Solaris 11 Express from text-mode CD. Leave the network unconfigured.
Login from the console and su - to root.
Set up the static IP addres:

#svcadm enable svc:/network/physical:default

#ipadm create-addr -T static -a 192.168.2.225 e1000g0/v4static

Set default gateway

# echo 192.168.2.222 > /etc/defaultrouter 

enable DNS name resolution

# cp /etc/nsswitch.dns /etc/nsswitch.conf

Set up DNS server and domain:
# cat /etc/resolv.conf
domain  smel.local
nameserver  192.168.2.221

If non-standard subnet partitioning is used, edit /etc/netmasks
Set up time synchronization from the AD server - time must be in sync for AD integration to work.
Create /etc/inet/ntp.conf file:

 
server 192.168.2.221 iburst
driftfile /var/ntp/ntp.drift
statsdir /var/ntp/ntpstats/
filegen peerstats file peerstats type day enable
filegen loopstats file loopstats type day enable

#svcadm enable svc:/network/ntp

#reboot

Setting up CIFS

User mappings
# idmap add 'winuser:*@smel.local' 'unixuser:*'
# idmap add 'wingroup:*@smel.local' 'unixgroup:*'

Create Solaris users with same names as AD users.I reuse original UIDs from my Linux server. Otherwise, specifying UID with -u argument would be extra.

# useradd -u 500  -d /volumes/tank/home/sergey -m sergey
...

Let Solaris know where WINS server is
# sharectl set -p wins_server_1=192.168.2.221 smb

Join Solaris CIFS into AD domain:

Set up Kerberos in /etc/krb/krb.conf file:

[libdefaults]
default_realm = SMEL.LOCAL
[realms]
SMEL.LOCAL = {
kpasswd_protocol = SET_CHANGE
kpasswd_server = ws2003.smel.local
admin_server = ws2003.smel.local
kdc = ws2003.smel.local
}
[domain_realm]
.smel.local = SMEL.LOCAL
smel.local = SMEL.LOCAL
[logging]
default = FILE:/var/krb5/kdc.log
kdc = FILE:/var/krb5/kdc.log
kdc_rotate = {
period = 1d
versions = 10
}
[appdefaults]
kinit = {
renewable = true
forwardable= true
}

Start CIFS service (does not hurt even if already started) 

#svcadm enable -r smb/server 

As long as network is working, ignore this message, if encountered:
svcadm: svc:/milestone/network depends on svc:/network/physical, which has multiple instances.

Finally, join the domain
# smbadm join -u Administrator SMEL.LOCAL
After joining SMEL.LOCAL the smb service will be restarted automatically.
Would you like to continue? [no]: yes
Enter domain password:
Joining SMEL.LOCAL ... this may take a minute ...
Successfully joined SMEL.LOCAL

At this point I could browse and modify the shares I have imported with the ZFS pool.
In case of fresh install, shares need to be configured now.

Oracle Solaris 11 Express

Yesterday I was making 'finishing touches' with Nexenta GUI.

First, I tried to transfer user accounts from my current Linux server together with theirs UID and GUID. The UID Linux used were in the 500-510 range. The UID Nexenta uses by default are starting with 1000. I tried to manually change UID in /etc/passwd and /etc/shadow from 1000 to 500. This has screwed things up so much, that nexenta could not boot. It was sitting with a message 'reading ZFS configuration' or such.

I also tried to rename the mis-spelled volume using Nexenta GUI. At one point I have realized that I'm about to delete all 1.5TB of data I was copying over last 2 days. I did not want that and I clicked 'Cancel' button. Data is gone anyway.

Thinking 'enough is enough', I have booted up commercial Solaris 11 Express in another virtual machine, took virtual disks from Nexenta, assigned them to Solaris 11 and imported the pool.
Wow! I must be out of Solaris world for too long. Unexpectedly to me, not just data and folders were there, they were already exported for me via NFS!

Copying data over again.

3rd hang of Nexenta in a row

I did a fresh install of NexentaStor, configured RAID-Z of 4 2TB drives and managed to copy almost all my archive of 1.5 TB over NFS. Now Nexenta hangs again.
Nexenta was about twice as fast as FreeNAS with same ZFS configuraiton. I was writing at 54 MB/s sustained.

Interesting.
root@nexenta:/# ls -l core
-rw------- 1 root root 3089329786 Dec 3 20:45 core
root@nexenta:/# file core
core: ELF 64-bit LSB core file AMD x86-64, version 1 (SYSV), SVR4-style, from '8*'

A few such cores, and root will be full.

root@nexenta:/# strings core |less
X"'7
CORE
zdb -D tank
CORE
i86pc

I guess this is a core from zdb, which man page says not to run unless asked by support. Nexenta runs it for reporting, apparently.

root 3900 766 0 22:00:34 ? 0:02 /usr/bin/perl /var/lib/nza/triggers/volume-check
root 4596 3900 0 22:13:31 ? 0:23 zdb -D tank

Let's disable this coring for now

root@nexenta:/# coreadm
global core file pattern:
global core file content: default
init core file pattern: core
init core file content: default
global core dumps: disabled
per-process core dumps: enabled
global setid core dumps: disabled
per-process setid core dumps: disabled
global core dump logging: disabled
root@nexenta:/# coreadm -d process

Hard drives for the virtualized NAS

My ESXi boots from a 40 GB SSD. This is just to boot ESXi itself, firewall VM, Active Directory VM and NAS VM. NAS VM would in turn organize 3 or 4 2TB drives in a RAID. This space will be served to the users and also additional VMs can be created there.

The large disk drives are under ESXi control initially. There are few ways how to make them available to the NAS VM:

1. Format the drives with the native ESXi filesystem VMF, or, in other words, create individual data stores on them. Create virtual drives on the data stores and pass to the NAS VM
2. Use raw device mapping in virtual mode ( vmkfstools -r). ESXi will be able to do snapshots on the drive. ESXi handles disk requests and accesses the disk on VM behalf.
   
3. Use raw device mapping in physical mode ( vmkfstools -z ). ESXi passes most of the disk commands to the drive. Drive is supposed to have no trace of ESXi on it, and can be later imported on a physical NAS machine.
   
4. If motherboard chipset supports VT-D device pass-through, pass whole SATA adapter (all ports at once) to the VM

I had in mind to implement #4, but the motherboard I bought could not do VT-D pass-through.

So, I started with #3. It worked till I put some real write load on the raid-z. Very soon all disks in the pool reported excessive errors and raid-z became degraded with all drives degraded. Scrubbing resulted in a few files being corrupted. Well, something does not really work with passing SCSI commands down to the SATA drives...

Then I tried #2. No disk errors anymore, and about 300 GB were written over NFS at rate up to 60MB/s . Then Nexenta hanged being out of memory (I could even reproduce this later) with some stack dump on the console and nothing in the /var/adm/messages log. Yuck. Maybe it's me enabling too much debugging?

So I decided to do fresh Nexenta install, but before doing that I revived FreeNAS virtual machine and configured it with 3, and then 4 drive raid-Z. I used #2 variant of the drive configuration.

I must say that Nexenta's GUI is very good, but I felt a relief when I was given FreeNAS GUI instead. I found write speed over NFS to ZFS on FreeNAS not as steady as on Nexenta. It was fluctuating from 20MB/s to 80 MB/s, with probably 40MB/s on average. FreeNAS was consuming both CPUs, but did not want much of memory ( 2GB the most). I feel that FreeNAS has satisfactory performance and is much more convenient than Nexenta. So I decided to give it more extensive testing, and possibly move it into production.

Free NAS products and Active Directory

Round 1 Integrating Samba 4 Active Directory

• Configured Samba 4 AD, so Windows 7 computer could join domain with no issues.
• FreeNAS - joined domain, can list the users. Users are not allowed in, though. Probably no magic to be expected - need to create local users and map them to AD users? Seems not able to create or save local users too. Tried to troubleshoot. Moved on.
• OpenFiler. Could not join Samba 4 domain. Could not add hard drive previously used by FreeNAS. User interface ran me in cycles. Shocked to see that the documentation had to be purchased. The web site is kind of empty. Moved on
• NexentaStore community edition. Does not use regular Samba. Had real trouble joining Samba4 domain. GUI has said it successfully joined it but I could not see the users. Invested lot of time troubleshooting, following all of the books on OpenSolaris/CIFS and Nexenta. Gave up.
  

Round 2. Integrating with Windows 2003 Active Directory

• Windows 7 client has no issues joining 2003 AD
  
• NexentaStore joined domain seamlessly. I have created local users and mapped them to the AD users. Web GUI wasn't clear on how to set individual permissions for the user's home directories, so I went to the command line and used the old good chmod. This part works now.
• FreeNAS joined domain, but dynamic DNS update did not happen. I have updated DNS manually. There was no need (or means) to map AD users to UNIX users. AD users were able to see the shares (as soon as I cared to synchronize time between AD server and FreeNAS server)