Hard drives for the virtualized NAS

My ESXi boots from a 40 GB SSD. This is just to boot ESXi itself, firewall VM, Active Directory VM and NAS VM. NAS VM would in turn organize 3 or 4 2TB drives in a RAID. This space will be served to the users and also additional VMs can be created there.

The large disk drives are under ESXi control initially. There are few ways how to make them available to the NAS VM:

1. Format the drives with the native ESXi filesystem VMF, or, in other words, create individual data stores on them. Create virtual drives on the data stores and pass to the NAS VM
2. Use raw device mapping in virtual mode ( vmkfstools -r). ESXi will be able to do snapshots on the drive. ESXi handles disk requests and accesses the disk on VM behalf.
   
3. Use raw device mapping in physical mode ( vmkfstools -z ). ESXi passes most of the disk commands to the drive. Drive is supposed to have no trace of ESXi on it, and can be later imported on a physical NAS machine.
   
4. If motherboard chipset supports VT-D device pass-through, pass whole SATA adapter (all ports at once) to the VM

I had in mind to implement #4, but the motherboard I bought could not do VT-D pass-through.

So, I started with #3. It worked till I put some real write load on the raid-z. Very soon all disks in the pool reported excessive errors and raid-z became degraded with all drives degraded. Scrubbing resulted in a few files being corrupted. Well, something does not really work with passing SCSI commands down to the SATA drives...

Then I tried #2. No disk errors anymore, and about 300 GB were written over NFS at rate up to 60MB/s . Then Nexenta hanged being out of memory (I could even reproduce this later) with some stack dump on the console and nothing in the /var/adm/messages log. Yuck. Maybe it's me enabling too much debugging?

So I decided to do fresh Nexenta install, but before doing that I revived FreeNAS virtual machine and configured it with 3, and then 4 drive raid-Z. I used #2 variant of the drive configuration.

I must say that Nexenta's GUI is very good, but I felt a relief when I was given FreeNAS GUI instead. I found write speed over NFS to ZFS on FreeNAS not as steady as on Nexenta. It was fluctuating from 20MB/s to 80 MB/s, with probably 40MB/s on average. FreeNAS was consuming both CPUs, but did not want much of memory ( 2GB the most). I feel that FreeNAS has satisfactory performance and is much more convenient than Nexenta. So I decided to give it more extensive testing, and possibly move it into production.