Skip to content

Single ESXi Host VSAN 6.2 Performance Stress Test

November 7, 2016

I recently bootstrapped a single ESXi host with VSAN 6.2 see Install VSAN 6.2 on a Single ESXi Host and wanted to test performance before installing any virtual machines.   My goal was to determine if everything is working – consider this a simple stress test, nothing more.  If you are interested in conducting more comprehensive Virtual SAN testing I would encourage you to read the Virtual SAN Performance Testing series by Wade Holmes on the VMware vSphere Blog.

I began by downloading the latest VMware I/O Analzyer fling – it is downloaded as a .zip file and needs to be extracted to expose the .OVA file.

Note for Mac Users: I was unable to unzip the archive by double-clicking on it or by attempting to open it with the Archive Utility.  Instead I had to use the command line.  I simply typed: unzip /Users/toddsimmons/Downloads/vmware-io-analyzer-1.6.2.zip I also had difficulty using the VMware Web Host Client.  I was unable to deploy the .OVA has a New virtual machine using Google Chrome Version 54, but was successful using Firefox 49.

To deploy the OVA file simply right-click on your ESXi host and select Create/Register VM. Next, select Deploy a virtual machine from an OVF or OVA file.  The remaining property pages are self-explanatory.  Before powering on the vmware-io-analyzer virtual machine I created a larger, 20 GB, Hard disk 2 by right-clicking on the virtual machine and choosing Edit Settings.

screen-shot-2016-11-07-at-1-54-25-am

Once you have finished, power-on the virtual machine.  Note: DHCP is required to lease an IP address.  Once the VM has powered on, open the console and login as root with the password vmware.

screen-shot-2016-11-07-at-1-57-24-amNow open a web browser and connect to the vmware-io-analyzer.  On the home page select Workload Configuration.

screen-shot-2016-11-07-at-1-53-07-am

In the Host list enter the IP Address of your ESXi host in the Hostname field along with the Root Password, then click Add New Host.

In the Add Workload Entry I used the Iometer Test Type and the 4k_100read_100rand.icf Workload

Under the Workload Configuration modify the Duration to 3600 sec (1-hour).  It is important that you set the duration to at least one hour, in the results below you can see that it takes some time to “warm up”.  When finished click the Run Now button.  Here are my results.

screen-shot-2016-11-07-at-12-38-11-am

screen-shot-2016-11-07-at-12-42-28-am

Here is the time series chart of the caching device SAMSUNG SM951 MZ-VPV256HDGL2  M.2 256GB PCIe x4 NVMe.

These are the results from the storage device SAMSUNG PM1635 MZ-ILS3T20 2.5″ 3.2TB SATA III 12G Mixed-use Enterprise Solid State Disk

Install VSAN 6.2 on a single ESXi Host

November 6, 2016

I recently purchased a single HPE DL160 Gen9 1U Server with following:

  • Single Intel Xeon X5 2609v4 CPU
  • 128 GB RAM
  • One 256 GB Samsung NVMe SSD
  • One 3 TB Samsung 12G SAS SSD
  • One 8 GB SanDisk USB Storage Media
  • I have installed the HPE Customized Image for VMware ESXi 6.0 u2 which uses Virtual SAN 6.2

I wanted to create a vsanDatastore on a single host, using a process known as bootstrapping, to host virtual machines for testing and development.

I used the steps that were outlined in the blog article from William Lam Quick Tip – VSAN 6.2 (vSphere 6.0 Update 2) now supports creating all-flash diskgroup using ESXCLI

I connected to my ESXi host via SSH and executed the following commands below (which are specific to my configuration) for more details on the purpose of each command I would encourage you to read William’s blog.

esxcli vsan policy setdefault -c vdisk -p “((\”hostFailuresToTolerate\” i1) (\”forceProvisioning\” i1))”

esxcli vsan policy setdefault -c vmnamespace -p “((\”hostFailuresToTolerate\” i1) (\”forceProvisioning\” i1))”

esxcli vsan network ipv4 add -i vmk0

esxcli vsan cluster new

esxcli vsan storage tag add -d naa.50000f0a0550a980 -t capacityFlash

esxcli vsan storage add -s t10.NVMe____SAMSUNG_MZVPV256HDGL2D000H1______________1031006164382500 -d naa.50000f0a0550a980

 

Install VMware VCSA using Mac OS X

November 6, 2016

There were a few gotchas involved so I wanted to write a quick post.

First I downloaded the VMware Client Integration Plug-in for Mac

After installing the Plug-in it appeared as though it was successful but both Firefox and Chrome browsers will report that it is not installed.  Once I followed the instructions in this KB Client Integration Plugin fails on Mac OSX 10.11.x (2144550) I was able to use the Plug-in.

Next I downloaded the latest VCSA .iso file from my.vmware.com.  The version  I used was VMware-VCSA-all-6.0.0-3634788

I then created a directory on my Desktop named ISO and copied the contents of the VCSA .iso to the new directory.

 

Finally, using a tip from William Lam, I modified the vcsa-setup.html file to check for the Mac OS X instead of Windows.

sed -i .bak ‘s/Win32/MacIntel/g’ vcsa-setup.html

Once finished I double-clicked on the vcsa-setup.html and began the installation.

screen-shot-2016-11-06-at-2-13-41-am

Note: The installation didn’t appear to finish, it was stuck on the “Starting the vSphere Web Client” task for an extended amount of time.  The installation did complete successfully however.

Installing ESXi 6 does not take 5-minutes!

November 6, 2016

If you ask most VMware Administrators how long it takes to install vSphere ESXi 6 the most common response is, “About 5-minutes”.  In fact, I recently had to install ESXi on a rack of 24-servers and I when I informed the customer that the installation of ESXi alone would take approximately 8-hours – they responded “I thought it only took 5-minutes to install ESXi?” Although the installation of ESXi is extremely, fast 5-minutes per server is really stretching the truth.

Here is a breakdown of the actual installation time for a single server:

Summary

Activity Time
Download ISO 1:00
Format USB/SD storage media – Extract ISO 3:42
Install USB/SD storage media into Server 1:00
Boot Server 1:35
Load ESXi 3:02
Complete ESXi Installation Screens 4:00
Shutdown Server 1:00
Reboot ESXi – Host is Ready  with DHCP Lease 4:01
Total Time 19:20

The installation of a single ESXi host actually takes approximately 20-minutes, assuming no mistakes are made. So why does everyone say “About 5-minutes?” my guess is that they are likely referring to the amount of time the spent completing the installation screens.

Note: This time is based on an HPE ProLiant DL160 Gen9 Server with 1 Intel Xeon E502609 v4 CPU and 128 GB of memory.

More Details

Download ISO

Although this seems like a simple task, it is common to download the wrong .ISO.  Many people aren’t aware that vendors such as Dell, HPE, IBM, and others provide a custom image that includes the necessary drivers.  You can locate the .ISO at http://my.vmware.com.  Today I am using a HPE DL160 Gen9 server so I downloaded the HPE Custom Image for VMware ESXi 6.0 U2 Install CD. 

Format USB/SD storage media – Extract ISO

Formatting a new USB/SD is straightforward, it may take a little longer if you have a stubborn partition that you cannot delete.  I use the Disk Utility on my Mac and erased a 8GB SanDisk USB stick using the MS-DOS (FAT) format and the Master Boot Record Scheme.

screen-shot-2016-11-05-at-6-57-33-pm

To Extract the ISO I used UNetbootin which allows you to create a bootable USB Drive.  I simply selected the HPE Custom Image ISO and then clicked OK.

screen-shot-2016-11-05-at-6-59-06-pm

Once the .ISO was extracted I properly ejected the USB media.

Install USB/SD storage media into Server

This was an easy task for me, my server has an internal USB Port, I simply removed the top cover installed the USB media and replaced the top cover.  Finished in under a minute!

Boot Server

My HPE Server takes 1 minute 35 seconds to boot – I have other servers in the office that take as long as 6 minutes to boot.  Server power-on, shutdown and reboot has the largest impact on the amount of time it takes to install ESXi since you will need to initially power on the server and then shut down and reboot after installation.

Load ESXi

The initial load of ESXi from the USB media took just over 3-minutes.

Complete ESXi Installation Screens

This is where you will asked to press F11 to accept the Product License, and to select the media to install ESXi.

Shutdown Server

My HPE Server takes 1-minute to gracefully shutdown at the end of the ESXi Installation.

Reboot ESXi – Host is Ready with DHCP lease

The last stage of the installation takes just over four minutes.  Its important to note that in the future anytime the ESXi host will need to be rebooted it will take a total of 5-minutes to be ready.

 

Running pfsense in a Virtual Machine on a QNAP TS-251

September 5, 2016

Introduction

I was considering consolidating some equipment in my home lab.  I wasn’t happy with the Surveillance Station software offered by QNAP and recently switched to a different NVR solution.  This allowed me to repurpose my QNAP TS-251 for other tasks.  I wanted to determine if the QNAP TS-251 could replace my existing pfsense SG-2220 Security Gateway Appliance.

One of my favorite features of QNAP storage appliances is is the built-in hypervisor – QNAP Virtualization Station which allows you to run virtual machines on your appliance.

Based on the hardware the TS-251 seemed like a perfect candidate for pfsense. Here are the specs for the TS-251:

CPU Intel Celeron J1800 dual-core 2.41 GHz Bay Trail D
RAM 8GB
Ethernet (2) Intel WGI210TA

I decided to create a virtual machine on my QNAP TS-251 to run pfsense, below are the steps I followed.

Configuration

First, you will need to download the latest version of pfsense.  I used 2.3.2-RELEASE (amd64) from pfsense.org/download

Screen Shot 2016-09-05 at 2.24.37 AM

Once I downloaded the .ISO installer I uploaded it to a shared directory on my QNAP TS-251 where I store all of my .ISO files.

Configure Physical Networking – The QNAP TS-251 has two physical network ports.  I connected one of the network ports (Physical Adpater 2) to my Cable Modem and the other (Physical Adapter 1) to my Cisco 3560CG-8PC-S Gigabit PoE Switch.

Configure Virtual Networking – Next I logged into my QNAP TS-251 and opened Virtualization Station 2.2 (Beta).  I clicked on Network Setting in the left menu bar and configured my Network Topology as follows:

Screen Shot 2016-09-05 at 2.35.18 AM

Create a New Custom VM -In the left menu bar I selected Create VM then in the middle pane I chose Create Custom VM.

Screen Shot 2016-09-05 at 2.32.15 AM

Note: I choose the settings above because I wanted to compare the performance to the virtual machine to my pfsense SG-2220 hardware appliance.  The SG-220 has an Intel “Rangeley” Atom C2338 1.7 Ghz CPU with 2GB DDR3L RAM.  Depending on the use case, you may be able reduce CPU and Memory resources.

When creating a New Custom VM you can only select a single Network  and it doesn’t allow you to select the Device Model.  After creating the virtual machine immediately go to Virtual Machine Settings.  Choose Add Device and make sure that this Network Adapter 2 Mode is Virtual Switch 2 and the Device Model is Virtual Gigabit Ethernet (VM Driver Required)

Screen Shot 2016-09-05 at 2.47.43 AM

Install pfsense – Now we are ready to install pfsense.  Attach the .ISO to the virtual machine, then power it on.

Screen Shot 2016-09-05 at 2.53.53 AM

During the installation I chose the Quick Easy Install and used the Std Kernel.  The WAN and LAN adapters are auto-detected – it appears as though it looks for the adapter that has DHCP and auto-assigns that as the WAN port.

Screen Shot 2016-09-05 at 2.57.55 AM

If the adapters are em0 and em1 instead of vtnet0 and vtnet1 then you forgot to change the Device Model to Virtual Gigabit Ethernet.

After the installation completed I attempted to ping the LAN interface 192.168.1.1 from my test machine.

Troubleshooting Tip: If you are unable to ping the LAN port after installing there is a good chance that you switched the interfaces.  Open the console of the VM and choose option 1 to Assign Interfaces properly.

Performance Tests

Testing Equipment:

There were two settings that had a significant impact on performance:

  • Hardware Checksum Offloading
  • Device Model Selection

During my initial testing I found that my download speed was awful 2.72 Mbps, that is not a typo it was 2.72 Mbps after multiple tests.  Checking the box to Disable hardware checksum offload restored my download speed to 255.53 Mbps.

Open a web browser connection to http://192.168.1.1 and logged in as admin with password pfsense.  From the top menu select System –> Advanced –> Network and then select the check box to disable Hardware Checksum Offloading.

Screen Shot 2016-09-05 at 3.08.24 AM

When I originally installed pfsense I used the Device Model Intel Gigabit Ethernet and although it provided a respectable download speed of 84.74 Mbps, switching to the Device Model Virtual Gigabit Ethernet provided the best download speed at 255.33 Mbps.

In addition to conducting several performance tests I was also curious about CPU utilization.  I monitored CPU Utilization during testing and found that utilization rates were relatively high spiking anywhere from 40% up to 60% during most tests.

Conclusion

As a result of the lower throughput, and potential risk to the performance of other applications running on the QNAP TS-251, I decided to keep my pfsense SG-2220 appliance. The SG-220 is silent, has a low power footprint, 6-9 watts during runtime – and provides outstanding performance.

Throughput – The pfsense SG-220 Security Gateway appliance is capable of download speeds of in excess of 350 Mbps consistently in my environment.  Although the pfsense virtual machine peaked at 255.53 Mbps I could not reliably reproduce that download speed, download speeds ranged from 160 – 255 Mbps when using the virtual machine.

Performance – During the download speed tests I found that the CPU Utilization on the QNAP TS-251 spiked to 60%.  I continue to use the QNAP TS-251 for other applications, the increased CPU overhead of the pfsense virtual machine could impact performance of those applications.  Although the SG-2220 also had higher CPU utilization rates during testing it is a dedicated appliance and does not impact the performance of my other applications

 

 

Identifying VMware Software Versions in your SDDC

January 12, 2016

I recently had to conduct a software inventory on a customer’s VMware Software Defined Datacenter (SDDC).  The environment consisted of the following VMware software products:

  • VMware ESXi
  • VMware vCenter
  • VMware NSX
  • VMware vRealize Operations Manager
  • VMware vRealize Log Insight

Below are the steps used to identify the current version of each product in the SDDC.

To determine the ESXi version use the vSphere Web Client
In the Inventory Pane (left) select an ESXi host, then click on the Summary tab (center).  Expand the Configuration.

Screen Shot 2016-01-11 at 2.49.55 PM

Note: Use the build number to determine the version of ESXi you are currently using.  You can find the latest chart here Correlating build numbers and versions of VMware products (1014508)

To determine the vCenter version use the vSphere Web Client
In the Inventory Pane (left) select a vCenter Server, then click on the Summary tab (center).  Expand Version Information.
Note: Don’t use Help —>About to determine the vSphere version.  Using Help —>About in the vSphere Web Client will give you the vSphere Web Client’s version.

Screen Shot 2016-01-11 at 2.51.19 PM

To determine the NSX Version use the vSphere Web Client                                                  From the Home Menu go to Networking and Security.  In the left pane select the NSX Managers link.  The Version is located in the Objects tab.
Screen Shot 2016-01-11 at 3.00.14 PM

For vRealize Operations open a web browser to the default Operations Manager web page then click About in the top menu bar.

Screen Shot 2016-01-11 at 2.53.47 PM

For vRealize Log Insight open a web browser to the default Log Insight web page            In the top right hand corner click on the drop down list then select About.

Screen Shot 2016-01-11 at 2.58.42 PM

 

The NFS server does not support NFS version 3 over TCP – QNAP

December 26, 2015

After recently powering-on an ESXi host I found that all of the NFS Shares hosted on my QNAP TS-231 storage appliance were disconnected.  I first attempted to reconnect by using the vSphere Client option Rescan All in the Configuration –> Storage –> Datastores pane.  When that was unsuccessful I attempted to manually remove and re-add the datastore.  Unfortunately that too was unsuccessful.  I didn’t want to restart my QNAP appliance so I decided to restart the NFS service.  First I enabled SSH on my QNAP appliance.

Screen Shot 2015-12-26 at 10.07.53 AM

Once SSH was enabled I connected to the QNAP appliance from my Mac by using the Terminal application ssh admin@192.168.1.10

The last step was to restart the NFS service.

Screen Shot 2015-12-26 at 10.10.14 AM

Once I successfully restarted the NFS Service, I re-opened the vSphere client and clicked on Rescan All.. once more.  This time access to my shares had been restored.

%d bloggers like this: