XenServer Performance and ScalabilityTesting

XenServer Scalability Configuration:

Generally speaking you should go more for a scale out scenario (higher number of VMs with lower CPU configuration) than vice versa in my experience. This is because IRQ handling in XenServer on a guest VM level is restricted to only 1 CPU. Hence if there is a log of IRQs a scaled-up VM could come to its limits. Max out hardware by higher number of VMs with lower number of vCPUs each should get better values.

Performance benchmark reports available by Cisco, HP, Dell, IBM, NetApp, EMC and Fuijts

• Cisco:
• http://www.cisco.com/go/vdi
•  http://www.cisco.com/go/citrix http://www.cisco.com/go/vxi
• http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/VXI/CVD/VXI_CVD_Citrix.html
• http://www.cisco.com/en/US/netsol/ns1099/index.html
• EMC: http://itzikr.wordpress.com/2011/06/08/citrix-xendesktopp-5-on-emc-vnx-match-made-in-heaven-part1/
• NetApp: http://www.netapp.com/us/library/technical-reports/tr-3915.html
• IBM: http://support.citrix.com/servlet/KbServlet/download/27455-102-657846/K-CTX-XD%20on%20IBM-Whitepaper_3.0.pdf
• HP: http://h18013.www1.hp.com/products/servers/virtualization/index.html?jumpid=ex_R2849/us/en/ISS/cclicks/4AA3-4089ENW/discover/clientvirtualization

Whatever the hypervisor XenServer, ESX 4/5, Hyper-V R2SP1, no reports of showing more than 110/120 VMs per hypervisor.

ProjectVRC - www.projectvrc.com has a number of hypervisor scalability reports created using LoginVSI the industry standard to measure user performance and the maximum amount of sessions a host can support with a decent performance.

XenServer Configuration Limits - http://support.citrix.com/article/CTX131047

Concurrent VMs per host	75
Concurrent XenDesktop VMs per host (2940 MB dom0 RAM)	75 – 130
Concurrent PVS VMs per host (2940 MB dom0 RAM)	75 – 130
Concurrent XenDesktop VMs using IntelliCache per host (2940 MB dom0 RAM)	50 – 130
Concurrent protected VMs per host with HA enabled	60

XenServer Performance Tweaking Items:

• Tuning XenServer for Maximum Scalability -  http://blogs.citrix.com/2011/08/08/tuning-xenserver-for-maximum-scalability/
• XenApp VM configuration (4VMs with 6vCPUs) – 180 Sessions http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns955/ns1099/solution_overview_c22-685528.pd
• Install Xentools on OS
• Must be the same Golden image other than XenTools
• Enable Multi-Pathing – http://support.citrix.com/article/CTX118791
• Enable Boot from SAN
• Dont over-provision vCPU for VMs. i.e. Match the Physical Cores to the vCPU for VMs
• Enable Intelicache
• Configure XenDesktops Groups and multiple XenServer Resources Pools – http://support.citrix.com/article/CTX120077
• Enable Optimize for Citrix XenApp on each XenApp VM template
• Configure performance monitoring scripts- http://support.citrix.com/article/CTX124157
• Increase DOM0 memory to 2490 MB – http://support.citrix.com/article/CTX124259
• Increase DOM0 CPU – http://support.citrix.com/article/CTX128922
• Use latest Service Pack and Hotfixes – http://support.citrix.com/product/xens/v5.6sp2/
• Configure Dymanic Memory Control for VMs
• XenServer Single Server Scalability - http://support.citrix.com/article/CTX124086
• Memory size for virtual machine on Xenserver  is Physical Mem – (Dom0 Memory + Xen Memory)
• Update Hardware Firmware
• Bond all NICS
• Xen Network Optimization in Details - http://wiki.xen.org/wiki/Network_Throughput_Guide
• XenServer White Papers
• http://community.citrix.com/display/adi/XenServer+for+XenApp
• http://www.citrix.com/English/ps2/products/documents_onecat.asp?contentid=683148&cid=White+Papers
• XenApp on XenServer Reference Architecture - http://support.citrix.com/article/CTX117922
• Enable XenServer Dynamic Memory Controller or Static Memory Size
• Check IOSTAT and watch for long waits. You might need to configure TCP offloading..
• http://virtualizationandstorage.wordpress.com/2011/06/21/xenserver-configuring-tcp-offload-script/
• http://www.xenappblog.com/2010/citrix-xenserver-slow-network-performance/
• http://support.citrix.com/article/CTX125157
• http://support.citrix.com/article/CTX117491
• Pin CPU for DOM0 - http://support.citrix.com/article/CTX117960
• Distributing Guest Traffic Over Physical CPUs in XenServer - http://support.citrix.com/article/CTX127970 & http://irqbalance.org/ & http://blog.benpiper.com/2011/08/improving-network-throughput-in-xenserver-using-irqbalance/
• The command to install irqbalance is:
• cat /proc/interrupts
• yum install irqbalance –enablerepo base
• service irqbalance star
• Enable NUMA on the hypervisor and add the below options to the /boot/extlinux.confboot file to pin the Dom0 vCPUs to a single NUMA node. (numa=on dom0_vcpus_pin dom0_max_vcpus=4) (Stay with default 4 vCPUs and use the 2940M for Dom0 memory)

Example line for xe-serial

———————————

1. XenServer (Serial)

kernel mboot.c32

append /boot/xen.gz numa=on dom0_vcpus_pin dom0_max_vcpus=4 com1=115200,8n1 console=com1,vga dom0_mem=2940M lowmem_emergency_pool=1M crashkernel=64M@32M — /boot/vmlinuz-2.6-xen root=LABEL=root-nrvrrkli ro console=tty0 xencons=hvc console=hvc0 — /boot/initrd-2.6-xen.img

Virtual Machine Performance Monitoring Tools

• Troubleshooting performance issues in a virtualized Environment – http://support.citrix.com/article/CTX125181
• Disk Bench Marking Tool – http://www.attotech.com/products/product.php?sku=Disk_Benchmark
• Microsoft Performance Advisor – http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=15506
• In order to compile the SPA reports just change to the SPA install dir of each machine (exchange and XA VMs) and execute: spacmd compile “System Overview” afterwards you can analyze the report within SPA by navigating to “reports”->”current”.
• XenTop – Displays real-time information about a Xen system and domains
• iostat
• hdparm
• Top – Provides a dymanic real-time view of a running system
• EdgeSight for LoadTesting - http://www.citrix.com/tv/#videos/3180
• LoginConsultants – http://www.loginconsultants.com/ & http://www.loginvsi.com/
• XenScope
• TaaS
• Performance VM
• SFlow - http://host-sflow.sourceforge.net/about.php
• EdgeSight for Load Testing - http://www.citrix.com/tv/#videos/3180
• Disable C-State - http://support.citrix.com/article/CTX127395
• Configure Switch Port for Xenserver - http://support.citrix.com/article/CTX123158
• Check the Known issues section for the release - http://support.citrix.com/article/CTX130418
• MRTG Tool - http://oss.oetiker.ch/mrtg/

 

XenServer TroubleShooting Tools

• xen-bugtool
• host-bugreport-upload
• host-crashdump-upload
• host-loglevel-set
• tail Logs
• XenTrace - http://support.citrix.com/article/CTX121583
• CTX117386 - http://support.citrix.com/article/CTX117386
• CTX126986 - http://support.citrix.com/article/CTX126986
• Xen Debugging - http://wiki.xen.org/xenwiki/XenDebugging