Monday 15 December 2014

Disk performance stats are missing from task manager

By default basic disk IO stats are not available in Windows Task Manager. This counter can be useful for a quick peek at what the disk is doing. For a more detailed look you need to use Resource Monitor.


To enable the Disk statistic in taskmgr, execute this command from an elevated CMD prompt.

diskperf -y

Thursday 25 September 2014

Windows servers keep resolving to a DHCP address

Windows servers with static IP addresses randomly loose their correct DNS IP address and start resolving to some random IP address for the DHCP scope.

This happens with Windows 2012 R2 DHCP and DNS servers. When a new server is connected to the network and powered up, it get's and DHCP address from the local DHCP server. After you fix a static IP address on the new server. It PINGs and DNS resolution works correctly, resolving the new IP address.
Later, on the hour normally. The new server starts to resolve to the old DHCP address. I checked the server to make sure that no Network Adapters have "Obtain and address automatically" but they did not.

Turns out that the 2012 DHCP updates/overwrites the DNS record to match the DHCP scope.

FIX: Find the rogue address within DHCP and delete it. On the new server issue an IPCONFIG /REGISTERDNS

Wednesday 10 September 2014

Configuring Dell PowerVault DR4100 with vRanger 7.1 and RDA (Rapid Data Access)

With the latest DR4100 software, version 3.1 and Dell vRanger 7.1 we can now use RDA or Rapid Data Access as a backup target.

Open the DR4100 web interface and select 'Containers' I have chosen to edit the default "backup" container but you can create a new on if you wish. 


Set the connection type to Rapid Data Access (RDA) as show below.


Next, select 'Clients' from the menu, select the RDA tab and click Edit Password in the top left.If you don't see this available, click the CIFS tab and RDA again. Seems to be an interface bug. Select the client type as RDA and enter a secure password.


Open the vRanger interface, selecting My Repositories, find Dell Rapid Data Access (RDA) at the bottom and select 'Add'

Enter a name, I have used DR4100, enter the IP address of the DR4100 on the LAN, enter the backup_user as the username and enter the password you set in the previous step.

Click OK, vRanger will check access to the DR4100 and close the dialogue box. You have finished and can now use the DR4100 as a backup target when creating a new job. I got over 200 MBps in my test backup with Hyper-V 2012 R2.




Dell PowerVault DR4100 becomes unreachable after update 3.1.0107.0

As with any installation of new hardware we should check the support website to make sure we have the latest firmware and software available.

After logging into the Dell support website there was a new update available for the DR4100 device I was installing. The patch was downloaded and the system was upgraded. The screen warns this can take a long time. After the update I rebooted the unit, unfortunately the unit was then unreachable from the LAN, I was able to gain access via the iDRAC interface.

Looking though the logs I could see that while the software had updated the firmware packages built in to the update bundle had not. This can happen if the iDRAC has started up too quickly.

The solution is to open the iDRAC console, login to the DR4100 with the 'administrator' user account and password and issue this command.

 maintenance --remote_access --racreset

After the iDRAC reboots, login to the console again and start the firmware manually.

system --upgrade

Enter the 'administrators' password and wait for the firmware to complete. After approximately 30 minutes the update should complete and ask for the restart.

system --reboot

Once the reboot has completed you should be able to access the web interface.

Tuesday 9 September 2014

Opening the agent port on Hyper-V hosts form Dell vRanger software

This PowerShell command needs to be executed from one of the cluster nodes. It will find all the nodes in the cluster, and create a firewall rule to allow TCP port 8081 inbound.


get-clusternode | ForEach-Object  {Invoke-command -computername $_.name -scriptblock {netsh advfirewall firewall add rule name="Dell vRanger remote agent" dir=in action=allow protocol=TCP localport=8081}}

Thursday 4 September 2014

Finding Dell firmware versions from PowerShell

Get-WmiObject -Namespace root\CIMV2\Dell -Class Dell_CMApplication -Filter "componentType = 'FRMW'" -ComputerName "localhost" | Select name,version

Thursday 19 June 2014

Data recovery from a damaged VHDX disk.

Had a situation with a VM that turned corrupt after been unfrozen. My feeling is this was the dreaded ODX issue.

The VM would boot and BSOD with Unmountable_Boot_Volume. Tried to mount the VHDX on the Hyper-V hosts and run a chkdsk /r but this wouldn't repair.

Veeam came to the rescue to recover the VM but the data was 36 hrs out of date.

We managed to recover the files by using this software.

http://www.eassos.com/partitionguru/featuredetails.php

Highly recommended.

Saturday 7 June 2014

Add-NetLbfoTeamMember : Validation failed and changes to the system are rolled back.

Had a situation where a NIC in a Dell Server would only negotiate 100Mbps. Turns out it was a dodgy network cable, but in the process I reapplied the Broadcom firmware on the NIC.

I had removed the NICfrom the LBFO team as part of the troubleshooting. After we found it was the cable I needed to add the NIC back to the NIC team.

When I tried I kept getting this error below:


Turns out that the firmware reflash had reset all the NIC advanced settings. Including the Jumbo frame configuration.

Set-NetAdapterAdvancedProperty -Name NIC1 -RegistryKeyword “*JumboPacket” -Registryvalue 9000

After I reconfigured the Jumbo Frame on the NIC it worked!


Friday 16 May 2014

Migrating VMs from Hyper-V 2008 R2 to 2012 R2 with VHDX using Dell AppAssure

I have been working on a project where the customer needs to migrate virtual machines from an existing Windows 2008 R2 Hyper-V cluster with VHD format disks to a new Windows 2012 R2 Hyper-V cluster. The new hardware employs Dell PowerEdge R620 servers.

It is well documented that the most favourable way to achieve this goal is by stopping the VMMS on the source 2008 R2 server, copy all the Virtual Machines files over including the virtual disk and configuration and import them into 2012 R2. 

http://technet.microsoft.com/en-us/library/jj574226.aspx

The problem with this is the amount of downtime required, as all the VMs on the source host must be powered off and copied in one hit. This could be for example 20 VMs and 4 TBs of data. That's fine if the business can accept this, but most won't!

Another solution is to use Dell AppAssure to perform a bare metal recovery to the new 2012 R2 host. This also allows you to end up with VMs using the new VHDX format even when the source VM was VHD! This is presuming you already have AppAssure as a backup solution, in this case the customer does.

The URL below is a link to the AppAssure KB article with the helicopter view of the operation. 
Performing a Bare Metal Restore for a Windows Machine

Here are the steps you will need to take:

Step 1: Create a new VM on the Windows 2012 R2 host. This is your opportunity to re-jig the virtual hardware. At this stage you can create new VHDX hard drives. These need to be the same size as the source or you can take this opportunity to increase the disk size.

Step 2: Take a final AppAssure snapshot of the VM running on the 2008 R2 host. To do this locate the VM in the AppAssure web GUI. Select the all the disks, as below. Then click 'Force Snapshot.
The job starts and you can monitor this via the Events tab in AppAssure.
After the snapshot has completed you need shut down the source VM ASAP, so that the snapshot doesn't become too dated.

Step 3: Record the VMs IP address information. Shut down the source VM using Hyper-V manager. Make sure you notify user first.




Step 4: Boot the destination VM, making sure you have attached the AppAssure recover ISO, which is detailed in the AppAssure KB above. After the AppAssure recovery environment has booted, note the IP address and Authentication Key on screen.
One of the great features of the recovery CD is that it already has the Hyper-V integration tools integrated to you don't need to fiddle about integrating synthetic drivers into you ISO!


Step 5: Using the AppAssure web GUI find the VM in the list and select 'Recovery Points' find the latest recovery point from the list. Select it and click 'Restore'


Step 6: Chose the destination as detailed below. Tick to confirm you have already booted from the recovery media ISO. Enter the IP address and Activation Code you noted down in step 4 and click next.
Step 7: Disk Mapping, Automatic volume mapping seem to do a good job of matching the source to the destination VHDX. Confirm and click next.

Click finish.

Step 8: Monitoring the recovery. In the AppAssure GUI select 'Events'. You can see that AppAssure is performing a roll-back of the snapshot data to the destination VM.

Step 9: After the recover has finished you can disconnect the AppAssure recovery media. At this point edit the setting of the destination VM and detach the network. This is so when the VM boots it doesn't get a DHCP address and screw up the DNS record. Remember this is a new VM with old disks so it will have a new NIC.

Step 10: Reboot the VM, once it has booted you will need to login with cached credentials and update the integration services. Set the IP address details and reboot. At this point you can edit the VM setting and re-attach the VM Network.

Step and repeat, jobs a good'un!!

Congratulations to the Dell AppAssure team, they really have made this a simple and efficient way to perform a migration! Please also note you could use this technique for moving from VMware to Hyper-V or vice-versa.

Notes: You need to disable or make sure Windows defrag is not active during the AppAssure snapshot, if not the VM won't boot after the restore. Also we saw mixed speeds from 2MB/s to 50MB/s. After the destination VM has booted for the first time you will see the Windows boot menu and abnormal shut down popup.
We managed to squeeze some extra performance out the of the AppAssure server by disabling C and C1e states in the BIOS, and configuring Windows PowerCfg of 'High Performance'

Thanks to James Allen for your assistance.

Thursday 24 April 2014

Dell MD Storage Array VMware vCenter Manager plug-in fails to load in vCenter

The Dell MD Storage Array VMware vCenter Manager plug-in is a great addition to vCenter for ease of management for MD3 series arrays.

After reading the release notes it states that if you already have VMware Update Manager installed in the same server you must change the Jetty SSL port from 8084 which is being used by VUM. I did as I was told and changed the port to 18084.

After opening vCenter and trying to enable the new plug-in you are faced with this error.


I added some firewall exceptions, but this did not help. I checked to make sure the Dell MD Storage Array VMware vCenter Application Service was started and it was. 

Did some digging around in the jetty folder.C:\Program Files\Dell\MD Storage Array VMware vCenter Plug-In\jetty and found that although I have specified port 8088 during the setup the XML file, wsconfig.xml had the wrong port!!!


The fix was to stop the Dell MD Storage Array VMware vCenter Application Service service, correct the port number in my case 8088 and start the service once more. Sorted!

While you in that folder you might want to edit appserver64.ini and reduce the amount of RAM java will drink. Unless you have a large amount of arrays to monitor reduce this to 128MB's 

#localize these lines  (Items to the right have the =)
service.name=Dell MD Storage Array VMware vCenter Application Service
service.description=Dell MD Storage Array VMware vCenter Application Service

#Do not localize below this point
service.class=com.loem.aa.webserver.WebServerService
service.id=DellMDStorageArrayApplicationServicevCP

classpath.1=lib\*.jar
service.startup=auto
#log=c:\logs.txt
vmarg.1=-Dconfig.file=C:\Program Files\Dell\MD Storage Array VMware vCenter Plug-In\jetty\wsconfig.xml
vmarg.2=-Dorg.apache.jasper.compiler.disablejsr199=true
vmarg.3=-Xmx128M
#vmarg.4=-Xdebug
#vmarg.5=-Xnoagent
#vmarg.6=-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=y
vm.location=C:\Program Files\Dell\MD Storage Array VMware vCenter Plug-In\jre\bin\server\jvm.dll

Note if the service won't start you might need to remove and reinstall using another port number. You can use netstat -an | find ":portnumber"  to see if a port is in use.

Tuesday 1 April 2014

Upgrading FTOS from USB

I have had some trouble lately upgrading FTOS via TFTP, the session starts but throws some errors. An easier way to upgrade the firmware or FTOS software is to use a USB stick:

FTOS needs to see a FAT32 formatted USB stick. I have seen plenty of times where the stick inserted with FAT32 isn't recognised with a file system. If this happens just get the switch to format the USB stick with this command:

format USBFLASH:

Once the stick is formatted copy the FTOS .bin file to the stick from a laptop and insert it back into the switch chassis.

Issue the command:

upgrade system usbflash: A:



After the file has copied to flash you issue the command reload to finish the process.

The process can take about 5 minutes. Please note in this example, I'm upgrading from version 8.x.x.x to 9.3.0.0 on a S4820T, in this case on reboot each physical port had to have a firmware upgrade from version 01.50 to 01.66 which took approximatly 10 minutes. Make sure you plan this into your downtime plan!

You can verify the firmware upgrade by issuing the show os-version command


Thursday 30 January 2014

EqualLogic, gathering diags

You speed up the EqualLogic diag by skipping the SNMP walk in section 14 by using these switches.

diag "-x 14"

VSS event ID 8194 after installing HIT 4.6

I have seen this with a few separate customers now. Event ID Error 8194 appears on Hyper-V 2012 hosts with the HIT installed as seen below.


Volume Shadow Copy Service error: Unexpected error querying for the IVssWriterCallback interface. hr = 0x80070005, Access is denied.

. This is often caused by incorrect security settings in either the writer or requestor process.

Operation:

Gathering Writer Data

Context:

Writer Class Id: {e8132975-6f93-4464-a53e-1050253ae220}

Writer Name: System Writer

Writer Instance ID: {c4cafe6b-9e92-49e6-b95e-35b7cd4c0cac}



The forum below intimates this has been seen before and points to the HIT.


http://social.technet.microsoft.com/Forums/en-US/c66d3ab4-44d7-4a24-bb4b-9b20b3ed56d1/backup-of-hyperv-2012-csv-intermittent-fails-with-error-0x80042301?forum=dpmhypervbackup

It would appear a working solution is to configure removing access to the network service account, what it actually does is to stop the call to VSS, hence no access denied error is reported. The double \\ is the output from the registry export and is required.


Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\VssAccessControl]
"NT Authority\\NetworkService"=dword:00000000


Didier Van Hoye, has some useful information on this...
workinghardinit.wordpress.com/.../dell-compellent-hardware-vss-provider-commvault-on-windows-server-2012-hyper-v-nodes-volume-shadow-copy-service-error-unexpected-error-querying-for-the-ivsswritercallback-interface-hr-0x80