July 01, 2015

Link to 3PAR Automatic Upgrades

Handy hints from 3PARDUDE

http://3pardude.com/category/upgrade-2/

Upgrade

Automating 3PAR Upgrades

Some of the ground work you have to do when planning a 3PAR upgrade can be quite time consuming, so I have been looking at ways to automate the process and make it quicker. Today’s post focuses on using scripts to speed things up. This will form part of a series on upgrades so don’t forget to connect with me on LinkedIn and Twitter to ensure you don’t miss any posts.

Check multipath policy

As a pre-upgrade step you will need to check that all Windows 2008 hosts using the MS MPIO driver have their multipath policy set to round robin. I used a script to make this step quicker.
The script uses Psexec to connect to a list of hosts and run mpclaim –s –d from the command line to check the current multipath policy. You can download Psexec from here .
The steps to check the host multipathing on multiple hosts via a single script are as follows:
1 Choose the server you want to run the script from and create a folder on it called C:\scripts. Copy psexec to this folder
2 In C:\scripts create a file called 3par_servers.txt Populate this with a list of the servers you wish to check for multipath configuration
3 Also in C:\scripts create a batch file called mpclaim.bat and enter the following command line into it:

mpclaim -s -d

4 Finally, open a command line on the machine from which you wish to run the script, change directory to C:\scripts and then enter

psexec -c -f @C:\scripts\3par_servers.txt C:\scripts\mpclaim.bat

5 You should then see the window populate with the information you require. An example of the output is below:

PsExec v1.71 - Execute processes remotely
Copyright (C) 2001-2006 Mark Russinovich
Sysinternals - www.sysinternals.com
\\Server1
C:\Windows\system32>mpclaim -s -d
For more information about a particular disk, use 'mpclaim -s -d #' where # is the MPIO disk number.
MPIO Disk    System Disk  LB Policy    DSM Name
-------------------------------------------------------------------------------
MPIO Disk4   Disk 5       RR           Microsoft DSM
MPIO Disk3   Disk 4       RR           Microsoft DSM
MPIO Disk2   Disk 3       RR           Microsoft DSM
MPIO Disk1   Disk 2       RR           Microsoft DSM
MPIO Disk0   Disk 1       RR           Microsoft DSM

Check LB policy appears as RR for all volumes.

Compatibility Check

Before you upgrade your 3PAR system you need to check the compatibility of all the components in the SAN, this can be time consuming particularly in a large environment. One component that can be particularly lengthy is gathering information on all the attached hosts. In order to make gathering the host information as easy as possible in conjunction with my colleague Rafal Kozaczek we have taken a PowerShell script (SystemInfo_(Excel).ps1) written by Amol Patil and modified it so that all the host information required to check host compatibility is gathered automatically for Windows servers.
I have included the modified version of the script at the end of this post. As ever you are welcome to use the script, but please check it is suitable for use in your environment and you do so entirely at your own risk.
You will need to run the script from a workstation with PowerShell and Excel installed.
1 Copy the PowerShell script found at the bottom of this post into Notepad and then save as 3PAR_Hosts.ps1 to the location C:\scripts on the machine you wish to run the script from

2 Open PowerShell as an administrator and change directory to c:\scripts

cd c:\scripts

3 Choose to run the script you just saved

3PAR_Hosts.ps1

4 Select option 1 to run the script against a list of hosts in a text file. When prompted enter the host list you created earlier 3par_servers.txt
The resulting spreadsheet will deliver all the information you need to complete the host checks. Columns A – I are the ones you will need, the other info was already pulled by the original script so I just left it in there.

The PowerShell code is in full below. You are free to use it but you do so entirely at your own risk, please also take the time to read through it before you run it so you fully understand what it does.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

#@=============================================

#@ FileName: SystemInfo_(Excel).ps1

#@=============================================

#@ Script Name: SystemInfo_(Excel)

#@ Created: [DATE_DMY] 20/03/2013

#@ Author: Amol Patil

#@ Email: amolsptech@gmail.com

#@ Edited by 3ParDude (Richard Arnold) / Rafal Kozaczek to gather host information for 3PAR upgrade. http://3ParDude.com

#@ Web:

#@ Requirements:

#@ OS:

#@ Keywords:

#@ Version History:

#@=============================================

#@ Purpose:

#@ To collect System Information local/remote

#@

#@=============================================

#@================Code Start===================

# ==============================================================================================

# Function SysInfo - collects information using WMI and places results in Excel Sheet

# ==============================================================================================

Function SysInfo {

foreach ($StrComputer in $colComputers){

$GenItems1 = gwmi Win32_ComputerSystem -Comp $StrComputer

$GenItems2 = gwmi Win32_OperatingSystem -Comp $StrComputer

$SysItems3 = gwmi Win32_WmiSetting -Comp $StrComputer

$MemItems1 = gwmi Win32_PhysicalMemory -Comp $StrComputer

$MemItems2 = gwmi -class MSFC_FCAdapterHBAAttributes -Comp $StrComputer -Namespace "root\WMI"

$MemItems3 = Invoke-Command -ComputerName $StrComputer -ScriptBlock {(Get-Command c:\windows\system32\drivers\mpio.sys).FileVersionInfo}

$NetItems = gwmi Win32_NetworkAdapterConfiguration -Comp $StrComputer  | where{$_.IPEnabled -eq "True" -and $_.DNSHostName -ne $NULL}

# Populate General Sheet(1) with information

foreach ($objItem in $GenItems1){

#$Sheet1.Cells.Item($intRow, 1) = $StrComputer

$Sheet1.Cells.Item($intRow, 1) = $objItem.Name

$Sheet1.Cells.Item($intRow, 15) = $objItem.Domain

$Sheet1.Cells.Item($intRow, 2) = $objItem.Manufacturer

$Sheet1.Cells.Item($intRow, 3) = $objItem.Model

$Sheet1.Cells.Item($intRow, 16) = $objItem.SystemType

$Sheet1.Cells.Item($intRow, 10) = $objItem.TotalPhysicalMemory / 1024 / 1024

}

foreach ($objItem in $GenItems2){

$Sheet1.Cells.Item($intRow, 4) = $objItem.Caption

$Sheet1.Cells.Item($intRow, 11) = $objItem.BuildNumber

$Sheet1.Cells.Item($intRow, 5) = $objItem.CSDVersion

$Sheet1.Cells.Item($intRow, 9) = $objItem.Version

$Sheet1.Cells.Item($intRow, 12) = $objItem.OSArchitecture

}

foreach ($objItem in $NetItems){

$Sheet1.Cells.Item($intRowNet, 13) = $objItem.IPAddress

}

foreach ($objItem in $MemItems2){

$Sheet1.Cells.Item($intRow, 6) = $objItem.Model

$Sheet1.Cells.Item($intRow, 7) = $objItem.DriverVersion

$Sheet1.Cells.Item($intRow, 8) = $objItem.FirmwareVersion

}

foreach ($objItem in $MemItems3){

$Sheet1.Cells.Item($intRow, 14) = $objItem.ProductVersion

}

$intRow = $intRow + 1

$intRowNet = $intRowNet + 1

}

}

#@=============================================

# ========================================================================

# Function Name 'HostList' - Enumerates Computer Names in a text file

# Create a text file and enter the names of each computer. One computer

# name per line. Supply the path to the text file when prompted.

# ========================================================================

Function HostList {

$strText = Read-Host "Enter the path for the text file"

$colComputers = Get-Content $strText

}

# ========================================================================

# Function Name 'Host' - Enumerates Computer from user input

# ========================================================================

Function Host {

$colComputers = Read-Host "Enter Computer Name or IP"

}

# ========================================================================

# Function Name 'LocalHost' - Enumerates Computer from user input

# ========================================================================

Function LocalHost {

$colComputers = $env:computername

}

#@=================================================================================================================================

#Gather info from user input.

Write-Host "********************************"               -ForegroundColor Green

Write-Host "System Information Inventory"                   -ForegroundColor Green

Write-Host "by: Amol Patil "                                -ForegroundColor Green

Write-Host "********************************"               -ForegroundColor Green

Write-Host " "

Write-Host "Which computer resources would you like in the report?"   -ForegroundColor Green

$strResponse = Read-Host "

[1] Computer Names from a File.

[2] Enter a Computer Name manually.

------OR------

[3] Local Computer.

"

If($strResponse -eq "1"){. HostList}

elseif($strResponse -eq "2"){. Host}

elseif($strResponse -eq "3"){. LocalHost}

else{Write-Host "You did not supply a correct response, `

Please run script again." -foregroundColor Red}

Write-Progress -Activity "Getting Inventory" -status "Running..." -id 1

#@=================================================================================================================================

#New Excel Application

$Excel = New-Object -Com Excel.Application

$Excel.visible = $True

# Create 2 worksheets

$Excel = $Excel.Workbooks.Add()

# Assign each worksheet to a variable and

# name the worksheet.

$Sheet1 = $Excel.Worksheets.Item(1)

$Sheet2 = $Excel.WorkSheets.Item(2)

$Sheet1.Name = "SysInfo"

$Sheet2.Name = "Software"

#Create Heading for General Sheet

$Sheet1.Cells.Item(1,1) = "Host Name"

$Sheet1.Cells.Item(1,2) = "Manufacturer"

$Sheet1.Cells.Item(1,3) = "Server Model"

$Sheet1.Cells.Item(1,4) = "Operating_System"

$Sheet1.Cells.Item(1,5) = "ServicePack"

$Sheet1.Cells.Item(1,6) = "HBA-Model"

$Sheet1.Cells.Item(1,7) = "HBA-DriverVersion"

$Sheet1.Cells.Item(1,8) = "HBA-FirmwareVersion"

$Sheet1.Cells.Item(1,9) = "MS MPIO Version"

$Sheet1.Cells.Item(1,10) = "Memory"

$Sheet1.Cells.Item(1,11) = "BuildNumber"

$Sheet1.Cells.Item(1,12) = "OSArchitecture"

$Sheet1.Cells.Item(1,13) = "IPAddress"

$Sheet1.Cells.Item(1,14) = "ProductVersion"

$Sheet1.Cells.Item(1,15) = "Domain"

$Sheet1.Cells.Item(1,16) = "SystemType"

$colSheets = ($Sheet1, $Sheet2)

foreach ($colorItem in $colSheets){

$intRow = 2

$intRowDisk = 2

$intRowSoft = 2

$intRowNet = 2

$WorkBook = $colorItem.UsedRange

$WorkBook.Interior.ColorIndex = 40

$WorkBook.Font.ColorIndex = 11

$WorkBook.Font.Bold = $True

}

If($credResponse -eq "y"){SysInfoCrd}

Else{SysInfo}

#Auto Fit all sheets in the Workbook

foreach ($colorItem in $colSheets){

$WorkBook = $colorItem.UsedRange

$WorkBook.EntireColumn.AutoFit()

#clear

}

Write-Host "@*******************************@" -ForegroundColor Green

Write-Host "The Report has been completed."  -ForeGroundColor Green

Write-Host "@*******************************@" -ForegroundColor Green

#@================Code End=====================

If you have found a way to speed up the process that you use, please let us all know in the comments.

Installing the 3PAR Management Console

The process for installing or upgrading the 3PAR Management Console is the same. Its dead easy but here are the screen shots to walk you through it. Before you begin you will need to download the 3PAR Management Console installation from the software updates and licencing website. You will need to know the software support contract (SAID) for your 3PAR system.
1 Before beginning disable any AV services on the system you are installing to
2 Mount the 3PAR Management Console using ISO Virtual CloneDrive or similar
3 Choose to run setup.exe in my case from inside the windows directory of the ISO
4 Just choose next at the introduction screen

5 Choose the install location and click next

6 A summary of the install appears, just click install

7 A couple of information screens appear during the install, no need for any input. Just wait for the install to complete.

8 That’s it just click done! If you are running 3PAR OS 3.1.3 or above don’t forget to check out the newer management console the 3PAR SSMC.

3 Par OS Upgrade – Part 3 Upgrade Checklist

As I went through the HP 3Par 3.1.3 Inform OS upgrade process I created a checklist of the key steps for quick reference. I have shared it below, but please double check this list against the official HP documentation to ensure all points relevant to your environment and version are covered. Any comments or additions you would like to see on the checklist get in touch through Twitter

A) Pre-Upgrade Checks

1. Hosts

Check compatibility of components. You can complete this step yourself by using SPOCK or send off a completed host worksheet to HP who will verify the compatibility for you.
Apply Windows multipath patches :
- KB275704 needs to be applied to Windows Server 2008, and 2008 R2
- KB2821052 needs to be applied to Windows Server 2012 and Windows Server 2008 R2
Check MPIO is set to round robin on Windows 2008 servers using the native MPIO driver. The process using psexec is covered in this post under point 2.

2. SAN

Load must be below 50% at the time of upgrade. Run statcpu and statport and check utilisation is below 50%
Check current version is on upgrade path. showversion
Check the system is healthy. checkhealth

There are some version specific checks when upgrading from 2.3.1 or 3.1.1 to 3.1.2 check HP 3par OS 3.1.X Upgrade Pre-planning Recommendations for more information

B) Day of upgrade

1. Pre-Upgrade

Load must be below 50% at the time of upgrade. Run statcpu and statport and check utilisation is below 50%
If possible suspended tasks that would cause a heavy load eg backups
Stop AO, DO, and RC tasks before the upgrade. Plus stop anything else that interacts with the system such as scripts

Stop system reporter. Stop windows service – 3Par System Reporter Sampler Service
Showtask – and check for any DO activity
Stop any software that directly interact with the SAN e.g. Recovery Manager, scripts etc.
Stop RC
Check no one is logged in showuserconn
Checkhealth again. checkhealth -svc detail performs a full health check on the system
checkupgrade to verify if the system is ready to undergo an online upgrade.
Check current connectivity status of the hosts showhost -pathsum
All upgrades until after 3.1.3 are performed by HP, so next HP should install the update for you.

2. Post Upgrade

Check version is as expected showversion –a -b

Verify connectivity of hosts showhost –pathsum, plus check monitoring software attached hosts etc. for any potential issues
Restart suspended tasks setsched -resume_all . Check schedules running again showsched
Change Windows 2008 and 2012 to host persona 15 sethost –persona 15 servername
Remove default CPG. removecpg
Restart system reporter windows service (3Par System Reporter Sampler Service) plus any other software you stopped such as Recovery Manager
Resume backups and any other systems you suspended to reduce the load on the SAN
Check for new CLI and management console from FTP supplied by HP or software depot. Check compatibility in SPOCK
Plus if you use any other 3Par software such as Recovery manager VSS provider etc check in the release notes for the latest version and then download via supplied FTP link from HP or from the software depot

Catch the full series of posts in which I ran through the upgrade process in detail: Part 1
Part 2
3ParDude

3Par OS Upgrade 3.1.3 – Part 2 Upgrade Day

Upgrade day was here I packed my lucky rabbit foot and headed into the office. Please note the screen shots below were taken during the last upgraded I posted about as I forgot to take any this time, the process was almost identical but obviously the version numbers you are going to see will differ from those in the screenshots. About an hour before the upgrade was due to begin I completed the following pre-upgrade steps and checks.

Pre-upgrade

Check CPU and port usage is below 50%. statcpu -iter 1, statport -iter 1
Suspend tasks. setsched -suspend_all, check tasks are suspended as expected showsched
Check for any DO activity. showtask –active, for any tasks that are active canceltask taskID
Stop system reporter by visiting the machine its installed on and stopping the Windows service

Check for any connected users who may be making changes to the system showuserconn
Check the connectivity of hosts before the upgrade. showhost –pathsum, I took a screenshot of this so I could verify connectivity was as before after the reboot of the first node
Verify health is OK to do upgradecheckhealth –svc
Check the system is ready for upgrade checkupgrade
Plus I suspended all backups so the system was as quiet as possible

Timings

Updating the Service Processor

The HP engineer first downloaded the update for the Service Processor and Inform OS. Next he disabled alerting in the Service Processor and chose to run the Service Processor update ISO. This stage was completed quite quickly and next he moved onto loading the Inform OS to the Service Processor.

Health Checks

Next was the health checks, again we moved quite quickly through this as I had ran most of them myself before the upgrade. In addition to the checks I ran above he also ran the following commands:
showsys –d, showversion –a –b, showpd –failed –degraded, shownet, showalert, shownode, showcage, showbattery, showport –d

Node Upgrade

The Inform OS update had already been loaded to the Service Processor so the next stage was to stage the new code to the controllers. This was achieved by connecting through SSH to the Service Processor and running a bunch of commands to transfer the files. When the upgrade was kicked off I took a handful of screenshots to show roughly what happens.
Frist the upgrade goes through some pre-upgrade checks

Next the staged software appears to be transferred so it is ready to be actively installed

Next node 0 reboots and picks up the new code

Post upgrade checks

Once both nodes were upgraded the HP engineer then ran the following checks shownode, showversion a – b, and then re-enabled scheduled tasks with setsched -resume_all

Cage and Drive Firmware Upgrade

Next it was time to upgrade the firmware of the cages, this was kicked off with the command starttask upgradecage –a. To check the task was running the following commands were run: showtask –active and then we were able to drill down for more details by running showtask –d taskID. Progress was also monitored by running showcage. In the screenshot below you can see that about half the cages were done at this stage with half on 320f and half on 320c (you can see this in the RevA and RevB column).

Once the cage firmware upgrade is completed it’s time to upgrade the firmware of the disks. Showpd –failed –degraded, those disks that require a firmware upgrade will show as degraded. To kick off the disk firmware upgrade run admithw. Progress can again be monitored though showtask and re-running Showpd –failed –degraded. To do all the disks and cages in our 7 cage system took about 1.5 hours.
Admithw appears to recreate the default CPG’s. I don’t like these to be there in case someone accidentally adds a VV to them so I did a showcpg to double check they contained no VV’s and then removed them with removecpg

Final Tasks

I then ended the remote session with the HP engineer and next set about updating all the attached Windows 2008 and 2012 Servers to host persona 15. Setting the host persona to 15 presents LUN’s in the manner which Windows expects and the good news is this means the application of KB2849097 is no longer part of the upgrade process. To set each Windows host to persona 15 I ran the following command:
sethost –persona 15 servername
Once this was done I set about the following final tasks:

Kicked backups off again
Restarted system reporter service on system reporter service
Checked for new alerts showalert –n
Check the hosts path showhost –pathsum
Ran a checkhealth
Checked all VM’s were online without issues
Checked our monitoring software
Updated software – CLI and management console. This was again downloaded from HP’s FTP site and was a simple case of just clicking next through the install wizard.

That’s was it successfully onto 3.1.3, interestingly once you are on 3.1.3 you can perform your own upgrades without HP having to install the software for you. This new process is explained in this excellent post by Bart Heungens.
Further Reading:
HP 3PAR OS Upgrade Pre-planning Guide
HP 3Par 3.1.3 Release Notes
HP 3PAR Windows Server 2012 and Windows Server 2008 Implementation Guide

If you missed the first part of the series catch it here.

3ParDude

3Par OS Upgrade 3.1.3 – Part 1: Pre-Upgrade Planning

I have been updating one of our 3Par systems to 3Par OS 3.1.3 MU1. This short series of posts will cover the process to successfully upgrade a 3Par system to 3.1.3
There are a number of pre-upgrade tasks that need completing before an upgrade can begin these are listed below under two categories host based tasks and SAN based tasks.

A) Host Tasks

1 Check compatibility of components – This is to ensure that you are running a tested configuration of components that have been proven to work together by HP. There are 2 ways to go about this. Firstly you can use SPOCK. This site contains all the compatibility information you will need to complete your own checks. Or you can complete a host worksheet and return it to HP who will then verify the compatibility of all your components and firmware versions. The components you need to check are fairly standard to any SAN upgrade – Server OS, multipath software, HBA’s and fabric switch firmware versions. I was lucky I didn’t need to update any firmware versions in order to proceed.
2 Check load balancing is set to round robin – This is a requirement for any Windows servers running 2008 and using the native MS MPIO driver. I used a script to make this step quicker.
I used the Microsoft command line application mpclaim to view the multipath configuration. Specifically I ran mpclaim –s –d from the command line.
To run the command line on multiple servers remotely without having to logon I used psexec. You can download it from here . Here is an excellent article on how to use it: psexec guide
In this case I used it in the following way
A Choose the server you want to run the script from and create a folder on it called C:\scripts. Copy psexec to this folder
B in C:\scripts create a file called 3par_servers.txt. Populate this with a list of the servers you wish to check for multipath configuration
C Also in C:\scripts create a batch fie called mpclaim.bat and enter the following command line into it mpclaim –s –d
D Finally, open a command line from the machine you wish to run the script on, change directory to C:\scripts and then enter c:\Scripts>psexec -c -f @C:\scripts\3par_servers.txt C:\scripts\mpclaim.bat
E You should then see the window populate with the information you require. An example of the output is below:
\\Server1
C:\Windows\system32>mpclaim -s -d
For more information about a particular disk, use ‘mpclaim -s -d #’ where # is the MPIO disk number.MPIO Disk   System Disk LB Policy   DSM Name
——————————————————————————-
MPIO Disk5   Disk 6       RR           Microsoft DSM
MPIO Disk4   Disk 5       RR           Microsoft DSM
MPIO Disk3   Disk 4       RR           Microsoft DSM
MPIO Disk2   Disk 3       RR           Microsoft DSM
MPIO Disk1   Disk 2       RR           Microsoft DSM

Check LB policy appears as RR for all volumes.

3 Windows MPIO Patches. Two patches needed applying to each Windows server running MS MPIO we had attached to the 3Par. The patches which are to ensure correct functioning of MS MPIO, in my case we pushed these out using WSUS.

KB275704 needs to be applied to Windows Server 2008, and 2008 R2

KB2821052 needs to be applied to Windows Server 2012 and Windows Server 2008 R2

B) SAN Tasks

The SAN tasks were much quicker than the host checks I just needed to check the following three things

1 Performance – The 3Par needs to be upgraded at a time of low utilisation, this is defined as the load on the CPU’s and ports being below 50% utilised.

Check CPU utilisation. statcpu -t -d 15 -iter 1 Must be less than 50%
Check port utilisation. statport -d 15 -iter 1 -ni

For each port, the average I/O, Kbytes per sec, needs to be less than 50%. The official documentation showed the following figures:
200,000 Kbytes per sec, for 4 Gb connections
400,000 Kbytes per sec, for 8 Gb connections ie less than 50%

2 Version Check – I just ran checkversion from the CLI and then checked in HP 3PAR Operating System Upgrade Pre-Planning Guide that the current version was on the upgrade path and if there were any special considerations .

3 Health Check – The system needs to be in a healthy state to allow an upgrade to go ahead so I ran a checkhealth from the command line and all came back healthy. If you do find any items flagged by the checkhealth you will need fix these items before proceeding with the upgrade.
That was all the pre upgrade tasks done so I went ahead booked in the upgrade with HP as only from releases post 3.1.3 onwards will the systems be customer upgradable.

Follow 3ParDude on Twitter here

HP 3Par Upgrade Checklist

As I went though the HP 3 Par Inform OS upgrade process I created a check list for quick reference of the things that need doing. I have shared it below, but please double check this list against the official HP documentation to ensure all points relevant to your environment and version you are upgrading to are covered.
Pre-Upgrade Checks

Hosts

Check compatibility of components. You can complete this step yourself by using SPOCK or send off a completed host worksheet to HP who will verify the compatibility for you. Items to checks for compatibility are: HBA model, HBA driver, HBA firmware if boot from SAN, MPIO driver, fabric switch firmware version. Depending on the results of the checks upgrade HBA, multipath and switch software as necessary
Check MPIO set to round robin on Windows 2008 servers using the native MPIO driver. Process using psexec covered here under point 2
LUN’s marked as offline following reboot – Apply KB2849097 to all connected Windows 2008 and 2012 hosts to stop this.
There are also some hotfixes related to running ESX hosts VM’s with pass through or Iscsi LUN’s. KB275704 and KB2821052

2. SAN

Load must be below 50% at the time of upgrade. Run statcpu and statport and check utilisation is below 50%
Check current version is on upgrade path. showversion
Check the system is healthy. checkhealth

Day of upgrade

Load must be below 50% at the time of upgrade. Run statcpu and statport and check utilisation is below 50%
If possible suspended tasks that would cause a heavy load eg backups

Stop AO, DO, and RC tasks before the upgrade. Plus stop anything else that interacts with:

Suspend AO and other scheduled tasks setsched -suspend_all
Stop system reporter. Stop windows service – 3Par System Reporter Sampler Service
Showtask – and check for any DO activity
Stop remote copy on both primary and DR site. stoprcopygroup [option] <group_name>,stoprcopy [options].
Stop any software that directly interact with the SAN e.g. Recovery Manager, scripts etc.

Check no one is logged in showuserconn
Checkhealth again. checkhealth -svc detail performs a full health check on the system
checkupgrade to verify if the system is ready to undergo an ONLINE upgrade.
Check current connectivity status of the hosts showhost -pathsum

All upgrades until after 3.1.3 are performed by HP. If you would like to see more details on this step check here Part 3

Post Upgrade

Check version is as expected showversion –a -b

Verify connectivity of hosts showhost –pathsum, plus check monitoring software attached hosts etc. for any potential issues
Restart suspended tasks setsched -resume_all . Check schedules running again showsched
Restart system reporter windows service (3Par System Reporter Sampler Service) plus any other software you stopped such as Recovery Manager
Resume backups and any other systems you suspended to reduce the load on the SAN
Restart Remote Copy
Check for new CLI and management console from FTP supplied by HP or software depot. Check compatibility in SPOCK
Plus if you use any other 3Par software such as Recovery manager VSS provider etc check in the release notes for the latest version and then download via supplied FTP link from HP or from the software depot
Remove default CPG. removecpg

Catch the full series of posts in which I ran through the upgrade process in detail, Part 1, Part 2, Part 3

HP 3Par Upgrade Part 3 – Upgrade Day!

Good times, upgrade completed successfully at the end of last week! We were planning to go to 3.1.2 MU3 but when HP got in touch with me on the day of the upgrade they advised 3.1.2 MU5 was available for our system, the requirements and pre-upgrade checks would be identical. MU5 also contains all previous patches and so there would be no need to run the individual patches we were planning. I asked what happened to MU4 as a number seemed to have been skipped in the sequence and found out that this version had been released specifically for an individual customer.

HP were due to perform the actual upgrade. About an hour before the upgrade was due to begin I completed the following pre-upgrade steps and checks.

Check CPU and port usage is below 50%. statcpu -iter 1, statport -iter 1
Suspend tasks. setsched -suspend_all, check tasks are suspended as expected showsched
Check for any DO activity. showtask –active, for any tasks that are active canceltask taskID
Stop system reporter by visiting the machine its installed on and stopping the Windows service

Check for any connected users who may be making changes to the system showuserconn
Check the connectivity of hosts before the upgrade. showhost –pathsum, I took a screenshot of this so I could verify connectivity was as before after the reboot of the first node
Verify health is OK to do upgradecheckhealth –svc
Check the system is ready for upgrade checkupgrade
Plus I suspended all backups so the system was as quiet as possible

Next it was time to hand over to HP. The high level steps and expected timing was as follows:
Updating New codes on the service Processor – 60 Minutes (non-intrusive, can be performed in advance, VIRTUAL ROOM)
Performing the pre-upgrade checks – 30 Minutes (non-intrusive)
Node Upgrade to the new InForm OS – (15 Minutes per node) + 5 pause time = 40 minutes
Performing Post Upgrade Check and patch installations – 30 Minutes (non-intrusive)
Drive cage and Drive Firmware update – 110 Minutes for 7 cages (will be run as a background task & monitored till completion (non-intrusive)

Updating the Service Processor
I connected with the HP representative, who was very helpful via a virtual room. From HP’s FTP site he downloaded the update for the Service Processor and Inform OS. Next he disabled alerting in the Service Processor and chose to run the Service Processor update ISO. This stage was completed quite quickly and next he moved onto loading the Inform OS to the Service Processor.
Health Checks
Next was the health checks, again we moved quite quickly through this as I had ran most of them myself before the upgrade. In addition to the checks I ran above he also ran the following commands:
showsys –d, showversion –a –b, showpd –failed –degraded, shownet, showalert, shownode, showcage, showbattery, showport –d
Node Upgrade
The Inform OS update had already been loaded to the Service Processor so the next stage was to stage the new code to the controllers. This was achieved by connecting through SSH to the Service Processor and running a bunch of commands to transfer the files. When the upgrade was kicked off I took a handful of screenshots to show roughly what happens.
Frist the upgrade goes through some pre-upgrade checks

Next the staged software appears to be transferred so it is ready to be actively installed

Next node 0 reboots and picks up the new code

There is then a pause between reboots of the nodes during which HP will allow you to check all looks OK. I checked our alerting software, checked all VM’s were still online and ran a showhost –pathsum to check that all paths and accessibility to nodes was OK. Before the last node reboots HP are able to roll the upgrade back in an online manner, once the last node has been upgraded this must be done offline. All looked good in my case so I let the upgrade continue.
Post upgrade checks
Once both nodes were upgraded the HP engineer then ran the following checks Shownode, Showversion a – b, and then re-enabled scheduled tasks with Setsched_resume_all
Cage and Drive Firmware Upgrade
Next it was time to upgrade the firmware of the cages, this was kicked off with the command starttask upgradecage –a. To check the task was running the following commands were run: showtask –active and then we were able to drill down for more details by running showtask –d taskID. Progress was also monitored by running showcage. In the screenshot below you can see that about half the cages were done at this stage with half on 320f and half on 320c (you can see this in the RevA and RevB column).

Once the cage firmware upgrade is completed it’s time to upgrade the firmware of the disks. Showpd –failed –degraded, those disks that require a firmware upgrade will show as degraded. To kick off the disk firmware upgrade run admithw. Progress can again be monitored though showtask and re-running Showpd –failed –degraded. To do all the disks and cages in our 7 cage system took about 1.5 hours.
Admithw appears to recreate the default CPG’s. I don’t like these to be there in case someone accidentally adds a VV to them so I did a showcpg to double check they contained no VV’s and then removed them with removecpg
I then ended the remote session with the HP engineer and took the following final steps

Kicked backups off again
Restarted system reporter service on system reporter service
Checked for new alerts showalert –n
Check the hosts path showhost –pathsum
Ran a checkhealth
Checked all VM’s were online without issues
Checked our monitoring software
Updated software – CLI and management console. This was again downloaded from HP’s FTP site and was a simple case of just clicking next through the install wizard.

That was it all done with zero down time or issues. For my first 3Par upgrade I was very pleased with how it had all run.

Catch parts one and two in this 3 Par upgrade series if you previously missed them

HP 3Par Upgrade Part 2 – Hosts

Over the past few days I have been completing all the pre-upgrade host checks for the 3Par OS upgrade to ensure a successful upgrade. Here are the steps I’ve taken:

1 Check compatibility of components – This is to ensure that you are running a tested configuration of components that have been proven to work together by HP. There are 2 ways to go about this. Firstly you can use SPOCK. This site contains all the compatibility information you will need to complete your own checks. Or you can complete a host worksheet and return it to HP who will then verify the compatibility of all your components and firmware versions. The components you need to check are fairly standard to any SAN upgrade – Server OS, multipath software, HBA’s and fabric switch firmware versions.
In my case as this is the upgrade of our largest datacentre I did both. My checks matched up with HP’s with only once cluster requiring a HBA driver upgrade. This upgrade is done so onto the next stage
2 Check load balancing is set to round robin – This is a requirement for any Windows servers running 2008 and using the native MS MPIO driver. As I have over 40 hosts to check I didn’t want to have to visit this manually. So I managed to get a script to do it, here is how:
I used the Microsoft command line application mpclaim to view the multipath configuration. Specifically I ran mpclaim –s –d from the command line.
To run the command line on multiple servers remotely without having to logon I used psexec. You can download it from here . Here is an excellent article on how to use it: psexec guide
In this case I used it in the following way
A Choose the server you want to run the script from and create a folder on it called C:\scripts. Copy psexec to this folder
B in C:\scripts create a file called 3par_servers.txt. Populate this with a list of the servers you wish to check for multipath configuration
C Also in C:\scripts create a batch fie called mpclaim.bat and enter the following command line into it mpclaim –s –d
D Finally, open a command line from the machine you wish to run the script on, change directory to C:\scripts and then enter c:\Scripts>psexec -c -f @C:\scripts\3par_servers.txt C:\scripts\mpclaim.bat
E You should then see the window populate with the information you require. An example of the output is below:
\\Server1
C:\Windows\system32>mpclaim -s -d
For more information about a particular disk, use ‘mpclaim -s -d #’ where # is the MPIO disk number.MPIO Disk   System Disk LB Policy   DSM Name
——————————————————————————-
MPIO Disk5   Disk 6       RR           Microsoft DSM
MPIO Disk4   Disk 5       RR           Microsoft DSM
MPIO Disk3   Disk 4       RR           Microsoft DSM
MPIO Disk2   Disk 3       RR           Microsoft DSM
MPIO Disk1   Disk 2       RR           Microsoft DSM

Check LB policy appears as RR for all volumes.
3 Preventing LUN’s being marked as offline following reboot – On the first Windows Server 2012 or Windows Server 2008 reboot following an HP 3PAR array firmware upgrade (whether a major upgrade or an MU update within the same release family) the Windows server will mark the HP 3PAR LUNs offline but the data remains intact. To prevent this it is recommended that KB2849097 is applied to all attached Windows 2008/2012 hosts

It is essentially a PowerShell that changes the registry value to 0 for HKLM\System\CurrentControlSet\Enum\SCSI\<device>\<instance>\DeviceParameters\Partmgr. The value is responsible for the state of HP 3PAR LUNs following an array firmware upgrade and a 0 indicates they stay online.

Windows Server 2008/2012 requires the PowerShell execution policy to be changed to RemoteSigned to allow execution of external scripts you can control this through a GPO. Or again amend through a PowerShell command

I got our PowerShell guy to look into if there was a way to the script against all hosts remotely but didn’t have much luck with this. It’s something I will have to look into for future upgrades, but on this occasion I had to log into each host individually and run the script.
So once you have ran the script in KB2849097 you can check its set the registry value as expected value by running: the PowerShell commands

Get-ItemProperty –path “HKLM:\SYSTEM\CurrentControlSet\Enum\SCSI\Disk*Ven_3PARdata*\*\Device Parameters\Partmgr” -Name Attributes

The value returned should then be 0

4 VM’s on ESX running pass through disks. The following wasn’t relevant to our environment but if you are running ESX with raw device mappings check out KB2754704 and KB2821052

That’s it host checks complete! Onto the next stage
If you missed the first part of this series catch it here:
HP 3Par Upgrade Part 1 – Planning

Inform OS Updates – Naming Convention

OS updates come in 3 varieties:
GA – is the general availability first release. Will generally require a reboot of the nodes
MU – are the mandatory updates. Will generally require a reboot of the nodes
P – are the patches, they are smaller fixes generally with no node reboot

Pre Upgrade Clean Up – Stage 2 Patch 25 Consistency Checks

Patch 25 was released earlier in the year and is a critical patch for 7000 series 3Par StoreServ systems running 3 and 4TB drives. The issue may occur when the checkpd diag command executes, which can occur during HP 3PAR OS upgrades, drive cage/PD firmware upgrades, or in some instances, cage replacements. Full details on the patch can be seen here:
Patch 25 release notes
Although we applied the patch soon after it was released we already had data on the 3TB drives. As data existed on the drives before the patch was applied it has been necessary to run some consistency checks on the system. This has to be done by HP who ran a script on the system and then sent the results away to engineering for analysis. The results have come back clean, so on to the next stage……

Search This Blog

Architect's Scratch list

Link to 3PAR Automatic Upgrades

Upgrade

Automating 3PAR Upgrades

Check multipath policy

Compatibility Check

Installing the 3PAR Management Console

3 Par OS Upgrade – Part 3 Upgrade Checklist

A) Pre-Upgrade Checks

1. Hosts

2. SAN

B) Day of upgrade

1. Pre-Upgrade

2. Post Upgrade

3Par OS Upgrade 3.1.3 – Part 2 Upgrade Day

Pre-upgrade

Timings

Updating the Service Processor

Health Checks

Node Upgrade

Post upgrade checks

Cage and Drive Firmware Upgrade

Final Tasks

3Par OS Upgrade 3.1.3 – Part 1: Pre-Upgrade Planning

A) Host Tasks

B) SAN Tasks

HP 3Par Upgrade Checklist

HP 3Par Upgrade Part 3 – Upgrade Day!

HP 3Par Upgrade Part 2 – Hosts

Inform OS Updates – Naming Convention

Pre Upgrade Clean Up – Stage 2 Patch 25 Consistency Checks

Comments

Post a Comment

Popular Posts

link to http://www.techopsguys.com/tag/3par/

Install Sticky Notes to Windows Server 2012