NetAppSnapMirrorAuditLogParser.ps1
|
<# .NOTES File Name: NetAppSnapMirrorAuditLogParser.ps1 .COMPONENT -NetApp PowerShell Toolkit: https://www.powershellgallery.com/packages/NetApp.ONTAP -ImportExcel module: https://www.powershellgallery.com/packages/ImportExcel (if exporting to Excel) .SYNOPSIS This script parses Clustered Data ONTAP SnapMirror audit logs that are either retrieved direct from the cluster or from a downloaded copy of a log (i.e. from AutoSupport). Version 1.6 Changed the script file name for publishing Moved to using current NetApp.ONTAP module Simplified connection to cluster Version 1.5 Added network compression ratio and snapmirror detail as additional output fields Version 1.4 Changed the way of retrieving log files from a cluster to better sort by order of files written Version 1.3: Added a "concurrency" field to show how many other SnapMirror transfers were transferring during any part of the time that particular SnapMirror transfer was transferring Version 1.2: Changed throughput to calculate at "start to end" instead of "request to end" Version 1.1: Added row numbers Changed data organization to objects Gave option to export to Excel format Changed time stamp to full date/time stamp to correct calculations Version: 1.0 - Original release .DESCRIPTION The SnapMirror audit log provides a running log of SnapMirror relationships. It shows starting, restarting, and ending information for each SnapMirror relationship. This script reads in the log file(s) and parses the information into a usable CSV file. The output produces the following columns in the file: -Source (the source of the SnapMirror relationship) -Destination (the destination of the SnapMirror relationship -Type (request, start, or end) -KB Transferred (produced only for end type entries) -Time Stamp (the time stamp associated with the log file entry) .PARAMETER Cluster The cluster management LIF IP address or resolvable DNS name for the cluster to connect to. .PARAMETER Username Username to use to connect to the cluster. .PARAMETER Password Password used to connect to the cluster. This is in clear text. If not provided you will be prompted for the password during the script and it will be obfuscated. .PARAMETER OutputFile A specific output file name to use. Without this parameter the default output file is named after the cluster name/IP address or the name of the input file specified along with a date stamp. .PARAMETER NumberOfFilesToParse By default all log files on the cluster are parsed. Specifying this parameter only the number passed will be parsed. .PARAMETER StartWithFileNumber By default the script parses logs starting at the most recent working back chronologically. If this parameter is passed the only logs that will be parsed when that log is reached in the order being reviewed through to the rest of the logs. .PARAMETER UseLocalLogFile This parameter will produce an open file window to prompt for a file to read to parse. You can select a file you downloaded from AutoSupport or copied directly from a cluster. When specifying this parameter, pass no other parameters. .PARAMETER ExportToExcel Exports to Excel XLSX format instead of CSV. You must have installed the ImportExcel module found here: https://github.com/dfinke/ImportExcel .EXAMPLE .\NetAppSnapMirrorAuditLogParser.ps1 Running without any parameters will prompt for all necessary values .EXAMPLE .\NetAppSnapMirrorAuditLogParser.ps1 -Cluster NetApp1 -Username admin -Password MyPassword Connects to the cluster named NetApp1 with the provided credentials. .EXAMPLE .\NetAppSnapMirrorAuditLogParser.ps1 -OutputFile MyResults.csv -NumberOfFilesToParse 3 -StartWithFileNumber 2 Prompts for cluster information and outputs to a file named MyResults.csv. Starts parsing when the second file is reached on each node while parsing a total of 3 files on each node. .EXAMPLE .\NetAppSnapMirrorAuditLogParser.ps1 -UseLocalLogFile This will provide a prompt to provide a local SnapMirror audit log file to parse. #> <#PSScriptInfo .VERSION 1.6 .GUID e4f02cb6-7fc6-4a68-bc0b-aad20d4c7726 .AUTHOR mcgue .COMPANYNAME .COPYRIGHT .TAGS .LICENSEURI .PROJECTURI .ICONURI .EXTERNALMODULEDEPENDENCIES .REQUIREDSCRIPTS .EXTERNALSCRIPTDEPENDENCIES .RELEASENOTES .PRIVATEDATA #> <# .DESCRIPTION The SnapMirror audit log provides a running log of SnapMirror relationships. It shows starting, restarting, and ending information for each SnapMirror relationship. This script reads in the log file(s) and parses the information into a usable CSV file. The output produces the following columns in the file: -Source (the source of the SnapMirror relationship) -Destination (the destination of the SnapMirror relationship -Type (request, start, or end) -KB Transferred (produced only for end type entries) -Time Stamp (the time stamp associated with the log file entry) #> #region Parameters and Variables [CmdletBinding(PositionalBinding=$False)] Param( [Parameter(Mandatory=$False)] [string]$Cluster, [Parameter(Mandatory=$False)] [string]$Username, [Parameter(Mandatory=$False)] [string]$Password, [Parameter(Mandatory=$False)] [string]$OutputFile, [Parameter(Mandatory=$false,ValueFromPipeline=$false)] [Switch]$UseLocalLogFile = $false, [Parameter(Mandatory=$false,ValueFromPipeline=$false)] [Switch]$ExportToExcel = $false, [Parameter(Mandatory=$False)] [int]$StartWithFileNumber = 1, [Parameter(Mandatory=$False)] [int]$NumberOfFilesToParse = -1 ) #Full output of either local audit log or all audit logs from cluster $SnapMirrorAuditLogContent = @() #Output of reduced log will be source,destination,type,KB transferred(if end),time stamp $Global:SnapMirrorAuditLogReduced = @() #Hash table for each SnapMirror transfer $SnapMirrorTransferHash=@{} $CollectedOutput = @() $FullCollectedOutput = @() $RowCounter = 0 # Check toolkit version If (!$UseLocalLogFile) { #Import module If (-Not (Get-Module NetApp.ONTAP)) { Import-Module NetApp.ONTAP } } #endregion #region Functions Function Get-OpenFile($initialDirectory) { [System.Reflection.Assembly]::LoadWithPartialName("System.windows.forms") | Out-Null $OpenFileDialog = New-Object System.Windows.Forms.OpenFileDialog $OpenFileDialog.initialDirectory = $initialDirectory $OpenFileDialog.filter = "All files (*.*)| *.*" $OpenFileDialog.ShowDialog() | Out-Null $OpenFileDialog.filename } Function Extract-SnapMirrorDetails ($SnapMirrorAuditResults) { ForEach ($Line in $SnapMirrorAuditResults) { $LineSplit = $Line -split '\s+' $SourceLocation = $Line.IndexOf("source=") $DestinationLocation = $Line.IndexOf("destination=") $TypeLocation = $Line.IndexOf("action=") $KBLocation = $Line.IndexOf("bytes_transferred=") $CompressionLocation = $Line.IndexOf("network_compression_ratio=") If ($SourceLocation -gt 0) { $Temp1 = $Line.Substring($SourceLocation) $Temp1 = $Temp1 -split '\s+' $Temp1 = $Temp1[0] -split '=' $SourceValue = $Temp1[1] } else { $SourceValue = "failure" } If ($DestinationLocation -gt 0) { $Temp1 = $Line.Substring($DestinationLocation) $Temp1 = $Temp1 -split '\s+' $Temp1 = $Temp1[0] -split '=' $DestinationValue = $Temp1[1] } else { $DestinationValue = "failure" } If ($KBLocation -gt 0) { $Temp1 = $Line.Substring($KBLocation) $Temp1 = $Temp1 -split '\s+' $Temp1 = $Temp1[0] -split '=' $KBValue = $Temp1[1] } else { $KBValue = "failure" } If ($TypeLocation -gt 0) { $Temp1 = $Line.Substring($TypeLocation) $Temp1 = $Temp1 -split '\s+' $Temp1 = $Temp1[0] -split '=' $TypeValue = $Temp1[1] } else { $TypeValue = "failure" } If ($CompressionLocation -gt 0) { $Temp1 = $Line.Substring($CompressionLocation) $Temp1 = $Temp1 -split '\s+' $Temp1 = $Temp1[0] -split '=' $CompressionValue = $Temp1[1] } else { $CompressionValue = "failure" } If ($SourceValue -eq "failure" -and $DestinationValue -eq "failure" -and $KBValue -eq "failure" -and $TypeValue -eq "failure" -and $CompressionValue -eq "failure") { $DateTimeStamp = "failure" } else { $Month = ConvertMonth ($LineSplit[1]) $TimeArray = $LineSplit[3] -split '\:' $DateTimeStamp = Get-Date -Year $LineSplit[5] -Month $Month -Day $LineSplit[2] -Hour $TimeArray[0] -Minute $TimeArray[1] -Second $TimeArray[2] $Temp1 = $LineSplit[6] $Temp1 = $Temp1 -split '\[' $SnapMirrorDetail = $Temp1[0] } $Global:SnapMirrorAuditLogReduced += ,@($SourceValue,$DestinationValue,$TypeValue,$KBValue,$DateTimeStamp,$CompressionValue,$SnapMirrorDetail) } } Function ConvertMonth ($AbbreviatedMonth) { switch ($AbbreviatedMonth.tolower()) { jan {Return [int]1} feb {Return [int]2} mar {Return [int]3} apr {Return [int]4} may {Return [int]5} jun {Return [int]6} jul {Return [int]7} aug {Return [int]8} sep {Return [int]9} oct {Return [int]10} nov {Return [int]11} dec {Return [int]12} default {Return 0} } } #endregion #region Main Body #Connect to the cluster If (!$UseLocalLogFile) { #Connect to the cluster If ($Cluster.Length -eq 0) { $Cluster = Read-host "Enter the cluster management LIF DNS name or IP address" } $Cluster = $Cluster.Trim() If ($Username.Length -eq 0) { $Username = "" } If ($Password.Length -eq 0) { $SecurePassword = "" } else { $SecurePassword = New-Object -TypeName System.Security.SecureString $Password.ToCharArray() | ForEach-Object {$SecurePassword.AppendChar($_)} } #Preparing credential object to pass If ($Username.Length -ne 0 -and $SecurePassword.Length -ne 0) { $Credentials = new-object -typename System.Management.Automation.PSCredential -argumentlist $Username, $SecurePassword } else { $Credentials = $Username } Write-Host "Attempting connection to $Cluster" $ClusterConnection = Connect-NcController -name $Cluster -Credential $Credentials #Only proceeding with valid connection to cluster If (!$ClusterConnection) { Write-Host "Unable to connect to NetApp cluster, please ensure all supplied information is correct and try again" -ForegroundColor Yellow Exit } #Get basic cluster information $ClusterInformation = Get-NcCluster $Nodes = Get-NcNode $NodeInformation = Get-NcNode Write-Host "Working with cluster:" $ClusterInformation.ClusterName Write-Host "Which contains the following Nodes:" $NodeInformation.Node } #Read from a local copied (from AutoSupport or direct from cluster) SnapMirror audit log file If ($UseLocalLogFile) { Write-Host "Select a saved snapmirror_audit file" $SnapMirrorAuditLogFile = Get-OpenFile Write-Host "Reading saved snapmirror_audit file" $SnapMirrorAuditLogContent = Get-Content $SnapMirrorAuditLogFile } else { #Need to read in SnapMirror audit logs from each node ForEach ($Node in $Nodes) { $FileCounter = 0 Write-Host "" If ($NumberOfFilesToParse -gt -1) { If ($StartWithFileNumber -gt 1) { Write-Host "Finding up to" $NumberOfFilesToParse "SnapMirror audit log files from node" $Node "starting with file" $StartWithFileNumber } else { Write-Host "Finding up to" $NumberOfFilesToParse "SnapMirror audit log files from node" $Node } } else { If ($StartWithFileNumber -gt 1) { Write-Host "Finding all SnapMirror audit log files from node" $Node "starting with file" $StartWithFileNumber } else { Write-Host "Finding all SnapMirror audit log files from node" $Node } } Write-Host "Retrieving SnapMirror audit log files from node" $Node $SnapMirrorAuditLogFileList = @() $LogDirectoryListingSnapMirrorAuditLogs = @() #Find all logs under /etc/log $LogDirectoryListingCommand = "system node run -node " + $Node + ' -command "priv set diag;ls /etc/log;priv set admin"' $LogDirectoryListingFull = Invoke-NcSsh $LogDirectoryListingCommand $StringLogDirectoryListingFull = $LogDirectoryListingFull.ToString() $ArrayLogDirectoryListingFull = $StringLogDirectoryListingFull.Split("`n") #Put in alphabetical order $ArrayLogDirectoryListingFull = $ArrayLogDirectoryListingFull | Sort-Object #Find only the snapmirror_audit files in this listing without the .log extension ForEach ($FileName in $ArrayLogDirectoryListingFull) { #See the following KB for layout of SnapMirror audit log files https://kb.netapp.com/support/index?page=content&id=3014284&locale=en_US If ($FileName -match "snapmirror_audit.log") { $SnapMirrorAuditLogFileList += $FileName } } #The high numbered files are most recent $SnapMirrorAuditLogFileList = $SnapMirrorAuditLogFileList | Sort-Object -Descending $NumberSkipped = 0 ForEach ($FileName in $SnapMirrorAuditLogFileList) { #Start with the file number if specified If (($StartWithFileNumber-1) -gt 0) { If ($NumberSkipped -lt ($StartWithFileNumber-1)) { #Do not read the file Write-Host "Skipping file named" $FileName $NumberSkipped++ Continue } } If (($NumberOfFilesToParse -eq -1) -or ($FileCounter -lt $NumberOfFilesToParse)) { $LogDirectoryListingSnapMirrorAuditLogs += $FileName #Increment the counter $FileCounter++ } } ForEach ($SnapMirrorAuditLogFile in $LogDirectoryListingSnapMirrorAuditLogs) { Write-Host "Reading SnapMirror audit log file named" $SnapMirrorAuditLogFile $ReadFileCommand = "system node run -node " + $Node + ' -command rdfile /etc/log/' + $SnapMirrorAuditLogFile $ReadFileResults = Invoke-NcSsh $ReadFileCommand $StringReadFileResults = $ReadFileResults.ToString() $ArrayReadFileResults = $StringReadFileResults.Split("`n") ForEach ($SnapMirrorAuditMessage in $ArrayReadFileResults) { $SnapMirrorAuditLogContent += $SnapMirrorAuditMessage } } } } #Create a formatted output of the results with just the needed details Extract-SnapMirrorDetails ($SnapMirrorAuditLogContent) #Start to process the results Write-Host "Now processing gathered SnapMirror audit log entries" #Get or create an output file If ($OutputFile -eq "") { #Create a file name formatted as ClusterNameDateStamp.csv $DateStamp = get-date -uformat "%Y-%m-%d@%H-%M-%S" If (!$UseLocalLogFile) { If ($ExportToExcel) { $OutputFile = $ClusterInformation.ClusterName + "_" + $DateStamp + ".xlsx" } else { $OutputFile = $ClusterInformation.ClusterName + "_" + $DateStamp + ".csv" } } else { If ($ExportToExcel) { $OutputFile = $SnapMirrorAuditLogFile + "_" + $DateStamp + ".xlsx" } else { $OutputFile = $SnapMirrorAuditLogFile + "_" + $DateStamp + ".csv" } } } #Parse the collected data ForEach ($AuditEntry in $Global:SnapMirrorAuditLogReduced) { $ArrayAuditEntry = $AuditEntry -split '\,' If ($SnapMirrorTransferHash.Item($ArrayAuditEntry[1]) -eq $null) { $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]) = @{} } #There is no request line in CDOT, only starts If ($ArrayAuditEntry[2].StartsWith("Start")) { $RequestEntry = @{ "Request" = $ArrayAuditEntry[4] } If ($SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).ContainsKey("Request")) { # do nothing as it has already been attempted to start once } else { $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]) += $RequestEntry } $StartEntry = @{ "Start" = $ArrayAuditEntry[4] } If ($SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).ContainsKey("Start")) { #This means SnapMirror was attempted to be started before, but failed to complete #for some reason, thus making the first start in the series the request $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).Remove("Start") } $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]) += $StartEntry } If ($ArrayAuditEntry[2].StartsWith("End")) { #Only need to proceed if SnapMirror completed in which this field would show KB entry If ($ArrayAuditEntry[3] -ne "failure") { [int64]$Bytes = $ArrayAuditEntry[3].TrimStart("(") #Convert bytes to KB $KB = ($Bytes/1024) $EndEntry = @{ "End" = $ArrayAuditEntry[4] } #Remove any previous entries for an end that didn't complete the transfer If ($SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).ContainsKey("End")) { $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).Remove("End") } else { $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]) += $EndEntry } If (($SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).ContainsKey("Request")) -And ` ($SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).ContainsKey("Start")) -And ` ($SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).ContainsKey("End"))) { $PowerShellObject = New-Object PSCustomObject $RowCounter++ $RequestTime = Get-Date $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).Request $start_time = Get-Date $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).Start $EndTime = Get-Date $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).End $RequestToEnd = ($EndTime - $RequestTime).TotalSeconds $StartToEnd = ($EndTime - $start_time).TotalSeconds $RequestToStart = ($start_time - $RequestTime).TotalSeconds $DataThroughput = [math]::Round($KB/$StartToEnd) $CompressionRatio = $ArrayAuditEntry[5] $Temp1 = $CompressionRatio -split '\:' $CompressionRatioCalc = $Temp1[0] $NetworkThroughput = [math]::Round($DataThroughput/$CompressionRatioCalc*8) $PowerShellObject | Add-Member -type NoteProperty -name Row -value $RowCounter $PowerShellObject | Add-Member -type NoteProperty -name Source -value $ArrayAuditEntry[0] $PowerShellObject | Add-Member -type NoteProperty -name Destination -value $ArrayAuditEntry[1] $PowerShellObject | Add-Member -type NoteProperty -name 'KB Transferred' -value $KB $PowerShellObject | Add-Member -type NoteProperty -name 'Request Time' -value $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).Request $PowerShellObject | Add-Member -type NoteProperty -name 'Start Time' -value $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).Start $PowerShellObject | Add-Member -type NoteProperty -name 'End Time' -value $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]).End $PowerShellObject | Add-Member -type NoteProperty -name 'Request to End' -value $RequestToEnd $PowerShellObject | Add-Member -type NoteProperty -name 'Start to End' -value $StartToEnd $PowerShellObject | Add-Member -type NoteProperty -name 'Request to Start' -value $RequestToStart $PowerShellObject | Add-Member -type NoteProperty -name 'Data Throughput KB/s' -value $DataThroughput $PowerShellObject | Add-Member -type NoteProperty -name 'Network Throughput Kb/s' -value $NetworkThroughput $PowerShellObject | Add-Member -type NoteProperty -name 'Network Compression Ratio' -value $CompressionRatio $PowerShellObject | Add-Member -type NoteProperty -name 'SnapMirror Detail' -value $ArrayAuditEntry[6] $CollectedOutput += $PowerShellObject #Null out the hash table to start fresh $SnapMirrorTransferHash.Item($ArrayAuditEntry[1]) = $null } } } } #Check for concurrency ForEach ($SnapMirrorTransfer in $CollectedOutput) { #Reset the counter to count only this running SnapMirror $ConcurrentSnapMirrorTransfers = 1 #Set the start and end times of the current SnapMirror transfer $MasterStartTime = $SnapMirrorTransfer.'Start Time' $MasterEndTime = $SnapMirrorTransfer.'End Time' #Find any other SnapMirror transfer that transfered any data between the start/end times of this transfer ForEach ($CheckSnapMirrorTransfer in $CollectedOutput) { $CheckStartTime = $CheckSnapMirrorTransfer.'Start Time' $CheckEndTime = $CheckSnapMirrorTransfer.'End Time' #CheckStart > MasterStart CheckEnd < MasterEnd CheckEnd > MasterStart #CheckStart < MasterStart CheckEnd < MasterEnd CheckEnd > MasterStart #CheckStart < MasterStart CheckEnd > MasterEnd CheckEnd > MasterStart #CheckStart > MasterStart CheckEnd > MasterEnd CheckStart < MasterEnd CheckEnd > MasterStart If ((($CheckStartTime -gt $MasterStartTime -and $CheckEndTime -lt $MasterEndTime) ` -or ($CheckStartTime -lt $MasterStartTime) ` -or ($CheckStartTime -gt $MasterStartTime -and $CheckEndTime -gt $MasterEndTime -and $CheckStartTime -lt $MasterEndTime)) ` -and ($CheckEndTime -gt $MasterStartTime)) { $ConcurrentSnapMirrorTransfers++ } } #Add this new concurrency number to the other fields $PowerShellObject = New-Object PSCustomObject $PowerShellObject | Add-Member -type NoteProperty -name Row -value $SnapMirrorTransfer.Row $PowerShellObject | Add-Member -type NoteProperty -name Source -value $SnapMirrorTransfer.Source $PowerShellObject | Add-Member -type NoteProperty -name Destination -value $SnapMirrorTransfer.Destination $PowerShellObject | Add-Member -type NoteProperty -name 'KB Transferred' -value $SnapMirrorTransfer.'KB Transferred' $PowerShellObject | Add-Member -type NoteProperty -name 'Request Time' -value $SnapMirrorTransfer.'Request Time' $PowerShellObject | Add-Member -type NoteProperty -name 'Start Time' -value $SnapMirrorTransfer.'Start Time' $PowerShellObject | Add-Member -type NoteProperty -name 'End Time' -value $SnapMirrorTransfer.'End Time' $PowerShellObject | Add-Member -type NoteProperty -name 'Request to End' -value $SnapMirrorTransfer.'Request to End' $PowerShellObject | Add-Member -type NoteProperty -name 'Start to End' -value $SnapMirrorTransfer.'Start to End' $PowerShellObject | Add-Member -type NoteProperty -name 'Request to Start' -value $SnapMirrorTransfer.'Request to Start' $PowerShellObject | Add-Member -type NoteProperty -name 'Data Throughput KB/s' -value $SnapMirrorTransfer.'Data Throughput KB/s' $PowerShellObject | Add-Member -type NoteProperty -name 'Network Throughput Kb/s' -value $SnapMirrorTransfer.'Network Throughput Kb/s' $PowerShellObject | Add-Member -type NoteProperty -name 'Concurrency' -value $ConcurrentSnapMirrorTransfers $PowerShellObject | Add-Member -type NoteProperty -name 'Network Compression Ratio' -value $SnapMirrorTransfer.'Network Compression Ratio' $PowerShellObject | Add-Member -type NoteProperty -name 'SnapMirror Detail' -value $SnapMirrorTransfer.'SnapMirror Detail' $FullCollectedOutput += $PowerShellObject } #Export to file Write-Host "Saving to output file named:" $OutputFile If ($ExportToExcel) { #Import the Excel module If (-Not (Get-Module ImportExcel)) { Import-Module ImportExcel } $FullCollectedOutput | Export-Excel -Path "$OutputFile" -WorkSheetname "SnapMirror Results" -BoldTopRow -AutoSize -FreezeTopRow -AutoFilter -Numberformat "#,##0" } else { $FullCollectedOutput | Export-Csv -Path $OutputFile -NoTypeInformation } Write-Host "Process complete" #endregion |