Test-ArcEsuChain.ps1
|
<#PSScriptInfo
.VERSION 1.0.2 .GUID 4e7fedde-ee1b-40e9-96c8-9c9706cb54d6 .AUTHOR Petar Ivanov .COMPANYNAME .COPYRIGHT (c) 2026 Petar Ivanov. All rights reserved. .TAGS Azure Arc ESU ExtendedSecurityUpdates WindowsServer2012 Certificate Revocation CRL OCSP Troubleshooting Diagnostics .LICENSEURI .PROJECTURI .ICONURI .EXTERNALMODULEDEPENDENCIES .REQUIREDSCRIPTS .EXTERNALSCRIPTDEPENDENCIES .RELEASENOTES 1.0.2 - CBS rollback signatures are now reported as WARN (evidence), never FAIL. CBS entries are historical by nature, so a hard FAIL was misleading - e.g. a rollback logged minutes before a fix would still flag after re-running. Each signature is shown with its latest timestamp and hit count so it can be correlated against the time of the fix / last attempt. The live verdict comes from the chain build and endpoint checks, which reflect current state. 1.0.1 - CBS log scan classified ESU rollback signatures by recency and consolidated to a single combined-regex pass. (Recency split superseded by 1.0.2.) 1.0.0 - Initial release. Diagnoses the Azure Arc-enabled ESU "The chain does not seem valid" patch-rollback issue on Windows Server 2012 / 2012 R2: certificate chain build (with and without revocation), required certificate stores, endpoint reachability with proxy-block detection, revocation cache, certutil verify, CBS log signatures, and an optional -CollectZip diagnostic bundle. Read-only. #> <# .SYNOPSIS Diagnoses the Azure Arc-enabled ESU "The chain does not seem valid" patch-rollback issue on Windows Server 2012 / 2012 R2. .DESCRIPTION Runs a comprehensive set of read-only checks on an Arc-enabled Windows Server 2012 / 2012 R2 machine where the latest ESU security update installs, reboots, then rolls back. It pinpoints WHICH of the known causes applies: * Missing / untrusted certificate in the license signing chain * Certificate chain present but REVOCATION cannot be checked (CRL/OCSP endpoint blocked by a proxy/firewall - e.g. Zscaler) * Old agent / missing Servicing Stack Update * License file / himds problems * Clock skew, blocked cert-download endpoint, root auto-update disabled The script only READS state (plus harmless network GETs). It changes nothing. .PARAMETER LicensePath Path to the Arc ESU license file. Defaults to the standard location. .PARAMETER Proxy Optional explicit proxy (e.g. http://proxy.contoso.com:8080) to use for the endpoint reachability tests. If omitted the machine/user default is used. .PARAMETER SkipNetwork Skip the live endpoint reachability tests (useful on air-gapped boxes). .PARAMETER CbsHours How many hours of CBS log history to scan for ESU rollback signatures. Default 72. .PARAMETER CollectZip Also collect all diagnostic outputs (report, chain, cert stores, proxy, url caches, certutil verify, filtered CBS ESU lines, the signing cert and license.json) into a single .zip for attaching to the case. Zips via .NET so it works on PowerShell 4.0. .PARAMETER OutputPath Optional explicit path for the -CollectZip output .zip. Defaults to the Desktop (or %TEMP%) as ArcEsuDiag_<host>_<timestamp>.zip. .EXAMPLE .\Test-ArcEsuChain.ps1 .EXAMPLE .\Test-ArcEsuChain.ps1 -Proxy "http://proxy.contoso.com:8080" .EXAMPLE .\Test-ArcEsuChain.ps1 -CollectZip .NOTES Run from an ELEVATED PowerShell prompt for complete results. Compatible with Windows PowerShell 4.0+ (Server 2012/2012 R2 default). #> [CmdletBinding()] param( [string] $LicensePath = "C:\ProgramData\AzureConnectedMachineAgent\Certs\license.json", [string] $Proxy, [switch] $SkipNetwork, [int] $CbsHours = 72, [switch] $CollectZip, [string] $OutputPath ) $ErrorActionPreference = "Continue" $ProgressPreference = "SilentlyContinue" # suppress Invoke-WebRequest progress noise on WinPS 5.1 $script:Findings = New-Object System.Collections.ArrayList # ----------------------------------------------------------------------------- helpers function Write-Section { param([string] $Title) Write-Host "" Write-Host ("=" * 78) -ForegroundColor DarkCyan Write-Host (" " + $Title) -ForegroundColor Cyan Write-Host ("=" * 78) -ForegroundColor DarkCyan } function Add-Finding { # Level: PASS / FAIL / WARN / INFO param([string] $Level, [string] $Check, [string] $Detail) $color = switch ($Level) { "PASS" { "Green" } "FAIL" { "Red" } "WARN" { "Yellow" } default { "Gray" } } $tag = "[{0}]" -f $Level Write-Host ("{0,-7}" -f $tag) -ForegroundColor $color -NoNewline Write-Host (" {0}" -f $Check) -ForegroundColor White -NoNewline if ($Detail) { Write-Host (" -> {0}" -f $Detail) -ForegroundColor Gray } else { Write-Host "" } [void]$script:Findings.Add([PSCustomObject]@{ Level = $Level; Check = $Check; Detail = $Detail }) } function Test-IsElevated { try { $id = [System.Security.Principal.WindowsIdentity]::GetCurrent() $p = New-Object System.Security.Principal.WindowsPrincipal($id) return $p.IsInRole([System.Security.Principal.WindowsBuiltInRole]::Administrator) } catch { return $false } } # Zip a directory in a way that works on PowerShell 4.0 (Server 2012 R2 default), # where Compress-Archive does not exist. Falls back to Compress-Archive if present. function New-ZipFromDir { param([string] $Dir, [string] $Zip) if (Test-Path $Zip) { Remove-Item $Zip -Force -ErrorAction SilentlyContinue } try { Add-Type -AssemblyName System.IO.Compression.FileSystem -ErrorAction Stop [System.IO.Compression.ZipFile]::CreateFromDirectory($Dir, $Zip) return $true } catch { try { Compress-Archive -Path (Join-Path $Dir '*') -DestinationPath $Zip -Force -ErrorAction Stop return $true } catch { return $false } } } # Performs an HTTP GET and classifies the outcome: # Reachable - got the real resource # BlockedByProxy - got an HTML interstitial (Zscaler/other) instead of the resource # Failed - timeout / connection error function Test-Endpoint { param( [string] $Url, [string] $Kind = "generic", # "crl", "cert" or "generic" [string] $ProxyArg ) $iwrArgs = @{ Uri = $Url; UseBasicParsing = $true; TimeoutSec = 30; ErrorAction = "Stop" } if ($ProxyArg) { $iwrArgs["Proxy"] = $ProxyArg; $iwrArgs["ProxyUseDefaultCredentials"] = $true } try { $resp = Invoke-WebRequest @iwrArgs $ctype = "" try { $ctype = [string]$resp.Headers["Content-Type"] } catch {} $looksHtml = ($ctype -match "text/html") $bytes = $null try { $bytes = $resp.Content } catch {} # Detect proxy block interstitial $raw = "" try { $raw = [string]$resp.RawContent } catch {} $blockHit = ($raw -match "(?i)zscaler|category_denied|website blocked|access denied|you don't have permission|proxy") if (($Kind -eq "crl" -or $Kind -eq "cert") -and ($looksHtml -or $blockHit)) { return [PSCustomObject]@{ Status = "BlockedByProxy"; Code = $resp.StatusCode; Detail = "HTML/interstitial returned instead of binary ($ctype)" } } if ($blockHit) { return [PSCustomObject]@{ Status = "BlockedByProxy"; Code = $resp.StatusCode; Detail = "proxy block page detected" } } return [PSCustomObject]@{ Status = "Reachable"; Code = $resp.StatusCode; Detail = "Content-Type=$ctype" } } catch { $msg = $_.Exception.Message # An HTTP error response (e.g. 400 from an OCSP base URL) still means the host is reachable if ($_.Exception.Response -ne $null) { $sc = $null try { $sc = [int]$_.Exception.Response.StatusCode } catch {} return [PSCustomObject]@{ Status = "Reachable"; Code = $sc; Detail = "HTTP error but host responded ($msg)" } } return [PSCustomObject]@{ Status = "Failed"; Code = $null; Detail = $msg } } } # ----------------------------------------------------------------------------- start Write-Host "" Write-Host " Arc-enabled ESU chain / rollback diagnostic" -ForegroundColor Cyan Write-Host " Reference: aka.ms/arc-esu-troubleshoot | $(Get-Date -Format 'dd/MM/yyyy HH:mm')" -ForegroundColor DarkGray $elevated = Test-IsElevated if (-not $elevated) { Add-Finding "WARN" "Not running elevated" "Some checks (urlcache, certutil verify) may be incomplete. Re-run as Administrator." } # 1 --------------------------------------------------------------------- environment Write-Section "1. Environment" try { $os = Get-CimInstance Win32_OperatingSystem -ErrorAction Stop $ver = [version]$os.Version $caption = $os.Caption $isSupported = ($ver.Major -eq 6 -and ($ver.Minor -eq 2 -or $ver.Minor -eq 3)) if ($isSupported) { Add-Finding "PASS" "Operating system" "$caption (build $($os.Version)) - in ESU scope" } else { Add-Finding "WARN" "Operating system" "$caption (build $($os.Version)) - this KI targets Server 2012 (6.2) / 2012 R2 (6.3)" } } catch { Add-Finding "INFO" "Operating system" "Could not query Win32_OperatingSystem: $($_.Exception.Message)" } Add-Finding "INFO" "PowerShell version" ("{0}" -f $PSVersionTable.PSVersion) try { $now = Get-Date $tz = [System.TimeZoneInfo]::Local.DisplayName Add-Finding "INFO" "System clock" ("{0:dd/MM/yyyy HH:mm:ss} ({1})" -f $now, $tz) # Crude skew sanity check against file time of a known system file is unreliable; just surface it. } catch {} # 2 ------------------------------------------------------------------ agent + himds Write-Section "2. Azure Connected Machine Agent" $agentVersion = $null try { $azcm = Get-Command azcmagent -ErrorAction SilentlyContinue if (-not $azcm) { $exe = "C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe" if (Test-Path $exe) { $azcm = $exe } } if ($azcm) { $verOut = & $(if ($azcm -is [string]) { $azcm } else { $azcm.Source }) version 2>$null if ($verOut) { $m = [regex]::Match(($verOut -join " "), "(\d+\.\d+)") if ($m.Success) { $agentVersion = [version]$m.Value } Add-Finding "INFO" "azcmagent version" ($verOut -join " ") } } if (-not $agentVersion) { $exe = "C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe" if (Test-Path $exe) { $fv = (Get-Item $exe).VersionInfo.ProductVersion $m = [regex]::Match($fv, "(\d+\.\d+)") if ($m.Success) { $agentVersion = [version]$m.Value } Add-Finding "INFO" "azcmagent file version" $fv } } if ($agentVersion) { if ($agentVersion -ge [version]"1.40") { Add-Finding "PASS" "Agent version >= 1.40" "$agentVersion" } else { Add-Finding "FAIL" "Agent version >= 1.40" "$agentVersion - OLD. Upgrade the Connected Machine Agent before anything else." } } else { Add-Finding "WARN" "Agent version" "Could not determine - is the Connected Machine Agent installed?" } } catch { Add-Finding "INFO" "Agent version" "Error: $($_.Exception.Message)" } try { $himds = Get-Service himds -ErrorAction Stop if ($himds.Status -eq "Running") { Add-Finding "PASS" "himds service" "Running" } else { Add-Finding "FAIL" "himds service" "$($himds.Status) - must be Running to serve the local ESU eligibility check" } } catch { Add-Finding "WARN" "himds service" "Not found: $($_.Exception.Message)" } # 3 ----------------------------------------------------------------- prerequisite SSU Write-Section "3. Servicing Stack Update prerequisite" # KB5037022 (2012) / KB5037021 (2012 R2) = April 2024 SSU or later removes the # requirement for the intermediary CA certs for the signed license. try { $hot = Get-HotFix -ErrorAction Stop | Where-Object { $_.HotFixID -match "KB\d+" } $ssuKbs = @("KB5037022","KB5037021") $found = $hot | Where-Object { $ssuKbs -contains $_.HotFixID } if ($found) { Add-Finding "PASS" "April 2024 SSU present" (($found | ForEach-Object { $_.HotFixID }) -join ", ") } else { $recent = $hot | Sort-Object InstalledOn -Descending | Select-Object -First 5 | ForEach-Object { $_.HotFixID } Add-Finding "WARN" "April 2024 SSU (KB5037022/KB5037021)" ("Not detected via Get-HotFix. A LATER SSU may still satisfy it. Recent hotfixes: " + ($recent -join ", ")) } } catch { Add-Finding "INFO" "Servicing Stack Update" "Get-HotFix failed: $($_.Exception.Message)" } # 4 -------------------------------------------------------------------- license file Write-Section "4. ESU license file" $signingCert = $null if (-not (Test-Path $LicensePath)) { Add-Finding "FAIL" "license.json present" "$LicensePath NOT found - is ESU linked to this machine? (try restarting himds)" } else { Add-Finding "PASS" "license.json present" $LicensePath try { $doc = Get-Content -Path $LicensePath -Raw | ConvertFrom-Json if (-not $doc.signature) { Add-Finding "FAIL" "license signature field" "Present file but no 'signature' field - license may be corrupt" } else { $sigBytes = [System.Convert]::FromBase64String($doc.signature) $signingCert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2(,$sigBytes) Add-Finding "PASS" "license signing certificate" ("Subject: {0}; Issuer: {1}; NotAfter: {2:dd/MM/yyyy}" -f $signingCert.Subject, $signingCert.Issuer, $signingCert.NotAfter) if ($signingCert.NotAfter -lt (Get-Date)) { Add-Finding "WARN" "signing cert validity" "Signing certificate is EXPIRED" } } } catch { Add-Finding "FAIL" "parse license.json" "Could not parse/decode: $($_.Exception.Message)" } } # 5 + 6 ------------------------------------------------------------- chain validation Write-Section "5. Certificate chain validation" $onlineOk = $null; $nocheckOk = $null; $chainStatuses = @(); $nocheckStatuses = @() $script:PrimaryCause = $null if ($signingCert) { # ONLINE (default) build - this is what the ESU installer effectively does try { $chain = New-Object System.Security.Cryptography.X509Certificates.X509Chain $chain.ChainPolicy.RevocationMode = "Online" $chain.ChainPolicy.RevocationFlag = "ExcludeRoot" $onlineOk = $chain.Build($signingCert) $chainStatuses = @($chain.ChainStatus | ForEach-Object { $_.Status.ToString() }) if ($onlineOk) { Add-Finding "PASS" "Chain build (revocation ON)" "Valid" } else { Add-Finding "FAIL" "Chain build (revocation ON)" ("Invalid. ChainStatus: " + (($chain.ChainStatus | ForEach-Object { $_.Status }) -join ", ")) foreach ($s in $chain.ChainStatus) { Add-Finding "INFO" (" status: " + $s.Status) $s.StatusInformation.Trim() } } Write-Host "" Write-Host " Chain (leaf -> root):" -ForegroundColor DarkGray $i = 0 foreach ($el in $chain.ChainElements) { Write-Host (" [{0}] {1}" -f $i, $el.Certificate.Issuer) -ForegroundColor DarkGray $i++ } $terminator = $chain.ChainElements[$chain.ChainElements.Count - 1].Certificate if ($terminator.Subject -match "DigiCert Global Root G2") { Add-Finding "PASS" "Chain terminates at" "DigiCert Global Root G2 (matches a known-good machine)" } else { Add-Finding "WARN" "Chain terminates at" ("{0} - a known-good chain bridges to DigiCert Global Root G2. The cross-signed 'Microsoft TLS RSA Root G2' (issued by DigiCert) may be missing from the Intermediate (CA) store." -f $terminator.Subject) } } catch { Add-Finding "INFO" "Chain build (online)" "Error: $($_.Exception.Message)" } # NO-CHECK build - isolates revocation from trust/path problems try { $chain2 = New-Object System.Security.Cryptography.X509Certificates.X509Chain $chain2.ChainPolicy.RevocationMode = "NoCheck" $nocheckOk = $chain2.Build($signingCert) $nocheckStatuses = @($chain2.ChainStatus | ForEach-Object { $_.Status.ToString() }) if ($nocheckOk) { Add-Finding "PASS" "Chain build (revocation OFF)" "Valid - the certificates/path are correct" } else { Add-Finding "FAIL" "Chain build (revocation OFF)" ("Invalid even without revocation. ChainStatus: " + (($chain2.ChainStatus | ForEach-Object { $_.Status }) -join ", ")) } } catch { Add-Finding "INFO" "Chain build (no-check)" "Error: $($_.Exception.Message)" } # Verdict - driven by the actual ChainStatus codes, not just the booleans $allStatus = @($chainStatuses + $nocheckStatuses) | Where-Object { $_ } | Select-Object -Unique if ($onlineOk -eq $true) { $script:PrimaryCause = "Healthy" Add-Finding "PASS" "DIAGNOSIS" "Chain validates fully online - the cert/revocation path is NOT the current blocker." } elseif ($allStatus -contains "NotTimeValid") { $script:PrimaryCause = "NotTimeValid" Add-Finding "WARN" "DIAGNOSIS" "A certificate in the chain is EXPIRED or the system clock is wrong (NotTimeValid). Check the signing cert NotAfter above and the system time. On a live machine himds should renew the license - restart himds; a captured/old license will show this." } elseif ($onlineOk -eq $false -and $nocheckOk -eq $true) { $script:PrimaryCause = "Revocation" Add-Finding "WARN" "DIAGNOSIS" "Certificates are CORRECT - failure is the REVOCATION check (CRL/OCSP unreachable). Focus on endpoint/proxy below." } elseif ($allStatus -contains "UntrustedRoot" -or $allStatus -contains "PartialChain") { $script:PrimaryCause = "MissingCert" Add-Finding "WARN" "DIAGNOSIS" "A certificate in the chain is MISSING/UNTRUSTED ($([string]::Join(', ',$allStatus))). Install the missing intermediate/root (see the store check above and the troubleshooting guide)." } else { $script:PrimaryCause = "Other" Add-Finding "WARN" "DIAGNOSIS" "Chain invalid - ChainStatus: $([string]::Join(', ',$allStatus)). Review the statuses above." } } else { Add-Finding "INFO" "Chain validation" "Skipped - no signing certificate loaded." } # 7 ------------------------------------------------------------- required cert stores Write-Section "6. Required certificates in local machine stores" function Test-CertInStore { param([string] $StoreName, [string] $MatchSubject, [string] $Friendly) try { $store = New-Object System.Security.Cryptography.X509Certificates.X509Store($StoreName, "LocalMachine") $store.Open("ReadOnly") $hit = $store.Certificates | Where-Object { $_.Subject -match [regex]::Escape($MatchSubject) } $store.Close() if ($hit) { Add-Finding "PASS" ("[$StoreName] $Friendly") "present" } else { Add-Finding "WARN" ("[$StoreName] $Friendly") "MISSING" } } catch { Add-Finding "INFO" ("[$StoreName] $Friendly") "store read error: $($_.Exception.Message)" } } Test-CertInStore "Root" "Microsoft TLS RSA Root G2" "Microsoft TLS RSA Root G2 (root)" Test-CertInStore "Root" "DigiCert Global Root G2" "DigiCert Global Root G2 (root)" Test-CertInStore "CA" "Microsoft TLS G2 RSA CA OCSP 16" "Microsoft TLS G2 RSA CA OCSP 16 (intermediate)" Test-CertInStore "CA" "Microsoft TLS RSA Root G2" "Microsoft TLS RSA Root G2 cross-signed (intermediate)" Test-CertInStore "CA" "Microsoft Azure RSA TLS Issuing CA 0" "Microsoft Azure RSA TLS Issuing CA 03/04 (intermediate)" # 8 -------------------------------------------------------------- SYSTEM WinHTTP proxy Write-Section "7. Machine (SYSTEM) WinHTTP proxy" # The ESU check runs as SYSTEM via WinHTTP - this proxy can differ from the user/IE proxy. try { $np = netsh winhttp show proxy 2>$null Add-Finding "INFO" "netsh winhttp show proxy" (($np | Where-Object { $_ -match "\S" }) -join " | ") } catch { Add-Finding "INFO" "WinHTTP proxy" "Could not query: $($_.Exception.Message)" } # 9 -------------------------------------------------------------- root auto-update key Write-Section "8. Root certificate auto-update policy" try { $regPath = "HKLM:\SOFTWARE\Policies\Microsoft\SystemCertificates\AuthRoot" $val = $null if (Test-Path $regPath) { $val = (Get-ItemProperty -Path $regPath -Name DisableRootAutoUpdate -ErrorAction SilentlyContinue).DisableRootAutoUpdate } if ($val -eq 1) { Add-Finding "WARN" "DisableRootAutoUpdate" "= 1 (root auto-update DISABLED). The OS cannot pull missing roots/CRLs automatically." } else { Add-Finding "PASS" "DisableRootAutoUpdate" "not set / 0 (auto root update allowed)" } } catch { Add-Finding "INFO" "DisableRootAutoUpdate" "reg read error: $($_.Exception.Message)" } # 10 ------------------------------------------------------------- revocation cache state Write-Section "9. Revocation (CRL/OCSP) cache state" foreach ($kind in @("CRL","OCSP")) { try { $out = certutil -urlcache $kind 2>$null $hits = $out | Where-Object { $_ -match "(?i)pkiops|digicert|microsoft" } if ($hits) { Add-Finding "INFO" "Cached $kind entries (relevant)" (($hits | Select-Object -First 6) -join " | ") } else { Add-Finding "INFO" "Cached $kind entries" "none relevant to the ESU chain" } } catch { Add-Finding "INFO" "Cached $kind entries" "certutil error: $($_.Exception.Message)" } } # 11 ------------------------------------------------------------ endpoint reachability Write-Section "10. Endpoint reachability (cert + revocation)" if ($SkipNetwork) { Add-Finding "INFO" "Endpoint tests" "Skipped (-SkipNetwork)" } else { if ($Proxy) { Add-Finding "INFO" "Using proxy" $Proxy } $targets = @( @{ Url = "http://www.microsoft.com/pkiops/certs/Microsoft%20TLS%20G2%20RSA%20CA%20OCSP%2016.crt"; Kind = "cert"; Name = "Microsoft pkiops CERT endpoint" }, @{ Url = "http://www.microsoft.com/pkiops/crl/Microsoft%20TLS%20RSA%20Root%20G2.crl"; Kind = "crl"; Name = "Microsoft pkiops CRL endpoint" }, @{ Url = "http://oneocsp.microsoft.com/"; Kind = "generic"; Name = "Microsoft OCSP responder" }, @{ Url = "http://crl3.digicert.com/DigiCertGlobalRootG2.crl"; Kind = "crl"; Name = "DigiCert CRL endpoint" }, @{ Url = "http://ocsp.digicert.com/"; Kind = "generic"; Name = "DigiCert OCSP responder" } ) foreach ($t in $targets) { $r = Test-Endpoint -Url $t.Url -Kind $t.Kind -ProxyArg $Proxy switch ($r.Status) { "Reachable" { Add-Finding "PASS" $t.Name ("reachable (HTTP {0}) {1}" -f $r.Code, $r.Detail) } "BlockedByProxy" { Add-Finding "FAIL" $t.Name ("BLOCKED by proxy/firewall - {0}. URL: {1}" -f $r.Detail, $t.Url) } default { Add-Finding "FAIL" $t.Name ("unreachable - {0}. URL: {1}" -f $r.Detail, $t.Url) } } } } # 12 ----------------------------------------------------------------- certutil verify Write-Section "11. certutil chain + revocation verify" if ($signingCert) { try { $tmp = Join-Path $env:TEMP ("esu_signcert_{0}.cer" -f $PID) [System.IO.File]::WriteAllBytes($tmp, $signingCert.RawData) $vo = certutil -f -urlfetch -verify $tmp 2>&1 Remove-Item $tmp -ErrorAction SilentlyContinue $key = $vo | Where-Object { $_ -match "(?i)Verifie|revocation|offline|ERROR|Cert is|failed|leaf|CRL|OCSP" } if ($key) { Write-Host " (key lines from certutil -verify)" -ForegroundColor DarkGray $key | Select-Object -First 25 | ForEach-Object { Write-Host (" " + $_) -ForegroundColor DarkGray } } # Match a genuine revocation failure phrase - NOT the flag constant CA_VERIFY_FLAGS_IGNORE_OFFLINE $offline = $vo | Where-Object { $_ -match "(?i)unable to check revocation|revocation server was offline|revocation status.*unknown" } if ($offline) { Add-Finding "FAIL" "certutil revocation" ("Revocation could not be checked -> " + ($offline | Select-Object -First 1).ToString().Trim()) } else { Add-Finding "INFO" "certutil verify" "completed (review lines above)" } } catch { Add-Finding "INFO" "certutil verify" "Error: $($_.Exception.Message)" } } else { Add-Finding "INFO" "certutil verify" "Skipped - no signing certificate." } # 13 ---------------------------------------------------------------- CBS log signatures Write-Section "12. CBS log ESU rollback signatures (historical evidence)" try { $cbsDir = "C:\Windows\Logs\CBS" $cut = (Get-Date).AddHours(-1 * $CbsHours) $logs = Get-ChildItem -Path $cbsDir -Filter "*.log" -ErrorAction Stop | Where-Object { $_.LastWriteTime -ge $cut } | Sort-Object LastWriteTime if (-not $logs) { Add-Finding "INFO" "CBS logs" "No .log files modified in the last $CbsHours h under $cbsDir" } else { # Ordered signature catalogue $cats = @( @{ Name = "Chain not valid (1633)"; Rx = "The chain does not seem valid|not eligible HRESULT_FROM_WIN32\(1633\)" }, @{ Name = "IMDS timeout (12002)"; Rx = "HRESULT_FROM_WIN32\(12002\)" }, @{ Name = "Different machine (12029)"; Rx = "different machine|HRESULT_FROM_WIN32\(12029\)" }, @{ Name = "Cert load failure"; Rx = "LoadCertificateToMemoryStore" }, @{ Name = "ESU rollback/uninstall"; Rx = "ESU: Uninstalled" } ) $combined = ($cats | ForEach-Object { $_.Rx }) -join "|" # single combined-regex pass (one pass for all signatures, vs one pass per signature) $hits = Select-String -Path ($logs.FullName) -Pattern $combined -ErrorAction SilentlyContinue if (-not $hits) { Add-Finding "INFO" "CBS signatures" "No known ESU rollback signatures found in scanned logs" } else { # CBS entries are HISTORICAL evidence, not a live pass/fail - report as WARN with the # latest timestamp so the engineer can judge whether it predates their fix/last attempt. foreach ($cat in $cats) { $catHits = @($hits | Where-Object { $_.Line -match $cat.Rx }) if ($catHits.Count -eq 0) { continue } $latest = $null foreach ($h in $catHits) { $ts = $null $m = [regex]::Match($h.Line, "^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})") if ($m.Success) { try { $ts = [datetime]::ParseExact($m.Groups[1].Value, "yyyy-MM-dd HH:mm:ss", [System.Globalization.CultureInfo]::InvariantCulture) } catch {} } if (-not $ts) { try { $ts = (Get-Item $h.Path).LastWriteTime } catch {} } if ($ts -and (-not $latest -or $ts -gt $latest)) { $latest = $ts } } if ($latest) { $whenTxt = "{0:dd/MM/yyyy HH:mm}" -f $latest } else { $whenTxt = "time unknown" } Add-Finding "WARN" ("CBS signature: " + $cat.Name) ("seen $($catHits.Count) time(s), latest $whenTxt - historical evidence; confirm it predates your fix / last attempt, not the current state") } Add-Finding "INFO" "CBS note" "CBS signatures are past events, not the live verdict. The current health is the chain build + endpoint checks above." } } } catch { Add-Finding "INFO" "CBS logs" "Scan error: $($_.Exception.Message)" } # ---------------------------------------------------------------------------- summary Write-Section "SUMMARY & RECOMMENDED ACTION" $fails = $script:Findings | Where-Object { $_.Level -eq "FAIL" } $warns = $script:Findings | Where-Object { $_.Level -eq "WARN" } Write-Host (" FAIL: {0} WARN: {1} PASS: {2}" -f ` ($fails | Measure-Object).Count, ($warns | Measure-Object).Count, ` (($script:Findings | Where-Object { $_.Level -eq 'PASS' }) | Measure-Object).Count) -ForegroundColor White Write-Host "" # Decision tree mapped to the KI if ($agentVersion -and $agentVersion -lt [version]"1.40") { Write-Host " >> Agent is below 1.40. Upgrade the agent + install the latest SSU, then rename license.json and restart himds." -ForegroundColor Yellow } elseif ($script:PrimaryCause -eq "NotTimeValid") { Write-Host " >> ROOT CAUSE: a certificate is EXPIRED or the system clock is wrong (NotTimeValid)." -ForegroundColor Yellow Write-Host " ACTION: confirm the signing cert NotAfter and the system time. On a live machine restart himds" -ForegroundColor Yellow Write-Host " so the license/cert is re-issued; if it persists, escalate. (An old captured license shows this.)" -ForegroundColor Yellow } elseif ($script:PrimaryCause -eq "Revocation") { Write-Host " >> ROOT CAUSE: revocation check cannot complete (certs are fine)." -ForegroundColor Yellow Write-Host " The CRL/OCSP endpoint is unreachable - almost certainly a proxy/firewall block." -ForegroundColor Yellow Write-Host " ACTION: allowlist http://www.microsoft.com/pkiops/ (certs, crl, ocsp) over HTTP/80," -ForegroundColor Yellow Write-Host " no SSL inspection, no proxy auth. Re-test, then retry the update." -ForegroundColor Yellow } elseif ($script:PrimaryCause -eq "MissingCert") { Write-Host " >> ROOT CAUSE: a certificate in the signing chain is missing/untrusted." -ForegroundColor Yellow Write-Host " ACTION: install the missing intermediate/root flagged above (see the troubleshooting guide)," -ForegroundColor Yellow Write-Host " then re-run this script - the chain should build to DigiCert Global Root G2." -ForegroundColor Yellow } elseif ($script:PrimaryCause -eq "Healthy") { Write-Host " >> Certificate chain + revocation are healthy. If the update still rolls back, check" -ForegroundColor Yellow Write-Host " the CBS signatures above (IMDS 12002 / himds / agent connectivity) or engage the" -ForegroundColor Yellow Write-Host " Windows servicing team per the support boundary." -ForegroundColor Yellow } else { Write-Host " >> Could not fully evaluate the chain (license/cert missing, or other ChainStatus)." -ForegroundColor Yellow Write-Host " Resolve the FAIL items above first, then re-run." -ForegroundColor Yellow } Write-Host "" Write-Host " Reference: https://learn.microsoft.com/azure/azure-arc/servers/troubleshoot-extended-security-updates" -ForegroundColor DarkGray Write-Host "" # ---------------------------------------------------------------------- collect bundle if ($CollectZip) { Write-Section "Collecting diagnostic bundle" try { $stamp = Get-Date -Format "yyyyMMdd-HHmmss" $host_ = $env:COMPUTERNAME $work = Join-Path $env:TEMP ("ArcEsuDiag_{0}_{1}" -f $host_, $stamp) New-Item -ItemType Directory -Force -Path $work | Out-Null # 00 - findings report + verdict $rep = New-Object System.Text.StringBuilder [void]$rep.AppendLine("Arc-enabled ESU chain / rollback diagnostic") [void]$rep.AppendLine("Reference: https://learn.microsoft.com/azure/azure-arc/servers/troubleshoot-extended-security-updates") [void]$rep.AppendLine(("Machine : {0}" -f $host_)) [void]$rep.AppendLine(("Run time: {0:dd/MM/yyyy HH:mm:ss}" -f (Get-Date))) [void]$rep.AppendLine(("Primary cause: {0}" -f $script:PrimaryCause)) [void]$rep.AppendLine(("License : {0}" -f $LicensePath)) [void]$rep.AppendLine("") [void]$rep.AppendLine(("{0,-6} {1}" -f "LEVEL","CHECK / DETAIL")) [void]$rep.AppendLine(("-" * 90)) foreach ($f in $script:Findings) { [void]$rep.AppendLine(("{0,-6} {1}" -f $f.Level, $f.Check)) if ($f.Detail) { [void]$rep.AppendLine((" -> {0}" -f $f.Detail)) } } Set-Content -Path (Join-Path $work "00_report.txt") -Value $rep.ToString() -Encoding UTF8 # 01 - environment $envTxt = @() try { $envTxt += (Get-CimInstance Win32_OperatingSystem | Select-Object Caption,Version,OSArchitecture,LastBootUpTime | Format-List | Out-String) } catch {} $envTxt += "PowerShell: $($PSVersionTable.PSVersion)" $envTxt += "Local time: $(Get-Date -Format 'dd/MM/yyyy HH:mm:ss') ($([System.TimeZoneInfo]::Local.DisplayName))" Set-Content -Path (Join-Path $work "01_environment.txt") -Value ($envTxt -join "`r`n") -Encoding UTF8 # 02 - agent try { $exe = "C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe" if (Test-Path $exe) { (& $exe version 2>&1) | Out-File (Join-Path $work "02_azcmagent_version.txt") (& $exe show 2>&1) | Out-File (Join-Path $work "02_azcmagent_show.txt") } } catch {} # 03 - hotfixes try { Get-HotFix | Sort-Object InstalledOn -Descending | Format-Table -AutoSize | Out-String | Set-Content (Join-Path $work "03_hotfixes.txt") } catch {} # 04 - chain detail (rebuild for the file) if ($signingCert) { $ct = New-Object System.Text.StringBuilder foreach ($mode in @("Online","NoCheck")) { try { $cc = New-Object System.Security.Cryptography.X509Certificates.X509Chain $cc.ChainPolicy.RevocationMode = $mode $okc = $cc.Build($signingCert) [void]$ct.AppendLine("=== RevocationMode=$mode Build=$okc ===") foreach ($el in $cc.ChainElements) { [void]$ct.AppendLine(" Issuer: " + $el.Certificate.Issuer) } foreach ($st in $cc.ChainStatus) { [void]$ct.AppendLine((" STATUS: {0} - {1}" -f $st.Status, $st.StatusInformation.Trim())) } [void]$ct.AppendLine("") } catch { [void]$ct.AppendLine(" build error ($mode): $($_.Exception.Message)") } } Set-Content -Path (Join-Path $work "04_chain.txt") -Value $ct.ToString() -Encoding UTF8 try { [System.IO.File]::WriteAllBytes((Join-Path $work "10_signingcert.cer"), $signingCert.RawData) } catch {} } # 05 - cert stores try { certutil -store Root 2>&1 | Out-File (Join-Path $work "05_certstore_root.txt") } catch {} try { certutil -store CA 2>&1 | Out-File (Join-Path $work "05_certstore_ca.txt") } catch {} # 06 - winhttp proxy try { netsh winhttp show proxy 2>&1 | Out-File (Join-Path $work "06_winhttp_proxy.txt") } catch {} # 07 - url caches try { certutil -urlcache CRL 2>&1 | Out-File (Join-Path $work "07_urlcache_crl.txt") } catch {} try { certutil -urlcache OCSP 2>&1 | Out-File (Join-Path $work "07_urlcache_ocsp.txt") } catch {} # 08 - certutil verify if ($signingCert) { try { $tmp = Join-Path $work "10_signingcert.cer" if (Test-Path $tmp) { certutil -f -urlfetch -verify $tmp 2>&1 | Out-File (Join-Path $work "08_certutil_verify.txt") } } catch {} } # 09 - root auto-update reg try { reg query "HKLM\SOFTWARE\Policies\Microsoft\SystemCertificates\AuthRoot" 2>&1 | Out-File (Join-Path $work "09_reg_AuthRoot.txt") } catch {} # 11 - CBS ESU lines (filtered, not the whole multi-MB logs) try { $cbsDir = "C:\Windows\Logs\CBS" $cut = (Get-Date).AddHours(-1 * $CbsHours) $logs = Get-ChildItem -Path $cbsDir -Filter "*.log" -ErrorAction SilentlyContinue | Where-Object { $_.LastWriteTime -ge $cut } if ($logs) { Select-String -Path ($logs.FullName) -Pattern "ESU:|chain does not seem valid|not eligible|different machine|LoadCertificate|HRESULT_FROM_WIN32|Extended Security Updates AI installer|rollback" -ErrorAction SilentlyContinue | ForEach-Object { "{0}({1}): {2}" -f (Split-Path $_.Path -Leaf), $_.LineNumber, $_.Line.Trim() } | Set-Content (Join-Path $work "11_cbs_esu_lines.txt") -Encoding UTF8 } } catch {} # copy the license file itself (small, useful for support) try { if (Test-Path $LicensePath) { Copy-Item $LicensePath (Join-Path $work "license.json") -ErrorAction SilentlyContinue } } catch {} # resolve output zip path if (-not $OutputPath) { $desk = [Environment]::GetFolderPath('Desktop') if (-not $desk -or -not (Test-Path $desk)) { $desk = $env:TEMP } $OutputPath = Join-Path $desk ("ArcEsuDiag_{0}_{1}.zip" -f $host_, $stamp) } $zipped = New-ZipFromDir -Dir $work -Zip $OutputPath Remove-Item $work -Recurse -Force -ErrorAction SilentlyContinue if ($zipped -and (Test-Path $OutputPath)) { Add-Finding "PASS" "Diagnostic bundle" $OutputPath Write-Host (" Bundle saved: {0}" -f $OutputPath) -ForegroundColor Green } else { Add-Finding "WARN" "Diagnostic bundle" "Could not create zip - artifacts left in $work" } } catch { Add-Finding "WARN" "Diagnostic bundle" "Collection failed: $($_.Exception.Message)" } Write-Host "" } |