Targets PowerShell 7.4+ on Linux, macOS, or Windows. The retry logic uses
[System.Net.Http.HttpRequestException]and[System.Net.Sockets.SocketException], which exist in both 5.1 and 7 but behave differently the code below is written against 7's semantics. If you're stuck on 5.1, the fallback catch on[System.Net.WebException]covers the older stack.
Invoke-RestMethod is one of the most-used cmdlets in operations scripting, and one of the most misused. The typical shape is a single line: $data = Invoke-RestMethod -Uri $url -Headers $headers. That works for exactly as long as the API stays up, refuses to rate-limit you, never times out, never has a load balancer that resets a connection, never returns a 5xx, and never paginates. In other words: never in production.
The rest of this post builds a resilient REST client the kind you'd actually run on a cron: exponential backoff with jitter, Retry-After honoured for 429 responses, idempotency keys for retryable writes, generic pagination, a Pester test suite you can trust, and TLS verification that stays on. Every function follows the same begin / process / end skeleton the rest of this blog uses, with try/catch around each external call.
What "Resilient" Actually Requires
A REST client is resilient when it satisfies all of the following at once:
- Retry transient failures. Network resets, DNS blips, 5xx responses, 429 rate-limits. Retry the specific ones that are safe to repeat.
- Back off exponentially, with jitter. Fixed intervals synchronise every client on the fleet into a thundering herd against the API's next uptime window. Randomised delays don't.
- Respect
Retry-After. When a server tells you when to come back, come back then not sooner, not on your own schedule. - Bound every attempt with a timeout. Without a timeout, a hung TCP connection blocks your pipeline until the OS decides otherwise typically minutes.
- Never retry non-idempotent writes blindly. A
POSTthat timed out may or may not have committed on the server. Use anIdempotency-Keyheader when the API supports it; refuse to retry when it doesn't. - Classify errors, not just count them. A 401 is not "try again in a second"; it's "your token is bad, stop." A 400 is "your payload is wrong, stop." A 502 is "try again". The retry policy has to know the difference.
Anything less is a script that works nine times out of ten and locks up on the tenth.
The Base Client Invoke-ResilientRestMethod
The core wrapper. Every other function in this post calls this one; the auth and pagination helpers layer on top.
function Invoke-ResilientRestMethod
{
<#
.SYNOPSIS
Wraps Invoke-RestMethod with retries, exponential backoff with jitter,
Retry-After honouring, and per-attempt timeouts.
.DESCRIPTION
Retries only on error classes that are safe to retry: 408, 425, 429,
500, 502, 503, 504, and network-level exceptions (DNS, connection
reset, socket timeout). Never retries 4xx client errors other than
the timeout / rate-limit codes. Honours the server's Retry-After
header when present (integer seconds or HTTP-date).
.PARAMETER Method
HTTP verb. POST/PUT/PATCH/DELETE are only retried when
-IdempotencyKey is supplied.
.PARAMETER IdempotencyKey
Optional value written into the 'Idempotency-Key' header. Required
for retrying non-GET methods.
.OUTPUTS
The response body (as returned by Invoke-RestMethod).
#>
[CmdletBinding()]
param
(
[Parameter(Mandatory = $true)]
[ValidateSet('GET','POST','PUT','PATCH','DELETE','HEAD','OPTIONS')]
[string]
$Method,
[Parameter(Mandatory = $true)]
[Uri]
$Uri,
[Parameter()]
[hashtable]
$Headers = @{},
[Parameter()]
[object]
$Body,
[Parameter()]
[string]
$ContentType = 'application/json',
[Parameter()]
[Microsoft.PowerShell.Commands.WebRequestSession]
$Session,
[Parameter()]
[int]
$MaxAttempts = 5,
[Parameter()]
[int]
$BaseDelayMs = 500,
[Parameter()]
[int]
$MaxDelayMs = 30000,
[Parameter()]
[int]
$TimeoutSec = 30,
[Parameter()]
[string]
$IdempotencyKey
)
begin
{
Write-Verbose -Message ('[Invoke-ResilientRestMethod] Begin {0} {1}' -f $Method, $Uri)
$retriableStatus = @(408, 425, 429, 500, 502, 503, 504)
$nonIdempotentVerbs = @('POST','PATCH')
$rng = [System.Random]::new()
}
process
{
if ($Method -in $nonIdempotentVerbs -and -not $IdempotencyKey)
{
# Fall through to a single attempt - the retry loop won't fire.
$singleAttempt = $true
}
else
{
$singleAttempt = $false
}
if ($IdempotencyKey)
{
$Headers['Idempotency-Key'] = $IdempotencyKey
}
for ($attempt = 1; $attempt -le $MaxAttempts; $attempt++)
{
$attemptParams = @{
Method = $Method
Uri = $Uri
Headers = $Headers
ContentType = $ContentType
TimeoutSec = $TimeoutSec
ErrorAction = 'Stop'
StatusCodeVariable = 'statusCode'
}
if ($PSBoundParameters.ContainsKey('Body'))
{
$attemptParams.Body = $Body
}
if ($Session)
{
$attemptParams.WebSession = $Session
}
try
{
$response = Invoke-RestMethod @attemptParams
Write-Verbose -Message ('[Invoke-ResilientRestMethod] {0} {1} -> {2} on attempt {3}' -f $Method, $Uri, $statusCode, $attempt)
return $response
}
catch [Microsoft.PowerShell.Commands.HttpResponseException]
{
$status = [int]$_.Exception.Response.StatusCode
if ($status -notin $retriableStatus -or $singleAttempt -or $attempt -eq $MaxAttempts)
{
Write-Error -Message ('[Invoke-ResilientRestMethod] {0} {1} returned {2} on final attempt {3}: {4}' -f $Method, $Uri, $status, $attempt, $_.Exception.Message) -ErrorAction Continue
throw
}
$delayMs = Get-BackoffDelay -Attempt $attempt -BaseMs $BaseDelayMs -MaxMs $MaxDelayMs -Response $_.Exception.Response -Rng $rng
Write-Verbose -Message ('[Invoke-ResilientRestMethod] {0} {1} returned {2} on attempt {3}; sleeping {4}ms' -f $Method, $Uri, $status, $attempt, $delayMs)
Start-Sleep -Milliseconds $delayMs
}
catch [System.Net.Http.HttpRequestException]
{
if ($singleAttempt -or $attempt -eq $MaxAttempts)
{
throw
}
$delayMs = Get-BackoffDelay -Attempt $attempt -BaseMs $BaseDelayMs -MaxMs $MaxDelayMs -Rng $rng
Write-Verbose -Message ('[Invoke-ResilientRestMethod] transport error on attempt {0}: {1}; sleeping {2}ms' -f $attempt, $_.Exception.Message, $delayMs)
Start-Sleep -Milliseconds $delayMs
}
catch [System.Net.Sockets.SocketException]
{
if ($singleAttempt -or $attempt -eq $MaxAttempts)
{
throw
}
$delayMs = Get-BackoffDelay -Attempt $attempt -BaseMs $BaseDelayMs -MaxMs $MaxDelayMs -Rng $rng
Write-Verbose -Message ('[Invoke-ResilientRestMethod] socket error on attempt {0}: {1}; sleeping {2}ms' -f $attempt, $_.Exception.Message, $delayMs)
Start-Sleep -Milliseconds $delayMs
}
catch [System.TimeoutException]
{
if ($singleAttempt -or $attempt -eq $MaxAttempts)
{
throw
}
$delayMs = Get-BackoffDelay -Attempt $attempt -BaseMs $BaseDelayMs -MaxMs $MaxDelayMs -Rng $rng
Write-Verbose -Message ('[Invoke-ResilientRestMethod] timeout on attempt {0}; sleeping {1}ms' -f $attempt, $delayMs)
Start-Sleep -Milliseconds $delayMs
}
catch
{
# Anything unclassified (auth cache exceptions, parameter validation, etc.)
# is not a candidate for retry. Bubble it out.
Write-Error -Message ('[Invoke-ResilientRestMethod] non-retriable failure: {0}' -f $_.Exception.Message) -ErrorAction Continue
throw
}
}
}
end
{
Write-Verbose -Message '[Invoke-ResilientRestMethod] End'
}
}
Two rules encoded above that trip most home-grown clients:
- Non-GETs without an idempotency key skip the retry loop. Retrying a
POSTthat already committed on the server produces duplicates the caller can't detect. Force the caller to opt in with a key. - The
Retry-Afterheader from the server wins. The backoff helper (below) reads it out of the response and skips its own computed delay when the server has spoken.
The Backoff Helper Exponential + Jitter + Retry-After
Isolating the delay math into its own function makes it independently testable and keeps Invoke-ResilientRestMethod readable.
function Get-BackoffDelay
{
<#
.SYNOPSIS
Computes the next retry delay in milliseconds.
.DESCRIPTION
Uses exponential backoff (2^attempt * base) capped at MaxMs, with
full jitter (a uniform random 0..computed). If the response
includes a Retry-After header (integer seconds or HTTP-date),
that value wins, clamped to MaxMs.
.OUTPUTS
System.Int32
#>
[CmdletBinding()]
[OutputType([int])]
param
(
[Parameter(Mandatory = $true)]
[int]
$Attempt,
[Parameter(Mandatory = $true)]
[int]
$BaseMs,
[Parameter(Mandatory = $true)]
[int]
$MaxMs,
[Parameter()]
$Response,
[Parameter(Mandatory = $true)]
[System.Random]
$Rng
)
begin
{
Write-Verbose -Message ('[Get-BackoffDelay] Begin attempt={0}' -f $Attempt)
}
process
{
if ($Response -and $Response.Headers -and $Response.Headers.Contains('Retry-After'))
{
$raw = ($Response.Headers.GetValues('Retry-After') | Select-Object -First 1)
[int] $seconds = 0
if ([int]::TryParse($raw, [ref] $seconds))
{
return [Math]::Min($seconds * 1000, $MaxMs)
}
[datetime] $when = [datetime]::MinValue
if ([datetime]::TryParse($raw, [ref] $when))
{
$delta = ($when.ToUniversalTime() - [datetime]::UtcNow).TotalMilliseconds
if ($delta -gt 0)
{
return [int][Math]::Min($delta, $MaxMs)
}
}
}
$capped = [Math]::Min([Math]::Pow(2, $Attempt) * $BaseMs, $MaxMs)
# Full jitter - uniform random between 0 and $capped.
return $Rng.Next(0, [int]$capped)
}
end
{
Write-Verbose -Message '[Get-BackoffDelay] End'
}
}
Full-jitter backoff is the pattern AWS documented years ago and it consistently outperforms fixed-interval or "exponential-with-a-small-random-nudge" schemes when many clients retry against the same recovering API. The Retry-After short-circuit means a server telling you "come back in 90 seconds" doesn't get hit at second 45 by an over-eager backoff.
Authentication OAuth Client Credentials
The most common auth mode for machine-to-machine REST calls. Cache the token, refresh before expiry, and reuse the resilient client for the token endpoint itself.
function Get-BearerToken
{
<#
.SYNOPSIS
Acquires (or reuses) an OAuth 2.0 client-credentials bearer token.
.DESCRIPTION
Caches the token in-memory keyed by (TokenUrl, ClientId, Scope).
Refreshes when the cached token has less than 60 seconds left.
Uses Invoke-ResilientRestMethod against the token endpoint so a
transient auth-service blip does not fail the outer call.
.OUTPUTS
System.String
#>
[CmdletBinding()]
[OutputType([string])]
param
(
[Parameter(Mandatory = $true)]
[Uri]
$TokenUrl,
[Parameter(Mandatory = $true)]
[string]
$ClientId,
[Parameter(Mandatory = $true)]
[System.Security.SecureString]
$ClientSecret,
[Parameter()]
[string]
$Scope
)
begin
{
Write-Verbose -Message '[Get-BearerToken] Begin'
if (-not $script:BearerTokenCache)
{
$script:BearerTokenCache = @{}
}
}
process
{
$cacheKey = '{0}|{1}|{2}' -f $TokenUrl, $ClientId, $Scope
$cached = $script:BearerTokenCache[$cacheKey]
if ($cached -and $cached.ExpiresAt -gt (Get-Date).AddSeconds(60))
{
Write-Verbose -Message '[Get-BearerToken] cache hit'
return $cached.AccessToken
}
$plain = [System.Net.NetworkCredential]::new('', $ClientSecret).Password
$body = @{
grant_type = 'client_credentials'
client_id = $ClientId
client_secret = $plain
}
if ($Scope)
{
$body.scope = $Scope
}
try
{
$reqParams = @{
Method = 'POST'
Uri = $TokenUrl
Body = $body
ContentType = 'application/x-www-form-urlencoded'
MaxAttempts = 4
}
$token = Invoke-ResilientRestMethod @reqParams
}
catch
{
throw ('Token acquisition against {0} failed: {1}' -f $TokenUrl, $_.Exception.Message)
}
$expiresIn = if ($token.expires_in) { [int]$token.expires_in } else { 3600 }
$script:BearerTokenCache[$cacheKey] = [pscustomobject] @{
AccessToken = $token.access_token
ExpiresAt = (Get-Date).AddSeconds($expiresIn)
}
$token.access_token
}
end
{
Write-Verbose -Message '[Get-BearerToken] End'
}
}
The ClientSecret is a SecureString on the boundary and is never logged. The module-scoped cache means one token acquisition per process per (URL, client, scope) triple; the refresh happens automatically when the cached token drops under 60 seconds of headroom.
Pagination Three Shapes, One Wrapper
Almost every REST API paginates one of three ways: cursor / next-token in the body, offset/limit as query parameters, or an RFC 5988 Link header. Rather than write three wrappers, expose the shape as a parameter and let one function iterate.
function Get-PaginatedResult
{
<#
.SYNOPSIS
Walks a paginated REST endpoint and streams every record.
.DESCRIPTION
Streams records as they arrive - callers can pipe into
Where-Object / Select-Object -First N without buffering the whole
result set. Supports three pagination shapes; select via -Style.
.PARAMETER Style
'Cursor' - response contains a NextCursorField pointing at the
next page; call with ?cursor=<value>.
'OffsetLimit' - append &offset=<n> to each request.
'LinkHeader' - Link header returns a rel="next" URL.
.PARAMETER NextCursorField
For Style=Cursor, the field on the response body that holds the
opaque cursor (e.g. 'nextPageToken', 'meta.nextCursor').
.PARAMETER ItemsField
Field holding the array of records on each page response
(e.g. 'value', 'items', 'results').
.OUTPUTS
The individual records, streamed.
#>
[CmdletBinding()]
param
(
[Parameter(Mandatory = $true)]
[Uri]
$Uri,
[Parameter()]
[hashtable]
$Headers = @{},
[Parameter(Mandatory = $true)]
[ValidateSet('Cursor','OffsetLimit','LinkHeader')]
[string]
$Style,
[Parameter()]
[string]
$ItemsField = 'value',
[Parameter()]
[string]
$NextCursorField,
[Parameter()]
[int]
$PageSize = 100,
[Parameter()]
[int]
$MaxPages = 1000
)
begin
{
Write-Verbose -Message ('[Get-PaginatedResult] Begin style={0}' -f $Style)
}
process
{
$currentUri = $Uri
$pageCount = 0
while ($currentUri -and $pageCount -lt $MaxPages)
{
$pageCount++
try
{
# The LinkHeader style needs the raw response to read headers,
# so use Invoke-WebRequest for that variant only.
if ($Style -eq 'LinkHeader')
{
$webParams = @{
Uri = $currentUri
Headers = $Headers
ErrorAction = 'Stop'
}
$raw = Invoke-WebRequest @webParams
$page = $raw.Content | ConvertFrom-Json
}
else
{
$page = Invoke-ResilientRestMethod -Method GET -Uri $currentUri -Headers $Headers
}
}
catch
{
throw ('Pagination failed on page {0} at {1}: {2}' -f $pageCount, $currentUri, $_.Exception.Message)
}
$items = $page.$ItemsField
if (-not $items)
{
Write-Verbose -Message ('[Get-PaginatedResult] page {0} empty; stopping' -f $pageCount)
return
}
$items
switch ($Style)
{
'Cursor'
{
$next = $page.$NextCursorField
$currentUri = if ($next) { '{0}?cursor={1}' -f $Uri, [Uri]::EscapeDataString($next) } else { $null }
}
'OffsetLimit'
{
if ($items.Count -lt $PageSize)
{
$currentUri = $null
}
else
{
$offset = $pageCount * $PageSize
$currentUri = '{0}?offset={1}&limit={2}' -f $Uri, $offset, $PageSize
}
}
'LinkHeader'
{
$linkHeader = $raw.Headers['Link']
if ($linkHeader -match '<([^>]+)>;\s*rel="next"')
{
$currentUri = [Uri]$matches[1]
}
else
{
$currentUri = $null
}
}
}
}
if ($pageCount -ge $MaxPages)
{
Write-Warning -Message ('[Get-PaginatedResult] MaxPages ({0}) reached; results may be truncated' -f $MaxPages)
}
}
end
{
Write-Verbose -Message '[Get-PaginatedResult] End'
}
}
Notice the emit-then-continue pattern: each page's items are pushed to the pipeline before the next request goes out. That means Get-PaginatedResult ... | Select-Object -First 5 really only fetches enough pages to yield five records the downstream cmdlet's short-circuit propagates upstream naturally.
Session Reuse Cookies and Connection Pooling
For any workflow that hits the same host repeatedly, feed the same WebRequestSession to every call. This shares the underlying HttpClient, reuses TCP + TLS connections, and preserves cookies.
function New-ApiSession
{
<#
.SYNOPSIS
Returns a reusable HTTP session object for downstream REST calls.
.DESCRIPTION
The session holds cookies and pools TCP connections. Pass it to
every Invoke-ResilientRestMethod call against the same host.
Pre-populates the Authorization header when Token is supplied.
.OUTPUTS
Microsoft.PowerShell.Commands.WebRequestSession
#>
[CmdletBinding()]
[OutputType([Microsoft.PowerShell.Commands.WebRequestSession])]
param
(
[Parameter()]
[string]
$UserAgent = 'PSResilientClient/1.0',
[Parameter()]
[string]
$Token
)
begin
{
Write-Verbose -Message '[New-ApiSession] Begin'
}
process
{
$session = [Microsoft.PowerShell.Commands.WebRequestSession]::new()
$session.UserAgent = $UserAgent
if ($Token)
{
$session.Headers['Authorization'] = ('Bearer {0}' -f $Token)
}
$session
}
end
{
Write-Verbose -Message '[New-ApiSession] End'
}
}
TLS Verification Never Turn It Off
The single worst pattern in REST scripting is:
# DON'T
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }
or the PowerShell 7 equivalent -SkipCertificateCheck. Both permanently disable certificate validation for the process, exposing every downstream API call to trivial man-in-the-middle interception. If you have a self-signed CA in an internal environment, add its cert to the OS trust store (or the SslProtocols/X509Chain-scoped custom callback for that one host); do not blanket-disable verification.
For certificate pinning against a known-good server certificate, use a custom X509Chain per call:
function Test-EndpointCertificate
{
<#
.SYNOPSIS
Validates that the TLS certificate presented by Uri matches an
expected SHA-256 thumbprint.
.DESCRIPTION
Opens a TCP connection, negotiates TLS, reads the leaf cert, and
compares its thumbprint to the pinned value. Does not perform any
HTTP request - use this before wiring up Invoke-ResilientRestMethod
in environments where you cannot accept a compromised CA.
.OUTPUTS
System.Management.Automation.PSCustomObject
#>
[CmdletBinding()]
[OutputType([pscustomobject])]
param
(
[Parameter(Mandatory = $true, ValueFromPipeline = $true)]
[Uri]
$Uri,
[Parameter(Mandatory = $true)]
[string]
$ExpectedSha256Thumbprint
)
begin
{
Write-Verbose -Message '[Test-EndpointCertificate] Begin'
}
process
{
$port = if ($Uri.Port -gt 0) { $Uri.Port } else { 443 }
try
{
$tcp = [System.Net.Sockets.TcpClient]::new()
$tcp.Connect($Uri.Host, $port)
$stream = $tcp.GetStream()
$ssl = [System.Net.Security.SslStream]::new($stream, $false, { $true }) # accept anything for inspection
$ssl.AuthenticateAsClient($Uri.Host)
$cert = [System.Security.Cryptography.X509Certificates.X509Certificate2]::new($ssl.RemoteCertificate)
$thumbprint = $cert.GetCertHashString('SHA256')
}
catch
{
return [pscustomobject] @{
Valid = $false
Reason = ('TLS handshake failed: {0}' -f $_.Exception.Message)
}
}
finally
{
if ($ssl) { $ssl.Dispose() }
if ($stream) { $stream.Dispose() }
if ($tcp) { $tcp.Dispose() }
}
$matched = $thumbprint -ieq $ExpectedSha256Thumbprint
[pscustomobject] @{
Valid = $matched
Reason = if ($matched) { 'Pin matched' } else { 'Thumbprint mismatch' }
ActualThumbprint = $thumbprint
Subject = $cert.Subject
NotAfter = $cert.NotAfter
}
}
end
{
Write-Verbose -Message '[Test-EndpointCertificate] End'
}
}
Run this in CI against every endpoint you rely on. Any thumbprint change is either a legitimate cert rotation (in which case update the pin and commit it) or an active MITM (in which case stop and investigate). Both should be loud.
Idempotency Keys for Retryable Writes
Some APIs accept an Idempotency-Key header on POST/PATCH requests. The server deduplicates repeat requests with the same key within a window (24h is common), returning the cached response for the second and later attempts. That lets the client retry a timed-out write safely.
function Submit-Payment
{
[CmdletBinding()]
param
(
[Parameter(Mandatory = $true)] [Uri] $ApiRoot,
[Parameter(Mandatory = $true)] [decimal]$Amount,
[Parameter(Mandatory = $true)] [string] $Recipient
)
begin
{
Write-Verbose -Message '[Submit-Payment] Begin'
}
process
{
# The idempotency key is derived from the semantic payload so a
# legitimate replay hits the same key. A random guid would let a
# crashed script re-submit a payment on next run.
$keyMaterial = '{0}|{1:F2}|{2:yyyy-MM-dd}' -f $Recipient, $Amount, (Get-Date)
$keyBytes = [System.Text.Encoding]::UTF8.GetBytes($keyMaterial)
$keyHash = [System.Security.Cryptography.SHA256]::HashData($keyBytes)
$keyString = -join ($keyHash | ForEach-Object { $_.ToString('x2') })
$reqParams = @{
Method = 'POST'
Uri = ('{0}/payments' -f $ApiRoot)
Body = @{ amount = $Amount; recipient = $Recipient } | ConvertTo-Json
IdempotencyKey = $keyString
MaxAttempts = 5
}
Invoke-ResilientRestMethod @reqParams
}
end
{
Write-Verbose -Message '[Submit-Payment] End'
}
}
The derived key is deterministic on the semantics of the call. A crash-then-resume of the outer script computes the same key, and the API deduplicates. A distinct payment on the same day would (correctly) produce a different key because the recipient or amount differ.
Testing With Pester
Retry logic is the classic case where "it worked when I tested it" means nothing there was no failure to retry. Mock Invoke-RestMethod to force each failure mode explicitly.
Describe 'Invoke-ResilientRestMethod' {
BeforeAll {
Import-Module $PSScriptRoot/../src/ResilientRest/ResilientRest.psd1 -Force
}
It 'retries a 503 up to MaxAttempts and eventually succeeds' {
$script:calls = 0
Mock -CommandName Invoke-RestMethod -MockWith {
$script:calls++
if ($script:calls -lt 3)
{
$status = 503
$ex = [Microsoft.PowerShell.Commands.HttpResponseException]::new(
'Service Unavailable',
[System.Net.Http.HttpResponseMessage]::new($status))
throw $ex
}
return @{ ok = $true }
}
$result = Invoke-ResilientRestMethod -Method GET -Uri 'https://api.example.com/x' -BaseDelayMs 1 -MaxAttempts 5
$result.ok | Should -BeTrue
$script:calls | Should -Be 3
}
It 'does NOT retry a 401' {
$script:calls = 0
Mock -CommandName Invoke-RestMethod -MockWith {
$script:calls++
$ex = [Microsoft.PowerShell.Commands.HttpResponseException]::new(
'Unauthorized',
[System.Net.Http.HttpResponseMessage]::new(401))
throw $ex
}
{ Invoke-ResilientRestMethod -Method GET -Uri 'https://api.example.com/x' -BaseDelayMs 1 -MaxAttempts 5 } |
Should -Throw
$script:calls | Should -Be 1
}
It 'refuses to retry a POST without an IdempotencyKey' {
$script:calls = 0
Mock -CommandName Invoke-RestMethod -MockWith {
$script:calls++
$ex = [Microsoft.PowerShell.Commands.HttpResponseException]::new(
'Bad Gateway',
[System.Net.Http.HttpResponseMessage]::new(502))
throw $ex
}
{ Invoke-ResilientRestMethod -Method POST -Uri 'https://api.example.com/x' -Body '{}' -BaseDelayMs 1 -MaxAttempts 5 } |
Should -Throw
$script:calls | Should -Be 1
}
It 'honours a Retry-After header' {
# Retry-After of "0" means "come back now" - three 429s and a success
# should complete in near-zero time.
$script:calls = 0
Mock -CommandName Invoke-RestMethod -MockWith {
$script:calls++
if ($script:calls -lt 4)
{
$resp = [System.Net.Http.HttpResponseMessage]::new(429)
$resp.Headers.Add('Retry-After', '0')
throw [Microsoft.PowerShell.Commands.HttpResponseException]::new('Too Many Requests', $resp)
}
return @{ ok = $true }
}
$sw = [System.Diagnostics.Stopwatch]::StartNew()
$result = Invoke-ResilientRestMethod -Method GET -Uri 'https://api.example.com/x' -MaxAttempts 5
$sw.Stop()
$result.ok | Should -BeTrue
$sw.ElapsedMilliseconds | Should -BeLessThan 500
}
}
The "does not retry a 401" and "refuses to retry a POST without an IdempotencyKey" tests are the ones you never write and desperately need. Every home-grown REST client eventually retries an auth failure eight times and hits an IP ban.
Wiring It All Together A Sample Client
Putting the pieces into one small client that talks to a rate-limited, paginated API using an OAuth-issued token:
function Get-Widget
{
[CmdletBinding()]
[OutputType([pscustomobject[]])]
param
(
[Parameter(Mandatory = $true)] [Uri] $ApiRoot,
[Parameter(Mandatory = $true)] [Uri] $TokenUrl,
[Parameter(Mandatory = $true)] [string] $ClientId,
[Parameter(Mandatory = $true)] [System.Security.SecureString] $ClientSecret,
[Parameter()] [string] $Category
)
begin
{
Write-Verbose -Message '[Get-Widget] Begin'
}
process
{
$tokenParams = @{
TokenUrl = $TokenUrl
ClientId = $ClientId
ClientSecret = $ClientSecret
Scope = 'widgets.read'
}
$token = Get-BearerToken @tokenParams
$session = New-ApiSession -Token $token
$uri = if ($Category)
{
'{0}/widgets?category={1}' -f $ApiRoot, [Uri]::EscapeDataString($Category)
}
else
{
'{0}/widgets' -f $ApiRoot
}
$pagerParams = @{
Uri = $uri
Headers = @{ Authorization = ('Bearer {0}' -f $token) }
Style = 'Cursor'
ItemsField = 'items'
NextCursorField = 'nextCursor'
PageSize = 200
}
Get-PaginatedResult @pagerParams
}
end
{
Write-Verbose -Message '[Get-Widget] End'
}
}
That's the shape a good REST client script wants: one thin function per API resource, all resilience concerns delegated to shared helpers.
Gotchas Earned the Hard Way
Invoke-RestMethodin PS 5.1 does NOT throw a typedHttpResponseException. It throws aSystem.Net.WebException. Catch that too if you support both. PS 7+ has both types.-TimeoutSecis per-attempt, not for the whole retry loop. Five attempts with a 30-second timeout is a 150-second worst case. CapMaxAttemptsaccordingly.- The default proxy is inherited from IE settings on Windows. In containers or headless Linux, an unset proxy variable can produce weird DNS failures. Set
-NoProxyexplicitly when you know the API is direct-reachable. - Cookies from
WebRequestSessiondo not clear between calls. If the API changes state via cookie (session tokens, CSRF), you want that behaviour; if it's stateless, discarding stale cookies is safer. ConvertTo-Json -Depthdefaults to 2. Silent data loss on nested payloads. Set it to20or higher for anything non-trivial.- A slow server can still respect
Retry-After. Don't override it because "the server should have been faster." If it says 60 seconds, sleep 60 seconds; a fleet of clients ignoringRetry-Afteris what killed the API in the first place. WebRequestSessionis not thread-safe. If you parallelise viaForEach-Object -Parallel, one session per parallel branch, not one session shared.
What to Do Next
Resilient REST is one of those areas where "good enough" is measurably worse than "correct": every unhandled 429 accumulates into a rate-limit ban, every timed-out POST retry can double-book a customer, every disabled TLS callback becomes a phishing-hunt three years later. The good news is that the correct pattern is small: one wrapper, one backoff helper, one paginator, and clear rules about when to retry.
Three concrete moves to bring your REST code under this pattern this week:
- Grep your repo for every naked
Invoke-RestMethodcall and count the retries. The ones with zero are your risk list. Wrap them inInvoke-ResilientRestMethodstarting with the ones on cron schedules those are the ones that page someone in the middle of the night when they don't. - Add
Test-EndpointCertificateto your CI pipeline for every internal API you own. A cert rotation shouldn't be a surprise; a MITM should be caught in seconds. The pin lives in the same YAML as the API URL. - Add the "does not retry 401" Pester test to your existing REST clients. If it fails, you have a client happily hammering an expired credential against production. Fix that specific case before touching anything else it's cheap to write, catches a class of production incident, and usually finds two or three real bugs per repo.
Pairs naturally with the SQL without an ORM post (same "narrow retry, dispose cleanly, test the failure modes" discipline applied to System.Data instead of HTTP), the secret management post (which is where the ClientSecret above should be coming from, not a config file), and the Pester tests post (for the mocking patterns that make the retry tests possible).
