Restore-AzWebAppSnapshot fails when large number of resources exist (sometimes) #26268
Description
Description
Restore-AzWebAppSnapshot can fail when a large number of resources exist in the subscription.
Restore-AzWebAppSnapshot calls the "resources" API which returns paginated results.
If the Web App of the Snapshot is not returned on the first "page" of results, the restore operation will fail with an InternalServerError
Note: a "filter" is passed to this API call, however, it appears that the Resources API is first retrieving all results, then only filtering using the first page.
I suspect the above as I have tested the API call using the Azure Resource Explorer (https://resources.azure.com/subscriptions/[subscription-guid-redacted]/resources)
We have 24 different apps and only have consistent issues restoring 5 of them with this call.
All 5 of the affected web apps do not appear on the first "page" of results in Azure Resource Explorer, while all working apps DO appear on the first page.
In addition, we have one web app that works some of the time.
Refreshing the results in Azure Resource Explorer shows that sometimes this app appears on the first page and sometimes it does not.
Note: In the debug output below you will see the call to the Resources API
https://management.azure.com/subscriptions/[redacted]/resources?$filter=resourceType eq 'Microsoft.Web/sites' and name eq '[redacted]'&api-version=2016-09-01
Observe it returns an empty array with a "nextLink"
Issue script & Debug output
$snapshot = (Get-AzWebAppSnapshot -Name $WebAppName -ResourceGroupName $ResourceGroupName)[0]
DEBUG: ============================ HTTP REQUEST ============================
HTTP Method:
GET
Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/snapshots?api-version=2021-01-15
Headers:
x-ms-client-request-id : 8b732e64-9590-447f-8ef0-a81b8b8f317c
Accept-Language : en-US
Body:
DEBUG: ============================ HTTP RESPONSE ============================
Status Code:
OK
Headers:
Cache-Control : no-cache
Pragma : no-cache
ETag : "1DB0EE97209AB0B"
Strict-Transport-Security : max-age=31536000; includeSubDomains
x-ms-request-id : 86bd8e45-aff8-4a31-8c31-059e0458e97e
X-AspNet-Version : 4.0.30319
X-Powered-By : ASP.NET
x-ms-ratelimit-remaining-subscription-reads: 249
x-ms-ratelimit-remaining-subscription-global-reads: 3749
x-ms-correlation-request-id : d27badb3-c7e0-474e-9c7e-44c996ba1411
x-ms-routing-request-id : EASTUS:20241010T204514Z:d27badb3-c7e0-474e-9c7e-44c996ba1411
X-Content-Type-Options : nosniff
X-Cache : CONFIG_NOCACHE
X-MSEdge-Ref : Ref A: E117A506F88B46559CD0664687396A12 Ref B: MNZ221060610029 Ref C: 2024-10-10T20:45:13Z
Date : Thu, 10 Oct 2024 20:45:14 GMT
Body:
{
"value": [
{
"id": "/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/snapshots/2024-09-11T00_25_46_4226984",
"name": "2024-09-11T00:25:46.4226984",
"type": "Microsoft.Web/sites/snapshots",
"properties": {
"time": "2024-09-11T00:25:46.4226984"
}
},
...
]
}Restore-AzWebAppSnapshot -Name $TargetWebAppName -ResourceGroupName $TargetResourceGroupName -InputObject $snapshot -Force -RecoverConfiguration -Debug
DEBUG: ============================ HTTP REQUEST ============================
HTTP Method:
GET
Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resources?$filter=resourceType eq 'Microsoft.Web/sites' and name eq '[redacted]'&api-version=2016-09-01
Headers:
x-ms-client-request-id : bb36d532-556d-4e7a-90a4-f7f55f0bafad
Accept-Language : en-US
Body:
DEBUG: ============================ HTTP RESPONSE ============================
Status Code:
OK
Headers:
Cache-Control : no-cache
Pragma : no-cache
x-ms-ratelimit-remaining-subscription-reads: 249
x-ms-ratelimit-remaining-subscription-global-reads: 3749
x-ms-request-id : 52ce3c1d-ac04-4795-bd9a-84a4e949f186
x-ms-correlation-request-id : 52ce3c1d-ac04-4795-bd9a-84a4e949f186
x-ms-routing-request-id : EASTUS:20241010T204621Z:52ce3c1d-ac04-4795-bd9a-84a4e949f186
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-Content-Type-Options : nosniff
X-Cache : CONFIG_NOCACHE
X-MSEdge-Ref : Ref A: E4B43C8E16E345328AA4FB9B3127D5CB Ref B: MNZ221060618037 Ref C: 2024-10-10T20:46:21Z
Date : Thu, 10 Oct 2024 20:46:20 GMT
Body:
{
"value": [],
"nextLink": "https://management.azure.com/subscriptions/[redacted]/resources?%24filter=resourceType+eq+%27Microsoft.Web%2fsites%27+and+name+eq+%27[redacted]%27&api-version=2016-09-01&%24skiptoken=rZFNa4NAEED%2fi2cPm9VcAj20XWMNzoplzXb3JkbqR1TQFaOh%2f73bIC2BhID0OgNvHvPORp2elJ%2fXZWdszkbdtCpL0lq18bHXEyMdd738yBDj2zHG%2b9ErmpxWoSU4DHQCRXk00hEhwUrb584JsFCiEEhU0TogyRowDF6NngzT6Jr%2bMVvwENMpQcCelWRg0VeEAu6MPvscBHcUMAeBC3ZAwKYsQjM7jTvVd%2fiOsHTDUZJwLXmkaLU9Sg2FybkICyYUJTKnTOayEBYtDtUMffQHgWUGPFwBdpR2Wf24SlIO%2bg8ocDUWvxd6d5LEmWSxy65c76gK7NlAIksWiQrYSwm5Vq32lc%2b9AapIBS6sxJRYAckyIIf8wvwy%2fyoG9VtzPOiW8xDitkxbfe067ea%2fwprXWW9yl0Q156Q3gUuCmr85byKXxJwd7%2fCWhPwG"
}
DEBUG: ============================ HTTP REQUEST ============================
HTTP Method:
POST
Absolute Uri:
https://management.azure.com/subscriptions/[redacted]/resourceGroups/[redacted]/providers/Microsoft.Web/sites/[redacted]/restoreSnapshot?api-version=2021-01-15
Headers:
x-ms-client-request-id : bb36d532-556d-4e7a-90a4-f7f55f0bafad
Accept-Language : en-US
Body:
{
"properties": {
"snapshotTime": "2024-10-10T19:49:23",
"recoverySource": {},
"overwrite": true,
"recoverConfiguration": true,
"ignoreConflictingHostNames": true,
"useDRSecondary": false
}
}
...
DEBUG: ============================ HTTP RESPONSE ============================
Status Code:
InternalServerError
Headers:
Cache-Control : no-cache
Cache-Control : no-cache
Pragma : no-cache
Strict-Transport-Security : max-age=31536000; includeSubDomains
X-AspNet-Version : 4.0.30319
X-Powered-By : ASP.NET
x-ms-failure-cause : service
x-ms-ratelimit-remaining-subscription-writes: 199
x-ms-ratelimit-remaining-subscription-global-writes: 2999
x-ms-request-id : e65257ab-3bba-4119-8811-d379bf759616
x-ms-correlation-request-id : e65257ab-3bba-4119-8811-d379bf759616
x-ms-routing-request-id : EASTUS:20241010T204644Z:e65257ab-3bba-4119-8811-d379bf759616
X-Content-Type-Options : nosniff
X-Cache : CONFIG_NOCACHE
X-MSEdge-Ref : Ref A: D5FB6060669A470A849BE4CBC9E966BF Ref B: MNZ221060610047 Ref C: 2024-10-10T20:46:44Z
Date : Thu, 10 Oct 2024 20:46:43 GMT
Body:
{
"Message": "An error has occurred."
}Environment data
Name Value
---- -----
PSVersion 7.2.23
PSEdition Core
GitCommitId 7.2.23
OS Microsoft Windows 10.0.22621
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
Module versions
ModuleType Version PreRelease Name ExportedCommands
---------- ------- ---------- ---- ----------------
Script 3.0.2 Az.Accounts {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault…}
Script 5.1.0 Az.KeyVault {Add-AzKeyVaultCertificate, Add-AzKeyVaultCertificateContact, Add-AzKeyVaultKey, Add-AzKe…
Script 6.13.0 Az.Resources {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment…}
Script 1.2.2 Az.TrafficManager {Add-AzTrafficManagerCustomHeaderToEndpoint, Add-AzTrafficManagerCustomHeaderToProfile, A…
Script 3.1.2 Az.Websites {Add-AzWebAppAccessRestrictionRule, Add-AzWebAppTrafficRouting, Edit-AzWebAppBackupConfig…
Error output
Message : Operation returned an invalid status code 'InternalServerError'
StackTrace : at Microsoft.Azure.Management.WebSites.WebAppsOperations.BeginRestoreSnapshotWithHttpMessagesAsync(String resourceGroupName, String name,
SnapshotRestoreRequest restoreRequest, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at Microsoft.Azure.Management.WebSites.WebAppsOperations.RestoreSnapshotWithHttpMessagesAsync(String resourceGroupName, String name,
SnapshotRestoreRequest restoreRequest, Dictionary`2 customHeaders, CancellationToken cancellationToken)
at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.RestoreSnapshotAsync(IWebAppsOperations operations, String
resourceGroupName, String name, SnapshotRestoreRequest restoreRequest, CancellationToken cancellationToken)
at Microsoft.Azure.Management.WebSites.WebAppsOperationsExtensions.RestoreSnapshot(IWebAppsOperations operations, String
resourceGroupName, String name, SnapshotRestoreRequest restoreRequest)
at Microsoft.Azure.Commands.WebApps.Utilities.WebsitesClient.RestoreSnapshot(String resourceGroupName, String webSiteName, String
slotName, SnapshotRestoreRequest restoreReq)
at Microsoft.Azure.Commands.WebApps.Cmdlets.BackupRestore.RestoreAzureWebAppSnapshot.<>c__DisplayClass17_0.<ExecuteCmdlet>b__0()
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ConfirmAction(Boolean force, String continueMessage, String
processMessage, String target, Action action, Func`1 useShouldContinue)
at Microsoft.Azure.Commands.ResourceManager.Common.AzureRMCmdlet.ConfirmAction(Boolean force, String continueMessage, String
processMessage, String target, Action action)
at Microsoft.Azure.Commands.WebApps.Cmdlets.BackupRestore.RestoreAzureWebAppSnapshot.ExecuteCmdlet()
at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.<>c__3`1.<ExecuteSynchronouslyOrAsJob>b__3_0(T c)
at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet, Action`1 executor)
at Microsoft.WindowsAzure.Commands.Utilities.Common.CmdletExtensions.ExecuteSynchronouslyOrAsJob[T](T cmdlet)
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()
Exception : Microsoft.Azure.Management.WebSites.Models.DefaultErrorResponseException
InvocationInfo : {Restore-AzWebAppSnapshot}
Line : Restore-AzWebAppSnapshot -Name [redacted] -ResourceGroupName [redacted] -InputObject $snapshot -Force
-RecoverConfiguration -Debug