[BUG]: VMSS agents go offline unexpectedly due to 401 auth error

[angaaruriakhil]

What happened?

Context: We're hosting VMSS in Azure with images built with runner-images code as Azure DevOps agents. This is the source code for Azure DevOps MS hosted agents. We use the operating systems: Windows 2022, Windows 2019, Ubuntu 22.04, Ubuntu 20.04 and they are all affected by this issue. The issue happens intermittently across scale sets hosted in multiple regions across all the operating systems mentioned.

We intermittently are seeing VMs failing with the error for at least the last month in the Diagnostics tab for each pool in Azure DevOps:

Pipeline agent went offline unexpectedly

Which then will cause the VM to go offline and skip the jobs it may be running. This is causing big problems for us as VMs unexpectedly go offline and if unlucky, this could be while they are running a business critical job. This has been happening for at least a month.

We have checked our proxy/all firewalls for networking blocks and there are no blocks reported from our VMSS subnets to any destination or port. All the outbound traffic being executed is allowed.

Saving an unhealthy Ubuntu 22.04 agent for investigation and investigating the logs under /agent/_diag shows that there is a 401 error (scrubbed excepts attached in log box, I don't want to share sensitive information). See the logs box for relevant logs.

We have similarly looked at the log files under:

• /var/log/waagent.log
• /var/log/azure/Microsoft.VisualStudio.Services.TeamServicesAgentLinux/enableagent.log
• /var/log/azure/Microsoft.VisualStudio.Services.TeamServicesAgentLinux/extension.log

All of which report nothing out of the ordinary.

There are similar issues reporting this problem here. #4826

As reported in #4826 , running the ./run.sh --diagnostics command, also reports an error writing to the log.

Versions

Azure DevOps Services
Images built with runner-images on October 15th 2024
Azure Pipelines Agent v3.246.0
WA Linux Agent v2.11.1.12

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu 22.04. Ubuntu 20.04, Windows 2022, Windows 2019

Version controll system

No response

Relevant log output

[2024-10-25 10:35:40Z WARN VisualStudioServices] GET request to https://dev.azure.com/{scrubbed}/_apis/distributedtask/pools/1166/messages timed out after 60 seconds.
[2024-10-25 10:36:15Z WARN VisualStudioServices] GET request to https://dev.azure.com/{scrubbed}/_apis/distributedtask/pools/1166/messages timed out after 60 seconds.
[2024-10-25 10:36:15Z ERR  MessageListener] System.TimeoutException: The HTTP request timed out after 00:01:00.
[2024-10-25 10:36:15Z INFO MessageListener] Retriable exception: The HTTP request timed out after 00:01:00.
[2024-10-25 10:36:15Z ERR  Terminal] WRITE ERROR: 2024-10-25 10:36:15Z: Agent connect error: The HTTP request timed out after 00:01:00.. Retrying until reconnected.

[2024-10-25 11:20:38Z INFO MessageListener] Sleeping for 13.802 seconds before retrying.
[2024-10-25 11:21:42Z INFO MessageListener] Sleeping for 7.404 seconds before retrying.
[2024-10-25 11:21:43Z INFO MessageListener] Sent GetAgentMessage to keep alive agent 926980, session '{scrubbed}'.
[2024-10-25 11:22:13Z WARN VisualStudioServices] Authentication failed with status code 401.

[2024-10-25 11:31:36Z INFO MessageListener] No message retrieved from session '{scrubbed}' within last 30 minutes.
Results from ./run.sh --diagnostics:

System.UnauthorizedAccessException: Access to the path '/agent/_diag/Agent_20241025-190446-utc.log' is denied. ---> System.IO.IOException: Permission denied --- End of inner exception stack trace --- at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func2 errorRewriter)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
at Microsoft.VisualStudio.Services.Agent.HostTraceListener.CreatePageLogWriter() in /mnt/vss/_work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostTraceListener.cs:line 178
at Microsoft.VisualStudio.Services.Agent.HostTraceListener..ctor(String logFileDirectory, String logFilePrefix, Int32 pageSizeLimit, Int32 retentionDays) in /mnt/vss/_work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostTraceListener.cs:line 50
at Microsoft.VisualStudio.Services.Agent.HostContext..ctor(HostType hostType, String logFile) in /mnt/vss/_work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostContext.cs:line 135
at Microsoft.VisualStudio.Services.Agent.Listener.Program.Main(String[] args) in /mnt/vss/_work/1/s/src/Agent.Listener/Program.cs:line 28
./run.sh: line 68: 24614 Aborted (core dumped) "$DIR"/bin/Agent.Listener run $*
`

[DenisNikulin5]

Hi @angaaruriakhil
The agent should acquire a new token if the old one is expired

Do you see any attempts in the logs before the VM is terminated?
It should look like this:

[2024-10-28 12:39:06Z WARN VisualStudioServices] Authentication failed with status code 401.
[2024-10-28 12:39:06Z VERB VisualStudioServices] Started authentication
[2024-10-28 12:39:06Z VERB VisualStudioServices] OAuth issued token provider instance 4920083 is acquiring a token
...
[2024-10-28 12:39:07Z VERB VisualStudioServices] OAuth issued token provider instance 4920083 acquired new token instance 38778824
[2024-10-28 12:39:07Z VERB VisualStudioServices] Finished authentication
[2024-10-28 12:39:07Z VERB VisualStudioServices] OAuth issued token provider instance 4920083 validated token instance 38778824

[angaaruriakhil]

Hi @DenisNikulin5 , No I don't see that in the logs.

It basically loops through this block of code and goes back to 401.

Let me know if more details are required, I scrubbed a lot to protect the details.

[2024-10-24 19:42:05Z WARN VisualStudioServices] Authentication failed with status code 401.
Cache-Control: no-cache
Pragma: no-cache
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
WWW-Authenticate: Bearer authorization_uri=https://login.microsoftonline.com/{scrubbed}, Basic realm="https://tfsprodcus6.visualstudio.com/", TFS-Federated
X-TFS-ProcessId: {scrubbed}
Strict-Transport-Security: max-age=31536000; includeSubDomains
ActivityId:  {scrubbed}
X-TFS-Session:  {scrubbed}
X-VSS-E2EID:  {scrubbed}
X-VSS-SenderDeploymentId:  {scrubbed}
X-Frame-Options: SAMEORIGIN
X-TFS-FedAuthRealm: https://tfsprodcus6.visualstudio.com/
X-TFS-FedAuthIssuer: https://dev.azure.com/{scrubbed}
X-VSS-AuthorizationEndpoint: https://vssps.dev.azure.com/{scrubbed}
X-VSS-ResourceTenant:  {scrubbed}
X-VSS-S2STargetService:  {scrubbed}/visualstudio.com
X-TFS-FedAuthRedirect: https://spsprodcus5.vssps.visualstudio.com/_signin?realm={scrubbed}
Request-Context: appId=cid-v1:{scrubbed}
Access-Control-Expose-Headers: Request-Context
X-Content-Type-Options: nosniff
X-Cache: CONFIG_NOCACHE
X-MSEdge-Ref: Ref A: {scrubbed} Ref B: CHGEDGE1009 Ref C: 2024-10-24T19:42:03Z
Date: Thu, 24 Oct 2024 19:42:03 GMT

[2024-10-24 19:42:03Z INFO RSAFileKeyManager] Loading RSA key parameters from file /agent/.credentials_rsaparams
[2024-10-24 19:42:03Z INFO RSAFileKeyManager] Loading RSA key parameters from file /agent/.credentials_rsaparams
[2024-10-24 19:42:04Z INFO VisualStudioServices] AAD Correlation ID for this token request: Unknown
[2024-10-24 19:42:04Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2024-10-24 19:42:04Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2024-10-24 19:42:04Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2024-10-24 19:42:04Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2024-10-24 19:42:04Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2024-10-24 19:42:04Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2024-10-24 19:42:05Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2024-10-24 19:42:05Z INFO CommandSettings] Arg 'work': ''
[2024-10-24 19:42:05Z INFO CommandSettings] Flag 'unattended': 'True'
[2024-10-24 19:42:05Z INFO PromptManager] ReadValue
[2024-10-24 19:42:05Z INFO CommandSettings] Flag 'disableloguploads': 'False'
[2024-10-24 19:42:05Z INFO CommandSettings] Flag 'restreamlogstofiles': 'False'
[2024-10-24 19:42:05Z INFO CommandSettings] Flag 'alwaysextracttask': 'False'
[2024-10-24 19:42:05Z INFO ConfigurationStore] Saving agent settings.
[2024-10-24 19:42:05Z INFO ConfigurationStore] Settings Saved.
[2024-10-24 19:42:05Z INFO Terminal] WRITE LINE: 2024-10-24 19:42:05Z: Settings Saved.
[2024-10-24 19:42:05Z INFO SystemDControlManager] Service name 'vsts.agent. {scrubbed}.Scale Set Pool - Linux {scrubbed}.vmsslnap{scrubbed}.service' display name 'Azure Pipelines Agent ({scrubbed}.Scale Set Pool - Linux {scrubbed}.vmsslnap{scrubbed})' will be used for service configuration.
[2024-10-24 19:42:05Z INFO HostContext] Well known directory 'Bin': '/agent/bin'
[2024-10-24 19:42:05Z INFO HostContext] Well known directory 'Root': '/agent'
[2024-10-24 19:42:05Z INFO HostContext] Well known directory 'Bin': '/agent/bin'
[2024-10-24 19:42:05Z INFO HostContext] Well known directory 'Bin': '/agent/bin'
[2024-10-24 19:42:05Z INFO HostContext] Well known directory 'Root': '/agent'
[2024-10-24 19:42:05Z INFO UnixUtil] Which: 'chmod'
[2024-10-24 19:42:05Z INFO UnixUtil] Location: '{matches.First()}'
[2024-10-24 19:42:05Z INFO UnixUtil] Running /usr/bin/chmod 755 "/agent/svc.sh"
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper] Starting process:
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   File name: '/usr/bin/chmod'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   Arguments: '755 "/agent/svc.sh"'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   Working directory: '/agent'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   Require exit code zero: 'True'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   Encoding web name:  ; code page: ''
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   Redirected STDIN: 'False'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   Persist current code page: 'False'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   Keep redirected STDIN open: 'False'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   High priority process: 'False'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   ContinueAfterCancelProcessTreeKillAttempt: 'False'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   Sigint timeout: '00:00:07.5000000'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   Sigterm timeout: '00:00:02.5000000'
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper]   Try to use graceful shutdown: False
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper] Updated oom_score_adj to 500 for PID: 4583.
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper] Process started with process id 4583, waiting for process exit.
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper] Exited process 4583 with exit code 0
[2024-10-24 19:42:05Z INFO ProcessInvokerWrapper] Finished process 4583 with exit code 0, and elapsed time 00:00:00.0023222.
[2024-10-24 19:42:05Z INFO ConfigurationStore] HasCredentials()
[2024-10-24 19:42:05Z INFO ConfigurationStore] stored True
[2024-10-24 19:42:05Z INFO CredentialManager] GetCredentialProvider
[2024-10-24 19:42:05Z INFO CredentialManager] Creating type OAuth
[2024-10-24 19:42:05Z INFO CredentialManager] Creating credential type: OAuth
[2024-10-24 19:42:05Z INFO RSAFileKeyManager] Loading RSA key parameters from file /agent/.credentials_rsaparams
[2024-10-24 19:42:05Z INFO ConfigurationManager] LoadSettings
[2024-10-24 19:42:05Z INFO ConfigurationStore] IsConfigured()
[2024-10-24 19:42:05Z INFO ConfigurationStore] IsConfigured: True
[2024-10-24 19:42:05Z INFO ConfigurationManager] Is configured: True
[2024-10-24 19:42:05Z INFO ConfigurationStore] Read setting file: 225 chars
[2024-10-24 19:42:05Z INFO ConfigurationManager] Settings Loaded
[2024-10-24 19:42:05Z WARN VisualStudioServices] Authentication failed with status code 401.

[HAldermo-Magnet]

We're seeing the same issue with a new VMSS Agent Pool. Existing pools seems unaffected.
Ubuntu 24.04 built using Azure Image builder. The same image works fine in other pools.

2024-12-10 14:19:34Z INFO AgentCapabilitiesProvider] Adding 'Agent.Name': 'xxxxxd00000E'
[2024-12-10 14:19:34Z INFO AgentCapabilitiesProvider] Adding 'Agent.OS': 'Linux'
[2024-12-10 14:19:34Z INFO AgentCapabilitiesProvider] Adding 'Agent.OSArchitecture': 'X64'
[2024-12-10 14:19:34Z INFO AgentCapabilitiesProvider] Adding 'InteractiveSession': 'True'
[2024-12-10 14:19:34Z INFO AgentCapabilitiesProvider] Adding 'Agent.Version': '3.248.0'
[2024-12-10 14:19:34Z INFO AgentCapabilitiesProvider] Adding 'Agent.ComputerName': 'xxxxxd00000E'
[2024-12-10 14:19:34Z INFO HostContext] Well known directory 'Bin': '/agent/bin'
[2024-12-10 14:19:34Z INFO HostContext] Well known directory 'Root': '/agent'
[2024-12-10 14:19:34Z INFO AgentCapabilitiesProvider] Adding 'Agent.HomeDirectory': '/agent'
[2024-12-10 14:19:34Z INFO Terminal] WRITE LINE: Connecting to the server.
[2024-12-10 14:19:35Z INFO Terminal] WRITE LINE: Successfully added the agent
[2024-12-10 14:19:35Z INFO ConfigurationManager] Agent server url resolve by server: 'https://dev.azure.com/xxxxx-dev/'.
[2024-12-10 14:19:35Z INFO ConfigurationStore] Saving OAuth credential @ /agent/.credentials
[2024-12-10 14:19:35Z INFO ConfigurationStore] Credentials Saved.
[2024-12-10 14:19:35Z INFO Terminal] WRITE LINE: Testing agent connection.
[2024-12-10 14:19:35Z INFO ConfigurationStore] HasCredentials()
[2024-12-10 14:19:35Z INFO ConfigurationStore] stored True
[2024-12-10 14:19:35Z INFO CredentialManager] GetCredentialProvider
[2024-12-10 14:19:35Z INFO CredentialManager] Creating type OAuth
[2024-12-10 14:19:35Z INFO CredentialManager] Creating credential type: OAuth
[2024-12-10 14:19:35Z INFO RSAFileKeyManager] Loading RSA key parameters from file /agent/.credentials_rsaparams
[2024-12-10 14:19:35Z INFO AgentServer] Establish connection with 100 seconds timeout.
[2024-12-10 14:19:35Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2024-12-10 14:19:35Z WARN VisualStudioServices] Authentication failed with status code 401.

[Steveiwonder]

Hey all, is there any temporary work around for this issue?

[mjblue]

Also seeing this issue, have rebuilt on latest ubuntu2204 images on 5/12 and the problem is still existing

[sicil1ano]

Hi. Like @angaaruriakhil mentioned in the OP, there's an issue with AzDevOps user used by the agent when running the ./run.sh --diagnostics command, because the user does not have enough permissions to use the agent directory.
The 401 errors are a red herring. The agents don't go offline due to 401 errors, but instead because of lack of adequate permissions not granted to AzDevOps to use agent directory.

adminuser@vmss-agents-142:/agent$ ./run.sh --diagnostics
Unhandled exception. System.UnauthorizedAccessException: Access to the path '/agent/_diag/Agent_20240604-105127-utc.log' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at Microsoft.VisualStudio.Services.Agent.HostTraceListener.CreatePageLogWriter() in /mnt/vss/_work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostTraceListener.cs:line 178
   at Microsoft.VisualStudio.Services.Agent.HostTraceListener..ctor(String logFileDirectory, String logFilePrefix, Int32 pageSizeLimit, Int32 retentionDays) in /mnt/vss/_work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostTraceListener.cs:line 50
   at Microsoft.VisualStudio.Services.Agent.HostContext..ctor(HostType hostType, String logFile) in /mnt/vss/_work/1/s/src/Microsoft.VisualStudio.Services.Agent/HostContext.cs:line 135
   at Microsoft.VisualStudio.Services.Agent.Listener.Program.Main(String[] args) in /mnt/vss/_work/1/s/src/Agent.Listener/Program.cs:line 28
./run.sh: line 68:  3425 Aborted                 (core dumped) "$DIR"/bin/Agent.Listener run $*

I executed the sudo chmod -R a+rwx command for the agent directory (be careful: it assigns read/write/execute permissions to ALL users recursively) and the agent finally worked properly.
Please note that the 401 errors are STILL reported in the logs, but they are not blocking, like I mentioned here.
I created another issue to solve this problem.

[yenneferofvengerberg]

We have the same issue on our containers randomly running the same image with agent (4.248.0). The logs are the same as others posted

[2025-01-12 08:51:33Z INFO ConfigurationStore] HasCredentials()
[2025-01-12 08:51:33Z INFO ConfigurationStore] stored True
[2025-01-12 08:51:33Z INFO CredentialManager] GetCredentialProvider
[2025-01-12 08:51:33Z INFO CredentialManager] Creating type OAuth
[2025-01-12 08:51:33Z INFO CredentialManager] Creating credential type: OAuth
[2025-01-12 08:51:33Z INFO RSAFileKeyManager] Loading RSA key parameters from file /azp/.credentials_rsaparams
[2025-01-12 08:51:33Z INFO Terminal] WRITE LINE: Connecting to the server.
[2025-01-12 08:51:33Z INFO MessageListener] Attempt to create session.
[2025-01-12 08:51:33Z INFO MessageListener] Connecting to the Agent Server...
[2025-01-12 08:51:33Z INFO AgentServer] Establish connection with 100 seconds timeout.
[2025-01-12 08:51:33Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2025-01-12 08:51:33Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2025-01-12 08:51:33Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2025-01-12 08:51:33Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2025-01-12 08:51:33Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2025-01-12 08:51:33Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2025-01-12 08:51:33Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2025-01-12 08:51:33Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2025-01-12 08:51:33Z INFO MessageListener] VssConnection created
[2025-01-12 08:51:33Z WARN VisualStudioServices] Authentication failed with status code 401.
Cache-Control: no-cache
Pragma: no-cache
P3P: CP="CAO DSP COR ADMa DEV CONo TELo CUR PSA PSD TAI IVDo OUR SAMi BUS DEM NAV STA UNI COM INT PHY ONL FIN PUR LOC CNT"
WWW-Authenticate: Bearer authorization_uri=https://login.microsoftonline.com/***REDACTED***, Basic realm="https://tfsprodweu5.visualstudio.com/", TFS-Federated
X-TFS-ProcessId: ***REDACTED***
Strict-Transport-Security: max-age=31536000; includeSubDomains
ActivityId: ***REDACTED***
X-TFS-Session: ***REDACTED***
X-VSS-E2EID: ***REDACTED***
X-VSS-SenderDeploymentId: ***REDACTED***
X-Frame-Options: SAMEORIGIN
X-TFS-FedAuthRealm: https://tfsprodweu5.visualstudio.com/
X-TFS-FedAuthIssuer: https://dev.azure.com/***REDACTED***/
X-VSS-AuthorizationEndpoint: https://vssps.dev.azure.com/***REDACTED***/
X-VSS-ResourceTenant: ***REDACTED***
X-VSS-S2STargetService: ***REDACTED***/visualstudio.com
X-TFS-FedAuthRedirect: https://spsprodweu4.vssps.visualstudio.com/_signin?realm=dev.azure.com&reply_to=https%3A%2F%2Fdev.azure.com%2F***REDACTED***%2F_apis%2Fdistributedtask%2Fpools%2F16%2Fsessions&redirect=**REDACTED**
Request-Context: appId=cid-v1:***REDACTED***
Access-Control-Expose-Headers: Request-Context
X-Content-Type-Options: nosniff
X-Cache: CONFIG_NOCACHE
X-MSEdge-Ref: Ref A: ***REDACTED*** Ref B: ROM30EDGE0121 Ref C: 2025-01-12T08:51:33Z
Date: Sun, 12 Jan 2025 08:51:32 GMT

[2025-01-12 08:51:33Z INFO RSAFileKeyManager] Loading RSA key parameters from file /azp/.credentials_rsaparams
[2025-01-12 08:51:33Z INFO RSAFileKeyManager] Loading RSA key parameters from file /azp/.credentials_rsaparams
[2025-01-12 08:51:34Z ERR  VisualStudioServices] POST request to https://vssps.dev.azure.com/***REDACTED***/_apis/oauth2/token failed. HTTP Status: BadRequest, AFD Ref: Ref A: ***REDACTED*** Ref B: ***REDACTED*** Ref C: 2025-01-12T08:51:33Z
[2025-01-12 08:51:34Z INFO VisualStudioServices] AAD Correlation ID for this token request: Unknown
[2025-01-12 08:51:34Z ERR  MessageListener] Catch exception during create session.
[2025-01-12 08:51:34Z ERR  MessageListener] Microsoft.VisualStudio.Services.OAuth.VssOAuthTokenRequestException: The signature is not valid.
   at Microsoft.VisualStudio.Services.OAuth.VssOAuthTokenProvider.OnGetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.IssuedTokenProvider.GetTokenOperation.GetTokenAsync(VssTraceActivity traceActivity)
   at Microsoft.VisualStudio.Services.Common.IssuedTokenProvider.GetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResourceVersion version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.Listener.MessageListener.CreateSessionAsync(CancellationToken token) in /mnt/vss/_work/1/s/src/Agent.Listener/MessageListener.cs:line 98
[2025-01-12 08:51:34Z INFO MessageListener] Retriable exception: The signature is not valid.
[2025-01-12 08:51:34Z ERR  Terminal] WRITE ERROR: 2025-01-12 08:51:34Z: Agent connect error: The signature is not valid.. Retrying until reconnected.
[2025-01-12 08:51:34Z INFO MessageListener] Sleeping for 30 seconds before retrying.
[2025-01-12 08:52:04Z INFO MessageListener] Attempt to create session.
[2025-01-12 08:52:04Z INFO MessageListener] Connecting to the Agent Server...
[2025-01-12 08:52:04Z INFO AgentServer] Establish connection with 100 seconds timeout.
[2025-01-12 08:52:04Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2025-01-12 08:52:04Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2025-01-12 08:52:04Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2025-01-12 08:52:04Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2025-01-12 08:52:04Z INFO AgentServer] Establish connection with 60 seconds timeout.
[2025-01-12 08:52:04Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2025-01-12 08:52:04Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2025-01-12 08:52:04Z INFO VisualStudioServices] Finished operation Location.GetConnectionData
[2025-01-12 08:52:04Z INFO MessageListener] VssConnection created
[2025-01-12 08:52:04Z WARN VisualStudioServices] Authentication failed with status code 401.

.run.sh --diagnostics reports no problem. Forcing chmod 777 on all files does not fix the issue. The container sits there looping to connect. Even though the error code is the same.
The issue seems not to present on hosts running ubuntu 22.04 with docker, while we see the issue on hosts running ubuntu 24.04 with containerd (both in the same kubernetes cluster)

[mgnisia]

also faced this issue, fix this at the moment with a cloud-init configuration:

#cloud-config
runcmd:
  - mkdir /agent
  - chmod -R a+rwx /agent

[Steveiwonder]

I was also experiencing this problem - It turns out unattaned-upgrade were enabled. In our logs we saw some packages being updated which trigger the walinuxagent.service to be restarted. This restart was tricking he devops backend into thinking the agent had gone offline and the vm was being destroyed.

We ran these commands

# Stop and disable unattended-upgrades service
sudo systemctl stop unattended-upgrades
sudo systemctl disable unattended-upgrades
sudo systemctl mask unattended-upgrades

# Remove unattended-upgrades package
sudo apt-get remove --purge -y unattended-upgrades

# Disable auto-updates in APT configuration
sudo sed -i 's/APT::Periodic::Update-Package-Lists "1";/APT::Periodic::Update-Package-Lists "0";/' /etc/apt/apt.conf.d/20auto-upgrades
sudo sed -i 's/APT::Periodic::Unattended-Upgrade "1";/APT::Periodic::Unattended-Upgrade "0";/' /etc/apt/apt.conf.d/20auto-upgrades

# Remove auto-upgrade configuration files (if exists)
sudo rm -f /etc/apt/apt.conf.d/20auto-upgrades
sudo rm -f /etc/apt/apt.conf.d/50unattended-upgrades

# Stop and disable automatic APT timers
sudo systemctl stop apt-daily.timer
sudo systemctl stop apt-daily-upgrade.timer
sudo systemctl disable apt-daily.timer
sudo systemctl disable apt-daily-upgrade.timer

[Sheludchenko]

@Steveiwonder's solution worked for me. The only thing I've changed is removed Disable auto-updates in APT configuration and Remove auto-upgrade configuration files because the folders weren't there in my case.

[EppO]

I concur, removing unattended-upgrades package on Ubuntu 24.04 prevents the agent to go offline after a little while. It didn't happen to me on 22.04 though.