AWS Lambda .NET Core - AzureDevOps failing - Unable to install global Amazon.Lambda.Tools

[3GDXC]

Describe the bug

##[error]Unable to install global Amazon.Lambda.Tools! The old package based version of Amazon.Lambda.Tools is now 
deprecated. Newer .NET core versions will need to use a newer hosted agent and the global tools (which this task auto 
installs). Refer to Microsoft's guide for the correct hosted agent for which hosted agent you need to use newer 
.NET Core versions:https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted

To reproduce
Add a deployment step to an existing pipeline as the last step

Expected behavior
after a successful build the deployment step should succeed.

Screenshots
Your Environment
Additional context

[hunterwerlla]

Can you give additional context on what build image you are using, if you are trying to use the global tools version, and your environment?

[3GDXC]

I'm using AzureDevOps with the default Azure Agent Pool, image of ubuntu 16.04; I have added the AWS for DevOps tooling via the market place on the organizations account.

[3GDXC]

If required I'm able to provide the full logs of the build/publish; however these are large.

[3GDXC]

this may be linked to issue #316

[3GDXC]

FYI, here is a more complete log output

2020-03-31T02:26:14.6646955Z ##[section]Starting: deploy .net core to lambda:  
2020-03-31T02:26:14.6676720Z ==============================================================================
2020-03-31T02:26:14.6677079Z Task         : AWS Lambda .NET Core
2020-03-31T02:26:14.6677624Z Description  : Builds, packages and deploys a .NET Core AWS Lambda function or serverless application. Optionally, the task can create a deployment package for subsequent deployment in another build or release pipeline
2020-03-31T02:26:14.6678142Z Version      : 1.6.0
2020-03-31T02:26:14.6678377Z Author       : Amazon Web Services
2020-03-31T02:26:14.6680493Z Help         : Please refer to [AWS Lambda Developer Guide](https://docs.aws.amazon.com/lambda/latest/dg/) for more information on working with AWS Lambda.

More information on this task can be found in the [task reference](https://docs.aws.amazon.com/vsts/latest/userguide/lambda-netcore-deploy.html).

####Task Permissions
This task requires permissions to call the following AWS service APIs (depending on selected task options, not all APIs may be used):
* lambda:CreateFunction
* lambda:UpdateFunctionCode
* lambda:GetFunctionConfiguration
* cloudformation:CreateChangeSet
* cloudformation:ExecuteChangeSet
* cloudformation:DescribeStackEvents
* cloudformation:DeleteStack
* cloudformation:DescribeChangeSet
* cloudformation:DescribeStacks
* s3:CreateBucket
* s3:GetBucketLocation

The task also requires permissions to upload your Lambda function or serverless application content to the specified Amazon S3 bucket. Depending on the size of the application bundle, either putObject or the S3 multi-part upload APIs may be used.
2020-03-31T02:26:14.6682748Z ==============================================================================
2020-03-31T02:26:14.9044849Z Configuring credentials for task
2020-03-31T02:26:14.9049127Z ...configuring AWS credentials from service endpoint 'da885225-9bf5-49ff-b29d-cbf5c9d1b9a9'
2020-03-31T02:26:14.9049871Z ...endpoint defines standard access/secret key credentials
2020-03-31T02:26:14.9096013Z Processing Lambda project at /home/vsts/work/1/s/src/DXC.BackOffice.ServiceHost
2020-03-31T02:26:14.9097350Z Reading existing aws-lambda-tools-defaults.json
2020-03-31T02:26:14.9098046Z Clearing out profile default so task credentials will be used.
2020-03-31T02:26:14.9105333Z Configuring region for task
2020-03-31T02:26:14.9109130Z ...configured to use region **eu-west-2**, defined in task.
2020-03-31T02:26:14.9155793Z [command]/usr/bin/dotnet tool install -g Amazon.Lambda.Tools
2020-03-31T02:26:16.7302584Z **/usr/share/dotnet/sdk/3.1.101/NuGet.targets(123,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/dxc-technology-ncle/_packaging/dxc-technology-ncle/nuget/v3/index.json.** [**/tmp/00vmq4is.fdf/restore.csproj**]
2020-03-31T02:26:16.7303788Z **/usr/share/dotnet/sdk/3.1.101/NuGet.targets(123,5): error :   Response status code does not indicate success: 401 (Unauthorized)**. [/tmp/00vmq4is.fdf/restore.csproj]
2020-03-31T02:26:16.7630902Z The tool package could not be restored.
2020-03-31T02:26:16.7631690Z Tool 'amazon.lambda.tools' failed to install. This failure may have been caused by:
2020-03-31T02:26:16.7631985Z 
2020-03-31T02:26:16.7632640Z * You are attempting to install a preview release and did not use the --version option to specify the version.
2020-03-31T02:26:16.7633238Z * A package by this name was found, but it was not a .NET Core tool.
2020-03-31T02:26:16.7633947Z * The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem.
2020-03-31T02:26:16.7634266Z * You mistyped the name of the tool.
2020-03-31T02:26:16.7634435Z 
2020-03-31T02:26:16.7635112Z For more reasons, including package naming enforcement, visit https://aka.ms/failure-installing-tool
2020-03-31T02:26:16.7731470Z 
2020-03-31T02:26:16.7782688Z [command]/usr/bin/dotnet tool update -g Amazon.Lambda.Tools
2020-03-31T02:26:18.3894891Z /usr/share/dotnet/sdk/3.1.101/NuGet.targets(123,5): error : Unable to load the service index for source https://pkgs.dev.azure.com/dxc-technology-ncle/_packaging/dxc-technology-ncle/nuget/v3/index.json. [/tmp/gp30sj20.rvz/restore.csproj]
2020-03-31T02:26:18.3896551Z /usr/share/dotnet/sdk/3.1.101/NuGet.targets(123,5): error :   Response status code does not indicate success: 401 (Unauthorized). [/tmp/gp30sj20.rvz/restore.csproj]
2020-03-31T02:26:18.4272389Z Tool 'amazon.lambda.tools' failed to update due to the following:
2020-03-31T02:26:18.4272770Z The tool package could not be restored.
2020-03-31T02:26:18.4273467Z Tool 'amazon.lambda.tools' failed to install. This failure may have been caused by:
2020-03-31T02:26:18.4273689Z 
2020-03-31T02:26:18.4274138Z * You are attempting to install a preview release and did not use the --version option to specify the version.
2020-03-31T02:26:18.4274530Z * A package by this name was found, but it was not a .NET Core tool.
2020-03-31T02:26:18.4274878Z * The required NuGet feed cannot be accessed, perhaps because of an Internet connection problem.
2020-03-31T02:26:18.4275193Z * You mistyped the name of the tool.
2020-03-31T02:26:18.4275345Z 
2020-03-31T02:26:18.4275798Z For more reasons, including package naming enforcement, visit https://aka.ms/failure-installing-tool
2020-03-31T02:26:18.4318547Z 
2020-03-31T02:26:18.4341065Z ##[error]Unable to install global Amazon.Lambda.Tools! The old package based version of Amazon.Lambda.Tools is now deprecated. Newer .NET core versions will need to use a newer hosted agent and the global tools (which this task auto installs). Refer to Microsoft's guide for the correct hosted agent for which hosted agent you need to use newer .NET Core versions:https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted
2020-03-31T02:26:18.4345930Z [command]/usr/bin/dotnet restore
2020-03-31T02:26:19.4393117Z   Restore completed in 36.27 ms for /home/vsts/work/1/s/src/DXC.BackOffice.ServiceHost/DXC.BackOffice.ServiceHost.csproj.
2020-03-31T02:26:19.4750227Z 
2020-03-31T02:26:19.4759878Z Beginning Serverless Deployment
2020-03-31T02:26:19.4793122Z [command]/usr/bin/dotnet lambda deploy-serverless --stack-name dxc-backoffice-stack-azuredevops --region eu-west-2 --s3-bucket dxc-backoffice-stack-s3-azuredevops --disable-interactive true
2020-03-31T02:26:19.9709084Z Could not execute because the specified command or file was not found.
2020-03-31T02:26:19.9709753Z Possible reasons for this include:
2020-03-31T02:26:19.9710774Z   * You misspelled a built-in dotnet command.
2020-03-31T02:26:19.9711823Z   * You intended to execute a .NET Core program, but dotnet-lambda does not exist.
2020-03-31T02:26:19.9712720Z   * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.
2020-03-31T02:26:19.9761116Z 
2020-03-31T02:26:19.9772187Z ##[error]Error: The process '/usr/bin/dotnet' failed with exit code 1
2020-03-31T02:26:19.9795899Z ##[section]Finishing: deploy .net core to lambda:  

the items highlighted in bold are very odd, give the build service has been given FULL access to the organizations artifact feed and the previous step in the build pipeline publishes to the same feed?

[3GDXC]

more info; I created a task that deletes the nuget.config after the build/publish but prior to the Lambda task; the follow is the output of the log

2020-03-31T02:49:06.2457368Z ##[section]Starting: deploy .net core to lambda:  
2020-03-31T02:49:06.2463435Z ==============================================================================
2020-03-31T02:49:06.2463761Z Task         : AWS Lambda .NET Core
2020-03-31T02:49:06.2464306Z Description  : Builds, packages and deploys a .NET Core AWS Lambda function or serverless application. Optionally, the task can create a deployment package for subsequent deployment in another build or release pipeline
2020-03-31T02:49:06.2464840Z Version      : 1.6.0
2020-03-31T02:49:06.2465061Z Author       : Amazon Web Services
2020-03-31T02:49:06.2467256Z Help         : Please refer to [AWS Lambda Developer Guide](https://docs.aws.amazon.com/lambda/latest/dg/) for more information on working with AWS Lambda.

More information on this task can be found in the [task reference](https://docs.aws.amazon.com/vsts/latest/userguide/lambda-netcore-deploy.html).

####Task Permissions
This task requires permissions to call the following AWS service APIs (depending on selected task options, not all APIs may be used):
* lambda:CreateFunction
* lambda:UpdateFunctionCode
* lambda:GetFunctionConfiguration
* cloudformation:CreateChangeSet
* cloudformation:ExecuteChangeSet
* cloudformation:DescribeStackEvents
* cloudformation:DeleteStack
* cloudformation:DescribeChangeSet
* cloudformation:DescribeStacks
* s3:CreateBucket
* s3:GetBucketLocation

The task also requires permissions to upload your Lambda function or serverless application content to the specified Amazon S3 bucket. Depending on the size of the application bundle, either putObject or the S3 multi-part upload APIs may be used.
2020-03-31T02:49:06.2469542Z ==============================================================================
2020-03-31T02:49:06.4911364Z Configuring credentials for task
2020-03-31T02:49:06.4912669Z ...configuring AWS credentials from service endpoint 'da885225-9bf5-49ff-b29d-cbf5c9d1b9a9'
2020-03-31T02:49:06.4913099Z ...endpoint defines standard access/secret key credentials
2020-03-31T02:49:06.4958790Z Processing Lambda project at /home/vsts/work/1/s/src/DXC.BackOffice.ServiceHost
2020-03-31T02:49:06.4959519Z Reading existing aws-lambda-tools-defaults.json
2020-03-31T02:49:06.4959903Z Clearing out profile default so task credentials will be used.
2020-03-31T02:49:06.4968276Z Configuring region for task
2020-03-31T02:49:06.4969168Z ...configured to use region eu-west-2, defined in task.
2020-03-31T02:49:06.5001414Z [command]/usr/bin/dotnet tool install -g Amazon.Lambda.Tools
2020-03-31T02:49:09.5327798Z Since you just installed the .NET Core SDK, you will need to logout or restart your session before running the tool you installed.
2020-03-31T02:49:09.5339993Z You can invoke the tool using the following command: dotnet-lambda
2020-03-31T02:49:09.5341691Z Tool 'amazon.lambda.tools' (version '4.0.0') was successfully installed.
2020-03-31T02:49:09.5453054Z 
2020-03-31T02:49:09.5504848Z [command]/usr/bin/dotnet restore
2020-03-31T02:49:11.0542859Z   Restore completed in 365.73 ms for /home/vsts/work/1/s/src/DXC.BackOffice.ServiceHost/DXC.BackOffice.ServiceHost.csproj.
2020-03-31T02:49:11.0895255Z 
2020-03-31T02:49:11.0902592Z Beginning Serverless Deployment
2020-03-31T02:49:11.0925788Z [command]/usr/bin/dotnet lambda deploy-serverless --stack-name dxc-backoffice-stack-azuredevops --region eu-west-2 --s3-bucket dxc-backoffice-stack-s3-azuredevops --disable-interactive true
2020-03-31T02:49:11.6358162Z Could not execute because the specified command or file was not found.
2020-03-31T02:49:11.6359124Z Possible reasons for this include:
2020-03-31T02:49:11.6360220Z   * You misspelled a built-in dotnet command.
2020-03-31T02:49:11.6361093Z   * You intended to execute a .NET Core program, but dotnet-lambda does not exist.
2020-03-31T02:49:11.6362152Z   * You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH.
2020-03-31T02:49:11.6409390Z 
2020-03-31T02:49:11.6440259Z ##[error]Error: The process '/usr/bin/dotnet' failed with exit code 1
2020-03-31T02:49:11.6468399Z ##[section]Finishing: deploy .net core to lambda:  

[3GDXC]

could this be the root issue ?

[command]/usr/bin/dotnet tool install -g Amazon.Lambda.Tools
Since you just installed the .NET Core SDK, you will need to logout or restart your session before running the tool you installed.
You can invoke the tool using the following command: dotnet-lambda
Tool 'amazon.lambda.tools' (version '4.0.0') was successfully installed.

[hunterwerlla]

That looks suspicious, it seems this is a pretty common issue in Azure Pipelines: dotnet/sdk#8999

[3GDXC]

@hunterwerlla yes, I would agree the second part of this issue around not finding the installed global tools looks to be a common AzureDevOps issue; the however the bug with regards the nuget.config is not great a work-around of deleting the file prior to the Lambda step seems to work (ie. allow the installation of the global tool to progress)

[3GDXC]

@hunterwerlla dotnet/sdk#9777 seems to suggest using the -tool-path switch do you know of this?

[fimiki]

@3GDXC, after thrashing around for a day trying to work around this problem using the ubuntu-20.04 Azure-hosted build agent, I eventually just added a command line script task immediately prior to the AWS Lamda .NET Core task, with the following script:

sudo dotnet tool install --tool-path=/usr/bin Amazon.Lambda.Tools --version 4.0.0

This worked. I'm not sure why the AWS task doesn't work and fails to find the global install of Amazon.Lambda.Tools . The default install location for dotnet global tools ($HOME/.dotnet/tools/) IS in the $PATH variable, and I tried doing a --tool-path install of Amazon.Lambda.Tools to that location just in case it was landing elsewhere, but still got the error: "You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH".

Since /usr/bin is the location from which dotnet-lambda deploy-serverless is run, forcing the tool to be installed in that location means the command will pick it up. I had to use "sudo" since the user doesn't have access to /usr/bin by default.

[gpomykala]

@3GDXC, after thrashing around for a day trying to work around this problem using the ubuntu-20.04 Azure-hosted build agent, I eventually just added a command line script task immediately prior to the AWS Lamda .NET Core task, with the following script:

sudo dotnet tool install --tool-path=/usr/bin Amazon.Lambda.Tools --version 4.0.0

This worked. I'm not sure why the AWS task doesn't work and fails to find the global install of Amazon.Lambda.Tools . The default install location for dotnet global tools ($HOME/.dotnet/tools/) IS in the $PATH variable, and I tried doing a --tool-path install of Amazon.Lambda.Tools to that location just in case it was landing elsewhere, but still got the error: "You intended to run a global tool, but a dotnet-prefixed executable with this name could not be found on the PATH".

Since /usr/bin is the location from which dotnet-lambda deploy-serverless is run, forcing the tool to be installed in that location means the command will pick it up. I had to use "sudo" since the user doesn't have access to /usr/bin by default.

Thanks, it solved my problem as well.