Skip to content

[wasm][debugger][tests] Fix crash in EvaluateOnCallFrameTests on CI #69073

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 10, 2022

Conversation

radical
Copy link
Member

@radical radical commented May 9, 2022

In TestHarnessProxy, we recently started adding proxy instances to a
static table, to enable getting the correct exit state from the client.
But the instances were not removed from the table, which prevented the
GC from collecting them.

This manifests as the test process crashing on systems with 8GB of
memory, when running EvaluateOnCallFrameTests. It would end up getting
to a resident size of 2.2-2.7G, and then crash after about 30 tests. With these changes, the
proxy gets collected, and the resident memory hovers around 500M.

Future work: this will change once we start using the upcoming app host
work, and thus moving the proxy to a separate process via dotnet run --debug.

radical added 2 commits May 9, 2022 20:36
In `TestHarnessProxy`, we recently started adding proxy instances to a
static table, to enable getting the correct exit state from the client.
But the instances were not removed from the table, which prevented the
GC from collecting them.

This manifests as the test process crashing on systems with 8GB of
memory, when running `EvaluateOnCallFrameTests`. It would end up getting
to a resident size of 2.2-2.7G, and then crash after about 30 tests. With these changes, the
proxy gets collected, and the resident memory hovers around 500M.

Future work: this will change once we start using the upcoming app host
work, and thus moving the proxy to a separate process via `dotnet run
--debug`.
@ghost ghost assigned radical May 9, 2022
@ghost ghost added the area-Debugger-mono label May 9, 2022
@ghost
Copy link

ghost commented May 9, 2022

Tagging subscribers to this area: @thaystg
See info in area-owners.md if you want to be subscribed.

Issue Details

In TestHarnessProxy, we recently started adding proxy instances to a
static table, to enable getting the correct exit state from the client.
But the instances were not removed from the table, which prevented the
GC from collecting them.

This manifests as the test process crashing on systems with 8GB of
memory, when running EvaluateOnCallFrameTests. It would end up getting
to a resident size of 2.2-2.7G, and then crash after about 30 tests. With these changes, the
proxy gets collected, and the resident memory hovers around 500M.

Future work: this will change once we start using the upcoming app host
work, and thus moving the proxy to a separate process via dotnet run --debug.

Author: radical
Assignees: radical
Labels:

area-Debugger-mono

Milestone: -

@radical radical added arch-wasm WebAssembly architecture test-bug Problem in test source code (most likely) labels May 9, 2022
@ghost
Copy link

ghost commented May 9, 2022

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

In TestHarnessProxy, we recently started adding proxy instances to a
static table, to enable getting the correct exit state from the client.
But the instances were not removed from the table, which prevented the
GC from collecting them.

This manifests as the test process crashing on systems with 8GB of
memory, when running EvaluateOnCallFrameTests. It would end up getting
to a resident size of 2.2-2.7G, and then crash after about 30 tests. With these changes, the
proxy gets collected, and the resident memory hovers around 500M.

Future work: this will change once we start using the upcoming app host
work, and thus moving the proxy to a separate process via dotnet run --debug.

Author: radical
Assignees: radical
Labels:

arch-wasm, test-bug, area-Debugger-mono

Milestone: -

@radical radical requested review from lewing and thaystg May 9, 2022 22:00
@radical radical marked this pull request as ready for review May 9, 2022 22:00
@radical radical requested a review from marek-safar as a code owner May 9, 2022 22:00
@radical
Copy link
Member Author

radical commented May 9, 2022

/azp run runtime-wasm

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@radical
Copy link
Member Author

radical commented May 10, 2022

The wasm/library test failures are #69085 .

@radical
Copy link
Member Author

radical commented May 10, 2022

The coreclr failures seem intermittent. And are unrelated to this PR.

/Users/runner/work/1/s/.dotnet/sdk/7.0.100-preview.3.22179.4/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/Users/runner/work/1/s/artifacts/log/Release/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/Users/runner/work/1/s/artifacts/toolset/7.0.0-beta.22255.2.txt /t:__WriteToolsetLocation /warnaserror /Users/runner/work/1/s/artifacts/toolset/restore.proj
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error : System.AggregateException: One or more errors occurred. (A task was canceled.)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :  ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Threading.Tasks.Task.GetExceptions(Boolean includeTaskCanceledExceptions)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.NuGetSdkResolver.NuGetSdkResolver.NuGetAbstraction.GetSdkResult(SdkReference sdk, Object nuGetVersion, SdkResolverContext context, SdkResultFactory factory)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.NuGetSdkResolver.NuGetSdkResolver.Resolve(SdkReference sdkReference, SdkResolverContext resolverContext, SdkResultFactory factory)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.SdkResolution.SdkResolverService.ResolveSdk(Int32 submissionId, SdkReference sdk, LoggingContext loggingContext, ElementLocation sdkReferenceLocation, String solutionPath, String projectPath, Boolean interactive, Boolean isRunningInVisualStudio)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.SdkResolution.CachingSdkResolverService.<>c__DisplayClass3_0.<ResolveSdk>b__2()
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Lazy`1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Lazy`1.CreateValue()
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.SdkResolution.CachingSdkResolverService.ResolveSdk(Int32 submissionId, SdkReference sdk, LoggingContext loggingContext, ElementLocation sdkReferenceLocation, String solutionPath, String projectPath, Boolean interactive, Boolean isRunningInVisualStudio)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.Evaluation.Evaluator`4.ExpandAndLoadImportsFromUnescapedImportExpressionConditioned(String directoryOfImportingFile, ProjectImportElement importElement, List`1& projects, SdkResult& sdkResult, Boolean throwOnFileNotExistsError)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.Evaluation.Evaluator`4.ExpandAndLoadImports(String directoryOfImportingFile, ProjectImportElement importElement, SdkResult& sdkResult)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.Evaluation.Evaluator`4.EvaluateImportElement(String directoryOfImportingFile, ProjectImportElement importElement)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.Evaluation.Evaluator`4.PerformDepthFirstPass(ProjectRootElement currentProjectOrImport)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.Evaluation.Evaluator`4.Evaluate()
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.Evaluation.Evaluator`4.Evaluate(IEvaluatorData`4 data, Project project, ProjectRootElement root, ProjectLoadSettings loadSettings, Int32 maxNodeCount, PropertyDictionary`1 environmentProperties, ILoggingService loggingService, IItemFactory`2 itemFactory, IToolsetProvider toolsetProvider, ProjectRootElementCacheBase projectRootElementCache, BuildEventContext buildEventContext, ISdkResolverService sdkResolverService, Int32 submissionId, EvaluationContext evaluationContext, Boolean interactive)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.Execution.ProjectInstance.Initialize(ProjectRootElement xml, IDictionary`2 globalProperties, String explicitToolsVersion, String explicitSubToolsetVersion, Int32 visualStudioVersionFromSolution, BuildParameters buildParameters, ILoggingService loggingService, BuildEventContext buildEventContext, ISdkResolverService sdkResolverService, Int32 submissionId, Nullable`1 projectLoadSettings, EvaluationContext evaluationContext)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.Execution.ProjectInstance..ctor(String projectFile, IDictionary`2 globalProperties, String toolsVersion, BuildParameters buildParameters, ILoggingService loggingService, BuildEventContext buildEventContext, ISdkResolverService sdkResolverService, Int32 submissionId, Nullable`1 projectLoadSettings)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.BuildRequestConfiguration.<>c__DisplayClass60_0.<LoadProjectIntoConfiguration>b__0()
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.BuildRequestConfiguration.InitializeProject(BuildParameters buildParameters, Func`1 loadProjectFromFile)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.RequestBuilder.BuildProject()
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.RequestBuilder.BuildProject()
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport()
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport()
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.RequestBuilder.RequestThreadProc(Boolean setThreadParameters)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.RequestBuilder.RequestThreadProc(Boolean setThreadParameters)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Threading.Tasks.Task`1.InnerInvoke()
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Threading.Tasks.Task.ExecuteEntry()
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.BackEnd.RequestBuilder.DedicatedThreadsTaskScheduler.<InjectThread>b__6_0()
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Threading.Thread.StartHelper.Callback(Object state)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error : --- End of stack trace from previous location ---
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error : 
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    --- End of inner exception stack trace ---
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error :    at Microsoft.Build.NuGetSdkResolver.NuGetSdkResolver.NuGetAbstraction.GetSdkResult(SdkReference sdk, Object nuGetVersion, SdkResolverContext context, SdkResultFactory factory)
/Users/runner/work/1/s/artifacts/toolset/restore.proj : error MSB4236: The SDK 'Microsoft.DotNet.Arcade.Sdk' specified could not be found.
Build failed with exit code 1. Check errors above.

@radical radical merged commit f730657 into dotnet:main May 10, 2022
@radical radical deleted the dbg-crash-fix branch May 10, 2022 04:34
@ghost ghost locked as resolved and limited conversation to collaborators Jun 9, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-wasm WebAssembly architecture area-Debugger-mono test-bug Problem in test source code (most likely)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants