[fix][EC][spark] resolve spark engine task execution failures and optimize logging#5420
Closed
v-kkhuang wants to merge 16 commits intoapache:dev-2.0.0from
Closed
[fix][EC][spark] resolve spark engine task execution failures and optimize logging#5420v-kkhuang wants to merge 16 commits intoapache:dev-2.0.0from
v-kkhuang wants to merge 16 commits intoapache:dev-2.0.0from
Conversation
Update project version to 2.0.0 in pom.xml and database scripts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…stance goes offline (#966) * #AI commit# 开发阶段: * 修复异常场景,当Entrance实例offline后,ParallelGroup缓存未更新,导致用户任务并发数计算错误 * #AI commit# 开发阶段: * 修复异常场景,当Entrance实例offline后,ParallelGroup缓存未更新,导致用户任务并发数计算错误 * #AI commit# 开发阶段: * 增加异常保护 * #AI commit# 开发阶段: * 修复异常场景,当Entrance实例offline后,ParallelGroup缓存未更新,导致用户任务并发数计算错误
…ge in hive tasks (#968) * #AI commit# 开发阶段: Hive任务禁止使用LOCATION功能 * #AI commit# 开发阶段: Hive任务禁止使用LOCATION功能,测试报告相关文件提交 * #AI commit# 开发阶段: * hive禁止location正则优化 * #AI commit# 开发阶段: entrance配置文件优化 * #AI commit# 开发阶段: 去除code打印 --------- Co-authored-by: Casion <casionone@gmail.com>
…#970) * #AI commit# 开发阶段: * 修复sr任务重试导致加载init_sql异常bug * 优化spark参数获取 * 管理台支持展示引擎版本 * * #AI commit# 开发阶段: * 增加开关和异常保护
… issues (#972) * #AI commit# 优化:修复keytab文件缓存导致的Full GC问题,添加keytabFile缓存机制 * #AI commit# 优化:修复keytab文件缓存导致的Full GC问题,添加keytabFile缓存机制 * #AI commit# 开发阶段: 增加异常保护
…e logs (#974) * #AI commit# 开发阶段: * optimize log printing and token masking * #AI commit# 开发阶段: * 增加hadoop客户端使用日志
…nection pool expansion (#976) * #AI commit# 开发阶段:Monitor模块优化 - 诊断日志自动清理、诊断功能拆分、连接池扩容 主要变更: 1. 新增DiagnosisLogClear.java - 诊断日志定时清理功能 2. 诊断功能配置化拆分 - 新增配置开关控制 3. Alert连接池扩容 - 5→20线程 4. MonitorConfig新增5个配置常量 * #AI commit# 开发阶段: 相关文档提交 * #AI commit# 开发阶段: monitor ai 相关文档提交 * #AI commit# 开发阶段: * monitor参数优化 * #AI commit# 开发阶段: * 新增诊断表数据清理 * #AI commit# 开发阶段: 去除日志地址 --------- Co-authored-by: Casion <casionone@gmail.com>
…figuration (#980) * #AI commit# 开发阶段: 异常注释修复 * #AI commit# 开发阶段: 修复依赖注入失败
…es-publicservice - Restore linkis-io_file-client dependency that was accidentally removed - Remove hbase-server dependency that was incorrectly added - This fixes missing jars in linkis-ps-publicservice lib directory: - linkis-io_file-client - linkis-computation-orchestrator - linkis-code-orchestrator - linkis-orchestrator-core Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…t SQL instead of invalid command (#983) - Fix getAllDatabases() method in DB2 SqlConnection classes - Replace invalid "list database directory" command with proper SQL query - Add configurable schema query SQL with default filtering system schemas - Query SYSCAT.SCHEMATA to get actual schema list from DB2 Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Background/Problem:
Currently, spark engine tasks fail during execution due to null pointer exceptions in logging initialization and exception handling issues. Additionally, dependency injection failures and inconsistent error logging affect system stability.
Purpose of Change:
To address these problems, this PR optimizes logging output, fixes spark init SQL exception handling, resolves dependency injection failures, and improves exception comments across multiple engine executors.
Value/Impact:
After the change, spark engine tasks execute more reliably with better error handling and logging, improving system stability and debugging capability.
Related issues/PRs
Related issues: close #5284
Related pr:none
Brief change log
Checklist