Skip to content

[fix][EC][spark] resolve spark engine task execution failures and optimize logging#5420

Closed
v-kkhuang wants to merge 16 commits intoapache:dev-2.0.0from
WeDataSphere:dev-2.0.0-bug-fix
Closed

[fix][EC][spark] resolve spark engine task execution failures and optimize logging#5420
v-kkhuang wants to merge 16 commits intoapache:dev-2.0.0from
WeDataSphere:dev-2.0.0-bug-fix

Conversation

@v-kkhuang
Copy link
Copy Markdown
Contributor

What is the purpose of the change

Background/Problem:
Currently, spark engine tasks fail during execution due to null pointer exceptions in logging initialization and exception handling issues. Additionally, dependency injection failures and inconsistent error logging affect system stability.

Purpose of Change:
To address these problems, this PR optimizes logging output, fixes spark init SQL exception handling, resolves dependency injection failures, and improves exception comments across multiple engine executors.

Value/Impact:
After the change, spark engine tasks execute more reliably with better error handling and logging, improving system stability and debugging capability.

Related issues/PRs

Related issues: close #5284
Related pr:none

Brief change log

  • Add CodeUtils utility class for common code operations
  • Optimize HDFSUtils keytab cache handling with null checks
  • Optimize logging output in spark, hive, flink, and other engine executors
  • Fix spark init SQL exception handling
  • Fix dependency injection failures in entrance module
  • Improve exception comments in multiple engine plugins
  • Update 2.0.0 feature documentation structure
  • Clean up obsolete test files and documentation

Checklist

  • I have read the Contributing Guidelines on pull requests.
  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible
  • If this is a code change: I have written unit tests to fully verify the new behavior.

aiceflower and others added 16 commits April 4, 2026 06:42
Update project version to 2.0.0 in pom.xml and database scripts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…stance goes offline (#966)

* #AI commit# 开发阶段: *  修复异常场景,当Entrance实例offline后,ParallelGroup缓存未更新,导致用户任务并发数计算错误

* #AI commit# 开发阶段: *  修复异常场景,当Entrance实例offline后,ParallelGroup缓存未更新,导致用户任务并发数计算错误

* #AI commit# 开发阶段: *  增加异常保护

* #AI commit# 开发阶段: * 修复异常场景,当Entrance实例offline后,ParallelGroup缓存未更新,导致用户任务并发数计算错误
…ge in hive tasks (#968)

* #AI commit# 开发阶段: Hive任务禁止使用LOCATION功能

* #AI commit# 开发阶段: Hive任务禁止使用LOCATION功能,测试报告相关文件提交

* #AI commit# 开发阶段: *  hive禁止location正则优化

* #AI commit# 开发阶段: entrance配置文件优化

* #AI commit# 开发阶段: 去除code打印

---------

Co-authored-by: Casion <casionone@gmail.com>
…#970)

* #AI commit# 开发阶段: * 修复sr任务重试导致加载init_sql异常bug
* 优化spark参数获取
* 管理台支持展示引擎版本
*

* #AI commit# 开发阶段: *  增加开关和异常保护
… issues (#972)

* #AI commit# 优化:修复keytab文件缓存导致的Full GC问题,添加keytabFile缓存机制

* #AI commit# 优化:修复keytab文件缓存导致的Full GC问题,添加keytabFile缓存机制

* #AI commit# 开发阶段: 增加异常保护
…e logs (#974)

* #AI commit# 开发阶段: *  optimize log printing and token masking

* #AI commit# 开发阶段: *  增加hadoop客户端使用日志
…nection pool expansion (#976)

* #AI commit# 开发阶段:Monitor模块优化 - 诊断日志自动清理、诊断功能拆分、连接池扩容

主要变更:
1. 新增DiagnosisLogClear.java - 诊断日志定时清理功能
2. 诊断功能配置化拆分 - 新增配置开关控制
3. Alert连接池扩容 - 5→20线程
4. MonitorConfig新增5个配置常量

* #AI commit# 开发阶段: 相关文档提交

* #AI commit# 开发阶段: monitor ai 相关文档提交

* #AI commit# 开发阶段: *  monitor参数优化

* #AI commit# 开发阶段: *  新增诊断表数据清理

* #AI commit# 开发阶段: 去除日志地址

---------

Co-authored-by: Casion <casionone@gmail.com>
…figuration (#980)

* #AI commit# 开发阶段: 异常注释修复

* #AI commit# 开发阶段: 修复依赖注入失败
…es-publicservice

- Restore linkis-io_file-client dependency that was accidentally removed
- Remove hbase-server dependency that was incorrectly added
- This fixes missing jars in linkis-ps-publicservice lib directory:
  - linkis-io_file-client
  - linkis-computation-orchestrator
  - linkis-code-orchestrator
  - linkis-orchestrator-core

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…t SQL instead of invalid command (#983)

- Fix getAllDatabases() method in DB2 SqlConnection classes
- Replace invalid "list database directory" command with proper SQL query
- Add configurable schema query SQL with default filtering system schemas
- Query SYSCAT.SCHEMATA to get actual schema list from DB2

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
@v-kkhuang v-kkhuang closed this Apr 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants