[RFC] Code Agent in CodeTrans

[letonghan]

This RFC proposes the integration of two Agent mechanisms into the CodeTrans Example to enhance the reliability, user experience, and code quality.
The goal is to minimize the propagation of erroneous code and improve the feasibility of automated code translation.

[yinghu5]

[Remind] @ftian1 please help to review the RFC, thank you!

[eero-t]

IMHO this RFC skips too many significant details:

• If translation is to compiled language, suitable build tools need to available for all of them
  • What languages this is targeted at; just Java?
• If resulting program relies on external components, those need to be identified & installed, before program can be successfully build or executed
  • Some dependencies can be GB sized
• There's no mention of how the execution is sandboxed
  • E.g. allowing network connectivity for malicious code would be rather serious (it could trivially do DOS attacks etc), but also innocent programs could be intended for network access
• Many programs do not run without input, but there's no mention of how input data provision is managed
  • Code translation may be also asked for individual functions, not whole programs

PS. Even if agent would do only code linting instead of execution, that would also need code dependencies to be installed/present (at least their API files).

[letonghan]

IMHO this RFC skips too many significant details:

• If translation is to compiled language, suitable build tools need to available for all of them

  • What languages this is targeted at; just Java?

• If resulting program relies on external components, those need to be identified & installed, before program can be successfully build or executed

  • Some dependencies can be GB sized

• There's no mention of how the execution is sandboxed

  • E.g. allowing network connectivity for malicious code would be rather serious (it could trivially do DOS attacks etc), but also innocent programs could be intended for network access

• Many programs do not run without input, but there's no mention of how input data provision is managed

  • Code translation may be also asked for individual functions, not whole programs

PS. Even if agent would do only code linting instead of execution, that would also need code dependencies to be installed/present (at least their API files).

Hi @eero-t , thanks for your detailed comments! I will update this RFC later, here's some responses of your questions:

• For build tool environment:
  • We do need to prepare different environment (considering using docker) for many languages, such as Python, Java, C#, GO and so on.
  • The dependencies will be installed in a seperate docker container for each task.
• For sandbox security:
  • A security policy is needed for docker container to prevent from malicious attack, this part will be updated in RFC in detail.
  • The network needs to be limited only for requirements installing, for example, using a white list.
• For program input:
  • We may ask user to provide a basic input and output case.

Please let me know if you have other suggestions.

[eero-t]

Please let me know if you have other suggestions.

@letonghan RFC needs to answer also following questions:

• What is done when code translation is asked for target language that agent does not support (but LLM does)?
• Why code is not linted (which typically finds more problems than running, and is easier)?
• What advantage building+running provides over code linting?

I'm also wondering whether each supported language would need its own additional RFC, as that that's rather complex, language specific topic (language versions, their upgrades, access to their module repositories, security etc).

[eero-t]

Will same agents be used also for CodeGen?

[eero-t]

Then there's also the performance aspect.

Users expect responses in seconds, but fetching code dependencies for building (or linting) the code could take minutes, maybe even tens of minutes.

Building the code also takes extra time, especially if agents need to do several rounds of builds to get translated code into fully buildable state.

Meaning that:

• User would need some feedback of the extra, time-consuming steps being performed
• When agent usage can induce large slowdowns, it would be nice if either:
  • UI would have an option to disable it (when perceived improvement is low enough), or
  • Application could cache query context (e.g. fetched dependencies)
    • This way it's only one-time (time/BW/CPU) cost per context, instead of user seeing large lags also for successive queries (and user feedback telling that app is doing dumbly same thing over-and-over again)
    • But it raises the need for context IDs / management, and question of how long such (large) contexts should persist?

[letonghan]

• What is done when code translation is asked for target language that agent does not support (but LLM does)?
• Why code is not linted (which typically finds more problems than running, and is easier)?
• What advantage building+running provides over code linting?

I'm also wondering whether each supported language would need its own additional RFC, as that that's rather complex, language specific topic (language versions, their upgrades, access to their module repositories, security etc).

I think using lint/bandit is also a great option for code checking.
This could be a two-step thing. For the firt step, agent will automatically check code with tools like lint and fix simple typos/faults. For the second step, the updated code will be executed to make sure it works.

[letonghan]

• User would need some feedback of the extra, time-consuming steps being performed

• When agent usage can induce large slowdowns, it would be nice if either:

  • UI would have an option to disable it (when perceived improvement is low enough), or

  • Application could cache query context (e.g. fetched dependencies)

    • This way it's only one-time (time/BW/CPU) cost per context, instead of user seeing large lags also for successive queries (and user feedback telling that app is doing dumbly same thing over-and-over again)
    • But it raises the need for context IDs / management, and question of how long such (large) contexts should persist?

Yes, building code sandbox, install dependencies, and execute it would take a lot of time.
As the two-steps thought above, I prefer to make link/execution as optional, which could be enabled/disabled in the web UI.

[lkk12014402]

• User would need some feedback of the extra, time-consuming steps being performed

• When agent usage can induce large slowdowns, it would be nice if either:

  • UI would have an option to disable it (when perceived improvement is low enough), or

  • Application could cache query context (e.g. fetched dependencies)

    • This way it's only one-time (time/BW/CPU) cost per context, instead of user seeing large lags also for successive queries (and user feedback telling that app is doing dumbly same thing over-and-over again)
    • But it raises the need for context IDs / management, and question of how long such (large) contexts should persist?

Yes, building code sandbox, install dependencies, and execute it would take a lot of time. As the two-steps thought above, I prefer to make link/execution as optional, which could be enabled/disabled in the web UI.

there is code execution tool https://github.com/QwenLM/Qwen-Agent/blob/main/qwen_agent/tools/code_interpreter.py

[minmin-intel]

Do we have customer requests for such code translation capabilities in OPEA? Is there a compelling need to invest engineering efforts in this? Shall we think about coding agent as a whole instead of just code translation or code generation? @letonghan @ftian1 @lkk12014402

[eero-t]

Shall we think about coding agent as a whole instead of just code translation or code generation?

Considering it for both makes more sense to me. At least I cannot quickly think of any difference between verifying / improving result for code translation, vs code generation.

[letonghan]

Hi @eero-t @minmin-intel , the RFC is updated.
The lint check tool and the code execution tool could be reused in both CodeTrans and CodeGen example, but I think we don't need to combine the RFCs here, since CodeGen RFC was already merged.
We can make sure it satisfies the needs of code agent, then develop and refine it in release v1.4.
Let's make sure this RFC be merged in v1.3 before middle April, thanks!

[eero-t]

This is much better now, but I still have few comments.

[eero-t]

The lint check tool and the code execution tool could be reused in both CodeTrans and CodeGen example, but I think we don't need to combine the RFCs here, since CodeGen RFC was already merged.

Ok.

(I do not see an overlap between #272 and this RFC, except both being RAG, but "before translation" phase is indeed specific just to code translation.)

[eero-t]

Still not fan of code execution, but RFC itself looks OK now. There are just a few inconsistencies that would be good to fix before merging.

(Doc updates have resulted with some things being repeated in multiple sections. It would help if details of each step are described only once, and removed from more generic sections.)