Add operator fusion pass into E2E pipeline but cause perfomance decline

[GuoningHuang]

Describe the bug
When I trying to add operator fusion pass such as flashattention or RMsNorm into E2E pass, the performance declined.
before I using norm opt, the E2E performance is:

after i using norm opt, the performance is:

To Reproduce
Reproduce this issue by #596 or #553.

[zhanghb97]

Nice report! I'm reviewing it.