Domino

[xrsrke]

the first PR is ready for review (i split it to two PRs), some left work for the next PR

• intra-layer overlapping (current overlapping communication within a layer), but if we do intra-layer overlapping, then we can almost totally overlapping the comm
• create an fixed buffer to concat hidden states (1st image)
• double check if there is cuda stream switching's overhead (4.3.1 in the 2nd image)
• minimize kernel launch overhead (cuda graph? 4.3.2 in 2nd image)

profiling results

profiling file: /fsx/phuc/temp/boom/issue2_domino_tp/nanotron/logs/profiles/_temp1_config_llama_domino/20250212-165229/ip-26-0-163-220_159222.1739379160555428025.pt.trace.json