Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added optional GPU Memory Logging #459

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

krishgera
Copy link

@krishgera krishgera commented Jan 29, 2025

This pull request introduces an optional --log-gpu-memory command-line flag in generate.py to log GPU memory usage at key points (immediately after loading the model, before generation, and after generation). By default, logging is disabled, so there is no impact on users who do not require memory tracking.

Changes :
Added a --log-gpu-memory argument.
Conditionally log GPU memory allocated/reserved at each relevant inference stage.

Rationale :
Simplifies troubleshooting for users running large models on limited GPU VRAM.
Maintains existing code paths when the flag is not used.

Testing :
Verified correct parsing of the new flag in local setups.
Observed expected memory logs appear only when --log-gpu-memory is enabled.

No additional dependencies or performance overhead for users who opt out of memory logging.

@Some1and2-XC
Copy link

You should remove the trailing indentation.

@krishgera krishgera force-pushed the feature/gpu-logging branch from 824e8b2 to a0a75d0 Compare January 29, 2025 19:53
@krishgera
Copy link
Author

You should remove the trailing indentation.

Fixed trailing indentation. Let me know if anything else needs adjustment.

@AlphaNewt7
Copy link

An interesting inclusion.

@krishgera
Copy link
Author

@mowentian can you review this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants