Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

google-batch provider % logging missing default #304

Open
rivershah opened this issue Jan 2, 2025 · 1 comment
Open

google-batch provider % logging missing default #304

rivershah opened this issue Jan 2, 2025 · 1 comment

Comments

@rivershah
Copy link

rivershah commented Jan 2, 2025

With the recent google-batch provider, I am noticing that any stdout / err with % is broken. Maybe it is parsed as old style strings?

For example when I print nvidia-smi which shows % gpu usage, I see this:

Thu Jan  2 22:40:08 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       Off |   00000000:00:04.0 Off |                    0 |
| N/A   66C    P8             14W /   70W |       3MiB /  15360MiB |      0%!D(MISSING)efault |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

0% is accurate but 0%!D(MISSING)efault is not

@rivershah rivershah changed the title google-batch provider % logging broken google-batch provider % logging missing default Jan 2, 2025
@wnojopra
Copy link
Contributor

wnojopra commented Jan 7, 2025

Thanks for reporting @rivershah . I'm able to reproduce something similar this with a quick test:

dsub \
   --provider google-batch \
   --project "${MY_PROJECT}" \
   --logging "${MY_BUCKET}" \
   --regions us-central1 \
   --command 'echo "%hel%lo%"'

In the resulting stdout file I see:
%!h(MISSING)el%!l(MISSING)o%!

Will update here as I learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants