Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grid engine support for terabyte (T) MEMTOT output from qhost, and cpu specifications #41

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Grid engine support for terabyte (T) MEMTOT output from qhost, and cpu specifications #41

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
16d9982
added support for terabyte (T) MEMTOT output from qhost
cooketho Sep 20, 2015
dcf101c
specify sge processors via -pe shm instead of -l num_proc
cooketho Sep 20, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions batchSystems/gridengine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@

class MemoryString:
def __init__(self, string):
if string[-1] == 'K' or string[-1] == 'M' or string[-1] == 'G':
if string[-1] == 'K' or string[-1] == 'M' or string[-1] == 'G' or string[-1] == 'T':
self.unit = string[-1]
self.val = float(string[:-1])
else:
Expand All @@ -60,6 +60,8 @@ def byteVal(self):
return self.val * 1048576
elif self.unit == 'G':
return self.val * 1073741824
elif self.unit == 'T':
return self.val * 1099511627776

def __cmp__(self, other):
return cmp(self.bytes, other.bytes)
Expand All @@ -69,7 +71,7 @@ def prepareQsub(cpu, mem):
"LD_LIBRARY_PATH=%s" % os.environ["LD_LIBRARY_PATH"]]
reqline = list()
if cpu is not None:
reqline.append("p="+str(cpu))
qsubline.extend(["-pe", "shm", str(int(cpu))])
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Toil already switched to -pe but it uses -pe smp instead of -pe shm. Do you think they are equivalent?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Hannes,

I don't think "-pe shm" and "-pe smp" are equivalent, but I'm not entirely
sure.

For example, on the computer I use, the results of the command qconf -spl
are:
orte
shm

So I don't think I can use the smp parallel environment on this particular
computer without making some changes.

I can't think of an elegant solution at the moment, but maybe you could
prompt the user for the name of the appropriate parallel environment, just
once, during package setup.

cheers,
Tom

On Tue, Oct 6, 2015 at 6:05 PM, Hannes Schmidt [email protected]
wrote:

In batchSystems/gridengine.py
#41 (comment):

@@ -69,7 +71,7 @@ def prepareQsub(cpu, mem):
"LD_LIBRARY_PATH=%s" % os.environ["LD_LIBRARY_PATH"]]
reqline = list()
if cpu is not None:

  •    reqline.append("p="+str(cpu))

  •    qsubline.extend(["-pe", "shm", str(int(cpu))])

Toil already switched to -pe but it uses -pe smp instead of -pe shm. Do
you think they are equivalent?

—
Reply to this email directly or view it on GitHub
https://github.com/benedictpaten/jobTree/pull/41/files#r41341883.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thank you.

if mem is not None:
reqline.append("vf="+str(mem/ 1024)+"K")
reqline.append("h_vmem="+str(mem/ 1024)+"K")
Expand Down