Use temp dir flag when making CSVs

[bethac07]

No description provided.

[shntnu]

Do you recollect details of this issue, @bethac07?

I thought it meant that we use use the -t flag when calling create_csv_from_xml.sh but that's probably not it

[bethac07]

I do believe that's it, yes.

[shntnu]

Hm – the flag has been defined but is not used https://github.com/broadinstitute/cellpainting_scripts/blob/master/create_csv_from_xml.sh#L30

It's possible that what we meant here was that create_csv_from_xml.sh is slow on EFS, so we should use a temp dir (and thus specify the temp flag) to copy the batch files locally on EBS, and then create load data. But I haven't had issues recently with create_csv_from_xml.sh.

So unless there's something else we need to address here, we could close this.

[bethac07]

The major issue I've run into with it is that it's very slow when working in parallel (like, runs in 5-10 minutes not run with parallel, takes many hours if you're doing more than a couple with parallel); I do not recall if there was once a reason we suspected it was EBS/EFS related (vs, say, network I/O), and my Slack investigations are not turning up anything.

[shntnu]

I lost the reply I was writing here, but the essence is that we'd need to address this broadinstitute/pe2loaddata#11 to fix this issue. I'll keep this open.