HPCC-30365 Add XRef Sasha service to K8s #19639
Conversation
- expandMask wasn't taking stripeNum and dirPerPart into account when creating the physical file name from the part mask - scanDirectory was calling itselg recursively on the dirPerPart directories causing files parts to be under different cDirDesc which prevented them from being logged as found.
|
Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-30365 Jirabot Action Result: |
| @@ -3890,6 +3890,7 @@ extern da_decl void parseFileName(const char *name,StringBuffer &mname,unsigned | |||
| mname.append(s); | |||
| num = pn; | |||
| max = mn; | |||
| break; | |||
There was a problem hiding this comment.
this change looks sensible, but what was happening before this change was made?
There was a problem hiding this comment.
Before the break was added, it would iterate over the tail of the string again i.e it would check each character in "_1_of_2" after it had already been processed for the part and max numbers.
dali/sasha/saxref.cpp
Outdated
| } | ||
| if (isDirPerPart) { | ||
| // MORE: Should maybe check this doesn't contain any subdirectories to make | ||
| // sure it is really a dirPerPart directory. Is an all numbers subdirectory valid in ecl? |
There was a problem hiding this comment.
in practice yes, a scope must have a leading alpha char. So worth clarifying the comment.
There was a problem hiding this comment.
Clarified comment.
dali/sasha/saxref.cpp
Outdated
| isDirPerPart = false; | ||
| } | ||
| if (isDirPerPart) { | ||
| // MORE: Should maybe check this doesn't contain any subdirectories to make |
There was a problem hiding this comment.
can check pdir.dirs after the scanDirectory, should be empty.
Let's add a check and throw an exception if not.
There was a problem hiding this comment.
Added check (pdir->dirs.ordinality()>0)
dali/sasha/saxref.cpp
Outdated
| unsigned maxMb = serverConfig->getPropInt("DfuXRef/@memoryLimit", DEFAULT_MAXMEMORY); | ||
| unsigned maxMb; | ||
| if (isContainerized()) { | ||
| const char *resourcedMemory = getComponentConfigSP()->queryProp("resources/@memory"); |
There was a problem hiding this comment.
now 'props' is saved, should used it instead of getComponentConfigSP() here.
There was a problem hiding this comment.
Fixed to use new props member.
system/jlib/jutil.cpp
Outdated
| const char *s=mask; | ||
| if (s) | ||
| while (*s) { | ||
| char next = *(s++); | ||
| if (next=='$') { | ||
| if (dirPerPart) | ||
| { |
There was a problem hiding this comment.
trivial/formatting: Allman vs K&R
system/jlib/jutil.cpp
Outdated
| { | ||
| const char * start = buf.str(); | ||
| const char * slash = start + buf.length(); | ||
| while (slash > start && *slash != PATHSEPCHAR) |
There was a problem hiding this comment.
'slash' at the start is potentially beyond the end of the allocated buffer I think.
system/jlib/jutil.cpp
Outdated
| while (slash > start && *slash != PATHSEPCHAR) | ||
| slash--; | ||
| buf.insert(slash-start, PATHSEPCHAR); | ||
| buf.insert((slash+1)-start, p+1); |
There was a problem hiding this comment.
not safe, buf may have reallocated after 1st insert, meaning 'slash' points to a free'd pointer.
There was a problem hiding this comment.
These insert also don't look right in that they're inserting "/", but the mask is typically something like "myfilename._$P$_of_10", so it's going to end up with e.g. "/5myfilename._5_of_10" afaics.
If it did find a slash, it would prob be okay, but there isn't necessarily a full path so there's no guarantee there's a slash.
There was a problem hiding this comment.
Also, the mask can be "myfilename._$P$_of_$N$" (see 'N' handleing below)
This insert code is going to be hit twice in that case, and therefore insert the dir-per-part directory twice...
There was a problem hiding this comment.
Reverted changes to expandMask in favor of storing prefix and scope directory paths in cFileDesc and building up full paths in getPartName instead.
system/jlib/jutil.cpp
Outdated
| { | ||
| if (stripeNum>0) | ||
| addPathSepChar(buf.append('d').append(stripeNum)); |
There was a problem hiding this comment.
hm, how is this strip dir going to be in right place?
There was a problem hiding this comment.
Removed changes to expandMask.
- Change cFileDesc to track the prefix and scope paths separately from name - When getPartName is called build up stripeNum and dirperpart in between prefix and scope paths - Find matching file parts by hashing full path minus stripeNum and dirperpart numbers
There was a problem hiding this comment.
As discussed in our meeting, there's are some issues with the way the info. is being scanned and stored in the hierarchical structure built up during the physical file scan and used subsequently.
Currently too much information is being stored per cFileDesc - as before, the file path shouldn't be needed, only the deduced filename mask.
Lost files are probably being misreported (at least during initial phase), because the paths they walk do not match the physical representation on disk (due to striping and dir-per-part directories).
saxref should:
- ensure runXRef is dealing with 1 plane a time (i.e. if multiple selected, process 1 at a time).
- store plane details (IPT) so accessible to other xref tasks during scan.
- during scanDirectories, if striped, detected when at stripe level, and 'skip'
- detect dir-per-part directory and 'skip'. But, check after recursing the dir, that contains no subdirs.
- Keep the filename mask only in cFileDesc
- Add scope/lfn to listOrphans. Build up in same way as 'basedir' is now.
Think we can get rid of baseDir altogether. - during scanLogicalFiles, parse file paths fetched from parts (add helper func), to remove stripe and dir-part-part, so can marry the pathing with the cDirDesc tree.
NB: LATER- may be better to assume that the cDirDesc is a representation of scopes, and walk scopes of logical file (+part endpoints), instead of getting part directories. - In listOrphans(cFileDesc), deduce file path from lfn, partNum and plane. Add a utility func that deduces and uses stripe num and dir-per-part if relevant (based on plane details).
| @@ -3834,7 +3835,8 @@ extern da_decl void parseFileName(const char *name,StringBuffer &mname,unsigned | |||
| } | |||
| } | |||
|
|
|||
| if (mname.isEmpty()) | |||
| // Assume that if prefix is passed in a match is required | |||
| if (prefix && prefix->isEmpty()) | |||
| throw makeStringExceptionV(-1, "Could not find matching prefix in plane definition for file %s", filename); | |||
There was a problem hiding this comment.
I think throwing an exception here, will mean 1 path that fails to match, will cause the whole xref process to fail?
True of any exception in parseFilename. NB: addFile only issues warnings.
addFile should likely have a try/catch and issue warnings in case of any exception in parseFilename.
There was a problem hiding this comment.
There's something that doesn't make sense here in fact I think..
How would the files not match prefix?
And, why is it scanning planes every file being added, to determine which plane the prefix path is in?
Given, we are specifically xref'ing a given plane, and therefore start at the prefix path..
This relates to HPCC-33151.
Let's discuss.
| mname.append((d+1)-name, name).append(cur-(tailSlash+1), tailSlash+1); | ||
| if (dirs) | ||
| dirs->append((d+1)-name, name); | ||
| mname.append(cur-(tailSlash+1), tailSlash+1); |
There was a problem hiding this comment.
common with line 3881, could go outside if/else
| } | ||
|
|
||
| StringBuffer &getPartName(StringBuffer &buf,unsigned p) | ||
| { | ||
| // In baremetal, buf can be prepoulated with replicate directory |
There was a problem hiding this comment.
The old code was:
StringBuffer &getPartName(StringBuffer &buf,unsigned p)
{
StringBuffer mask;
getName(mask);
return expandMask(buf, mask, p, N);
}
If buf is prepopulated, didn't that mean that as it was, it added the whole path again?
There was a problem hiding this comment.
This feels like it's masking a problem, rather than the correct fix.
Something not quite right here..
the cFileDesc (and cDirDesc) contain the name only, not a path.
getPartName should be returning an expanded form of the name mask only, as it was before.
Type of change:
Checklist:
Smoketest:
Testing: