Supporting all subdirectories in paths

[lucasnorman]

This is a suggestion for a nice to have. Consider the below conf file:

fsImagePath : $FSIMAGE_PATH

skipFileDistributionForGroupStats : true

skipFileDistributionForUserStats : false


paths:
  - '/data/*'

skipFileDistributionForPathStats : true

skipFileDistributionForPathSetStats : true

I would like metrics displayed for all the subdirectories under data. This way I can create a Prometheus query that automatically picks up new paths added and I don't have to add them to the conf file. Currently an error is thrown saying the path could not be found. Please let me know your thoughts.

[marcelmay]

Thx alot for the suggestion.

Paths support regex matching for child directories, this should bring the result you want (metric per child directory):

paths:
  - '/data/.*'

Note the dot-and-star .* in the example, matching any subdirectory.

[lucasnorman]

Thank you for getting back to me. This does work but only one level down. Is there a way to define something like ``` - '/.*' ``` And it pick up all directories in hdfs?

…

On Tue, Apr 27, 2021, 2:38 PM Marcel May ***@***.***> wrote: Thx alot for the suggestion. Paths support regex matching for child directories, this should bring the result you want (metric per child directory): paths: - '/data/.*' Note the dot-and-star .* in the example, matching any subdirectory. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#45 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGIXQXLBWSXCLB7PUAJ43QDTK4VFVANCNFSM43VOSJKA> .

[marcelmay]

For now you'd have to specify all the child dirs.

Out of curiosity, what is your use case for for-all-dirs instead of path-aggregated metrics?
How many dirs would that be?

In a classical datalake scenario, I'd recommend to either use eg by user or group stats.
Or for assets with mixed-user-and-groups a top level asset approach like /datalake/.* or the pathSets option (multiple paths aggregated as metric).