Technical issues with Dakota

[JavierGOrdonnez]

Technical Issues reported by Cedric. These specifically concern the python wheel (itis-dakota), they may or may not arise when using the dakota executable or when using dakota as a lib. These have been observed and mostly resolved through time.

We will try to address them as individual sub-cases, where we can discuss the best solution and make sure that this package consistently integrates a solution.

• variables and functions names are case insensitive, and some of the Ascii
  non alphanumeric chars are non valid.
  Suggestion : have a systematic name conversion internally into x1, x2, x3,...
  For input variables, and y1, y2,... For output var. Make this invisible to the
  user, systematic for Dakota. This solves all pb and allows you to easily
  manipulate/search variables internally: for ex when gathering data from
  files/stdout which only output values in a specific order, you can easily
  reassign them to xi or yi. Whenever the user/osparc interact with variables the
  conversion will take place, they never see these internal names, only those
  they choose. Very clean.

• Seeds must be non zero, positive, floats are accepted but shouldn't be used imo.
  Floats are bad due to rounding errors (rounding may vary with hardware) and
  offer no advantage over integers. I suggest to build a map into strictly
  positive unsigned 64 bits ints. My guess is that Dakota just static casts the
  values into a uint64 on the C++ side anyway.

• Isolation issues: When calling dakota methods successively I've ran into the
  pb where some sort of state remains between calls. In some instances data may
  even be shared. Isolating dakota calls into distinct processes solves the pb
  (distinct threads are not enough!).

• The many Dakota methods are inconsistent regarding whether they support
  batch or non-batch modes. Some methods even support batch everywhere except
  during the last iteration. The best way seems to let a wrapping function detect
  the mode requested by Dakota at runtime at every call/iteration; this can be
  done by simply detecting whether a list or dict is passed in by Dakota.

• Dakota generated output on stdout can overflow the OS stdout buffer. We need
  a special wrapper to direct very large textual output into files. Most python
  packages I found are not able to handle such output sizes properly, I built a
  little py file to do this right.

• Ill conditioned matrices happen occasinally when correlation matrices are
  involved (GP modeling, optimization, etc). These cannot be totally avoided,
  they depend on user input. They must be handled at run time.

• Multi objective optimization does not always output solutions that are
  strictly pareto. Some non-dominant values are kept (prob to smooth out the
  spread on the pareto surface). This can become an issue when relying on the
  pareto property (hypervolume metric, etc).

• Tabular data file are sometimes inconsistent across methods. Don't let the
  user directly interact/modify them. I think it's more robust to handle all data
  on our side and present the results to the in a consistent way.

• Parsing for data in stdout seems quite reliable, but the format is not
  consistent across methods and versions.

• Handling hd5 data files might be dangerous if you need to run multiple
  Dakota calls or methods. I've managed to avoid them so far.

• Manual prescaling is very often necessary. My suggestion is to normalize all
  input variables between 0 and 1 or using log (offering both options will be
  useful), without the user/osparc noticing similarly to the variable naming
  conversion case.

• A manual base surrogate may be useful if the user only wants to play with
  finite datasets. A Dakota workflow should offer both possibilities: use a full
  function/study as base, or build a base surrogate based on user (finite) data.

• Effectiveness of optimization highly depends on the Dakota parametrization.
  The unrecommended "elitism" keyword seems to help spreading the data along the
  pareto.

• GP modeling should offer a nugget option. Important for noisy input.

• Model validation is very incomplete. Dakota only offers to compute few
  basic metrics (MAE, RMSE, etc) without disclosing the residuals.

• Inverse uncertainty quantification (calibration) does not seem to work
  well: the computed posteriors may lack precision for some practical
  applications.

• Surrogate model serialization only works under very specific conditions.
  For GP alone, various dakota packages may handle this very differently and it
  is recomended to use the same dakota configuration files to save and reload
  models. A more generic and robust approach is to simply retrain surrogates on
  the fly based on saved training data sets, and never assume a reloaded model
  will be exactly the same.

• Dakota GP modeling essentially uses an elaborate version of simple kriging:
  no variograms involved, process means are estimated outside of the kriging
  equations. This may imply inaccuracies in the presence of complicated types of
  anisotropies. Appropriately transforming the input space a priory may improve
  surrogate accuracy. In the global case we cannot expect dakota to always create
  good models. This issue is mostly inexistent for local GP surrogates (for many
  of the surrogate based methods).

[wvangeit]

@cbujard Could you do a triage of the issues above.
I would say at least three categories:

• Issues that could/should be solved in the core Dakota lib -> Create issue in core dakota repository
• Issues that are only happening in the wheel, and can't be solved in core Dakota -> Create issue in itis-dakota repo
• Issues, and that seem to be the majority, that might have to be solved in some wrapper(?). Should we create a repo for such a wrapper, or can you make suggestions on how to solve the other issues?

[wvangeit]

@cbujard did you already have time to look at this? thx.

[cbujard]

Here is a categorization.

Category C essentially are straightforward technicalities that should systematically be handled by some wrapper code. A "generic" wrapper class built on a known dakota configurations should probably be able to handle these.

Category D issues are more context dependent and less straightforward: most of them would need further discussion on how to handle them in tandem with the user. Some may require our own solutions. We may want to use a dedicated repo to develop satisfactory solutions overtime.

Category E is just a warning.

A) Dakota lib issues: None

B) Dakota wheel issues:

isolation issue (3): an immediate workaround possible at the wrapper code level.

C) Technical issues to be automatically handled in wrapper code:

variables and functions names (1): can naturally be implemented within a wrapping class that internally keeps track of all input/output variables based on variable order (xi, yj). 

seeds (2): use a well balanced surjective mapping onto unsigned ints for proper reproducibility. 

isolation workaround (3): run each dakota call in a separate process.

batch vs non-batch inconsistency (4): this can be detected by analysing the input data type (list vs dict).

stdout capture overflow (5): most capture libs won't work here, needs a dedicated code. 

ill conditioned matrices (6): this naturally happens and cannot be fully avoided.

stdout format inconsistency (9): each dakota version may change this, each dakota method has its own format.

surrogate model serialization (17): we may assume each model can be rebuilt from a finite sample set, but one cannot assume the model will be the same.

D) Context/user dependent modeling issues to be handled in wrapper code:

non-pareto results (7): this should be handled with an option to provide true paretos, otherwise can break algorithms or indicators that expect true pareto sets.

manual prescaling (11): this can significantly improves the results as dakota typically expects the caller to handle this. The type of prescaling depends on the context, we may want to offer two types: linear normalization, log rescaling. 

nugget option (14): this is a must whenever a model is to be trained on data coming from noisy measurments/models. 

improper modeling in highly anisotropic contexts (18): dakota cannot handle all types of anisotropies when building global surrogates.

provide real time base surrogate model construction if no callable is available (12): in practice this may be very useful as some user may not have access to a full simulator.

optimization parametrization (13): we need to give the user some flexibility here, very dependent on context.

incomplete validation (15): dakota does not offer enough for validation of (its own) models, we need to bring some extra solutions.

ineffective inverse UQ (16): reformulating calibration as optimization problems may provide usable workflow.

dakota sampling (19): it's very basic and might not be enough to some users. There is an option for custom/user provided sample sets. We may also want to add more general sampling options. 

E) Potential issue:

global nature of Hd5 data (10): to be tested.

[JavierGOrdonnez]

Hi @cbujard @wvangeit I agree with most of what you discussed above, and the need for a package to unify our approaches and efforts.

I would like to propose to use this Thursday's meeting to do a longer session of joint programming, and lay the foundation of such package together. I believe many of the problems above have already been addressed by Cedric in his scripts or by me in my mmux-python, so we could quickly re-implement them as part of the joint repo, and rewrite them as needed to fit the defined requirements. Also, it is a chance to peer-review our codes, define automatic testing, and make them as generic and reliable as possible, as well as document the usage of all functionality.

For items of the list which require development effort, we could generate the corresponding issues and description of requirements during the joint programming session, and work on them in the following weeks / months according to priority.

PS @cbujard it would be good to keep in mind how to include techniques, learnings and solutions from Melanie & Aiping experiences so they can eventually also use this joint framework rather than their individual, harder to maintain solutions.

[JavierGOrdonnez]

Inputs from 12th Aug 2025 meeting

• Create repo as private GitHub
• Protect main branch & require review
• Integration tests - be able to run previous projects (@cbujard and @JavierGOrdonnez ) with this package
• Unit-tests : do one PR for each separate issue, create a test of how it SHOULD work, and then copy / create the code that solves this test

Features / Structure

• Have a class that takes in either a callable or a CSV file; a dakota file; ...
• Include functionality to create correct Dakota files
• Include automatic validation of Dakota files (seeds, etc)
• Include metadata (inc version) in the CSV file to ensure backwards-compatibility (@wvangeit)
• After Dakota implementation, be able to generate our own models using the Class
• Include tests that compare Dakota file building vs. a pre-generated dakota file
• Run evth in in a separate subfolder, for total reproducibility -- and have an "Activity Center" file where it gets logged what the user has run at different points.
• Include tests that output stdout for a fixed input config & data stays the same - so that we can adapt the output parsing functions if necessary.

User Story

• A user calls a factory class / function that generates a DakotaProject object
• then they can use the "run" method to execute
• have output handling / parsing for each method (as it is very heterogeneous)