Skip to content

add constrained optimizers #1980

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Jun 6, 2025
Merged

add constrained optimizers #1980

merged 8 commits into from
Jun 6, 2025

Conversation

yetongumich
Copy link
Contributor

@yetongumich yetongumich commented Jan 15, 2025 •
edited by dellaert
Loading

This PR introduces constrained optimizers that solve constrained nonlinear least-squares problems.

The contents of this PR includes

  • Representation of the constrained optimization problem (ConstrainedOptProblem.*) which includes (1) nonlinear least-squares cost function represented as a factor graph; (2) nonlinear equality constraints; (3) nonlinear inequality constraints.
  • Abstract class for constrained optimizer (ConstrainedOptimizer.h).
  • Implementation of two popular constrained optimization methods: the penalty method (PenaltyOptimizer.*) and the augmented Lagrangian method (AugmentedLagrangianOptimizer.*). The implementation follows L. Vandenberghe's slides.
  • Since both optimizers requires iteratively solving unconstrained optimization problems with merit functions (cost function + penalty functions for constraints), additional classes (InequalityPenaltyFunction.*) are introduced to represent the penalty functions. In order to express certain merit functions (e.g., augmented Lagrangian function) in nonlinear least square forms, additional factors (BiasedFactor.h and ConstResidualFactor.h) are introduced.

@dellaert
Copy link
Member

Please provide a nice PR comment :-)

@dellaert
Copy link
Member

Please update PR comment, make it draft PR until this is ready to review. Also, before that, refraing from pushing commits as it will trigger CI every time.

@dellaert
Copy link
Member

@yetongumich Please update PR comment, make it draft PR

@yetongumich yetongumich requested a review from dellaert February 19, 2025 19:32
@dellaert
Copy link
Member

CI on windows fails: maybe missing GTSAm_EXPORT? Do check the readme on that

@yetongumich
Copy link
Contributor Author

@dellaert This PR is ready for review

dellaert
dellaert reviewed Apr 19, 2025
View reviewed changes
dellaert
dellaert reviewed Jun 2, 2025
View reviewed changes
Copy link
Member

@dellaert dellaert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really cool !!!

Ajay, Bruce, and I looked at the code in more depth than I could by myself.

We also found that constraint factors could use a bit more documentation (as well as check naming convention there as well?)

gtsam/constrained/AugmentedLagrangianOptimizer.h Outdated
* where
* c = ||lambda_e||^2 / (2 * mu_e) + ||lambda_i||^2 / (2 * d)
* is a constant term.
* Notice: a term (0.5 d * ||g(x)||^2) is added to incorporate Lagrange
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We (Frank, Ayaj, Bruce) were trying to read this carefully and are not entire following - and also does not seem to be there on Vandenberghe slides?

Copy link
Contributor Author

@yetongumich yetongumich Jun 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is inspired by Vandenberghe slide number 23. However, the slides only consider equality constraints. I made some extension to incorporate inequality constraints using a similar strategy to rewrite the augmented lagrangian function in least-squares form. I have updated the comments, and more detailed derivation is available in this overleaf document: https://www.overleaf.com/read/rzzgkwdrwnbh#a1f9f5

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment to a source

dellaert
dellaert reviewed Jun 2, 2025
View reviewed changes
dellaert
dellaert reviewed Jun 2, 2025
View reviewed changes
dellaert
dellaert approved these changes Jun 6, 2025
View reviewed changes
Copy link
Member

@dellaert dellaert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Merge at will. We could add (and benchmark) Nocedal scheme for updating $\mu$ in a separate PR.

dellaert
dellaert reviewed Jun 6, 2025
View reviewed changes
gtsam/constrained/AugmentedLagrangianOptimizer.cpp

// Construct merit function.
const NonlinearFactorGraph augmentedLagrangian =
augmentedLagrangianFunction(newState);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this->augmentedLagrangianFunction()

gtsam/constrained/AugmentedLagrangianOptimizer.h Outdated
* where
* c = ||lambda_e||^2 / (2 * mu_e) + ||lambda_i||^2 / (2 * d)
* is a constant term.
* Notice: a term (0.5 d * ||g(x)||^2) is added to incorporate Lagrange
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment to a source

@dellaert dellaert merged commit 6c6db8a into develop Jun 6, 2025
36 checks passed
@dellaert dellaert deleted the feature/constrained_optimizer branch June 6, 2025 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dellaert dellaert dellaert approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@yetongumich @dellaert @yetongatech