Persistence Extensions: Riemann Sums #4461

mherwege · 2024-11-22T10:15:52Z

At the moment, it is not possible to easily calculate the integral value of the curve represented by persisted values. This is useful for calculating e.g. total energy consumed (in kWh) from instantaneous power meter readings (in W).

The current sum persistence actions calculate a sum without considering the time dimension, and therefore are only useful when we have a guaranteed constant interval between persisted values for this.

The average actions do use a time weighting in their calculation. Multiplying the average with the total duration considered would give an approximation. However, the average calculations assume a constant value over the bucket and take the value at the start of the bucket. Using this average to calculate an integration would also mean an extra calculation (already divided in the action code and multiplied again in the rule).
An alternative approach to get an integral value is by setting up an integrated item and a rule that adds value * duration since previous change/upate, triggered on base item change. This works and is flexible in its calculation (value can be approximated based on previous and new item state). It suffers from things like unexpected shutdowns, where the base item and its aggretation may run out of sync.

This PR proposes to create a group of new actions, riemannSum , that will calculate the RiemannSum as an approximation for the integral value. Analoguous to the other actions, there are variants taking (or not) startDate, endDate and serviceId as input.
There is one extra key parameter, the Riemann type, representing the type of approximation used. Valid values are:

RiemannType.left: takes the persisted value at the start of the bucket to represent the value for the whole bucket. This is most useful when there is a persistOnChange strategy and the values represent a step function. An example would be dynamic electricity rates, as they will effectively be constant inside the bucket.
RiemannType.right: takes the persisted value at the end of the bucket.
RiemannType.trapezoidal: takes the average of the persisted value at the start end the end of the bucket, effectively making a linear interpolation to fit the curve. This type is most useful when the real values change continuously. It can be used for any persistence strategy and any interval.
RiemannType.midpoint: uses 3 persisted values and uses the middle of the values as an approximation for the value halfway in the interval between the middle of point 1 and 2 and the middle of point 2 and 3. This is the best approximation when the real values change continuously, the persistence intervals are short and the bucket sizes between persisted values are relatively constant.

The existing average, variance and deviation actions have been enhanced to use the results of the Riemann sum calculations. They now have variants with an extra parameter where this type can be selected.
For backward compatibility and consistency between defaults of the different actions, the action versions for riemannSum, average, variance and deviation without a Riemann type parameter all use RiemannType.left as a default.

Tests have been enhanced end confirmed the existing actions with the existing parameters still behave as before.

If this PR is accepted the scripting libraries should be enhanced to cover these new actions and the new parameters in the existing actions.

For testing, a compiled jar can be downloaded from here.

mherwege · 2024-11-22T11:24:40Z

Tests run fine locally. I am trying to figure out why they do not work in the PR.

mherwege · 2024-11-22T17:08:48Z

I put in some extra test logging to try to analyze where the difference is between my succeeding local tests and the failing online tests.

mherwege · 2024-11-23T22:11:49Z

All test are passing now and extra logging has been removed. This is ready for review.

...persistence/src/main/java/org/openhab/core/persistence/extensions/PersistenceExtensions.java

jimtng · 2024-11-26T00:51:07Z

This class is getting very cumbersome with all the parameter permutations!

mherwege · 2024-11-26T13:30:37Z

This class is getting very cumbersome with all the parameter permutations!

Totally agree. Do you have any suggestions? As it is all about static methods in one class, it is not easy to split without introducing a lot of breaking changes. The private methods could move to one or more utility classes, but I am not sure this will make maintaining it any easier.

jimtng · 2024-11-30T09:08:35Z

...persistence/src/main/java/org/openhab/core/persistence/extensions/PersistenceExtensions.java

    }

    private static @Nullable State internalVarianceBetween(Item item, @Nullable ZonedDateTime begin,
-            @Nullable ZonedDateTime end, @Nullable String serviceId) {
+            @Nullable ZonedDateTime end, RiemannType type, @Nullable String serviceId) {


This comment applies to all the other similar methods.

Whilst implementing this into the jruby helper library, I came across one particular issue.

Could you please make the type parameter @Nullable, and provide the default (i.e. RiemannType.LEFT) inside the internalXxxxXxxx methods, and pass null by the callers? This would:

DRY and collect all the defaults into one place (in the internalXxxx method) rather than in the callers

Make it easy for the helper libraries to also call it with null to indicate "default", and therefore letting core decides what the default is.

As it is now, I'd either have to also hard-code this default in the helper library, thus maintaining a duplicate default, or conditionally call one of the overloaded methods that doesn't take the type argument in order to use the "default" type.

Yes, I will do that. Makes sense.

mherwege · 2024-11-30T19:38:24Z

@florian-h05 I was looking into making these new methods available in JS. I don’t know how to properly handle the RiemannType enum arguments. Does a Java enum work properly in GraalVM JS scripting? Or should I be using string arguments in this PR? That may make it harder as well as the serviceId argument is also a string and will make it impossible to distinguish in varargs.

florian-h05 · 2024-11-30T21:22:32Z

I would expose the RiemannType enum as a static member of the ItemPersistence class, I think that should work (even though enums do not natively exist in JS).

holgerfriedrich

@mherwege sorry for the long delay....
I have just seen that I already wrote in prep to the OH4.3 release "The most interesting pending feature is probably #4461, as it is useful for computations regarding energy consumption." Though, I did not manage to review this until now. 🙈

Overall, LGTM!

I left a few comments.
There is a minor conflict, could you please rebase?

Finally, I think we need some user doc for this (this could be based on the text you already added to the PR description, and might also include pictures from #4439).

holgerfriedrich · 2025-03-23T09:30:56Z

...ersistence/src/test/java/org/openhab/core/persistence/extensions/TestPersistenceService.java

+        }
+    }
+
+    static double riemannSum(@Nullable Integer beginYear, @Nullable Integer endYear, RiemannType type) {


Could you please rename this method?
It is easy to mix this up with riemannSum from the persistence extension, taking a different number of arguments....

...istence/src/test/java/org/openhab/core/persistence/extensions/PersistenceExtensionsTest.java

Signed-off-by: Mark Herwege <[email protected]>

mherwege · 2025-03-24T14:47:41Z

Finally, I think we need some user doc for this (this could be based on the text you already added to the PR description, and might also include pictures from #4439).

I totally agree. I intended to work on documentation once there was progress on this and the related PRs for javascript (openhab/openhab-js#401) and Blockly (openhab/openhab-webui#2893).

mherwege · 2025-03-24T15:18:11Z

@holgerfriedrich The CI build fails on a test that was not touched in the last commit. It builds fine locally with no test failures. I don't see what the issue can be.

EDIT: and just adjusting a few comments in the last commit makes the tests pass again. I have no clue.

Signed-off-by: Mark Herwege <[email protected]>

holgerfriedrich

LGTM.
Thanks for this effort! I am looking forward to using this on my production system.

openhab-bot · 2025-03-25T00:44:02Z

This pull request has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/jruby-scripting-official-helper-library/145072/33

dandjo · 2025-03-25T10:28:17Z

Thanks everyone for this!

Refs on openhab/openhab-core#4461. Signed-off-by: Mark Herwege <[email protected]>

…2893) Depends on openhab/openhab-js#401 Depends on openhab/openhab-core#4461 This PR extends the Blockly block getting a statistical value from persistence to be aligned with core and js scripting extensions. The added statistical function is Riemann sum. This is an approximation of the integral value and can e.g. be used to calculate an approximation for energy consumption (in kWh) when the instantanous power (in W) is persisted. The existing sum method calculates a naive sum, ignoring the time dimension and is not applicable for this. It could be used only if the persistence interval is constant by using the sum and multiplying with the interval duration. There are multiple types of Riemann sum calculations depending on which value is used as an approximation in each bucket. The ones implemented in core (and js scripting) are: left, right, trapezoidal and midpoint. If the Riemann sum statistical method block is selected in the block, a parameter for this will also be shown, defaulting to left. As average, variance and deviation statistical methods are based on Riemann sum calculations (the current average calculation assumes Riemann sums of type left), these methods now also have this extra type input parameter (defaulting to left). Note that in most cases, the trapezoidal (or midpoint) methods would result in better accuracy. However, for backward compatibility reasons, the default has been kept on left. --------- Also-by: Florian Hotze <[email protected]> Signed-off-by: Mark Herwege <[email protected]>

mherwege requested a review from a team as a code owner November 22, 2024 10:15

mherwege force-pushed the riemann_sum branch 2 times, most recently from cbf98c2 to 5c76129 Compare November 22, 2024 10:33

mherwege force-pushed the riemann_sum branch from 8516df7 to 39a0bbe Compare November 22, 2024 17:07

andrewfg suggested changes Nov 24, 2024

View reviewed changes

...persistence/src/main/java/org/openhab/core/persistence/extensions/PersistenceExtensions.java Show resolved Hide resolved

jimtng mentioned this pull request Nov 30, 2024

Persistence: support riemann_sum methods openhab/openhab-jruby#362

Merged

jimtng reviewed Nov 30, 2024

View reviewed changes

This was referenced Dec 1, 2024

Extend Persistence Extensions with Riemann Sums openhab/openhab-js#401

Merged

[blockly] Riemann sum persistence extension and js tern definitions openhab/openhab-webui#2893

Merged

holgerfriedrich added the enhancement An enhancement or new feature of the Core label Dec 4, 2024

mherwege force-pushed the riemann_sum branch from 380d8ff to d2ca5aa Compare January 6, 2025 11:46

mherwege force-pushed the riemann_sum branch from e905777 to 1894e4d Compare February 17, 2025 16:42

holgerfriedrich reviewed Mar 23, 2025

View reviewed changes

mherwege added 2 commits March 24, 2025 15:11

riemann sums

df0de82

Signed-off-by: Mark Herwege <[email protected]>

reviewer feedback

2d45133

Signed-off-by: Mark Herwege <[email protected]>

mherwege force-pushed the riemann_sum branch from 1894e4d to 2d45133 Compare March 24, 2025 14:44

improve comments

a581539

Signed-off-by: Mark Herwege <[email protected]>

holgerfriedrich approved these changes Mar 24, 2025

View reviewed changes

holgerfriedrich merged commit f171ecc into openhab:main Mar 24, 2025
2 checks passed

holgerfriedrich added this to the 5.0 milestone Mar 24, 2025

mherwege deleted the riemann_sum branch March 24, 2025 18:58

mherwege mentioned this pull request Mar 25, 2025

Riemann sum documentation openhab/openhab-docs#2477

Merged

florian-h05 pushed a commit to openhab/openhab-js that referenced this pull request Mar 27, 2025

Extend Persistence Extensions with Riemann Sums (#401)

5f647b2

Refs on openhab/openhab-core#4461. Signed-off-by: Mark Herwege <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Persistence Extensions: Riemann Sums #4461

Persistence Extensions: Riemann Sums #4461

mherwege commented Nov 22, 2024 •

edited

Loading

mherwege commented Nov 22, 2024

mherwege commented Nov 22, 2024

mherwege commented Nov 23, 2024

jimtng commented Nov 26, 2024

mherwege commented Nov 26, 2024

jimtng Nov 30, 2024 •

edited

Loading

mherwege Nov 30, 2024

mherwege Nov 30, 2024

mherwege commented Nov 30, 2024 •

edited

Loading

florian-h05 commented Nov 30, 2024

holgerfriedrich left a comment •

edited

Loading

holgerfriedrich Mar 23, 2025

mherwege Mar 24, 2025

mherwege commented Mar 24, 2025

mherwege commented Mar 24, 2025 •

edited

Loading

holgerfriedrich left a comment

openhab-bot commented Mar 25, 2025

dandjo commented Mar 25, 2025

Persistence Extensions: Riemann Sums #4461

Persistence Extensions: Riemann Sums #4461

Conversation

mherwege commented Nov 22, 2024 • edited Loading

mherwege commented Nov 22, 2024

mherwege commented Nov 22, 2024

mherwege commented Nov 23, 2024

jimtng commented Nov 26, 2024

mherwege commented Nov 26, 2024

jimtng Nov 30, 2024 • edited Loading

Choose a reason for hiding this comment

mherwege Nov 30, 2024

Choose a reason for hiding this comment

mherwege Nov 30, 2024

Choose a reason for hiding this comment

mherwege commented Nov 30, 2024 • edited Loading

florian-h05 commented Nov 30, 2024

holgerfriedrich left a comment • edited Loading

Choose a reason for hiding this comment

holgerfriedrich Mar 23, 2025

Choose a reason for hiding this comment

mherwege Mar 24, 2025

Choose a reason for hiding this comment

mherwege commented Mar 24, 2025

mherwege commented Mar 24, 2025 • edited Loading

holgerfriedrich left a comment

Choose a reason for hiding this comment

openhab-bot commented Mar 25, 2025

dandjo commented Mar 25, 2025

mherwege commented Nov 22, 2024 •

edited

Loading

jimtng Nov 30, 2024 •

edited

Loading

mherwege commented Nov 30, 2024 •

edited

Loading

holgerfriedrich left a comment •

edited

Loading

mherwege commented Mar 24, 2025 •

edited

Loading