Update documentation on hdbpp. #356

dlacoste-esrf · 2020-10-01T08:35:52Z

Update hdbpp documentation.

break down the big document under services in smaller files for easier
management.

break down the big document under services in smaller files for easier management.

lorenzopivetta · 2020-10-30T09:53:39Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+Commands
+~~~~~~~~
+
+The commands availabile in the are summarized in commands-table.


Suggest to rephrase: "The commands available are summarized in commands-table."

Thanks, done

lorenzopivetta · 2020-10-30T09:58:33Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+The commands availabile in the are summarized in commands-table.
+
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ArchiverAdd            | add a new instance to the archivers list; the instance must have been already created and configured via jive/astor and the device shall be running; as per release adding an device to an existing instance is not supported   |


We should explicit the release in the second part of the phrase. Moreover may be rephrased as "in the current design, only one ConfigurationManager device per instance is supported"

Actually reading this sentence, I don't understand what it means so it needs to be explicited, agreed, but I don't know how. This command is to add an eventsubscriber to the list, isn't it ? I don't understand this per release part at all… a device of what to an instance of what ?

Indeed, these are two different points. Let's just remove the last part of the sentence. E.g. the ne one can read "add a new instance to the archivers list; the instance must have been already created and configured via jive/astor and the device shall be running"

The other point is that, currently, you cannot deploy two devices in the same ConfigurationManager instance. For sure it is not relevant here.

Thanks for the explanation, done.

lorenzopivetta · 2020-10-30T10:01:17Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ArchiverAdd            | add a new instance to the archivers list; the instance must have been already created and configured via jive/astor and the device shall be running; as per release adding an device to an existing instance is not supported   |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ArchiverRemove         | remove an from the list; neither the device instance nor the attributes configured are removed from the database                                                                                                                |


"remove an Archiver from..."
This is missing from almost all descriptions (was a latex \def in the original format...)

thank you, especially for pointing out all the places it was missing !

lorenzopivetta · 2020-10-30T10:02:16Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| AttributeAdd           | add an attribute to archiving                                                                                                                                                                                                   |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| AttributeAssign        | assign attribute to                                                                                                                                                                                                             |


assign attribute to Archiver

lorenzopivetta · 2020-10-30T10:02:53Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| AttributeAssign        | assign attribute to                                                                                                                                                                                                             |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| AttributeGetArchiver   | return in charge of attribute                                                                                                                                                                                                   |


return Archiver in charge of attribute

lorenzopivetta · 2020-10-30T10:05:11Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| Context                | set context to all managed archivers                                                                                                                                                                                            |
+------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| ResetStatistics        | reset statistics of and all                                                                                                                                                                                                     |


reset statistics of ConfigurationManager and all Archivers

lorenzopivetta · 2020-10-30T10:13:22Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+
+Table 1: Configuration Manager Commands.
+
+Note that the list of managed is stored into the ArchiverList device


Note that the list of managed Archivers is...

lorenzopivetta · 2020-10-30T10:14:07Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+Attributes
+~~~~~~~~~~
+
+The attributes of the are summarized in attributes-table.


The attributes of the ConfigurationManager device are...

lorenzopivetta · 2020-10-30T10:26:39Z

source/administration/services/hdbpp/hdbpp-es-interface.rst

+Table 2: Event Subscriber Attributes.
+
+The class and device properties availabile for configuration are shown
+in table. According to TANGO


I am missing something here…
table 1 are the command, and this one should be 2 no ?

Previous one is Table 2. But line 111 we're talking about properties, so this must read "...table 3".

Got it, thx.

lorenzopivetta · 2020-10-30T11:09:43Z

source/administration/services/hdbpp/hdbpp-es-interface.rst

+
+    The device dynamically laods the configured library configured (using dlopen()) during the device initialization.
+    See :ref:`database-interface` section for more information.
+


Not sure i remember correctly, but in a recent email exchange my understanding was the dynamic loading mechanism to be kept in place.

I went a bit ahead of myself here, I changed it.

lorenzopivetta · 2020-10-30T11:13:18Z

source/administration/services/hdbpp/hdbpp-es-interface.rst

+
+The first two attributes will be archived in both RUN and SHUTDOWN
+contexts; the last three only when in RUN.
+


Following is a general description of the Configuration manager. Suggest to move at the beginning of hdbpp-cm-interface.rst

good idea.
I am not sure the hdbpp-cm-interface.rst is the best place as it is supposed to describe the interface, this is more high level considerations here, but it still fits better than in the es-interface.
Thx for the suggestion

Another option i to move it in the general section...

lorenzopivetta · 2020-10-30T11:14:33Z

source/tools-and-extensions/archiving/HDB++.rst

@@ -41,7 +41,7 @@ Elements
 +--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 | HDB++ Diagnostic         | Standalone JAVA application that visualizes both the status of all the archiver device servers and the overall archiving system                                                                                                                                                                                                                                                                                                     |
 +--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Archiving DB             | Specific Database devoted to storing attribute values. It can be either Mysql or Cassandra for the moment.                                                                                                                                                                                                                                                                                                                          |
+| Archiving DB             | Specific Database devoted to storing attribute values. The currently supported backend are Mysql, Cassandra or Timescaledb.                                                                                                                                                                                                                                                                                                         |


Mysql, Cassanadra, Postgres or TimescaleDB

Indeed postgres as well, as a collateral, but it works.

Well, there's a native implementation for Postgres made by the guys in Russia...

ok, cool, I didn't know about that !

added missing "archiver" and "configuration manager" in hdbpp-cm-interface fixed warning on libhdbpp dynamic loading not supported. moved cm information from es page to cm one. added postgres as a supported backend, as it is in a way.

add table number

gscalam · 2020-11-20T09:07:32Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+workload over a large number of event subscriber shows up.
+The configuration manager device server will assist in
+the operations of adding, editing, moving, deleting an attribute the
+archiving system. All the configuration parameters, such as polling


Something is missing in "deleting an attribute the archiving system", maybe "deleting an attribute from the archiving system" ?

Indeed, thanks

gscalam · 2020-11-20T09:14:36Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+
+        -  to the specified device server
+
+#. move an attribute from an device server to another one


"from a device server"

Well spotted, thanks

gscalam · 2020-11-20T09:14:46Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+The configuration shall be possible via the device server API as well as
+via a dedicated GUI interface; the GUI just use the provided API.
+
+The may also expose a certain number of attributes to give the status of


The ConfigurationManager may

Damn it ! missed some, thanks.

gscalam · 2020-11-20T09:16:49Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+The may also expose a certain number of attributes to give the status of
+what is going on:
+
+-  total number of


total number of Archivers ?

gscalam · 2020-11-20T09:24:00Z

source/administration/services/hdbpp/hdbpp-cm-interface.rst

+~~~~~~~~~~~~~~~~
+
+--------------------+--------------------------------------------------------+
+| LibConfiguration   | configuration parameters for backend support library   |


Should we add a reference to different libhdbpp-xxx implementations documentation for details on syntax and possible parameters of LibConfiguration or should we add a section here in the documentation?
Same comment in hdbpp-es-interface where there is a small example of configuration for MySQL (incomplete).

actually the configurationmanager does not depend on the backends anymore… we should remove this libconfiguration…
And maybe add some stuff on the hdbpp-es side. I am not sure a full section is necessary for this, it is pretty straightforward.

gscalam · 2020-11-20T09:50:10Z

source/tools-and-extensions/archiving/HDB++.rst

@@ -58,12 +58,32 @@ Elements



-HDB++ inherits the database structure from the existing Tango Historical Data Base (:ref:`hdb_archiving`) and introduces new storage architecture possibilities, better internal diagnostic capabilities and an optimized API. Its design allows storing data into traditional database management systems such as MySQL or into NoSQL database such as Apache Cassandra.
+HDB++ inherits the database structure from the existing Tango Historical Data Base (:ref:`hdb_archiving`) and introduces new storage architecture possibilities, better internal diagnostic capabilities and an optimized API. Different backends to store the data can be implemented through an unified interface, currently Timescaledb, MySQL and Apache Cassandra are supported.



Missing PostgreSQL and ElasticSearch

Added both of them, and a comment on cassandra support being dropped as suggested by @bourtemb

gscalam · 2020-11-20T09:53:45Z

source/tools-and-extensions/archiving/HDB++.rst

@@ -41,7 +41,7 @@ Elements
 +--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
 | HDB++ Diagnostic         | Standalone JAVA application that visualizes both the status of all the archiver device servers and the overall archiving system                                                                                                                                                                                                                                                                                                     |
 +--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-| Archiving DB             | Specific Database devoted to storing attribute values. It can be either Mysql or Cassandra for the moment.                                                                                                                                                                                                                                                                                                                          |
+| Archiving DB             | Specific Database devoted to storing attribute values. The currently supported backend are Mysql, Cassandra, PostgreSQL or Timescaledb.                                                                                                                                                                                                                                                                                             |
 +--------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+


Missing ElasticSearch. There is a preliminary implementaion (libhdbpp-elk) that is probably going to be finalized and maintained by SKA. At the same time Cassandra is not going to be maintained. Should we mention it here?

I think we should mention that Cassandra backend is not going to be maintained and maybe on line 61 too.

Done and done, thanks

gscalam · 2020-11-20T09:55:21Z

source/tools-and-extensions/archiving/HDB++.rst

+* **libhdb++mysql**: HDB++ schema support for MySQL back-end
+* **libhdb++cassandra**: Cassandra back-end implementation of libhdb++
+* **libhdb++timescale**: Timescaledb back-end implementation of libhdb++
+


Same comment here about supported backends as above.

Indeed, indeed. Thanks

gscalam · 2020-11-20T10:01:51Z

source/tools-and-extensions/archiving/HDB++.rst

 .. _hdbpp-cm: https://github.com/tango-controls-hdbpp/hdbpp-cm
 .. _hdbpp-es: https://github.com/tango-controls-hdbpp/hdbpp-es
 .. _hdbpp-cm-es: https://github.com/tango-controls-hdbpp/hdbpp-cm-es
 .. _libhdbpp: https://github.com/tango-controls-hdbpp/libhdbpp
 .. _libhdbpp-mysql: https://github.com/tango-controls-hdbpp/libhdbpp-mysql
 .. _libhdbpp-mysql-legacy: https://github.com/tango-controls-hdbpp/libhdbpp-mysql-legacy
 .. _libhdbpp-cassandra: https://github.com/tango-controls-hdbpp/libhdbpp-cassandra
+.. _libhdbpp-timescale: https://github.com/tango-controls-hdbpp/libhdbpp-timescale
 .. _CassandraMonitor: https://github.com/tango-controls-hdbpp/CassandraMonitor


I would add link to libhdbpp-postgresql and libhdbpp-elk

It should be ok, now, or did I miss some other projects ?

It's WIP, but hdbpp-benchmark could be mentioned as "a project to compare performances of different HDB++ backends using dockers"

gscalam · 2020-11-20T10:02:25Z

source/tools-and-extensions/archiving/HDB++.rst



 Source code
 -----------

 The source code is available on GitHub in the following repositories:

+.. _hdbpp-timescale-project: https://github.com/tango-controls-hdbpp/hdbpp-timescale-project
 .. _hdbpp-cm: https://github.com/tango-controls-hdbpp/hdbpp-cm


hdbpp-mysql-project exists as well

…ra as out for support

Update documentation on hdbpp.

35a226e

break down the big document under services in smaller files for easier management.

dlacoste-esrf requested review from lorenzopivetta, bourtemb and gscalam October 1, 2020 08:35

lorenzopivetta reviewed Oct 30, 2020

View reviewed changes

dlacoste-esrf added 2 commits October 30, 2020 15:24

Fix some typos.

cbee3a8

added missing "archiver" and "configuration manager" in hdbpp-cm-interface fixed warning on libhdbpp dynamic loading not supported. moved cm information from es page to cm one. added postgres as a supported backend, as it is in a way.

Remove misplaced information for archiverAdd command documentation

cbc4540

add table number

gscalam reviewed Nov 20, 2020

View reviewed changes

dlacoste-esrf added 4 commits November 20, 2020 11:41

Fix grammar and missing words

69af211

Add missing ref to elastic search and postgresql project, tag Cassand…

d818104

…ra as out for support

fix malformed table

ad8c8ad

Add hdbpp-benchmark project

169cd86


		Table 1: Configuration Manager Commands.

		Note that the list of managed is stored into the ArchiverList device


		The device dynamically laods the configured library configured (using dlopen()) during the device initialization.
		See :ref:`database-interface` section for more information.


		The first two attributes will be archived in both RUN and SHUTDOWN
		contexts; the last three only when in RUN.


		- to the specified device server

		#. move an attribute from an device server to another one

		@@ -58,12 +58,32 @@ Elements



		HDB++ inherits the database structure from the existing Tango Historical Data Base (:ref:`hdb_archiving`) and introduces new storage architecture possibilities, better internal diagnostic capabilities and an optimized API. Its design allows storing data into traditional database management systems such as MySQL or into NoSQL database such as Apache Cassandra.
		HDB++ inherits the database structure from the existing Tango Historical Data Base (:ref:`hdb_archiving`) and introduces new storage architecture possibilities, better internal diagnostic capabilities and an optimized API. Different backends to store the data can be implemented through an unified interface, currently Timescaledb, MySQL and Apache Cassandra are supported.

Update documentation on hdbpp. #356

Are you sure you want to change the base?

Update documentation on hdbpp. #356

Conversation

dlacoste-esrf commented Oct 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lorenzopivetta Oct 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lorenzopivetta Oct 30, 2020 • edited Loading

Choose a reason for hiding this comment

lorenzopivetta Oct 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lorenzopivetta Oct 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lorenzopivetta Oct 30, 2020 •

edited

Loading

lorenzopivetta Oct 30, 2020 •

edited

Loading

lorenzopivetta Oct 30, 2020 •

edited

Loading

lorenzopivetta Oct 30, 2020 •

edited

Loading