IcingaDB: add changes queue & RedisConnection enhancements

[yhabteab]

This pull request introduces a new runtime changes queue to IcingaDB, along with several enhancements to the RedisConnection class. These changes aim to improve the memory footprint and number of duplicate (and thus superfluous) Redis queries. The problem of duplicate queries has been a long-standing issue in IcingaDB, and some hacky workarounds have been implemented in the past to mitigate it. This PR takes a more systematic approach as Julian described in #10186 to address the root cause. I will try to summarize the key changes below:

Changes Queue

A new changes queue has been introduced to IcingaDB, which allows for batching of all runtime updates for a given object in an efficient manner. The changes queue works as outlined below:

Before going into more detail, we should clarify what we mean by "changes". In this context, changes refer to any event
that requires a Redis write operation. This includes, but is not limited to:

• Object creation and deletion
• Object modification (e.g., updating attributes)
• Relationship updates (e.g., adding or removing links between objects)
• State related updates (e.g., state changes, acknowledgements, etc.)
• Dependency config and state updates

This new queue does not cover any history related writes, those types of events follow a different path and are not
affected by this change. The focus here is solely on runtime object changes that affect the normal non-historical operation of IcingaDB. Consequently, history and heartbeat related writes use their own dedicated Redis connection and do not interfere with any of the changes described here.

Now, here is how the changes queue operates:

When an object is modified, instead of immediately writing the changes to Redis, the object pointer is pushed onto the queue with a corresponding flag indicating the type of change required. As long as the object remains in the queue, any subsequent Redis write requests concerning that object are merged into the existing queued dirty bits. This means that no matter how many times e.g., a OnStateChange is triggered for a given object, only a single write operation will be performed when it is finally popped from the queue. Do note that an object can have multiple dirty bits set, so if both its attributes and state are modified while in the queue, a state and config update will be sent when it is processed.

The consumer of the changes queue is a new background worker that pops objects from the queue and performs the necessary Redis write operations. This worker doesn't immediately process objects as they are enqueued; instead, it waits for a short period (currently set to 1000ms) to allow for more changes to accumulate and be merged. After this wait period, the worker serializes the queued objects according to their dirty bits and sends the appropriate Redis commands. Though, there's also another restriction in place: when the used RedisConnection reaches a certain number of pending commands (currently set to 512), the worker won't dequeue any more objects from the changes queue until the pending commands drop below that threshold. This ensures that we don't unnecessarily waste memory by serializing too many objects in advance, if the Redis server isn't able to keep up.

To accommodate this new changes queue, quite a number of existing code has been refactored, so that we no longer perform immediate writes to Redis. Additionally, the RedisConnection class has been enhanced to support this new workflow.

RedisConnection Enhancements

Several enhancements have been made to the RedisConnection class to better support the changes queue and improve overall efficiency:

1. The RedisConnection now supports to safely disconnect connections if required. Previously, IcingaDB starts a separate connection for each config object type (e.g., host, service, etc.) to perfectly parallelize the initial sync phase. However, once the initial sync is complete, these separate connections are no longer necessary, but since there was no way to disconnect them, they remained open until an Icinga 2 restart (you can just verify this by running netstat -ant | grep 6380). With this enhancement, IcingaDB closes these extra connections right after the initial sync.
2. As a consequence of the changes queue, the Redis queries internal queue has significantly been simplified. As opposed to the previous version which used a complex data structure to correctly manage the query priorities, this version uses a std::deque for the write queue and a simple mechanism to insert high-priority items at the front. By default, items are processed in FIFO order, but if someone wants to immediately send a high-priority query it will be placed at the front of the queue (remember std::deque allows efficient insertion at both ends), and will overtake any normal priority items already queued. However, if there are already high-priority items in the queue, the new high-priority item will be inserted after them but still before any normal priority items, ensuring that all high-priority items are processed in the order they were enqueued.
3. And while I'm at it, I also took the chance to improve the WriteQueueItem type by replacing the previously used ridiculously verbose query types by a more compact std::variant based approach. This not only reduces memory usage but also makes clearer that each item represents exactly one of a defined set of query types and nothing else.

Now, IcingaDB is subscribed to the OnNextCheckChanged signal and not the dummy OnNextCheckUpdated signal anymore. Though, that dummy signal is still there since the IDO relies on it. The only behavioural change in IcingaDB as opposed to before is that the oldest pending Redis query is determined only on the primary Redis connection (the one used for history and heartbeats). If you guys think this is a problem, I can look into a way to have IcingaDB consider all connections when determining the oldest pending query.

resolves #10186

[jschmidt-icinga]

I didn't test extensively yet, but from a quick test everything seems to work as expected. I also can't speak much for the logic and performance implications of when to send which events to Redis since I barely touched that until now. I'll continue to look at this in the coming days and see if I can test this more thoroughly.

[jschmidt-icinga]

This introduces some tight bidirectional coupling between the IcingaDB (even though it's just the -ti file here) and RedisConnection objects. It would be nice if this could be avoided, though copying all the members doesn't seem elegant either.

[yhabteab]

I personally think, that this is 100% better than the previous version of this, and I don't see a problem with the bidirectional coupling either, since the RedisConnection class is meant to be used by the IcingaDB class only. Obviously, this is not that perfect either but if you have a better approach in mind, then feel free to suggest it.

[jschmidt-icinga]

I had already thought about it and couldn't come up with anything good to suggest, other than designing it that way from the start. One option would be to maybe make the members string_views, which would be fine here, since the connection object has a longer lifetime than the icingadb object, but might become an implementation detail when/if other classes want to use this class. Or just suck up the (small) memory cost and leave things as they were.

It's not the worst thing in the world either way. We have this kind of coupling in many places, like (JsonRpc|HttpServer)Connection<->ApiListener, but it is kind a ugly, design-wise.

Regarding RedisConnection being only used by IcingaDB: Recently I was briefly looking at caching Perfdata in Redis for persistence in case the target services go offline. It's always nice to at least keep the option open of using a class like that for something else in the future.

[yhabteab]

Or just suck up the (small) memory cost and leave things as they were.

It's not just about the memory cost but due to this ridiculously long parameter passing I would have to use. With master branch there are two places like this, and this PR add another one. So, decided simply squash them with approach.

icinga2/lib/icingadb/icingadb.cpp

Lines 84 to 86 in 35fdea8

	m_Rcon = new RedisConnection(GetHost(), GetPort(), GetPath(), GetUsername(), GetPassword(), GetDbIndex(),
	GetEnableTls(), GetInsecureNoverify(), GetCertPath(), GetKeyPath(), GetCaPath(), GetCrlPath(),
	GetTlsProtocolmin(), GetCipherList(), GetConnectTimeout(), GetDebugInfo());

icinga2/lib/icingadb/icingadb.cpp

Lines 94 to 96 in 35fdea8

	RedisConnection::Ptr con = new RedisConnection(GetHost(), GetPort(), GetPath(), GetUsername(), GetPassword(), GetDbIndex(),
	GetEnableTls(), GetInsecureNoverify(), GetCertPath(), GetKeyPath(), GetCaPath(), GetCrlPath(),
	GetTlsProtocolmin(), GetCipherList(), GetConnectTimeout(), GetDebugInfo(), m_Rcon);

Recently I was briefly looking at caching Perfdata in Redis for persistence in case the target services go offline. It's always nice to at least keep the option open of using a class like that for something else in the future.

If we ever end up using RedisConnection for other purposes, then we would definitely have to move it somewhere else. And while doing that, there will be other design decisions to make, I would consider this tiny bit of coupling acceptable for now that can further be improved later.

[yhabteab]

Please the updated code now. I've introduced a helper struct for all the parameters instead and will be copied only once.

[yhabteab]

I'll continue to look at this in the coming days and see if I can test this more thoroughly.

At the moment the integration tests from the Icinga DB repository are the only way to stress test this PR thoroughly. I've been running them ever since the initial implementation and I have almost gone crazy due to a subtle race condition that only showed up when running those tests.

[jschmidt-icinga]

a subtle race condition that only showed up when running those tests.

Can you describe this in a bit more detail so I know what to look out for, i.e. the symptoms of the race condition?

[yhabteab]

Can you describe this in a bit more detail so I know what to look out for, i.e. the symptoms of the race condition?

Well, the obvious symptom is that the integration tests (specifically the Redundancy Group ones) will sporadically fail because either Icinga 2 didn't sent a delete command when it's supposed to, or deleted something that it shouldn't have. Generally speaking, if the tests don't succeed then there is a bug in here.

[julianbrost]

Don't consider this a full review, just what I noticed at first glance.

[julianbrost]

Wait up to 5 seconds? Doesn't this just always wait 5 seconds?

And why are the read/write loops only woken after that?

[yhabteab]

Wait up to 5 seconds? Doesn't this just always wait 5 seconds?

Yes, it always waits 5 seconds.

And why are the read/write loops only woken after that?

Doesn't make any difference but I've moved these above the timer now.

[julianbrost]

That number should be just big enough that there are enough queries in flight so that the throughput isn't limited by latency. For any other queries, it's more desirable to stay in the worker queue than the Redis query queue, because in the first, they can still be combined with other operations. My intuition is that the number is bigger than it needs to be.

Alternatively, could it be feasible to check the state of the Redis connection so that we write queries until the write operations would block. so that we don't have (many) more queries in flight than the send and receive queues can hold.

[yhabteab]

I've reduced the number to 128 now but I'll have to think more thoroughly about the alternative approaches you suggested.

[julianbrost]

I'd rather initialize the field individually, i.e.:

RedisConnInfo::Ptr connInfo = new RedisConnInfo();
connInfo.EnableTls = GetEnableTls();
// ...

Then you can check whether the values are passed correctly here without needing IDE annotations (or very carefully checking them one by one) and also can just omit the following constructor:

icinga2/lib/icingadb/redisconnection.hpp

Lines 76 to 110 in ac841b3

	RedisConnInfo(
	bool enableTls,
	bool tlsInsecureNoverify,
	int port,
	int dbIndex,
	double connectTimeout,
	const String& host,
	const String& path,
	const String& user,
	const String& password,
	const String& tlsCertPath,
	const String& tlsKeyPath,
	const String& tlsCaPath,
	const String& tlsCrlPath,
	const String& tlsProtocolMin,
	const String& tlsCipherList,
	const DebugInfo& debugInfo
	)
	: EnableTls{enableTls},
	TlsInsecureNoverify{tlsInsecureNoverify},
	Port{port},
	DbIndex{dbIndex},
	ConnectTimeout{connectTimeout},
	Host{host},
	Path{path},
	User{user},
	Password{password},
	TlsCertPath{tlsCertPath},
	TlsKeyPath{tlsKeyPath},
	TlsCaPath{tlsCaPath},
	TlsCrlPath{tlsCrlPath},
	TlsProtocolMin{tlsProtocolMin},
	TlsCipherList{tlsCipherList},
	DbgInfo{debugInfo}
	{}

[yhabteab]

The members can't be constfied that way though. I've chosen this approach to make all the struct fields immutable.

[julianbrost]

Why was the nice initialization syntax ({.x = 42}) only added in C++20 (when it's already in C99)? :(

Anyway, I think we could still get a reasonable level of const-correctness if there was an RedisConnInfo::ConstPtr IcingaDB::GetRedisConnInfo() that does the initialization and is the only place where there's any non-const use of RedisConnInfo.

[yhabteab]

Changed.

[julianbrost]

Is there any case where the value of redisKey does not already imply what the value of hasChecksum will be? Or in other words: is there any redisKey where this function is called with both hasChecksum = true and hasChecksum = false?

If not, we shouldn't need to pass it around all the way but just have a place where we can look it up. That could be this function by just replacing all the uses of hasChecksum with true/false constants depending on the type.

However, I'm not too sure about the enum approach. Even though there's now this big switch here, there are still other places that map enum values to actual keys and also key names defined in other places. What do you think about the following idea? Instead of an enum, just make RedisKey a struct with members like Key, Checksum and basically instantiate it for the individual keys in global constants and passing them around as const RedisKey& (or const RedisKey* if there are places that don't like references).

[yhabteab]

Is there any case where the value of redisKey does not already imply what the value of hasChecksum will be? Or in other words: is there any redisKey where this function is called with both hasChecksum = true and hasChecksum = false?

No, there isn't such but only because it's easier to figure out on the call site whether that Redis key has also a corresponding checksum key to be deleted.

Even though there's now this big switch here, there are still other places that map enum values to actual keys and also key names defined in other places.

Yes, there are still places that use key names and I didn't refactor them yet because changing one of those would require basically a rewrite of the initial dump process. Though, I don't see any code part that maintains a map of enum values to actual Redis keys, so I'm not sure what you're referring to with this.

Instead of an enum, just make RedisKey a struct with members like Key, Checksum and basically instantiate it for the individual keys in global constants and passing them around as const RedisKey& (or const RedisKey* if there are places that don't like references).

I can't tell you yet whether this would make anything easier, but I can try it out.

[yhabteab]

Instead of an enum, just make RedisKey a struct with members like Key, Checksum and basically instantiate it for the individual keys in global constants and passing them around as const RedisKey& (or const RedisKey* if there are places that don't like references).

I can't tell you yet whether this would make anything easier, but I can try it out.

Now, after I've made a mental image of it, how exactly should the global map look like? Most of the types don't just map to a single Redis Key, but can span multiple keys (like Command types with their args/env/customvars). Or is your suggestion to fill that struct with all these kind of sub-keys as well, which would bloat the struct a lot?

[julianbrost]

Now, after I've made a mental image of it, how exactly should the global map look like?

My thoughts don't involve a map at all. Instead of the enum, I'd suggest something like this:

struct RedisKey
{
	std::string Key;
	bool Checksum;
};

const RedisKey RedisKeyUserGroupCustomVar {"icinga:usergroup:customvar", true};
// ...

Which wouldn't be longer than this function but could replace it and should also be helpful in other places.

Though, I don't see any code part that maintains a map of enum values to actual Redis keys, so I'm not sure what you're referring to with this.

I was thinking that such a change would also benefit other places if all Redis key names were basically available as a constant. So for example, that one could then use something like RedisKeyRedundancyGroupState.Key:

icinga2/lib/icingadb/icingadb-objects.cpp

Lines 1404 to 1407 in ac841b3

	addDependencyStateToStream(m_PrefixConfigObject + "redundancygroup:state", stateAttrs);
	addDependencyStateToStream(m_PrefixConfigObject + "dependency:edge:state", sharedGroupState);
	AddDataToHmSets(hMSets, RedisKey::RedundancyGroupState, depGroup->GetIcingaDBIdentifier(), stateAttrs);
	AddDataToHmSets(hMSets, RedisKey::DependencyEdgeState, depGroup->GetIcingaDBIdentifier(), sharedGroupState);

[yhabteab]

Which wouldn't be longer than this function but could replace it and should also be helpful in other places.

The keys aren't statically known and we have such code parts all over the place. How would make listing all the possible variables statically this any better? It wouldn't just be longer but will quite triple (if not even quadruple) the amount of code we would need to build in order to figure out which variable to use.

auto& typeCvs (hMSets[m_PrefixConfigObject + typeName + ":customvar"]);
auto& allCvs (hMSets[m_PrefixConfigObject + "customvar"]);

I was thinking that such a change would also benefit other places if all Redis key names were basically available as a constant. So for example, that one could then use something like RedisKeyRedundancyGroupState.Key:

Like I said before, all the leftovers that still use such string literals are kinda special cases and refactoring them as part of this PR would be quite out of scope. But following your example, I could also outsource this huge switch into a separate function that yields the corresponding Redis key for a given RedisKey enum value when needed and some of the string literals in this file could be replaced that way. But I fail to see the real benefit of having a huge list of static variables as opposed to this enum based approach.