PHOENIX-7727 Eliminate IndexMetadataCache RPCs by leveraging server PTable cache

[virajjasani]

Jira: PHOENIX-7727 Eliminate IndexMetadataCache RPCs by leveraging server PTable cache and
retrieve IndexMaintainer objects for each active index from the PTable object.

To optimize rpc calls, use it only when all of these conditions are met:

1. Use server metadata feature is enabled (enabled by default).
2. New index design is used (IndexRegionObserver coproc).
3. Table is not of type System.
4. Either table has mutable indexes or server side handling of immutable indexes is enabled.
5. Table's UPDATE_CACHE_FREQUENCY is not ALWAYS. This ensures IndexRegionObserver does not have to make additional getTable() rpc call with each batchMutate() rpc call.
6. Table's UPDATE_CACHE_FREQUENCY is ALWAYS but addServerCache() rpc call is needed due to the size of mutations. Unless expensive addServerCache() rpc call is required, client can attach index maintainer mutation attribute so that IndexRegionObserver does not have to make additional getTable() rpc call with each batchMutate() rpc call with small mutation size (size < phoenix.index.mutableBatchSizeThreshold value).

If above conditions do not match and if the mutation size is greater than "phoenix.index.mutableBatchSizeThreshold" value, however if none of the data table indexes need to be sent to server (only index in state other than DISABLE, CREATE_DISABLE, PENDING_ACTIVE need to be sent to server), do not use expensive addServerCache() rpc call.

[virajjasani]

Note: add a test to validate num of getTable() rpc calls to syscat for table with UPDATE_CACHE_FREQUENCY > 0 and ALWAYS.

For table with UCF as ALWAYS, we might run into double the rpc calls to syscat:

1. UPSERT and SELECT execution at client side requiring PTable
2. IndexRegionObserver requiring PTable to get index maintainers using getTable()

Unfortunately, 2nd one is getting introduced with this change.

@tkhurana

[virajjasani]

How about we let the mutations with batch size < 3 and UCF as ALWAYS continue to send index maintainer as part of mutation attribute?

For other cases, we use new approach:

1. Any case with UCF other than ALWAYS
2. UCF as ALWAYS and mutation size >= 3, so that we at least avoid sending IndexMaintainer as separate rpc call always

[virajjasani]

Latest build with latest commit: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-2321/18/