feat: allocating by function executor

[seriousben]

Context

As part of moving responsibilities from the executors to the server, we need to make the server aware of function executors.

What

This PR replaces the allocations_by_fn indexes by function_executors_by_executor and allocations_by_executor.

Details:

• Metrics and /internal/allocations are function executor aware
• ExecutorMetadata only gets upserted if the heartbeat state hash changes.
• ExecutorUpserted state change now triggers reconcilation in the graph processor
• Current executor reconciliation does two things: 1) ensure all current FE respect current dev + allow list + 2) remove any FE that has been removed on the executors. We also reallocate immediately any tasks that gets deallocated.
• Task Allocation now is structured differently:
  • 1. get_executor_candidates
  • 2. select_executor
  • 3. ensure function executor
  • 4. create allocation
• A new migration reallocates any existing allocation in order to make sure all allocations have a function executor ID associated to them

Testing

• Test Executor upsert + reconciliation logic
• Test allocation leveraging different function executors

Contribution Checklist

• If a Python package was changed, please run make fmt in the package directory.
• If the server was changed, please run make fmt in server/.
• Make sure all PR Checks are passing.

[seriousben]

This was a gap before. We would start any DB on version zero without migrating.

[eabatalov]

fyi, test failed:

======================================================================
FAIL: test_function_routing (__main__.TestFunctionAllowlist)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/indexify/indexify/indexify/tests/./cli/test_function_allowlist.py", line 227, in test_function_routing
    assert (
AssertionError: Not all executors were used: {'executor_dev': 800, 'executor_a': 0, 'executor_b': 0, 'executor_c': 0}

[eabatalov]

Approving but please fix the test failure unless it's a known issue. Seems like it's not.