Code

[controller] log compaction#1282

Code

Draft

WhitneyDeng wants to merge 58 commits intolinkedin:mainlinkedin/venice:mainfrom WhitneyDeng:log-compactionWhitneyDeng/venice:log-compactionCopy head branch name to clipboard

+946-6

Commits

Commits on Oct 29, 2024

• [controller] log compaction

  

  Whitney Deng

  committed
  ff3c1ae

  Copy full SHA for ff3c1ae

Commits on Oct 30, 2024

• [controller][server] complete log compaction candidate selection helper methods VeniceParentHelixAdmin::isCompactionReady VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours

  

  Whitney Deng

  committed
  6a7903d

  Copy full SHA for 6a7903d

• [controller][server][style] put log compaction criterion on newline for readability & maintainability

  

  Whitney Deng

  committed
  62fcf64

  Copy full SHA for 62fcf64

• [controller][server][documentation] VeniceParentHelixAdmin::getStoresForCompaction & VeniceParentHelixAdmin::isLastCompactionTimeOlderThanThresholdHours JavaDocs description

  Show description for 189ed58
  

  Whitney Deng

  committed
  189ed58

  Copy full SHA for 189ed58

Commits on Oct 31, 2024

• [controller][server][documentation] VeniceParentHelixAdmin::isCompactionReady & helper methods

  

  Whitney Deng

  committed
  8101408

  Copy full SHA for 8101408

• [controller][server][documentation] filter out stores where current version is not found

  

  Whitney Deng

  committed
  8160e5b

  Copy full SHA for 8160e5b

Commits on Nov 1, 2024

• [controller][server][unit test] Unit test for VeniceParentHelixAdmin::filterStoresForCompaction

  Show description for c28728c
  

  Whitney Deng

  committed
  c28728c

  Copy full SHA for c28728c

• [controller][server][unit test] move testFilterStoresForCompaction to TestVeniceParentHelixAdmin

  Show description for 5e481cb
  

  Whitney Deng

  committed
  5e481cb

  Copy full SHA for 5e481cb

• [controller][server][unit test][style] switch actual & expected value argument order

  

  Whitney Deng

  committed
  2a0f3f2

  Copy full SHA for 2a0f3f2

Commits on Nov 4, 2024

• [controller][server][unit test][style] switch actual & expected value argument order

  

  Whitney Deng

  committed
  f6ca080

  Copy full SHA for f6ca080

Commits on Nov 6, 2024

• [controller][server][style][PR] make VeniceParentHelixAdmin::isCompactionReady package private

  

  Whitney Deng

  committed
  bf73b48

  Copy full SHA for bf73b48

• [controller][server][style][PR] revert wildcard import

  

  Whitney Deng

  committed
  77d83c0

  Copy full SHA for 77d83c0

• [controller][server][style][PR] delete wildcard import

  

  Whitney Deng

  committed
  af12106

  Copy full SHA for af12106

• [controller][server][PR] change Admin::getStoresForCompaction return data type ArrayList -> List

  

  Whitney Deng

  committed
  e82e659

  Copy full SHA for e82e659

Commits on Nov 7, 2024

• [controller][server][PR]

  Show description for 373c208
  

  Whitney Deng

  committed
  373c208

  Copy full SHA for 373c208

Commits on Nov 21, 2024

• [controller]

  Show description for 2b11f9d
  

  Whitney Deng

  committed
  2b11f9d

  Copy full SHA for 2b11f9d

• [controller] create RepushOrchestratorProvider

  Show description for 1a7f015
  

  Whitney Deng

  committed
  1a7f015

  Copy full SHA for 1a7f015

• [controller] create LogCompactionService & initialise instance in VeniceController

  

  Whitney Deng

  committed
  1df10d9

  Copy full SHA for 1df10d9

• [controller][PR] change ArrayList to List datatype in contracts

  

  Whitney Deng

  committed
  4652dc9

  Copy full SHA for 4652dc9

Commits on Nov 22, 2024

• [controller][PR] move VeniceParentHelixAdmin::getStoresForCompaction logic to VeniceHelixAdmin

  

  Whitney Deng

  committed
  da5455c

  Copy full SHA for da5455c

• [controller] add triggerRepush endpoint

  Show description for f30f705
  

  Whitney Deng

  committed
  f30f705

  Copy full SHA for f30f705

• [controller] change repush terminology to compactStore

  Show description for 24c97a1
  

  Whitney Deng

  committed
  24c97a1

  Copy full SHA for 24c97a1

Commits on Nov 23, 2024

• [controller] create LogCompactionService

  Show description for 3991b22
  

  Whitney Deng

  committed
  3991b22

  Copy full SHA for 3991b22

• [controller][config] add SCHEDULED_LOG_COMPACTION_THREAD_COUNT config

  Show description for 43f5c40
  

  Whitney Deng

  committed
  43f5c40

  Copy full SHA for 43f5c40

• [controller][config][style] delete SCHEDULED_LOG_COMPACTION_THREAD_COUNT annotations & comments

  

  Whitney Deng

  committed
  f638e40

  Copy full SHA for f638e40

Commits on Nov 25, 2024

• [controller] Refactor VeniceHelixAdmin::getStoresForCompaction logic into CompactionManager

  Show description for 2f3966d
  

  Whitney Deng

  committed
  2f3966d

  Copy full SHA for 2f3966d

• [controller] add scheduled.log.compaction.interval.HR config for executor setup in LogCompactionService::startInner

  

  Whitney Deng

  committed
  8655b96

  Copy full SHA for 8655b96

Commits on Nov 26, 2024

• [controller][config] change schedule.log.compaction.interval.ms config time unit from hour to millisecond

  

  Whitney Deng

  committed
  1f44e45

  Copy full SHA for 1f44e45

• [controller][test] TestLogCompactionService::testScheduledExecution

  

  Whitney Deng

  committed
  e0bd4c9

  Copy full SHA for e0bd4c9

• [controller][config] change schedule.log.compaction.interval.ms config time unit from hour to millisecond

  

  Whitney Deng

  committed
  275b4ec

  Copy full SHA for 275b4ec

• [controller][config] create RepushOrchestrator interface

  Show description for cf6861b
  

  Whitney Deng

  committed
  cf6861b

  Copy full SHA for cf6861b

Commits on Nov 27, 2024

• [controller][PR]

  Show description for 201631d
  

  Whitney Deng

  committed
  201631d

  Copy full SHA for 201631d

Commits on Dec 5, 2024

• [controller] return type ArrayList > List

  

  Whitney Deng

  committed
  bcdb3da

  Copy full SHA for bcdb3da

Commits on Dec 6, 2024

• [controller][PR] revert wildcard import

  

  Whitney Deng

  committed
  2c65d3c

  Copy full SHA for 2c65d3c

• [controller][PR] prepend DEFAULT to public static final field in CompactionManager

  

  Whitney Deng

  committed
  9a9cc1d

  Copy full SHA for 9a9cc1d

• [controller][PR] move logcompaction tests to same package

  Show description for 5457c79
  

  Whitney Deng

  committed
  5457c79

  Copy full SHA for 5457c79

• test commit after renaming branch from log-compaction to whdeng/log-compaction-repush

  

  Whitney Deng

  committed
  083af27

  Copy full SHA for 083af27

• test commit after reverting branch name from whdeng/log-compaction-repush to log-compaction

  

  Whitney Deng

  committed
  dc0835e

  Copy full SHA for dc0835e

• [controller][PR] add and update docs and logs

  

  Whitney Deng

  committed
  a9a5aeb

  Copy full SHA for a9a5aeb

• [controller][PR] revert wildcard imports

  

  Whitney Deng

  committed
  5ac6468

  Copy full SHA for 5ac6468

Commits on Dec 9, 2024

• [controller][PR] add VeniceControllerClusterConfig::isLogCompactionEnabled feature flat

  Show description for c4dc3de
  

  Whitney Deng

  committed
  c4dc3de

  Copy full SHA for c4dc3de

• [controller][PR] add VeniceControllerMultiClusterConfig::isLogCompactionEnabled

  

  Whitney Deng

  committed
  7a851d9

  Copy full SHA for 7a851d9

• [controller] add LogCompactionTask::triggerSource

  Show description for b93c6fc
  

  Whitney Deng

  committed
  b93c6fc

  Copy full SHA for b93c6fc

Commits on Dec 10, 2024

• [controller][PR] in ControllerClient::getStoresForCompaction, modify QueryParams initialisation to take explicit clusterName directly

  

  Whitney Deng

  committed
  5a5dddb

  Copy full SHA for 5a5dddb

• [controller][PR] add ControllerClient::triggerRepush TODO

  Show description for 3f641f5
  

  Whitney Deng

  committed
  3f641f5

  Copy full SHA for 3f641f5

Commits on Dec 11, 2024

• [controller] repush job response handling flow

  Show description for 98171a5
  

  Whitney Deng

  committed
  98171a5

  Copy full SHA for 98171a5

Commits on Dec 12, 2024

• [controller] add/modify RepushJobResponse fields

  Show description for 8253c82
  

  Whitney Deng

  committed
  8253c82

  Copy full SHA for 8253c82

• [controller] catch unhandled Throwable

  Show description for 7eec4be
  

  Whitney Deng

  committed
  7eec4be

  Copy full SHA for 7eec4be

• [controller] VeniceHelixAdmin: only initialise RepushOrchestrator & CompactionManager if log compaction is enabled

  

  Whitney Deng

  committed
  759c2ae

  Copy full SHA for 759c2ae

Commits on Dec 13, 2024

• [controller] setup test controller in TestLogCompactionService

  Show description for 8a9dc9f
  

  Whitney Deng

  committed
  8a9dc9f

  Copy full SHA for 8a9dc9f

Commits on Dec 17, 2024

• [controller] add config for time since last log compaction threshold

  

  Whitney Deng

  committed
  87294ca

  Copy full SHA for 87294ca

• [controller] ConfigKeys::TIME_SINCE_LAST_LOG_COMPACTION_THRESHOLD_MS cont'd: update CompactionManager init

  

  Whitney Deng

  committed
  e6d2bb4

  Copy full SHA for e6d2bb4

Commits on Dec 18, 2024

• [controller][PR] lambda function instead of Optional

  

  Whitney Deng

  committed
  ec5acaa

  Copy full SHA for ec5acaa

Commits on Dec 19, 2024

• [controller] added integration test for scheduled log compaction

  Show description for 8a321ca
  

  Whitney Deng

  committed
  8a321ca

  Copy full SHA for 8a321ca

• [controller] incomplete integration test for log compaction in TestLogCompaction

  Show description for c5376ca
  

  Whitney Deng

  committed
  c5376ca

  Copy full SHA for c5376ca

• [controller] todo to extract TestHybrid::testHybridStoreLogCompaction to TestLogCompactionService

  

  Whitney Deng

  committed
  3a677e5

  Copy full SHA for 3a677e5

Commits on Jan 27, 2025

• [controller] pass repush orchestrator configs as VeniceProperties

  Show description for 0e205c8
  

  Whitney Deng

  committed
  0e205c8

  Copy full SHA for 0e205c8

• [format] comment spacing

  Show description for 1ddc155
  

  Whitney Deng

  committed
  1ddc155

  Copy full SHA for 1ddc155