poller: make recheck timing configurable

Longer recheck is useful if the poller crashes and we want to catch
up with old work. Make it configurable via:

[poller]
recheck_period=3
recheck_lookback=9

The above are defaults.

Signed-off-by: Jakub Kicinski <[email protected]>

Author: kuba-moo

CONFIG.rst

@@ -0,0 +1,26 @@
+Config syntax
+~~~~~~~~~~~~~
+
+This document describes the fields of the config file and their meaning.
+
+poller
+======
+
+Section configuring patch ingest.
+
+recheck_period
+--------------
+
+During normal operation poller fetches only the new patches - patches which
+were sent since the previous check (minus 10 minutes to account for email lag).
+
+To catch patches which got stuck in the email systems for longer, or got sent with
+a date in the past poller will periodically scan patchwork looking back further into
+the past.
+
+``recheck_period`` defines the period of the long scans in hours.
+
+recheck_lookback
+----------------
+
+Defines the length of the long history scan, see ``recheck_period``.

pw_poller.py

@@ -78,6 +78,9 @@ def __init__(self) -> None:
         self.seen_series = set(self._state['done_series'])
         self.done_series = self.seen_series.copy()
 
+        self._recheck_period = config.getint('poller', 'recheck_period', fallback=3)
+        self._recheck_lookback = config.getint('poller', 'recheck_lookback', fallback=9)
+
     def init_state_from_disk(self) -> None:
         try:
             with open('poller.state', 'r') as f:
@@ -200,9 +203,9 @@ def run(self) -> None:
                 req_time = datetime.datetime.utcnow()
 
                 # Decide if this is a normal 4 minute history poll or big scan of last 12 hours
-                if prev_big_scan + datetime.timedelta(hours=3) < req_time:
+                if prev_big_scan + datetime.timedelta(hours=self._recheck_period) < req_time:
                     big_scan = True
-                    since = prev_big_scan - datetime.timedelta(hours=9)
+                    since = prev_big_scan - datetime.timedelta(hours=self._recheck_lookback)
                     log_open_sec(f"Big scan of last 12 hours at {req_time} since {since}")
                 else:
                     big_scan = False