Fix some issues with empty daemon autoscale pools

Summary:
Ref T10811. I'm not sure this causes any real problems, but I caught a couple of errors in the production logs in this vein:

```
[13-Jun-2016 14:26:55 UTC] [2016-06-13 14:26:55] ERROR 8: Undefined index:  at [/core/lib/libphutil/src/daemon/PhutilDaemonOverseer.php:302]
[13-Jun-2016 14:26:55 UTC] arcanist(head=stable, ref.master=29839e8c72c5, ref.stable=7b0aac5c6f31), libcore(), phabricator(head=stable, ref.stable=b26ecb189e48), phutil(head=stable, ref.master=992abe4a420c, ref.stable=52748950bb36), services(head=stable, ref.master=5b51b63027b7, ref.stable=66cf59af4580)
[13-Jun-2016 14:26:55 UTC]   #0 PhutilDaemonOverseer::updateAutoscale() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:224]
[13-Jun-2016 14:26:55 UTC]   phacility#1 PhutilDaemonOverseer::run() called at [<phabricator>/scripts/daemon/launch_daemon.php:20]
[13-Jun-2016 14:26:55 UTC] [2016-06-13 14:26:55] EXCEPTION: (InvalidArgumentException) Argument 1 passed to PhutilDaemonOverseer::getAutoscaleProperty() must be an instance of PhutilDaemonHandle, null given, called in /core/lib/libphutil/src/daemon/PhutilDaemonOverseer.php on line 303 and defined at [<phutil>/src/error/PhutilErrorHandler.php:200]
[13-Jun-2016 14:26:55 UTC] arcanist(head=stable, ref.master=29839e8c72c5, ref.stable=7b0aac5c6f31), libcore(), phabricator(head=stable, ref.stable=b26ecb189e48), phutil(head=stable, ref.master=992abe4a420c, ref.stable=52748950bb36), services(head=stable, ref.master=5b51b63027b7, ref.stable=66cf59af4580)
[13-Jun-2016 14:26:55 UTC]   #0 PhutilErrorHandler::handleError(integer, string, string, integer, array) called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:278]
[13-Jun-2016 14:26:55 UTC]   phacility#1 PhutilDaemonOverseer::getAutoscaleProperty(NULL, string, integer) called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:303]
[13-Jun-2016 14:26:55 UTC]   phacility#2 PhutilDaemonOverseer::updateAutoscale() called at [<phutil>/src/daemon/PhutilDaemonOverseer.php:224]
[13-Jun-2016 14:26:55 UTC]   phacility#3 PhutilDaemonOverseer::run() called at [<phabricator>/scripts/daemon/launch_daemon.php:20]
```

These are caused by attempting to update empty autoscale pools. Currently, autoscale pools read configuration from the first daemon in the pool, but obviously that doesn't work if the pool is empty.

This could occur when shutting down or possibly while reloading daemons.

Also, add a diagnostic message bout pool autoscaling. This might be a little too chatty, but maybe useful for some other issues.

Test Plan:
  - A reliable way to hit this was `bin/phd debug task --autoscale`, then `^C` it. It no longer errors.
  - I used `bin/worker flood` to artificially create a lot of work, and saw the pool autoscale up and down properly.
  - I launched daemons normally (no autoscale).
  - I faked daemons as busy and saw the pool autoscale to full.
  - I changed config and saw daemons automatically reload properly.

Reviewers: chad

Reviewed By: chad

Maniphest Tasks: T10811

Differential Revision: https://secure.phabricator.com/D16110

Author: epriestley

Diff for: src/daemon/PhutilDaemonOverseer.php

@@ -25,6 +25,7 @@ final class PhutilDaemonOverseer extends Phobject {
   private $lastPidfile;
   private $startEpoch;
   private $autoscale = array();
+  private $autoscaleConfig = array();
 
   public function __construct(array $argv) {
     PhutilServiceProfiler::getInstance()->enableDiscardMode();
@@ -183,6 +184,18 @@ public function run() {
       $daemon->setTraceMemory($this->traceMemory);
 
       $this->addDaemon($daemon, $config);
+
+      $group = idx($config['autoscale'], 'group');
+      if (strlen($group)) {
+        if (isset($this->autoscaleConfig[$group])) {
+          throw new Exception(
+            pht(
+              'Two daemons are part of the same autoscale group ("%s"). '.
+              'Each daemon autoscale group must be unique.',
+              $group));
+        }
+        $this->autoscaleConfig[$group] = $config;
+      }
     }
 
     $should_reload = false;
@@ -271,17 +284,14 @@ private function removeDaemon(PhutilDaemonHandle $daemon) {
   }
 
   private function getAutoscaleGroup(PhutilDaemonHandle $daemon) {
-    return $this->getAutoscaleProperty($daemon, 'group');
-  }
-
-  private function getAutoscaleProperty(
-    PhutilDaemonHandle $daemon,
-    $key,
-    $default = null) {
-
     $id = $daemon->getDaemonID();
     $autoscale = $this->daemons[$id]['config']['autoscale'];
-    return idx($autoscale, $key, $default);
+    return idx($autoscale, 'group');
+  }
+
+  private function getAutoscaleProperty($group_key, $key, $default = null) {
+    $config = $this->autoscaleConfig[$group_key]['autoscale'];
+    return idx($config, $key, $default);
   }
 
   public function didBeginWork(PhutilDaemonHandle $daemon) {
@@ -298,11 +308,14 @@ public function didBeginIdle(PhutilDaemonHandle $daemon) {
   }
 
   public function updateAutoscale() {
+    if ($this->inGracefulShutdown) {
+      return;
+    }
+
     foreach ($this->autoscale as $group => $daemons) {
-      $daemon = $this->daemons[head_key($daemons)]['handle'];
-      $scaleup_duration = $this->getAutoscaleProperty($daemon, 'up', 2);
-      $max_pool_size = $this->getAutoscaleProperty($daemon, 'pool', 8);
-      $reserve = $this->getAutoscaleProperty($daemon, 'reserve', 0);
+      $scaleup_duration = $this->getAutoscaleProperty($group, 'up', 2);
+      $max_pool_size = $this->getAutoscaleProperty($group, 'pool', 8);
+      $reserve = $this->getAutoscaleProperty($group, 'reserve', 0);
 
       // Don't scale a group if it is already at the maximum pool size.
       if (count($daemons) >= $max_pool_size) {
@@ -344,7 +357,7 @@ public function updateAutoscale() {
       }
 
       if ($should_scale) {
-        $config = $this->daemons[$daemon_id]['config'];
+        $config = $this->autoscaleConfig[$group];
 
         $config['autoscale']['clone'] = true;
 
@@ -359,6 +372,13 @@ public function updateAutoscale() {
             'autoscale' => $config['autoscale'],
           ));
 
+        $this->logMessage(
+          'AUTO',
+          pht(
+            'Scaling pool "%s" up to %s daemon(s).',
+            $group,
+            new PhutilNumber(count($daemons) + 1)));
+
         $this->addDaemon($clone, $config);
 
         // Don't scale more than one pool up per iteration. Otherwise, we could