Made Script::escapeString() not encode bytes above 0x7F, in turn allowing for script's parameter string values to be influenced by charset settings;

Added a $full parameter to Script::escapeString() that encodes all bytes (binary mode, basically);
Made Script::append() escape stream values in full, meaning that script stream values are NOT influenced by charset settings;
Util::fileGetContents() properly strips out the success token from the result when in streaming mode (by copying the stream to a new stream);
Util::count() now has a second argument passed as the request's "from" argument;
Streamlined Util::exec() result collection creation;
Doc and CS fixes.

Author: boenrobot

RELEASE-1.0.0b6

@@ -1,7 +1,7 @@
 Util stuff, mostly.
 
 * __BREAKING CHANGES:__
-    - Removed all $mode arguments from all Countable implementations (turns out no stable 5.6+ releases uses it...), and the COUNT_RECURSIVE implementations (which were mostly useless anyway). Util::count() has the $query argument as its first and only one.
+    - Removed all $mode arguments from all Countable implementations (turns out no stable 5.6+ releases uses it...), and the COUNT_RECURSIVE implementations (which were mostly useless anyway). Util::count() has the $query argument as its first one, and a new $from argument as a second one.
     - All Util CRUD methods throw RouterErrorException when the router returns an error. Previously, you'd need to inspect the returned value to check if there's an error.
     - Util::edit() is no longer an alias of Util::set(). It's now its own method that can modify or unset a single specified property.
     - Util::fileGetContents() now throws RouterErrorException if the file doesn't exist or if there are other problems with getting its contents. Previously, it returned FALSE for such cases.
@@ -11,6 +11,8 @@ Util stuff, mostly.
     - comment()
     - getCurrentTime()
     - newRequest()
+* Script::escapeString() no longer escapes bytes above 0x7F. This means that parameter string values are now influenced by charset settings, and thus no additional conversion is necessary. A new second argument can be set to TRUE to escape all bytes, including the previously untouched ASCII alphanumeric characters and underscores.
+* Stream parameters in Script::append() (and inherently, Script::prepare() and Util::exec()) now have ALL bytes escaped.
 * Script::parseValue() now supports letter notation for time (1h2m3s), not just double colon notation (01:02:03), modeled after RouterOS. Related to that is also that leading zeroes, and zero minutes and seconds are now optional (e.g. "1:" is a valid way of saying 1 hour). Sub-second information is rounded up to the nearest second on current PHP versions (future versions are expected to support sub-second information in DateInterval by allowing seconds to be a double; The code currently attempts to give DateInterval a double, falling back to rounding to a second).
 * Script::parseValue() now recognizes dates in the "M/d/Y H:i:s" format (used across RouterOS), and turns that into a DateTime object for that time (or midnight in UTC if the time part is omitted).
 * Util::getAll() now throws a NotSupportedException if the arguments "follow", "follow-only" or "count-only" are used. The first two, because PHP would hang (since Client::sendSync() is used under the hood), and the last one because it's unredable in the returned output (use Util::count() instead).

src/PEAR2/Net/RouterOS/Script.php

@@ -80,7 +80,7 @@ class Script
      *
      * @return mixed Depending on RouterOS type detected:
      *     - "nil" (the string "[]") or "nothing" (empty string) - NULL.
-     *     - "number" - int or double for large values.
+     *     - "num" - int or double for large values.
      *     - "bool" - a boolean.
      *     - "array" - an array, with the keys and values processed recursively.
      *     - "time" - a {@link DateInterval} object.
@@ -147,7 +147,7 @@ public static function parseValueToString($value)
      *
      * @return string|bool|int|double|null Depending on RouterOS type detected:
      *     - "nil" (the string "[]") or "nothing" (empty string) - NULL.
-     *     - "number" - int or double for large values.
+     *     - "num" - int or double for large values.
      *     - "bool" - a boolean.
      *     - Unrecognized type - casted to a string, unmodified.
      */
@@ -192,6 +192,7 @@ public static function parseValueToDateTime(
         $value,
         DateTimeZone $timezone = null
     ) {
+        $previous = null;
         $value = (string)$value;
         if ('' !== $value && preg_match(
             '#^
@@ -221,12 +222,13 @@ public static function parseValueToDateTime(
                     $timezone
                 );
             } catch (E $e) {
-                return $value;
+                $previous = $e;
             }
         }
         throw new ParserException(
             'The supplied value can not be converted to a DateTime',
-            ParserException::CODE_DATETIME
+            ParserException::CODE_DATETIME,
+            $previous
         );
     }
 
@@ -406,7 +408,7 @@ public static function parseValueToArray(
      * variables to it.
      *
      * This is particularly useful when you're creating scripts that you don't
-     * want to execute right now (as with {@link static::exec()}, but instead
+     * want to execute right now (as with {@link Util::exec()}, but instead
      * you want to store it for later execution, perhaps by supplying it to
      * "/system scheduler".
      *
@@ -420,9 +422,9 @@ public static function parseValueToArray(
      *     Array values are automatically processed with
      *     {@link static::escapeValue()}. Streams are also supported, and are
      *     processed in chunks, each with
-     *     {@link static::escapeString()}. Processing starts from the current
-     *     position to the end of the stream, and the stream's pointer stays at
-     *     the end after reading is done.
+     *     {@link static::escapeString()} with all bytes being escaped.
+     *     Processing starts from the current position to the end of the stream,
+     *     and the stream's pointer is left untouched after the reading is done.
      *
      * @return resource A new PHP temporary stream with the script as contents,
      *     with the pointer back at the start.
@@ -456,9 +458,9 @@ public static function prepare(
      *     Array values are automatically processed with
      *     {@link static::escapeValue()}. Streams are also supported, and are
      *     processed in chunks, each with
-     *     {@link static::escapeString()}. Processing starts from the current
-     *     position to the end of the stream, and the stream's pointer stays at
-     *     the end after reading is done.
+     *     {@link static::escapeString()} with all bytes being escaped.
+     *     Processing starts from the current position to the end of the stream,
+     *     and the stream's pointer is left untouched after the reading is done.
      *
      * @return int The number of bytes written to $stream is returned,
      *     and the pointer remains where it was after the write
@@ -481,7 +483,7 @@ public static function append(
                 $bytes += $writer->send('"');
                 while ($reader->isAvailable() && $reader->isDataAwaiting()) {
                     $bytes += $writer->send(
-                        static::escapeString(fread($pvalue, $chunkSize))
+                        static::escapeString(fread($pvalue, $chunkSize), true)
                     );
                 }
                 $bytes += $writer->send("\";\n");
@@ -518,7 +520,7 @@ public static function escapeValue($value)
     {
         switch(gettype($value)) {
         case 'NULL':
-            $value = '';
+            $value = '[]';
             break;
         case 'integer':
             $value = (string)$value;
@@ -573,22 +575,28 @@ public static function escapeValue($value)
      * larger string first), and you can be sure there won't be any code
      * injections coming from it.
      *
-     * For the sake of brevity of the output, alphanumeric characters and
-     * underscores are left untouched
+     * By default, for the sake of brevity of the output, ASCII alphanumeric
+     * characters and underscores are left untouched. And for the sake of
+     * character conversion, bytes above 0x7F are also left untouched.
      *
      * @param string $value Value to be escaped.
+     * @param bool   $full  Whether to escape all bytes in the string, including
+     *     ASCII alphanumeric characters, underscores and bytes above 0x7F.
      *
      * @return string The escaped value.
      *
-     * @internal Why leave ONLY those characters and not also others?
+     * @internal Why leave ONLY those ASCII characters and not also others?
      *     Because those can't in any way be mistaken for language constructs,
      *     unlike many other "safe inside strings, but not outside" ASCII
      *     characters, like ",", ".", "+", "-", "~", etc.
      */
-    public static function escapeString($value)
+    public static function escapeString($value, $full = false)
     {
+        if ($full) {
+            return self::_escapeCharacters(array($value));
+        }
         return preg_replace_callback(
-            '/[^\\_A-Za-z0-9]+/S',
+            '/[^\\_A-Za-z0-9\\x80-\\xFF]+/S',
             array(__CLASS__, '_escapeCharacters'),
             $value
         );
@@ -597,8 +605,9 @@ public static function escapeString($value)
     /**
      * Escapes a character for a RouterOS scripting context.
      *
-     * Escapes a character for a RouterOS scripting context. Intended to only be
-     * called for non-alphanumeric characters.
+     * Escapes a character for a RouterOS scripting context.
+     * Intended to only be called by {@link self::escapeString()} for the
+     * matching strings.
      *
      * @param array $chars The matches array, expected to contain exactly one
      *     member, in which is the whole string to be escaped.

src/PEAR2/Net/RouterOS/Util.php

@@ -226,9 +226,9 @@ public function newRequest(
      *     Array values are automatically processed with
      *     {@link static::escapeValue()}. Streams are also supported, and are
      *     processed in chunks, each processed with
-     *     {@link static::escapeString()}. Processing starts from the current
-     *     position to the end of the stream, and the stream's pointer is left
-     *     untouched after the reading is done.
+     *     {@link static::escapeString()} with all bytes being escaped.
+     *     Processing starts from the current position to the end of the stream,
+     *     and the stream's pointer is left untouched after the reading is done.
      *     Note that the script's (generated) name is always added as the
      *     variable "_", which will be inadvertently lost if you overwrite it
      *     from here.
@@ -296,42 +296,32 @@ public function exec(
         $request->setArgument('numbers', $name);
         $removeResult = $this->client->sendSync($request);
 
+        $results = new ResponseCollection(
+            array_merge(
+                $addResult->toArray(),
+                $runResult->toArray(),
+                $removeResult->toArray()
+            )
+        );
+
         if (count($runResult->getAllOfType(Response::TYPE_ERROR)) > 0) {
             throw new RouterErrorException(
                 'Error when running script',
                 RouterErrorException::CODE_SCRIPT_RUN_ERROR,
                 null,
-                new ResponseCollection(
-                    array_merge(
-                        $addResult->toArray(),
-                        $runResult->toArray(),
-                        $removeResult->toArray()
-                    )
-                )
+                $results
             );
         }
         if (count($removeResult->getAllOfType(Response::TYPE_ERROR)) > 0) {
             throw new RouterErrorException(
                 'Error when removing script',
                 RouterErrorException::CODE_SCRIPT_REMOVE_ERROR,
                 null,
-                new ResponseCollection(
-                    array_merge(
-                        $addResult->toArray(),
-                        $runResult->toArray(),
-                        $removeResult->toArray()
-                    )
-                )
+                $results
             );
         }
 
-        return new ResponseCollection(
-            array_merge(
-                $addResult->toArray(),
-                $runResult->toArray(),
-                $removeResult->toArray()
-            )
-        );
+        return $results;
     }
 
     /**
@@ -928,17 +918,27 @@ public function move($numbers, $destination = null)
      * queries are allowed as a criteria, in contrast with
      * {@link static::find()}, where numbers and callbacks are allowed also.
      *
-     * @param Query|null $query A query to filter items by. Without it, all items
-     *     are included in the count.
+     * @param Query|null           $query A query to filter items by.
+     *     Without it, all items are included in the count.
+     * @param string|resource|null $from  A comma separated list of item IDs.
+     *     Any items in the set that still exist at the time of couting
+     *     are included in the final tally. Note that the $query filters this
+     *     set further (i.e. the item must be in the list AND match the $query).
+     *     Leaving the value to NULL means all matching items at the current
+     *     menu are included in the count.
      *
      * @return int The number of items, or -1 on failure (e.g. if the
      *     current menu does not have a "print" command or items to be counted).
      */
-    public function count(Query $query = null)
+    public function count(Query $query = null, $from = null)
     {
-        $result = $this->client->sendSync(
-            new Request($this->menu . '/print count-only=""', $query)
-        )->end()->getProperty('ret');
+        $countRequest = new Request(
+            $this->menu . '/print count-only=""',
+            $query
+        );
+        $countRequest->setArgument('from', $from);
+        $result = $this->client->sendSync($countRequest)->end()
+            ->getProperty('ret');
 
         if (null === $result) {
             return -1;
@@ -1147,7 +1147,11 @@ public function fileGetContents($filename, $tmpScriptName = null)
             if (Stream::isStream($message)) {
                 $successToken = fread($message, 1/*strlen('&')*/);
                 if ('&' === $successToken) {
-                    return $message;
+                    $messageCopy = fopen('php://temp', 'r+b');
+                    stream_copy_to_stream($message, $messageCopy);
+                    rewind($messageCopy);
+                    fclose($message);
+                    return $messageCopy;
                 }
                 rewind($message);
             } elseif (strpos($message, '&') === 0) {