Adding CJK support

Author: s-n-g

Changelog

@@ -1,3 +1,11 @@
+2019-10-18 s-n-g
+    * CJK Unified Ideographs supported by the line edittor
+    * On python 2, trying to edit a station whose name contains
+      non-ASCII characters is prohibited and will end up in
+      displaying a relevant message
+    * Search term will not be lost when resizing the window
+    * Fixing issues with presenting search history
+
 2019-09-08 s-n-g
     * Adding station editor ("a" and "A" to add a station, "e" to edit)
     * Line editor supports unlimited string length

README.html

@@ -37,6 +37,7 @@ <h2 id="table-of-contents">Table of contents <span style="padding-left: 10px;"><
 <li><a href="#config-file">Config file</a></li>
 <li><a href="#about-playlist-files">About Playlist files</a></li>
 <li><a href="#search-function">Search function</a></li>
+<li><a href="#line-editor">Line editor</a></li>
 <li><a href="#moving-stations-around">Moving stations around</a></li>
 <li><a href="#specifying-stations-encoding">Specifying stations’ encoding</a></li>
 <li><a href="#player-detection-selection">Player detection / selection</a></li>
@@ -184,6 +185,16 @@ <h2 id="search-function">Search function <span style="padding-left: 10px;"><sup
 <p>One can always get help by pressing the “<strong>?</strong>” key.</p>
 <p>After a search term has been successfully found (search is case insensitive), next occurrence can be obtained using the “<strong>n</strong>” key and previous occurrence can be obtained using the “<strong>N</strong>” key.</p>
 <p style="margin: 1.5em 4em 0 4em; text-indent: -2.5em;"><strong>Note:</strong> <strong>Python 2</strong> users are confined in typing ASCII characters only.</p>
+<h2 id="line-editor">Line editor <span style="padding-left: 10px;"><sup style="font-size: 50%"><a href="#" title="Go to top of the page">Top</a></sup></style></h2>
+<p><strong>PyRadio</strong> “<em>Search function</em>” and “<em>Station edior</em>” use a <em>line editor</em> to permit typing and editing stations’ data.</p>
+<p>The <em>line editor</em> works both on <strong>Python 2</strong> and <strong>Python 3</strong>, but does not provide the same functionality for both versions:</p>
+<ul>
+<li>In <strong>Python 2</strong>, only ASCII characters can be inserted.</li>
+<li>In <strong>Python 3</strong>, no such restriction exists. Furthermore, using CJK characters is also supported.</li>
+</ul>
+<h3 id="cjk-characters-support">CJK characters support</h3>
+<p>The <em>line editor</em> supports the insertion of <a target="_blank" href="https://en.wikipedia.org/wiki/CJK_Unified_Ideographs">CJK Unified Ideographs</a>, as described on <a target="_blank" href="https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)">CJK Unified Ideographs (Unicode block)</a> also known as URO, abbreviation of Unified Repertoire and Ordering. These characters, although encoded as a single code-point (character), actually take up a 2-character space, when rendered on the terminal.</p>
+<p>A depiction of the editor’s behavior can be seen at this image: <a target="_blank" href="https://members.hellug.gr/sng/pyradio/pyradio-editor.jpg">pyradio-editor.jpg</a>.</p>
 <h2 id="moving-stations-around">Moving stations around <span style="padding-left: 10px;"><sup style="font-size: 50%"><a href="#" title="Go to top of the page">Top</a></sup></style></h2>
 <p>Rearranging the order of the stations in the playlist is another feature <strong>PyRadio</strong> offers.</p>
 <p>All you have to do is specify the <em>source</em> station (the station to be moved) and the position it will be moved to (<em>target</em>).</p>

README.md

@@ -13,6 +13,7 @@ Ben Dowling - [https://github.com/coderholic](https://github.com/coderholic)
 * [Config file](#config-file)
 * [About Playlist files](#about-playlist-files)
 * [Search function](#search-function)
+* [Line editor](#line-editor)
 * [Moving stations around](#moving-stations-around)
 * [Specifying stations' encoding](#specifying-stations-encoding)
 * [Player detection / selection](#player-detection-selection)
@@ -223,6 +224,22 @@ After a search term has been successfully found (search is case insensitive), ne
 
 **Note:** **Python 2** users are confined in typing ASCII characters only.
 
+## Line editor
+
+**PyRadio** "*Search function*" and "*Station edior*" use a *line editor* to permit typing and editing stations' data. 
+
+The *line editor* works both on **Python 2** and **Python 3**, but does not provide the same functionality for both versions:
+
+
+* In **Python 2**, only ASCII characters can be inserted.
+* In **Python 3**, no such restriction exists. Furthermore, using CJK characters is also supported.
+
+### CJK characters support
+
+The *line editor* supports the insertion of [CJK Unified Ideographs](https://en.wikipedia.org/wiki/CJK_Unified_Ideographs), as described on [CJK Unified Ideographs (Unicode block)](https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)) also known as URO, abbreviation of Unified Repertoire and Ordering. These characters, although encoded as a single code-point (character), actually take up a 2-character space, when rendered on the terminal.
+
+A depiction of the editor's behavior can be seen at this image: [pyradio-editor.jpg](https://members.hellug.gr/sng/pyradio/pyradio-editor.jpg).
+
 ## Moving stations around
 
 Rearranging the order of the stations in the playlist is another feature **PyRadio** offers.

pyradio.1

@@ -243,7 +243,17 @@ After a search term has been successfully found (search is case insensitive), ne
 \fBPython 2\fR users are confined in typing ASCII characters only.
 
 
+.SH CJK CHARACTERS SUPPORT
 
+The \fIline editor\fR supports the insertion of \fICJK Unified Ideographs [1]\fR, as described on \fICJK Unified Ideographs (Unicode block) [2]\fR, also known as URO, abbreviation of Unified Repertoire and Ordering. These characters, although encoded as a single code-point (character), actually take up a 2-character space, when rendered on the terminal.
+
+A depiction of the editor's behavior can be seen at this image:
+
+\fIhttps://members.hellug.gr/sng/pyradio/pyradio-editor.jpg\fR
+
+[1] \fIhttps://en.wikipedia.org/wiki/CJK_Unified_Ideographs\fR
+
+[2] \fIhttps://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)\fR
 
 
 .SH MOVING STATIONS AROUND

pyradio/edit.py

@@ -32,6 +32,8 @@ def __init__(self, parent, width, begin_y, begin_x, **kwargs):
             self._range_command='range'
 
     def show(self, parent_win, repaint=False):
+        if repaint:
+            tmp = self.string
         if parent_win is not None:
             self.parent_win = parent_win
 
@@ -51,25 +53,30 @@ def show(self, parent_win, repaint=False):
                     self._caption_win.addstr(0, x-1, '┤'.encode('utf-8'), self.box_color)
                 except:
                     pass
-        if not repaint:
+        if repaint:
+            self.string = tmp
+            self.keep_restore_data()
+            self._caption_win.refresh()
+            self.refreshEditWindow(opening=True)
+        else:
             self.string = self._displayed_string = ''
-            self._curs_pos = self._disp_curs_pos = 0
+            self._first = self._curs_pos = self._disp_curs_pos = 0
             self._edit_win.erase()
             self._edit_win.chgat(0, 0, 1, self.cursor_color)
             if self._has_history:
                 self._input_history.reset_index()
-        self._caption_win.refresh()
-        self._edit_win.refresh()
+            self._caption_win.refresh()
+            self._edit_win.refresh()
 
     def _get_history_next(self):
         """ callback function for key down """
         if self._has_history:
-            self.string = self._input_history.return_history(1)
+            self.string = self._input_history.return_history(1, self.string)
 
     def _get_history_previous(self):
         """ callback function for key up """
         if self._has_history:
-            self.string = self._input_history.return_history(-1)
+            self.string = self._input_history.return_history(-1, self.string)
 
     def get_next(self, a_list, start=0, stop=None):
         if self.string:
@@ -485,7 +492,7 @@ def keypress(self, char):
                     self._encoding = 'utf-8'
                 self._orig_encoding = self._encoding
                 for i in range(0,2):
-                    self._line_editor[i]._reset_position = True
+                    self._line_editor[i]._go_to_end()
             elif self._focus <= 1:
                 """
                  Returns:

pyradio/simple_curses_widgets.py

@@ -212,13 +212,22 @@ def string(self, val):
         self._go_to_end()
 
     def _is_cjk(self):
+        """ Check if string contains CJK characters.
+            If string is empty reset history index """
         old_cjk = self._cjk
         if len(self.string) == cjklen(self.string):
             self._cjk = False
         else:
             self._cjk = True
+        if self.string == '' and self._has_history:
+            self._input_history.reset_index()
         if logger.isEnabledFor(logging.DEBUG) and self._cjk != old_cjk:
-                logger.debug('CJK editing is {}'.format('ON' if self._cjk else 'OFF'))
+                logger.debug('=== CJK editing is {} ==='.format('ON' if self._cjk else 'OFF'))
+
+    def keep_restore_data(self):
+        """ Keep a copy of current editor state
+            so that it can be restored later. """
+        self._restore_data = [ self.string, self._displayed_string, self._curs_pos, self._disp_curs_pos, self._first ]
 
     def getmaxyx(self):
         return self._caption_win.getmaxyx()
@@ -320,9 +329,7 @@ def refreshEditWindow(self, opening=False):
 
         if self.focused:
             if logger.isEnabledFor(logging.DEBUG):
-                logger.debug('refreshEditWindow: first={0}, curs={1}, dcurs={2}, max={3}'.format(self._first, self._curs_pos, self._disp_curs_pos, self._max_chars_to_display))
-                logger.debug('refreshEditWindow:      full string: "{}"'.format(self.string))
-                logger.debug('refreshEditWindow: displayed string: "{}"'.format(self._displayed_string))
+                logger.debug('refreshEditWindow:\n  first={0}, curs={1}, dcurs={2}, max={3}\n  len={4}, cjklen={5}\n  string="{6}"\n  len={7}, cjklen={8}\n  disstr="{9}"'.format(self._first, self._curs_pos, self._disp_curs_pos, self._max_chars_to_display, len(self.string), cjklen(self.string), self.string, len(self._displayed_string), cjklen(self._displayed_string), self._displayed_string))
             self._edit_win.chgat(0, self._disp_curs_pos, 1, self.cursor_color)
 
         self._edit_win.refresh()
@@ -498,52 +505,60 @@ def _previous_word(self):
                         break
             if pos == 0:
                 # word_delimiter not found:
-                self._curs_pos = 0
-                self._first =0
+                self._go_to_start()
+                return
             else:
                 # word delimiter found
                 if str_len < self._max_chars_to_display or \
                         pos >= self._first:
                     # pos is on screen
-                    #logger.error('DE 1 pos = {0}, first = {1}, curs = {2}, len = {3}, max = {4}'.format(pos, self._first, self._curs_pos, str_len, self._max_chars_to_display))
                     self._curs_pos = pos - self._first + 1
-                    #logger.error('DE 2 pos = {0}, first = {1}, curs = {2}, len = {3}, max = {4}'.format(pos, self._first, self._curs_pos, str_len, self._max_chars_to_display))
                 else:
-                    #logger.error('DE 3 pos = {0}, first = {1}, curs = {2}, len = {3}, max = {4}'.format(pos, self._first, self._curs_pos, str_len, self._max_chars_to_display))
                     self._first = n + 1
                     self._curs_pos = 0
+                    self._displayed_string = self.string[self._first:self._first+self._max_chars_to_display]
+                self._disp_curs_pos = cjklen(self._displayed_string[:self._curs_pos])
+                while cjklen(self._displayed_string) > self._max_chars_to_display:
+                    self._displayed_string = self._displayed_string[:-1]
 
     def _next_word(self):
-        pos = cjklen(self._string)
-        str_len = pos
+        if self._at_end_of_sting():
+            return
+        if self._first + self._curs_pos + 1 >= len(self.string):
+            self._go_to_end()
+            return
+        pos = 0
         for n in range(self._first + self._curs_pos + 1, len(self.string)):
             if self._string[n] in self._word_delim:
                 pos = n
                 break
-        if pos == str_len:
-            # word delimiter not found
-            self._first = str_len - self._max_chars_to_display
-            if self._first < 0:
-                self._first = 0
-            self._curs_pos = pos - self._first
-            #logger.error('DE x pos = {0}, first = {1}, curs = {2}, len = {3}, max = {4}'.format(pos, self._first, self._curs_pos, str_len, self._max_chars_to_display))
-        else:
-            # word delimiter found
-            if str_len < self._max_chars_to_display or \
-                    pos < self._first + self._max_chars_to_display:
+        if pos >= len(self.string):
+            pos = 0
+        if pos > 0:
+            if pos < len(self._displayed_string):
                 # pos is on screen
-                self._curs_pos = pos - self._first + 1
+                self._curs_pos = pos + 1 - self._first
+                self._disp_curs_pos = cjklen(self._displayed_string[:self._curs_pos])
             else:
-                # pos is not on screen
-                #logger.error('DE 1 pos = {0}, len = {1}, max = {2}'.format(pos, str_len, self._max_chars_to_display))
-                self._first = pos
-                self._curs_pos = 1
-                pos = 0
-                while str_len - (self._first + pos + 1) < self._max_chars_to_display:
-                    pos -= 1
-                self._first = self._first + pos + 1
-                self._curs_pos = self._curs_pos + abs(pos) - 1
-                #logger.error('DE 2 pos = {0}, len = {1}, max = {2}'.format(pos, str_len, self._max_chars_to_display))
+                if pos < self._first + len(self._displayed_string):
+                    # pos is on middle and on screen
+                    self._curs_pos = pos - self._first + 1
+                    self._disp_curs_pos = cjklen(self._displayed_string[:self._curs_pos])
+                else:
+                    # pos is off screen
+                    self._first = 0
+                    self._curs_pos = pos + 2
+                    self._displayed_string = tmp = self.string[:self._curs_pos]
+                    while cjklen(tmp) > self._max_chars_to_display:
+                        self._first += 1
+                        tmp = self._displayed_string[self._first:]
+                    self._displayed_string = tmp
+                    self._curs_pos = len(self._displayed_string) - 1
+                    self._disp_curs_pos = cjklen(self._displayed_string[:-1])
+
+        else:
+            # word delimiter not found
+            self._go_to_end()
 
     def _go_right(self):
         if self.string and not self._at_end_of_sting():
@@ -606,13 +621,14 @@ def _go_left(self):
                     while cjklen(self._displayed_string) > self._max_chars_to_display:
                         self._displayed_string = self._displayed_string[:-1]
             else:
+                logger.error('simple')
                 self._go_left_simple()
 
     def _go_left_simple(self):
         if len(self.string) < self._max_chars_to_display:
-                self._curs_pos -= 1
-                if self._curs_pos < 0:
-                    self._curs_pos = 0
+            self._curs_pos -= 1
+            if self._curs_pos < 0:
+                self._curs_pos = 0
         else:
             if self._curs_pos == 0:
                 self._first -= 1
@@ -680,10 +696,9 @@ def keypress(self, win, char):
             # display help
             if logger.isEnabledFor(logging.DEBUG):
                 logger.debug('action: help')
-            self._restore_data = [ self.string, self._displayed_string, self._curs_pos, self._disp_curs_pos, self._first ]
+            self.keep_restore_data()
             return 2
 
-
         elif char in (curses.KEY_ENTER, ord('\n'), ord('\r')):
             """ ENTER """
             if logger.isEnabledFor(logging.DEBUG):
@@ -745,7 +760,7 @@ def keypress(self, win, char):
             if self.string:
                 if logger.isEnabledFor(logging.DEBUG):
                     logger.debug('action: LEFT')
-            self._go_left()
+                self._go_left()
 
         elif char in (curses.KEY_HOME, curses.ascii.SOH):
             """ KEY_HOME, ^A """
@@ -880,6 +895,8 @@ def keypress(self, win, char):
                     else:
                         self._string = self._string[:self._first + self._curs_pos] + chr(char) + self._string[self._first + self._curs_pos:]
                         self._add_to_end = False
+                        self._curs_pos+=1
+                        self._disp_curs_pos = self._curs_pos
                         self._displayed_string=self.string[self._first:self._first+self._max_chars_to_display]
             else:
                 if platform.startswith('win'):
@@ -894,6 +911,7 @@ def keypress(self, win, char):
                     self._displayed_string=self._string[self._first:self._first+self._curs_pos]
                 else:
                     self._string = self._string[:self._first + self._curs_pos] + char + self._string[self._first + self._curs_pos:]
+                    self._curs_pos+=1
                     self._add_to_end = False
                     self._displayed_string=self.string[self._first:self._first+self._max_chars_to_display]
             if self._add_to_end:
@@ -910,6 +928,10 @@ def keypress(self, win, char):
                 # adding to middle of string
                 while cjklen(self._displayed_string) > self._max_chars_to_display:
                     self._displayed_string=self._displayed_string[:-1]
+                if self._cjk:
+                    self._disp_curs_pos = cjklen(self._displayed_string[:self._curs_pos])
+                else:
+                    self._disp_curs_pos=self._curs_pos
 
         self.refreshEditWindow()
         return 1
@@ -954,7 +976,7 @@ def get_check_next_byte():
                 if is_wide(out) and not self._cjk:
                     self._cjk = True
                     if logger.isEnabledFor(logging.DEBUG):
-                        logger.debug('CJK editing is ON')
+                        logger.debug('=== CJK editing is ON ===')
         return out
 
     def _encode_string(self, data):
@@ -1004,8 +1026,8 @@ def run(self):
 class SimpleCursesLineEditHistory(object):
 
     def __init__(self):
-        self._history = []
-        self._active_history_index = -1
+        self._history = [ '' ]
+        self._active_history_index = 0
 
     def add_to_history(self, a_string):
         if a_string:
@@ -1021,17 +1043,19 @@ def add_to_history(self, a_string):
                 self._history.append(a_string)
             self._active_history_index = len(self._history)
 
-    def return_history(self, direction):
+    def return_history(self, direction, current_string):
         if self._history:
             self._active_history_index += direction
             if self._active_history_index <= -1:
                 self._active_history_index = len(self._history) - 1
             elif self._active_history_index >= len(self._history):
                 self._active_history_index = 0
             ret =  self._history[self._active_history_index]
+            if ret.lower() == current_string.lower():
+                return self.return_history(direction, current_string)
             return ret
         return ''
 
     def reset_index(self):
-        self._active_history_index = -1
+        self._active_history_index = 0