[PATCH 1 of 2 stable] curses: do not initialize LC_ALL to user settings (issue6358)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH 1 of 2 stable] curses: do not initialize LC_ALL to user settings (issue6358)

Manuel Jacob
# HG changeset patch
# User Manuel Jacob <[hidden email]>
# Date 1593360165 -7200
#      Sun Jun 28 18:02:45 2020 +0200
# Branch stable
# Node ID 298402eac2b0ccbd70f399929f084599f8b2c4e4
# Parent  2632c1ed8f34988fc918d70ccc40b01c20c08a53
# EXP-Topic with_lc_type
curses: do not initialize LC_ALL to user settings (issue6358)

701341f57ceb moved the setlocale() call to right before curses was used. This
didn’t fully solve the problem it was supposed to solve (locale-dependent
functions, like date formatting/parsing and str methods on Python 2), but only
postponed it.

Initializing LC_CTYPE seems to be sufficient for curses to work correctly.
Therefore LC_CTYPE is set while curses is used and reset afterwards. Some
locale-dependent str methods might behave differently on Python 2 while curses
is used, but that shouldn’d be a problem.

diff --git a/hgext/histedit.py b/hgext/histedit.py
--- a/hgext/histedit.py
+++ b/hgext/histedit.py
@@ -1710,11 +1710,8 @@
         ctxs = []
         for i, r in enumerate(revs):
             ctxs.append(histeditrule(ui, repo[r], i))
-        # Curses requires setting the locale or it will default to the C
-        # locale. This sets the locale to the user's default system
-        # locale.
-        locale.setlocale(locale.LC_ALL, '')
-        rc = curses.wrapper(functools.partial(_chisteditmain, repo, ctxs))
+        with util.with_lc_ctype():
+            rc = curses.wrapper(functools.partial(_chisteditmain, repo, ctxs))
         curses.echo()
         curses.endwin()
         if rc is False:
diff --git a/mercurial/crecord.py b/mercurial/crecord.py
--- a/mercurial/crecord.py
+++ b/mercurial/crecord.py
@@ -574,14 +574,12 @@
     """
     ui.write(_(b'starting interactive selection\n'))
     chunkselector = curseschunkselector(headerlist, ui, operation)
-    # This is required for ncurses to display non-ASCII characters in
-    # default user locale encoding correctly.  --immerrr
-    locale.setlocale(locale.LC_ALL, '')
     origsigtstp = sentinel = object()
     if util.safehasattr(signal, b'SIGTSTP'):
         origsigtstp = signal.getsignal(signal.SIGTSTP)
     try:
-        curses.wrapper(chunkselector.main)
+        with util.with_lc_ctype():
+            curses.wrapper(chunkselector.main)
         if chunkselector.initexc is not None:
             raise chunkselector.initexc
         # ncurses does not restore signal handler for SIGTSTP
diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -22,6 +22,7 @@
 import gc
 import hashlib
 import itertools
+import locale
 import mmap
 import os
 import platform as pyplatform
@@ -3626,3 +3627,32 @@
         if not (byte & 0x80):
             return result
         shift += 7
+
+
+# Passing the '' locale means that the locale should be set according to the
+# user settings (environment variables).
+# Python sometimes avoids setting the global locale settings. When interfacing
+# with C code (e.g. the curses module or the Subversion bindings), the global
+# locale settings must be initialized correctly. Python 2 does not initialize
+# the global locale settings on interpreter startup. Python 3 sometimes
+# initializes LC_CTYPE, but not consistently at least on Windows. Therefore we
+# explicitly initialize it to get consistent behavior if it's not already
+# initialized. Since CPython commit 177d921c8c03d30daa32994362023f777624b10d,
+# LC_CTYPE is always initialized. If we require Python 3.8+, we should re-check
+# if we can remove this code.
+@contextlib.contextmanager
+def with_lc_ctype():
+    oldloc = locale.setlocale(locale.LC_CTYPE, None)
+    if oldloc == 'C':
+        try:
+            try:
+                locale.setlocale(locale.LC_CTYPE, '')
+            except locale.Error:
+                # The likely case is that the locale from the environment
+                # variables is unknown.
+                pass
+            yield
+        finally:
+            locale.setlocale(locale.LC_CTYPE, oldloc)
+    else:
+        yield
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

[PATCH 2 of 2 stable] convert: set LC_CTYPE around calls to Subversion bindings

Manuel Jacob
# HG changeset patch
# User Manuel Jacob <[hidden email]>
# Date 1593360165 -7200
#      Sun Jun 28 18:02:45 2020 +0200
# Branch stable
# Node ID b732cbd649021133921b24d8c81247a142150831
# Parent  298402eac2b0ccbd70f399929f084599f8b2c4e4
# EXP-Topic with_lc_type
convert: set LC_CTYPE around calls to Subversion bindings

The Subversion bindings require that LC_CTYPE is set. However, we don’t want to
set it all the time, as it changes the behavior of str methods on Python 2. The
taken approach is hopefully fine-grained enough to not trigger any
locale-specfic behavior of the str methods and coarse-grained enough to not
clutter the code.

Emulating the with-statement behavior in before() and after() should be safe, as
after() is always called when before() is called. hgext.convert.hg takes a
similar approach.

diff --git a/hgext/convert/subversion.py b/hgext/convert/subversion.py
--- a/hgext/convert/subversion.py
+++ b/hgext/convert/subversion.py
@@ -187,13 +187,14 @@
     """Fetch SVN log in a subprocess and channel them back to parent to
     avoid memory collection issues.
     """
-    if svn is None:
-        raise error.Abort(
-            _(b'debugsvnlog could not load Subversion python bindings')
-        )
+    with util.with_lc_ctype():
+        if svn is None:
+            raise error.Abort(
+                _(b'debugsvnlog could not load Subversion python bindings')
+            )
 
-    args = decodeargs(ui.fin.read())
-    get_log_child(ui.fout, *args)
+        args = decodeargs(ui.fin.read())
+        get_log_child(ui.fout, *args)
 
 
 class logstream(object):
@@ -420,18 +421,19 @@
         self.url = geturl(url)
         self.encoding = b'UTF-8'  # Subversion is always nominal UTF-8
         try:
-            self.transport = transport.SvnRaTransport(url=self.url)
-            self.ra = self.transport.ra
-            self.ctx = self.transport.client
-            self.baseurl = svn.ra.get_repos_root(self.ra)
-            # Module is either empty or a repository path starting with
-            # a slash and not ending with a slash.
-            self.module = urlreq.unquote(self.url[len(self.baseurl) :])
-            self.prevmodule = None
-            self.rootmodule = self.module
-            self.commits = {}
-            self.paths = {}
-            self.uuid = svn.ra.get_uuid(self.ra)
+            with util.with_lc_ctype():
+                self.transport = transport.SvnRaTransport(url=self.url)
+                self.ra = self.transport.ra
+                self.ctx = self.transport.client
+                self.baseurl = svn.ra.get_repos_root(self.ra)
+                # Module is either empty or a repository path starting with
+                # a slash and not ending with a slash.
+                self.module = urlreq.unquote(self.url[len(self.baseurl) :])
+                self.prevmodule = None
+                self.rootmodule = self.module
+                self.commits = {}
+                self.paths = {}
+                self.uuid = svn.ra.get_uuid(self.ra)
         except svn.core.SubversionException:
             ui.traceback()
             svnversion = b'%d.%d.%d' % (
@@ -477,7 +479,8 @@
             )
 
         try:
-            self.head = self.latest(self.module, latest)
+            with util.with_lc_ctype():
+                self.head = self.latest(self.module, latest)
         except SvnPathNotFound:
             self.head = None
         if not self.head:
@@ -494,6 +497,13 @@
             self.wc = None
         self.convertfp = None
 
+    def before(self):
+        self.with_lc_ctype = util.with_lc_ctype()
+        self.with_lc_ctype.__enter__()
+
+    def after(self):
+        self.with_lc_ctype.__exit__(None, None, None)
+
     def setrevmap(self, revmap):
         lastrevs = {}
         for revid in revmap:
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

[PATCH stable V2] curses: do not initialize LC_ALL to user settings (issue6358)

Manuel Jacob
In reply to this post by Manuel Jacob
# HG changeset patch
# User Manuel Jacob <[hidden email]>
# Date 1593360165 -7200
#      Sun Jun 28 18:02:45 2020 +0200
# Branch stable
# Node ID 1ebf068d88ead4e0b538bf6f8e3fcc9dac49ed16
# Parent  2632c1ed8f34988fc918d70ccc40b01c20c08a53
# EXP-Topic with_lc_type
curses: do not initialize LC_ALL to user settings (issue6358)

701341f57ceb moved the setlocale() call to right before curses was used. This
didn’t fully solve the problem it was supposed to solve (locale-dependent
functions, like date formatting/parsing and str methods on Python 2), but only
postponed it.

Initializing LC_CTYPE seems to be sufficient for curses to work correctly.
Therefore LC_CTYPE is set while curses is used and reset afterwards. Some
locale-dependent str methods might behave differently on Python 2 while curses
is used, but that shouldn’d be a problem.

diff --git a/hgext/histedit.py b/hgext/histedit.py
--- a/hgext/histedit.py
+++ b/hgext/histedit.py
@@ -201,7 +201,6 @@
     termios = None
 
 import functools
-import locale
 import os
 import struct
 
@@ -1710,11 +1709,8 @@
         ctxs = []
         for i, r in enumerate(revs):
             ctxs.append(histeditrule(ui, repo[r], i))
-        # Curses requires setting the locale or it will default to the C
-        # locale. This sets the locale to the user's default system
-        # locale.
-        locale.setlocale(locale.LC_ALL, '')
-        rc = curses.wrapper(functools.partial(_chisteditmain, repo, ctxs))
+        with util.with_lc_ctype():
+            rc = curses.wrapper(functools.partial(_chisteditmain, repo, ctxs))
         curses.echo()
         curses.endwin()
         if rc is False:
diff --git a/mercurial/crecord.py b/mercurial/crecord.py
--- a/mercurial/crecord.py
+++ b/mercurial/crecord.py
@@ -10,7 +10,6 @@
 
 from __future__ import absolute_import
 
-import locale
 import os
 import re
 import signal
@@ -574,14 +573,12 @@
     """
     ui.write(_(b'starting interactive selection\n'))
     chunkselector = curseschunkselector(headerlist, ui, operation)
-    # This is required for ncurses to display non-ASCII characters in
-    # default user locale encoding correctly.  --immerrr
-    locale.setlocale(locale.LC_ALL, '')
     origsigtstp = sentinel = object()
     if util.safehasattr(signal, b'SIGTSTP'):
         origsigtstp = signal.getsignal(signal.SIGTSTP)
     try:
-        curses.wrapper(chunkselector.main)
+        with util.with_lc_ctype():
+            curses.wrapper(chunkselector.main)
         if chunkselector.initexc is not None:
             raise chunkselector.initexc
         # ncurses does not restore signal handler for SIGTSTP
diff --git a/mercurial/util.py b/mercurial/util.py
--- a/mercurial/util.py
+++ b/mercurial/util.py
@@ -22,6 +22,7 @@
 import gc
 import hashlib
 import itertools
+import locale
 import mmap
 import os
 import platform as pyplatform
@@ -3626,3 +3627,32 @@
         if not (byte & 0x80):
             return result
         shift += 7
+
+
+# Passing the '' locale means that the locale should be set according to the
+# user settings (environment variables).
+# Python sometimes avoids setting the global locale settings. When interfacing
+# with C code (e.g. the curses module or the Subversion bindings), the global
+# locale settings must be initialized correctly. Python 2 does not initialize
+# the global locale settings on interpreter startup. Python 3 sometimes
+# initializes LC_CTYPE, but not consistently at least on Windows. Therefore we
+# explicitly initialize it to get consistent behavior if it's not already
+# initialized. Since CPython commit 177d921c8c03d30daa32994362023f777624b10d,
+# LC_CTYPE is always initialized. If we require Python 3.8+, we should re-check
+# if we can remove this code.
+@contextlib.contextmanager
+def with_lc_ctype():
+    oldloc = locale.setlocale(locale.LC_CTYPE, None)
+    if oldloc == 'C':
+        try:
+            try:
+                locale.setlocale(locale.LC_CTYPE, '')
+            except locale.Error:
+                # The likely case is that the locale from the environment
+                # variables is unknown.
+                pass
+            yield
+        finally:
+            locale.setlocale(locale.LC_CTYPE, oldloc)
+    else:
+        yield
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH stable V2] curses: do not initialize LC_ALL to user settings (issue6358)

Yuya Nishihara
On Mon, 29 Jun 2020 02:16:13 +0200, Manuel Jacob wrote:
> # HG changeset patch
> # User Manuel Jacob <[hidden email]>
> # Date 1593360165 -7200
> #      Sun Jun 28 18:02:45 2020 +0200
> # Branch stable
> # Node ID 1ebf068d88ead4e0b538bf6f8e3fcc9dac49ed16
> # Parent  2632c1ed8f34988fc918d70ccc40b01c20c08a53
> # EXP-Topic with_lc_type
> curses: do not initialize LC_ALL to user settings (issue6358)

Queued for stable, thanks.
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel