D8337: pycompat: change argv conversion semantics

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

D8337: pycompat: change argv conversion semantics

valentin.gatienbaron (Valentin Gatien-Baron)
indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Use of os.fsencode() to convert Python's sys.argv back to bytes
  was not correct because it isn't the logically inverse operation
  from what CPython was doing under the hood.
 
  This commit changes the logic for doing the str -> bytes
  conversion. This required a separate implementation for
  POSIX and Windows.
 
  The Windows behavior is arguably not ideal. The previous
  behavior on Windows was leading to failing tests, such as
  test-http-branchmap.t, which defines a utf-8 branch name
  via a command argument. Previously, Mercurial's argument
  parser looked to be receiving wchar_t bytes in some cases.
  After this commit, behavior on Windows is compatible with
  Python 2, where CPython did not implement `int wmain()` and
  Windows was performing a Unicode to ANSI conversion on the
  wchar_t native command line.
 
  Arguably better behavior on Windows would be for Mercurial to
  preserve the original Unicode sequence coming from Python and
  to wrap this in a bytes-like type so we can round trip safely.
  But, this would be new, backwards incompatible behavior. My
  goal for this commit was to converge Mercurial behavior on
  Python 3 on Windows to fix busted tests. And I believe I was
  successful, as this commit fixes 9 tests on my Windows
  machine and 14 tests in the AWS CI environment!

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D8337

AFFECTED FILES
  mercurial/pycompat.py

CHANGE DETAILS

diff --git a/mercurial/pycompat.py b/mercurial/pycompat.py
--- a/mercurial/pycompat.py
+++ b/mercurial/pycompat.py
@@ -98,6 +98,7 @@
     import codecs
     import functools
     import io
+    import locale
     import struct
 
     if os.name == r'nt' and sys.version_info >= (3, 6):
@@ -148,15 +149,36 @@
     stdout = sys.stdout.buffer
     stderr = sys.stderr.buffer
 
-    # Since Python 3 converts argv to wchar_t type by Py_DecodeLocale() on Unix,
-    # we can use os.fsencode() to get back bytes argv.
-    #
-    # https://hg.python.org/cpython/file/v3.5.1/Programs/python.c#l55
-    #
-    # On Windows, the native argv is unicode and is converted to MBCS bytes
-    # since we do enable the legacy filesystem encoding.
     if getattr(sys, 'argv', None) is not None:
-        sysargv = list(map(os.fsencode, sys.argv))
+        # On POSIX, the char** argv array is converted to Python str using
+        # Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which isn't
+        # directly callable from Python code. So, we need to emulate it.
+        # Py_DecodeLocale() calls mbstowcs() and falls back to mbrtowc() with
+        # surrogateescape error handling on failure. These functions take the
+        # current system locale into account. So, the inverse operation is to
+        # .encode() using the system locale's encoding and using the
+        # surrogateescape error handler. The only tricky part here is getting
+        # the system encoding correct, since `locale.getlocale()` can return
+        # None. We fall back to the filesystem encoding if lookups via `locale`
+        # fail, as this seems like a reasonable thing to do.
+        #
+        # On Windows, the wchar_t **argv is passed into the interpreter as-is.
+        # Like POSIX, we need to emulate what Py_EncodeLocale() would do. But
+        # there's an additional wrinkle. What we really want to access is the
+        # ANSI codepage representation of the arguments, as this is what
+        # `int main()` would receive if Python 3 didn't define `int wmain()`
+        # (this is how Python 2 worked). To get that, we encode with the mbcs
+        # encoding, which will pass CP_ACP to the underlying Windows API to
+        # produce bytes.
+        if os.name == r'nt':
+            sysargv = [a.encode("mbcs", "ignore") for a in sys.argv]
+        else:
+            encoding = (
+                locale.getlocale()[1]
+                or locale.getdefaultlocale()[1]
+                or sys.getfilesystemencoding()
+            )
+            sysargv = [a.encode(encoding, "surrogateescape") for a in sys.argv]
 
     bytechr = struct.Struct('>B').pack
     byterepr = b'%r'.__mod__



To: indygreg, #hg-reviewers
Cc: mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D8337: pycompat: change argv conversion semantics

valentin.gatienbaron (Valentin Gatien-Baron)
indygreg added a comment.
indygreg added a subscriber: yuja.


  @yuja I'd appreciate your eyes on this since you have a firm grasp on Windows/Unicode matters...

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8337/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8337

To: indygreg, #hg-reviewers
Cc: yuja, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

Re: D8337: pycompat: change argv conversion semantics

Yuya Nishihara
In reply to this post by valentin.gatienbaron (Valentin Gatien-Baron)
> -    # Since Python 3 converts argv to wchar_t type by Py_DecodeLocale() on Unix,
> -    # we can use os.fsencode() to get back bytes argv.
> -    #
> -    # https://hg.python.org/cpython/file/v3.5.1/Programs/python.c#l55
> -    #
> -    # On Windows, the native argv is unicode and is converted to MBCS bytes
> -    # since we do enable the legacy filesystem encoding.
>      if getattr(sys, 'argv', None) is not None:
> -        sysargv = list(map(os.fsencode, sys.argv))
> +        # On POSIX, the char** argv array is converted to Python str using
> +        # Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which isn't
> +        # directly callable from Python code. So, we need to emulate it.
> +        # Py_DecodeLocale() calls mbstowcs() and falls back to mbrtowc() with
> +        # surrogateescape error handling on failure. These functions take the
> +        # current system locale into account. So, the inverse operation is to
> +        # .encode() using the system locale's encoding and using the
> +        # surrogateescape error handler. The only tricky part here is getting
> +        # the system encoding correct, since `locale.getlocale()` can return
> +        # None. We fall back to the filesystem encoding if lookups via `locale`
> +        # fail, as this seems like a reasonable thing to do.
> +        #
> +        # On Windows, the wchar_t **argv is passed into the interpreter as-is.
> +        # Like POSIX, we need to emulate what Py_EncodeLocale() would do. But
> +        # there's an additional wrinkle. What we really want to access is the
> +        # ANSI codepage representation of the arguments, as this is what
> +        # `int main()` would receive if Python 3 didn't define `int wmain()`
> +        # (this is how Python 2 worked). To get that, we encode with the mbcs
> +        # encoding, which will pass CP_ACP to the underlying Windows API to
> +        # produce bytes.
> +        if os.name == r'nt':
> +            sysargv = [a.encode("mbcs", "ignore") for a in sys.argv]

On Windows, my assumption was os.fsencode() == .encode("mbcs") if
sys._enablelegacywindowsfsencoding(). So this looks good to me.
Perhaps, the "ignore" error mode would match the legacy Windows behavior.

> +        else:
> +            encoding = (
> +                locale.getlocale()[1]
> +                or locale.getdefaultlocale()[1]
> +                or sys.getfilesystemencoding()
> +            )
> +            sysargv = [a.encode(encoding, "surrogateescape") for a in sys.argv]

I'm not pretty sure if the locale encoding is the encoding Py_DecodeLocale()
would use. There are many ifdefs for `__APPLE__`. The doc says use
`os.fsencode()`, but that's no longer valid (or wrong from the start)?

https://docs.python.org/3/library/sys.html#sys.argv

Something might be changed around 3.7 or 3.8. Since bytes argv handling
has been moved from `int main()` to `preconfig.c`, things could become
more dynamic. But I don't know. Just my guess.

Overall, the new code looks good, but I have no idea if that's more correct.
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D8337: pycompat: change argv conversion semantics

valentin.gatienbaron (Valentin Gatien-Baron)
yuja added a comment.


  > - # Since Python 3 converts argv to wchar_t type by Py_DecodeLocale() on Unix,
  > - # we can use os.fsencode() to get back bytes argv.
  > - #
  > - # https://hg.python.org/cpython/file/v3.5.1/Programs/python.c#l55
  > - #
  > - # On Windows, the native argv is unicode and is converted to MBCS bytes
  > - # since we do enable the legacy filesystem encoding. if getattr(sys, 'argv', None) is not None:
  > - sysargv = list(map(os.fsencode, sys.argv))
  >
  > +        # On POSIX, the char** argv array is converted to Python str using
  > +        # Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which isn't
  > +        # directly callable from Python code. So, we need to emulate it.
  > +        # Py_DecodeLocale() calls mbstowcs() and falls back to mbrtowc() with
  > +        # surrogateescape error handling on failure. These functions take the
  > +        # current system locale into account. So, the inverse operation is to
  > +        # .encode() using the system locale's encoding and using the
  > +        # surrogateescape error handler. The only tricky part here is getting
  > +        # the system encoding correct, since `locale.getlocale()` can return
  > +        # None. We fall back to the filesystem encoding if lookups via `locale`
  > +        # fail, as this seems like a reasonable thing to do.
  > +        #
  > +        # On Windows, the wchar_t **argv is passed into the interpreter as-is.
  > +        # Like POSIX, we need to emulate what Py_EncodeLocale() would do. But
  > +        # there's an additional wrinkle. What we really want to access is the
  > +        # ANSI codepage representation of the arguments, as this is what
  > +        # `int main()` would receive if Python 3 didn't define `int wmain()`
  > +        # (this is how Python 2 worked). To get that, we encode with the mbcs
  > +        # encoding, which will pass CP_ACP to the underlying Windows API to
  > +        # produce bytes.
  > +        if os.name == r'nt':
  > +            sysargv = [a.encode("mbcs", "ignore") for a in sys.argv]
 
  On Windows, my assumption was os.fsencode() == .encode("mbcs") if
  sys._enablelegacywindowsfsencoding(). So this looks good to me.
  Perhaps, the "ignore" error mode would match the legacy Windows behavior.
 
  > +        else:
  > +            encoding = (
  > +                locale.getlocale()[1]
  > +                or locale.getdefaultlocale()[1]
  > +                or sys.getfilesystemencoding()
  > +            )
  > +            sysargv = [a.encode(encoding, "surrogateescape") for a in sys.argv]
 
  I'm not pretty sure if the locale encoding is the encoding Py_DecodeLocale()
  would use. There are many ifdefs for `__APPLE__`. The doc says use
  `os.fsencode()`, but that's no longer valid (or wrong from the start)?
 
  https://docs.python.org/3/library/sys.html#sys.argv
 
  Something might be changed around 3.7 or 3.8. Since bytes argv handling
  has been moved from `int main()` to `preconfig.c`, things could become
  more dynamic. But I don't know. Just my guess.
 
  Overall, the new code looks good, but I have no idea if that's more correct.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8337/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8337

To: indygreg, #hg-reviewers
Cc: yuja, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D8337: pycompat: change argv conversion semantics

valentin.gatienbaron (Valentin Gatien-Baron)
In reply to this post by valentin.gatienbaron (Valentin Gatien-Baron)
mharbison72 added a comment.


  This looks like it also fixes the phabricator test on Windows, which diverged by creating a different hash for the `create alpha for phabricator test €` commit.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8337/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8337

To: indygreg, #hg-reviewers
Cc: mharbison72, yuja, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D8337: pycompat: change argv conversion semantics

valentin.gatienbaron (Valentin Gatien-Baron)
In reply to this post by valentin.gatienbaron (Valentin Gatien-Baron)
mharbison72 added a comment.


  In D8337#124542 <https://phab.mercurial-scm.org/D8337#124542>, @yuja wrote:
 
  > On Windows, my assumption was os.fsencode() == .encode("mbcs") if
  > sys._enablelegacywindowsfsencoding(). So this looks good to me.
  > Perhaps, the "ignore" error mode would match the legacy Windows behavior.
 
  Does this mean it could be a problem running from source on Windows?  For example, `hg version` (as opposed to `hg.exe version`) seems to be equivalent to `python hg`, which obviously doesn't have the proper environment variable or C API option to enable legacy mode.  Should there be code early on that detects this and warns/aborts?

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8337/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8337

To: indygreg, #hg-reviewers
Cc: mharbison72, yuja, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

Re: D8337: pycompat: change argv conversion semantics

Yuya Nishihara
>   > On Windows, my assumption was os.fsencode() == .encode("mbcs") if
>   > sys._enablelegacywindowsfsencoding(). So this looks good to me.
>   > Perhaps, the "ignore" error mode would match the legacy Windows behavior.
>  
>   Does this mean it could be a problem running from source on Windows?  For example, `hg version` (as opposed to `hg.exe version`) seems to be equivalent to `python hg`, which obviously doesn't have the proper environment variable or C API option to enable legacy mode.  Should there be code early on that detects this and warns/aborts?

It's hacked up by pycompat.py so `python hg` should be fine.
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D8337: pycompat: change argv conversion semantics

valentin.gatienbaron (Valentin Gatien-Baron)
yuja added a comment.


  >   > On Windows, my assumption was os.fsencode() == .encode("mbcs") if
  >   > sys._enablelegacywindowsfsencoding(). So this looks good to me.
  >   > Perhaps, the "ignore" error mode would match the legacy Windows behavior.
  >   Does this mean it could be a problem running from source on Windows?  For example, `hg version` (as opposed to `hg.exe version`) seems to be equivalent to `python hg`, which obviously doesn't have the proper environment variable or C API option to enable legacy mode.  Should there be code early on that detects this and warns/aborts?
 
  It's hacked up by pycompat.py so `python hg` should be fine.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8337/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8337

To: indygreg, #hg-reviewers
Cc: mharbison72, yuja, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D8337: pycompat: change argv conversion semantics

valentin.gatienbaron (Valentin Gatien-Baron)
In reply to this post by valentin.gatienbaron (Valentin Gatien-Baron)
Closed by commit rHG00e0c5c06ed5: pycompat: change argv conversion semantics (authored by indygreg).
This revision was automatically updated to reflect the committed changes.
This revision was not accepted when it landed; it landed in state "Needs Review".

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D8337?vs=20903&id=20948

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8337/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8337

AFFECTED FILES
  mercurial/pycompat.py

CHANGE DETAILS

diff --git a/mercurial/pycompat.py b/mercurial/pycompat.py
--- a/mercurial/pycompat.py
+++ b/mercurial/pycompat.py
@@ -98,6 +98,7 @@
     import codecs
     import functools
     import io
+    import locale
     import struct
 
     if os.name == r'nt' and sys.version_info >= (3, 6):
@@ -148,15 +149,36 @@
     stdout = sys.stdout.buffer
     stderr = sys.stderr.buffer
 
-    # Since Python 3 converts argv to wchar_t type by Py_DecodeLocale() on Unix,
-    # we can use os.fsencode() to get back bytes argv.
-    #
-    # https://hg.python.org/cpython/file/v3.5.1/Programs/python.c#l55
-    #
-    # On Windows, the native argv is unicode and is converted to MBCS bytes
-    # since we do enable the legacy filesystem encoding.
     if getattr(sys, 'argv', None) is not None:
-        sysargv = list(map(os.fsencode, sys.argv))
+        # On POSIX, the char** argv array is converted to Python str using
+        # Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which isn't
+        # directly callable from Python code. So, we need to emulate it.
+        # Py_DecodeLocale() calls mbstowcs() and falls back to mbrtowc() with
+        # surrogateescape error handling on failure. These functions take the
+        # current system locale into account. So, the inverse operation is to
+        # .encode() using the system locale's encoding and using the
+        # surrogateescape error handler. The only tricky part here is getting
+        # the system encoding correct, since `locale.getlocale()` can return
+        # None. We fall back to the filesystem encoding if lookups via `locale`
+        # fail, as this seems like a reasonable thing to do.
+        #
+        # On Windows, the wchar_t **argv is passed into the interpreter as-is.
+        # Like POSIX, we need to emulate what Py_EncodeLocale() would do. But
+        # there's an additional wrinkle. What we really want to access is the
+        # ANSI codepage representation of the arguments, as this is what
+        # `int main()` would receive if Python 3 didn't define `int wmain()`
+        # (this is how Python 2 worked). To get that, we encode with the mbcs
+        # encoding, which will pass CP_ACP to the underlying Windows API to
+        # produce bytes.
+        if os.name == r'nt':
+            sysargv = [a.encode("mbcs", "ignore") for a in sys.argv]
+        else:
+            encoding = (
+                locale.getlocale()[1]
+                or locale.getdefaultlocale()[1]
+                or sys.getfilesystemencoding()
+            )
+            sysargv = [a.encode(encoding, "surrogateescape") for a in sys.argv]
 
     bytechr = struct.Struct('>B').pack
     byterepr = b'%r'.__mod__



To: indygreg, #hg-reviewers
Cc: mharbison72, yuja, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel