[PATCH rebased] pycompat: use os.fsencode() to re-encode sys.argv
# HG changeset patch
# User Manuel Jacob <[hidden email]>
# Date 1593002661 -7200
# Wed Jun 24 14:44:21 2020 +0200
# Node ID 6f504f329b13265d5f67b03023c96978539d5df7
# Parent b61dc3c1a7f6c0c7c54bb0b6b6de53b6011cfebd
# EXP-Topic use_os_fsencode
pycompat: use os.fsencode() to re-encode sys.argv
Historically, the previous code made sense, as Py_EncodeLocale() and
fs.fsencode() could possibly use different encodings. However, this is not the
case anymore for Python 3.2, which uses the locale encoding as the filesystem
encoding (this is not true for later Python versions, but see below). See
a source and more background information.
Using os.fsencode() is safer, as the documentation for sys.argv says that it can
be used to get the original bytes. When doing further changes, the Python
developers will take care that this continues to work.
One concrete case where os.fsencode() is more correct is when enabling Python's
UTF-8 mode. Py_DecodeLocale() will use UTF-8 in this case. Our previous code
would have encoded it using the locale encoding (which might be different),
whereas os.fsencode() will encode it with UTF-8.
Since we don’t claim to support the UTF-8 mode, this is not really a bug and the
patch can go to the default branch. It might be a good idea to not commit this
to the stable branch, as it could in theory introduce regressions.
if getattr(sys, 'argv', None) is not None:
# On POSIX, the char** argv array is converted to Python str using
- # Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which isn't
- # directly callable from Python code. So, we need to emulate it.
- # Py_DecodeLocale() calls mbstowcs() and falls back to mbrtowc() with
- # surrogateescape error handling on failure. These functions take the
- # current system locale into account. So, the inverse operation is to
- # .encode() using the system locale's encoding and using the
- # surrogateescape error handler. The only tricky part here is getting
- # the system encoding correct, since `locale.getlocale()` can return
- # None. We fall back to the filesystem encoding if lookups via `locale`
- # fail, as this seems like a reasonable thing to do.
+ # Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which
+ # isn't directly callable from Python code. In practice, os.fsencode()
+ # can be used instead (this is recommended by Python's documentation
+ # for sys.argv).
# On Windows, the wchar_t **argv is passed into the interpreter as-is.
# Like POSIX, we need to emulate what Py_EncodeLocale() would do. But
@@ -178,19 +172,7 @@
if os.name == r'nt':
sysargv = [a.encode("mbcs", "ignore") for a in sys.argv]
- def getdefaultlocale_if_known():
- return locale.getdefaultlocale()
- except ValueError:
- return None, None
- encoding = (
- or getdefaultlocale_if_known()
- or sys.getfilesystemencoding()
- sysargv = [a.encode(encoding, "surrogateescape") for a in sys.argv]
+ sysargv = [fsencode(a) for a in sys.argv]
Re: [PATCH rebased] pycompat: use os.fsencode() to re-encode sys.argv
On Thu, 25 Jun 2020 05:43:26 +0200, Manuel Jacob wrote:
> # HG changeset patch
> # User Manuel Jacob <[hidden email]>
> # Date 1593002661 -7200
> # Wed Jun 24 14:44:21 2020 +0200
> # Node ID 6f504f329b13265d5f67b03023c96978539d5df7
> # Parent b61dc3c1a7f6c0c7c54bb0b6b6de53b6011cfebd
> # EXP-Topic use_os_fsencode
> pycompat: use os.fsencode() to re-encode sys.argv