Python 2 removal and thg packaging status

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Python 2 removal and thg packaging status

Raphaël Gomès
Hello all,

As you all know, Mercurial's codebase is still burdened by Python 2
support. Patches still
need to be adapted for backwards compat, some Python niceties still
cannot be used,
and the Heptapod CI which is used on a near majority of the patches we
collectively land
is still putting in double shifts as well.

At the last (virtual) sprint, everybody present agreed that the time had
long passed to drop
Python 2 support, but that it couldn't be done until TortoiseHg's Python
3 Windows packaging
story was straightened out. Where are we standing on the matter? Are we
close?

I am CC'ing Greg who proposed to help at the time, as well as Matt, the
maintainer, since they
might know better than anyone where we're currently standing.

Thanks,
Raphaël

_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 2 removal and thg packaging status

Raphaël Gomès
+ CC actually Greg, oops

On 4/3/21 6:18 PM, Raphaël Gomès wrote:

> Hello all,
>
> As you all know, Mercurial's codebase is still burdened by Python 2
> support. Patches still
> need to be adapted for backwards compat, some Python niceties still
> cannot be used,
> and the Heptapod CI which is used on a near majority of the patches we
> collectively land
> is still putting in double shifts as well.
>
> At the last (virtual) sprint, everybody present agreed that the time
> had long passed to drop
> Python 2 support, but that it couldn't be done until TortoiseHg's
> Python 3 Windows packaging
> story was straightened out. Where are we standing on the matter? Are
> we close?
>
> I am CC'ing Greg who proposed to help at the time, as well as Matt,
> the maintainer, since they
> might know better than anyone where we're currently standing.
>
> Thanks,
> Raphaël
>
> _______________________________________________
> Mercurial-devel mailing list
> [hidden email]
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 2 removal and thg packaging status

Matt Harbison-2
In reply to this post by Raphaël Gomès
On Sat, Apr 3, 2021 at 12:18 PM Raphaël Gomès <[hidden email]> wrote:

>
> Hello all,
>
> As you all know, Mercurial's codebase is still burdened by Python 2
> support. Patches still
> need to be adapted for backwards compat, some Python niceties still
> cannot be used,
> and the Heptapod CI which is used on a near majority of the patches we
> collectively land
> is still putting in double shifts as well.
>
> At the last (virtual) sprint, everybody present agreed that the time had
> long passed to drop
> Python 2 support, but that it couldn't be done until TortoiseHg's Python
> 3 Windows packaging
> story was straightened out. Where are we standing on the matter? Are we
> close?

I've been making progress on adding type hints and fixing issues in
the TortoiseHg code.  The good news is there are ~13 files left that
are excluded.  The bad news is I know for a fact that some of the
*included* files have type mismatches that pytype isn't detecting.
Some of this could be helped by moving to py3.  (e.g. pytype seems to
not recognize `foo[b'bar']` as calling `__getitem__()` on an object,
and treats it as returning Any, even when there's type info on the
function.  Subclassing typing.Mapping seems to help in some cases, but
classes like `config.config` and `localrepo.localrepository` really
can't subclass that because they have functions like `items()` with a
signature that differs from the base class.  It also seems to struggle
with @annotations.)  IDK for sure how far I am through adding type
hints, because it's so non linear by nature.  If I had to guess, maybe
50%?

Stepping back from the thg work, there's also core work that still
needs to be done.  I just ran the tests locally and got 19 failures
(excluding fixes I'm about the send), and skipping 114 which are
mostly never going to work on Windows because of symlink, executable
bit, casefolding, etc.  Some of those skips are test-convert-* other
than git, so I assume that's a mess.  I don't care about that, but
somebody might.  Of the failures, 4 are test-gendoc-*, so IDK how
important those are.  Some of the other failures are... concerning.
But they may have simple fixes.  There are runtime issues that are not
detected by tests, such as
https://bz.mercurial-scm.org/show_bug.cgi?id=6446 and getting no
output for commands that spin up a pager.  (I *think* the latter is
limited to running `hg` instead of `hg.exe` in a venv, so maybe this
is fixable by compiling wrapper.exe and not installing the `hg` script
when running `setup.py` on Windows.)  I also need to finish up the py3
porting series for mercurial_keyring, as that still has a few issues.

I think that it would be beneficial to have a beta period for thg
where both py2 and py3 are built (like we did with hg), to find some
of the issues that pytype isn't catching.  I'll probably regret this
because IDK how long I can sustain the current pace, but if we can get
the Windows and Mac packaging done for 5.8, maybe 5.8 can be the beta
period and 5.9 is py3 only?  (It might be OK if the packaging isn't
ready until 5.8.1, since Yuya seems to be OK with landing packaging
and py3 fixes on stable if they aren't crazy.)  I think Greg made some
MSI related enhancement to PyOxidizer recently, but IDK what the means
for the state of things, or how far away a macOS *.app bundle is.
Worst case scenario, I can build thg 5.9 with hg 5.8.2 if thg progress
can't keep pace.  I just don't want that going on forever.

> I am CC'ing Greg who proposed to help at the time, as well as Matt, the
> maintainer, since they
> might know better than anyone where we're currently standing.
>
> Thanks,
> Raphaël
>
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 2 removal and thg packaging status

Augie Fackler-2


On Apr 3, 2021, at 6:34 PM, Matt Harbison <[hidden email]> wrote:

On Sat, Apr 3, 2021 at 12:18 PM Raphaël Gomès <[hidden email]> wrote:

Hello all,

As you all know, Mercurial's codebase is still burdened by Python 2
support. Patches still
need to be adapted for backwards compat, some Python niceties still
cannot be used,
and the Heptapod CI which is used on a near majority of the patches we
collectively land
is still putting in double shifts as well.

At the last (virtual) sprint, everybody present agreed that the time had
long passed to drop
Python 2 support, but that it couldn't be done until TortoiseHg's Python
3 Windows packaging
story was straightened out. Where are we standing on the matter? Are we
close?

I've been making progress on adding type hints and fixing issues in
the TortoiseHg code.  The good news is there are ~13 files left that
are excluded.  The bad news is I know for a fact that some of the
*included* files have type mismatches that pytype isn't detecting.
Some of this could be helped by moving to py3.  (e.g. pytype seems to
not recognize `foo[b'bar']` as calling `__getitem__()` on an object,
and treats it as returning Any, even when there's type info on the
function.  Subclassing typing.Mapping seems to help in some cases, but
classes like `config.config` and `localrepo.localrepository` really
can't subclass that because they have functions like `items()` with a
signature that differs from the base class.  It also seems to struggle
with @annotations.)  IDK for sure how far I am through adding type
hints, because it's so non linear by nature.  If I had to guess, maybe
50%?

Stepping back from the thg work, there's also core work that still
needs to be done.  I just ran the tests locally and got 19 failures
(excluding fixes I'm about the send), and skipping 114 which are
mostly never going to work on Windows because of symlink, executable
bit, casefolding, etc.  Some of those skips are test-convert-* other
than git, so I assume that's a mess.  I don't care about that, but
somebody might.  Of the failures, 4 are test-gendoc-*, so IDK how
important those are.  Some of the other failures are... concerning.
But they may have simple fixes.  There are runtime issues that are not
detected by tests, such as
https://bz.mercurial-scm.org/show_bug.cgi?id=6446 and getting no
output for commands that spin up a pager.  (I *think* the latter is
limited to running `hg` instead of `hg.exe` in a venv, so maybe this
is fixable by compiling wrapper.exe and not installing the `hg` script
when running `setup.py` on Windows.)  I also need to finish up the py3
porting series for mercurial_keyring, as that still has a few issues.

I think that it would be beneficial to have a beta period for thg
where both py2 and py3 are built (like we did with hg), to find some
of the issues that pytype isn't catching.  I'll probably regret this
because IDK how long I can sustain the current pace, but if we can get
the Windows and Mac packaging done for 5.8, maybe 5.8 can be the beta
period and 5.9 is py3 only?  (It might be OK if the packaging isn't
ready until 5.8.1, since Yuya seems to be OK with landing packaging
and py3 fixes on stable if they aren't crazy.)  I think Greg made some
MSI related enhancement to PyOxidizer recently, but IDK what the means
for the state of things, or how far away a macOS *.app bundle is.
Worst case scenario, I can build thg 5.9 with hg 5.8.2 if thg progress
can't keep pace.  I just don't want that going on forever.

I think this plan is sensible and you should proceed with it.


I am CC'ing Greg who proposed to help at the time, as well as Matt, the
maintainer, since they
might know better than anyone where we're currently standing.

Thanks,
Raphaël

_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel


_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 2 removal and thg packaging status

Gregory Szorc
My plan of attack was to implement support for packaging in PyOxidizer and have thg (and Mercurial) use this functionality (with or without the traditional PyOxidizer-building-the-binary approach). However, I've been sidetracked by various things. Ugh, COVID.

I'll try to take a look at what features I still need to implement in PyOxidizer to proceed with this plan in the next week or so.

On Mon, Apr 12, 2021 at 11:52 AM Augie Fackler <[hidden email]> wrote:


On Apr 3, 2021, at 6:34 PM, Matt Harbison <[hidden email]> wrote:

On Sat, Apr 3, 2021 at 12:18 PM Raphaël Gomès <[hidden email]> wrote:

Hello all,

As you all know, Mercurial's codebase is still burdened by Python 2
support. Patches still
need to be adapted for backwards compat, some Python niceties still
cannot be used,
and the Heptapod CI which is used on a near majority of the patches we
collectively land
is still putting in double shifts as well.

At the last (virtual) sprint, everybody present agreed that the time had
long passed to drop
Python 2 support, but that it couldn't be done until TortoiseHg's Python
3 Windows packaging
story was straightened out. Where are we standing on the matter? Are we
close?

I've been making progress on adding type hints and fixing issues in
the TortoiseHg code.  The good news is there are ~13 files left that
are excluded.  The bad news is I know for a fact that some of the
*included* files have type mismatches that pytype isn't detecting.
Some of this could be helped by moving to py3.  (e.g. pytype seems to
not recognize `foo[b'bar']` as calling `__getitem__()` on an object,
and treats it as returning Any, even when there's type info on the
function.  Subclassing typing.Mapping seems to help in some cases, but
classes like `config.config` and `localrepo.localrepository` really
can't subclass that because they have functions like `items()` with a
signature that differs from the base class.  It also seems to struggle
with @annotations.)  IDK for sure how far I am through adding type
hints, because it's so non linear by nature.  If I had to guess, maybe
50%?

Stepping back from the thg work, there's also core work that still
needs to be done.  I just ran the tests locally and got 19 failures
(excluding fixes I'm about the send), and skipping 114 which are
mostly never going to work on Windows because of symlink, executable
bit, casefolding, etc.  Some of those skips are test-convert-* other
than git, so I assume that's a mess.  I don't care about that, but
somebody might.  Of the failures, 4 are test-gendoc-*, so IDK how
important those are.  Some of the other failures are... concerning.
But they may have simple fixes.  There are runtime issues that are not
detected by tests, such as
https://bz.mercurial-scm.org/show_bug.cgi?id=6446 and getting no
output for commands that spin up a pager.  (I *think* the latter is
limited to running `hg` instead of `hg.exe` in a venv, so maybe this
is fixable by compiling wrapper.exe and not installing the `hg` script
when running `setup.py` on Windows.)  I also need to finish up the py3
porting series for mercurial_keyring, as that still has a few issues.

I think that it would be beneficial to have a beta period for thg
where both py2 and py3 are built (like we did with hg), to find some
of the issues that pytype isn't catching.  I'll probably regret this
because IDK how long I can sustain the current pace, but if we can get
the Windows and Mac packaging done for 5.8, maybe 5.8 can be the beta
period and 5.9 is py3 only?  (It might be OK if the packaging isn't
ready until 5.8.1, since Yuya seems to be OK with landing packaging
and py3 fixes on stable if they aren't crazy.)  I think Greg made some
MSI related enhancement to PyOxidizer recently, but IDK what the means
for the state of things, or how far away a macOS *.app bundle is.
Worst case scenario, I can build thg 5.9 with hg 5.8.2 if thg progress
can't keep pace.  I just don't want that going on forever.

I think this plan is sensible and you should proceed with it.


I am CC'ing Greg who proposed to help at the time, as well as Matt, the
maintainer, since they
might know better than anyone where we're currently standing.

Thanks,
Raphaël

_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel


_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel