D6754: contrib: proof of concept script to build Mac packages without system python

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

D6754: contrib: proof of concept script to build Mac packages without system python

indygreg (Gregory Szorc)
mharbison72 created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  This isn't close to being ready- I'm just looking for ideas before I get too far
  off into the weeds.  It will build a *.pkg that does install, and both
  `hg version` and `hg debuginstall` look healthy.  As noted in the script though,
  there are still references to /Library/Frameworks/Python.framework (which is the
  python.org installed stuff), so it may not work on a fresh system.
 
  At this point, I'm wondering if we should just curl the source for OpenSSL and
  Python, and build the non-framework version with the destination set to the
  proper directory.  The packaging for thg does this (though it probably doesn't
  have to), and it takes about 10 minutes on a Mac Mini.  5 Minutes of that is
  running the tests on the python build, so maybe there's a way to shut that off.
 
  As far as alternatives, I started down the path of using virtualenv here, and
  shipping that.  I'm skeptical of the experimental nature of --relocatable, and
  grepping on the tree showed the build path in the *.pyc files.  I have no idea
  if that matters.  I left pip in place so that things like evolve could be
  installed, but it errored out when specifying `--user`.  I didn't look into why.
 
  I also looked at pyinstaller.  I couldn't figure out how to convince it to
  bundle the C extensions, and it appears to append its own output when the hg
  script exits with a non zero value.  I briefly looked at bbfreeze and cx_freeze,
  but didn't get as far.  I've had enough frustration with py2exe and py2app that
  I'm not wanting to struggle with yet another packager.
 
  Ultimately I think we should keep the target for building against system python
  (there was some special functionality unlocked with a specifically crafted dummy
  cert), and this should possibly be a different target.  (I don't see a way to
  switch between the two by simply overriding $PYTHON the way this is going.)  I
  basically copy/pasted the makefile as a starting point, but moved it into a
  shell script so the environment could be activated.  This either needs to be
  moved to contrib/, or folded back into the makefile.  I kinda lean towards the
  former, and have that called by make.  The readme in the installer will also
  have to be conditionalize/updated, since it mentions relying on system python.
 
  It would be nice to open up the range of supported platforms back to say, 10.9.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6754

AFFECTED FILES
  make_osx.sh

CHANGE DETAILS

diff --git a/make_osx.sh b/make_osx.sh
new file mode 100755
--- /dev/null
+++ b/make_osx.sh
@@ -0,0 +1,87 @@
+#!/bin/sh
+
+set -e
+
+# Extra args passed in, not version
+OSXVERSIONFLAGS=
+
+# TODO: default MACOSX_DEPLOYMENT_TARGET=10.9
+# TODO: adjust ${OSXVER} accordingly
+
+rm -rf build/mercurial
+python -m virtualenv build/mercurial/usr/local/mercurial
+source build/mercurial/usr/local/mercurial/bin/activate
+
+# TODO: Don't install this into the virtual environment. Also, use docutils-python3
+python -m pip install docutils
+
+python setup.py install --optimize=1  \
+  --root=build/mercurial/ --prefix=/usr/local/mercurial
+
+make -C doc all install DESTDIR="`pwd`/build/mercurial/"
+
+# Place a bogon .DS_Store file in the target dir so we can be
+# sure it doesn't get included in the final package.
+touch build/mercurial/.DS_Store
+
+# install zsh completions - this location appears to be
+# searched by default as of macOS Sierra.
+install -d build/mercurial/usr/local/share/zsh/site-functions/
+install -m 0644 contrib/zsh_completion build/mercurial/usr/local/share/zsh/site-functions/_hg
+
+# install bash completions - there doesn't appear to be a
+# place that's searched by default for bash, so we'll follow
+# the lead of Apple's git install and just put it in a
+# location of our own.
+install -d build/mercurial/usr/local/hg/contrib/
+install -m 0644 contrib/bash_completion build/mercurial/usr/local/hg/contrib/hg-completion.bash
+
+make -C contrib/chg \
+  HGPATH=/usr/local/bin/hg \
+  PYTHON=python \
+  HGEXTDIR=/usr/local/mercurial/lib/python2.7/site-packages/hgext \
+  DESTDIR=../../build/mercurial \
+  PREFIX=/usr/local \
+  clean install
+
+# Keep pip around so that things like evolve can be installed.
+rm -f build/mercurial/usr/local/mercurial/bin/rst* \
+      build/mercurial/usr/local/mercurial/bin/activate* \
+      build/mercurial/usr/local/mercurial/bin/wheel \
+      build/mercurial/usr/local/mercurial/bin/easy_install
+
+# LANG=C' to avoid "sed: RE error: illegal byte sequence"
+LANG=C sed -i '' -e "s|`pwd`/build/mercurial||g" build/mercurial/usr/local/mercurial/bin/*
+
+# The virtualenv contains a symlink back to the real python install.  We can
+# either slim this down somehow (to avoid distributing whatever random packages
+# are already installed), or build python from source with a custom --dest.
+mkdir -p build/mercurial/usr/local/mercurial/Library/Frameworks/Python.framework/Versions
+cp -R /Library/Frameworks/Python.framework/Versions/2.7 build/mercurial/usr/local/mercurial/Library/Frameworks/Python.framework/Versions
+ln -s -f /usr/local/mercurial/Python.framework/Versions/2.7/Python build/mercurial/usr/local/mercurial/.Python
+
+# Running /usr/local/mercurial/Python.framework/Versions/2.7/bin/python2.7 and
+# printing sys.paths lists out stuff in /Library/Frameworks/Python.framework,
+# so this may not be totally detached from a standard python.org installation.
+# sys.exec_prefix and sys.prefix also point there.  That stuff appears to live
+# in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.py.
+# The various *.pyc files that are copied over also have /Library/Frameworks strings
+# in them.  Then again, so do *.pyc files in a virtualenv.
+ln -s /usr/local/mercurial/bin/hg build/mercurial/usr/local/bin/hg
+
+mkdir -p ${OUTPUTDIR:-dist}
+
+HGVER=$(python2 contrib/genosxversion.py ${OSXVERSIONFLAGS} \
+       build/mercurial/usr/local/mercurial/lib/python2.7/site-packages/mercurial/__version__.py) && \
+OSXVER=$(sw_vers -productVersion | cut -d. -f1,2) && \
+PYVER=$(python -V 2>&1 | cut -d' ' -f2 | cut -d. -f1,2) && \
+pkgbuild --filter .DS_Store --root build/mercurial/ \
+  --identifier org.mercurial-scm.mercurial \
+  --version "${HGVER}" \
+  build/mercurial.pkg && \
+
+productbuild --distribution contrib/packaging/macosx/distribution.xml \
+  --package-path build/ \
+  --version "${HGVER}" \
+  --resources contrib/packaging/macosx/ \
+  "${OUTPUTDIR:-dist/}"/Mercurial-"${HGVER}"-py"${PYVER}"-macosx"${OSXVER}".pkg



To: mharbison72, #hg-reviewers
Cc: mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D6754: contrib: proof of concept script to build Mac packages without system python

indygreg (Gregory Szorc)
indygreg added a comment.


  This looks promising.
 
  I don't want to tell you not to work on this, but my bold plan is to get Mercurial using PyOxidizer and leaning on PyOxidizer for packaging. This will require shipping a Python 3 Mercurial, however.
 
  Until PyOxidizer is ready, you may want to look into Beeware's Briefcase tool for packaging macOS applications. It also provides its own self-contained Python distribution. I'm not sure if it works with Python 2.7, however.
 
  You may also be interested in https://github.com/indygreg/python-build-standalone for self-contained Python distributions. Only works with Python 3.7+ at the moment now, though. The pre-built distributions on GitHub having a working Python install in them. All the Python C extensions are statically linked into the `python` executable.

INLINE COMMENTS

> make_osx.sh:16
> +# TODO: Don't install this into the virtual environment. Also, use docutils-python3
> +python -m pip install docutils
> +

We should be pinning versions and hashes when installing anything from the network. For reproducibility and security.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D6754/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D6754

To: mharbison72, #hg-reviewers
Cc: indygreg, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D6754: contrib: proof of concept script to build Mac packages without system python

indygreg (Gregory Szorc)
In reply to this post by indygreg (Gregory Szorc)
mharbison72 added a comment.
mharbison72 added a subscriber: durin42.


  In D6754#99255 <https://phab.mercurial-scm.org/D6754#99255>, @indygreg wrote:
 
  > This looks promising.
  > I don't want to tell you not to work on this, but my bold plan is to get Mercurial using PyOxidizer and leaning on PyOxidizer for packaging. This will require shipping a Python 3 Mercurial, however.
 
  Thanks for the pointers.
 
  I'm broadly aware of the plans to use PyOxidizer for py3, and hope to use it for thg too.  I started this with nothing more in mind than making the installer support more than 10.14.  But since it links against the system python, setting the MACOS_DEPLOYMENT_TARGET (or whatever it's called) errors out.  So I tried allowing PYTHON to be overridden in the makefile, so it could be pointed to the python.org install and go back to 10.9.  But @durin42 locked it into the system python in da1848f07c6a <https://phab.mercurial-scm.org/rHGda1848f07c6ac14a748e5dea826342922f686480>, and things just kinda snowballed from there.  He mentioned on IRC that a pyinstaller dependency wouldn't be great, and also maybe we don't want to be completely tied to PyOxidizer.  (I took it as don't preclude other methods, but maybe that needs to be clarified.)
 
  > Until PyOxidizer is ready, you may want to look into Beeware's Briefcase tool for packaging macOS applications. It also provides its own self-contained Python distribution. I'm not sure if it works with Python 2.7, however.
 
  There at least appears to be a package visible in pip27, but the docs talk about building a *.app, which isn't going to be useful for us.
 
  > You may also be interested in https://github.com/indygreg/python-build-standalone for self-contained Python distributions. Only works with Python 3.7+ at the moment now, though. The pre-built distributions on GitHub having a working Python install in them. All the Python C extensions are statically linked into the `python` executable.
 
  I'll try to find time to play with this in the next week or so.  I'm not sure what it would take to support py2, but I'd rather everybody poured their effort in elsewhere at this point.  @durin42 mentioned on IRC that it's long past time to get off the system python, but the only real deadline is that it won't be in 10.16 a year from now.  Hopefully the codebase is still supporting py2 then, but maybe the py3 support is good enough that we don't need an official py2 installer?  End of py2 support doesn't seem like a big deal, unless some horrible TLS bug is found.
 
  I suppose a reasonable fix for supporting more OS versions is to spin up a 10.9 VM and build there.  I don't have one right now, but I do have a machine that is going to be setup to host various Mac VMs, and I don't have a problem building the installer on it if there's interest.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D6754/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D6754

To: mharbison72, #hg-reviewers
Cc: durin42, indygreg, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D6754: contrib: proof of concept script to build Mac packages without system python

indygreg (Gregory Szorc)
In reply to this post by indygreg (Gregory Szorc)
durin42 added a comment.


  This is a start, but I suspect where we should end up with this is downloading and compiling the whole Python distribution, rather than relying on a virtualenv to be portable?

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D6754/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D6754

To: mharbison72, #hg-reviewers
Cc: durin42, indygreg, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D6754: contrib: proof of concept script to build Mac packages without system python

indygreg (Gregory Szorc)
In reply to this post by indygreg (Gregory Szorc)
mharbison72 added a comment.


  In D6754#100159 <https://phab.mercurial-scm.org/D6754#100159>, @durin42 wrote:
 
  > This is a start, but I suspect where we should end up with this is downloading and compiling the whole Python distribution, rather than relying on a virtualenv to be portable?
 
  Agreed.  I think we will need to build OpenSSL too, and bundle certifi?
 
  The script for building thg makes it look easy, but the result I ended up with failed to support zlib and sqlite3.  (We need the latter too, right?)  Maybe it's an issue with non-framework builds, as that's the only difference I'm aware of from what thg build is doing.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D6754/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D6754

To: mharbison72, #hg-reviewers
Cc: durin42, indygreg, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel