D8351: hgcli: customize for Mercurial

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

D8351: hgcli: customize for Mercurial

valentin.gatienbaron (Valentin Gatien-Baron)
indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Now that we have a shiny new PyOxidizer-based hgcli project, let's
  customize it for Mercurial!
 
  This commit replaces the auto-generated pyoxidizer.bzl with one
  that installs Mercurial from the local source repository.
 
  A README.md with build instructions has been added.
 
  The Cargo.toml file has been updated to reflect the proper license
  and reference the added README.md.
 
  In my Linux environment, running the test suite yields 27 failures.
 
  It's worth noting the run time of the test harness on Linux on my
  Ryzen 3950X:
 
  before: 378s wall; 9982s user; 1195s sys
  after:  353s wall; 8996s user;  958s sys
  % orig: 93.4 wall;  90.1  user; 80.2 sys
 
  While I haven't measured explicitly, I suspect the performance win is
  due to in-memory resource loading (which is known to be faster than
  Python's filesystem importer).

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D8351

AFFECTED FILES
  rust/hgcli/Cargo.toml
  rust/hgcli/README.md
  rust/hgcli/pyoxidizer.bzl

CHANGE DETAILS

diff --git a/rust/hgcli/pyoxidizer.bzl b/rust/hgcli/pyoxidizer.bzl
--- a/rust/hgcli/pyoxidizer.bzl
+++ b/rust/hgcli/pyoxidizer.bzl
@@ -1,147 +1,53 @@
-# This file defines how PyOxidizer application building and packaging is
-# performed. See the pyoxidizer crate's documentation for extensive
-# documentation on this file format.
+ROOT = CWD + "/../.."
 
-# Obtain the default PythonDistribution for our build target. We link
-# this distribution into our produced executable and extract the Python
-# standard library from it.
-def make_dist():
-    return default_python_distribution()
+def make_exe():
+    dist = default_python_distribution()
+
+    code = "import hgdemandimport; hgdemandimport.enable(); from mercurial import dispatch; dispatch.run()"
 
-# Configuration files consist of functions which define build "targets."
-# This function creates a Python executable and installs it in a destination
-# directory.
-def make_exe(dist):
-    # This variable defines the configuration of the
-    # embedded Python interpreter.
-    python_config = PythonInterpreterConfig(
-        #     bytes_warning=0,
-        #     dont_write_bytecode=True,
-        #     ignore_environment=True,
-        #     inspect=False,
-        #     interactive=False,
-        #     isolated=False,
-        #     legacy_windows_fs_encoding=False,
-        #     legacy_windows_stdio=False,
-        #     no_site=True,
-        #     no_user_site_directory=True,
-        #     optimize_level=0,
-        #     parser_debug=False,
-        #     stdio_encoding=None,
-        #     unbuffered_stdio=False,
-        #     filesystem_importer=False,
-        #     sys_frozen=False,
-        #     sys_meipass=False,
-        #     sys_paths=None,
-        #     raw_allocator=None,
-        #     terminfo_resolution="dynamic",
-        #     terminfo_dirs=None,
-        #     use_hash_seed=False,
-        #     verbose=0,
-        #     write_modules_directory_env=None,
-        #     run_eval=None,
-        #     run_module=None,
-        #     run_noop=False,
-        #     run_repl=True,
+    config = PythonInterpreterConfig(
+        raw_allocator = "system",
+        run_eval = code,
+        # We want to let the user load extensions from the file system
+        filesystem_importer = True,
+        # We need this to make resourceutil happy, since it looks for sys.frozen.
+        sys_frozen = True,
+        legacy_windows_stdio = True,
     )
 
-    # The run_eval, run_module, run_noop, and run_repl arguments are mutually
-    # exclusive controls over what the interpreter should do once it initializes.
-    #
-    # run_eval -- Run the specified string value via `eval()`.
-    # run_module -- Import the specified module as __main__ and run it.
-    # run_noop -- Do nothing.
-    # run_repl -- Start a Python REPL.
-    #
-    # These arguments can be ignored if you are providing your own Rust code for
-    # starting the interpreter, as Rust code has full control over interpreter
-    # behavior.
-
-    # Produce a PythonExecutable from a Python distribution, embedded
-    # resources, and other options. The returned object represents the
-    # standalone executable that will be built.
     exe = dist.to_python_executable(
-        name = "hgcli",
-        config = python_config,
-        # Embed all extension modules, making this a fully-featured Python.
+        name = "hg",
+        resources_policy = "prefer-in-memory-fallback-filesystem-relative:lib",
+        config = config,
+        # Extension may depend on any Python functionality. Include all
+        # extensions.
         extension_module_filter = "all",
-
-        # Only package the minimal set of extension modules needed to initialize
-        # a Python interpreter. Many common packages in Python's standard
-        # library won't work with this setting.
-        #extension_module_filter='minimal',
-
-        # Only package extension modules that don't require linking against
-        # non-Python libraries. e.g. will exclude support for OpenSSL, SQLite3,
-        # other features that require external libraries.
-        #extension_module_filter='no-libraries',
-
-        # Only package extension modules that don't link against GPL licensed
-        # libraries.
-        #extension_module_filter='no-gpl',
-
-        # Include Python module sources. This isn't strictly required and it does
-        # make binary sizes larger. But having the sources can be useful for
-        # activities such as debugging.
-        include_sources = True,
-
-        # Whether to include non-module resource data/files.
-        include_resources = False,
-
-        # Do not include functionality for testing Python itself.
-        include_test = False,
     )
 
-    # Invoke `pip install` with our Python distribution to install a single package.
-    # `pip_install()` returns objects representing installed files.
-    # `add_in_memory_python_resources()` adds these objects to the binary,
-    # marking them for in-memory loading.
-    #exe.add_in_memory_python_resources(dist.pip_install(["appdirs"]))
+    exe.add_python_resources(dist.pip_install([ROOT]))
 
-    # Invoke `pip install` using a requirements file and add the collected resources
-    # to our binary.
-    #exe.add_in_memory_python_resources(dist.pip_install(["-r", "requirements.txt"]))
+    return exe
+
+def make_install(exe):
+    m = FileManifest()
 
-    # Read Python files from a local directory and add them to our embedded
-    # context, taking just the resources belonging to the `foo` and `bar`
-    # Python packages.
-    #exe.add_in_memory_python_resources(dist.read_package_root(
-    #    path="/src/mypackage",
-    #    packages=["foo", "bar"],
-    #))
+    # `hg` goes in root directory.
+    m.add_python_resource(".", exe)
 
-    # Discover Python files from a virtualenv and add them to our embedded
-    # context.
-    #exe.add_in_memory_python_resources(dist.read_virtualenv(path="/path/to/venv"))
+    templates = glob(
+        include = [ROOT + "/mercurial/templates/**/*"],
+        strip_prefix = ROOT + "/mercurial/",
+    )
+    m.add_manifest(templates)
 
-    # Filter all resources collected so far through a filter of names
-    # in a file.
-    #exe.filter_from_files(files=["/path/to/filter-file"]))
-
-    # Return our `PythonExecutable` instance so it can be built and
-    # referenced by other consumers of this target.
-    return exe
+    return m
 
 def make_embedded_resources(exe):
     return exe.to_embedded_resources()
 
-def make_install(exe):
-    # Create an object that represents our installed application file layout.
-    files = FileManifest()
-
-    # Add the generated executable to our install layout in the root directory.
-    files.add_python_resource(".", exe)
-
-    return files
-
-# Tell PyOxidizer about the build targets defined above.
-register_target("dist", make_dist)
-register_target("exe", make_exe, depends = ["dist"], default = True)
-register_target("resources", make_embedded_resources, depends = ["exe"], default_build_script = True)
-register_target("install", make_install, depends = ["exe"])
-
-# Resolve whatever targets the invoker of this configuration file is requesting
-# be resolved.
+register_target("exe", make_exe)
+register_target("app", make_install, depends = ["exe"], default = True)
+register_target("embedded", make_embedded_resources, depends = ["exe"], default_build_script = True)
 resolve_targets()
 
 # END OF COMMON USER-ADJUSTED SETTINGS.
diff --git a/rust/hgcli/README.md b/rust/hgcli/README.md
new file mode 100644
--- /dev/null
+++ b/rust/hgcli/README.md
@@ -0,0 +1,50 @@
+# Oxidized Mercurial
+
+This project provides a Rust implementation of the Mercurial (`hg`)
+version control tool.
+
+Under the hood, the project uses
+[PyOxidizer](https://github.com/indygreg/PyOxidizer) to embed a Python
+interpreter in a binary built with Rust. At run-time, the Rust `fn main()`
+is called and Rust code handles initial process startup. An in-process
+Python interpreter is started (if needed) to provide additional
+functionality.
+
+# Building
+
+This project currently requires an unreleased version of PyOxidizer
+(0.7.0-pre). For best results, build the exact PyOxidizer commit
+as defined in the `pyoxidizer.bzl` file:
+
+    $ git clone https://github.com/indygreg/PyOxidizer.git
+    $ cd PyOxidizer
+    $ git checkout <Git commit from pyoxidizer.bzl>
+    $ cargo build --release
+
+Then build this Rust project using the built `pyoxidizer` executable::
+
+    $ /path/to/pyoxidizer/target/release/pyoxidizer build
+
+If all goes according to plan, there should be an assembled application
+under `build/<arch>/debug/app/` with an `hg` executable:
+
+    $ build/x86_64-unknown-linux-gnu/debug/app/hg version
+    Mercurial Distributed SCM (version 5.3.1+433-f99cd77d53dc+20200331)
+    (see https://mercurial-scm.org for more information)
+
+    Copyright (C) 2005-2020 Matt Mackall and others
+    This is free software; see the source for copying conditions. There is NO
+    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+
+# Running Tests
+
+To run tests with a built `hg` executable, you can use the `--with-hg`
+argument to `run-tests.py`. But there's a wrinkle: many tests run custom
+Python scripts that need to `import` modules provided by Mercurial. Since
+these modules are embedded in the produced `hg` executable, a regular
+Python interpreter can't access them! To work around this, set `PYTHONPATH`
+to the Mercurial source directory. e.g.:
+
+    $ cd /path/to/hg/src/tests
+    $ PYTHONPATH=`pwd`/.. python3.7 run-tests.py \
+        --with-hg `pwd`/../rust/hgcli/build/x86_64-unknown-linux-gnu/debug/app/hg
diff --git a/rust/hgcli/Cargo.toml b/rust/hgcli/Cargo.toml
--- a/rust/hgcli/Cargo.toml
+++ b/rust/hgcli/Cargo.toml
@@ -4,8 +4,12 @@
 build = "build.rs"
 authors = ["Gregory Szorc <[hidden email]>"]
 edition = "2018"
+license = "GPL-2.0"
+readme = "README.md"
 
-# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+[[bin]]
+name = "hg"
+path = "src/main.rs"
 
 [dependencies]
 jemallocator-global = { version = "0.3", optional = true }



To: indygreg, #hg-reviewers
Cc: mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D8351: hgcli: customize for Mercurial

valentin.gatienbaron (Valentin Gatien-Baron)
marmoute added a comment.
marmoute accepted this revision.


  Seems overall good. I added a request for comments.

INLINE COMMENTS

> pyoxidizer.bzl:15
> +        sys_frozen = True,
> +        legacy_windows_stdio = True,
>      )

It would be nice to have a comment abotu why this is necesssary (as for the other entry in this call)

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8351/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8351

To: indygreg, #hg-reviewers, marmoute
Cc: marmoute, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

D8351: hgcli: customize for Mercurial

valentin.gatienbaron (Valentin Gatien-Baron)
In reply to this post by valentin.gatienbaron (Valentin Gatien-Baron)
Closed by commit rHGbc847878f4c0: hgcli: customize for Mercurial (authored by indygreg).
This revision was automatically updated to reflect the committed changes.
This revision was not accepted when it landed; it landed in state "Needs Review".

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST UPDATE
  https://phab.mercurial-scm.org/D8351?vs=20926&id=20938

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8351/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8351

AFFECTED FILES
  rust/hgcli/Cargo.toml
  rust/hgcli/README.md
  rust/hgcli/pyoxidizer.bzl

CHANGE DETAILS

diff --git a/rust/hgcli/pyoxidizer.bzl b/rust/hgcli/pyoxidizer.bzl
--- a/rust/hgcli/pyoxidizer.bzl
+++ b/rust/hgcli/pyoxidizer.bzl
@@ -1,147 +1,53 @@
-# This file defines how PyOxidizer application building and packaging is
-# performed. See the pyoxidizer crate's documentation for extensive
-# documentation on this file format.
+ROOT = CWD + "/../.."
 
-# Obtain the default PythonDistribution for our build target. We link
-# this distribution into our produced executable and extract the Python
-# standard library from it.
-def make_dist():
-    return default_python_distribution()
+def make_exe():
+    dist = default_python_distribution()
+
+    code = "import hgdemandimport; hgdemandimport.enable(); from mercurial import dispatch; dispatch.run()"
 
-# Configuration files consist of functions which define build "targets."
-# This function creates a Python executable and installs it in a destination
-# directory.
-def make_exe(dist):
-    # This variable defines the configuration of the
-    # embedded Python interpreter.
-    python_config = PythonInterpreterConfig(
-        #     bytes_warning=0,
-        #     dont_write_bytecode=True,
-        #     ignore_environment=True,
-        #     inspect=False,
-        #     interactive=False,
-        #     isolated=False,
-        #     legacy_windows_fs_encoding=False,
-        #     legacy_windows_stdio=False,
-        #     no_site=True,
-        #     no_user_site_directory=True,
-        #     optimize_level=0,
-        #     parser_debug=False,
-        #     stdio_encoding=None,
-        #     unbuffered_stdio=False,
-        #     filesystem_importer=False,
-        #     sys_frozen=False,
-        #     sys_meipass=False,
-        #     sys_paths=None,
-        #     raw_allocator=None,
-        #     terminfo_resolution="dynamic",
-        #     terminfo_dirs=None,
-        #     use_hash_seed=False,
-        #     verbose=0,
-        #     write_modules_directory_env=None,
-        #     run_eval=None,
-        #     run_module=None,
-        #     run_noop=False,
-        #     run_repl=True,
+    config = PythonInterpreterConfig(
+        raw_allocator = "system",
+        run_eval = code,
+        # We want to let the user load extensions from the file system
+        filesystem_importer = True,
+        # We need this to make resourceutil happy, since it looks for sys.frozen.
+        sys_frozen = True,
+        legacy_windows_stdio = True,
     )
 
-    # The run_eval, run_module, run_noop, and run_repl arguments are mutually
-    # exclusive controls over what the interpreter should do once it initializes.
-    #
-    # run_eval -- Run the specified string value via `eval()`.
-    # run_module -- Import the specified module as __main__ and run it.
-    # run_noop -- Do nothing.
-    # run_repl -- Start a Python REPL.
-    #
-    # These arguments can be ignored if you are providing your own Rust code for
-    # starting the interpreter, as Rust code has full control over interpreter
-    # behavior.
-
-    # Produce a PythonExecutable from a Python distribution, embedded
-    # resources, and other options. The returned object represents the
-    # standalone executable that will be built.
     exe = dist.to_python_executable(
-        name = "hgcli",
-        config = python_config,
-        # Embed all extension modules, making this a fully-featured Python.
+        name = "hg",
+        resources_policy = "prefer-in-memory-fallback-filesystem-relative:lib",
+        config = config,
+        # Extension may depend on any Python functionality. Include all
+        # extensions.
         extension_module_filter = "all",
-
-        # Only package the minimal set of extension modules needed to initialize
-        # a Python interpreter. Many common packages in Python's standard
-        # library won't work with this setting.
-        #extension_module_filter='minimal',
-
-        # Only package extension modules that don't require linking against
-        # non-Python libraries. e.g. will exclude support for OpenSSL, SQLite3,
-        # other features that require external libraries.
-        #extension_module_filter='no-libraries',
-
-        # Only package extension modules that don't link against GPL licensed
-        # libraries.
-        #extension_module_filter='no-gpl',
-
-        # Include Python module sources. This isn't strictly required and it does
-        # make binary sizes larger. But having the sources can be useful for
-        # activities such as debugging.
-        include_sources = True,
-
-        # Whether to include non-module resource data/files.
-        include_resources = False,
-
-        # Do not include functionality for testing Python itself.
-        include_test = False,
     )
 
-    # Invoke `pip install` with our Python distribution to install a single package.
-    # `pip_install()` returns objects representing installed files.
-    # `add_in_memory_python_resources()` adds these objects to the binary,
-    # marking them for in-memory loading.
-    #exe.add_in_memory_python_resources(dist.pip_install(["appdirs"]))
+    exe.add_python_resources(dist.pip_install([ROOT]))
 
-    # Invoke `pip install` using a requirements file and add the collected resources
-    # to our binary.
-    #exe.add_in_memory_python_resources(dist.pip_install(["-r", "requirements.txt"]))
+    return exe
+
+def make_install(exe):
+    m = FileManifest()
 
-    # Read Python files from a local directory and add them to our embedded
-    # context, taking just the resources belonging to the `foo` and `bar`
-    # Python packages.
-    #exe.add_in_memory_python_resources(dist.read_package_root(
-    #    path="/src/mypackage",
-    #    packages=["foo", "bar"],
-    #))
+    # `hg` goes in root directory.
+    m.add_python_resource(".", exe)
 
-    # Discover Python files from a virtualenv and add them to our embedded
-    # context.
-    #exe.add_in_memory_python_resources(dist.read_virtualenv(path="/path/to/venv"))
+    templates = glob(
+        include = [ROOT + "/mercurial/templates/**/*"],
+        strip_prefix = ROOT + "/mercurial/",
+    )
+    m.add_manifest(templates)
 
-    # Filter all resources collected so far through a filter of names
-    # in a file.
-    #exe.filter_from_files(files=["/path/to/filter-file"]))
-
-    # Return our `PythonExecutable` instance so it can be built and
-    # referenced by other consumers of this target.
-    return exe
+    return m
 
 def make_embedded_resources(exe):
     return exe.to_embedded_resources()
 
-def make_install(exe):
-    # Create an object that represents our installed application file layout.
-    files = FileManifest()
-
-    # Add the generated executable to our install layout in the root directory.
-    files.add_python_resource(".", exe)
-
-    return files
-
-# Tell PyOxidizer about the build targets defined above.
-register_target("dist", make_dist)
-register_target("exe", make_exe, depends = ["dist"], default = True)
-register_target("resources", make_embedded_resources, depends = ["exe"], default_build_script = True)
-register_target("install", make_install, depends = ["exe"])
-
-# Resolve whatever targets the invoker of this configuration file is requesting
-# be resolved.
+register_target("exe", make_exe)
+register_target("app", make_install, depends = ["exe"], default = True)
+register_target("embedded", make_embedded_resources, depends = ["exe"], default_build_script = True)
 resolve_targets()
 
 # END OF COMMON USER-ADJUSTED SETTINGS.
diff --git a/rust/hgcli/README.md b/rust/hgcli/README.md
new file mode 100644
--- /dev/null
+++ b/rust/hgcli/README.md
@@ -0,0 +1,50 @@
+# Oxidized Mercurial
+
+This project provides a Rust implementation of the Mercurial (`hg`)
+version control tool.
+
+Under the hood, the project uses
+[PyOxidizer](https://github.com/indygreg/PyOxidizer) to embed a Python
+interpreter in a binary built with Rust. At run-time, the Rust `fn main()`
+is called and Rust code handles initial process startup. An in-process
+Python interpreter is started (if needed) to provide additional
+functionality.
+
+# Building
+
+This project currently requires an unreleased version of PyOxidizer
+(0.7.0-pre). For best results, build the exact PyOxidizer commit
+as defined in the `pyoxidizer.bzl` file:
+
+    $ git clone https://github.com/indygreg/PyOxidizer.git
+    $ cd PyOxidizer
+    $ git checkout <Git commit from pyoxidizer.bzl>
+    $ cargo build --release
+
+Then build this Rust project using the built `pyoxidizer` executable::
+
+    $ /path/to/pyoxidizer/target/release/pyoxidizer build
+
+If all goes according to plan, there should be an assembled application
+under `build/<arch>/debug/app/` with an `hg` executable:
+
+    $ build/x86_64-unknown-linux-gnu/debug/app/hg version
+    Mercurial Distributed SCM (version 5.3.1+433-f99cd77d53dc+20200331)
+    (see https://mercurial-scm.org for more information)
+
+    Copyright (C) 2005-2020 Matt Mackall and others
+    This is free software; see the source for copying conditions. There is NO
+    warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+
+# Running Tests
+
+To run tests with a built `hg` executable, you can use the `--with-hg`
+argument to `run-tests.py`. But there's a wrinkle: many tests run custom
+Python scripts that need to `import` modules provided by Mercurial. Since
+these modules are embedded in the produced `hg` executable, a regular
+Python interpreter can't access them! To work around this, set `PYTHONPATH`
+to the Mercurial source directory. e.g.:
+
+    $ cd /path/to/hg/src/tests
+    $ PYTHONPATH=`pwd`/.. python3.7 run-tests.py \
+        --with-hg `pwd`/../rust/hgcli/build/x86_64-unknown-linux-gnu/debug/app/hg
diff --git a/rust/hgcli/Cargo.toml b/rust/hgcli/Cargo.toml
--- a/rust/hgcli/Cargo.toml
+++ b/rust/hgcli/Cargo.toml
@@ -4,8 +4,12 @@
 build = "build.rs"
 authors = ["Gregory Szorc <[hidden email]>"]
 edition = "2018"
+license = "GPL-2.0"
+readme = "README.md"
 
-# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+[[bin]]
+name = "hg"
+path = "src/main.rs"
 
 [dependencies]
 jemallocator-global = { version = "0.3", optional = true }



To: indygreg, #hg-reviewers, marmoute
Cc: marmoute, mercurial-devel
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel