[PATCH 1 of 3 V3] run-tests: change test identity from a path to a dict

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH 1 of 3 V3] run-tests: change test identity from a path to a dict

Jun Wu
# HG changeset patch
# User Jun Wu <[hidden email]>
# Date 1493491247 25200
#      Sat Apr 29 11:40:47 2017 -0700
# Node ID 93f69acc3e8c998b3a2c03b01273505a8fc6fe42
# Parent  7040f5131454b0ae9117ec10a9f33352a04746a3
# Available At https://bitbucket.org/quark-zju/hg-draft
#              hg pull https://bitbucket.org/quark-zju/hg-draft -r 93f69acc3e8c
run-tests: change test identity from a path to a dict

Previously, we use path to identify a test. A later patch adds more
information so a path is not enough to identify a test. So we change it to a
dictionary.

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -1690,5 +1690,5 @@ class TestSuite(unittest.TestSuite):
                 num_tests[0] += 1
                 if getattr(test, 'should_reload', False):
-                    return self._loadtest(test.path, num_tests[0])
+                    return self._loadtest(test, num_tests[0])
                 return test
             if not os.path.exists(test.path):
@@ -1788,5 +1788,5 @@ class TestSuite(unittest.TestSuite):
                             num_tests[0] += 1
                             tests.append(
-                                self._loadtest(test.name, num_tests[0]))
+                                self._loadtest(test, num_tests[0]))
                         else:
                             tests.append(test)
@@ -2101,4 +2101,5 @@ class TestRunner(object):
             def sortkey(f):
                 # run largest tests first, as they tend to take the longest
+                f = f['path']
                 try:
                     return perf[f]
@@ -2268,9 +2269,14 @@ class TestRunner(object):
                 args = os.listdir(b'.')
 
-        return [t for t in args
+        return [{'path': t} for t in args
                 if os.path.basename(t).startswith(b'test-')
                     and (t.endswith(b'.py') or t.endswith(b'.t'))]
 
     def _runtests(self, tests):
+        def _reloadtest(test, i):
+            # convert a test back to its description dict
+            desc = {'path': test.path}
+            return self._gettest(desc, i)
+
         try:
             if self._installdir:
@@ -2286,5 +2292,5 @@ class TestRunner(object):
                 orig = list(tests)
                 while tests:
-                    if os.path.exists(tests[0] + ".err"):
+                    if os.path.exists(tests[0]['path'] + ".err"):
                         break
                     tests.pop(0)
@@ -2310,5 +2316,5 @@ class TestRunner(object):
                               runs_per_test=self.options.runs_per_test,
                               showchannels=self.options.showchannels,
-                              tests=tests, loadtest=self._gettest)
+                              tests=tests, loadtest=_reloadtest)
             verbosity = 1
             if self.options.verbose:
@@ -2357,5 +2363,6 @@ class TestRunner(object):
         map to a known type.
         """
-        lctest = test.lower()
+        path = test['path']
+        lctest = path.lower()
         testcls = Test
 
@@ -2365,5 +2372,5 @@ class TestRunner(object):
                 break
 
-        refpath = os.path.join(self._testdir, test)
+        refpath = os.path.join(self._testdir, path)
         tmpdir = os.path.join(self._hgtmp, b'child%d' % count)
 
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

[PATCH 2 of 3 V3] runtests: rename test description dict to testdesc

Jun Wu
# HG changeset patch
# User Jun Wu <[hidden email]>
# Date 1494994660 25200
#      Tue May 16 21:17:40 2017 -0700
# Node ID 487161ec82a581324060d6300184206b5a179322
# Parent  93f69acc3e8c998b3a2c03b01273505a8fc6fe42
# Available At https://bitbucket.org/quark-zju/hg-draft
#              hg pull https://bitbucket.org/quark-zju/hg-draft -r 487161ec82a5
runtests: rename test description dict to testdesc

Previously the word "test" was used for both a Test instance and a path or
test dict. This patch renames them so it's clear that "testdesc" is the
dict, and "test" is the instance.

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -2068,9 +2068,9 @@ class TestRunner(object):
 
             self._checktools()
-            tests = self.findtests(args)
+            testdescs = self.findtests(args)
             if options.profile_runner:
                 import statprof
                 statprof.start()
-            result = self._run(tests)
+            result = self._run(testdescs)
             if options.profile_runner:
                 statprof.stop()
@@ -2081,7 +2081,7 @@ class TestRunner(object):
             os.umask(oldmask)
 
-    def _run(self, tests):
+    def _run(self, testdescs):
         if self.options.random:
-            random.shuffle(tests)
+            random.shuffle(testdescs)
         else:
             # keywords for slow tests
@@ -2119,5 +2119,5 @@ class TestRunner(object):
                     perf[f] = val / 1000.0
                     return perf[f]
-            tests.sort(key=sortkey)
+            testdescs.sort(key=sortkey)
 
         self._testdir = osenvironb[b'TESTDIR'] = getattr(
@@ -2249,5 +2249,5 @@ class TestRunner(object):
 
         try:
-            return self._runtests(tests) or 0
+            return self._runtests(testdescs) or 0
         finally:
             time.sleep(.1)
@@ -2273,5 +2273,5 @@ class TestRunner(object):
                     and (t.endswith(b'.py') or t.endswith(b'.t'))]
 
-    def _runtests(self, tests):
+    def _runtests(self, testdescs):
         def _reloadtest(test, i):
             # convert a test back to its description dict
@@ -2290,14 +2290,14 @@ class TestRunner(object):
 
             if self.options.restart:
-                orig = list(tests)
-                while tests:
-                    if os.path.exists(tests[0]['path'] + ".err"):
+                orig = list(testdescs)
+                while testdescs:
+                    if os.path.exists(testdescs[0]['path'] + ".err"):
                         break
-                    tests.pop(0)
-                if not tests:
+                    testdescs.pop(0)
+                if not testdescs:
                     print("running all tests")
-                    tests = orig
+                    testdescs = orig
 
-            tests = [self._gettest(t, i) for i, t in enumerate(tests)]
+            tests = [self._gettest(d, i) for i, d in enumerate(testdescs)]
 
             failed = False
@@ -2357,5 +2357,5 @@ class TestRunner(object):
         return port
 
-    def _gettest(self, test, count):
+    def _gettest(self, testdesc, count):
         """Obtain a Test by looking at its filename.
 
@@ -2363,5 +2363,5 @@ class TestRunner(object):
         map to a known type.
         """
-        path = test['path']
+        path = testdesc['path']
         lctest = path.lower()
         testcls = Test
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

[PATCH 3 of 3 V3] run-tests: support multiple cases in .t test

Jun Wu
In reply to this post by Jun Wu
# HG changeset patch
# User Jun Wu <[hidden email]>
# Date 1495001431 25200
#      Tue May 16 23:10:31 2017 -0700
# Node ID 7505f5ff2346c15cec1bf24fdf99122b305d8f4f
# Parent  487161ec82a581324060d6300184206b5a179322
# Available At https://bitbucket.org/quark-zju/hg-draft
#              hg pull https://bitbucket.org/quark-zju/hg-draft -r 7505f5ff2346
run-tests: support multiple cases in .t test

Sometimes we want to run similar tests with slightly different
configurations. Previously we duplicate the test files. This patch
introduces special "#testcases" syntax that allows a single .t file to
contain multiple test cases.

Defined cases could be tested using "#if".

For example, if a test should behave the same with or without an
experimental flag, we can add the following to the .t header:

    #testcases default experimental-a
    #if experimental-a
      $ cat >> $HGRCPATH << EOF
      > [experimental]
      > feature=a
      > EOF
    #endif

The "experimental-a" block won't be executed when running the "default" test
case.

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -221,4 +221,16 @@ def parselistfiles(files, listtype, warn
     return entries
 
+def parsettestcases(path):
+    """read a .t test file, return a set of test case names"""
+    cases = set()
+    try:
+        with open(path, 'rb') as f:
+            for l in f:
+                if l.startswith(b'#testcases '):
+                    cases.update(l[11:].split())
+    except IOError:
+        pass
+    return cases
+
 def getparser():
     """Obtain the OptionParser used by the CLI."""
@@ -592,4 +604,5 @@ class Test(unittest.TestCase):
         self.name = _strpath(self.bname)
         self._testdir = os.path.dirname(path)
+        self._tmpname = os.path.basename(path)
         self.errpath = os.path.join(self._testdir, b'%s.err' % self.bname)
 
@@ -651,5 +664,5 @@ class Test(unittest.TestCase):
                 raise
 
-        name = os.path.basename(self.path)
+        name = self._tmpname
         self._testtmp = os.path.join(self._threadtmp, name)
         os.mkdir(self._testtmp)
@@ -1061,4 +1074,17 @@ class TTest(Test):
     ESCAPEMAP.update({b'\\': b'\\\\', b'\r': br'\r'})
 
+    def __init__(self, path, *args, **kwds):
+        # accept an extra "case" parameter
+        case = None
+        if 'case' in kwds:
+            case = kwds.pop('case')
+        self._case = case
+        self._allcases = parsettestcases(path)
+        super(TTest, self).__init__(path, *args, **kwds)
+        if case:
+            self.name += b' (case %s)' % case
+            self.errpath = b'%s.%s.err' % (self.errpath[:-4], case)
+            self._tmpname += b'-%s' % case
+
     @property
     def refpath(self):
@@ -1116,4 +1142,18 @@ class TTest(Test):
         return True, None
 
+    def _iftest(self, args):
+        # implements "#if"
+        reqs = []
+        for arg in args:
+            if arg.startswith(b'no-') and arg[3:] in self._allcases:
+                if arg[3:] == self._case:
+                    return False
+            elif arg in self._allcases:
+                if arg != self._case:
+                    return False
+            else:
+                reqs.append(arg)
+        return self._hghave(reqs)[0]
+
     def _parsetest(self, lines):
         # We generate a shell script which outputs unique markers to line
@@ -1173,5 +1213,5 @@ class TTest(Test):
                 if skipping is not None:
                     after.setdefault(pos, []).append('  !!! nested #if\n')
-                skipping = not self._hghave(lsplit[1:])[0]
+                skipping = not self._iftest(lsplit[1:])
                 after.setdefault(pos, []).append(l)
             elif l.startswith(b'#else'):
@@ -2269,7 +2309,19 @@ class TestRunner(object):
                 args = os.listdir(b'.')
 
-        return [{'path': t} for t in args
-                if os.path.basename(t).startswith(b'test-')
-                    and (t.endswith(b'.py') or t.endswith(b'.t'))]
+        tests = []
+        for t in args:
+            if not (os.path.basename(t).startswith(b'test-')
+                    and (t.endswith(b'.py') or t.endswith(b'.t'))):
+                continue
+            if t.endswith(b'.t'):
+                # .t file may contain multiple test cases
+                cases = sorted(parsettestcases(t))
+                if cases:
+                    tests += [{'path': t, 'case': c} for c in sorted(cases)]
+                else:
+                    tests.append({'path': t})
+            else:
+                tests.append({'path': t})
+        return tests
 
     def _runtests(self, testdescs):
@@ -2277,4 +2329,7 @@ class TestRunner(object):
             # convert a test back to its description dict
             desc = {'path': test.path}
+            case = getattr(test, '_case', None)
+            if case:
+                desc['case'] = case
             return self._gettest(desc, i)
 
@@ -2292,5 +2347,10 @@ class TestRunner(object):
                 orig = list(testdescs)
                 while testdescs:
-                    if os.path.exists(testdescs[0]['path'] + ".err"):
+                    desc = testdescs[0]
+                    if 'case' in desc:
+                        errpath = '%s.%s.err' % (desc['path'], desc['case'])
+                    else:
+                        errpath = '%s.err' % desc['path']
+                    if os.path.exists(errpath):
                         break
                     testdescs.pop(0)
@@ -2375,4 +2435,7 @@ class TestRunner(object):
         tmpdir = os.path.join(self._hgtmp, b'child%d' % count)
 
+        # extra keyword parameters. 'case' is used by .t tests
+        kwds = dict((k, testdesc[k]) for k in ['case'] if k in testdesc)
+
         t = testcls(refpath, tmpdir,
                     keeptmpdir=self.options.keep_tmpdir,
@@ -2385,5 +2448,5 @@ class TestRunner(object):
                     hgcommand=self._hgcommand,
                     usechg=bool(self.options.with_chg or self.options.chg),
-                    useipv6=useipv6)
+                    useipv6=useipv6, **kwds)
         t.should_reload = True
         return t
diff --git a/tests/test-run-tests.t b/tests/test-run-tests.t
--- a/tests/test-run-tests.t
+++ b/tests/test-run-tests.t
@@ -901,2 +901,77 @@ support for bisecting failed tests autom
   python hash seed: * (glob)
   [1]
+
+  $ cd ..
+
+Test cases in .t files
+======================
+  $ mkdir cases
+  $ cd cases
+  $ cat > test-cases-abc.t <<'EOF'
+  > #testcases A B C
+  >   $ V=B
+  > #if A
+  >   $ V=A
+  > #endif
+  > #if C
+  >   $ V=C
+  > #endif
+  >   $ echo $V | sed 's/A/C/'
+  >   C
+  > #if C
+  >   $ [ $V = C ]
+  > #endif
+  > #if A
+  >   $ [ $V = C ]
+  >   [1]
+  > #endif
+  > #if no-C
+  >   $ [ $V = C ]
+  >   [1]
+  > #endif
+  >   $ [ $V = D ]
+  >   [1]
+  > EOF
+  $ rt
+  .
+  --- $TESTTMP/anothertests/cases/test-cases-abc.t
+  +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
+  @@ -7,7 +7,7 @@
+     $ V=C
+   #endif
+     $ echo $V | sed 's/A/C/'
+  -  C
+  +  B
+   #if C
+     $ [ $V = C ]
+   #endif
+  
+  ERROR: test-cases-abc.t (case B) output changed
+  !.
+  Failed test-cases-abc.t (case B): output changed
+  # Ran 3 tests, 0 skipped, 0 warned, 1 failed.
+  python hash seed: * (glob)
+  [1]
+
+--restart works
+
+  $ rt --restart
+  
+  --- $TESTTMP/anothertests/cases/test-cases-abc.t
+  +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
+  @@ -7,7 +7,7 @@
+     $ V=C
+   #endif
+     $ echo $V | sed 's/A/C/'
+  -  C
+  +  B
+   #if C
+     $ [ $V = C ]
+   #endif
+  
+  ERROR: test-cases-abc.t (case B) output changed
+  !.
+  Failed test-cases-abc.t (case B): output changed
+  # Ran 2 tests, 0 skipped, 0 warned, 1 failed.
+  python hash seed: * (glob)
+  [1]
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 2 of 3 V3] runtests: rename test description dict to testdesc

Yuya Nishihara
In reply to this post by Jun Wu
On Wed, 17 May 2017 08:36:43 -0700, Jun Wu wrote:
> # HG changeset patch
> # User Jun Wu <[hidden email]>
> # Date 1494994660 25200
> #      Tue May 16 21:17:40 2017 -0700
> # Node ID 487161ec82a581324060d6300184206b5a179322
> # Parent  93f69acc3e8c998b3a2c03b01273505a8fc6fe42
> # Available At https://bitbucket.org/quark-zju/hg-draft
> #              hg pull https://bitbucket.org/quark-zju/hg-draft -r 487161ec82a5
> runtests: rename test description dict to testdesc

Queued the first two, thanks.
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 3 of 3 V3] run-tests: support multiple cases in .t test

Yuya Nishihara
In reply to this post by Jun Wu
On Wed, 17 May 2017 08:36:44 -0700, Jun Wu wrote:
> # HG changeset patch
> # User Jun Wu <[hidden email]>
> # Date 1495001431 25200
> #      Tue May 16 23:10:31 2017 -0700
> # Node ID 7505f5ff2346c15cec1bf24fdf99122b305d8f4f
> # Parent  487161ec82a581324060d6300184206b5a179322
> # Available At https://bitbucket.org/quark-zju/hg-draft
> #              hg pull https://bitbucket.org/quark-zju/hg-draft -r 7505f5ff2346
> run-tests: support multiple cases in .t test

> --- a/tests/run-tests.py
> +++ b/tests/run-tests.py
> @@ -221,4 +221,16 @@ def parselistfiles(files, listtype, warn
>      return entries
>  
> +def parsettestcases(path):
> +    """read a .t test file, return a set of test case names"""
> +    cases = set()
> +    try:
> +        with open(path, 'rb') as f:
> +            for l in f:
> +                if l.startswith(b'#testcases '):
> +                    cases.update(l[11:].split())
> +    except IOError:
> +        pass

Nit: maybe better to check errno?

> +    def __init__(self, path, *args, **kwds):
> +        # accept an extra "case" parameter
> +        case = None
> +        if 'case' in kwds:
> +            case = kwds.pop('case')
> +        self._case = case
> +        self._allcases = parsettestcases(path)
> +        super(TTest, self).__init__(path, *args, **kwds)
> +        if case:
> +            self.name += b' (case %s)' % case
> +            self.errpath = b'%s.%s.err' % (self.errpath[:-4], case)
> +            self._tmpname += b'-%s' % case

It appears we have (name, bname) pair, which is (str, bytes) respectively.
Maybe one of them could be a @property.
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 3 of 3 V3] run-tests: support multiple cases in .t test

Jun Wu
Excerpts from Yuya Nishihara's message of 2017-05-18 22:04:09 +0900:
> It appears we have (name, bname) pair, which is (str, bytes) respectively.
> Maybe one of them could be a @property.

"b" seems to be "basename", not "bytes". So they are different. "name" is
human-facing, like "testname (case XXX)", while "bname" is the filename.
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH 3 of 3 V3] run-tests: support multiple cases in .t test

Yuya Nishihara
On Thu, 18 May 2017 12:58:53 -0700, Jun Wu wrote:
> Excerpts from Yuya Nishihara's message of 2017-05-18 22:04:09 +0900:
> > It appears we have (name, bname) pair, which is (str, bytes) respectively.
> > Maybe one of them could be a @property.
>
> "b" seems to be "basename", not "bytes". So they are different. "name" is
> human-facing, like "testname (case XXX)", while "bname" is the filename.

That's true in your patch. 'bname' would be originally a bytes name per
8505eb1bafb1, but changing the meaning should be okay.
_______________________________________________
Mercurial-devel mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel