HGWeb/Jenkins problems

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

HGWeb/Jenkins problems

Jérôme Godbout
Hi,
I have intermittent problems with my hgweb with .cgi and apache (everything seem to work fine), I can perform a full checkout without issue with TortoiseHG under Windows machines to the web address, but sometime I got this error when doing the checkout into Jenkins:

hg pull --rev default
08:07:21 pulling from http://MyHost/MyRepos/
08:07:21 searching for changes
08:07:21 adding changesets
08:07:21 adding manifests
08:07:21 transaction abort!
08:07:21 rollback completed
08:07:21 abort: stream ended unexpectedly (got 0 bytes, expected 4)

I can start a build and end up with this, and start again and it will work just fine. The apache server access log only have 200 return code for the hgweb virtual host and the main access. Nothing into the error log. I'm a bit confuse about what can cause this, I double checked the network config of all machine imply into this and nothing seem to be wrong (ipV4, ipV6, hostname, domain) everything seem to be fine and ping work flawlessly under 0.75 ms. Anybody have a clue or a way to check what happen into those case? what is happening that close the stream (host unreachable, timeout, ...)?

Any tips is welcome here, I'm a bit at lost, I don't have anything into logs except this abort and I don't understand why.

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


_______________________________________________
Mercurial mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial
Reply | Threaded
Open this post in threaded view
|

Re: HGWeb/Jenkins problems

Jérôme Godbout
Hi, I sometime end up with the following error too (made 2 repos pull that work from the same host and one that failed, retry work for all 3 of them):

10:02:12 [CAD_CPP_Packaging] $ hg showconfig paths.default
10:02:12 [CAD_CPP_Packaging] $ hg pull --rev Designer-OnCall
10:02:15 pulling from http://MyHost/MyRepos/
10:02:15 searching for changes
10:02:15 ** unknown exception encountered, please report by visiting
10:02:15 ** https://mercurial-scm.org/wiki/BugTracker
10:02:15 ** Python 2.7.10 (default, May 23 2015, 09:44:00) [MSC v.1500 64 bit (AMD64)]
10:02:15 ** Mercurial Distributed SCM (version 4.0.1)
10:02:15 ** Extensions loaded: 
10:02:15 Traceback (most recent call last):
10:02:15   File "hg", line 49, in <module>
10:02:15   File "mercurial\dispatch.pyo", line 60, in run
10:02:15   File "mercurial\dispatch.pyo", line 126, in dispatch
10:02:15   File "mercurial\dispatch.pyo", line 216, in _runcatch
10:02:15   File "mercurial\dispatch.pyo", line 225, in callcatch
10:02:15   File "mercurial\dispatch.pyo", line 205, in _runcatchfunc
10:02:15   File "mercurial\dispatch.pyo", line 901, in _dispatch
10:02:15   File "mercurial\dispatch.pyo", line 650, in runcommand
10:02:15   File "mercurial\dispatch.pyo", line 909, in _runcommand
10:02:15   File "mercurial\dispatch.pyo", line 898, in <lambda>
10:02:15   File "mercurial\util.pyo", line 1037, in check
10:02:15   File "mercurial\commands.pyo", line 5912, in pull
10:02:15   File "mercurial\exchange.pyo", line 1215, in pull
10:02:15   File "mercurial\exchange.pyo", line 1353, in _pullbundle2
10:02:15   File "mercurial\wireproto.pyo", line 417, in getbundle
10:02:15   File "mercurial\bundle2.pyo", line 661, in getunbundler
10:02:15   File "mercurial\changegroup.pyo", line 38, in readexactly
10:02:15   File "mercurial\util.pyo", line 1660, in read
10:02:15   File "mercurial\util.pyo", line 1632, in splitbig
10:02:15   File "mercurial\httppeer.pyo", line 38, in zgenerator
10:02:15 zlib.error: Error -3 while decompressing: invalid code lengths set
10:02:15 ERROR: Failed to pull

Seem like I got some network problems, is there any way to troubleshoot this? I feel like the server are well setup and work most of the time and sometime I end up with those nasty error. Network fluke or data corruption or is it simply packet lost? Look like some data are not making it from the hgweb server to the Jenkins server for some obscure reason.

If someone have a way to troubleshoot this intermittent problems on a network I would be glad. 

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  +1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Wed, Aug 30, 2017 at 10:05 AM, Jérôme Godbout <[hidden email]> wrote:
Hi,
I have intermittent problems with my hgweb with .cgi and apache (everything seem to work fine), I can perform a full checkout without issue with TortoiseHG under Windows machines to the web address, but sometime I got this error when doing the checkout into Jenkins:

hg pull --rev default
08:07:21 pulling from http://MyHost/MyRepos/
08:07:21 searching for changes
08:07:21 adding changesets
08:07:21 adding manifests
08:07:21 transaction abort!
08:07:21 rollback completed
08:07:21 abort: stream ended unexpectedly (got 0 bytes, expected 4)

I can start a build and end up with this, and start again and it will work just fine. The apache server access log only have 200 return code for the hgweb virtual host and the main access. Nothing into the error log. I'm a bit confuse about what can cause this, I double checked the network config of all machine imply into this and nothing seem to be wrong (ipV4, ipV6, hostname, domain) everything seem to be fine and ping work flawlessly under 0.75 ms. Anybody have a clue or a way to check what happen into those case? what is happening that close the stream (host unreachable, timeout, ...)?

Any tips is welcome here, I'm a bit at lost, I don't have anything into logs except this abort and I don't understand why.

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.



_______________________________________________
Mercurial mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial
Reply | Threaded
Open this post in threaded view
|

Re: HGWeb/Jenkins problems

Jérôme Godbout
My webhg server is CentOS 7 and have the following installed version of python/mercurial:

mercurial-4.0.0.9_rc
python-2.7.5 [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

don't known if there's some compatibility issue between those version.

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  +1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Thu, Aug 31, 2017 at 10:15 AM, Jérôme Godbout <[hidden email]> wrote:
Hi, I sometime end up with the following error too (made 2 repos pull that work from the same host and one that failed, retry work for all 3 of them):

10:02:12 [CAD_CPP_Packaging] $ hg showconfig paths.default
10:02:12 [CAD_CPP_Packaging] $ hg pull --rev Designer-OnCall
10:02:15 pulling from http://MyHost/MyRepos/
10:02:15 searching for changes
10:02:15 ** unknown exception encountered, please report by visiting
10:02:15 ** https://mercurial-scm.org/wiki/BugTracker
10:02:15 ** Python 2.7.10 (default, May 23 2015, 09:44:00) [MSC v.1500 64 bit (AMD64)]
10:02:15 ** Mercurial Distributed SCM (version 4.0.1)
10:02:15 ** Extensions loaded: 
10:02:15 Traceback (most recent call last):
10:02:15   File "hg", line 49, in <module>
10:02:15   File "mercurial\dispatch.pyo", line 60, in run
10:02:15   File "mercurial\dispatch.pyo", line 126, in dispatch
10:02:15   File "mercurial\dispatch.pyo", line 216, in _runcatch
10:02:15   File "mercurial\dispatch.pyo", line 225, in callcatch
10:02:15   File "mercurial\dispatch.pyo", line 205, in _runcatchfunc
10:02:15   File "mercurial\dispatch.pyo", line 901, in _dispatch
10:02:15   File "mercurial\dispatch.pyo", line 650, in runcommand
10:02:15   File "mercurial\dispatch.pyo", line 909, in _runcommand
10:02:15   File "mercurial\dispatch.pyo", line 898, in <lambda>
10:02:15   File "mercurial\util.pyo", line 1037, in check
10:02:15   File "mercurial\commands.pyo", line 5912, in pull
10:02:15   File "mercurial\exchange.pyo", line 1215, in pull
10:02:15   File "mercurial\exchange.pyo", line 1353, in _pullbundle2
10:02:15   File "mercurial\wireproto.pyo", line 417, in getbundle
10:02:15   File "mercurial\bundle2.pyo", line 661, in getunbundler
10:02:15   File "mercurial\changegroup.pyo", line 38, in readexactly
10:02:15   File "mercurial\util.pyo", line 1660, in read
10:02:15   File "mercurial\util.pyo", line 1632, in splitbig
10:02:15   File "mercurial\httppeer.pyo", line 38, in zgenerator
10:02:15 zlib.error: Error -3 while decompressing: invalid code lengths set
10:02:15 ERROR: Failed to pull

Seem like I got some network problems, is there any way to troubleshoot this? I feel like the server are well setup and work most of the time and sometime I end up with those nasty error. Network fluke or data corruption or is it simply packet lost? Look like some data are not making it from the hgweb server to the Jenkins server for some obscure reason.

If someone have a way to troubleshoot this intermittent problems on a network I would be glad. 

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Wed, Aug 30, 2017 at 10:05 AM, Jérôme Godbout <[hidden email]> wrote:
Hi,
I have intermittent problems with my hgweb with .cgi and apache (everything seem to work fine), I can perform a full checkout without issue with TortoiseHG under Windows machines to the web address, but sometime I got this error when doing the checkout into Jenkins:

hg pull --rev default
08:07:21 pulling from http://MyHost/MyRepos/
08:07:21 searching for changes
08:07:21 adding changesets
08:07:21 adding manifests
08:07:21 transaction abort!
08:07:21 rollback completed
08:07:21 abort: stream ended unexpectedly (got 0 bytes, expected 4)

I can start a build and end up with this, and start again and it will work just fine. The apache server access log only have 200 return code for the hgweb virtual host and the main access. Nothing into the error log. I'm a bit confuse about what can cause this, I double checked the network config of all machine imply into this and nothing seem to be wrong (ipV4, ipV6, hostname, domain) everything seem to be fine and ping work flawlessly under 0.75 ms. Anybody have a clue or a way to check what happen into those case? what is happening that close the stream (host unreachable, timeout, ...)?

Any tips is welcome here, I'm a bit at lost, I don't have anything into logs except this abort and I don't understand why.

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.




_______________________________________________
Mercurial mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial
Reply | Threaded
Open this post in threaded view
|

Re: HGWeb/Jenkins problems

Matt Harbison-2
On Thu, 31 Aug 2017 10:22:59 -0400, Jérôme Godbout <[hidden email]>  
wrote:

> My webhg server is CentOS 7 and have the following installed version of
> python/mercurial:
>
> mercurial-4.0.0.9_rc
> python-2.7.5 [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]
>
> don't known if there's some compatibility issue between those version.

You may be hitting something that others have occasionally reported (and  
hasn't been tracked down yet, AFAIK).

https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096435.html
https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-June/099528.html

The only thing that I can suggest is to upgrade Mercurial to the latest  
release, to see if the "got 0 bytes, expected 4" message changes to  
something more useful.

FWIW, one coworker was getting these messages (and also similar  
decompressing errors, and SSL errors when using https), but pretty  
consistently.  He swapped network cards, and hasn't had a problem since.  
I've got another coworker trying to pilot Mercurial who is also running  
into these issues (using the same model of network card), but I haven't  
had time to really troubleshoot yet.  But I think you said your problem is  
intermittent, so unfortunately, that probably won't help you.
_______________________________________________
Mercurial mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial
Reply | Threaded
Open this post in threaded view
|

Re: HGWeb/Jenkins problems

Jérôme Godbout
I'm trying some apache settings to change the behavior of the jenkins proxy that may be related to the problems we see. I will try to keep in touch if this change anything or solve the problems. The problems might be on the Jenkins side with the proxy.
SetEnv proxy-nokeepalive 1
Hope this fix the issue, but I'm not sure it will.

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  +1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Mon, Sep 4, 2017 at 9:01 PM, Matt Harbison <[hidden email]> wrote:
On Thu, 31 Aug 2017 10:22:59 -0400, Jérôme Godbout <[hidden email]> wrote:

My webhg server is CentOS 7 and have the following installed version of
python/mercurial:

mercurial-4.0.0.9_rc
python-2.7.5 [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

don't known if there's some compatibility issue between those version.

You may be hitting something that others have occasionally reported (and hasn't been tracked down yet, AFAIK).

https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096435.html
https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-June/099528.html

The only thing that I can suggest is to upgrade Mercurial to the latest release, to see if the "got 0 bytes, expected 4" message changes to something more useful.

FWIW, one coworker was getting these messages (and also similar decompressing errors, and SSL errors when using https), but pretty consistently.  He swapped network cards, and hasn't had a problem since.  I've got another coworker trying to pilot Mercurial who is also running into these issues (using the same model of network card), but I haven't had time to really troubleshoot yet.  But I think you said your problem is intermittent, so unfortunately, that probably won't help you.


_______________________________________________
Mercurial mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial
Reply | Threaded
Open this post in threaded view
|

Re: HGWeb/Jenkins problems

Matt Harbison-2
(Back to the list)

On Sep 12, 2017, at 10:04 AM, Jérôme Godbout <[hidden email]> wrote:

I have the lastest package for CentOS7 available 4.0.0.9_rc, maybe I should go back to 3.X ?

I would uninstall this, and use pip to install the latest.  Unless you know that some version of 3.x works, in which case, please report that back.  I'd be curious what the first version is where this issue is noticeable.

I still have intermittent problems with the hgweb (all other machine work without issue), but polling from Jenkins still give the same error every day:
hg pull --rev default
pulling from http://HGSERVER/Bodycad/
searching for changes
adding changesets
adding manifests
transaction abort!
rollback completed
abort: stream ended unexpectedly (got 0 bytes, expected 4)

The abort message may be different with newer versions.  That info would be good to know.  Also, adding --traceback to the command will print a stack where the failure occurs.

Redmine also work for a few hour and after the repository page return:
500
An error occurred when trying to access the repository: hg exited with non-zero status: 255

I can browse and I can access the code without issue, checkout work for most other repos without a problems, hg verify don't see any issue. Note, the repos have some sub-repos, which can be large, is that something that could cause that much problems? Is this a timeout problems? I have 0 idea what is wrong

Yeah, the problem seems to appear when using the wire protocol.  Checkout, verify and browsing don't touch this path.  Subrepos shouldn't matter for this.

Here's the apache config (I don't even have the security put yet)
<VirtualHost *:80>
        ServerName HGSERVER
        RewriteEngine on
        ProxyRequests off
        ProxyPreserveHost on
        AllowEncodedSlashes NoDecode
        ErrorLog "/var/log/httpd/vhost-mercurial-error.log"
        CustomLog "/var/log/httpd/vhost-mercurial-access.log" common

        LimitRequestLine 32760

        ################
        # hgweb

        RewriteRule                     ^/staticHG/(.*) "/usr/lib64/python2.7/site-packages/mercurial/templates/static/$1" [L]

        RewriteRule                     ^/(?!git|static)(.*)          "/var/www/cgi-bin/hgweb.cgi/$1" [L]
        ProxyPassReverse        /                       "/var/www/cgi-bin/hgweb.cgi/"

        # Enable access to static folder for hgweb
        <Directory "/usr/lib64/python2.7/site-packages/mercurial/templates/static/">
                Require all granted
        </Directory>

        # Enable execution of cgi-bin scripts
        <Directory "/var/www/cgi-bin/">
                Require all granted
                Options ExecCGI FollowSymLinks MultiViews Indexes
                AddHandler cgi-script .cgi
        </Directory>
</VirtualHost>

Here the hgweb config file, nothing too weird here
[paths]
Bodycad = /mnt/hg3/Bodycad/**
Jenkinsfile = /mnt/hg3/Jenkinsfile/*
JenkinsBuildHelperScripts = /mnt/hg3/JenkinsBuildHelperScripts/*

[web]
contact = [hidden email]
description = Laboratoire Bodycad repositories
allow_archive = zip
push_ssl = False
allow_push = *
allow_pull = True
allow_read = *
encoding = cp1251
baseurl = /
staticurl = /staticHG/
verbose = True

The only weird thing is that the files are located on another server with CIFS share mount locally under /mnt/hg3 with the following options:
//FILESERVER/share/hg_repositories /mnt/hg3 cifs auto,credentials=/.win_credentials,uid=apache,gid=apache,dir_mode=0777,file_mode=0777 0 0

The only thing I see into the access log (the error log is empty) is that the other tools are trying to use hg-history get request, which is not available, they fallback to raw-file which seem to work.
[12/Sep/2017:09:36:04 -0400] "GET /Bodycad/hg-history/78ec7dcc602d/MyFolder/MyFiles.cpp HTTP/1.1" 400 1592
[12/Sep/2017:09:37:50 -0400] "GET /Bodycad/raw-file/78ec7dcc602d/MyFolder/MyFiles.cpp HTTP/1.1" 200 5483

I've never seen the file browsing over http fail (and I suspect it's ok here).  The entries in the log for a pull or clone look different, and that's the I/O that would be of interest here.

Any idea? This is painful situation that prevent us from working properly with redmine/reviewboard, the whole workflow get broken because of this.

Other than upgrading and trying to collect more diagnostics, I don't, unfortunately.  We are supposed to be setting up a ReviewBoard server (and I thought a coworker had one setup already for testing purposes, without a problem).  I'll let you know if I run into similar issues.

Maybe you can mount the cifs share read-only on the Jenkins server, so that the repos don't need to be cloned/pulled over http to it?


On Tue, Sep 5, 2017 at 9:28 AM, Jérôme Godbout <[hidden email]> wrote:
I'm trying some apache settings to change the behavior of the jenkins proxy that may be related to the problems we see. I will try to keep in touch if this change anything or solve the problems. The problems might be on the Jenkins side with the proxy.
SetEnv proxy-nokeepalive 1
Hope this fix the issue, but I'm not sure it will.

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Mon, Sep 4, 2017 at 9:01 PM, Matt Harbison <[hidden email]> wrote:
On Thu, 31 Aug 2017 10:22:59 -0400, Jérôme Godbout <[hidden email]> wrote:

My webhg server is CentOS 7 and have the following installed version of
python/mercurial:

mercurial-4.0.0.9_rc
python-2.7.5 [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

don't known if there's some compatibility issue between those version.

You may be hitting something that others have occasionally reported (and hasn't been tracked down yet, AFAIK).

https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096435.html
https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-June/099528.html

The only thing that I can suggest is to upgrade Mercurial to the latest release, to see if the "got 0 bytes, expected 4" message changes to something more useful.

FWIW, one coworker was getting these messages (and also similar decompressing errors, and SSL errors when using https), but pretty consistently.  He swapped network cards, and hasn't had a problem since.  I've got another coworker trying to pilot Mercurial who is also running into these issues (using the same model of network card), but I haven't had time to really troubleshoot yet.  But I think you said your problem is intermittent, so unfortunately, that probably won't help you.



_______________________________________________
Mercurial mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial
Reply | Threaded
Open this post in threaded view
|

Re: HGWeb/Jenkins problems

Jérôme Godbout
I did update all mercurial on all build machine (Jenkins/ReviewBoard/HgWeb) with python pip 3.4.1. Sadly this did not changed the problems at all :-( 
Since I don't have a full control over the command line Jenkins used (only it's .hgrc config like setup) how can I add the traceback you are looking for?

pull = --traceback --debug

into which section [section] I should add this or something like this I guess???

here's the full polling jenkins log message:

Started on Sep 13, 2017 10:15:00 AM
[CAD_CPP_ContinuousBuild@script] $ hg pull --rev VS2015
pulling from http://HGSERVER/Jenkinsfile
no changes found
[CAD_CPP_ContinuousBuild@script] $ hg log --rev VS2015 --template {node}
[CAD_CPP_ContinuousBuild@script] $ hg log --rev VS2015 --template {rev}
no polling baseline in c:\Jenkins\workspace\CAD_CPP\CAD_CPP_ContinuousBuild@tmp\Jenkinsfile on JGMachine
[CAD_CPP_ContinuousBuild] $ hg pull --rev default
pulling from http://HGSERVER/Bodycad/
searching for changes adding changesets adding manifests adding file changes transaction abort! rollback completed abort: stream ended unexpectedly (got 0 bytes, expected 4) [CAD_CPP_ContinuousBuild] $ hg log --rev default --template {node} [CAD_CPP_ContinuousBuild] $ hg log --rev default --template {rev} Done. Took 11 sec No changes

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  +1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Tue, Sep 12, 2017 at 11:29 PM, Matt Harbison <[hidden email]> wrote:
(Back to the list)

On Sep 12, 2017, at 10:04 AM, Jérôme Godbout <[hidden email]> wrote:

I have the lastest package for CentOS7 available 4.0.0.9_rc, maybe I should go back to 3.X ?

I would uninstall this, and use pip to install the latest.  Unless you know that some version of 3.x works, in which case, please report that back.  I'd be curious what the first version is where this issue is noticeable.

I still have intermittent problems with the hgweb (all other machine work without issue), but polling from Jenkins still give the same error every day:
hg pull --rev default
pulling from http://HGSERVER/Bodycad/
searching for changes
adding changesets
adding manifests
transaction abort!
rollback completed
abort: stream ended unexpectedly (got 0 bytes, expected 4)

The abort message may be different with newer versions.  That info would be good to know.  Also, adding --traceback to the command will print a stack where the failure occurs.

Redmine also work for a few hour and after the repository page return:
500
An error occurred when trying to access the repository: hg exited with non-zero status: 255

I can browse and I can access the code without issue, checkout work for most other repos without a problems, hg verify don't see any issue. Note, the repos have some sub-repos, which can be large, is that something that could cause that much problems? Is this a timeout problems? I have 0 idea what is wrong

Yeah, the problem seems to appear when using the wire protocol.  Checkout, verify and browsing don't touch this path.  Subrepos shouldn't matter for this.

Here's the apache config (I don't even have the security put yet)
<VirtualHost *:80>
        ServerName HGSERVER
        RewriteEngine on
        ProxyRequests off
        ProxyPreserveHost on
        AllowEncodedSlashes NoDecode
        ErrorLog "/var/log/httpd/vhost-mercurial-error.log"
        CustomLog "/var/log/httpd/vhost-mercurial-access.log" common

        LimitRequestLine 32760

        ################
        # hgweb

        RewriteRule                     ^/staticHG/(.*) "/usr/lib64/python2.7/site-packages/mercurial/templates/static/$1" [L]

        RewriteRule                     ^/(?!git|static)(.*)          "/var/www/cgi-bin/hgweb.cgi/$1" [L]
        ProxyPassReverse        /                       "/var/www/cgi-bin/hgweb.cgi/"

        # Enable access to static folder for hgweb
        <Directory "/usr/lib64/python2.7/site-packages/mercurial/templates/static/">
                Require all granted
        </Directory>

        # Enable execution of cgi-bin scripts
        <Directory "/var/www/cgi-bin/">
                Require all granted
                Options ExecCGI FollowSymLinks MultiViews Indexes
                AddHandler cgi-script .cgi
        </Directory>
</VirtualHost>

Here the hgweb config file, nothing too weird here
[paths]
Bodycad = /mnt/hg3/Bodycad/**
Jenkinsfile = /mnt/hg3/Jenkinsfile/*
JenkinsBuildHelperScripts = /mnt/hg3/JenkinsBuildHelperScripts/*

[web]
contact = [hidden email]
description = Laboratoire Bodycad repositories
allow_archive = zip
push_ssl = False
allow_push = *
allow_pull = True
allow_read = *
encoding = cp1251
baseurl = /
staticurl = /staticHG/
verbose = True

The only weird thing is that the files are located on another server with CIFS share mount locally under /mnt/hg3 with the following options:
//FILESERVER/share/hg_repositories /mnt/hg3 cifs auto,credentials=/.win_credentials,uid=apache,gid=apache,dir_mode=0777,file_mode=0777 0 0

The only thing I see into the access log (the error log is empty) is that the other tools are trying to use hg-history get request, which is not available, they fallback to raw-file which seem to work.
[12/Sep/2017:09:36:04 -0400] "GET /Bodycad/hg-history/78ec7dcc602d/MyFolder/MyFiles.cpp HTTP/1.1" 400 1592
[12/Sep/2017:09:37:50 -0400] "GET /Bodycad/raw-file/78ec7dcc602d/MyFolder/MyFiles.cpp HTTP/1.1" 200 5483

I've never seen the file browsing over http fail (and I suspect it's ok here).  The entries in the log for a pull or clone look different, and that's the I/O that would be of interest here.

Any idea? This is painful situation that prevent us from working properly with redmine/reviewboard, the whole workflow get broken because of this.

Other than upgrading and trying to collect more diagnostics, I don't, unfortunately.  We are supposed to be setting up a ReviewBoard server (and I thought a coworker had one setup already for testing purposes, without a problem).  I'll let you know if I run into similar issues.

Maybe you can mount the cifs share read-only on the Jenkins server, so that the repos don't need to be cloned/pulled over http to it?


On Tue, Sep 5, 2017 at 9:28 AM, Jérôme Godbout <[hidden email]> wrote:
I'm trying some apache settings to change the behavior of the jenkins proxy that may be related to the problems we see. I will try to keep in touch if this change anything or solve the problems. The problems might be on the Jenkins side with the proxy.
SetEnv proxy-nokeepalive 1
Hope this fix the issue, but I'm not sure it will.

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Mon, Sep 4, 2017 at 9:01 PM, Matt Harbison <[hidden email]> wrote:
On Thu, 31 Aug 2017 10:22:59 -0400, Jérôme Godbout <[hidden email]> wrote:

My webhg server is CentOS 7 and have the following installed version of
python/mercurial:

mercurial-4.0.0.9_rc
python-2.7.5 [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

don't known if there's some compatibility issue between those version.

You may be hitting something that others have occasionally reported (and hasn't been tracked down yet, AFAIK).

https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096435.html
https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-June/099528.html

The only thing that I can suggest is to upgrade Mercurial to the latest release, to see if the "got 0 bytes, expected 4" message changes to something more useful.

FWIW, one coworker was getting these messages (and also similar decompressing errors, and SSL errors when using https), but pretty consistently.  He swapped network cards, and hasn't had a problem since.  I've got another coworker trying to pilot Mercurial who is also running into these issues (using the same model of network card), but I haven't had time to really troubleshoot yet.  But I think you said your problem is intermittent, so unfortunately, that probably won't help you.




_______________________________________________
Mercurial mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial
Reply | Threaded
Open this post in threaded view
|

Re: HGWeb/Jenkins problems

Jérôme Godbout
If I browse the repos files I can end up on this stack trace telling me the repos file is corrupted!?! If I clone from the drive, I don't get any error like this, so I surely have something wrong going on?!? Do I have a corrupted repos that is a total mess and may explain why I have problems only with this repos. Any way to above verify to see if anything can be detected or even fixed?
 
 
<class 'mercurial.error.RevlogError'>
Python 2.7.5: /usr/bin/python
Wed Sep 13 11:48:46 2017

A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred.

 /var/www/cgi-bin/hgweb.cgi in ()
     10 from mercurial.hgweb import hgweb, wsgicgi
     11 
     12 application = hgweb(config)
=>   13 wsgicgi.launch(application)
     14 
wsgicgi = <module 'mercurial.hgweb.wsgicgi' from '/usr/lib...on2.7/site-packages/mercurial/hgweb/wsgicgi.pyc'>, wsgicgi.launch = <function launch>, application = <mercurial.hgweb.hgwebdir_mod.hgwebdir object>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/wsgicgi.py in launch(application=<mercurial.hgweb.hgwebdir_mod.hgwebdir object>)
     83     content = application(environ, start_response)
     84     try:
=>   85         for chunk in content:
     86             write(chunk)
     87         if not headers_sent:
chunk = '/Bodycad/file/1262998fd98a">browse</a></li>\n</ul...odycad/Osteotomy/Steps/Plates/PlateModel.qml</a> ', content = <generator object run_wsgi>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/hgwebdir_mod.py in run_wsgi(self=<mercurial.hgweb.hgwebdir_mod.hgwebdir object>, req=<mercurial.hgweb.request.wsgirequest object>)
    223         profile = self.ui.configbool('profiling', 'enabled')
    224         with profiling.profile(self.ui, enabled=profile):
=>  225             for r in self._runwsgi(req):
    226                 yield r
    227 
r = '/Bodycad/file/1262998fd98a">browse</a></li>\n</ul...odycad/Osteotomy/Steps/Plates/PlateModel.qml</a> ', self = <mercurial.hgweb.hgwebdir_mod.hgwebdir object>, self._runwsgi = <bound method hgwebdir._runwsgi of <mercurial.hgweb.hgwebdir_mod.hgwebdir object>>, req = <mercurial.hgweb.request.wsgirequest object>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/hgweb_mod.py in run_wsgi(self=<mercurial.hgweb.hgweb_mod.hgweb object>, req=<mercurial.hgweb.request.wsgirequest object>)
    314             profile = repo.ui.configbool('profiling', 'enabled')
    315             with profiling.profile(repo.ui, enabled=profile):
=>  316                 for r in self._runwsgi(req, repo):
    317                     yield r
    318 
r = '/Bodycad/file/1262998fd98a">browse</a></li>\n</ul...odycad/Osteotomy/Steps/Plates/PlateModel.qml</a> ', self = <mercurial.hgweb.hgweb_mod.hgweb object>, self._runwsgi = <bound method hgweb._runwsgi of <mercurial.hgweb.hgweb_mod.hgweb object>>, req = <mercurial.hgweb.request.wsgirequest object>, repo = <mercurial.localrepo.visiblefilteredrepo object>
 /usr/lib64/python2.7/site-packages/mercurial/util.py in increasingchunks(source=<generator object _flatten>, min=4096, max=65536)
    900     buf = []
    901     blen = 0
=>  902     for chunk in source:
    903         buf.append(chunk)
    904         blen += len(chunk)
chunk = '</td>\n</tr>\n<tr>\n <th class="diffstat">diffstat</th>\n <td class="diffstat">\n ', source = <generator object _flatten>
 /usr/lib64/python2.7/site-packages/mercurial/templater.py in _flatten(thing=<generator object runtemplate>)
   1109         yield pycompat.bytestr(thing)
   1110     else:
=> 1111         for i in thing:
   1112             i = templatekw.unwraphybrid(i)
   1113             if isinstance(i, bytes):
i = '</td>\n</tr>\n<tr>\n <th class="diffstat">diffstat</th>\n <td class="diffstat">\n ', thing = <generator object runtemplate>
 /usr/lib64/python2.7/site-packages/mercurial/templater.py in runtemplate(context=<mercurial.templater.engine object>, mapping={'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, template=[(<function runsymbol>, 'header'), (<function runstring>, '\n<title>'), (<function runfilter>, ((<function runsymbol>, 'repo'), <function escape>)), (<function runstring>, ': '), (<function runfilter>, ((<function runsymbol>, 'node'), <function short>)), (<function runstring>, '</title>\n</head>\n<body>\n<div class="container">\n<div class="menu">\n<div class="logo">\n<a href="'), (<function runsymbol>, 'logourl'), (<function runstring>, '">\n<img src="'), (<function runfilter>, ((<function runsymbol>, 'staticurl'), <function urlescape>)), (<function runsymbol>, 'logoimg'), (<function runstring>, '" alt="mercurial" /></a>\n</div>\n<ul>\n <li><a href="'), (<function runfilter>, ((<function runsymbol>, 'url'), <function urlescape>)), (<function runstring>, 'shortlog/'), (<function runsymbol>, 'symrev'), (<function runmap>, (<function runsymbol>, 'sessionvars', <function runtemplate>, [(<function runsymbol>, 'separator'), (<function runsymbol>, 'name'), (<function runstring>, '='), (<function runfilter>, ((...), <function urlescape>))])), (<function runstring>, '">log</a></li>\n <li><a href="'), (<function runfilter>, ((<function runsymbol>, 'url'), <function urlescape>)), (<function runstring>, 'graph/'), (<function runsymbol>, 'symrev'), (<function runmap>, (<function runsymbol>, 'sessionvars', <function runtemplate>, [(<function runsymbol>, 'separator'), (<function runsymbol>, 'name'), (<function runstring>, '='), (<function runfilter>, ((...), <function urlescape>))])), ...])
    381 def runtemplate(context, mapping, template):
    382     for func, data in template:
=>  383         yield func(context, mapping, data)
    384 
    385 def buildfilter(exp, context):
func = <function runsymbol>, context = <mercurial.templater.engine object>, mapping = {'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, data = 'diffsummary'
 /usr/lib64/python2.7/site-packages/mercurial/templater.py in runsymbol(context=<mercurial.templater.engine object>, mapping={'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, key='diffsummary', default='')
    372             v = default
    373     if callable(v):
=>  374         return v(**pycompat.strkwargs(mapping))
    375     return v
    376 
v = <function <lambda>>, global pycompat = <module 'mercurial.pycompat' from '/usr/lib64/python2.7/site-packages/mercurial/pycompat.pyc'>, pycompat.strkwargs = <function identity>, mapping = {'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/webutil.py in (**x={'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...})
    425         changesetbranch=showbranch,
    426         files=files,
=>  427         diffsummary=lambda **x: diffsummary(diffstatsgen),
    428         diffstat=diffstats,
    429         archives=web.archivelist(ctx.hex()),
global diffsummary = <function diffsummary>, x = {'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, diffstatsgen = <generator object diffstatgen>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/webutil.py in diffsummary(statgen=<generator object diffstatgen>)
    542     '''Return a short summary of the diff.'''
    543 
=>  544     stats, maxname, maxtotal, addtotal, removetotal, binary = next(statgen)
    545     return _(' %d files changed, %d insertions(+), %d deletions(-)\n') % (
    546              len(stats), addtotal, removetotal)
stats undefined, maxname undefined, maxtotal undefined, addtotal undefined, removetotal undefined, binary undefinedbuiltin next = <built-in function next>, statgen = <generator object diffstatgen>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/webutil.py in diffstatgen(ctx=<changectx 1262998fd98a>, basectx=<changectx 2778fddf3bc3>)
    534     '''Generator function that provides the diffstat data.'''
    535 
=>  536     stats = patch.diffstatdata(util.iterlines(ctx.diff(basectx)))
    537     maxname, maxtotal, addtotal, removetotal, binary = patch.diffstatsum(stats)
    538     while True:
stats undefinedglobal patch = <module 'mercurial.patch' from '/usr/lib64/python2.7/site-packages/mercurial/patch.pyc'>, patch.diffstatdata = <function diffstatdata>, global util = <module 'mercurial.util' from '/usr/lib64/python2.7/site-packages/mercurial/util.pyc'>, util.iterlines = <function iterlines>, ctx = <changectx 1262998fd98a>, ctx.diff = <bound method changectx.diff of <changectx 1262998fd98a>>,basectx = <changectx 2778fddf3bc3>
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in diffstatdata(lines=<generator object iterlines>)
   2712     inheader = False
   2713 
=> 2714     for line in lines:
   2715         if line.startswith('diff'):
   2716             addresult()
line undefinedlines = <generator object iterlines>
 /usr/lib64/python2.7/site-packages/mercurial/util.py in iterlines(iterator=<generator object diff>)
   2470 
   2471 def iterlines(iterator):
=> 2472     for chunk in iterator:
   2473         for line in chunk.splitlines():
   2474             yield line
chunk undefinediterator = <generator object diff>
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in diff(repo=<mercurial.localrepo.visiblefilteredrepo object>, node1=<changectx 2778fddf3bc3>, node2=<changectx 1262998fd98a>, match=None, changes=None, opts=<mercurial.mdiff.diffopts object>, losedatafn=None, prefix='', relroot='', copy=None)
   2319                                    changes=changes, opts=opts,
   2320                                    losedatafn=losedatafn, prefix=prefix,
=> 2321                                    relroot=relroot, copy=copy):
   2322         text = ''.join(sum((list(hlines) for hrange, hlines in hunks), []))
   2323         if header and (text or len(header) > 1):
relroot = '', copy = None
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in trydiff(repo=<mercurial.localrepo.visiblefilteredrepo object>, revs=['2778fddf3bc3', '1262998fd98a'], ctx1=<changectx 2778fddf3bc3>, ctx2=<changectx 1262998fd98a>, modified=['QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Producers/PlateProducer.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Producers/PlateSegmentProducer.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Ui2D/PlateContextualMenu.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Ui2D/SegmentContextualMenu.qml', 'QmlScripts/Common/Nurbs/Examples/ExampleBlendLinkProfile.qml', 'QmlScripts/Common/Nurbs/Examples/ExampleNurbsBlendProfile.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/LinkModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/LinkProducer.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/SegmentModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/SegmentProducer.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/StructureModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/SurfaceModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/TrajectoryModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/TrajectoryProducer.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/qmldir'], added=['QmlScripts/Common/Nurbs/ProfileBlending/LongestEdgeSurfaceIsoCurve.qml'], removed=[], copy={}, getfilectx=<function getfilectx>, opts=<mercurial.mdiff.diffopts object>, losedatafn=None, prefix='', relroot='')
   2583         flag2 = None
   2584         if f1:
=> 2585             fctx1 = getfilectx(f1, ctx1)
   2586             if opts.git or losedatafn:
   2587                 flag1 = ctx1.flags(f1)
fctx1 = None, getfilectx = <function getfilectx>, f1 = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', ctx1 = <changectx 2778fddf3bc3>
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in getfilectx(f='QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', ctx=<changectx 2778fddf3bc3>)
   2349                 if len(cache) > 20:
   2350                     del cache[order.popleft()]
=> 2351                 cache[f] = fctx.filelog()
   2352             else:
   2353                 order.remove(f)
cache = {}, f = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', fctx = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, fctx.filelog = <bound method filectx.filelog of <filectx QmlScr...eotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>>
 /usr/lib64/python2.7/site-packages/mercurial/context.py in filelog(self=<filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>)
    746         return self._flags
    747     def filelog(self):
=>  748         return self._filelog
    749     def rev(self):
    750         return self._changeid
self = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, self._filelog = <mercurial.filelog.filelog object>
 /usr/lib64/python2.7/site-packages/mercurial/util.py in __get__(self=<mercurial.util.propertycache object>, obj=<filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, type=<class 'mercurial.context.filectx'>)
    821         self.name = func.__name__
    822     def __get__(self, obj, type=None):
=>  823         result = self.func(obj)
    824         self.cachevalue(obj, result)
    825         return result
result undefinedself = <mercurial.util.propertycache object>, self.func = <function _filelog>, obj = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>
 /usr/lib64/python2.7/site-packages/mercurial/context.py in _filelog(self=<filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>)
    669     @propertycache
    670     def _filelog(self):
=>  671         return self._repo.file(self._path)
    672 
    673     @propertycache
self = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, self._repo = <mercurial.localrepo.visiblefilteredrepo object>, self._repo.file = <bound method visiblefilteredrepo.file of <mercurial.localrepo.visiblefilteredrepo object>>, self._path = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml'
 /usr/lib64/python2.7/site-packages/mercurial/localrepo.py in file(self=<mercurial.localrepo.visiblefilteredrepo object>, f='QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml')
    989         if f[0] == '/':
    990             f = f[1:]
=>  991         return filelog.filelog(self.svfs, f)
    992 
    993     def changectx(self, changeid):
global filelog = <module 'mercurial.filelog' from '/usr/lib64/python2.7/site-packages/mercurial/filelog.pyc'>, filelog.filelog = <class 'mercurial.filelog.filelog'>, self = <mercurial.localrepo.visiblefilteredrepo object>, self.svfs = <mercurial.store._fncachevfs object>, f = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml'
 /usr/lib64/python2.7/site-packages/mercurial/filelog.py in __init__(self=<mercurial.filelog.filelog object>, opener=<mercurial.store._fncachevfs object>, path='QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml')
     43     def __init__(self, opener, path):
     44         super(filelog, self).__init__(opener,
=>   45                         "/".join(("data", path + ".i")))
     46 
     47     def read(self, node):
path = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml'
 /usr/lib64/python2.7/site-packages/mercurial/revlog.py in __init__(self=<mercurial.filelog.filelog object>, opener=<mercurial.store._fncachevfs object>, indexfile='data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i', datafile=None, checkambig=False)
    377             d = self._io.parseindex(indexdata, self._inline)
    378         except (ValueError, IndexError):
=>  379             raise RevlogError(_("index %s is corrupted") % (self.indexfile))
    380         self.index, nodemap, self._chunkcache = d
    381         if nodemap is not None:
global RevlogError = <class 'mercurial.error.RevlogError'>, global _ = <function gettext>, self = <mercurial.filelog.filelog object>, self.indexfile = 'data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i'

<class 'mercurial.error.RevlogError'>: index data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i is corrupted 
      args = ('index data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i is corrupted',) 
      hint = None 
      message = 'index data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i is corrupted'


bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  +1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Wed, Sep 13, 2017 at 10:21 AM, Jérôme Godbout <[hidden email]> wrote:
I did update all mercurial on all build machine (Jenkins/ReviewBoard/HgWeb) with python pip 3.4.1. Sadly this did not changed the problems at all :-( 
Since I don't have a full control over the command line Jenkins used (only it's .hgrc config like setup) how can I add the traceback you are looking for?

pull = --traceback --debug

into which section [section] I should add this or something like this I guess???

here's the full polling jenkins log message:

Started on Sep 13, 2017 10:15:00 AM
[CAD_CPP_ContinuousBuild@script] $ hg pull --rev VS2015
pulling from http://HGSERVER/Jenkinsfile
no changes found
[CAD_CPP_ContinuousBuild@script] $ hg log --rev VS2015 --template {node}
[CAD_CPP_ContinuousBuild@script] $ hg log --rev VS2015 --template {rev}
no polling baseline in c:\Jenkins\workspace\CAD_CPP\CAD_CPP_ContinuousBuild@tmp\Jenkinsfile on JGMachine
[CAD_CPP_ContinuousBuild] $ hg pull --rev default
pulling from http://HGSERVER/Bodycad/
searching for changes adding changesets adding manifests adding file changes transaction abort! rollback completed abort: stream ended unexpectedly (got 0 bytes, expected 4) [CAD_CPP_ContinuousBuild] $ hg log --rev default --template {node} [CAD_CPP_ContinuousBuild] $ hg log --rev default --template {rev} Done. Took 11 sec No changes

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Tue, Sep 12, 2017 at 11:29 PM, Matt Harbison <[hidden email]> wrote:
(Back to the list)

On Sep 12, 2017, at 10:04 AM, Jérôme Godbout <[hidden email]> wrote:

I have the lastest package for CentOS7 available 4.0.0.9_rc, maybe I should go back to 3.X ?

I would uninstall this, and use pip to install the latest.  Unless you know that some version of 3.x works, in which case, please report that back.  I'd be curious what the first version is where this issue is noticeable.

I still have intermittent problems with the hgweb (all other machine work without issue), but polling from Jenkins still give the same error every day:
hg pull --rev default
pulling from http://HGSERVER/Bodycad/
searching for changes
adding changesets
adding manifests
transaction abort!
rollback completed
abort: stream ended unexpectedly (got 0 bytes, expected 4)

The abort message may be different with newer versions.  That info would be good to know.  Also, adding --traceback to the command will print a stack where the failure occurs.

Redmine also work for a few hour and after the repository page return:
500
An error occurred when trying to access the repository: hg exited with non-zero status: 255

I can browse and I can access the code without issue, checkout work for most other repos without a problems, hg verify don't see any issue. Note, the repos have some sub-repos, which can be large, is that something that could cause that much problems? Is this a timeout problems? I have 0 idea what is wrong

Yeah, the problem seems to appear when using the wire protocol.  Checkout, verify and browsing don't touch this path.  Subrepos shouldn't matter for this.

Here's the apache config (I don't even have the security put yet)
<VirtualHost *:80>
        ServerName HGSERVER
        RewriteEngine on
        ProxyRequests off
        ProxyPreserveHost on
        AllowEncodedSlashes NoDecode
        ErrorLog "/var/log/httpd/vhost-mercurial-error.log"
        CustomLog "/var/log/httpd/vhost-mercurial-access.log" common

        LimitRequestLine 32760

        ################
        # hgweb

        RewriteRule                     ^/staticHG/(.*) "/usr/lib64/python2.7/site-packages/mercurial/templates/static/$1" [L]

        RewriteRule                     ^/(?!git|static)(.*)          "/var/www/cgi-bin/hgweb.cgi/$1" [L]
        ProxyPassReverse        /                       "/var/www/cgi-bin/hgweb.cgi/"

        # Enable access to static folder for hgweb
        <Directory "/usr/lib64/python2.7/site-packages/mercurial/templates/static/">
                Require all granted
        </Directory>

        # Enable execution of cgi-bin scripts
        <Directory "/var/www/cgi-bin/">
                Require all granted
                Options ExecCGI FollowSymLinks MultiViews Indexes
                AddHandler cgi-script .cgi
        </Directory>
</VirtualHost>

Here the hgweb config file, nothing too weird here
[paths]
Bodycad = /mnt/hg3/Bodycad/**
Jenkinsfile = /mnt/hg3/Jenkinsfile/*
JenkinsBuildHelperScripts = /mnt/hg3/JenkinsBuildHelperScripts/*

[web]
contact = [hidden email]
description = Laboratoire Bodycad repositories
allow_archive = zip
push_ssl = False
allow_push = *
allow_pull = True
allow_read = *
encoding = cp1251
baseurl = /
staticurl = /staticHG/
verbose = True

The only weird thing is that the files are located on another server with CIFS share mount locally under /mnt/hg3 with the following options:
//FILESERVER/share/hg_repositories /mnt/hg3 cifs auto,credentials=/.win_credentials,uid=apache,gid=apache,dir_mode=0777,file_mode=0777 0 0

The only thing I see into the access log (the error log is empty) is that the other tools are trying to use hg-history get request, which is not available, they fallback to raw-file which seem to work.
[12/Sep/2017:09:36:04 -0400] "GET /Bodycad/hg-history/78ec7dcc602d/MyFolder/MyFiles.cpp HTTP/1.1" 400 1592
[12/Sep/2017:09:37:50 -0400] "GET /Bodycad/raw-file/78ec7dcc602d/MyFolder/MyFiles.cpp HTTP/1.1" 200 5483

I've never seen the file browsing over http fail (and I suspect it's ok here).  The entries in the log for a pull or clone look different, and that's the I/O that would be of interest here.

Any idea? This is painful situation that prevent us from working properly with redmine/reviewboard, the whole workflow get broken because of this.

Other than upgrading and trying to collect more diagnostics, I don't, unfortunately.  We are supposed to be setting up a ReviewBoard server (and I thought a coworker had one setup already for testing purposes, without a problem).  I'll let you know if I run into similar issues.

Maybe you can mount the cifs share read-only on the Jenkins server, so that the repos don't need to be cloned/pulled over http to it?


On Tue, Sep 5, 2017 at 9:28 AM, Jérôme Godbout <[hidden email]> wrote:
I'm trying some apache settings to change the behavior of the jenkins proxy that may be related to the problems we see. I will try to keep in touch if this change anything or solve the problems. The problems might be on the Jenkins side with the proxy.
SetEnv proxy-nokeepalive 1
Hope this fix the issue, but I'm not sure it will.

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Mon, Sep 4, 2017 at 9:01 PM, Matt Harbison <[hidden email]> wrote:
On Thu, 31 Aug 2017 10:22:59 -0400, Jérôme Godbout <[hidden email]> wrote:

My webhg server is CentOS 7 and have the following installed version of
python/mercurial:

mercurial-4.0.0.9_rc
python-2.7.5 [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

don't known if there's some compatibility issue between those version.

You may be hitting something that others have occasionally reported (and hasn't been tracked down yet, AFAIK).

https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096435.html
https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-June/099528.html

The only thing that I can suggest is to upgrade Mercurial to the latest release, to see if the "got 0 bytes, expected 4" message changes to something more useful.

FWIW, one coworker was getting these messages (and also similar decompressing errors, and SSL errors when using https), but pretty consistently.  He swapped network cards, and hasn't had a problem since.  I've got another coworker trying to pilot Mercurial who is also running into these issues (using the same model of network card), but I haven't had time to really troubleshoot yet.  But I think you said your problem is intermittent, so unfortunately, that probably won't help you.





_______________________________________________
Mercurial mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial
Reply | Threaded
Open this post in threaded view
|

Re: HGWeb/Jenkins problems

Jérôme Godbout
This only happen with recent repos revision and for a little delais, after a while it seem to work fine. Look like something is caching the repos data somehow. Could this be CIFS share? Apache? Python mercurial use some cache?

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  +1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Wed, Sep 13, 2017 at 11:53 AM, Jérôme Godbout <[hidden email]> wrote:
If I browse the repos files I can end up on this stack trace telling me the repos file is corrupted!?! If I clone from the drive, I don't get any error like this, so I surely have something wrong going on?!? Do I have a corrupted repos that is a total mess and may explain why I have problems only with this repos. Any way to above verify to see if anything can be detected or even fixed?
 
 
<class 'mercurial.error.RevlogError'>
Python 2.7.5: /usr/bin/python
Wed Sep 13 11:48:46 2017

A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred.

 /var/www/cgi-bin/hgweb.cgi in ()
     10 from mercurial.hgweb import hgweb, wsgicgi
     11 
     12 application = hgweb(config)
=>   13 wsgicgi.launch(application)
     14 
wsgicgi = <module 'mercurial.hgweb.wsgicgi' from '/usr/lib...on2.7/site-packages/mercurial/hgweb/wsgicgi.pyc'>, wsgicgi.launch = <function launch>, application = <mercurial.hgweb.hgwebdir_mod.hgwebdir object>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/wsgicgi.py in launch(application=<mercurial.hgweb.hgwebdir_mod.hgwebdir object>)
     83     content = application(environ, start_response)
     84     try:
=>   85         for chunk in content:
     86             write(chunk)
     87         if not headers_sent:
chunk = '/Bodycad/file/1262998fd98a">browse</a></li>\n</ul...odycad/Osteotomy/Steps/Plates/PlateModel.qml</a> ', content = <generator object run_wsgi>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/hgwebdir_mod.py in run_wsgi(self=<mercurial.hgweb.hgwebdir_mod.hgwebdir object>, req=<mercurial.hgweb.request.wsgirequest object>)
    223         profile = self.ui.configbool('profiling', 'enabled')
    224         with profiling.profile(self.ui, enabled=profile):
=>  225             for r in self._runwsgi(req):
    226                 yield r
    227 
r = '/Bodycad/file/1262998fd98a">browse</a></li>\n</ul...odycad/Osteotomy/Steps/Plates/PlateModel.qml</a> ', self = <mercurial.hgweb.hgwebdir_mod.hgwebdir object>, self._runwsgi = <bound method hgwebdir._runwsgi of <mercurial.hgweb.hgwebdir_mod.hgwebdir object>>, req = <mercurial.hgweb.request.wsgirequest object>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/hgweb_mod.py in run_wsgi(self=<mercurial.hgweb.hgweb_mod.hgweb object>, req=<mercurial.hgweb.request.wsgirequest object>)
    314             profile = repo.ui.configbool('profiling', 'enabled')
    315             with profiling.profile(repo.ui, enabled=profile):
=>  316                 for r in self._runwsgi(req, repo):
    317                     yield r
    318 
r = '/Bodycad/file/1262998fd98a">browse</a></li>\n</ul...odycad/Osteotomy/Steps/Plates/PlateModel.qml</a> ', self = <mercurial.hgweb.hgweb_mod.hgweb object>, self._runwsgi = <bound method hgweb._runwsgi of <mercurial.hgweb.hgweb_mod.hgweb object>>, req = <mercurial.hgweb.request.wsgirequest object>, repo = <mercurial.localrepo.visiblefilteredrepo object>
 /usr/lib64/python2.7/site-packages/mercurial/util.py in increasingchunks(source=<generator object _flatten>, min=4096, max=65536)
    900     buf = []
    901     blen = 0
=>  902     for chunk in source:
    903         buf.append(chunk)
    904         blen += len(chunk)
chunk = '</td>\n</tr>\n<tr>\n <th class="diffstat">diffstat</th>\n <td class="diffstat">\n ', source = <generator object _flatten>
 /usr/lib64/python2.7/site-packages/mercurial/templater.py in _flatten(thing=<generator object runtemplate>)
   1109         yield pycompat.bytestr(thing)
   1110     else:
=> 1111         for i in thing:
   1112             i = templatekw.unwraphybrid(i)
   1113             if isinstance(i, bytes):
i = '</td>\n</tr>\n<tr>\n <th class="diffstat">diffstat</th>\n <td class="diffstat">\n ', thing = <generator object runtemplate>
 /usr/lib64/python2.7/site-packages/mercurial/templater.py in runtemplate(context=<mercurial.templater.engine object>, mapping={'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, template=[(<function runsymbol>, 'header'), (<function runstring>, '\n<title>'), (<function runfilter>, ((<function runsymbol>, 'repo'), <function escape>)), (<function runstring>, ': '), (<function runfilter>, ((<function runsymbol>, 'node'), <function short>)), (<function runstring>, '</title>\n</head>\n<body>\n<div class="container">\n<div class="menu">\n<div class="logo">\n<a href="'), (<function runsymbol>, 'logourl'), (<function runstring>, '">\n<img src="'), (<function runfilter>, ((<function runsymbol>, 'staticurl'), <function urlescape>)), (<function runsymbol>, 'logoimg'), (<function runstring>, '" alt="mercurial" /></a>\n</div>\n<ul>\n <li><a href="'), (<function runfilter>, ((<function runsymbol>, 'url'), <function urlescape>)), (<function runstring>, 'shortlog/'), (<function runsymbol>, 'symrev'), (<function runmap>, (<function runsymbol>, 'sessionvars', <function runtemplate>, [(<function runsymbol>, 'separator'), (<function runsymbol>, 'name'), (<function runstring>, '='), (<function runfilter>, ((...), <function urlescape>))])), (<function runstring>, '">log</a></li>\n <li><a href="'), (<function runfilter>, ((<function runsymbol>, 'url'), <function urlescape>)), (<function runstring>, 'graph/'), (<function runsymbol>, 'symrev'), (<function runmap>, (<function runsymbol>, 'sessionvars', <function runtemplate>, [(<function runsymbol>, 'separator'), (<function runsymbol>, 'name'), (<function runstring>, '='), (<function runfilter>, ((...), <function urlescape>))])), ...])
    381 def runtemplate(context, mapping, template):
    382     for func, data in template:
=>  383         yield func(context, mapping, data)
    384 
    385 def buildfilter(exp, context):
func = <function runsymbol>, context = <mercurial.templater.engine object>, mapping = {'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, data = 'diffsummary'
 /usr/lib64/python2.7/site-packages/mercurial/templater.py in runsymbol(context=<mercurial.templater.engine object>, mapping={'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, key='diffsummary', default='')
    372             v = default
    373     if callable(v):
=>  374         return v(**pycompat.strkwargs(mapping))
    375     return v
    376 
v = <function <lambda>>, global pycompat = <module 'mercurial.pycompat' from '/usr/lib64/python2.7/site-packages/mercurial/pycompat.pyc'>, pycompat.strkwargs = <function identity>, mapping = {'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/webutil.py in (**x={'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...})
    425         changesetbranch=showbranch,
    426         files=files,
=>  427         diffsummary=lambda **x: diffsummary(diffstatsgen),
    428         diffstat=diffstats,
    429         archives=web.archivelist(ctx.hex()),
global diffsummary = <function diffsummary>, x = {'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, diffstatsgen = <generator object diffstatgen>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/webutil.py in diffsummary(statgen=<generator object diffstatgen>)
    542     '''Return a short summary of the diff.'''
    543 
=>  544     stats, maxname, maxtotal, addtotal, removetotal, binary = next(statgen)
    545     return _(' %d files changed, %d insertions(+), %d deletions(-)\n') % (
    546              len(stats), addtotal, removetotal)
stats undefined, maxname undefined, maxtotal undefined, addtotal undefined, removetotal undefined, binary undefinedbuiltin next = <built-in function next>, statgen = <generator object diffstatgen>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/webutil.py in diffstatgen(ctx=<changectx 1262998fd98a>, basectx=<changectx 2778fddf3bc3>)
    534     '''Generator function that provides the diffstat data.'''
    535 
=>  536     stats = patch.diffstatdata(util.iterlines(ctx.diff(basectx)))
    537     maxname, maxtotal, addtotal, removetotal, binary = patch.diffstatsum(stats)
    538     while True:
stats undefinedglobal patch = <module 'mercurial.patch' from '/usr/lib64/python2.7/site-packages/mercurial/patch.pyc'>, patch.diffstatdata = <function diffstatdata>, global util = <module 'mercurial.util' from '/usr/lib64/python2.7/site-packages/mercurial/util.pyc'>, util.iterlines = <function iterlines>, ctx = <changectx 1262998fd98a>, ctx.diff = <bound method changectx.diff of <changectx 1262998fd98a>>,basectx = <changectx 2778fddf3bc3>
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in diffstatdata(lines=<generator object iterlines>)
   2712     inheader = False
   2713 
=> 2714     for line in lines:
   2715         if line.startswith('diff'):
   2716             addresult()
line undefinedlines = <generator object iterlines>
 /usr/lib64/python2.7/site-packages/mercurial/util.py in iterlines(iterator=<generator object diff>)
   2470 
   2471 def iterlines(iterator):
=> 2472     for chunk in iterator:
   2473         for line in chunk.splitlines():
   2474             yield line
chunk undefinediterator = <generator object diff>
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in diff(repo=<mercurial.localrepo.visiblefilteredrepo object>, node1=<changectx 2778fddf3bc3>, node2=<changectx 1262998fd98a>, match=None, changes=None, opts=<mercurial.mdiff.diffopts object>, losedatafn=None, prefix='', relroot='', copy=None)
   2319                                    changes=changes, opts=opts,
   2320                                    losedatafn=losedatafn, prefix=prefix,
=> 2321                                    relroot=relroot, copy=copy):
   2322         text = ''.join(sum((list(hlines) for hrange, hlines in hunks), []))
   2323         if header and (text or len(header) > 1):
relroot = '', copy = None
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in trydiff(repo=<mercurial.localrepo.visiblefilteredrepo object>, revs=['2778fddf3bc3', '1262998fd98a'], ctx1=<changectx 2778fddf3bc3>, ctx2=<changectx 1262998fd98a>, modified=['QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Producers/PlateProducer.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Producers/PlateSegmentProducer.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Ui2D/PlateContextualMenu.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Ui2D/SegmentContextualMenu.qml', 'QmlScripts/Common/Nurbs/Examples/ExampleBlendLinkProfile.qml', 'QmlScripts/Common/Nurbs/Examples/ExampleNurbsBlendProfile.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/LinkModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/LinkProducer.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/SegmentModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/SegmentProducer.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/StructureModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/SurfaceModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/TrajectoryModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/TrajectoryProducer.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/qmldir'], added=['QmlScripts/Common/Nurbs/ProfileBlending/LongestEdgeSurfaceIsoCurve.qml'], removed=[], copy={}, getfilectx=<function getfilectx>, opts=<mercurial.mdiff.diffopts object>, losedatafn=None, prefix='', relroot='')
   2583         flag2 = None
   2584         if f1:
=> 2585             fctx1 = getfilectx(f1, ctx1)
   2586             if opts.git or losedatafn:
   2587                 flag1 = ctx1.flags(f1)
fctx1 = None, getfilectx = <function getfilectx>, f1 = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', ctx1 = <changectx 2778fddf3bc3>
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in getfilectx(f='QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', ctx=<changectx 2778fddf3bc3>)
   2349                 if len(cache) > 20:
   2350                     del cache[order.popleft()]
=> 2351                 cache[f] = fctx.filelog()
   2352             else:
   2353                 order.remove(f)
cache = {}, f = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', fctx = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, fctx.filelog = <bound method filectx.filelog of <filectx QmlScr...eotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>>
 /usr/lib64/python2.7/site-packages/mercurial/context.py in filelog(self=<filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>)
    746         return self._flags
    747     def filelog(self):
=>  748         return self._filelog
    749     def rev(self):
    750         return self._changeid
self = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, self._filelog = <mercurial.filelog.filelog object>
 /usr/lib64/python2.7/site-packages/mercurial/util.py in __get__(self=<mercurial.util.propertycache object>, obj=<filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, type=<class 'mercurial.context.filectx'>)
    821         self.name = func.__name__
    822     def __get__(self, obj, type=None):
=>  823         result = self.func(obj)
    824         self.cachevalue(obj, result)
    825         return result
result undefinedself = <mercurial.util.propertycache object>, self.func = <function _filelog>, obj = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>
 /usr/lib64/python2.7/site-packages/mercurial/context.py in _filelog(self=<filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>)
    669     @propertycache
    670     def _filelog(self):
=>  671         return self._repo.file(self._path)
    672 
    673     @propertycache
self = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, self._repo = <mercurial.localrepo.visiblefilteredrepo object>, self._repo.file = <bound method visiblefilteredrepo.file of <mercurial.localrepo.visiblefilteredrepo object>>, self._path = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml'
 /usr/lib64/python2.7/site-packages/mercurial/localrepo.py in file(self=<mercurial.localrepo.visiblefilteredrepo object>, f='QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml')
    989         if f[0] == '/':
    990             f = f[1:]
=>  991         return filelog.filelog(self.svfs, f)
    992 
    993     def changectx(self, changeid):
global filelog = <module 'mercurial.filelog' from '/usr/lib64/python2.7/site-packages/mercurial/filelog.pyc'>, filelog.filelog = <class 'mercurial.filelog.filelog'>, self = <mercurial.localrepo.visiblefilteredrepo object>, self.svfs = <mercurial.store._fncachevfs object>, f = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml'
 /usr/lib64/python2.7/site-packages/mercurial/filelog.py in __init__(self=<mercurial.filelog.filelog object>, opener=<mercurial.store._fncachevfs object>, path='QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml')
     43     def __init__(self, opener, path):
     44         super(filelog, self).__init__(opener,
=>   45                         "/".join(("data", path + ".i")))
     46 
     47     def read(self, node):
path = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml'
 /usr/lib64/python2.7/site-packages/mercurial/revlog.py in __init__(self=<mercurial.filelog.filelog object>, opener=<mercurial.store._fncachevfs object>, indexfile='data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i', datafile=None, checkambig=False)
    377             d = self._io.parseindex(indexdata, self._inline)
    378         except (ValueError, IndexError):
=>  379             raise RevlogError(_("index %s is corrupted") % (self.indexfile))
    380         self.index, nodemap, self._chunkcache = d
    381         if nodemap is not None:
global RevlogError = <class 'mercurial.error.RevlogError'>global _ = <function gettext>, self = <mercurial.filelog.filelog object>, self.indexfile = 'data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i'

<class 'mercurial.error.RevlogError'>: index data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i is corrupted 
      args = ('index data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i is corrupted',) 
      hint = None 
      message = 'index data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i is corrupted'


bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Wed, Sep 13, 2017 at 10:21 AM, Jérôme Godbout <[hidden email]> wrote:
I did update all mercurial on all build machine (Jenkins/ReviewBoard/HgWeb) with python pip 3.4.1. Sadly this did not changed the problems at all :-( 
Since I don't have a full control over the command line Jenkins used (only it's .hgrc config like setup) how can I add the traceback you are looking for?

pull = --traceback --debug

into which section [section] I should add this or something like this I guess???

here's the full polling jenkins log message:

Started on Sep 13, 2017 10:15:00 AM
[CAD_CPP_ContinuousBuild@script] $ hg pull --rev VS2015
pulling from http://HGSERVER/Jenkinsfile
no changes found
[CAD_CPP_ContinuousBuild@script] $ hg log --rev VS2015 --template {node}
[CAD_CPP_ContinuousBuild@script] $ hg log --rev VS2015 --template {rev}
no polling baseline in c:\Jenkins\workspace\CAD_CPP\CAD_CPP_ContinuousBuild@tmp\Jenkinsfile on JGMachine
[CAD_CPP_ContinuousBuild] $ hg pull --rev default
pulling from http://HGSERVER/Bodycad/
searching for changes adding changesets adding manifests adding file changes transaction abort! rollback completed abort: stream ended unexpectedly (got 0 bytes, expected 4) [CAD_CPP_ContinuousBuild] $ hg log --rev default --template {node} [CAD_CPP_ContinuousBuild] $ hg log --rev default --template {rev} Done. Took 11 sec No changes

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Tue, Sep 12, 2017 at 11:29 PM, Matt Harbison <[hidden email]> wrote:
(Back to the list)

On Sep 12, 2017, at 10:04 AM, Jérôme Godbout <[hidden email]> wrote:

I have the lastest package for CentOS7 available 4.0.0.9_rc, maybe I should go back to 3.X ?

I would uninstall this, and use pip to install the latest.  Unless you know that some version of 3.x works, in which case, please report that back.  I'd be curious what the first version is where this issue is noticeable.

I still have intermittent problems with the hgweb (all other machine work without issue), but polling from Jenkins still give the same error every day:
hg pull --rev default
pulling from http://HGSERVER/Bodycad/
searching for changes
adding changesets
adding manifests
transaction abort!
rollback completed
abort: stream ended unexpectedly (got 0 bytes, expected 4)

The abort message may be different with newer versions.  That info would be good to know.  Also, adding --traceback to the command will print a stack where the failure occurs.

Redmine also work for a few hour and after the repository page return:
500
An error occurred when trying to access the repository: hg exited with non-zero status: 255

I can browse and I can access the code without issue, checkout work for most other repos without a problems, hg verify don't see any issue. Note, the repos have some sub-repos, which can be large, is that something that could cause that much problems? Is this a timeout problems? I have 0 idea what is wrong

Yeah, the problem seems to appear when using the wire protocol.  Checkout, verify and browsing don't touch this path.  Subrepos shouldn't matter for this.

Here's the apache config (I don't even have the security put yet)
<VirtualHost *:80>
        ServerName HGSERVER
        RewriteEngine on
        ProxyRequests off
        ProxyPreserveHost on
        AllowEncodedSlashes NoDecode
        ErrorLog "/var/log/httpd/vhost-mercurial-error.log"
        CustomLog "/var/log/httpd/vhost-mercurial-access.log" common

        LimitRequestLine 32760

        ################
        # hgweb

        RewriteRule                     ^/staticHG/(.*) "/usr/lib64/python2.7/site-packages/mercurial/templates/static/$1" [L]

        RewriteRule                     ^/(?!git|static)(.*)          "/var/www/cgi-bin/hgweb.cgi/$1" [L]
        ProxyPassReverse        /                       "/var/www/cgi-bin/hgweb.cgi/"

        # Enable access to static folder for hgweb
        <Directory "/usr/lib64/python2.7/site-packages/mercurial/templates/static/">
                Require all granted
        </Directory>

        # Enable execution of cgi-bin scripts
        <Directory "/var/www/cgi-bin/">
                Require all granted
                Options ExecCGI FollowSymLinks MultiViews Indexes
                AddHandler cgi-script .cgi
        </Directory>
</VirtualHost>

Here the hgweb config file, nothing too weird here
[paths]
Bodycad = /mnt/hg3/Bodycad/**
Jenkinsfile = /mnt/hg3/Jenkinsfile/*
JenkinsBuildHelperScripts = /mnt/hg3/JenkinsBuildHelperScripts/*

[web]
contact = [hidden email]
description = Laboratoire Bodycad repositories
allow_archive = zip
push_ssl = False
allow_push = *
allow_pull = True
allow_read = *
encoding = cp1251
baseurl = /
staticurl = /staticHG/
verbose = True

The only weird thing is that the files are located on another server with CIFS share mount locally under /mnt/hg3 with the following options:
//FILESERVER/share/hg_repositories /mnt/hg3 cifs auto,credentials=/.win_credentials,uid=apache,gid=apache,dir_mode=0777,file_mode=0777 0 0

The only thing I see into the access log (the error log is empty) is that the other tools are trying to use hg-history get request, which is not available, they fallback to raw-file which seem to work.
[12/Sep/2017:09:36:04 -0400] "GET /Bodycad/hg-history/78ec7dcc602d/MyFolder/MyFiles.cpp HTTP/1.1" 400 1592
[12/Sep/2017:09:37:50 -0400] "GET /Bodycad/raw-file/78ec7dcc602d/MyFolder/MyFiles.cpp HTTP/1.1" 200 5483

I've never seen the file browsing over http fail (and I suspect it's ok here).  The entries in the log for a pull or clone look different, and that's the I/O that would be of interest here.

Any idea? This is painful situation that prevent us from working properly with redmine/reviewboard, the whole workflow get broken because of this.

Other than upgrading and trying to collect more diagnostics, I don't, unfortunately.  We are supposed to be setting up a ReviewBoard server (and I thought a coworker had one setup already for testing purposes, without a problem).  I'll let you know if I run into similar issues.

Maybe you can mount the cifs share read-only on the Jenkins server, so that the repos don't need to be cloned/pulled over http to it?


On Tue, Sep 5, 2017 at 9:28 AM, Jérôme Godbout <[hidden email]> wrote:
I'm trying some apache settings to change the behavior of the jenkins proxy that may be related to the problems we see. I will try to keep in touch if this change anything or solve the problems. The problems might be on the Jenkins side with the proxy.
SetEnv proxy-nokeepalive 1
Hope this fix the issue, but I'm not sure it will.

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Mon, Sep 4, 2017 at 9:01 PM, Matt Harbison <[hidden email]> wrote:
On Thu, 31 Aug 2017 10:22:59 -0400, Jérôme Godbout <[hidden email]> wrote:

My webhg server is CentOS 7 and have the following installed version of
python/mercurial:

mercurial-4.0.0.9_rc
python-2.7.5 [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

don't known if there's some compatibility issue between those version.

You may be hitting something that others have occasionally reported (and hasn't been tracked down yet, AFAIK).

https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096435.html
https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-June/099528.html

The only thing that I can suggest is to upgrade Mercurial to the latest release, to see if the "got 0 bytes, expected 4" message changes to something more useful.

FWIW, one coworker was getting these messages (and also similar decompressing errors, and SSL errors when using https), but pretty consistently.  He swapped network cards, and hasn't had a problem since.  I've got another coworker trying to pilot Mercurial who is also running into these issues (using the same model of network card), but I haven't had time to really troubleshoot yet.  But I think you said your problem is intermittent, so unfortunately, that probably won't help you.






_______________________________________________
Mercurial mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial
Reply | Threaded
Open this post in threaded view
|

Re: HGWeb/Jenkins problems

Jérôme Godbout
Sorry for the noise, but if this might help someone later down the road, look like the samba share and the cifs mount were caching the drive data, which was resulting into problems between server acknowledging what was existing into the repos and not finding part of the data.

So the solution was this:
  1. All machine mounting a CIFS drive now use cache=none into the fstab
  2. The Samba CIFS server now use oplocks = no to prevent problems
I'm not sure this fix everything, but most stuff start working like it was intended. Sorry about all the noise, the error was not intuitive nor giving any good clue about what was wrong, all the log were not telling anything either, it was not so funny to debug and understand what was wrong here. This Linux migration was not as smooth as I would have love too but we are getting there!
Thanks,
Jerome

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  +1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Wed, Sep 13, 2017 at 11:55 AM, Jérôme Godbout <[hidden email]> wrote:
This only happen with recent repos revision and for a little delais, after a while it seem to work fine. Look like something is caching the repos data somehow. Could this be CIFS share? Apache? Python mercurial use some cache?

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Wed, Sep 13, 2017 at 11:53 AM, Jérôme Godbout <[hidden email]> wrote:
If I browse the repos files I can end up on this stack trace telling me the repos file is corrupted!?! If I clone from the drive, I don't get any error like this, so I surely have something wrong going on?!? Do I have a corrupted repos that is a total mess and may explain why I have problems only with this repos. Any way to above verify to see if anything can be detected or even fixed?
 
 
<class 'mercurial.error.RevlogError'>
Python 2.7.5: /usr/bin/python
Wed Sep 13 11:48:46 2017

A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred.

 /var/www/cgi-bin/hgweb.cgi in ()
     10 from mercurial.hgweb import hgweb, wsgicgi
     11 
     12 application = hgweb(config)
=>   13 wsgicgi.launch(application)
     14 
wsgicgi = <module 'mercurial.hgweb.wsgicgi' from '/usr/lib...on2.7/site-packages/mercurial/hgweb/wsgicgi.pyc'>, wsgicgi.launch = <function launch>, application = <mercurial.hgweb.hgwebdir_mod.hgwebdir object>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/wsgicgi.py in launch(application=<mercurial.hgweb.hgwebdir_mod.hgwebdir object>)
     83     content = application(environ, start_response)
     84     try:
=>   85         for chunk in content:
     86             write(chunk)
     87         if not headers_sent:
chunk = '/Bodycad/file/1262998fd98a">browse</a></li>\n</ul...odycad/Osteotomy/Steps/Plates/PlateModel.qml</a> ', content = <generator object run_wsgi>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/hgwebdir_mod.py in run_wsgi(self=<mercurial.hgweb.hgwebdir_mod.hgwebdir object>, req=<mercurial.hgweb.request.wsgirequest object>)
    223         profile = self.ui.configbool('profiling', 'enabled')
    224         with profiling.profile(self.ui, enabled=profile):
=>  225             for r in self._runwsgi(req):
    226                 yield r
    227 
r = '/Bodycad/file/1262998fd98a">browse</a></li>\n</ul...odycad/Osteotomy/Steps/Plates/PlateModel.qml</a> ', self = <mercurial.hgweb.hgwebdir_mod.hgwebdir object>, self._runwsgi = <bound method hgwebdir._runwsgi of <mercurial.hgweb.hgwebdir_mod.hgwebdir object>>, req = <mercurial.hgweb.request.wsgirequest object>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/hgweb_mod.py in run_wsgi(self=<mercurial.hgweb.hgweb_mod.hgweb object>, req=<mercurial.hgweb.request.wsgirequest object>)
    314             profile = repo.ui.configbool('profiling', 'enabled')
    315             with profiling.profile(repo.ui, enabled=profile):
=>  316                 for r in self._runwsgi(req, repo):
    317                     yield r
    318 
r = '/Bodycad/file/1262998fd98a">browse</a></li>\n</ul...odycad/Osteotomy/Steps/Plates/PlateModel.qml</a> ', self = <mercurial.hgweb.hgweb_mod.hgweb object>, self._runwsgi = <bound method hgweb._runwsgi of <mercurial.hgweb.hgweb_mod.hgweb object>>, req = <mercurial.hgweb.request.wsgirequest object>, repo = <mercurial.localrepo.visiblefilteredrepo object>
 /usr/lib64/python2.7/site-packages/mercurial/util.py in increasingchunks(source=<generator object _flatten>, min=4096, max=65536)
    900     buf = []
    901     blen = 0
=>  902     for chunk in source:
    903         buf.append(chunk)
    904         blen += len(chunk)
chunk = '</td>\n</tr>\n<tr>\n <th class="diffstat">diffstat</th>\n <td class="diffstat">\n ', source = <generator object _flatten>
 /usr/lib64/python2.7/site-packages/mercurial/templater.py in _flatten(thing=<generator object runtemplate>)
   1109         yield pycompat.bytestr(thing)
   1110     else:
=> 1111         for i in thing:
   1112             i = templatekw.unwraphybrid(i)
   1113             if isinstance(i, bytes):
i = '</td>\n</tr>\n<tr>\n <th class="diffstat">diffstat</th>\n <td class="diffstat">\n ', thing = <generator object runtemplate>
 /usr/lib64/python2.7/site-packages/mercurial/templater.py in runtemplate(context=<mercurial.templater.engine object>, mapping={'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, template=[(<function runsymbol>, 'header'), (<function runstring>, '\n<title>'), (<function runfilter>, ((<function runsymbol>, 'repo'), <function escape>)), (<function runstring>, ': '), (<function runfilter>, ((<function runsymbol>, 'node'), <function short>)), (<function runstring>, '</title>\n</head>\n<body>\n<div class="container">\n<div class="menu">\n<div class="logo">\n<a href="'), (<function runsymbol>, 'logourl'), (<function runstring>, '">\n<img src="'), (<function runfilter>, ((<function runsymbol>, 'staticurl'), <function urlescape>)), (<function runsymbol>, 'logoimg'), (<function runstring>, '" alt="mercurial" /></a>\n</div>\n<ul>\n <li><a href="'), (<function runfilter>, ((<function runsymbol>, 'url'), <function urlescape>)), (<function runstring>, 'shortlog/'), (<function runsymbol>, 'symrev'), (<function runmap>, (<function runsymbol>, 'sessionvars', <function runtemplate>, [(<function runsymbol>, 'separator'), (<function runsymbol>, 'name'), (<function runstring>, '='), (<function runfilter>, ((...), <function urlescape>))])), (<function runstring>, '">log</a></li>\n <li><a href="'), (<function runfilter>, ((<function runsymbol>, 'url'), <function urlescape>)), (<function runstring>, 'graph/'), (<function runsymbol>, 'symrev'), (<function runmap>, (<function runsymbol>, 'sessionvars', <function runtemplate>, [(<function runsymbol>, 'separator'), (<function runsymbol>, 'name'), (<function runstring>, '='), (<function runfilter>, ((...), <function urlescape>))])), ...])
    381 def runtemplate(context, mapping, template):
    382     for func, data in template:
=>  383         yield func(context, mapping, data)
    384 
    385 def buildfilter(exp, context):
func = <function runsymbol>, context = <mercurial.templater.engine object>, mapping = {'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, data = 'diffsummary'
 /usr/lib64/python2.7/site-packages/mercurial/templater.py in runsymbol(context=<mercurial.templater.engine object>, mapping={'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, key='diffsummary', default='')
    372             v = default
    373     if callable(v):
=>  374         return v(**pycompat.strkwargs(mapping))
    375     return v
    376 
v = <function <lambda>>, global pycompat = <module 'mercurial.pycompat' from '/usr/lib64/python2.7/site-packages/mercurial/pycompat.pyc'>, pycompat.strkwargs = <function identity>, mapping = {'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/webutil.py in (**x={'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...})
    425         changesetbranch=showbranch,
    426         files=files,
=>  427         diffsummary=lambda **x: diffsummary(diffstatsgen),
    428         diffstat=diffstats,
    429         archives=web.archivelist(ctx.hex()),
global diffsummary = <function diffsummary>, x = {'archives': <generator object archivelist>, 'author': 'Jerome Godbout', 'basenode': '2778fddf3bc392886dca63b81d12dabb112ac09a', 'bookmarks': [], 'branch': [{'name': 'JG'}], 'branches': [{'name': 'JG'}], 'changesetbookmark': <generator object showbookmark>, 'changesetbranch': [{'name': 'JG'}], 'changesettag': <generator object showtag>, 'child': <function <lambda>>, ...}, diffstatsgen = <generator object diffstatgen>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/webutil.py in diffsummary(statgen=<generator object diffstatgen>)
    542     '''Return a short summary of the diff.'''
    543 
=>  544     stats, maxname, maxtotal, addtotal, removetotal, binary = next(statgen)
    545     return _(' %d files changed, %d insertions(+), %d deletions(-)\n') % (
    546              len(stats), addtotal, removetotal)
stats undefined, maxname undefined, maxtotal undefined, addtotal undefined, removetotal undefined, binary undefinedbuiltin next = <built-in function next>, statgen = <generator object diffstatgen>
 /usr/lib64/python2.7/site-packages/mercurial/hgweb/webutil.py in diffstatgen(ctx=<changectx 1262998fd98a>, basectx=<changectx 2778fddf3bc3>)
    534     '''Generator function that provides the diffstat data.'''
    535 
=>  536     stats = patch.diffstatdata(util.iterlines(ctx.diff(basectx)))
    537     maxname, maxtotal, addtotal, removetotal, binary = patch.diffstatsum(stats)
    538     while True:
stats undefinedglobal patch = <module 'mercurial.patch' from '/usr/lib64/python2.7/site-packages/mercurial/patch.pyc'>, patch.diffstatdata = <function diffstatdata>, global util = <module 'mercurial.util' from '/usr/lib64/python2.7/site-packages/mercurial/util.pyc'>, util.iterlines = <function iterlines>, ctx = <changectx 1262998fd98a>, ctx.diff = <bound method changectx.diff of <changectx 1262998fd98a>>,basectx = <changectx 2778fddf3bc3>
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in diffstatdata(lines=<generator object iterlines>)
   2712     inheader = False
   2713 
=> 2714     for line in lines:
   2715         if line.startswith('diff'):
   2716             addresult()
line undefinedlines = <generator object iterlines>
 /usr/lib64/python2.7/site-packages/mercurial/util.py in iterlines(iterator=<generator object diff>)
   2470 
   2471 def iterlines(iterator):
=> 2472     for chunk in iterator:
   2473         for line in chunk.splitlines():
   2474             yield line
chunk undefinediterator = <generator object diff>
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in diff(repo=<mercurial.localrepo.visiblefilteredrepo object>, node1=<changectx 2778fddf3bc3>, node2=<changectx 1262998fd98a>, match=None, changes=None, opts=<mercurial.mdiff.diffopts object>, losedatafn=None, prefix='', relroot='', copy=None)
   2319                                    changes=changes, opts=opts,
   2320                                    losedatafn=losedatafn, prefix=prefix,
=> 2321                                    relroot=relroot, copy=copy):
   2322         text = ''.join(sum((list(hlines) for hrange, hlines in hunks), []))
   2323         if header and (text or len(header) > 1):
relroot = '', copy = None
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in trydiff(repo=<mercurial.localrepo.visiblefilteredrepo object>, revs=['2778fddf3bc3', '1262998fd98a'], ctx1=<changectx 2778fddf3bc3>, ctx2=<changectx 1262998fd98a>, modified=['QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Producers/PlateProducer.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Producers/PlateSegmentProducer.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Ui2D/PlateContextualMenu.qml', 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/Ui2D/SegmentContextualMenu.qml', 'QmlScripts/Common/Nurbs/Examples/ExampleBlendLinkProfile.qml', 'QmlScripts/Common/Nurbs/Examples/ExampleNurbsBlendProfile.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/LinkModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/LinkProducer.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/SegmentModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/SegmentProducer.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/StructureModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/SurfaceModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/TrajectoryModel.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/TrajectoryProducer.qml', 'QmlScripts/Common/Nurbs/ProfileBlending/qmldir'], added=['QmlScripts/Common/Nurbs/ProfileBlending/LongestEdgeSurfaceIsoCurve.qml'], removed=[], copy={}, getfilectx=<function getfilectx>, opts=<mercurial.mdiff.diffopts object>, losedatafn=None, prefix='', relroot='')
   2583         flag2 = None
   2584         if f1:
=> 2585             fctx1 = getfilectx(f1, ctx1)
   2586             if opts.git or losedatafn:
   2587                 flag1 = ctx1.flags(f1)
fctx1 = None, getfilectx = <function getfilectx>, f1 = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', ctx1 = <changectx 2778fddf3bc3>
 /usr/lib64/python2.7/site-packages/mercurial/patch.py in getfilectx(f='QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', ctx=<changectx 2778fddf3bc3>)
   2349                 if len(cache) > 20:
   2350                     del cache[order.popleft()]
=> 2351                 cache[f] = fctx.filelog()
   2352             else:
   2353                 order.remove(f)
cache = {}, f = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml', fctx = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, fctx.filelog = <bound method filectx.filelog of <filectx QmlScr...eotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>>
 /usr/lib64/python2.7/site-packages/mercurial/context.py in filelog(self=<filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>)
    746         return self._flags
    747     def filelog(self):
=>  748         return self._filelog
    749     def rev(self):
    750         return self._changeid
self = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, self._filelog = <mercurial.filelog.filelog object>
 /usr/lib64/python2.7/site-packages/mercurial/util.py in __get__(self=<mercurial.util.propertycache object>, obj=<filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, type=<class 'mercurial.context.filectx'>)
    821         self.name = func.__name__
    822     def __get__(self, obj, type=None):
=>  823         result = self.func(obj)
    824         self.cachevalue(obj, result)
    825         return result
result undefinedself = <mercurial.util.propertycache object>, self.func = <function _filelog>, obj = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>
 /usr/lib64/python2.7/site-packages/mercurial/context.py in _filelog(self=<filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>)
    669     @propertycache
    670     def _filelog(self):
=>  671         return self._repo.file(self._path)
    672 
    673     @propertycache
self = <filectx QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml@2778fddf3bc3>, self._repo = <mercurial.localrepo.visiblefilteredrepo object>, self._repo.file = <bound method visiblefilteredrepo.file of <mercurial.localrepo.visiblefilteredrepo object>>, self._path = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml'
 /usr/lib64/python2.7/site-packages/mercurial/localrepo.py in file(self=<mercurial.localrepo.visiblefilteredrepo object>, f='QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml')
    989         if f[0] == '/':
    990             f = f[1:]
=>  991         return filelog.filelog(self.svfs, f)
    992 
    993     def changectx(self, changeid):
global filelog = <module 'mercurial.filelog' from '/usr/lib64/python2.7/site-packages/mercurial/filelog.pyc'>, filelog.filelog = <class 'mercurial.filelog.filelog'>, self = <mercurial.localrepo.visiblefilteredrepo object>, self.svfs = <mercurial.store._fncachevfs object>, f = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml'
 /usr/lib64/python2.7/site-packages/mercurial/filelog.py in __init__(self=<mercurial.filelog.filelog object>, opener=<mercurial.store._fncachevfs object>, path='QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml')
     43     def __init__(self, opener, path):
     44         super(filelog, self).__init__(opener,
=>   45                         "/".join(("data", path + ".i")))
     46 
     47     def read(self, node):
path = 'QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml'
 /usr/lib64/python2.7/site-packages/mercurial/revlog.py in __init__(self=<mercurial.filelog.filelog object>, opener=<mercurial.store._fncachevfs object>, indexfile='data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i', datafile=None, checkambig=False)
    377             d = self._io.parseindex(indexdata, self._inline)
    378         except (ValueError, IndexError):
=>  379             raise RevlogError(_("index %s is corrupted") % (self.indexfile))
    380         self.index, nodemap, self._chunkcache = d
    381         if nodemap is not None:
global RevlogError = <class 'mercurial.error.RevlogError'>global _ = <function gettext>, self = <mercurial.filelog.filelog object>, self.indexfile = 'data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i'

<class 'mercurial.error.RevlogError'>: index data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i is corrupted 
      args = ('index data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i is corrupted',) 
      hint = None 
      message = 'index data/QmlScripts/Bodycad/Osteotomy/Steps/Plates/PlateModel.qml.i is corrupted'


bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Wed, Sep 13, 2017 at 10:21 AM, Jérôme Godbout <[hidden email]> wrote:
I did update all mercurial on all build machine (Jenkins/ReviewBoard/HgWeb) with python pip 3.4.1. Sadly this did not changed the problems at all :-( 
Since I don't have a full control over the command line Jenkins used (only it's .hgrc config like setup) how can I add the traceback you are looking for?

pull = --traceback --debug

into which section [section] I should add this or something like this I guess???

here's the full polling jenkins log message:

Started on Sep 13, 2017 10:15:00 AM
[CAD_CPP_ContinuousBuild@script] $ hg pull --rev VS2015
pulling from http://HGSERVER/Jenkinsfile
no changes found
[CAD_CPP_ContinuousBuild@script] $ hg log --rev VS2015 --template {node}
[CAD_CPP_ContinuousBuild@script] $ hg log --rev VS2015 --template {rev}
no polling baseline in c:\Jenkins\workspace\CAD_CPP\CAD_CPP_ContinuousBuild@tmp\Jenkinsfile on JGMachine
[CAD_CPP_ContinuousBuild] $ hg pull --rev default
pulling from http://HGSERVER/Bodycad/
searching for changes adding changesets adding manifests adding file changes transaction abort! rollback completed abort: stream ended unexpectedly (got 0 bytes, expected 4) [CAD_CPP_ContinuousBuild] $ hg log --rev default --template {node} [CAD_CPP_ContinuousBuild] $ hg log --rev default --template {rev} Done. Took 11 sec No changes

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Tue, Sep 12, 2017 at 11:29 PM, Matt Harbison <[hidden email]> wrote:
(Back to the list)

On Sep 12, 2017, at 10:04 AM, Jérôme Godbout <[hidden email]> wrote:

I have the lastest package for CentOS7 available 4.0.0.9_rc, maybe I should go back to 3.X ?

I would uninstall this, and use pip to install the latest.  Unless you know that some version of 3.x works, in which case, please report that back.  I'd be curious what the first version is where this issue is noticeable.

I still have intermittent problems with the hgweb (all other machine work without issue), but polling from Jenkins still give the same error every day:
hg pull --rev default
pulling from http://HGSERVER/Bodycad/
searching for changes
adding changesets
adding manifests
transaction abort!
rollback completed
abort: stream ended unexpectedly (got 0 bytes, expected 4)

The abort message may be different with newer versions.  That info would be good to know.  Also, adding --traceback to the command will print a stack where the failure occurs.

Redmine also work for a few hour and after the repository page return:
500
An error occurred when trying to access the repository: hg exited with non-zero status: 255

I can browse and I can access the code without issue, checkout work for most other repos without a problems, hg verify don't see any issue. Note, the repos have some sub-repos, which can be large, is that something that could cause that much problems? Is this a timeout problems? I have 0 idea what is wrong

Yeah, the problem seems to appear when using the wire protocol.  Checkout, verify and browsing don't touch this path.  Subrepos shouldn't matter for this.

Here's the apache config (I don't even have the security put yet)
<VirtualHost *:80>
        ServerName HGSERVER
        RewriteEngine on
        ProxyRequests off
        ProxyPreserveHost on
        AllowEncodedSlashes NoDecode
        ErrorLog "/var/log/httpd/vhost-mercurial-error.log"
        CustomLog "/var/log/httpd/vhost-mercurial-access.log" common

        LimitRequestLine 32760

        ################
        # hgweb

        RewriteRule                     ^/staticHG/(.*) "/usr/lib64/python2.7/site-packages/mercurial/templates/static/$1" [L]

        RewriteRule                     ^/(?!git|static)(.*)          "/var/www/cgi-bin/hgweb.cgi/$1" [L]
        ProxyPassReverse        /                       "/var/www/cgi-bin/hgweb.cgi/"

        # Enable access to static folder for hgweb
        <Directory "/usr/lib64/python2.7/site-packages/mercurial/templates/static/">
                Require all granted
        </Directory>

        # Enable execution of cgi-bin scripts
        <Directory "/var/www/cgi-bin/">
                Require all granted
                Options ExecCGI FollowSymLinks MultiViews Indexes
                AddHandler cgi-script .cgi
        </Directory>
</VirtualHost>

Here the hgweb config file, nothing too weird here
[paths]
Bodycad = /mnt/hg3/Bodycad/**
Jenkinsfile = /mnt/hg3/Jenkinsfile/*
JenkinsBuildHelperScripts = /mnt/hg3/JenkinsBuildHelperScripts/*

[web]
contact = [hidden email]
description = Laboratoire Bodycad repositories
allow_archive = zip
push_ssl = False
allow_push = *
allow_pull = True
allow_read = *
encoding = cp1251
baseurl = /
staticurl = /staticHG/
verbose = True

The only weird thing is that the files are located on another server with CIFS share mount locally under /mnt/hg3 with the following options:
//FILESERVER/share/hg_repositories /mnt/hg3 cifs auto,credentials=/.win_credentials,uid=apache,gid=apache,dir_mode=0777,file_mode=0777 0 0

The only thing I see into the access log (the error log is empty) is that the other tools are trying to use hg-history get request, which is not available, they fallback to raw-file which seem to work.
[12/Sep/2017:09:36:04 -0400] "GET /Bodycad/hg-history/78ec7dcc602d/MyFolder/MyFiles.cpp HTTP/1.1" 400 1592
[12/Sep/2017:09:37:50 -0400] "GET /Bodycad/raw-file/78ec7dcc602d/MyFolder/MyFiles.cpp HTTP/1.1" 200 5483

I've never seen the file browsing over http fail (and I suspect it's ok here).  The entries in the log for a pull or clone look different, and that's the I/O that would be of interest here.

Any idea? This is painful situation that prevent us from working properly with redmine/reviewboard, the whole workflow get broken because of this.

Other than upgrading and trying to collect more diagnostics, I don't, unfortunately.  We are supposed to be setting up a ReviewBoard server (and I thought a coworker had one setup already for testing purposes, without a problem).  I'll let you know if I run into similar issues.

Maybe you can mount the cifs share read-only on the Jenkins server, so that the repos don't need to be cloned/pulled over http to it?


On Tue, Sep 5, 2017 at 9:28 AM, Jérôme Godbout <[hidden email]> wrote:
I'm trying some apache settings to change the behavior of the jenkins proxy that may be related to the problems we see. I will try to keep in touch if this change anything or solve the problems. The problems might be on the Jenkins side with the proxy.
SetEnv proxy-nokeepalive 1
Hope this fix the issue, but I'm not sure it will.

bodycad
Jerome Godbout
Software Developer
2035 rue du Haut-Bord, Québec, QC, Canada. G1N 4R7
T:  <a href="tel:(418)%20527-1388" value="+14185271388" target="_blank">+1 418 527-1388
E: [hidden email]
www.bodycad.com

The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure. If you are not the intended recipient of this message or their agent, or if this message has been addressed to you in error, please immediately alert the sender by reply email and then delete this message and any attachments. If you are not the intended recipient, you are hereby notified that any use, dissemination, copying, or storage of this message or its attachments is strictly prohibited. 

Le contenu de ce message et les pièces jointes sont destinés uniquement pour le(s) destinataire(s) et peuvent contenir des informations confidentielles et / ou privilégiées qui peuvent être protégées légalement contre toute divulgation. Si vous n'êtes pas le destinataire de ce message ou son agent, ou si ce message vous a été adressé par erreur, s’il vous plaît avertir immédiatement l'expéditeur par courriel de réponse, puis supprimer ce message et les pièces jointes. Si vous n'êtes pas le destinataire prévu, vous êtes par la présente informé que toute utilisation, diffusion, copie, ou stockage de ce message ou de ses pièces jointes est strictement interdit.


On Mon, Sep 4, 2017 at 9:01 PM, Matt Harbison <[hidden email]> wrote:
On Thu, 31 Aug 2017 10:22:59 -0400, Jérôme Godbout <[hidden email]> wrote:

My webhg server is CentOS 7 and have the following installed version of
python/mercurial:

mercurial-4.0.0.9_rc
python-2.7.5 [GCC 4.8.5 20150623 (Red Hat 4.8.5-11)]

don't known if there's some compatibility issue between those version.

You may be hitting something that others have occasionally reported (and hasn't been tracked down yet, AFAIK).

https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-April/096435.html
https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-June/099528.html

The only thing that I can suggest is to upgrade Mercurial to the latest release, to see if the "got 0 bytes, expected 4" message changes to something more useful.

FWIW, one coworker was getting these messages (and also similar decompressing errors, and SSL errors when using https), but pretty consistently.  He swapped network cards, and hasn't had a problem since.  I've got another coworker trying to pilot Mercurial who is also running into these issues (using the same model of network card), but I haven't had time to really troubleshoot yet.  But I think you said your problem is intermittent, so unfortunately, that probably won't help you.







_______________________________________________
Mercurial mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial
Reply | Threaded
Open this post in threaded view
|

Re: HGWeb/Jenkins problems

Matt Harbison-2
In reply to this post by Jérôme Godbout
On Wed, 13 Sep 2017 10:21:29 -0400, Jérôme Godbout <[hidden email]>  
wrote:

> I did update all mercurial on all build machine  
> (Jenkins/ReviewBoard/HgWeb)
> with python pip 3.4.1. Sadly this did not changed the problems at all :-(
> Since I don't have a full control over the command line Jenkins used  
> (only
> it's .hgrc config like setup) how can I add the traceback you are looking
> for?
>
> pull = --traceback --debug
>
> into which section [section] I should add this or something like this I
> guess???

See `hg help config.ui.traceback`
_______________________________________________
Mercurial mailing list
[hidden email]
https://www.mercurial-scm.org/mailman/listinfo/mercurial