[fossil-users] commit crashes after http redirection

[Ashwin Hirschi]

Hello everyone,

recently, my team started using Fossil. It took me a while to move our
existing code revisions to Fossil repositories. But it was worth it:
everyone's very pleased with what Fossil offers. So, I'm really glad we
made the jump.

Unfortunately, it looks we've also run into strange crashes related to
(HTTP) redirection. Since many team members work from home, their IP
addresses jump around a lot. To help people find each other, we've set up
a simple redirection service.

In this system, each team member has an associated, fixed URL. And
whenever a HTTP GET or POST request on such a URL occurs, the service
redirects the client to the actual (registered) IP address of the
corresponding member.

Strangely enough, this works well *most* of the time. Fossil picks up the
302 redirect, reports it and goes about its business. Regular "fossil
sync" commands seem to work fine, as do other commands that deal with
remote repositories.

But oddly, 9 out of 10 "fossil commit" commands will fail. The application
crashes always seem to occur when Fossil tries to push the changes to the
remote repository. Sometimes the commited changes were still transmitted,
sometimes not...

As I'm currently unable to build and/or debug Fossil, I've taken up
reading and analysing the Fossil source code to try and find the cause.
But I'm hoping others may have run into similar issues or are perhaps able
to reproduce our problems?

By the way, we're all using the official version 1.29 build and are
running in auto-sync mode.

Ashwin.

[bch]

Certainly a crash is rarely the correct behaviour.

Is there a chance that the dynamic endpoint is switched-out part way
through a transfer, and *that* is the cause for the crash ?

-bch

Post by Ashwin Hirschi
Hello everyone,
recently, my team started using Fossil. It took me a while to move our
everyone's very pleased with what Fossil offers. So, I'm really glad we
made the jump.
Unfortunately, it looks we've also run into strange crashes related to
(HTTP) redirection. Since many team members work from home, their IP
addresses jump around a lot. To help people find each other, we've set up
a simple redirection service.
In this system, each team member has an associated, fixed URL. And
whenever a HTTP GET or POST request on such a URL occurs, the service
redirects the client to the actual (registered) IP address of the
corresponding member.
Strangely enough, this works well *most* of the time. Fossil picks up the
302 redirect, reports it and goes about its business. Regular "fossil
sync" commands seem to work fine, as do other commands that deal with
remote repositories.
But oddly, 9 out of 10 "fossil commit" commands will fail. The application
crashes always seem to occur when Fossil tries to push the changes to the
remote repository. Sometimes the commited changes were still transmitted,
sometimes not...
As I'm currently unable to build and/or debug Fossil, I've taken up
reading and analysing the Fossil source code to try and find the cause.
But I'm hoping others may have run into similar issues or are perhaps able
to reproduce our problems?
By the way, we're all using the official version 1.29 build and are
running in auto-sync mode.
Ashwin.
_______________________________________________
fossil-users mailing list
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

[Richard Hipp]

Post by bch
Certainly a crash is rarely the correct behaviour.
Is there a chance that the dynamic endpoint is switched-out part way
through a transfer, and *that* is the cause for the crash ?

Even then, it shouldn't crash.

Are you trying to build on Unix/Mac or on Windows? Did you follow the
instructions at https://www.fossil-scm.org/fossil/doc/tip/www/build.wiki ?

[bch]

Post by Richard Hipp

Post by bch
Certainly a crash is rarely the correct behaviour.
Is there a chance that the dynamic endpoint is switched-out part way
through a transfer, and *that* is the cause for the crash ?

Even then, it shouldn't crash.

Agreed. I was mostly trying to introduce idea that the 30x redirect
itself might not be blame. Andy has request for core files, etc. which
if present, may be enlightening...

Post by Richard Hipp
Are you trying to build on Unix/Mac or on Windows? Did you follow the
instructions at https://www.fossil-scm.org/fossil/doc/tip/www/build.wiki ?
--
D. Richard Hipp

[Ashwin Hirschi]

Post by Richard Hipp
Are you trying to build on Unix/Mac or on Windows? Did you follow the
instructions at https://www.fossil-scm.org/fossil/doc/tip/www/build.wiki ?

I'm unable to build or debug Fossil because I've just switched to a new
(Windows) machine. We're still in the process of putting things in place.
So, at the moment, only the tools related to my regular day-to-day work
are up & running.

In other words, for now I'm stuck at browsing the Fossil source code and
hoping maybe someone on the list is able to reproduce the problem.

Ashwin.

[Andy Bradford]

In other words, for now I'm stuck at browsing the Fossil source code
and hoping maybe someone on the list is able to reproduce the problem.

I've used Fossil with redirected sites before so I may be able to look
at this later, not sure how far I'll get.

I believe Fossil does have a maximum redirection limit so it should
automatically handle the case where you redirect from your central site
to the remote IP, and then the remote IP decides to redirect elsewhere.

Thanks,

Andy
--
TAI64 timestamp: 40000000548a414b

[Alek Paunov]

Hi Ashwin,

Post by Ashwin Hirschi

Post by Richard Hipp
Are you trying to build on Unix/Mac or on Windows? Did you follow the
instructions at
https://www.fossil-scm.org/fossil/doc/tip/www/build.wiki ?

I'm unable to build or debug Fossil because I've just switched to a new
(Windows) machine. We're still in the process of putting things in
place. So, at the moment, only the tools related to my regular
day-to-day work are up & running.
In other words, for now I'm stuck at browsing the Fossil source code and
hoping maybe someone on the list is able to reproduce the problem.

Few hours ago, Andy committed a patch for you to test, but you say above
that you lack C dev environment at this machine.

If this is still a obstacle for you, we may try to build a Virtual Box
VM with Linux and mingw ready for producing windows binaries, and place
the image somewhere for download.

Actually, Jan Nijtmans probably have something already prepared for such
cases.

Kind regards,
Alek

[Ashwin Hirschi]

Post by Alek Paunov
Few hours ago, Andy committed a patch for you to test, but you say above
that you lack C dev environment at this machine.

It's good to hear Andy may have found the cause of the crashes and has
created a fix.

I intend to have the required tool chain installed on my new machine
sometime next week.

Trying out a fresh Fossil build will then be the first thing I'll do!

Ashwin.

[Richard Hipp]

Post by Ashwin Hirschi
I intend to have the required tool chain installed on my new machine
sometime next week.

Installing Msys+Mingw+awk.exe+Tcl takes less than an hour (modulo corporate
computer-lockdown nonsense). And its all free. And I think only the first
two elements are required to just do a build.

[Andy Bradford]

Post by Ashwin Hirschi
It's good to hear Andy may have found the cause of the crashes and has
created a fix.

One detail that I failed to ask... what OS is this on? You mentioned
having a Windows install to deal with, but I wasn't certain if that was
where you were actually seeing the redirect errors, or if that was just
a distraction that you had to deal with before you could get more
details.

Thanks,

Andy

[Ashwin Hirschi]

Post by Andy Bradford

Post by Ashwin Hirschi
It's good to hear Andy may have found the cause of the crashes and has
created a fix.

One detail that I failed to ask... what OS is this on?

The crashes happened on a Windows 8.1 machine.

Ashwin.

[Ashwin Hirschi]

Post by bch
Certainly a crash is rarely the correct behaviour.

Indeed [;-)].

Post by bch
Is there a chance that the dynamic endpoint is switched-out part way
through a transfer, and *that* is the cause for the crash ?

Good point, but... no, in all cases the end-points where still valid &
available afterwards. And if it turned out the commited changes were not
pushed to the remote repository, I could always recover by doing a "fossil
sync".

Ashwin.

[Andy Bradford]

Post by Ashwin Hirschi
Good point, but... no, in all cases the end-points where still valid &
available afterwards. And if it turned out the commited changes were
not pushed to the remote repository, I could always recover by doing a
"fossil sync".

But we want to avoid this because effectively you aren't able to take
advantage of autosync being enabled in this scenario. Will you provide a
core file output?

At least the minimum use:

gdb fossil /path/to/fossil.core

Then type: bt

Thanks,

Andy
--
TAI64 timestamp: 40000000548a40d1

[Andy Bradford]

Post by Ashwin Hirschi
recently, my team started using Fossil. It took me a while to move our
everyone's very pleased with what Fossil offers. So, I'm really glad
we made the jump.

Welcome and thanks for giving Fossil a try!

Post by Ashwin Hirschi
In this system, each team member has an associated, fixed URL. And
whenever a HTTP GET or POST request on such a URL occurs, the service
redirects the client to the actual (registered) IP address of the
corresponding member.

Let me see if I understand this correctly... You have a bunch of team
members, each hosting their own fossil repositories on their own unique
(and sometimes changing) IP addresses. You have a redirection server
in a central location that knows about these unique IPs and will
dynamically update when the team member's IP address changes. Other team
members will clone a given team member's fossil via the redirection
service instead of the unique and changing IP address. So in essence,
you've made a central location for where your Fossil users will clone
to, but where the clones reside is distributed and dynamically handled
by the redirection service?

Definitely an interesting setup.

You may have already considered the opposite: What if instead of the
redirection, you just provided a Fossil hosting location? Each user
could have their own project hosted in their own directory and Fossil
could serve all of them from there? Then they would clone like:

fossil clone http://redirection.server/user/project project

Of course each user would probably need their own account on
redirection.server to maintain their files for their account.

Post by Ashwin Hirschi
But oddly, 9 out of 10 "fossil commit" commands will fail. The
application crashes always seem to occur when Fossil tries to push the
changes to the remote repository. Sometimes the commited changes were
still transmitted, sometimes not...

Any chance you can share the error that you're getting? Is it actually
crashing (e.g. segfault) or is it just erroring out with an error? I
suppose the latter question will be answered if you post an example.

Thanks,

Andy
--
TAI64 timestamp: 40000000548a2fe4

[Ashwin Hirschi]

Post by Andy Bradford
Welcome and thanks for giving Fossil a try!

Thanks & my pleasure.

Although our experience with Fossil is still limited, we're all finding
Fossil very easy to like! [:-)]

Post by Andy Bradford

Post by Ashwin Hirschi
In this system, each team member has an associated, fixed URL. And
whenever a HTTP GET or POST request on such a URL occurs, the service
redirects the client to the actual (registered) IP address of the
corresponding member.

Let me see if I understand this correctly... You have a bunch of team
members, each hosting their own fossil repositories on their own unique
(and sometimes changing) IP addresses. You have a redirection server
in a central location that knows about these unique IPs and will
dynamically update when the team member's IP address changes.

Yes.

Post by Andy Bradford
Other team members will clone a given team member's fossil via the
redirection service instead of the unique and changing IP address.So
in essence, you've made a central location for where your Fossilusers
will clone to, but where the clones reside is distributed and
dynamically handled by the redirection service?

Precisely.

So, the redirection service does nothing but help people reach other. If a
POST or GET reaches the service, the only thing it really does is redirect
the client (here: Fossil) to the actual IP address of the intended person.

Post by Andy Bradford
Definitely an interesting setup.

Thanks [;-)].

Post by Andy Bradford
You may have already considered the opposite: What if instead of the
redirection, you just provided a Fossil hosting location? Each user
could have their own project hosted in their own directory and Fossil
fossil clone http://redirection.server/user/project project
Of course each user would probably need their own account on
redirection.server to maintain their files for their account.

Exactly. We wanted to circumvent that. In a way, different people are "in
charge" of different parts of the software. So, I guess in that respect it
also felt more natural to distribute the "master repositories" (in a
manner of speaking).

Post by Andy Bradford

Post by Ashwin Hirschi
But oddly, 9 out of 10 "fossil commit" commands will fail. The
application crashes always seem to occur when Fossil tries to push the
changes to the remote repository. Sometimes the commited changes were
still transmitted, sometimes not...

Any chance you can share the error that you're getting? Is it actually
crashing (e.g. segfault) or is it just erroring out with an error? I
suppose the latter question will be answered if you post an example.

Unfortunately, it's segfaulting. So I have no error messages whatsoever I
can share with people...

Ashwin.

[Andy Bradford]

Post by Ashwin Hirschi
Unfortunately, it looks we've also run into strange crashes related to
(HTTP) redirection. Since many team members work from home, their IP
addresses jump around a lot. To help people find each other, we've set
up a simple redirection service.

I just committed a change that may address your issue. I haven't been
able to reproduce the exact problem that you described (possibly because
I don't have the redirect setup exactly like you did), but while
investigating the code, I did find a problem which may be the cause.

Basically, on a redirect Fossil did not completely reinitialize the
address to which it was connecting. For a host where the redirect is
simply to a new URI on the same host, this is not a problem, but for a
redirection service as you have setup, I'm not sure how it ever worked
because the address is going to be different for each redirect.

If you get a chance, please try:

http://www.fossil-scm.org/index.html/info/6e7cb7f27a190702593a2a54cbe5340453a13d74

Or if you don't yet have Fossil cloned:

http://www.fossil-scm.org/index.html/zip/fossil-6e7cb7f27a.zip?uuid=6e7cb7f27a190702593a2a54cbe5340453a13d74

If this doesn't take care of the problem, more details about how the
redirection is configured would be helpful, e.g. what kind of Redirect
rules do you have in place? Apache or some other setup?

Or even better would be some debug info from the core file if you can
get it.

Thanks,

Andy

[Ashwin Hirschi]

Post by Andy Bradford
I just committed a change that may address your issue. I haven't been
able to reproduce the exact problem that you described (possibly because
I don't have the redirect setup exactly like you did), but while
investigating the code, I did find a problem which may be the cause.
Basically, on a redirect Fossil did not completely reinitialize the
address to which it was connecting. For a host where the redirect is
simply to a new URI on the same host, this is not a problem, but for a
redirection service as you have setup, I'm not sure how it ever worked
because the address is going to be different for each redirect.

I managed to build Fossil with the patches that were submitted last week
(version [522cf5f66d]).

Unfortunately, those changes don't prevent commits from crashing Fossil...
(though, obviously, they may help solve part of the problem!)

Running a Fossil commit from gdb on my Windows 8.1 machine results in:

Program received signal SIGSEGV, Segmentation fault.
0x005159f9 in BIO_ctrl ()

Strangely enough, only a single frame is being reported in the backtrace...

(gdb) bt
#0 0x005159f9 in BIO_ctrl ()
(gdb) down
Bottom (innermost) frame selected; you cannot go down.
(gdb) up
Initial frame selected; you cannot go up.
(gdb)

No other frames are available. Apparently, I'm not creating the debug
build correctly? I must admit I'm also wholly unfamiliar with gdb...

As far as my debug build goes: I enabled the FOSSIL_ENABLE_SYMBOLS define
in win/Makefile.mingw, performed a make clean (plus: make clean-zlib &
make clean-openssl) and then simply ran make again. The executable size
jumped from 3.5 Mb to nearly 6.5 Mb. So, that looked okay... What must I
do to get a more comprehensive backtrace?

The good news is that it looks like we have a new clue as to what might be
wrong. Our redirection service not only translates (fixed) member URLs to
their varying IP addresses, but also switches from HTTP to HTTPS.

So, for the full auto-sync commit scenario something like this happens:

1. pull from remote repository
1a. fossil tries http://service/member/etc
1b. fossil picks up redirection and uses https://member-ip/etc { <=
switching to HTTPS! }
2. commit to local repository
3. push to remote repository
3a. fossil tries http://service/member/etc { <= back to HTTP }
3b. fossil picks up redirection and uses https://member-ip/etc { <=
HTTPS, again }

I strongly suspect the crash happens during 3b, possibly because of
lingering SSL issues from 1b.

This would also explain why Andy could not reproduce our exact problem. My
apologies for not mentioning HTTPS earlier, I did not think it played a
part during my initial post.

In any case, does this help track down the problem?

Ashwin.

[Andy Bradford]

This would also explain why Andy could not reproduce our exact
problem. My apologies for not mentioning HTTPS earlier, I did not
think it played a part during my initial post.
In any case, does this help track down the problem?

Yes, it certainly does make a difference. I can take a look at the
switching from HTTP to HTTPS scenario you mentioned later on (unless
someone beats me to it), though I suspect you're right that something is
left lingering from a previous SSL setup.

Also, I'm not very familiar with building it on Windows so someone else
might have to assist in that regard, but what you have provided is at
least sufficient to look in the right place.

Thanks,

Andy
--
TAI64 timestamp: 40000000548f772f

[Ashwin Hirschi]

I thought I'd take a quick look at the Fossil source code and how it deals
with redirects. Though I'm an utter newbie on both Fossil and its
internals, there are several things that strike me as odd.

For instance, it looks wrong that redirects are parsed (around line 342 in
http.c) *before* the transport is closed (a few lines after). I mean, the
*new* URL data is now passed on to the close routines, while I'd say the
old parameters should be used here. In other words, should the transport
close calls not come *before* the url-parsing?

Also, in http_ssl.c I cannot help but notice that ssl_close checks for
iBio to not be NULL, even though it's never initialised at the top (near
line 46). My suggestion here would be to always initialise to NULL *and*
also make sure that the ssl_close function assigns NULL to iBio after the
BIO_reset & BIO_free_all calls.

Combine the 2 points above, and I'd say that the "late" transport closing
(i.e. after the new URL data is in place), triggers an incorrect call to
ssl_close (because the URL is *now* an HTTPS one!), which in turn is
tricked into closing the SSL system even though it was never initialised
before[?!].

I haven't tried anything out, yet. But I thought I'd throw this on the
list so see what you guys think. So, am I reading this right?

Ashwin.

[Richard Hipp]

These seem like reasonable suggestions so I added them.

But I'm not able to recreate the problem. As a test, I created a redirect
CGI program at http://www.cvstrac.org/redirect that redirects to
https://www.fossil-scm.org/. (Notice the HTTP to HTTPS transition.) So if
you do

fossil clone http://www.cvstrac.org/redirect x1.fossil

It should clone fossil. And that appears to work, both before and after
the modifications above.

Post by Ashwin Hirschi
I thought I'd take a quick look at the Fossil source code and how it deals
with redirects. Though I'm an utter newbie on both Fossil and its
internals, there are several things that strike me as odd.
For instance, it looks wrong that redirects are parsed (around line 342 in
http.c) *before* the transport is closed (a few lines after). I mean, the
*new* URL data is now passed on to the close routines, while I'd say the
old parameters should be used here. In other words, should the transport
close calls not come *before* the url-parsing?
Also, in http_ssl.c I cannot help but notice that ssl_close checks for
iBio to not be NULL, even though it's never initialised at the top (near
line 46). My suggestion here would be to always initialise to NULL *and*
also make sure that the ssl_close function assigns NULL to iBio after the
BIO_reset & BIO_free_all calls.
Combine the 2 points above, and I'd say that the "late" transport closing
(i.e. after the new URL data is in place), triggers an incorrect call to
ssl_close (because the URL is *now* an HTTPS one!), which in turn is
tricked into closing the SSL system even though it was never initialised
before[?!].
I haven't tried anything out, yet. But I thought I'd throw this on the
list so see what you guys think. So, am I reading this right?
Ashwin.
_______________________________________________
fossil-users mailing list
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

[Andy Bradford]

It should clone fossil. And that appears to work, both before and
after the modifications above.

I was able to reproduce the problem (which only happened with autosync;
triggered by commit), and can confirm that your changes (as suggested by
Ashwin) have corrected it in my environment:

$ fossil ci -m test
Autosync: http://***@remote/oldfossil/oldfossil.cgi/new
redirect to https://remote/fossil/fossil.cgi/new
Round-trips: 1 Artifacts sent: 0 received: 0
Pull finished with 747 bytes sent, 3143 bytes received
New_Version: 080129766ef6a2bf92d7db3468df5b2975169355
Autosync: http://***@remote/oldfossil/oldfossil.cgi/new
redirect to https://remote/fossil/fossil.cgi/new
Segmentation fault (core dumped)
$ gdb fossil fossil.core
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-unknown-openbsd5.4"...
Core was generated by `fossil'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/lib/libssl.so.19.0...done.
Loaded symbols for /usr/lib/libssl.so.19.0
Reading symbols from /usr/lib/libcrypto.so.22.0...done.
Loaded symbols for /usr/lib/libcrypto.so.22.0
Reading symbols from /usr/lib/libz.so.4.1...done.
Loaded symbols for /usr/lib/libz.so.4.1
Reading symbols from /usr/lib/libc.so.69.0...done.
Loaded symbols for /usr/lib/libc.so.69.0
Reading symbols from /usr/libexec/ld.so...done.
Loaded symbols for /usr/libexec/ld.so
#0 SSL_shutdown (s=0x898fda00)
at /usr/src/lib/libssl/ssl/../src/ssl/ssl_lib.c:1011
1011 if (s->handshake_func == 0)
(gdb) bt
#0 SSL_shutdown (s=0x898fda00)
at /usr/src/lib/libssl/ssl/../src/ssl/ssl_lib.c:1011
#1 0x0bd5b665 in ssl_ctrl (b=0x83b13c00, cmd=1, num=0, ptr=0x0)
at /usr/src/lib/libssl/ssl/../src/ssl/bio_ssl.c:310
#2 0x0dc2b72a in BIO_ctrl (b=0x83b13c00, cmd=1, larg=0, parg=0x0)
at /usr/src/lib/libssl/crypto/../src/crypto/bio/bio_lib.c:370
#3 0x1c03575f in ssl_close () at http_ssl.c:174
#4 0x1c036a9c in transport_close (pUrlData=0x3c090b54) at http_transport.c:203
#5 0x1c035014 in http_exchange (pSend=0xcfbbfee8, pReply=0xcfbbfed0,
useLogin=1, maxRedirect=19) at http.c:346
#6 0x1c086854 in client_sync (syncFlags=3, configRcvMask=0, configSendMask=0)
at xfer.c:1590
#7 0x1c068fe4 in autosync (flags=3) at sync.c:76
#8 0x1c06907b in autosync_loop (flags=3, nTries=1) at sync.c:87
#9 0x1c015a9c in commit_cmd () at checkin.c:2008
#10 0x1c045b0c in main (argc=4, argv=0xcfbc02a8) at main.c:760

And here it is with the changes:

$ /tmp/fossil ci -m httpimproved
Autosync: http://***@remote/oldfossil/oldfossil.cgi/new
redirect to https://remote/fossil/fossil.cgi/new
Round-trips: 1 Artifacts sent: 0 received: 0
Pull finished with 743 bytes sent, 811 bytes received
New_Version: 2db78e625f39fec00548de4e0a03d7beac672be8
Autosync: http://***@remote/oldfossil/oldfossil.cgi/new
redirect to https://remote/fossil/fossil.cgi/new
Round-trips: 1 Artifacts sent: 2 received: 0
Sync finished with 1241 bytes sent, 860 bytes received

Andy

[Ashwin Hirschi]

Post by Andy Bradford
I was able to reproduce the problem (which only happened with autosync;
triggered by commit), and can confirm that your changes (as suggested by

I ran several tests here as well. It looks like the "redirected commits"
now work fine. Great!

I'm pleased we managed to fix this one quickly. Many thanks to everyone
involved!

Ashwin.