[fossil-users] cgi on Mac problem

[j. van den hoff]

hi list,

disclaimer: I have no experience whatsoever with apache, cgi, and
webservers in general. so maybe I'm missing something stupid...

I'm trying for the first time to setup a fossil repo for http access via
the cgi route on macosx 10.8.4. if someone has done this for this OS maybe
he can give me some help. I've tried to follow the howto step by step but
failed, getting the dreaded

"Internal Server Error
The server encountered an internal error or misconfiguration and was
unable to complete your request."

message from apache. what I did so far (apart from activating apache):

1.
putting a file `repo.cgi' with this content:

#!{my-path-to-fossil-executable}
repository: {my-path-to-fslrepo}

into /Library/WebServer/CGI-Executables and made it executable for
`everybody'.

2.
giving read/write access to everybody for the respective repo (is this
really required?).

3.
putting a test-file `first.cgi' with this content:

#!/usr/bin/perl -wT
print "Content-type: text/html\n\n";
print "<h2>Hello, World!</h2>\n";

in the same location as `repo.cgi' and giving it the same permissions.

4.
verified that local access to the repo works just fine, including
web-access via the stand-alone webserver (so the repo is not corrupt,
closed, or anything like that)

5.
what I see:

-- access to `http:{mymachine}/cgi-bin/first.cgi' works just fine (I do
get the `hello world' page)

-- access to `http:{mymachine}/cgi-bin/repo.cgi' gives the `internal
server error' in the browser and the message "Premature end of script
headers: repo.cgi" in the apache error log

which, I understand, can have many reasons (e.g. wrong permissions) but
boils down to the html headers not being generated as expected.


any help in these matters would be appreciated.


j.

[Mark Janssen]

On Sun, Sep 29, 2013 at 4:59 PM, j. van den hoff

Post by j. van den hoff
hi list,
<snip>
-- access to `http:{mymachine}/cgi-bin/**first.cgi' works just fine (I
do get the `hello world' page)
-- access to `http:{mymachine}/cgi-bin/**repo.cgi' gives the `internal
server error' in the browser and the message "Premature end of script
headers: repo.cgi" in the apache error log
which, I understand, can have many reasons (e.g. wrong permissions) but
boils down to the html headers not being generated as expected.
any help in these matters would be appreciated.

Trying `fossil cgi /Library/WebServer/CGI-**Executables/repo.cgi` should
give a more descriptive error.

Mark

[j. van den hoff]

thanks for responding. just discovered that actually it were the
permissions of the `fossil' executable which where insufficient (not
executable for the rest of the world...). but I tried your recommendation
anyway and get the output

Status: 501 Not Implemented
X-Frame-Options: SAMEORIGIN
Cache-control: no-cache
Content-Type: text/html; charset=utf-8
Content-Length: 66

which I presume is normal (despite me not making sense of the first two
lines ...), right?

j.

Post by Mark Janssen
On Sun, Sep 29, 2013 at 4:59 PM, j. van den hoff

Post by j. van den hoff
hi list,
<snip>
-- access to `http:{mymachine}/cgi-bin/**first.cgi' works just fine (I
do get the `hello world' page)
-- access to `http:{mymachine}/cgi-bin/**repo.cgi' gives the `internal
server error' in the browser and the message "Premature end of script
headers: repo.cgi" in the apache error log
which, I understand, can have many reasons (e.g. wrong permissions) but
boils down to the html headers not being generated as expected.
any help in these matters would be appreciated.

Trying `fossil cgi /Library/WebServer/CGI-**Executables/repo.cgi` should
give a more descriptive error.
Mark

[Stephan Beal]

On Sun, Sep 29, 2013 at 5:28 PM, j. van den hoff

Post by j. van den hoff
thanks for responding. just discovered that actually it were the
permissions of the `fossil' executable which where insufficient (not
executable for the rest of the world...). but I tried your recommendation
anyway and get the output
Status: 501 Not Implemented

That one is new to me, but luckily it only appears one time in the sources:

/*
** Send a reply indicating that the HTTP request was malformed
*/
static NORETURN void malformed_request(const char *zMsg){
cgi_set_status(501, "Not Implemented");
cgi_printf(
"<html><body><p>Bad Request: %s</p></body></html>\n", zMsg
);
cgi_reply();
fossil_exit(0);
}

Post by j. van den hoff
X-Frame-Options: SAMEORIGIN
Cache-control: no-cache
Content-Type: text/html; charset=utf-8
Content-Length: 66
which I presume is normal (despite me not making sense of the first two
lines ...), right?

What request are you sending it? What are the 66 bytes after the header?
That will show us at what point (in the C sources) it's failing.

Regarding the docs: the current docs elude (vaguely) to the permissions for
that setup, but i'll improve them a bit in a few minutes.

[j. van den hoff]

I tried the command recommended by marc jansen in his reply to my original
question:

fossil cgi /Library/WebServer/CGI-**Executables/repo.cgi

which yielded ( I unintentionally skipped the last line in my previous
post):

8<-------------------------------------------------------
Status: 501 Not Implemented
X-Frame-Options: SAMEORIGIN
Cache-control: no-cache
Content-Type: text/html; charset=utf-8
Content-Length: 66

<html><body><p>Bad Request: missing REQUEST_URI</p></body></html>
8<-------------------------------------------------------

any more information I can provide?

j.

Post by Stephan Beal
On Sun, Sep 29, 2013 at 5:28 PM, j. van den hoff

Post by j. van den hoff
thanks for responding. just discovered that actually it were the
permissions of the `fossil' executable which where insufficient (not
executable for the rest of the world...). but I tried your
recommendation
anyway and get the output
Status: 501 Not Implemented

/*
** Send a reply indicating that the HTTP request was malformed
*/
static NORETURN void malformed_request(const char *zMsg){
cgi_set_status(501, "Not Implemented");
cgi_printf(
"<html><body><p>Bad Request: %s</p></body></html>\n", zMsg
);
cgi_reply();
fossil_exit(0);
}

Post by j. van den hoff
X-Frame-Options: SAMEORIGIN
Cache-control: no-cache
Content-Type: text/html; charset=utf-8
Content-Length: 66
which I presume is normal (despite me not making sense of the first two
lines ...), right?

What request are you sending it? What are the 66 bytes after the header?
That will show us at what point (in the C sources) it's failing.
Regarding the docs: the current docs elude (vaguely) to the permissions for
that setup, but i'll improve them a bit in a few minutes.

[Stephan Beal]

On Sun, Sep 29, 2013 at 5:45 PM, j. van den hoff

Post by j. van den hoff
<html><body><p>Bad Request: missing REQUEST_URI</p></body></html>

That's an indication that Apache is missing something. The env vars
REQUEST_URI and SCRIPT_NAME, both specified by CGI, are required in the
environment:

http://www.cgi101.com/book/ch3/text.html

(i mention SCRIPT_NAME b/c the check for those two vars is in the same
place in the source code)

i've never seen that happen on an Apache-based config, and i have never set
up Fossil as a CGI under any different web server :/. i.e. i'm confused and
don't have any suggestions at the moment.

[j. van den hoff]

Post by Stephan Beal
On Sun, Sep 29, 2013 at 5:45 PM, j. van den hoff

Post by j. van den hoff
<html><body><p>Bad Request: missing REQUEST_URI</p></body></html>

That's an indication that Apache is missing something. The env vars
REQUEST_URI and SCRIPT_NAME, both specified by CGI, are required in the
http://www.cgi101.com/book/ch3/text.html
(i mention SCRIPT_NAME b/c the check for those two vars is in the same
place in the source code)
i've never seen that happen on an Apache-based config, and i have never set
up Fossil as a CGI under any different web server :/. i.e. i'm confused and
don't have any suggestions at the moment.

and I have no clue in these matters anyway. but it seems to work
nevertheless. one remaining issue is the following:

I've put the cgi-script in the standard location on the mac with is a
directory only modifiable by root. now, if I access the server, most
things work just fine (logging in as different users, looking at timeline
and diffs etc.) but what does _not_ work is to download a tarball or
zip-file, where the attempt to do this leads to the message:

The requested URL
/cgi-bin/repo.cgi/tarball/reponame-0253aa7c17e75b5a.tar.gz was not found
on this server.

from this I guess that `fossil' tries ot create a subdir "tarball" within
the cgi-script directory (or does it?) which it is not able to do (since
only writable by root, not by the cgi-user). if this is correct: what is
the canonical solution to the problem? is it really necessary (and
unproblematic) to make the cgi-script dir world writable?

j.

[Richard Hipp]

The requested URL /cgi-bin/repo.cgi/tarball/**
reponame-0253aa7c17e75b5a.tar.**gz was not found on this server.
from this I guess that `fossil' tries ot create a subdir "tarball" within
the cgi-script directory (or does it?) which it is not able to do (since
only writable by root, not by the cgi-user). if this is correct: what is
the canonical solution to the problem? is it really necessary (and
unproblematic) to make the cgi-script dir world writable?

No, Fossil generates the tarball on-the-fly and transmits it back over the
wire without every writing to disk. I'm not sure what the problem might be.

Are you sure you have the "Download ZIP" capability (which also covers
tarballs) turned on?

[j. van den hoff]

On Mon, Sep 30, 2013 at 11:06 AM, j. van den hoff

The requested URL /cgi-bin/repo.cgi/tarball/**
reponame-0253aa7c17e75b5a.tar.**gz was not found on this server.
from this I guess that `fossil' tries ot create a subdir "tarball" within
the cgi-script directory (or does it?) which it is not able to do (since
only writable by root, not by the cgi-user). if this is correct: what is
the canonical solution to the problem? is it really necessary (and
unproblematic) to make the cgi-script dir world writable?

No, Fossil generates the tarball on-the-fly and transmits it back over the
wire without every writing to disk. I'm not sure what the problem might be.

I see.

Are you sure you have the "Download ZIP" capability (which also covers
tarballs) turned on?

at least I did not intentionally switch it off... what I actually did, for
testing purposes, is to set the permissions/capabilities as follows for
anonymous and nobody:

anonymous: hmncz
nobody: j

but the reported error occurs when I'm logged in as the Setup/Super-user
(capabilities: s). I presume `s' implies everything else? but if not so:
the download fails also if I explicitly add `z' for the setup user.

if I can provide any more information please let me know.

j.

[Stephan Beal]

On Mon, Sep 30, 2013 at 5:25 PM, j. van den hoff

Post by j. van den hoff
anonymous: hmncz
nobody: j
but the reported error occurs when I'm logged in as the Setup/Super-user
the download fails also if I explicitly add `z' for the setup user.

"s"etup is kind of a "s"uperuser, but the permissions are not additive
there (at least, not IIRC). Setup users have a few rights which admins do
not, e.g. editing the site's configuration info. Admins can add users, but
cannot (e.g.) change the access rights of a user with Setup permissions. In
my experience the owner of a repo, and perhaps one or two others, have s+a
permissions, then perhaps a few others have admin permissions, so that they
can set up users and do a few of the mundane admin tasks, but they cannot
modify or lock out the Setup user(s).

If you'll grep the sources for "perm.Setup" and "perm.Admin" the difference
should become a bit clearer. It is possible to have Setup permission
without Admin permissions, in which case you can set up the basic
configuration but will be denied access to some other bits. e.g. an Admin
can shun but a Setup user cannot (not sure why, but that's what the code
tells me).

Post by j. van den hoff
if I can provide any more information please let me know.

i'm still at a loss.

[j. van den hoff]

I think I have found the reason for the problem: it is caused by the
presence of a `/' in the project name. specifically in my case, the error
message was:

8<--------
The requested URL
/cgi-bin/repo.cgi/zip/PET/MR+Bug+List-0253aa7c17e75b5a.zip was not found
on this server.
8<--------

and it's the `/' in `PET/MR' which obviously is interpreted as part of the
path, since it is not quoted. if I change the project name by replacing
PET/MR --> PET-MR I can download the tarball/zip file without problem.

so I think either fossil should quote any occurence of `/' when
constructing the URL (if possible -- I'm not sure if it is) or silently
replace any `/' by something else (as it already does with any blanks
which are replaced by `+'). a `-' actually might be the best choice for
replacing the `/'.

does this make sense?

j.

On Mon, Sep 30, 2013 at 11:06 AM, j. van den hoff

The requested URL /cgi-bin/repo.cgi/tarball/**
reponame-0253aa7c17e75b5a.tar.**gz was not found on this server.
from this I guess that `fossil' tries ot create a subdir "tarball" within
the cgi-script directory (or does it?) which it is not able to do (since
only writable by root, not by the cgi-user). if this is correct: what is
the canonical solution to the problem? is it really necessary (and
unproblematic) to make the cgi-script dir world writable?

No, Fossil generates the tarball on-the-fly and transmits it back over the
wire without every writing to disk. I'm not sure what the problem might be.
Are you sure you have the "Download ZIP" capability (which also covers
tarballs) turned on?

[Stephan Beal]

On Mon, Sep 30, 2013 at 9:07 PM, j. van den hoff

Post by j. van den hoff
so I think either fossil should quote any occurence of `/' when
constructing the URL (if possible -- I'm not sure if it is) or silently
replace any `/' by something else (as it already does with any blanks which
are replaced by `+'). a `-' actually might be the best choice for replacing
the `/'.

My personal preference would be to restrict the project name field to only
characters which are legal in Fossil filenames (note that fossil disallows
some which are legal in Unix but not on Windows, like ":").

[j. van den hoff]

Post by Stephan Beal
On Mon, Sep 30, 2013 at 9:07 PM, j. van den hoff

Post by j. van den hoff
so I think either fossil should quote any occurence of `/' when
constructing the URL (if possible -- I'm not sure if it is) or silently
replace any `/' by something else (as it already does with any blanks which
are replaced by `+'). a `-' actually might be the best choice for replacing
the `/'.

My personal preference would be to restrict the project name field to only
characters which are legal in Fossil filenames (note that fossil disallows
some which are legal in Unix but not on Windows, like ":").

yes, that would be possible. but still: at least blanks _are_ allowed in
file names, but not in the URL and are therefore replaced by `+' and I
think this is the more reasonable approach. I would argue that it's
preferable to allow free text for the project name (including blank,
slash, colon etc.) and just map all these to something valid (maybe all of
them to `+' just like it is already done for blanks) when the "tarball
URL" is constructed: after all, the project name is just the descriptive
title at the top of the web-page and it feels not right to enforce that
this title is also a "valid file name".

[Stephan Beal]

On Mon, Sep 30, 2013 at 9:57 PM, j. van den hoff

after all, the project name is just the descriptive title at the top of the
web-page and it feels not right to enforce that this title is also a "valid
file name".

If i'm not mistaken, the project's name is used as the top-most directory
name in the tar/zip file, which means you might not be able to unpack the
such a zip/tar if it contains ':' or '/' in it. Encoding it for the URL is
not a problem (in principal, at least), though there might be a corner case
or three waiting for us there.

Way back in the old days, the zip files (fossil didn't have tar back then)
unpacked everything into the current directory. At some point we added an
"artificial" subdir based off of the project's name and the version UUID.
At the time it was assumed (my fault!) that project names would be "simple"
in nature, but that was (in hindsight a poor assumption).

In baseline_zip_cmd() the code can be found which replaces space with
underscore, but that's just a very basic form of escaping. To handle /, :,
\, *, and probably a few other potentially problematic ones, we'd need to
expand that (and the corresponding tar counterpart) a bit (but it's almost
bedtime here, so i'm not going to touch it tonight ;).

Ah, here's a workaround which might work: pass a name=abcdef option and it
will use that as the base name, but if it's longer than 10 bytes then it
truncates it to 10 bytes (see zip.c:baseline_zip_page()). i haven't checked
the corresponding tar code to see if it has a similar feature.

[j. van den hoff]

Post by Stephan Beal
On Mon, Sep 30, 2013 at 9:57 PM, j. van den hoff

after all, the project name is just the descriptive title at the top of the
web-page and it feels not right to enforce that this title is also a "valid
file name".

If i'm not mistaken, the project's name is used as the top-most directory
name in the tar/zip file, which means you might not be able to unpack the
such a zip/tar if it contains ':' or '/' in it. Encoding it for the URL is
not a problem (in principal, at least), though there might be a corner case
or three waiting for us there.
Way back in the old days, the zip files (fossil didn't have tar back then)
unpacked everything into the current directory. At some point we added an
"artificial" subdir based off of the project's name and the version UUID.
At the time it was assumed (my fault!) that project names would be "simple"
in nature, but that was (in hindsight a poor assumption).
In baseline_zip_cmd() the code can be found which replaces space with
underscore, but that's just a very basic form of escaping. To handle /, :,
\, *, and probably a few other potentially problematic ones, we'd need to
expand that (and the corresponding tar counterpart) a bit (but it's almost
bedtime here, so i'm not going to touch it tonight ;).
Ah, here's a workaround which might work: pass a name=abcdef option and it
will use that as the base name, but if it's longer than 10 bytes then it
truncates it to 10 bytes (see zip.c:baseline_zip_page()). i haven't checked
the corresponding tar code to see if it has a similar feature.

anything which works will be fine I'd say ;-).
ideally, though, using the project name (after replacing all potentially
offending characters by `+' or `_' or similar) plus sha1 would seem the
best naming scheme in my view.

[j. van den hoff]

coming back to this problem of "problematic" character such as `/' causing
tar/zip download to fail since the
URL generated by fossil is invalid: would it not be sensible to simply use
as name the sha1 hash itself plus the suitable extension,
e.g something like

ae9ca63258d41bda7b7ca968b65ddecaef1a742b.zip (or, preferably,
abbreviated to 10 chars of the hash as in the timeline)?

this seems to have 3 advantages:

1) no interference anymore with user's choice of project name and valid
file name on all platforms
2) file name still sufficiently "unique" to avoid accidental names clashes
on the local disk
3) it should be the path of least resistance regarding fixing it in the
source code

renaming of the unpacked archive directory to something more recognizable
(if so desired) might be left as an exercise to the user I'd say ...


j.

Post by Stephan Beal
On Mon, Sep 30, 2013 at 9:57 PM, j. van den hoff

after all, the project name is just the descriptive title at the top of the
web-page and it feels not right to enforce that this title is also a "valid
file name".

If i'm not mistaken, the project's name is used as the top-most directory
name in the tar/zip file, which means you might not be able to unpack the
such a zip/tar if it contains ':' or '/' in it. Encoding it for the URL is
not a problem (in principal, at least), though there might be a corner case
or three waiting for us there.
Way back in the old days, the zip files (fossil didn't have tar back then)
unpacked everything into the current directory. At some point we added an
"artificial" subdir based off of the project's name and the version UUID.
At the time it was assumed (my fault!) that project names would be "simple"
in nature, but that was (in hindsight a poor assumption).
In baseline_zip_cmd() the code can be found which replaces space with
underscore, but that's just a very basic form of escaping. To handle /, :,
\, *, and probably a few other potentially problematic ones, we'd need to
expand that (and the corresponding tar counterpart) a bit (but it's almost
bedtime here, so i'm not going to touch it tonight ;).
Ah, here's a workaround which might work: pass a name=abcdef option and it
will use that as the base name, but if it's longer than 10 bytes then it
truncates it to 10 bytes (see zip.c:baseline_zip_page()). i haven't checked
the corresponding tar code to see if it has a similar feature.

[Stephan Beal]

On Tue, Oct 8, 2013 at 2:31 PM, j. van den hoff

Post by j. van den hoff
coming back to this problem of "problematic" character such as `/' causing
tar/zip download to fail since the
URL generated by fossil is invalid: would it not be sensible to simply use
as name the sha1 hash itself plus the suitable extension,
e.g something like
ae9ca63258d41bda7b7ca968b65dde**caef1a742b.zip (or, preferably,
abbreviated to 10 chars of the hash as in the timeline)?

That does indeed sound like the most bullet-proof mechanism, though perhaps
not the most user-friendly. Once downloaded, the user is going to
practically be forced to rename it.

Post by j. van den hoff
1) no interference anymore with user's choice of project name and valid
file name on all platforms
2) file name still sufficiently "unique" to avoid accidental names clashes
on the local disk

3) it should be the path of least resistance regarding fixing it in the

Post by j. van den hoff
source code

Agreed on all 3 points, but it sure would be ugly. :/

Post by j. van den hoff
renaming of the unpacked archive directory to something more recognizable
(if so desired) might be left as an exercise to the user I'd say ...

The vast majority of project names unpack just fine currently. This
proposal (while arguably better at a technical level) increases the amount
of work for everyone who _doesn't_ use special characters in their project
names. That's my biggest (possibly only) argument against it. It also might
complicate scripting because one now no longer has some common string to
use as a wildcard basis (except for *.zip/*.tar, which may be too generic
for some scripting purposes).

i'd like to see this combined with David G.'s hash-to-name idea, so that
the Zip files get named like ORANGE BATTERY HORSE ;).

[j. van den hoff]

Post by Stephan Beal
On Tue, Oct 8, 2013 at 2:31 PM, j. van den hoff

Post by j. van den hoff
coming back to this problem of "problematic" character such as `/' causing
tar/zip download to fail since the
URL generated by fossil is invalid: would it not be sensible to simply use
as name the sha1 hash itself plus the suitable extension,
e.g something like
ae9ca63258d41bda7b7ca968b65dde**caef1a742b.zip (or, preferably,
abbreviated to 10 chars of the hash as in the timeline)?

That does indeed sound like the most bullet-proof mechanism, though perhaps
not the most user-friendly. Once downloaded, the user is going to
practically be forced to rename it.

I would say: as he is currently, too, in general. I suspect for most users
it will seem natural to choose a descriptive free text project name and
then the zipfile name ends up as something like (this is a real world
example...):

sd+--+a+drop-in+replacement+for+`cd'-ae9ca63258d41bda.zip

which is not especially inviting to keep this as the final name of the
unpacked archive dir...

Post by Stephan Beal

Post by j. van den hoff
1) no interference anymore with user's choice of project name and valid
file name on all platforms
2) file name still sufficiently "unique" to avoid accidental names clashes
on the local disk

3) it should be the path of least resistance regarding fixing it in the

Post by j. van den hoff
source code

Agreed on all 3 points, but it sure would be ugly. :/

see above: I don't think it is _that_ beautiful right now. ;-)

Post by Stephan Beal

Post by j. van den hoff
renaming of the unpacked archive directory to something more
recognizable
(if so desired) might be left as an exercise to the user I'd say ...

The vast majority of project names unpack just fine currently. This

I don't have any statistics for that (but it might be true). nevertheless
it is _very_ easy to stumble over this right now and it seems fossil would
benefit from fixing this issue.

and also see my example above: that one does unpack just fine but I'm not
happy with that name either and sure would change it soon.

Post by Stephan Beal
proposal (while arguably better at a technical level) increases the amount
of work for everyone who _doesn't_ use special characters in their project
names. That's my biggest (possibly only) argument against it. It also might
complicate scripting because one now no longer has some common string to
use as a wildcard basis (except for *.zip/*.tar, which may be too generic
for some scripting purposes).

not sure whether that is a widespread demand (handle multiple downloaded
zipfiles via scripting). but if it is, one could prepend some fixed
string, e.g. use a name like

fossil-ae9ca63258d41bda.zip

that should at least handle all within-project activities related to
handling multiple zip files from the same timeline.

what do others think in this regards?

Post by Stephan Beal
i'd like to see this combined with David G.'s hash-to-name idea, so that
the Zip files get named like ORANGE BATTERY HORSE ;).

well ...

[Richard Hipp]

On Tue, Oct 8, 2013 at 9:15 AM, j. van den hoff

I suspect for most users it will seem natural to choose a descriptive
free text project name and then the zipfile name ends up as something like
sd+--+a+drop-in+replacement+**for+`cd'-ae9ca63258d41bda.zip
which is not especially inviting to keep this as the final name of the
unpacked archive dir...

I checked in a change that encodes things in the admittedly ugly way. It
is ugly but it does work, pending a final resolution of the problem.

I further observed that having "/" characters in the project name was never
a problem for Firefox, on which I do most of my testing. Which browser was
the issue originally observed on?

[Stephan Beal]

Post by Richard Hipp

I suspect for most users it will seem natural to choose a descriptive
free text project name and then the zipfile name ends up as something like
sd+--+a+drop-in+replacement+**for+`cd'-ae9ca63258d41bda.zip

OTOH, this had not come up until recently (not that i can recall, anyway,
but my memory is certainly not an authoritative information source!).

which is not especially inviting to keep this as the final name of the

Post by Richard Hipp

unpacked archive dir...

I further observed that having "/" characters in the project name was
never a problem for Firefox, on which I do most of my testing. Which
browser was the issue originally observed on?

i think it's more the unpacking which would be the problem in that case,
since we'd be creating a synthetic ZIP directory entry which contains a
slash in it. (Speculation - haven't tried!)

[Richard Hipp]

Trunk now escapes just a handful of characters that might cause problems
(\/*?[]). And a comment has been added to the entry box for the "Project
Name" encouraging admins to avoid the use of those special characters.

[j. van den hoff]

Post by Richard Hipp
Trunk now escapes just a handful of characters that might cause problems
(\/*?[]). And a comment has been added to the entry box for the "Project
Name" encouraging admins to avoid the use of those special characters.

thanks for fixing this (mostly/completely?).

but I still believe that constructing the name for the zip file (and top
level dir in the archive)
from the (essentially free text) project name is not optimal. of course
one can take care to avoid "problematic characters" but one should not
need to.

e.g., I also use fossil for tracking changes to latex-documents
(manuscripts of journal articles) where project title typically looks like
"a paper addressing something or other". there can be
single/double/back-quotes in this name, parentheses, or whatever, and
several of these characters can cause problems or at least annoyances if
later used in the shell/command line (even if a valid file name results
there might be chars special to the shell). and quoting all these still
makes for difficult to read/handle file names.

I view the download facility mainly as a convenience for "anonymous" to
get hold of the managed files for a certain revision without the need
to clone the whole repo. at least in this use case it sure would suffice
to use a fixed_string+sha1 as the identifier (e.g. `fossil' plus the
sha1). actually I feel it would be easier for the user. so what are the
benefits of using project-name derived (and usually unwieldy) long names
for the generated archives?

[Richard Hipp]

On Tue, Oct 8, 2013 at 10:02 AM, j. van den hoff

Post by j. van den hoff
I view the download facility mainly as a convenience for "anonymous" to
get hold of the managed files for a certain revision without the need
to clone the whole repo. at least in this use case it sure would suffice
to use a fixed_string+sha1 as the identifier (e.g. `fossil' plus the sha1).
actually I feel it would be easier for the user. so what are the benefits
of using project-name derived (and usually unwieldy) long names for the
generated archives?

I think the point is that the "Project Name" is suppose to be that short
prefix, not a long description of the project. So for your repo, the
"Project Name" should be "veedeehjay-papers" or "articles" or "papers",
not "Journal Articles (LaTeX) Collected by J. Van Den Hoff".

[j. van den hoff]

Post by Richard Hipp
On Tue, Oct 8, 2013 at 10:02 AM, j. van den hoff

Post by j. van den hoff
I view the download facility mainly as a convenience for "anonymous" to
get hold of the managed files for a certain revision without the need
to clone the whole repo. at least in this use case it sure would suffice
to use a fixed_string+sha1 as the identifier (e.g. `fossil' plus the sha1).
actually I feel it would be easier for the user. so what are the benefits
of using project-name derived (and usually unwieldy) long names for the
generated archives?

I think the point is that the "Project Name" is suppose to be that short
prefix, not a long description of the project. So for your repo, the

that obviously was/is your attitude here, that I have understood.

Post by Richard Hipp
"Project Name" should be "veedeehjay-papers" or "articles" or "papers",
not "Journal Articles (LaTeX) Collected by J. Van Den Hoff".

i'm not collecting them, but writing them (one by one, each in a different
repo). so the project name might be the running header (short title) of
the paper, e.g., which makes perfect sense to me. but anyway.

I can adjust to short (and necessarily no longer really descriptive)
project names if need be. but it just seems that it would be desirable to
(have the ability to) get a main header in the fossil-generated webpages
(that's where the project name ends up after all) which is just that: a
reasonable header/title whose content is dictated by the considerations
one usually applies when creating a title for a document (article, thesis,
book, whatever), namely of being reasonable concise and descriptive and
_not_ by the demand "avoid all the following chars since otherwise you get
silly names when downloading (and long names all the time if you use a
'real' project name instead of an artificially shortened one)"

seemingly I have a different preference here than you ...

[Richard Hipp]

The latest trunk has (yet another) configuration option that allows you to
specify the name prefix for generated tarballs and ZIP archives. If the
"Abbreviated Name" is omitted, the Project Name is used.

[j. van den hoff]

Post by Richard Hipp
The latest trunk has (yet another) configuration option that allows you to
specify the name prefix for generated tarballs and ZIP archives. If the
"Abbreviated Name" is omitted, the Project Name is used.

that sounds like a good solution. thanks.

[j. van den hoff]

Post by Richard Hipp
The latest trunk has (yet another) configuration option that allows you to
specify the name prefix for generated tarballs and ZIP archives. If the
"Abbreviated Name" is omitted, the Project Name is used.

a minor thing, but still -- I would propose the following changes w.r.t.
this fix:

1. move the new `abbreviated name' into the same frame and immediately
below the `project name' (or at least mv the `abbreviated name' above
`project description').
2. call it something different (e.g. `archive name prefix')
3. change the explanation for this field to something like "specify a
name prefix for generated tarballs and zip files. If left empty the
project name is used."

I believe this would help to make the user aware that project name and
abbreviated name are both potentially influencing the tarball name prefix.
more important, the current explanation is a bit misleading since there is
of course no need to make the prefix an abbreviation of the project name.
one might very well decide to use

a) a static string (e.g. "fossil_archive")
b) the basename of the fossil repo (as proposed by ron wilson -- this
could be a sensible default for this field I presume)
c) the name of the respective application/document contained in the repo

[Ron Wilson]

Post by Richard Hipp
I think the point is that the "Project Name" is suppose to be that short
prefix, not a long description of the project. So for your repo, the
"Project Name" should be "veedeehjay-papers" or "articles" or "papers",
not "Journal Articles (LaTeX) Collected by J. Van Den Hoff".

My team often gets documents with file names like that from the
requirements team.

What about URL encoding? Richard's example would yield:
"Journal+Articles+%28LaTeX%29+Collected+by+J.+Van+Den+Hoff"

Admittedly still ugly, but URL encoding has been around for many years.

[Jan Nijtmans]

Post by Ron Wilson
"Journal+Articles+%28LaTeX%29+Collected+by+J.+Van+Den+Hoff"
Admittedly still ugly, but URL encoding has been around for many years.

That wouldn't help on Windows, where characters like
" * : < > ? | \
cannot be used in filenames ('[' and ']' are OK, B.T.W.).

Richard, could you please add '>', '<' and '|' to the replacement
set (or allow me to add them .....) ?

Regards,
Jan Nijtmans

[Ron Wilson]

Post by Jan Nijtmans

Post by Ron Wilson
"Journal+Articles+%28LaTeX%29+Collected+by+J.+Van+Den+Hoff"

That wouldn't help on Windows, where characters like
" * : < > ? | \
cannot be used in filenames ('[' and ']' are OK, B.T.W.).

Hmm. The * didn't get encoded, but the others did:
Journal+Articles%2FStories%3F+%3CLaTeX%3E+by+%22J*+Van+Den+Hoff%22

Maybe an error in the implementation of the URL encoder at
http://www.url-encode-decode.com/

[j. van den hoff]

Post by Richard Hipp
On Tue, Oct 8, 2013 at 9:15 AM, j. van den hoff

I suspect for most users it will seem natural to choose a descriptive
free text project name and then the zipfile name ends up as something like
sd+--+a+drop-in+replacement+**for+`cd'-ae9ca63258d41bda.zip
which is not especially inviting to keep this as the final name of the
unpacked archive dir...

I checked in a change that encodes things in the admittedly ugly way. It
is ugly but it does work, pending a final resolution of the problem.
I further observed that having "/" characters in the project name was never
a problem for Firefox, on which I do most of my testing. Which browser was
the issue originally observed on?

opera (but I think I've tried with chrome, too). and the problem occurred
when testing access to a "cgi-repo" on a web-server (as explained in my
original post). I've just tried to reproduce the problem locally (when
using `fossil ui'): in this case I see something different, namely that
everything in the project name up to and including the (in this case
single) `/' is omitted from the generated zipfile name, e.g.,

"something/or-other" becomes "or-other-0253aa7c17e75b5a.zip"

(which works, of course, but still is not preferable over a name just
consisting of the sha1 has in my view)

but no problem related to "invalid URL" as previously with the "cgi-repo".



if need be I can of course test it with the "cgi-repo" again. just let me
know

[Doug Currie]

coming back to this problem of "problematic" character such as `/' causing tar/zip download to fail since the
URL generated by fossil is invalid: would it not be sensible to simply use as name the sha1 hash itself plus the suitable extension,

My repos integrate with packaging/build systems (some not under my control) that would break if the archive name was changed by fossil. Accessing URLs like

http://lua.sqlite.org/index.cgi/zip/lsqlite3_fsl09b.zip?uuid=fsl_9b

shouldn't add a SHA to the archive name.

e

[Ron Wilson]

Post by Doug Currie
My repos integrate with packaging/build systems (some not under my
control) that would break if the archive name was changed by fossil.
Accessing URLs like
http://lua.sqlite.org/index.cgi/zip/lsqlite3_fsl09b.zip?uuid=fsl_9b
shouldn't add a SHA to the archive name.

We are talking about the generation of a name for a Zip/tar file when a
user selects the option to download a Zip/tar archive from the repository.

Now it occurs to me that we already have a valid file name to use as a base
for the Zip/tar file: The base name of the .fossil file. That file gets
created and named before the project name is set, so would be a reasonable
candidate for the base name of the generated Zip/tar file.

[j. van den hoff]

Post by Ron Wilson

Post by Doug Currie
My repos integrate with packaging/build systems (some not under my
control) that would break if the archive name was changed by fossil.
Accessing URLs like
http://lua.sqlite.org/index.cgi/zip/lsqlite3_fsl09b.zip?uuid=fsl_9b
shouldn't add a SHA to the archive name.

We are talking about the generation of a name for a Zip/tar file when a
user selects the option to download a Zip/tar archive from the
repository.
Now it occurs to me that we already have a valid file name to use as a base
for the Zip/tar file: The base name of the .fossil file. That file gets
created and named before the project name is set, so would be a reasonable
candidate for the base name of the generated Zip/tar file.

I believe the recent fix (to make the prefix user-settable) goes in the
right direction (and is really sufficient as far as I'm concerned), so
what you propose might be another option here: a) define explicit prefix,
b) select "use basename of repo", c) use project name. let the developers
decide ...

[Stephan Beal]

On Sun, Sep 29, 2013 at 4:59 PM, j. van den hoff

Post by j. van den hoff
"Internal Server Error
The server encountered an internal error or misconfiguration and was
unable to complete your request."

Unfortunately, there are about 32032.20 giga-zillion things which can cause
that. The first place to look is in your Apache error log file. Where that
file is, is system-specific and Apple does everything differently so i
can't even guess where they've put it (or whether they've hidden it away
from mortal users).

Post by j. van den hoff
1.
...2.
giving read/write access to everybody for the respective repo (is this
really required?).

It depends on the setup. For CGIs, this will normally be either the www
user (sometimes called "httpd", "apache", or "www") or the user who's
account owns the CGI script. For testing purposes, just give the repo file
AND its directly permissions 0666 and 0777, respectively.

Post by j. van den hoff
3.
#!/usr/bin/perl -wT
print "Content-type: text/html\n\n";
print "<h2>Hello, World!</h2>\n";

That header is not HTTP-conformant. You're missing the result code. See:

http://www.jmarshall.com/easy/http/#responseline


-- access to `http:{mymachine}/cgi-bin/**repo.cgi' gives the `internal

Post by j. van den hoff
server error' in the browser and the message "Premature end of script
headers: repo.cgi" in the apache error log

In my experience, that normally happens when the output contains extra
garbage, e.g. debug output, or an assertion has failed.

Post by j. van den hoff
which, I understand, can have many reasons (e.g. wrong permissions) but
boils down to the html headers not being generated as expected.
any help in these matters would be appreciated.

The first place to check is ALWAYS the Apache error log - oftentimes lots
of useful info (or even just small hints) ends up there.

Things to check which come to mind:

- is the WHOLE path (ALL (sub)directories) leading to the CGI file readable
for the www user? i think the +x bit is not necessary on the dirs here
because +x for a dir means (basically) "ls" permissions, which you don't
need if you know the name of the file (as you do here).

- Same thing for the Fossil binary: the WHOLE path leading it up to it must
be readable.

- Use wget or curl to fetch the page and send the output to the console or
a temp file. Those two tools use different options for sending the output
to a file and i always forget which one uses which, so see --help first.

[Stephan Beal]

Post by Stephan Beal
http://www.jmarshall.com/easy/http/#responseline

Another tip, taken from that page:

try sending the request with telnet and see what comes back:

telnet localhost PORT_NUMBER

GET /timeline HTTP/1.0

Assuming Mac even has telnet (it's rarely used much nowadays... normally as
a debugging tool for network connections of other types).

[Mark Janssen]

Post by Stephan Beal

Post by Stephan Beal
http://www.jmarshall.com/easy/http/#responseline

telnet localhost PORT_NUMBER
GET /timeline HTTP/1.0
Assuming Mac even has telnet (it's rarely used much nowadays... normally
as a debugging tool for network connections of other types).

You could also use something like:

echo "GET /home\n\n" | sudo su - www-data -c "fossil http
/path/to/repository"

to get more info.

[j. van den hoff]

On Sun, 29 Sep 2013 17:19:09 +0200, Stephan Beal <***@googlemail.com>
wrote:

....

Post by Stephan Beal
- Same thing for the Fossil binary: the WHOLE path leading it up to it must
be readable.

....

that was it (and I discovered it myself some 100 ms before your mail
arrived -- not enough time to send the "SOLVED" mail ;-)).
I _was_ afraid it would be something stupid...

I'll try to keep and remember the other trouble-shooting tips. thanks a
lot.

j.

[Stephan Beal]

On Sun, Sep 29, 2013 at 5:25 PM, j. van den hoff

Post by j. van den hoff
that was it (and I discovered it myself some 100 ms before your mail
arrived -- not enough time to send the "SOLVED" mail ;-)).
I _was_ afraid it would be something stupid...
I'll try to keep and remember the other trouble-shooting tips. thanks a
lot.

That's not a stupid problem at all - it bites me almost every time i set up
a new machine :/. But i've seen it often enough to include that in my list
of things to check first when it doesn't work ;).

[j. van den hoff]

Post by Stephan Beal
On Sun, Sep 29, 2013 at 5:25 PM, j. van den hoff

Post by j. van den hoff
that was it (and I discovered it myself some 100 ms before your mail
arrived -- not enough time to send the "SOLVED" mail ;-)).
I _was_ afraid it would be something stupid...
I'll try to keep and remember the other trouble-shooting tips. thanks a
lot.

That's not a stupid problem at all - it bites me almost every time i set

good to know ;-)

Post by Stephan Beal
up
a new machine :/. But i've seen it often enough to include that in my list
of things to check first when it doesn't work ;).

maybe one could add a few of these things to the fossil howto? should many
hit new users, too, I guess.

[Stephan Beal]

On Sun, Sep 29, 2013 at 5:30 PM, j. van den hoff

Post by j. van den hoff
maybe one could add a few of these things to the fossil howto? should many
hit new users, too, I guess.

Added:

http://fossil-scm.org/index.html/doc/tip/www/server.wiki#cgi

[j. van den hoff]

Post by Stephan Beal
On Sun, Sep 29, 2013 at 5:30 PM, j. van den hoff

Post by j. van den hoff
maybe one could add a few of these things to the fossil howto? should many
hit new users, too, I guess.

http://fossil-scm.org/index.html/doc/tip/www/server.wiki#cgi

very good. such a "check list" (maybe suitably augmented in due time
according to future problems becoming apparent on the mailing list) will
save many people from waisting some hours of their lives (like I did
today...), I'm sure.

j.