RFC: letting modules phone home to check for new releases

[Derek Wright]

(if you're impatient and won't read this message ;) please comment here:
http://drupal.org/node/94154 -- "module page could say if new
releases are available").

one of the many benefits of the new release system is that contrib
maintainers can now make an official release when they think their
users should upgrade. this is particularly important for security
fixes.

my goal is for the admin/build/modules page to tell you if you're
running out-of-date code. project.module just has to provide an RSS
feed for new release nodes and/or an XML-RPC interface so that
something on the modules page can either display the aggregated feed
items in a block, or the UI could directly display the needs-update
info. for example, you visit your modules page and the row for one
of your installed contribs is now bright red with a giant warning:

"SECURITY UPDATE RELEASED FOR THIS VERSION, YOU ARE URGED TO UPGRADE
IMMEDIATELY"

wouldn't that be slick? ;)

my goal is to get this into core someday. it's too late in 5.x for
that. however, since the modules page is all happy FAPI, we could do
this via a form_alter()'ing contrib for 5.x and see that eventually
migrate into core in 6.x or beyond.

why am i asking for comments?

there needs to be some more data in the .info files for 5.x modules
if a contrib is going to be able to do this properly. as of today,
the packaging script on drupal.org that creates the downloadable
tarballs will automatically append the version of your module into
the .info files[1]. this is b/c the modules page in core is about to
be enhanced to display this data[2]. however, just knowing the
version and the human-readable name of the module isn't enough for a
5.x contrib to be able to find out if your code is out of date. we
also need something along the lines of:

; Project identification
Project home = drupal.org
Project name = views

then, the soon-to-be-written check_updates.module can get this info
and use it to construct the right URLs for the XML-RPC calls or to
find the right RSS feed, etc.

the packaging script can easily add this info automatically, but i
need to finalize exactly what the fields are going to be called and
what data they should really hold. i don't want to just hard-code a
full URL since the project.module changes aren't done to provide the
info yet, and we might change our mind on exactly the right path to
use (and XML-RPC vs. RSS, etc). also, we can't assume *every* module
installed on your site is from drupal.org, since that would break
things for 3rd-party modules or custom-built ones. in that case,
"Project home" should point to something else, or not exist at all.
similarly, using a "Project name" instead of a "Project nid" is both
more sane for end users to grok, and provides a level of indirection
in case project nodes ever move around, get deleted and re-created,
whatever.

i *really* want to get this data into the .info files ASAP so that
there aren't many 5.x contribs out in the wild that are missing it.
however, i don't want to just unilaterally decide the fields and
format of the values without any input from the rest of you. so,
please comment ASAP here:

http://drupal.org/node/94154

thanks!
-derek

[1] http://drupal.org/node/97286
"automatically insert human-readable version into .info files"

[2] http://drupal.org/node/87298
"Add version info to modules page"

[Bèr Kessels]

Hi,

The way I handle this on Sympal is far easier, maybe a good idea:
I add a predefined feed in aggregator, and have a PHP block preconfigured that
shows the last item in that feed to certain roles.

If you like KISS something down this line may be more pragmatic and easier to
achieve. It works rather well, be it, that I cannot (nor want to) focus on
certain modules, something that you seem to want to introduce also.

Bèr

[Robert Douglass]

Derek's plan is much more robust and not a lot more complicated than
RSS. People ignore RSS. But if they look at their modules page and right
next to the Views module it says "New version available", the'll notice
that. I really like Derek's plan.

Derek, the fields you proposed look sane an unobtrusive. How, though
will it support the case you mentioned where the module comes from a 3rd
party vendor? Don't you need an XML-RPC endpoint or or some other
absolute URL? I know you mentioned you want to avoid hardcoding these,
but is it reasonable to expect that you can avoid doing so?

-Robert

Post by Bèr Kessels
Hi,
I add a predefined feed in aggregator, and have a PHP block preconfigured that
shows the last item in that feed to certain roles.
If you like KISS something down this line may be more pragmatic and easier to
achieve. It works rather well, be it, that I cannot (nor want to) focus on
certain modules, something that you seem to want to introduce also.
Bèr

[adrian rossouw]

Post by Robert Douglass
Derek, the fields you proposed look sane an unobtrusive. How,
though will it support the case you mentioned where the module
comes from a 3rd party vendor? Don't you need an XML-RPC endpoint
or or some other absolute URL? I know you mentioned you want to
avoid hardcoding these, but is it reasonable to expect that you can
avoid doing so?

I think a block that shows up for uid no 1, which lists security
announcements, would be cool too

Kind of a security release announcement block. Even if it's just a
little line of text in the body.

[Karoly Negyesi]

Post by adrian rossouw
I think a block that shows up for uid no 1, which lists security
announcements, would be cool too
Kind of a security release announcement block. Even if it's just a
little line of text in the body.

Huh? Last I checked drupal.org/security had an RSS feed and Drupal core had an aggregator which can do blocks and we have excellent control over block visibility.

[Bèr Kessels]

Post by Karoly Negyesi
Huh? Last I checked drupal.org/security had an RSS feed and Drupal core had
an aggregator which can do blocks and we have excellent control over block
visibility.

This is almost exactly what I mean. Its works now, needs no patches, no
modules and no develoment/. It can be done now.

Bèr

[Earl Miles]

Post by Bèr Kessels

Post by Karoly Negyesi
Huh? Last I checked drupal.org/security had an RSS feed and Drupal core had
an aggregator which can do blocks and we have excellent control over block
visibility.

This is almost exactly what I mean. Its works now, needs no patches, no
modules and no develoment/. It can be done now.

Yes, but does it work for just the versions of modules you currently have on
your site?

No.

The RSS feed works for ALL modules. That's great. There's 500 modules and in
general I probably care about maybe 15 or 20. That's a lot of data to sift
through, needlessly. RSS won't let me search for data on a specific module
(Because RSS feeds only go back so far).

What we're proposing here *is* simple, and it doesn't try to use RSS for
something it's not actually particularly good for. If it were good for that, we
would already have figured out a way to use it.

[Karoly Negyesi]

Post by Bèr Kessels

Post by Karoly Negyesi
Huh? Last I checked drupal.org/security had an RSS feed and Drupal core had
an aggregator which can do blocks and we have excellent control over block
visibility.

This is almost exactly what I mean. Its works now, needs no patches, no
modules and no develoment/. It can be done now.

Ah sorry. My mail has been misunderstood. I reacted to Adrian's mail re. security.

In overall this solution is not enough because you many modules but by far not all from contribs, so neither a per module feed or an eat-it-all feed won't suffice. So definitely we need more code in project.

[Bèr Kessels]

Post by Karoly Negyesi
In overall this solution is not enough because you many modules but by far
not all from contribs, so neither a per module feed or an eat-it-all feed
won't suffice. So definitely we need more code in project.

I never said it would be enough. All I said is that, instead of talking for
ages here about that utopian solution, then get into developing some whole
new release-pushing-xmlrpc-system, we can offer a default RSS feed with block
NOW.

As in: now. Not something in some future after some unknown development cycle.

Bèr

[Arnab Nandi]

What about simply having an iframe to
https://drupal.org/modulecheck/modulename, which returns a "You're
running the latest version" normally, or a button saying "click here
to upgrade" otherwise. The button essentially pionts to the changelog
/ install notes. (later, the button can point to the installer which
autoupgrades stuff).

-Arnab

Post by Bèr Kessels

Post by Karoly Negyesi
In overall this solution is not enough because you many modules but by far
not all from contribs, so neither a per module feed or an eat-it-all feed
won't suffice. So definitely we need more code in project.

I never said it would be enough. All I said is that, instead of talking for
ages here about that utopian solution, then get into developing some whole
new release-pushing-xmlrpc-system, we can offer a default RSS feed with block
NOW.
As in: now. Not something in some future after some unknown development cycle.
Bèr
--
sympal.nl

[Corey Bordelon]

It looks like with the suggestions for XML-RPC calls would allow the
modulecheck to be run with the cron hook. That way, everytime an admin looks
at their modules page, it wouldn't be a call to drupal.org, like an iframe
would. Also, I'm not crazy about iframes, but I could in be a minority on
that point.

Post by Arnab Nandi
What about simply having an iframe to
https://drupal.org/modulecheck/modulename, which returns a "You're
running the latest version" normally, or a button saying "click here
to upgrade" otherwise. The button essentially pionts to the changelog
/ install notes. (later, the button can point to the installer which
autoupgrades stuff).
-Arnab

Post by Bèr Kessels

Post by Karoly Negyesi
In overall this solution is not enough because you many modules but by

far

Post by Bèr Kessels

Post by Karoly Negyesi
not all from contribs, so neither a per module feed or an eat-it-all

feed

Post by Bèr Kessels

Post by Karoly Negyesi
won't suffice. So definitely we need more code in project.

I never said it would be enough. All I said is that, instead of talking

for

Post by Bèr Kessels
ages here about that utopian solution, then get into developing some

whole

Post by Bèr Kessels
new release-pushing-xmlrpc-system, we can offer a default RSS feed with

block

Post by Bèr Kessels
NOW.
As in: now. Not something in some future after some unknown development

cycle.

Post by Bèr Kessels
BÚr
--
Drupal, Ruby on Rails and Joomla! development: webschuur.com | Drupal
sympal.nl

--
http://www.arnab.org

[Johan Forngren]

I don't like iframes either, but it is a lot simpler to set up than xml-rpc.
Allthough, I would prefer a xmlrpc solution, or both.

Post by Corey Bordelon
It looks like with the suggestions for XML-RPC calls would allow the
modulecheck to be run with the cron hook. That way, everytime an admin looks
at their modules page, it wouldn't be a call to drupal.org, like an iframe
would. Also, I'm not crazy about iframes, but I could in be a minority on
that point.

Post by Arnab Nandi
What about simply having an iframe to
https://drupal.org/modulecheck/modulename, which returns a "You're
running the latest version" normally, or a button saying "click here
to upgrade" otherwise. The button essentially pionts to the changelog
/ install notes. (later, the button can point to the installer which
autoupgrades stuff).
-Arnab

Post by Bèr Kessels

Post by Karoly Negyesi
In overall this solution is not enough because you many modules but

by far

Post by Bèr Kessels

Post by Karoly Negyesi
not all from contribs, so neither a per module feed or an eat-it-all

feed

Post by Bèr Kessels

Post by Karoly Negyesi
won't suffice. So definitely we need more code in project.

I never said it would be enough. All I said is that, instead of

talking for

Post by Bèr Kessels
ages here about that utopian solution, then get into developing some

whole

Post by Bèr Kessels
new release-pushing-xmlrpc-system, we can offer a default RSS feed

with block

Post by Bèr Kessels
NOW.
As in: now. Not something in some future after some unknown

development cycle.

Post by Bèr Kessels
BÚr
--
Drupal, Ruby on Rails and Joomla! development: webschuur.com | Drupal
sympal.nl

--
http://www.arnab.org

--
// Johan Forngren

/****************************************
Email: ***@forngren.com
Site: http://johan.forngren.com/
Skype: forngren
MSN: ***@forngren.com
IRC: forngren @ freenode
Jabber/GTalk: ***@gmail.com
****************************************/

[Earl Miles]

Post by Johan Forngren
I don't like iframes either, but it is a lot simpler to set up than
xml-rpc. Allthough, I would prefer a xmlrpc solution, or both.

Since Drupal already has a good xmlrpc library, xmlrpc is actually very simple.
It's actually easier than trying to parse an RSS feed, in fact.

[Johan Forngren]

Indeed, but not all servers has xml-rpc or require some kind configuration
to enable it. I think it's the masses who need this function the most, but
also the ones who is less likely to set up xml-rpc.

Post by Earl Miles

Post by Johan Forngren
I don't like iframes either, but it is a lot simpler to set up than
xml-rpc. Allthough, I would prefer a xmlrpc solution, or both.

Since Drupal already has a good xmlrpc library, xmlrpc is actually very simple.
It's actually easier than trying to parse an RSS feed, in fact.

--
// Johan Forngren

/****************************************
Email: ***@forngren.com
Site: http://johan.forngren.com/
Skype: forngren
MSN: ***@forngren.com
IRC: forngren @ freenode
Jabber/GTalk: ***@gmail.com
****************************************/

[andrew morton]

Post by Johan Forngren
Indeed, but not all servers has xml-rpc or require some kind configuration
to enable it. I think it's the masses who need this function the most, but
also the ones who is less likely to set up xml-rpc.

Provided that PHP has the proper extensions configured, Drupal "out of
the box" with XML-RPC. I'm not sure what the exact percentage of hosts
that have the xml extension but according to the PHP docs (
http://us3.php.net/xml ) it's enabled by default.

[Earl Miles]

Post by Bèr Kessels

Post by Karoly Negyesi
Huh? Last I checked drupal.org/security had an RSS feed and Drupal core had
an aggregator which can do blocks and we have excellent control over block
visibility.

This is almost exactly what I mean. Its works now, needs no patches, no
modules and no develoment/. It can be done now.

Two other points.

1) It's hard to argue that highly targeted security announcements are a bad
idea. If 1) drupal.org tracks what modules have security problems and the
solution, and 2) a site can ask drupal.org for this information an all its
modules, sites owners can be properly informed. We know that many people
continue to run with old versions of the software. We can reduce that number a lot.

2) This information can be given via drupal_set_message on /admin, not via
blocks which can be misconfigured due to user error. We can be certain that the
right messages are delivered to the right people.

[Richard Morse]

Post by adrian rossouw

Post by Robert Douglass
Derek, the fields you proposed look sane an unobtrusive. How,
though will it support the case you mentioned where the module
comes from a 3rd party vendor? Don't you need an XML-RPC endpoint
or or some other absolute URL? I know you mentioned you want to
avoid hardcoding these, but is it reasonable to expect that you
can avoid doing so?

I think a block that shows up for uid no 1, which lists security
announcements, would be cool too

Sorry for being late on this, but I just wanted to comment that any
such setup would be useful to be able to be configured for users
other than 1. I rarely use user 1 -- only when I need to run
update.php -- and as such, I would never see such a block.

Thanks,
Ricky

[Robert Douglass]

Just to clarify: the reason I think XML-RPC is superior to RSS in this
case (and worth the extra effort), is that it can request information
for the exact branches of the modules installed. We'd have to work extra
hard to build a tailored RSS feed for every Drupal installation on the
planet, add extra fields to the feed, parse them out etc., whereas with
XML-RPC it is all much more comfortable. I like that Derek's proposal
will allow us to display things like:

"You are using views-4-7--1.1. views-4-7--1.2 is available. This is a
security update"
as well as:
"You are using views-4-7--1.1. views-4-7--1.2 is the latest release in
this branch. views-4-7--2.0 is also compatible with your Drupal
installation and is available."

Post by Robert Douglass
Derek's plan is much more robust and not a lot more complicated than
RSS. People ignore RSS. But if they look at their modules page and
right next to the Views module it says "New version available", the'll
notice that. I really like Derek's plan.
Derek, the fields you proposed look sane an unobtrusive. How, though
will it support the case you mentioned where the module comes from a
3rd party vendor? Don't you need an XML-RPC endpoint or or some other
absolute URL? I know you mentioned you want to avoid hardcoding these,
but is it reasonable to expect that you can avoid doing so?
-Robert

Post by Bèr Kessels
Hi,
I add a predefined feed in aggregator, and have a PHP block
preconfigured that shows the last item in that feed to certain roles.
If you like KISS something down this line may be more pragmatic and
easier to achieve. It works rather well, be it, that I cannot (nor
want to) focus on certain modules, something that you seem to want to
introduce also.
Bèr

[Earl Miles]

Post by Robert Douglass
Just to clarify: the reason I think XML-RPC is superior to RSS in this
case (and worth the extra effort), is that it can request information
for the exact branches of the modules installed. We'd have to work extra
hard to build a tailored RSS feed for every Drupal installation on the
planet, add extra fields to the feed, parse them out etc., whereas with
XML-RPC it is all much more comfortable. I like that Derek's proposal
"You are using views-4-7--1.1. views-4-7--1.2 is available. This is a
security update"
"You are using views-4-7--1.1. views-4-7--1.2 is the latest release in
this branch. views-4-7--2.0 is also compatible with your Drupal
installation and is available."

Also, the 'extra effort' is tiny. Seriously, support for xmlrpc is already
there. Support for having Drupal phone home to drupal.org is already there.

All that this proposal does is add a couple of extra xmlrpc methods that can be
called, which allows for a contrib module to utilize these methods and let
operators know about important data *that is tailored to them*.

[Steven Wittens]

Post by Derek Wright
my goal is to get this into core someday. it's too late in 5.x for
that. however, since the modules page is all happy FAPI, we could
do this via a form_alter()'ing contrib for 5.x and see that
eventually migrate into core in 6.x or beyond.

Typically, modules will only go out of date long after you install
them. So, you would only notice outdated modules when you come back
to admin/modules for something else.

Perhaps this could be added to the status report instead. We already
have an 'outreach' from the status report to /admin for critical
errors and we could do something similar for out of date versions.
Given that version checking can be applied not only to modules, but
also themes and even core, it seems a bit out of place on the modules
page as well.

Plus, it allows us to present it in a simple, compact format. Some
examples:

------------
Drupal 5.2 (up to date)
(Checkmark icon)
------------
Drupal 5.1 (minor update available)
(Error icon) A new maintenance release is available for Drupal 5.x
and should be installed immediately to prevent critical security bugs.
------------
Drupal 4.7.4 (major update available)
(Warning icon) A new major release of Drupal is available which
includes many new features.
------------

------------
Modules Up to date
------------
Modules Minor updates available
Updates are available for the following modules:
* Pirate
* Views
(Warning icon)
------------
Modules Major updates available
A critical security fix is available for the following modules and
should be installed immediately:
* Hack your site.module
------------

The idea of the status report has always been that it should only
present important information. Contrast this with, for example,
presenting a long table of module names with a column that says "up
to date" or "updates available": it forces the admin to sift through
the (possibly long) list him/herself. It is much more efficient to
summarize the information.

Of course, an additional outreach could be added to admin/modules
too, but the update information should IMO not live there.

Steven Wittens

[Derek Wright]

Post by Steven Wittens
Perhaps this could be added to the status report instead.

i'm down with this, only not "instead". personally, once
check_updates.module (or whatever) does it's daily ping home and has
a locally cached table of out-of-date modules (and potentially themes
+ core), i'd be thrilled to see the admin literally bombarded with
the fact they've got out-of-date (potentially insecure) modules
installed:

1) on the modules page (since it's about modules, and that's where
they'll be seeing the versions of their installed modules in the
first place).

2) in the status report (grand idea)

3) as a little block that automatically appears any time you visit
admin/*

then, it'd be pretty hard to miss the fact you're out of date, and
that you should do something about it.

none of this is meant to replace the drupal.org/security RSS feed,
and people are encouraged to use that however they wish. this whole
proposal just supplements that mechanism with a more direct approach
that tells people exactly what they have to know.

once we start putting more work into security audits of contrib, our
poor security RSS feed is going to start turning into the boy that
cried wolf. :( of course, we must continue, but more and more, SAs
are going to go out and people are going to say "oh, i don't care
about that".

now that we have the means, i want to see us use them to put only the
relevant data, and all of it, exactly where the admins need it.

thanks,
-derek

[Boris Mann]

Post by Derek Wright
i *really* want to get this data into the .info files ASAP so that
there aren't many 5.x contribs out in the wild that are missing it.
however, i don't want to just unilaterally decide the fields and
format of the values without any input from the rest of you. so,
http://drupal.org/node/94154

Adding the extra information is a great idea...we have our own little
repository / update system, and with a different "home", different sites
could, for instance, keep different distributions up to date.

HOWEVER, the phone home and XML-RPC stuff makes me *very* nervous from a
security perspective. I would want to have some real hard core folks examine
and document information flow end to end and looking for vulnerabilities --
ideally some external folks as well. We will need to review all
Drupal.orgprocesses as well as the receiving code.

There has been other talk about auto-downloading various information. Same
comment there -- huge security risk, needs 100x as much review, and even
then I'm nervous about it....

[Oswald Jaskolla]

Hi,

I am currently working on a system to automatically install modules (i.e
download and unpack into the appropriate directory) and upgrade modules
(i.e. overwrite present files). The architecture is finished:

- - A module for the GUI and the actual work (not fully implemented yet)
- - A client that provides necessary information to the "GUI and
functionality" module (one implementation is nearly finished, it lacks
the ability to provide the actual module files)
- - A server that serves information (module files, version information
etc.) to client (not implemented, because the client does not need a
server)

My implementation of the client uses a working copy of the HEAD branch
of CVS (that's why I don't need a server at the moment), but there are
numerous other possibilities:
- - CVS (without a working copy)
- - XML-RPC
- - FTP
- - webcvs ... {whatever}

I wouldn't have mentioned this effort in this early stage (it's far from
alpha) but there are three points I have to make:
1. I wouldn't want us to duplicate code or go into different
(a.k.a incompatible) directions.
2. My module would also benefit from more precise information in the
.info-files
3. I, too, like to have access to the projects via XML-RPC.

So, what do you think? I tried to understand the other posts of this
discussion, but they seem to revolve around technical aspects that are
not really an issue (at least not at this point of the development
(sure, making it into core would be great)).

Greetings,
- --
Oswald Jaskolla
Ingenieurbüro Richard Schieferdecker
Kreuzherrenstraße 2
D-52062 Aachen

Tel.: 02 41 / 409 54 43
Fax: 02 41 / 477 05 199
mobil: 01 64 / 941 06 75
eMail: ***@schieferdecker.com

[Derek Wright]

Post by Oswald Jaskolla
So, what do you think?

i hope you don't take this personally, by i'm *very* opposed to the
kind of system you're building.

the security implications of giving your website permission to
overwrite itself automatically are *HORRIFYING*. i'd never install
such a thing, and i'd never advocate anyone else should install such
a thing.


the kind of system i'm building is just an automated way to tell the
human site admins: "your code is out of date" (and if true, "and
insecure") and nag them mercilessly until they upgrade the stale
module(s) to the latest, secure version(s). it's still the human's
task to perform the upgrade itself.

this manual upgrade could itself be further automated, but a high-
privileged admin user must run this automated script themselves, just
like they have to run update.php themselves. building and providing
a tool that can "do it all" for you is asking for security hell, and
therefore defeats the purpose of what i'm trying to accomplish (make
it easier and therefore more likely for drupal sites to remain secure).


anyway, i'm willing to coordinate, and further discuss design/
implementation issues, but i can't emphasize enough how bad i think a
fully-automated system for upgrading a drupal site would be.

maybe i'm misunderstanding your design/proposal, but after re-reading
your message a few times, it's pretty clear you're marching down the
path towards what i'd consider "the dark side". ;)

sorry,
-derek

[Gerhard Killesreiter]

Post by Oswald Jaskolla
So, what do you think?

i hope you don't take this personally, by i'm *very* opposed to the kind
of system you're building.
the security implications of giving your website permission to overwrite
itself automatically are *HORRIFYING*. i'd never install such a thing,
and i'd never advocate anyone else should install such a thing.

The underlying idea is so silly I'd not want us to host the resulting
code on drupal.org.

the kind of system i'm building is just an automated way to tell the
human site admins: "your code is out of date" (and if true, "and
insecure") and nag them mercilessly until they upgrade the stale
module(s) to the latest, secure version(s). it's still the human's task
to perform the upgrade itself.

That's a nice thing and very useful. Do you plan to check if a module
has been altered after download, too?

Cheers,
Gerhard

[Derek Wright]

Do you plan to check if a module has been altered after download, too?

no, i wasn't planning to do that. the md5 hashes we have are on the
whole tarball, not per file. so an easy, obvious way to detect this
doesn't immediately spring to mind, short of doing a bunch more work
to calculate, store, and advertise the per-file md5 hashes at d.o. [1]

i was just planning to go with the version listed in the .info file
and comparing that to what "home" thinks the latest version of that
module is and reporting (in various places) if they disagree.

in the first (and probably only) implementation, "home" would be
drupal.org. if someone else wants to make what i'm doing work to
phone home to other sites where other modules are hosted, they should
feel free to do so. again, i'm happy to discuss/coordinate/share,
but i'm not going to worry about it until people who really want this
functionality a) exist, b) step forward, and c) do some of their own
work to make it happen.

cheers,
-derek

[1] i guess the packaging script could compute the md5 hashes for all
the files in a tarball, and write a big section of those filename =>
md5 hash mappings into the .info file itself. then,
check_updates.module could compare the values advertised in the .info
file with the real values as installed on the file system. i'm not
sure that's worth doing, but it'd probably work. ;)

[Darren Oh]

Why so many hostile reactions to Oswald's project? Read his message
again. He's not proposing a site that updates itself. He's promoting
automatic downloads and automatic uprades. An administrator could
select the modules to be installed or upgraded and it would happen
automatically.

What he asked for was collaboration on a system to detect which
modules have updates available.

What he's working on is inevitable. Manual module installation and
upgrading creates many opportunities for inexperienced users to make
mistakes.

[Earl Miles]

Post by Darren Oh
What he's working on is inevitable. Manual module installation and
upgrading creates many opportunities for inexperienced users to make
mistakes.

Yes, and dww and I (and webchick and neclimdul and a handfull of others too)
have been working toward *this very goal* very hard for the last 4 or 5 months.

None of this is occurring in a vacuum.

One of the major motivations of the new release system project was put in place
a method for your modules to tell if they're out of date.

The map's been here for awhile and we've been working toward it. Just because
some guy has some system that he's been working on in isolation doesn't mean we
should all suddenly drop what we've been busting our asses on for months over.
If he wants to join the work we've been doing, that's great, but he's got to
join the work that's already been done. It's not like we've been working
privately and behind the scenes. There's a huge trail of issues and
conversations about this set of projects that have led up to it.

Post by Darren Oh
So, what do you think? I tried to understand the other posts of this
discussion, but they seem to revolve around technical aspects that are
not really an issue (at least not at this point of the development
(sure, making it into core would be great)).

If the technical aspects we're discussing aren't really an issue, I'm not sure
what the issues are. The technical aspects we're discussing are, in my opinion,
the last impediment to being able to actually write this system. The rest of it
is just sitting down and writing a module to do it according to the spec we lay
out in these discussions.

[Gerhard Killesreiter]

Post by Darren Oh
Why so many hostile reactions to Oswald's project? Read his message
again. He's not proposing a site that updates itself. He's promoting
automatic downloads and automatic uprades.

Where's the difference between "a sites that updates itself" and
"automatic upgrades"? I can't spot any.

Post by Darren Oh
An administrator could select
the modules to be installed or upgraded and it would happen automatically.
What he asked for was collaboration on a system to detect which modules
have updates available.
What he's working on is inevitable. Manual module installation and
upgrading creates many opportunities for inexperienced users to make
mistakes.

Inexperienced users shouldn't be running a public service such as a
website. They should read and follow the docs and thus be no longer
inexperienced.

Cheers,
Gerhard

[Darren Oh]

Post by Gerhard Killesreiter
Where's the difference between "a sites that updates itself" and
"automatic upgrades"? I can't spot any.

Think about it. This is the process recommended in the manual:

1. Select the module you want to install or upgrade.
2. Download the module to your computer.
3. Unpack the module.
4. Start your FTP client.
5. Connect to your site.
6. Switch to the correct directory (whatever that is).
7. Upload the module directory to your site.
8. Enable the module (if installing) or run update.php (if upgrading).

Steps 2 through 8 would become one step. Automatic installation or
automatic upgrade seemed clear enough to me. What would you call it?

[Earl Miles]

Post by Darren Oh

Post by Gerhard Killesreiter
Where's the difference between "a sites that updates itself" and
"automatic upgrades"? I can't spot any.

1. Select the module you want to install or upgrade.
2. Download the module to your computer.
3. Unpack the module.
4. Start your FTP client.
5. Connect to your site.
6. Switch to the correct directory (whatever that is).
7. Upload the module directory to your site.
8. Enable the module (if installing) or run update.php (if upgrading).
Steps 2 through 8 would become one step. Automatic installation or
automatic upgrade seemed clear enough to me. What would you call it?

For security and stability reasons it's more complex than that.

[Darren Oh]

Post by Earl Miles

Post by Darren Oh

Post by Gerhard Killesreiter
Where's the difference between "a sites that updates itself" and
"automatic upgrades"? I can't spot any.

1. Select the module you want to install or upgrade.
2. Download the module to your computer.
3. Unpack the module.
4. Start your FTP client.
5. Connect to your site.
6. Switch to the correct directory (whatever that is).
7. Upload the module directory to your site.
8. Enable the module (if installing) or run update.php (if
upgrading).
Steps 2 through 8 would become one step. Automatic installation
or automatic upgrade seemed clear enough to me. What would you
call it?

For security and stability reasons it's more complex than that.

All right, then. Steps 2 through 7 could become one step.
Administrators get a prompt when update.php needs to be run. The
point is, what should this be called if not an automatic installation
or upgrade? Maybe replace automatic with automated? Or scripted?

Derek's concerns about permissions could be addressed by using FTP
rather than PHP to replace files. The point is to making things
easier for users. The system he's designing would help administrators
select the modules to upgrade (step 1).

Anyway, as has been said before, some system like this is inevitable
in the future. I just hoped to dampen the negative reaction.

[Derek Wright]

Post by Darren Oh
Anyway, as has been said before, some system like this is
inevitable in the future.

thanks in large part to the system i wrote and put in place. ;) now,
i'm taking responsibility for that, and making sure that no one uses
the fruits of my labors to build any insecure tools in the name of
"usability".

Post by Darren Oh
I just hoped to dampen the negative reaction.

then help design a secure system that doesn't have the failings that
are generating the negative reactions. we're not opposed to making
it easier to update your site when it's out of date (as merlin said,
a small # of us have been busting our asses to make that happen for
months now). what we're opposed to are the careless approaches to
security being put forward (no personal offense to anyone in
particular).

i want this to *help* the security team, not make our lives worse.

that's why i'm reacting negatively.

hope that's clear...

thanks,
-derek

[Larry Garfield]

Post by Derek Wright
i want this to *help* the security team, not make our lives worse.
that's why i'm reacting negatively.

Drupal 5 now has the ability to run from the command line, right? So make the
actual replace/upgrade process a command line only script. Checking which
new modules are available can and should be an automated process. Actually
doing the upgrade should be something you have go to well out of your way to
do.

For the sysadmins in the crowd, there's nothing wrong with putting "apt-get
update" in your crontab. "apt-get upgrade" should only be run by the
sysadmin as root, by hand, and therefore when you know you're doing something
dangerous. :-)

So:
1) System auto-updates list of latest versions and lets you know if action is
necessary.
2) Admin goes to command line and runs drupal-get.php download
3) Admin sets site to offline.
4) Admin runs drupal-get.php backup
5) Admin runs drupal-get.php upgrade
6) Admin runs update.php (or that can be done automatically by #5 if it's
safe)
7) Admin brings site back online.

The hard parts (downloading and unpacking) are all automated, but only when
the admin is on the command line (and therefore running as the owner of the
files anyway), and therefore (hopefully) knows exactly what he's doing. Yes,
this excludes people who are on a host that doesn't give them a shell. Given
the file permissions involved, I don't really know of a way around that.

*sits back to have the security holes pointed out in the above.* :-)
--
Larry Garfield AIM: LOLG42
***@garfieldtech.com ICQ: 6817012

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson

[Gabor Hojtsy]

Post by Larry Garfield
For the sysadmins in the crowd, there's nothing wrong with putting "apt-get
update" in your crontab. "apt-get upgrade" should only be run by the
sysadmin as root, by hand, and therefore when you know you're doing something
dangerous. :-)
The hard parts (downloading and unpacking) are all automated, but only when
the admin is on the command line (and therefore running as the owner of the
files anyway), and therefore (hopefully) knows exactly what he's doing. Yes,
this excludes people who are on a host that doesn't give them a shell. Given
the file permissions involved, I don't really know of a way around that.
*sits back to have the security holes pointed out in the above.* :-)

How many shared hosts have a command line feature for their users? This
discussion is around easing the update pain for inexperienced users. They
are surely not running their site on their own servers.

The problem with the above, even if you only consider "apt-get update"
automatically done is that incompatibilities can happen. Previous releases
of Drupal have seen database updates needed between minor versions, and
even big chunks of code replaced / modified. Like the XML-RPC layer
completely replaced and released as a minor update. Or the form key
requirement added and released as a minor update. There is a strong
possibility that at the time when that update happens, your contrib
modules are not yet up to date (especially those you might have added
yourself, or downloaded and set up from a third party vendor - not
drupal.org). Now think that you have a modified theme, not some stock
theme from drupal.org. That cannot be updated, although you might need to
add the form key stuff into the code. No automatic update will do this for
you.

The problem is that customized software like Drupal is extremely hard to
update automatically (and most of the time it will fail due to being
uncertain that some of your components which are not possible to update
might not be compatible with stuff which is possible to update). And this
is only about updating minor releases...

As far as I see, noone saves you from testing the updates / upgrades
yourself, and ensuring that everything works. There is a place for humans
here.

Gabor

[Bèr Kessels]

And again...

Post by Darren Oh
1. Select the module you want to install or upgrade.
2. Download the module to your computer.
3. Unpack the module.
4. Start your FTP client.
5. Connect to your site.
6. Switch to the correct directory (whatever that is).
7. Upload the module directory to your site.
8. Enable the module (if installing) or run update.php (if upgrading).
Steps 2 through 8 would become one step. Automatic installation or  
automatic upgrade seemed clear enough to me. What would you call it

... we forget that a great lot of Drupas sites are, could be, or should be
installed with TOOLS.

As in: bryghts own tools, capistrano, ssh scripts, ispconfig installers, plesk
API implementations etceteras.

In that case the *properly designed* install and upgrade *framework* would
allow webGUI tools like:
1. Select the module you want to install or upgrade.
2. Install the module in your site.
3. Enable the module (if installing).
Or like
1. Type a domainname (example.com)
2. Select what type of site you want (from a list of install profiles and/or
other CMSes)
3. Log in, into your newly installed and preconfigured site

Such a webgui CAN (but should not, IMO) live inside your very own Drupal site.
It can also be ran from within plesk, or ispconfig. Or on the CLI by one
sysadmin with too much time.

So, I again, plead for a framework. An API. A well thought out foundation. An
architecture.
Not a GUI installer that needs FTP and SSH access and that requires you to
open up (rather) insecure permissions on certain files. Not an
in-site-upgrade system that forces you to run with the most insecure
permissions on teh intarweb.
No.
A framework.

Then the Good Folks Who Develop Cool stuff can build install GUIs on top of
that. Or a capistrano recipe. Or we can give the redistributors a good
alternative to develop on top of, instead of bashing the hell out of them on
digg and drupal.org for distributing "old" Drupal versions.

Bèr

[Dries Buytaert]

Post by Bèr Kessels
So, I again, plead for a framework. An API. A well thought out
foundation. An
architecture.
Not a GUI installer that needs FTP and SSH access and that requires you to
open up (rather) insecure permissions on certain files. Not an
in-site-upgrade system that forces you to run with the most insecure
permissions on teh intarweb.
No.
A framework.

I agree that we should play nice with third-party installers. That
said, I think you might want to do a cost-benefit analysis: either
work on your own Sympal scripts, or team up with Derek, Earl,
webchick et al to govern and improve certain design aspects of the
core installer. We both know what strategy is going to win in the
long term.

--
Dries Buytaert :: http://www.buytaert.net/

[Bèr Kessels]

We both know what strategy is going to win in the  
long term.

Yup. Be it that capistrano works NOW. And so do these scripts. they allow me
to install stuff already.
So on the "short term" I also know what strategy brings me most benefit.

But that said: the main reason that I continued on the framework and API
(commandline) road is not because I felt like "doing my own thing". The
reason is twofold:

When The installer was discussed, I explained my plans in rahter great
detail. At that moment I could hardly show code. I could only show where I
wanted to head and explain (which I did) why I thought that framework route
was the one to take. People, including Dries, clearly indicated that this was
not the desired route at that time: Drupal needed a web-based installer ASAP,
not a framework in some future. Fine, but I decided to continue on what I
felt the correct route, because I have no need (scratch itch) for a webbased
installer.

When the first installer was released we discussed a profile-management
system (distros). By that time, I already had a working draft of my own
installer with profiles. I presented that on this list, but it was ignored,
with one exception: A technical sidenote by Moshe. And frankly I don't feel
like spending loads of efforts in defending an idea, where people have little
interest in. I prefer to spend that time on making stuff work better. Not on
telling people that it might work better then something else. So I decided to
let it go and see what would come out, and whether or not I would like the
core distro-system better (I don't) and then convert my own into that.

So here are my plans:
* Move all my scripts to capistrano. It makes no sense to reinvent the wheel
(sympal PHP scripts are a Drupal-specific PHP version of a much more mature
Ruby system called capistrano)
* During that time, keep on using the scripts, so possibly I will integrate
the new profile/distro stuff in PHP, before moving over to capistrano.

Questions:
Would capistrano serve as a default Drupal install/update framework? Are we
afraid of Ruby (as in: not rails, but the language)? Do we want to depend on
a third party tool for our deployment framework?

Personal my answers are: yes, no, yes. But I'd like to hear if this can even
be considered for core :)

Bèr

[Gerhard Killesreiter]

Bèr Kessels wrote:

Note: I am not very much interested in installers/updaters since I
mostly only run one site at a time.

Post by Bèr Kessels
Would capistrano serve as a default Drupal install/update framework? Are we
afraid of Ruby (as in: not rails, but the language)? Do we want to depend on
a third party tool for our deployment framework?

I don't think I'd use a tool written in "yet another language".

Post by Bèr Kessels
Personal my answers are: yes, no, yes. But I'd like to hear if this can even
be considered for core :)

That's IMO a rather clear case of "Forget it".

Cheers,
Gerhard

[Robert Douglass]

I've got nothing against Ruby or any other language, but there's no way
in hell I'd support making another language a requirement for Drupal.

Post by Gerhard Killesreiter

Post by Bèr Kessels
Personal my answers are: yes, no, yes. But I'd like to hear if this
can even be considered for core :)

That's IMO a rather clear case of "Forget it".
Cheers,
Gerhard

[Gabor Hojtsy]

Post by Bèr Kessels
Would capistrano serve as a default Drupal install/update framework? Are we
afraid of Ruby (as in: not rails, but the language)? Do we want to depend on
a third party tool for our deployment framework?
Personal my answers are: yes, no, yes. But I'd like to hear if this can even
be considered for core :)

We are solving an end user problem with in installer/updater. How many of
our users (and user prospects) will host on servers having capistrano
installed? If we are depending on capistrano, how many servers will we be
compatible with?

Gabor

[Corey Bordelon]

I'm a little late to nitpick details, but capistrano doesn't need to be on
the host/server. It needs to be installed on the user's computer as a sort
of client. The other software requirement (besides ruby) is a version
control system that has the files you want to install elsewhere (that's
probably optional, too). I believe capistrano uses ssh to login and run the
commands in it's scripts. Think of it like a super ssh/scripting client
that runs multiple commands on multiple machines in parallel. If that
thought scares you, all I can say is "be afraid, be very afraid" ;)

<opinion-based-almost-rant>
I don't think BÚr wants capistrano to be the "one installer to rule them
all" for everyone. I think our needs are similar; I need something that can
allow me to deploy/update multiple drupal sites on multiple hosts. That's
not the use case for the newbie setting up their first site.

PEAR has an interesting installer, especially with the channels concept that
was introduced in their last major update (over 1.5 years ago). Why can't
we adapt a system to play nicely with that, so that a drupal install is as
easy as "pear install drupal/drupal.core"? (I know their are other
technical things needed before that can happen, but it's an option that
we're getting very close to being able to do).

If said framework existed, we could use the same infrastructure to power
whatever "human-interaction required" update system that many seem to favor
with the system for those of us who have "multiple sites to update and
better things to do than be playing with some @!#$%^ tarballs of modules".
</opinion-based-almost-rant>

-- Corey Bordelon <http://drupal.org/user/23318>

Post by Bèr Kessels

Post by Bèr Kessels
Would capistrano serve as a default Drupal install/update framework? Are

we

Post by Bèr Kessels
afraid of Ruby (as in: not rails, but the language)? Do we want to

depend on

Post by Bèr Kessels
a third party tool for our deployment framework?
Personal my answers are: yes, no, yes. But I'd like to hear if this can

even

Post by Bèr Kessels
be considered for core :)

We are solving an end user problem with in installer/updater. How many of
our users (and user prospects) will host on servers having capistrano
installed? If we are depending on capistrano, how many servers will we be
compatible with?
Gabor

[Dries Buytaert]

Post by Bèr Kessels
When The installer was discussed, I explained my plans in rahter great
detail. At that moment I could hardly show code. I could only show where I
wanted to head and explain (which I did) why I thought that
framework route
was the one to take. People, including Dries, clearly indicated that this was
not the desired route at that time: Drupal needed a web-based
installer ASAP,
not a framework in some future.

I don't remember saying any of that, but my memory has failed on more
than one occasion. However, I do remember advocating the need to
play nice with installers like Debian's and Gentoo's. My opinions on
this issue have been very clear for at least 6-12 months -- that is
plenty of time to align your work with that of core.

Of course, you're more than welcome to work on Sympal. There is
nothing wrong with that. :)

Post by Bèr Kessels
* Move all my scripts to capistrano. It makes no sense to reinvent the wheel
(sympal PHP scripts are a Drupal-specific PHP version of a much more mature
Ruby system called capistrano)
* During that time, keep on using the scripts, so possibly I will integrate
the new profile/distro stuff in PHP, before moving over to capistrano.

Have fun! I haven't looked at Capistrano, but we can't require people
to install Ruby. Then again, if this is only meant to be used by a
small subset of power-users, it might fly. For various reasons, I'd
prefer a PHP-based solution though (eg. maintainability).

--
Dries Buytaert :: http://www.buytaert.net/

[Bèr Kessels]

 Then again, if this is only meant to be used by a  
small subset of power-users, it might fly.  For various reasons, I'd  
prefer a PHP-based solution though (eg. maintainability).

Yup, me too.

However, it seems to become more common over time. You will notice the Ruby
(on rails) enabled webhosts popping up all over the place already.

As said elsewhere in the thread, capistrano is not something to be installed
on the server, but rather on a client. So the server neds no Ruby, the client
needs that.

Add to that, the fact that wordpress, Joomla! and most RoR apps are
capistranised, and you can see a great opportunity.

But all that in consideration, dependency on another language has two issues:
we need to have people in the dev teams who speak such a language, and we
will be requiring a lot more for a simple installation.

Still, I think capistrano is very well suited for the task, it seems to become
a sort of de-facto standard in deployment-land and it has all the stuff
running and working, while such a PHP application is not even existing.

More links and investigations for Drupal capistrano on
http://groups.drupal.org/node/1461

Bèr

[Richard Morse]

Post by Derek Wright

Do you plan to check if a module has been altered after download, too?

no, i wasn't planning to do that. the md5 hashes we have are on
the whole tarball, not per file. so an easy, obvious way to detect
this doesn't immediately spring to mind, short of doing a bunch
more work to calculate, store, and advertise the per-file md5
hashes at d.o. [1]

One thing about this is that it would be good to allow admins to say
"yes, this file has been modified, but I know about it so don't tell
me again". Occasionally I make small changes to a module (example: I
modified biblio.module to hide ability to export XML, because I don't
want it visible), and I don't want to be plagued by massive warnings
(or worse, automatic disabling) because I customized a module.

Thanks,
Ricky

[Sammy Spets]

While I agree with Derek about the security issues there can be
ways around it.

For example, functions that perform the update do so only when a file
(in the site root directory) contains some random characters chosen by
the system for that upgrade session. The file must be manually created
prior to the upgrade though a way to generate the file is provided. The
administrator can only download to their computer and then upload to
server.

Derek is absolutely wright (sic) that security is a BIG issue. I'd also
not use it without measures similar to the above.

Cheers,
--
Sammy Spets
Synerger Pty Ltd
http://synerger.com

Post by Derek Wright

Post by Oswald Jaskolla
So, what do you think?

i hope you don't take this personally, by i'm *very* opposed to the
kind of system you're building.
the security implications of giving your website permission to
overwrite itself automatically are *HORRIFYING*. i'd never install
such a thing, and i'd never advocate anyone else should install such
a thing.
the kind of system i'm building is just an automated way to tell the
human site admins: "your code is out of date" (and if true, "and
insecure") and nag them mercilessly until they upgrade the stale
module(s) to the latest, secure version(s). it's still the human's
task to perform the upgrade itself.
this manual upgrade could itself be further automated, but a high-
privileged admin user must run this automated script themselves, just
like they have to run update.php themselves. building and providing
a tool that can "do it all" for you is asking for security hell, and
therefore defeats the purpose of what i'm trying to accomplish (make
it easier and therefore more likely for drupal sites to remain secure).
anyway, i'm willing to coordinate, and further discuss design/
implementation issues, but i can't emphasize enough how bad i think a
fully-automated system for upgrading a drupal site would be.
maybe i'm misunderstanding your design/proposal, but after re-reading
your message a few times, it's pretty clear you're marching down the
path towards what i'd consider "the dark side". ;)
sorry,
-derek

[Derek Wright]

The file must be manually created prior to the upgrade though a way
to generate the file is provided.

you lost me. ;) if the admin has to manually type in the random key
to allow the "automatic" update to do its thing, what's the point of
calling it "automatic" and giving the website filesystem write access
to clobber itself?

why not just say "human intervention is REQUIRED"?

1) put site into system maintenance mode
2) backup DB and filesystem
3) run some "fetch_latest.php" script that knows all your installed
modules, downloads the updated tarballs, unpacks them into place, etc
4) run update.php
5) test to make sure life is good
6) leave system maintenance mode and return to live operation.

i'm happy to see steps #2 and #3 as automated as possible, but #3
should definitely run as the high-privileged admin's uid, not the uid
of the webserver process itself.

anything less would be uncivilized... ;)

-derek

[Sammy Spets]

You're saying I called it automatic, when I didn't!

We're talking about the same thing. My example is a way to ensure the
update procedure is legitimately run by the admin user and thus can use
the webserver process uid to replace files (which is necessary for the
files to be usable by Drupal). It's an example and not a suggestion or a
"do it this way".

Aside from all that there is a need to reduce downtime of a site.
Putting the site into maintenance mode and then downloading the
replacement modules increases the downtime. Would be nice if the
solution ends up with a way to download updated modules before going
into maintenance mode.

So an addendum to Derek's workflow:

1) download updated modules into a temporary filesystem location
2) put site into system maintenance mode
3) backup DB and filesystem
4) run some "upgrade.php" script that unpacks the modules into place
5) run update.php
6) test to make sure life is good
7) leave system maintenance mode and return to live operation.

Cheers,
--
Sammy Spets
Synerger Pty Ltd
http://synerger.com

Post by Derek Wright

The file must be manually created prior to the upgrade though a way
to generate the file is provided.

you lost me. ;) if the admin has to manually type in the random key
to allow the "automatic" update to do its thing, what's the point of
calling it "automatic" and giving the website filesystem write access
to clobber itself?
why not just say "human intervention is REQUIRED"?
1) put site into system maintenance mode
2) backup DB and filesystem
3) run some "fetch_latest.php" script that knows all your installed
modules, downloads the updated tarballs, unpacks them into place, etc
4) run update.php
5) test to make sure life is good
6) leave system maintenance mode and return to live operation.
i'm happy to see steps #2 and #3 as automated as possible, but #3
should definitely run as the high-privileged admin's uid, not the uid
of the webserver process itself.
anything less would be uncivilized... ;)
-derek

[Derek Wright]

Post by Sammy Spets
My example is a way to ensure the
update procedure is legitimately run by the admin user and thus can use
the webserver process uid to replace files (which is necessary for the
files to be usable by Drupal).

no offense, but this is exactly the kind of sloppy approach to file
system permissions that leads to sites getting hacked and drupal
getting a bad name. :(

your Drupal source files *DO NOT* have to be writable by the
webserver process uid for "the files to be usable by Drupal" (!!!).
the "files" directory is a special case, which is why a) it's been a
source of numerous security issues and b) it should be handled with
intensely defensive programming by anything that's touching it.

that aside, your Drupal source files must be _readable_ by the
webserver process uid, period.

your Drupal source files *SHOULD* be _writable_ only by a *DIFFERENT*
process uid, namely, your "high-privilege admin" user (could be root,
but doesn't have to be, so long as it's *NOT* the same as your httpd).


until people completely understand this, they'll a) continue to fill
up this thread with confusing posts that don't advance the end goal,
and b) continue to setup insecure websites.

"automatic" upgrades, in whatever form, imply that some automatic
service is going to be overwriting things in modules/* under your doc
root. that requires that the automatic service has write access, and
that means trouble.

unless the human (with the different uid, and higher powers to do
damage) runs a script to trigger the upgrade (after backups, etc),
life sucks.

so, if we're all clear on that, and we want to discuss exactly how
much can be automated into this script, what the UI looks like, etc,
etc, that's all fine and good, but a) it's premature (i'm just asking
about agreement on what goes in the .info files to make all this
possible) and b) it's a different discussion than "automatic site
upgrades"...

-derek


p.s. sure, friendly amendment about downloading to a tmp directory
and only putting the site in maintenance mode for as little time as
possible is totally reasonable. i wasn't being as careful about the
exact steps in that reply, since my main point was *THE HUMANS MUST
DO IT* or life sucks.

p.p.s. did i mention that human intervention is required for this to
be a good idea? ;)

[Sammy Spets]

Again i'll emphasize that it was an example and NOT a suggestion. You're
jumping all over it like it's on fire. A little hot under the collar?

I'm going to bow out of this discussion while I still have a head on my
shoulders. I'm obviously out of my league.
--
Sammy Spets
Synerger Pty Ltd
http://synerger.com

Post by Derek Wright

Post by Sammy Spets
My example is a way to ensure the
update procedure is legitimately run by the admin user and thus can use
the webserver process uid to replace files (which is necessary for the
files to be usable by Drupal).

no offense, but this is exactly the kind of sloppy approach to file
system permissions that leads to sites getting hacked and drupal
getting a bad name. :(
your Drupal source files *DO NOT* have to be writable by the
webserver process uid for "the files to be usable by Drupal" (!!!).
the "files" directory is a special case, which is why a) it's been a
source of numerous security issues and b) it should be handled with
intensely defensive programming by anything that's touching it.
that aside, your Drupal source files must be _readable_ by the
webserver process uid, period.
your Drupal source files *SHOULD* be _writable_ only by a *DIFFERENT*
process uid, namely, your "high-privilege admin" user (could be root,
but doesn't have to be, so long as it's *NOT* the same as your httpd).
until people completely understand this, they'll a) continue to fill
up this thread with confusing posts that don't advance the end goal,
and b) continue to setup insecure websites.
"automatic" upgrades, in whatever form, imply that some automatic
service is going to be overwriting things in modules/* under your doc
root. that requires that the automatic service has write access, and
that means trouble.
unless the human (with the different uid, and higher powers to do
damage) runs a script to trigger the upgrade (after backups, etc),
life sucks.
so, if we're all clear on that, and we want to discuss exactly how
much can be automated into this script, what the UI looks like, etc,
etc, that's all fine and good, but a) it's premature (i'm just asking
about agreement on what goes in the .info files to make all this
possible) and b) it's a different discussion than "automatic site
upgrades"...
-derek
p.s. sure, friendly amendment about downloading to a tmp directory
and only putting the site in maintenance mode for as little time as
possible is totally reasonable. i wasn't being as careful about the
exact steps in that reply, since my main point was *THE HUMANS MUST
DO IT* or life sucks.
p.p.s. did i mention that human intervention is required for this to
be a good idea? ;)

[Anton]

Post by Derek Wright
your Drupal source files *DO NOT* have to be writable by the
webserver process uid for "the files to be usable by Drupal" (!!!).
the "files" directory is a special case, which is why a) it's been a
source of numerous security issues and b) it should be handled with
intensely defensive programming by anything that's touching it.

Yep, a very important aspect of web application security is to protect
the web application from itself. The web server can't tell the
difference between a legitimate request to overwrite application files
and one that has manipulated the application into doing so. Trust
nothing (or more practically trust as little as possible).

Ideally the web server and the web application shouldn't trust each
other. Changes to the application should be done "out of band" ie not
through the same medium the rest of the internet accesses the
application.

Of course, on a shared host there will be compromises that need to be
made. For instance phpsuexec effectively violates the "out of band"
idea by running the site as the "out of band" admin user, but it adds
protection against other sites on a shared host which is arguably more
important. When not on a shared host it would be better not to use
phpsuexec so running the app and changing the app are separated from
each other.

Hopefully that wasn't offtopic. Apologies for rehashing something most
of you are already fully aware of, I just wanted to reiterate what
Derek was saying in a slightly different way to make sure any less
experienced developers understood his point.

[Oswald Jaskolla]

Wow,

I am currently working on a system to automatically install modules.

looks like I really hit a nerve there. So let me clarify a few things:

- - Downloading and installing is only done on explicit request of the
administrator. I am not Microsoft.
- - Downloaded files are not less safe because they are downloaded via
PHP. There is currently no checksumming available and apart from
developers nobody looks into the code to see if it was tampered with.
- - There are a lot of drupal installations for development and testing,
that do not have the same security needs as production sites have.
- - Typo3 does it.

The only security issue remaining is having write access to the modules
directory. If the actual downloading and unpacking is done via a one
time cron job, this cron job could temporarily alter the access mode of
the target directory, minimizing the time that the directory is writable.

Greetings,
- --
Oswald Jaskolla
Ingenieurbüro Richard Schieferdecker
Kreuzherrenstraße 2
52062 Aachen

Tel.: 02 41 / 409 54 43
Fax: 02 41 / 477 05 199
mobil: 01 64 / 941 06 75
eMail: ***@schieferdecker.com

[Darrel O'Pry]

write perms to modules directory from drupal as web server user is
really hard for me to swallow....

any package managers like script should be run from the command line as
a privileged user. should do it's set job and be bullet proof.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Wow,

I am currently working on a system to automatically install modules.

- - Downloading and installing is only done on explicit request of the
administrator. I am not Microsoft.
- - Downloaded files are not less safe because they are downloaded via
PHP. There is currently no checksumming available and apart from
developers nobody looks into the code to see if it was tampered with.
- - There are a lot of drupal installations for development and testing,
that do not have the same security needs as production sites have.
- - Typo3 does it.
The only security issue remaining is having write access to the modules
directory. If the actual downloading and unpacking is done via a one
time cron job, this cron job could temporarily alter the access mode of
the target directory, minimizing the time that the directory is writable.
Greetings,
- --
Oswald Jaskolla
Ingenieurbüro Richard Schieferdecker
Kreuzherrenstraße 2
52062 Aachen
Tel.: 02 41 / 409 54 43
Fax: 02 41 / 477 05 199
mobil: 01 64 / 941 06 75
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFFZCSquinSHQ/4/T4RAsUmAJ4sTVuIs5eKpQgOCn9sZ6QvOub7YwCeN39w
pnLSOei74O+fQkwTaHF1sho=
=aIUQ
-----END PGP SIGNATURE-----

[Darren Oh]

Post by Darrel O'Pry
write perms to modules directory from drupal as web server user is
really hard for me to swallow....
any package managers like script should be run from the command line as
a privileged user. should do it's set job and be bullet proof.

Let's not forget that very few users use the command line to work
with Drupal. Let's also not make unnecessary assumptions about how an
automated module install or upgrade would work. The security issues
will be worked out if people share more ideas for how it can be done
than for how it can't be done.

Oswald was asking to collaborate. The negative reactions give the
impression that some people would rather work on their own. Not very
open source. Am I missing some history here?

[Larry Garfield]

Post by Darren Oh

Post by Darrel O'Pry
write perms to modules directory from drupal as web server user is
really hard for me to swallow....
any package managers like script should be run from the command line as
a privileged user. should do it's set job and be bullet proof.

Let's not forget that very few users use the command line to work
with Drupal. Let's also not make unnecessary assumptions about how an
automated module install or upgrade would work. The security issues
will be worked out if people share more ideas for how it can be done
than for how it can't be done.

I don't see how it can be done any other way.

The apache user should NOT have write access to code files.

To upgrade modules, you need write access to code files.

Therefore, you either need to temporarily give apache write access to code
files (which you can't do from within a web app running in apache, obviously)
or run the upgrade as a user that already has write access to them.

The Drupal 5 installer already says "please manually give apache write access"
before it runs (something you need a command line for, or a toggle in an FTP
client) and "hey, lock down access now" afterward for settings.php. That
kinda works for one file, but for an entire file tree I don't see how that
would work.

Post by Darren Oh
Oswald was asking to collaborate. The negative reactions give the
impression that some people would rather work on their own. Not very
open source. Am I missing some history here?

I think the "history" is "we're already working on it in way X, it's a hard
problem, please don't confuse us with way Y that has problems that way X is
already trying to address". Note that I am not part of the "we" in that
sentence; I am just summarizing what I've seen said in the past year. :-)
--
Larry Garfield AIM: LOLG42
***@garfieldtech.com ICQ: 6817012

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson

[Steven Wittens]

Post by Larry Garfield
Therefore, you either need to temporarily give apache write access to code
files (which you can't do from within a web app running in apache, obviously)
or run the upgrade as a user that already has write access to them.

Are we sure that you can't change the owner of the current process
through Apache? We can execute arbitrary shell commands, if needed.
The ideal solution could then be a script that can be invoked both
from the web and from the command-line.

Through the browser, it would ask for your local username/password,
and then perform the upgrade tasks (only from a very limited set of
commands, e.g. unpacking module files and copying them into the right
dir). From the command-line, it would just assume the current user is
the right one already.

Steven Wittens

[Larry Garfield]

Post by Steven Wittens
Are we sure that you can't change the owner of the current process
through Apache? We can execute arbitrary shell commands, if needed.
The ideal solution could then be a script that can be invoked both
from the web and from the command-line.

As far as I know, the only way to change the user the process is running as is
with the suexec apache module, in which case the process runs as the user
that owns the PHP script that is running. Otherwise, changing the owner of a
running process requires root access, something no web app should ever have.

Think about it from the other side: If you had a PHP script that could decide
to change the user it's running as to some arbitrary user, would YOU want it
on your server? I wouldn't.

Post by Steven Wittens
Through the browser, it would ask for your local username/password,
and then perform the upgrade tasks (only from a very limited set of
commands, e.g. unpacking module files and copying them into the right
dir). From the command-line, it would just assume the current user is
the right one already.
Steven Wittens

All I can think of here off the top of my head would be exec()ing su, but
again any shared host that makes that possible I don't want to touch.

Although, there are web control panels for the system itself, like webmin.
I'm not entirely sure how they do their thing. That may be something to look
into, but I still expect that any shared web host worth the money is going to
not allow a normal user to run anything like that, on principle.
--
Larry Garfield AIM: LOLG42
***@garfieldtech.com ICQ: 6817012

"If nature has made any one thing less susceptible than all others of
exclusive property, it is the action of the thinking power called an idea,
which an individual may exclusively possess as long as he keeps it to
himself; but the moment it is divulged, it forces itself into the possession
of every one, and the receiver cannot dispossess himself of it." -- Thomas
Jefferson

[Bèr Kessels]

Post by Larry Garfield
Although, there are web control panels for the system itself, like webmin.
  I'm not entirely sure how they do their thing.  That may be something to
look into, but I still expect that any shared web host worth the money is
going to not allow a normal user to run anything like that, on principle.

Webmin has its own webserver compiled, running on a different port. This
server runs with root(alike) permissions. When someone compromises such a
tool, he/she can do anything from within a browser.

I have already played with another option, being a single instance of
lighthttp running as a different user under a different UID, with root
permissions. That lighthttp serves a single hardened Drupal site Once logged
in on that site, certain modules can speak for example to sympal scripts,
e.g. to install a new multisite. OR simply exectute exec() tasks on the
server, as root.
However, Drupal was/is not secure enough *IMHO* to handle such a critical
task. E.g. too much issues with XSS and so were released last year, to serve
such a critical task.
But the idea works: Drupal can be used as a vhost management tool.

NOTE: Webmin is not very secure either (see the long list of security issues
on their site), but its architecture allows for better security
configurations.

Bèr

[Darrel O'Pry]

Post by Steven Wittens

Post by Larry Garfield
Therefore, you either need to temporarily give apache write access to code
files (which you can't do from within a web app running in apache, obviously)
or run the upgrade as a user that already has write access to them.

Are we sure that you can't change the owner of the current process
through Apache? We can execute arbitrary shell commands, if needed.
The ideal solution could then be a script that can be invoked both
from the web and from the command-line.
Through the browser, it would ask for your local username/password,
and then perform the upgrade tasks (only from a very limited set of
commands, e.g. unpacking module files and copying them into the right
dir). From the command-line, it would just assume the current user is
the right one already.
Steven Wittens

A well written & setuid script would do the job. You can call it
through system or exec I believe. There are many applications that use
this technique. The first that come to mind or qmail and majordomo. I
believe windows even supports a run as user option for its files.

Something to keep in mind about such a script is it should in no way
receive input from the browser. It should only be triggered, and should
still be paranoid about its config files... aka they're still owned by
the proper non-apache uid, don't contain any sort of exploit like
code...

At the end of the day I'm more concerned about other websites on shared
servers and partitioning risk between multiple sites on the same server.
As it is any code injection bugs in drupal could me a lost db, and the
real value of most websites is what's in the database.

.darrel.

[Steven Peck]

-----Original Message-----
From: development-***@drupal.org on behalf of Darrel O'Pry
Sent: Mon 11/27/2006 6:48 AM
To: ***@drupal.org
Subject: Re: [development] RFC: letting modules phone home to checkfor new releases

Post by Steven Wittens

Post by Larry Garfield
Therefore, you either need to temporarily give apache write access to code
files (which you can't do from within a web app running in apache, obviously)
or run the upgrade as a user that already has write access to them.

Are we sure that you can't change the owner of the current process
through Apache? We can execute arbitrary shell commands, if needed.
The ideal solution could then be a script that can be invoked both
from the web and from the command-line.
Through the browser, it would ask for your local username/password,
and then perform the upgrade tasks (only from a very limited set of
commands, e.g. unpacking module files and copying them into the right
dir). From the command-line, it would just assume the current user is
the right one already.
Steven Wittens

A well written & setuid script would do the job. You can call it
through system or exec I believe. There are many applications that use
this technique. The first that come to mind or qmail and majordomo. I
believe windows even supports a run as user option for its files.

Something to keep in mind about such a script is it should in no way
receive input from the browser. It should only be triggered, and should
still be paranoid about its config files... aka they're still owned by
the proper non-apache uid, don't contain any sort of exploit like
code...

At the end of the day I'm more concerned about other websites on shared
servers and partitioning risk between multiple sites on the same server.
As it is any code injection bugs in drupal could me a lost db, and the
real value of most websites is what's in the database.

.darrel.

and I'd be interested in seeing if it would work through IIS or Apache on Windows.

[Boris Mann]

Post by Darren Oh

Post by Darrel O'Pry
write perms to modules directory from drupal as web server user is
really hard for me to swallow....
any package managers like script should be run from the command line as
a privileged user. should do it's set job and be bullet proof.

Let's not forget that very few users use the command line to work
with Drupal. Let's also not make unnecessary assumptions about how an
automated module install or upgrade would work. The security issues
will be worked out if people share more ideas for how it can be done
than for how it can't be done.

It's likely that the best path is command line tools first -- "the market"
can build whatever GUI tools are desired on top of it. At some point, some
of those may make it to Drupal directly, but taking the responsibility for
remote upgrades is not something I would not necessarily want to expose the
community to.

Lots of people are still thinking about individual websites. Think
Fantastico and other host scripts that can drive command line hooks into
Drupal as privileged users.

Other options -- like pointing your website at different repositories
(supported by Derek's outlined framework) -- are also taken into
consideration.

Oswald was asking to collaborate. The negative reactions give the

Post by Darren Oh
impression that some people would rather work on their own. Not very
open source. Am I missing some history here?

No, you're not....you're missing frustration with a core (in both senses of
the word) group of people who have played nice and developed stuff directly
for the core of Drupal, undergoing excruciating levels of issue threads to
have all their work torn apart and improved upon. And then someone drops in
with an alternate system.

What would have been nice: Review the existing threads, post new issues
pointing to sandbox / module code, and integrate. That can still
happen....there is likely some good code to be integrated.

Secondly, Oswald didn't cover much in the nature of security. That's what I
put my 2¢ in on....security, security, security.

[Chris Johnson]

The whole shared-hosting without shell access and restricted Apache server
access to files has been well-analyzed and a solution implemented by Allie
Micka of Pajunas Interactive -- and she's a great Drupal developer to boot.
So if you want to solve this problem of creating install tools that will
really work in a secure fashion on shared-hosting providers, it would be
really worth the trouble to get her involved in the conversation. :-)

..chrisxj

[Bèr Kessels]

There already is an automated installer.... sigh.

Post by Oswald Jaskolla
I am currently working on a system to automatically install modules (i.e
download and unpack into the appropriate directory) and upgrade modules

http://cvs.drupal.org/viewcvs/drupal/contributions/modules/sympal_scripts/module_fetcher/

[Robert Douglass]

Bèr, have you ever written a tutorial on how to use the Sympal scripts?
I think a lot of people are unaware of what they can do and how you use
them. Sorry if you have, and I missed it =)

Post by Bèr Kessels
There already is an automated installer.... sigh.

Post by Oswald Jaskolla
I am currently working on a system to automatically install modules (i.e
download and unpack into the appropriate directory) and upgrade modules

http://cvs.drupal.org/viewcvs/drupal/contributions/modules/sympal_scripts/module_fetcher/

[Bèr Kessels]

Post by Robert Douglass
Bèr, have you ever written a tutorial on how to use the Sympal scripts?
I think a lot of people are unaware of what they can do and how you use
them. Sorry if you have, and I missed it =)

I haven't, mostly because not all stuff is made by me, and because it all is
still alpha-stage. Meaning: stuff works allright, but stuff can change Just
Like That, when I feel like it, or when I am in a bad mood :). Potentially
letting you loose loads of data, for example.

The closest you can get to a manual is this document of the commandline
reference, for once its ready:
http://webschuur.com/node/637

Its both a reference and a statusdoc, because I tell below the reference what
is implemented and what not.

Bèr