Discussion:
[Dovecot] Thinking Outside the Box - Extending IMAP
Marc Perkel
2007-05-13 13:55:59 UTC
Permalink
Here's some thoughts I'd like to throw out there. I know it's not
standard IMAP protocol but someone has to try new ideas first and I want
to see what people (Timo) think of this.

IMAP establishes a connection between the client and the server.
Wouldn't it be great if it could be a conduit to let custom Thunderbird
plugins talk to custom server application over the IMAP interface? For
example, personalized server settings. Suppose for example I want
Thunderbird to edit my server side white lists or black lists or any
other setting? Wouldn't it be nice if IMAP supported these changes? The
connection is made. It's a secure connection that's been authenticated.
Lets use it!

Here's my initial thoughts on this. Suppose we extended IMAP to include
an EXECUTE command as follows:

EXECUTE command parameter, parameter ....

On the server side is a config file that has the commands that execute
will allow and what programs they run. When the execute command is seen
by Dovecot then Dovecot runs the program in the list with the parameters
passed. For example, suppose there is a command to add a user to a
server side blacklist.

100 execute blacklist add joe at smith.com
100 ok

Dovecot would open a two way connection to the server application
allowing the client to talk to any application that is configured and
can send and receive text. The connection persists until the server end
terminates or the client closes the connection.

With a tool like this one can write generic applications easily that
would greatly expand what email clients can do interacting with the
server. Not only can setting be changes but you could interact with
server side calendars, pick up voice messages from phone systems, run
any sort of groupware, all over a generic IMAP connection with this
simple extension.

Example:

100 EXECUTE calendar
100 ok
100 list schedule today 8:00 10:00
100 8:00 make coffee
100 9:00 meeting with boss
100 9:30 Call Joe Blow at 415-555-1212
100 ok
100 quit
100 ok

One thing I'd like to use it for is an outgoing SMTP connection to send
outgoing email over IMAP. A session might look like this:

999 EXECUTE smtp
999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26 -0700
999 helo ctyme.com
999 250 darwin.ctyme.com Hello localhost [127.0.0.1]
999 mail from:marc at perkel.com
999 250 OK
999 rcpt to:dovecot at dovecot.org
999 250 Accepted
......
999 quit
999 OK

Who likes this idea?
Andy Shellam
2007-05-13 14:00:36 UTC
Permalink
I love this idea! I know it wouldn't be part of the standard IMAP
specification, and you'd probably have to have an extra capability such
as CUSTOMEXECUTE or something in the IMAP capabilities, which
Thunderbird would have to look for before it allows the use of the plugins.

As you suggested, this would make it so much easier to create an
Exchange-style account with Thunderbird/other client software.

Anyone else?

Andy.
Post by Marc Perkel
Here's some thoughts I'd like to throw out there. I know it's not
standard IMAP protocol but someone has to try new ideas first and I
want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server.
Wouldn't it be great if it could be a conduit to let custom
Thunderbird plugins talk to custom server application over the IMAP
interface? For example, personalized server settings. Suppose for
example I want Thunderbird to edit my server side white lists or black
lists or any other setting? Wouldn't it be nice if IMAP supported
these changes? The connection is made. It's a secure connection that's
been authenticated. Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to
EXECUTE command parameter, parameter ....
On the server side is a config file that has the commands that execute
will allow and what programs they run. When the execute command is
seen by Dovecot then Dovecot runs the program in the list with the
parameters passed. For example, suppose there is a command to add a
user to a server side blacklist.
100 execute blacklist add joe at smith.com
100 ok
Dovecot would open a two way connection to the server application
allowing the client to talk to any application that is configured and
can send and receive text. The connection persists until the server
end terminates or the client closes the connection.
With a tool like this one can write generic applications easily that
would greatly expand what email clients can do interacting with the
server. Not only can setting be changes but you could interact with
server side calendars, pick up voice messages from phone systems, run
any sort of groupware, all over a generic IMAP connection with this
simple extension.
100 EXECUTE calendar
100 ok
100 list schedule today 8:00 10:00
100 8:00 make coffee
100 9:00 meeting with boss
100 9:30 Call Joe Blow at 415-555-1212
100 ok
100 quit
100 ok
One thing I'd like to use it for is an outgoing SMTP connection to
999 EXECUTE smtp
999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26 -0700
999 helo ctyme.com
999 250 darwin.ctyme.com Hello localhost [127.0.0.1]
999 mail from:marc at perkel.com
999 250 OK
999 rcpt to:dovecot at dovecot.org
999 250 Accepted
......
999 quit
999 OK
Who likes this idea?
!DSPAM:37,4647190f343946551815980!
Marc Perkel
2007-05-13 14:18:15 UTC
Permalink
Post by Andy Shellam
I love this idea! I know it wouldn't be part of the standard IMAP
specification, and you'd probably have to have an extra capability
such as CUSTOMEXECUTE or something in the IMAP capabilities, which
Thunderbird would have to look for before it allows the use of the plugins.
As you suggested, this would make it so much easier to create an
Exchange-style account with Thunderbird/other client software.
Anyone else?
Andy.
Thanks Andy. You literally could do anything with it. All you have to do
is create a server and a client and let IMAP establish a connection.
Seems to me it would be pretty easy to code up. The executed program
would need environment variables set so that the server side would know
the login name and IP address of the connection in case you wanted to
restrict or log that.

So - who wants to code it up? I'll write a few server end pieces to say
edit text files to be used for server side block lists with Exim. Need
someone to code it into Dovecot, and need someone to write a Thunderbird
plugin that would add a "blacklist sender" button to send the command.

How hard can it be? Right?
Andy Shellam
2007-05-13 14:25:49 UTC
Permalink
I vote for making a custom Thunderbird plugin that will have a standard
"blacklist/whitelist" for the exact sender or the whole domain (like
Outlook 2003's junk handling.) E.g. I could black/whitelist
"marc at perkel.com" or "perkel.com." This would then send the "execute
blacklist <data>" command and the server-side will then be custom to
whatever you need.

E.g. you say your black/whitelists are in text files, mine are in a
PostgreSQL database, but the front-end client interface would be exactly
the same regardless.

It depends if Thunderbird plugins can interact with the IMAP connection,
as most of the plugins I've seen are all client-side functions.

Andy.
Post by Marc Perkel
Post by Andy Shellam
I love this idea! I know it wouldn't be part of the standard IMAP
specification, and you'd probably have to have an extra capability
such as CUSTOMEXECUTE or something in the IMAP capabilities, which
Thunderbird would have to look for before it allows the use of the plugins.
As you suggested, this would make it so much easier to create an
Exchange-style account with Thunderbird/other client software.
Anyone else?
Andy.
Thanks Andy. You literally could do anything with it. All you have to
do is create a server and a client and let IMAP establish a
connection. Seems to me it would be pretty easy to code up. The
executed program would need environment variables set so that the
server side would know the login name and IP address of the connection
in case you wanted to restrict or log that.
So - who wants to code it up? I'll write a few server end pieces to
say edit text files to be used for server side block lists with Exim.
Need someone to code it into Dovecot, and need someone to write a
Thunderbird plugin that would add a "blacklist sender" button to send
the command.
How hard can it be? Right?
!DSPAM:37,46471e50343942291814979!
Marc Perkel
2007-05-19 14:06:51 UTC
Permalink
OK - many of you responded very positively to this idea and I'd really
like to see it happen. My gut level feeling is that this wouldn't be too
hard to get working. So I'm willing to kick in $500 to make it happen.
And if any of you who like this idea agree then maybe others could
pledge something.

If this works I'm going to find a Thunderbird programmer to write some
simple plugins that will put up some buttons to send simple messages to
a server to do a few simple things like one line messages to add or
remove email addresses from white/black lists.

I believe this feature is the next big revolution in expanding email
functionality and I'm willing to put some bucks behind making it happen.
I'm not a rich guy but with my $500 maybe I can get someone interested.
Maybe other will be willing to kick in a few dollars or euros to make it
happen.

Thoughts?
Post by Andy Shellam
I love this idea! I know it wouldn't be part of the standard IMAP
specification, and you'd probably have to have an extra capability
such as CUSTOMEXECUTE or something in the IMAP capabilities, which
Thunderbird would have to look for before it allows the use of the plugins.
As you suggested, this would make it so much easier to create an
Exchange-style account with Thunderbird/other client software.
Anyone else?
Andy.
Post by Marc Perkel
Here's some thoughts I'd like to throw out there. I know it's not
standard IMAP protocol but someone has to try new ideas first and I
want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server.
Wouldn't it be great if it could be a conduit to let custom
Thunderbird plugins talk to custom server application over the IMAP
interface? For example, personalized server settings. Suppose for
example I want Thunderbird to edit my server side white lists or
black lists or any other setting? Wouldn't it be nice if IMAP
supported these changes? The connection is made. It's a secure
connection that's been authenticated. Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to
EXECUTE command parameter, parameter ....
On the server side is a config file that has the commands that
execute will allow and what programs they run. When the execute
command is seen by Dovecot then Dovecot runs the program in the list
with the parameters passed. For example, suppose there is a command
to add a user to a server side blacklist.
100 execute blacklist add joe at smith.com
100 ok
Dovecot would open a two way connection to the server application
allowing the client to talk to any application that is configured and
can send and receive text. The connection persists until the server
end terminates or the client closes the connection.
With a tool like this one can write generic applications easily that
would greatly expand what email clients can do interacting with the
server. Not only can setting be changes but you could interact with
server side calendars, pick up voice messages from phone systems, run
any sort of groupware, all over a generic IMAP connection with this
simple extension.
100 EXECUTE calendar
100 ok
100 list schedule today 8:00 10:00
100 8:00 make coffee
100 9:00 meeting with boss
100 9:30 Call Joe Blow at 415-555-1212
100 ok
100 quit
100 ok
One thing I'd like to use it for is an outgoing SMTP connection to
999 EXECUTE smtp
999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26 -0700
999 helo ctyme.com
999 250 darwin.ctyme.com Hello localhost [127.0.0.1]
999 mail from:marc at perkel.com
999 250 OK
999 rcpt to:dovecot at dovecot.org
999 250 Accepted
......
999 quit
999 OK
Who likes this idea?
!DSPAM:37,4647190f343946551815980!
Timo Sirainen
2007-05-19 14:47:37 UTC
Permalink
Post by Marc Perkel
I believe this feature is the next big revolution in expanding email
functionality and I'm willing to put some bucks behind making it happen.
I'm not a rich guy but with my $500 maybe I can get someone interested.
Maybe other will be willing to kick in a few dollars or euros to make it
happen.
Thoughts?
My thoughts:

- It should be pretty easy add new commands with plugins.

- Unless the commands are at least becoming an accepted standard (or
possibly otherwise widely used), I won't include them in the main
Dovecot tarball. But they can be listed in the Wiki of course.

- I'm not really interested in implementing it. I'd rather get all the
other things done that I already have in my TODO list. Also it looks
like I'm going to get paid for many of them (I'll probably write
something about this later when everything is clear).

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20070519/2da8c524/attachment.pgp
Marc Perkel
2007-05-19 16:03:27 UTC
Permalink
Post by Timo Sirainen
Post by Marc Perkel
I believe this feature is the next big revolution in expanding email
functionality and I'm willing to put some bucks behind making it happen.
I'm not a rich guy but with my $500 maybe I can get someone interested.
Maybe other will be willing to kick in a few dollars or euros to make it
happen.
Thoughts?
- It should be pretty easy add new commands with plugins.
- Unless the commands are at least becoming an accepted standard (or
possibly otherwise widely used), I won't include them in the main
Dovecot tarball. But they can be listed in the Wiki of course.
- I'm not really interested in implementing it. I'd rather get all the
other things done that I already have in my TODO list. Also it looks
like I'm going to get paid for many of them (I'll probably write
something about this later when everything is clear).
My offer is for anyone who will write it. Who wants to make $500?
Marc Perkel
2007-05-19 16:09:39 UTC
Permalink
Post by Timo Sirainen
Post by Marc Perkel
I believe this feature is the next big revolution in expanding email
functionality and I'm willing to put some bucks behind making it happen.
I'm not a rich guy but with my $500 maybe I can get someone interested.
Maybe other will be willing to kick in a few dollars or euros to make it
happen.
Thoughts?
- It should be pretty easy add new commands with plugins.
- Unless the commands are at least becoming an accepted standard (or
possibly otherwise widely used), I won't include them in the main
Dovecot tarball. But they can be listed in the Wiki of course.
- I'm not really interested in implementing it. I'd rather get all the
other things done that I already have in my TODO list. Also it looks
like I'm going to get paid for many of them (I'll probably write
something about this later when everything is clear).
That's acceptable not being a documented feature. Probably better. This
is somewhat just totally experimental with the idea that if it gets
exciting then the idea will become a standard and probably anything
developed for it could be changed to meet the standard.

Dave McGuire
2007-05-13 19:22:15 UTC
Permalink
Post by Marc Perkel
Here's some thoughts I'd like to throw out there. I know it's not
standard IMAP protocol but someone has to try new ideas first and I
want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server.
Wouldn't it be great if it could be a conduit to let custom
Thunderbird plugins talk to custom server application over the IMAP
interface? For example, personalized server settings. Suppose for
example I want Thunderbird to edit my server side white lists or
black lists or any other setting? Wouldn't it be nice if IMAP
supported these changes? The connection is made. It's a secure
connection that's been authenticated. Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to
EXECUTE command parameter, parameter ....
On the server side is a config file that has the commands that
execute will allow and what programs they run. When the execute
command is seen by Dovecot then Dovecot runs the program in the
list with the parameters passed. For example, suppose there is a
command to add a user to a server side blacklist.
100 execute blacklist add joe at smith.com
100 ok
...

I think this is a fantastic idea! I'm always opposed to
proprietary extensions to standardized protocols, but in this case,
it may be useful enough to gain acceptance. With Dovecot (as far as
I can tell) quickly becoming the dominant IMAP server, it probably
has the "push" to be able to pull this off.

Of course to make it useful you'd need some client
support...that's even more work than implementing it in the server,
multiple clients 'n all..

-Dave
--
Dave McGuire
Port Charlotte, FL
Timo Sirainen
2007-05-13 19:32:48 UTC
Permalink
Post by Marc Perkel
IMAP establishes a connection between the client and the server.
Wouldn't it be great if it could be a conduit to let custom Thunderbird
plugins talk to custom server application over the IMAP interface? For
example, personalized server settings. Suppose for example I want
Thunderbird to edit my server side white lists or black lists or any
other setting? Wouldn't it be nice if IMAP supported these changes? The
connection is made. It's a secure connection that's been authenticated.
Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to include
EXECUTE command parameter, parameter ....
I think this could be done in almost standard way with METADATA
extension
(http://www.ietf.org/internet-drafts/draft-daboo-imap-annotatemore-11.txt). It allows setting and getting any key=value pairs in the server (also for mailboxes). Then you could have a Dovecot plugin that listens for some specific keys and handles them in whatever way you want.
Post by Marc Perkel
100 execute blacklist add joe at smith.com
100 ok
METADATA doesn't really support adding/removing, but it could be kludged
something like this:

1 setmetadata "" "/x-blacklist/joe at smith.com" (value.priv "1")

Although the joe at smith.com can't contain "*" or "%" characters. If
that's needed then the string could be base64 encoded. Or the address
could be in the value and the key would contain some kind of a unique
ID.

Listing blacklist would work like:

1 getmetadata "/x-blacklist/* value.priv"

Anyway all of this would mean that if client implements a standard
METADATA browser you could edit these kinds of lists without any special
support from the client.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20070513/26471e8e/attachment.pgp
Marc Perkel
2007-05-14 03:37:15 UTC
Permalink
Timo,

Think outside the box on this. Create a new standard. Sometimes someone
has to make the move and lead the standard instead of following it. This
is new territory. Let's just try it and see where it goes.
Timo Sirainen
2007-05-13 19:46:37 UTC
Permalink
Post by Marc Perkel
One thing I'd like to use it for is an outgoing SMTP connection to send
999 EXECUTE smtp
999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26 -0700
999 helo ctyme.com
999 250 darwin.ctyme.com Hello localhost [127.0.0.1]
999 mail from:marc at perkel.com
999 250 OK
999 rcpt to:dovecot at dovecot.org
999 250 Accepted
......
999 quit
999 OK
And as for this, I think embedding SMTP inside IMAP session only
complicates things. LEMONADE support
(http://www.ietf.org/html.charters/lemonade-charter.html
http://www.lemonadeformobiles.com/index.html) is apparently being
implemented in all kinds of mobile clients and probably will get used a
lot more.

They're not using port 25 for sending messages, they're using port 587
which requires authentication (RFC 4409). So instead of trying to create
a new non-standard implementation, why not just use the submission port,
which is pretty much guaranteed to be the standard future way of sending
messages?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20070513/78900c5e/attachment.pgp
Gabriel Millerd
2007-05-14 00:36:02 UTC
Permalink
Post by Marc Perkel
Here's some thoughts I'd like to throw out there. I know it's not
standard IMAP protocol but someone has to try new ideas first and I want
to see what people (Timo) think of this.
Somewhat off topic.

I long time ago rather than making just a intranet/extranet website
that dealt out reports I made a imap daemon in perl that would also
serve the same report. Easily navigated virtual folders and subject
lines that helped with the content and attachments.

The backend would handle the authentication, new/read/flag, caching
the reports if needed, automatic folder subscription.

This was very useful for people that live in outlook, need to get
price lists on the road. The phrase 'awesome the noon call center
report is done' isn't something that you often think you would hear,
but is priceless.

I also made a imap front end to bugzilla in much the same manner for
people that couldn't handle bugzilla but that I wanted to be involved
(project managers / executives). Product Folder->Component
Subfolder->Threaded Bugs all coupled with a smtp2bugzilla gateway.

I find imap's ability to be secure and easy to integrate into the
personalized desktop. For some things its better than the browser.

Only thing that will get you in some of the above examples is a maniac
anti-virus administrator that wants all the outlook copies to rip
through your server frequently ... ouch.
--
Gabriel Millerd
Vulpes Velox
2007-05-14 02:25:27 UTC
Permalink
On Sun, 13 May 2007 06:55:59 -0700
Post by Marc Perkel
Here's some thoughts I'd like to throw out there. I know it's not
standard IMAP protocol but someone has to try new ideas first and I
want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server.
Wouldn't it be great if it could be a conduit to let custom
Thunderbird plugins talk to custom server application over the IMAP
interface? For example, personalized server settings. Suppose for
example I want Thunderbird to edit my server side white lists or
black lists or any other setting? Wouldn't it be nice if IMAP
supported these changes? The connection is made. It's a secure
connection that's been authenticated. Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to
EXECUTE command parameter, parameter ....
On the server side is a config file that has the commands that
execute will allow and what programs they run. When the execute
command is seen by Dovecot then Dovecot runs the program in the
list with the parameters passed. For example, suppose there is a
command to add a user to a server side blacklist.
100 execute blacklist add joe at smith.com
100 ok
I like the idea of having to and from added to the w/b list. That way
it would be way nicer to use with SpamAssassin.
Post by Marc Perkel
Dovecot would open a two way connection to the server application
allowing the client to talk to any application that is configured
and can send and receive text. The connection persists until the
server end terminates or the client closes the connection.
With a tool like this one can write generic applications easily
that would greatly expand what email clients can do interacting
with the server. Not only can setting be changes but you could
interact with server side calendars, pick up voice messages from
phone systems, run any sort of groupware, all over a generic IMAP
connection with this simple extension.
100 EXECUTE calendar
100 ok
100 list schedule today 8:00 10:00
100 8:00 make coffee
100 9:00 meeting with boss
100 9:30 Call Joe Blow at 415-555-1212
100 ok
100 quit
100 ok
One thing I'd like to use it for is an outgoing SMTP connection to
999 EXECUTE smtp
999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26
-0700 999 helo ctyme.com
999 250 darwin.ctyme.com Hello localhost [127.0.0.1]
999 mail from:marc at perkel.com
999 250 OK
999 rcpt to:dovecot at dovecot.org
999 250 Accepted
......
999 quit
999 OK
Who likes this idea?
I love that idea. Especially if it could run it through a specified
program for deliver as well as passing the username, from, and
password to the program. It may seem like a silly idea to pass that
to the program, but it does have a awesome usage. That way the
backend program can fetch the outgoing SMTP server through some
method and figure out how to send it. Been kicking around writing
something like that for QPSMTPD. It would be nice for with
squirrelmail.
Marc Perkel
2007-05-14 13:20:29 UTC
Permalink
Post by Vulpes Velox
On Sun, 13 May 2007 06:55:59 -0700
Post by Marc Perkel
Here's some thoughts I'd like to throw out there. I know it's not
standard IMAP protocol but someone has to try new ideas first and I
want to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server.
Wouldn't it be great if it could be a conduit to let custom
Thunderbird plugins talk to custom server application over the IMAP
interface? For example, personalized server settings. Suppose for
example I want Thunderbird to edit my server side white lists or
black lists or any other setting? Wouldn't it be nice if IMAP
supported these changes? The connection is made. It's a secure
connection that's been authenticated. Lets use it!
Here's my initial thoughts on this. Suppose we extended IMAP to
EXECUTE command parameter, parameter ....
On the server side is a config file that has the commands that
execute will allow and what programs they run. When the execute
command is seen by Dovecot then Dovecot runs the program in the
list with the parameters passed. For example, suppose there is a
command to add a user to a server side blacklist.
100 execute blacklist add joe at smith.com
100 ok
I like the idea of having to and from added to the w/b list. That way
it would be way nicer to use with SpamAssassin.
Post by Marc Perkel
Dovecot would open a two way connection to the server application
allowing the client to talk to any application that is configured
and can send and receive text. The connection persists until the
server end terminates or the client closes the connection.
With a tool like this one can write generic applications easily
that would greatly expand what email clients can do interacting
with the server. Not only can setting be changes but you could
interact with server side calendars, pick up voice messages from
phone systems, run any sort of groupware, all over a generic IMAP
connection with this simple extension.
100 EXECUTE calendar
100 ok
100 list schedule today 8:00 10:00
100 8:00 make coffee
100 9:00 meeting with boss
100 9:30 Call Joe Blow at 415-555-1212
100 ok
100 quit
100 ok
One thing I'd like to use it for is an outgoing SMTP connection to
999 EXECUTE smtp
999 220 darwin.ctyme.com ESMTP Exim 4.67 Sun, 13 May 2007 06:52:26
-0700 999 helo ctyme.com
999 250 darwin.ctyme.com Hello localhost [127.0.0.1]
999 mail from:marc at perkel.com
999 250 OK
999 rcpt to:dovecot at dovecot.org
999 250 Accepted
......
999 quit
999 OK
Who likes this idea?
I love that idea. Especially if it could run it through a specified
program for deliver as well as passing the username, from, and
password to the program. It may seem like a silly idea to pass that
to the program, but it does have a awesome usage. That way the
backend program can fetch the outgoing SMTP server through some
method and figure out how to send it. Been kicking around writing
something like that for QPSMTPD. It would be nice for with
squirrelmail.
I would think that when Dovecot executed the server end program that
they would pass through in environment variables all the user name and
connection info so that the server had that info for lookups.
Eric Rostetter
2007-05-14 03:43:22 UTC
Permalink
This post might be inappropriate. Click to display it.
Curtis Maloney
2007-05-18 01:17:53 UTC
Permalink
Post by Eric Rostetter
Post by Marc Perkel
Here's my initial thoughts on this. Suppose we extended IMAP to include
You'd be better off either making these "generic action verbs" like
"add" and "remove" (if each is meant to be self-sufficient, one-line
actions), or calling them "modes" (if the idea is to switch contexts
to some other protocol such as smtp, calendars, etc).
Generic action verbs are handy, but a LOT less flexible. They limit me to that
set of actions for _all_ possible extensions... and whilst a good set can be
found for many cases, the contortions it will force people into for cases that
don't fit... well...
Post by Eric Rostetter
Post by Marc Perkel
On the server side is a config file
Yes, that would be critical IMHO, or at least that there was some
centralized way to control it, and not Dovecot-specific if you wanted
the idea to spread past dovecot.
A general config form I like. My biggest concern on reading this was it opens
one enormous can of security worms - albeit it self-inflicted by admins,
essentially, Dovecot will still get blamed for making it easy.
Post by Eric Rostetter
Post by Marc Perkel
With a tool like this one can write generic applications easily that
would greatly expand what email clients can do interacting with the
server.
Sure, but why not just use a protocol for each, rather than layering
them on top of IMAP? Do you really think the overhead of opening a new
connection is so great? Debugging sure would be easier if they are
separate...
Whilst it does sort of make a step towards "IMAP as RPC", I think the nice thing
about IMAP is the basic protocol level is quite solid - how to send/parse
various data types, etc, is clear and clean, and is independent, pretty much, of
the command set you implement. Take ACAP as an example - same low-level
protocol, whole different command set.

Add to that the IMAP connection has already dealt with handshaking,
authentication, and encryption...

Now, from what I've seen of Dovecot's innards, it probably wouldn't be too
difficult to write your own protocol using the imap base, and build your own
daemon...
Post by Eric Rostetter
Post by Marc Perkel
Who likes this idea?
Not me. I think using different protocols for things...
Now, that doesn't mean dovecot couldn't support more protocols. Just
as it already supports multiple protocols (pop3 and imap) it could add
others... No reason not to, and not reason to piggy back them through
the IMAP session, IMHO.
As you've probably guessed, I'm somewhat in favour of this approach. Dovecot as
a base for more protocols. I like the IMAP base protocols, and Dovecot already
provides the parsing and command dispatch for such.

One day... One day I'll have time... and in that time I'll try to implement a
Calendar protocol, because I'm sick of all these polling-based implementations.
But that will have to be after I hook LigHTTPd and Apache into Dovecot Auth,
and .... all the billion other things I want to do :(

--
Curtis Maloney
cmaloney at cardgate.net
Johannes Berg
2007-05-14 12:16:47 UTC
Permalink
Post by Marc Perkel
Here's some thoughts I'd like to throw out there. I know it's not
standard IMAP protocol but someone has to try new ideas first and I want
to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server.
Wouldn't it be great if it could be a conduit to let custom Thunderbird
plugins talk to custom server application over the IMAP interface?
Why do you always want to stuff everything into IMAP?
Post by Marc Perkel
For
example, personalized server settings.
Isn't there some protocol similar to IMAP that solves this?
Post by Marc Perkel
Who likes this idea?
I strongly disagree with this idea. Too little definition, too much
server dependence, not portable across installations.

IMHO defining some behaviour that is so little related to the original
purpose of IMAP is counterproductive.

Besides, why do you need to do this with IMAP? There's a protocol that
supports all this already, it's called ssh. You can even tunnel pre-auth
IMAP tunnels through the same ssh connection :)

johannes
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20070514/89d161f7/attachment.pgp
andy.shellam-lists
2007-05-14 13:41:19 UTC
Permalink
I disagree about SSH.

Firstly, how do virtual users fit into your proposed setup?
Secondly, as a service provider to the general public, the absolute LAST thing
I want to be doing is opening up SSH access to my servers.

Mark has a valid point in that you have to connect to the server via IMAP to
get your mail, why should you have to have a second protocol to do other
things with the same mailbox? And why worry about a whole second set of
authentication when you've got a pre-authenticated connection ready and
waiting?

I agree it's not portable, and not ideal (ie. look at M$ Exchange's handling
of custom server features), but Timo's suggestion of using the METADATA
extension may strike the ideal balance between an extensible feature and the
IMAP standard.

Andy.
Post by Johannes Berg
Post by Marc Perkel
Here's some thoughts I'd like to throw out there. I know it's not
standard IMAP protocol but someone has to try new ideas first and I want
to see what people (Timo) think of this.
IMAP establishes a connection between the client and the server.
Wouldn't it be great if it could be a conduit to let custom Thunderbird
plugins talk to custom server application over the IMAP interface?
Why do you always want to stuff everything into IMAP?
Post by Marc Perkel
For
example, personalized server settings.
Isn't there some protocol similar to IMAP that solves this?
Post by Marc Perkel
Who likes this idea?
I strongly disagree with this idea. Too little definition, too much
server dependence, not portable across installations.
IMHO defining some behaviour that is so little related to the original
purpose of IMAP is counterproductive.
Besides, why do you need to do this with IMAP? There's a protocol that
supports all this already, it's called ssh. You can even tunnel pre-auth
IMAP tunnels through the same ssh connection :)
johannes
_________________________________________________________
DISCLAIMER

This e-mail was sent through a Mail Network server.
The Mail Network accepts no liability for it's content.
Eric Rostetter
2007-05-14 16:39:53 UTC
Permalink
Post by andy.shellam-lists
I disagree about SSH.
Good for you.
Post by andy.shellam-lists
Firstly, how do virtual users fit into your proposed setup?
You can setup a ssh tunnel on the server on any port. The user then
sets up to connect to that port. The authentication can be done anyway
you want, or not at all. We're not talking ssh logins to the server,
we're talking ssh tunneling.
Post by andy.shellam-lists
Secondly, as a service provider to the general public, the absolute LAST thing
I want to be doing is opening up SSH access to my servers.
Why? An SSH tunnel can be useful, and can increase rather than decrease
your security.

Now, if you meant logins, that would be a different story.
Post by andy.shellam-lists
Mark has a valid point in that you have to connect to the server via IMAP to
get your mail, why should you have to have a second protocol to do other
things with the same mailbox?
Because some of these things may not involve the same mailbox? And because
it makes for several smaller code bases which are then easier to secure,
audit, debug, etc. (instead of one massive code base that is harder to
secure, audit, debug, etc). And because then the authors only need to know
and understand one protocol, instead of trying to know and understand any
number of protocols that they may not use, have any interest in, etc.
And so that those who only want 1 protocol can install only that protocol,
and not have to worry about all the others? And, well, you should get
the idea by now.
Post by andy.shellam-lists
And why worry about a whole second set of
authentication when you've got a pre-authenticated connection ready and
waiting?
That was what the SSH connection was about I think.
Post by andy.shellam-lists
I agree it's not portable, and not ideal (ie. look at M$ Exchange's handling
of custom server features), but Timo's suggestion of using the METADATA
extension may strike the ideal balance between an extensible feature and the
IMAP standard.
Yes, but it only would handle some situations (maybe whitelist/blacklist)
and not others (like SMTP) so it isn't what the original question was
about really.
Post by andy.shellam-lists
Andy.
--
Eric Rostetter
The Department of Physics
The University of Texas at Austin

Go Longhorns!
Andy Shellam
2007-05-14 17:49:07 UTC
Permalink
Hi Eric,

SSH tunnelling requires an SSH login though, right? Even if that's via
PAM or other authentication method, it's another set of authentication
to set up, why not keep it all in the same system (i.e. Dovecot?) I
wouldn't allow SSH access connections to a public-facing server, mainly
for the fact I'd have to disclose the details to my user-base, which the
majority are non-techies, and it'd be harder to audit logins (there's
only my team that login to each server so it's easy to spot if someone's
intruding/attempting to.)

SSH is for tunneling a connection from the user's client machine to the
server which encrypts data over the connection. Why should the average
user have to download a piece of software (such as PuTTY), configure an
SSH session and tunnel, connect to SSH, then run their usual mail client
just to blacklist/whitelist a sender? It'd open up a world of
interesting problems for our support guys and is a massive overhead
that's unnecessary.

Why would a user logged into their mailbox want to make changes to
another user's mailbox? More importantly, why would I, as the
administrator, want to let them? If one user owns more than 1 mailbox,
then they just switch over to the other account - like myself, my
Thunderbird has both my -lists account and my normal account in the same
window. It takes one click to switch accounts.

I also disagree with your comment about code security. The fact it
requires another system, in whatever fashion, means there's a completely
different code-base which could pose even more of a security problem
rather than extending an existing code-base which is already well-known
(and guaranteed) for it's security anyway? As Timo is offering 1,000
EUR to find a security hole, I very much doubt he'd release an extension
that isn't secure. The way it's been suggested, Dovecot would run an
external program to set/get the metadata anyway.

The comment about another set of authentication was, if you've already
established an IMAP connection, why not use it for something else?
If you take it as you said with SSH authentication, the user has to have
authenticated via SSH first (to open the tunnel), then authenticates via
IMAP. Sure if your IMAP component recognises an SSH tunnelled
connection, and doesn't require authentication for it (not sure if
Dovecot can be set up like this), then that closes the door for using
your mail account on a different machine without having to set up the
SSH tunnel as well. Using the suggested metadata extension, any
IMAP-capable client could be configured to use it with no dependencies
on anything else.

The original question was suggesting that Dovecot could extend the IMAP
protocol to do other things (SMTP was an example, as was black- and
white-listing.)

Andy.
Post by Eric Rostetter
Post by andy.shellam-lists
I disagree about SSH.
Good for you.
Post by andy.shellam-lists
Firstly, how do virtual users fit into your proposed setup?
You can setup a ssh tunnel on the server on any port. The user then
sets up to connect to that port. The authentication can be done anyway
you want, or not at all. We're not talking ssh logins to the server,
we're talking ssh tunneling.
Post by andy.shellam-lists
Secondly, as a service provider to the general public, the absolute LAST thing
I want to be doing is opening up SSH access to my servers.
Why? An SSH tunnel can be useful, and can increase rather than decrease
your security.
Now, if you meant logins, that would be a different story.
Post by andy.shellam-lists
Mark has a valid point in that you have to connect to the server via IMAP to
get your mail, why should you have to have a second protocol to do other
things with the same mailbox?
Because some of these things may not involve the same mailbox? And because
it makes for several smaller code bases which are then easier to secure,
audit, debug, etc. (instead of one massive code base that is harder to
secure, audit, debug, etc). And because then the authors only need to know
and understand one protocol, instead of trying to know and understand any
number of protocols that they may not use, have any interest in, etc.
And so that those who only want 1 protocol can install only that protocol,
and not have to worry about all the others? And, well, you should get
the idea by now.
Post by andy.shellam-lists
And why worry about a whole second set of
authentication when you've got a pre-authenticated connection ready and
waiting?
That was what the SSH connection was about I think.
Post by andy.shellam-lists
I agree it's not portable, and not ideal (ie. look at M$ Exchange's handling
of custom server features), but Timo's suggestion of using the METADATA
extension may strike the ideal balance between an extensible feature and the
IMAP standard.
Yes, but it only would handle some situations (maybe whitelist/blacklist)
and not others (like SMTP) so it isn't what the original question was
about really.
Post by andy.shellam-lists
Andy.
Eric Rostetter
2007-05-14 18:11:43 UTC
Permalink
Post by Andy Shellam
Hi Eric,
SSH tunnelling requires an SSH login though, right?
No. SSH tunneling connects to a service, and that service can require
authentication if it so desires. SSH tunneling is just port forwarding
via SSH. It can be used to tunnel (and hence encrypt) pop3, imap, smtp,
popassd, telnet, ftp, etc.

The client/server applications using an SSH tunnel will carry out their own
authentication procedures (if any) the same way they would without
the encrypted tunnel.
Post by Andy Shellam
Even if that's via
PAM or other authentication method, it's another set of authentication
to set up
No, it uses the existing authentication already in the service, if any,
and no additional authentication is needed.
Post by Andy Shellam
why not keep it all in the same system (i.e. Dovecot?) I
I never moved it out. I only moved the encryption and some session
managment out.
Post by Andy Shellam
wouldn't allow SSH access connections to a public-facing server, mainly
for the fact I'd have to disclose the details to my user-base, which
If you are already running IMAP, then you are already exposing this.
Tunneling the IMAP via SSH exposes nothing more than the IMAP exposes,
except that you are using an SSH tunnel (meaning if they can find an
SSH vulnerability, then can crack you).
Post by Andy Shellam
the majority are non-techies, and it'd be harder to audit logins
(there's only my team that login to each server so it's easy to spot if
someone's intruding/attempting to.)
No. It may be harder to diagnose problems, but not harder to audit logins.
The logins are no different than before, only the transport is.
Post by Andy Shellam
SSH is for tunneling a connection from the user's client machine to the
server which encrypts data over the connection. Why should the average
user have to download a piece of software (such as PuTTY), configure an
SSH session and tunnel, connect to SSH, then run their usual mail
client just to blacklist/whitelist a sender? It'd open up a world of
interesting problems for our support guys and is a massive overhead
that's unnecessary.
Yes, that would be fairly rediculous. You'd have to have support in
your mail client or OS to make it worth doing.
Post by Andy Shellam
Why would a user logged into their mailbox want to make changes to
another user's mailbox?
Hopefully they wouldn't be allowed to. But one user might own multiple
mailboxes, and want to manipulate them all. Plus there could be shared
mailboxes, etc.
Post by Andy Shellam
More importantly, why would I, as the
administrator, want to let them? If one user owns more than 1 mailbox,
then they just switch over to the other account - like myself, my
Thunderbird has both my -lists account and my normal account in the
same window. It takes one click to switch accounts.
Not sure why you bring this up actually. But there are of course obscure
reasons (want to blacklist across all accounts, and not do it manually
for each one, etc).
Post by Andy Shellam
I also disagree with your comment about code security. The fact it
requires another system, in whatever fashion, means there's a
completely different code-base which could pose even more of a security
problem rather than extending an existing code-base which is already
well-known (and guaranteed) for it's security anyway? As Timo is
offering 1,000 EUR to find a security hole, I very much doubt he'd
release an extension that isn't secure. The way it's been suggested,
Dovecot would run an external program to set/get the metadata anyway.
Yes, and so, the external program would be another code base not controlled
by Timo, so... Well, you do the math.

There are various theories about small vs large code bases. But if you
look at security statistics, you will see small code bases almost always
win over big code bases, if all other things are held equal. Of course,
people can debate that until the cows come home, can you can't really prove
it one way or the other, and there are always exceptions.
Post by Andy Shellam
The comment about another set of authentication was, if you've already
established an IMAP connection, why not use it for something else?
That was the whole point of the SSH connection.
Post by Andy Shellam
If you take it as you said with SSH authentication, the user has to
have authenticated via SSH first (to open the tunnel), then
First, it was not my idea to use SSH. I was simply trying to correct
your assumption that SSH tunnels would require an authentication. Second,
they don't have to authenticate via SSH first, so your assumption is wrong.
Post by Andy Shellam
the SSH tunnel as well. Using the suggested metadata extension, any
IMAP-capable client could be configured to use it with no dependencies
on anything else.
As I pointed out, that only addresses issues like black/whitelist,
not issues like SMTP via IMAP.
Post by Andy Shellam
The original question was suggesting that Dovecot could extend the IMAP
protocol to do other things (SMTP was an example, as was black- and
white-listing.)
Yes, and no solution provided so far has addressed both of those in any
real way. The SSH one at least has potential to do so, whereas I see no
way the METADATA solution could possible address both of those issues.

METADATA is for storing and retrieving data, not for performing an action.
So, yes, it is a possible solution for things like updating a blacklist
or whitelist, but not for things like sendmail an e-mail (unless you want
to create a batch mechanism for doing so).
Post by Andy Shellam
Andy.
--
Eric Rostetter
The Department of Physics
The University of Texas at Austin

Go Longhorns!
Johannes Berg
2007-05-15 09:23:32 UTC
Permalink
Post by Eric Rostetter
You can setup a ssh tunnel on the server on any port. The user then
sets up to connect to that port. The authentication can be done anyway
you want, or not at all. We're not talking ssh logins to the server,
we're talking ssh tunneling.
Actually, I was thinking ssh logins :)

What I was thinking is this:

* user logs in via ssh
* ssh exposes a bunch of services like sftp and doesn't expose shell
(iirc this is possible), except the service wouldn't be sftp but
'imap'
* the imap service you provide is a pre-authenticated imap session so
that authentication/encryption is in ssh. I read my mail this way all
the time.
* the ssh also provides a few other services that you can use

Yes, this'll need a new client, but no, it won't require installing
putty, it'll require adding an ssh-transport to the imap protocol in
your mail client. Evolution is capable of doing that already by a
preauth tunnel command. But you need to modify the client anyway, so
this isn't a concern.

Thus, what you get is exactly what you want: a service that provides
multiple virtual services within a single existing connection. You ought
to be able to partition users within ssh, maybe not with openssh but
surely with some other implementation, virtual users should be possible
too, there's nothing in the ssh protocol that forbids this.

Now, at this point, you might want to support imap-only clients. But
that's probably easy too, just set up *dovecot* to do the authentication
(might sound odd first) and then write a dovecot PAM module that allows
ssh to authenticate against dovecot's auth service. Trivial, really.
Then you can *also* expose plain imap (with TLS) on another port and
therefore support legacy clients.

johannes
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 190 bytes
Desc: This is a digitally signed message part
Url : http://dovecot.org/pipermail/dovecot/attachments/20070515/55c5e34a/attachment.pgp
Eric Rostetter
2007-05-15 14:30:00 UTC
Permalink
Post by Johannes Berg
Post by Eric Rostetter
You can setup a ssh tunnel on the server on any port. The user then
sets up to connect to that port. The authentication can be done anyway
you want, or not at all. We're not talking ssh logins to the server,
we're talking ssh tunneling.
Actually, I was thinking ssh logins :)
Huh... Not sure why, but... This sounds like it would require both
ssh server modifications and e-mail client modifications. As such,
you may not get a lot of buy in to your idea. At that point, you're
almost half way to creating a new protocol anyway...
Post by Johannes Berg
* the imap service you provide is a pre-authenticated imap session so
that authentication/encryption is in ssh. I read my mail this way all
the time.
* the ssh also provides a few other services that you can use
Seems to me that instead of adding plugins to dovecot and the e-mail client,
you've added "subsystems" and plugins to the ssh server and e-mail client.
So you've just traded one server/client combination for another.
Post by Johannes Berg
Thus, what you get is exactly what you want: a service that provides
multiple virtual services within a single existing connection.
But since you've had to modify the client and server, why not just do this
with any old client/server protocol? What is so special about ssh in this
case?

I'd rather just tunnel the imap via ssh, and use the existing ssh tunnel
to do pre-auth for other services... Seems more trivial, as we're only
modifying the client, not the server... But what do I know/care. I've
always been happy with multiple protocols.

One reason I like multiple protocols, each with their own server code, is
that it scales well. I can put each service on a separate machine if I
need to, I can re-prioritize them individually, I can proxy them with ease,
etc. When you start jamming lots of protocols into one code base, not only
is it harder to audit and debug, it is harder to scale. Yes, you can still
scale with load balancers and such, but that introduces additional cost and
complexity which isn't needed when the services are isolated.

But, I guess not every one needs to scale, and not everyone is on the
server end (and yes, things always look different from the client end).
Post by Johannes Berg
johannes
--
Eric Rostetter
The Department of Physics
The University of Texas at Austin

Go Longhorns!
Loading...