Discussion:
WIN32 native ... lets start?!?
Joerg Hessdoerfer
2002-05-16 11:47:45 UTC
Permalink
Hi all,

I followed the various threads regarding this for some time now. My current
situation is:

I'm working at a company which does industrial automation, and does it's own
custom products. We try to be cross-platform, but it's a windoze world, as
far as most measurement devices or PLCs are concerned. We also employ
databases for various tasks (including simple ones as holding configuration
data, but also hammering production data into it at a rate of several hundred
records/sec.)
Well, we would *love* to use PostgreSQL in most our projects and products,
(and we do already use it in some), because it has proven to be very reliable
and quite fast.

So, I'm faced with using PostgreSQL on windows also (you can't always put a
Linux box besides). We do this using cygwin, but it's a bit painful ;-)
(although it works!).

Thinking about the hreads I read, it seems there are 2 obstacles to native PG
on W:

1.) no fork,
2.) no SYSV IPC

Ok, 1.) is an issue, but there's a fork() in MinGW, so it's 'just' going to
be a bit slow on new connections to the DB, right?? But this could be sorted
out once we *have* a native WIN32 build.

The second one's a bit harder, but... I'm currently trying to find time to do
a minimal implementation of SYSV IPC on WIN32 calls, just enough to get PG up
(doesn't need msg*() for example, right?).
As far as I understand it, we would not need to have IPC items around *after*
all backends and postmaster have gone away, or? Then there's no need for a
'daemon' process like in cygwin.

So, my route would be to get it to run *somehow* without paying attention to
speed and not to change much of the existing code, THEN see how we could get
rid of fork() on windows.

What do you guys think? Anyone up to join efforts? (I'll start the IPC thingy
anyway, as an exercise, and see where I'll end).

Greetings,
Joerg

P.s.: thanks for a great database system!!
--
Leading SW developer - S.E.A GmbH
Mail: ***@sea-gmbh.com
WWW: http://www.sea-gmbh.com

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Marc G. Fournier
2002-05-16 13:38:25 UTC
Permalink
Actually, take a look at the thread starting at:

http://archives.postgresql.org/pgsql-hackers/2002-05/msg00665.php

Right now, IMHO, the big show stopper is passing global variables to the
child processes in Windows ... the above thread talks about a method of
pulling together the global variables *cleanly* that Tom seems to feel
wouldn't add much in the way of long term maintenance headaches ... *and*,
as I understand it, would provide us with a means to use threading in
future developments if deemed appropriate ...
From what I read by those 'in the know' about Windows programming, if we
could centralize the global variables somewhat, using CreateProcess in
Windows shouldn't be a big deal, eliminiating the whole fork() headache
...
Hi all,
I followed the various threads regarding this for some time now. My current
I'm working at a company which does industrial automation, and does it's own
custom products. We try to be cross-platform, but it's a windoze world, as
far as most measurement devices or PLCs are concerned. We also employ
databases for various tasks (including simple ones as holding configuration
data, but also hammering production data into it at a rate of several hundred
records/sec.)
Well, we would *love* to use PostgreSQL in most our projects and products,
(and we do already use it in some), because it has proven to be very reliable
and quite fast.
So, I'm faced with using PostgreSQL on windows also (you can't always put a
Linux box besides). We do this using cygwin, but it's a bit painful ;-)
(although it works!).
Thinking about the hreads I read, it seems there are 2 obstacles to native PG
1.) no fork,
2.) no SYSV IPC
Ok, 1.) is an issue, but there's a fork() in MinGW, so it's 'just' going to
be a bit slow on new connections to the DB, right?? But this could be sorted
out once we *have* a native WIN32 build.
The second one's a bit harder, but... I'm currently trying to find time to do
a minimal implementation of SYSV IPC on WIN32 calls, just enough to get PG up
(doesn't need msg*() for example, right?).
As far as I understand it, we would not need to have IPC items around *after*
all backends and postmaster have gone away, or? Then there's no need for a
'daemon' process like in cygwin.
So, my route would be to get it to run *somehow* without paying attention to
speed and not to change much of the existing code, THEN see how we could get
rid of fork() on windows.
What do you guys think? Anyone up to join efforts? (I'll start the IPC thingy
anyway, as an exercise, and see where I'll end).
Greetings,
Joerg
P.s.: thanks for a great database system!!
--
Leading SW developer - S.E.A GmbH
WWW: http://www.sea-gmbh.com
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Bruce Momjian
2002-06-05 04:33:44 UTC
Permalink
OK, I think I am now caught up on the Win32/cygwin discussion, and would
like to make some remarks.

First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies working on Win32
ports of PostgreSQL. We always said there would not be PostgreSQL forks
if we were doing our job to meet user needs. Well, obviously, a number
of groups see a need for a better Win32 port and we aren't meeting that
need, so they are. I believe this is one of the few cases where groups
are going out on their own because we are falling behind.

So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.

INSTALLER
---------

We clearly need an installer that is zero-hassle for users. We need to
decide on a direction for this.

GUI
---

We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?

BINARY
------

This is the big daddy. It is broken down into several sections:

FORK()

How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?

THREADING

Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)

IPC

We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?

ENVIRONMENT

Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?

---------------------------------------------------------------------------

As a roadmap, it would be good to get consensus on as many of these
items as possible so people can start working in these areas. We can
keep a web page of decisions we have made to help rally developers to
the project.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Mark kirkwood
2002-06-05 07:38:52 UTC
Permalink
Post by Bruce Momjian
OK, I think I am now caught up on the Win32/cygwin discussion, and would
like to make some remarks.
First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies working on Win32
ports of PostgreSQL. We always said there would not be PostgreSQL forks
if we were doing our job to meet user needs. Well, obviously, a number
of groups see a need for a better Win32 port and we aren't meeting that
need, so they are. I believe this is one of the few cases where groups
are going out on their own because we are falling behind.
So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.
INSTALLER
---------
We clearly need an installer that is zero-hassle for users. We need to
decide on a direction for this.
GUI
---
We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?
BINARY
------
FORK()
How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?
THREADING
Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)
IPC
We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?
ENVIRONMENT
Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?
---------------------------------------------------------------------------
As a roadmap, it would be good to get consensus on as many of these
items as possible so people can start working in these areas. We can
keep a web page of decisions we have made to help rally developers to
the project.
--
Bruce Momjian | http://candle.pha.pa.us
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026
---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
Is it worth looking at how the mysql crowd did their win32 port -
(or is that intrinsically a _bad_thing_ to do..) ?

(I am guessing that is why their sources requires c++ ....)

regards

Mark


---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Bruce Momjian
2002-06-05 15:18:29 UTC
Permalink
Post by Mark kirkwood
Is it worth looking at how the mysql crowd did their win32 port -
(or is that intrinsically a _bad_thing_ to do..) ?
(I am guessing that is why their sources requires c++ ....)
Absolutely worth seeing how MySQL does it. They use cygwin, and I
assume they aren't seeing the fork() issue because they are threaded,
and perhaps they don't use SysV IPC like we do.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Igor Kovalenko
2002-06-05 19:32:13 UTC
Permalink
I might be naive here, but would not proper threading model remove the need
for fork() altogether? On both Unix and Win32? Should not be too hard to
come up with abstraction which encapsulates POSIX, BeOS and Win32 threads...
I am not sure how universal POSIX threads are by now. Any important Unix
platforms which don't support them yet?

This has downside of letting any bug to kill the whole thing. On the bright
side, performance should be better on some platforms (note however, Apache
group still can't come up with implementation of threaded model which would
provide better performance than forked or other models). The need to deal
with possibility of 'alien' postmaster running along with orphaned backends
would also be removed since there would be only one process.

Issue of thread safety of code will come up undoubtedly and some things will
probably have to be revamped. But in long term this is probably best way if
you want to have efficient and uniform Unix AND Win32 implementations.

I am not too familiar with Win32. Speaking about POSIX threads, it would be
something like a thread pool with low & high watermarks. Main thread would
handle thread pool and hand over requests to worker threads (blocked on
condvar). How does that sound?

-- igor

----- Original Message -----
From: "Bruce Momjian" <***@candle.pha.pa.us>
To: "PostgreSQL-development" <pgsql-***@postgresql.org>
Sent: Tuesday, June 04, 2002 11:33 PM
Subject: [HACKERS] Roadmap for a Win32 port
Post by Bruce Momjian
OK, I think I am now caught up on the Win32/cygwin discussion, and would
like to make some remarks.
First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies working on Win32
ports of PostgreSQL. We always said there would not be PostgreSQL forks
if we were doing our job to meet user needs. Well, obviously, a number
of groups see a need for a better Win32 port and we aren't meeting that
need, so they are. I believe this is one of the few cases where groups
are going out on their own because we are falling behind.
So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.
INSTALLER
---------
We clearly need an installer that is zero-hassle for users. We need to
decide on a direction for this.
GUI
---
We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?
BINARY
------
FORK()
How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?
THREADING
Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)
IPC
We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?
ENVIRONMENT
Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?
--------------------------------------------------------------------------
-
Post by Bruce Momjian
As a roadmap, it would be good to get consensus on as many of these
items as possible so people can start working in these areas. We can
keep a web page of decisions we have made to help rally developers to
the project.
--
Bruce Momjian | http://candle.pha.pa.us
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026
---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/users-lounge/docs/faq.html
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Bruce Momjian
2002-06-05 20:05:02 UTC
Permalink
Post by Igor Kovalenko
I might be naive here, but would not proper threading model remove the need
for fork() altogether? On both Unix and Win32? Should not be too hard to
come up with abstraction which encapsulates POSIX, BeOS and Win32 threads...
I am not sure how universal POSIX threads are by now. Any important Unix
platforms which don't support them yet?
This has downside of letting any bug to kill the whole thing. On the bright
side, performance should be better on some platforms (note however, Apache
group still can't come up with implementation of threaded model which would
provide better performance than forked or other models). The need to deal
with possibility of 'alien' postmaster running along with orphaned backends
would also be removed since there would be only one process.
Issue of thread safety of code will come up undoubtedly and some things will
probably have to be revamped. But in long term this is probably best way if
you want to have efficient and uniform Unix AND Win32 implementations.
I am not too familiar with Win32. Speaking about POSIX threads, it would be
something like a thread pool with low & high watermarks. Main thread would
handle thread pool and hand over requests to worker threads (blocked on
condvar). How does that sound?
Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Thomas Lockhart
2002-06-05 22:02:33 UTC
Permalink
...
Post by Bruce Momjian
Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.
I would think that we would build on our strengths of having a fork/exec
model for separate clients. A threaded model *could* benefit individual
clients who are doing queries on multiprocessor servers, and I would be
supportive of efforts to enable that.

But the requirements for that may be less severe than for managing
multiple clients within the same process, and imho there is not strong
requirement to enable the latter for our current crop of well supported
targets. If it came for free then great, but if it came with a high cost
then the choice is not as obvious. It is also not a *requirement* if we
were instead able to do the multiple threads for a single client
scenerio first.

- Thomas

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Steve Howe
2002-06-06 01:05:11 UTC
Permalink
Hello Thomas,

Wednesday, June 5, 2002, 7:02:33 PM, you wrote:

TL> ...
Post by Bruce Momjian
Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.
TL> I would think that we would build on our strengths of having a fork/exec
TL> model for separate clients. A threaded model *could* benefit individual
TL> clients who are doing queries on multiprocessor servers, and I would be
TL> supportive of efforts to enable that.
Just a note - this is also the solution adopted by Interbase/Firebird
and it seems interesting. They already had the same problems
PostgreSQL has been under today.
Those interested in read about Interbase's architeture, please refer
to http://community.borland.com/article/0,1410,23217,00.html.
"Classic" is the fork() model, and the "SuperServer" is the threaded
model.
-------------
Best regards,
Steve Howe mailto:***@carcass.dhs.org


---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Jon Franz
2002-06-05 22:50:46 UTC
Permalink
One note: SGI developers discovered they could get amazing performance using
as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
utilizing thier 'state threads' model. Please see:

http://state-threads.sourceforge.net/docs/st.html

Thus, on platforms where it can be supported, we should keep in mind that a
hybrid multiprocess/multithreaded postgresql might be the fastest
solution...


----- Original Message -----
From: "Bruce Momjian" <***@candle.pha.pa.us>
To: "Igor Kovalenko" <***@motorola.com>
Cc: "PostgreSQL-development" <pgsql-***@postgresql.org>
Sent: Wednesday, June 05, 2002 4:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Bruce Momjian
Post by Igor Kovalenko
I might be naive here, but would not proper threading model remove the need
for fork() altogether? On both Unix and Win32? Should not be too hard to
come up with abstraction which encapsulates POSIX, BeOS and Win32 threads...
I am not sure how universal POSIX threads are by now. Any important Unix
platforms which don't support them yet?
This has downside of letting any bug to kill the whole thing. On the bright
side, performance should be better on some platforms (note however, Apache
group still can't come up with implementation of threaded model which would
provide better performance than forked or other models). The need to deal
with possibility of 'alien' postmaster running along with orphaned backends
would also be removed since there would be only one process.
Issue of thread safety of code will come up undoubtedly and some things will
probably have to be revamped. But in long term this is probably best way if
you want to have efficient and uniform Unix AND Win32 implementations.
I am not too familiar with Win32. Speaking about POSIX threads, it would be
something like a thread pool with low & high watermarks. Main thread would
handle thread pool and hand over requests to worker threads (blocked on
condvar). How does that sound?
Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.
--
Bruce Momjian | http://candle.pha.pa.us
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026
---------------------------(end of broadcast)---------------------------
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Neil Conway
2002-06-06 00:05:44 UTC
Permalink
On Wed, 5 Jun 2002 18:50:46 -0400
Post by Jon Franz
One note: SGI developers discovered they could get amazing performance using
as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
utilizing thier 'state threads' model.
I think ST is designed for network I/O-bound apps -- last I checked,
disk I/O will still block an entire ST process. While you can get around
that by using another process to do disk I/O, it sounds like ST won't be
that useful.

However, Chris KL. (I believe) raised the idea of using POSIX AIO for
PostgreSQL. Without having looked into it extensively, this technique
sounds promising. Perhaps someone who has looked into this further
(e.g. someone from Redhat) can comment?

Cheers,

Neil
--
Neil Conway <***@rogers.com>
PGP Key ID: DB3C29FC

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Bruce Momjian
2002-06-06 00:53:15 UTC
Permalink
Post by Neil Conway
On Wed, 5 Jun 2002 18:50:46 -0400
Post by Jon Franz
One note: SGI developers discovered they could get amazing performance using
as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
utilizing thier 'state threads' model.
I think ST is designed for network I/O-bound apps -- last I checked,
disk I/O will still block an entire ST process. While you can get around
that by using another process to do disk I/O, it sounds like ST won't be
that useful.
However, Chris KL. (I believe) raised the idea of using POSIX AIO for
PostgreSQL. Without having looked into it extensively, this technique
sounds promising. Perhaps someone who has looked into this further
(e.g. someone from Redhat) can comment?
I know Red Hat is interested in AIO. Only a few OS's support it so it
was hard to get exited about it at the time, but with threading, a
AIO-specific module could be attempted.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Igor Kovalenko
2002-06-06 02:54:29 UTC
Permalink
I think SGI gets amazing performance because they have very good (efficient)
synchronisation primitives on SGI. Some proprietary light-weight mutexes.
Using threaded or mixed model just by itself is not going to do a miracle.
Threads will save you some context switch time, but that will probably
translate into lower CPU usage rather than performance boost. And if your
mutexes are not fast or awkwardly implemented (say Linux), it might be even
worse. Apache is not all that fast on Linux as on SGI, whatever model you
chose. I also doubt that purely threaded model would be slower than mixed
one.

Now about the AIO model. It is useful when you need to do something else
while I/O requests are being processed as long as platform does it in some
useful way. If all you can do is to submit requests and keep sitting in
select/poll then AIO does not buy you anything you can't get by just using
threaded model. However, if you can tag the requests and set up
notifications, then few I/O threads could handle relatively large number of
requests from different clients. Note, this means you don't have any
association between clients and servers at all, there is pool of generic I/O
threads which serve requests from whoever they come. It saves system
resources and scales very well. It also provides interesting possibilities
for fault recovery - since handlers are generic all the state information
would have to be kept in some kind of global context area. That area can be
saved into persistent memory or dumped onto disk and *recovered* after a
forced restart. Server and library could be designed in such a way that
clients may continue where they left with a recoverable error.

In POSIX AIO model you can tag requests and set up notifications via
synchronous signals. You wait for them *synchronously* in 'waiter' thread
via sigwaitinfo() and avoid the headache of asynchronous signals hitting you
any time... Unfortunately on some platforms (Solaris) the depth of
synchronous signal queue is fixed at magic value 32 (and not adjustable).
This may not be a problem if you're sure that waiting thread will be able to
drain the queue faster than it gets filled with notifications... but I'm not
sure there is a portable way to guarantee that, so you need to check for
overloads and handle them... that complicates things. On Solaris you also
need a mile of compiler/linker switches to even get this scheme to work and
I am afraid other platforms may not support it at all (but then again, they
may not support AIO to begin with).

And speaking about getting best of all worlds. Note how Apache spent nearly
3 years developing their portable Multi-Processing Modules scheme. What they
got for that is handful of models neither of which perform noticeably better
than original pre-fork() model. Trying to swallow all possible ways to
handle things on all possible platforms usually does not produce very fast
code. It tends to produce very complex code with mediocre performance and
introduces extra complexity into configuration process. If you consider all
that was done mostly to support Win32, one might doubt if it was worth the
while.

What I am trying to say is, extra complexity in model to squeeze few percent
of performance is not a wise investment of time and efforts. On Win32 you
don't really compete in terms of performance. You compete in terms of
easyness and features. Spend 3 years trying to support Windows and Unix in
most optimal way including all subvariants of Unix ... meanwhile MSFT will
come up with some bundled SQL server. It probably will have more features
since they will spend time doing features rather than inventing a model to
support gazillion of platforms. Chances are, it will be faster too - due to
better integration with OS and better compiler.

I am not in position to tell you what to do guys. But if I was asked, I'd
say supporting Win32 is only worth it if it comes as a natural result of a
simple, coherent and uniform model applied to Unix. Threaded model may not
have as much inherent stability as forked/mixed, but it has inherent
simplicity and better Unix/Windows/BeOS portability. It can be done faster
and simpler code will make work on features easier.

Regards,
- Igor

"There are 2 ways to design an efficient system - first is to design it so
complex that there are no obvious deficiencies, second is to design it so
simple that there are obviously no deficiencies. Second way is much harder"
(author unknown to me)


----- Original Message -----
From: "Neil Conway" <***@klamath.dyndns.org>
To: "Jon Franz" <***@one.net>
Cc: <pgsql-***@postgresql.org>
Sent: Wednesday, June 05, 2002 7:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Neil Conway
On Wed, 5 Jun 2002 18:50:46 -0400
Post by Jon Franz
One note: SGI developers discovered they could get amazing performance using
as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
utilizing thier 'state threads' model.
I think ST is designed for network I/O-bound apps -- last I checked,
disk I/O will still block an entire ST process. While you can get around
that by using another process to do disk I/O, it sounds like ST won't be
that useful.
However, Chris KL. (I believe) raised the idea of using POSIX AIO for
PostgreSQL. Without having looked into it extensively, this technique
sounds promising. Perhaps someone who has looked into this further
(e.g. someone from Redhat) can comment?
Cheers,
Neil
--
PGP Key ID: DB3C29FC
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Bruce Momjian
2002-06-06 04:59:36 UTC
Permalink
Post by Igor Kovalenko
I think SGI gets amazing performance because they have very good (efficient)
synchronization primitives on SGI. Some proprietary light-weight mutexes.
Using threaded or mixed model just by itself is not going to do a miracle.
Threads will save you some context switch time, but that will probably
translate into lower CPU usage rather than performance boost. And if your
mutexes are not fast or awkwardly implemented (say Linux), it might be even
worse. Apache is not all that fast on Linux as on SGI, whatever model you
chose. I also doubt that purely threaded model would be slower than mixed
one.
Let me throw out an idea. I have been mentioning full fork, light
fork(copy globals only), and threading as possible solutions.

Another idea uses neither threading nor copying. It is the old system
we used before I removed exec() from our code. We used to pass the
database name as an argument to an exec'ed postgres binary that
continued with the database connection.

We removed the exec, then started moving what we could into the
postmaster so each backend didn't need to do the initialization.

One solution is to return to that for Win32 only, so instead of doing:

initialization()
want for connection()
fork backend()

we do for Win32:

want for connection()
exec backend()
initialization()

It wouldn't be hard to do. We would still do CreateProcess rather than
CreateThread, but it eliminates the fork/threading issues. We don't
know the database before the connection arrives, so we don't do a whole
lot of initialization.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Jan Wieck
2002-06-06 13:35:14 UTC
Permalink
Post by Bruce Momjian
Let me throw out an idea. I have been mentioning full fork, light
fork(copy globals only), and threading as possible solutions.
Another idea uses neither threading nor copying. It is the old system
we used before I removed exec() from our code. We used to pass the
database name as an argument to an exec'ed postgres binary that
continued with the database connection.
We removed the exec, then started moving what we could into the
postmaster so each backend didn't need to do the initialization.
initialization()
want for connection()
fork backend()
want for connection()
exec backend()
initialization()
Summarizes pretty much what we discussed Monday on the phone.
Except that the postmaster still has to initialize the shared
memory and other stuff. It's just that the backends and
helper processes need to reinitialize themself (attach).
Post by Bruce Momjian
It wouldn't be hard to do. We would still do CreateProcess rather than
CreateThread, but it eliminates the fork/threading issues. We don't
know the database before the connection arrives, so we don't do a whole
lot of initialization.
All I see so far is the reading of the postgresql.conf, the
pg_hba.conf and the password files. Nothing fancy and the
postmaster could easily write out a binary content only file
that the backends then read, eliminating the parsing
overhead.

The bad news is that Tom is right. We did a terrible job in
using the new side effect, that the shared memory segment is
at the same address in all forked processes, after removing
the need to reattach.

In detail the XLog code, the FreeSpaceMap code and the
"shared memory" hashtable code now use pointers, located in
shared memory. For the XLog and FreeSpace code this is
understandable, because they where developed under the fork()
only model. But the dynahash code used offsets only until
v7.1!

All three (no claim that that's all) make it impossible to
ever have someone attaching to the shared memory from the
outside. So with these moves we made the shared memory a
"Postmaster and children" only thing. Raises the question,
why we need an IPC key at all any more.

Anyhow, looks as if I can get that fork() vs. fork()+exec()
feature done pretty soon. It'll be controlled by another
Postmaster commandline switch. After cleaning up the mess I
did to get it working quick, I'll provide a patch for
discussion.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== ***@Yahoo.com #



---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Bruce Momjian
2002-06-06 15:06:18 UTC
Permalink
Post by Jan Wieck
Post by Bruce Momjian
initialization()
want for connection()
fork backend()
want for connection()
exec backend()
initialization()
Summarizes pretty much what we discussed Monday on the phone.
Except that the postmaster still has to initialize the shared
memory and other stuff. It's just that the backends and
helper processes need to reinitialize themself (attach).
Yes, obviously I simplified, and I do believe our optimizations are
helping on Unix. It is just that I think for Win32 the fork is more
harmful than removing those optimizations.

One thing that may not have been clear is that we don't need to play
with globals at all. We just pass whatever info we want to the child
via command-line arguments, rather than shared memory.
Post by Jan Wieck
Post by Bruce Momjian
It wouldn't be hard to do. We would still do CreateProcess rather than
CreateThread, but it eliminates the fork/threading issues. We don't
know the database before the connection arrives, so we don't do a whole
lot of initialization.
All I see so far is the reading of the postgresql.conf, the
pg_hba.conf and the password files. Nothing fancy and the
postmaster could easily write out a binary content only file
that the backends then read, eliminating the parsing
overhead.
Yes, that is clearly possible. Another option is to just write out a
no-comments, no-whitespace version of each file and just have the
backends read those. The advantage is that we can use the same code to
read them, and I don't think it would be any slower than a binary file.
Post by Jan Wieck
The bad news is that Tom is right. We did a terrible job in
using the new side effect, that the shared memory segment is
at the same address in all forked processes, after removing
the need to reattach.
In detail the XLog code, the FreeSpaceMap code and the
"shared memory" hashtable code now use pointers, located in
shared memory. For the XLog and FreeSpace code this is
understandable, because they where developed under the fork()
only model. But the dynahash code used offsets only until
v7.1!
All three (no claim that that's all) make it impossible to
ever have someone attaching to the shared memory from the
outside. So with these moves we made the shared memory a
"Postmaster and children" only thing. Raises the question,
why we need an IPC key at all any more.
Well, we could force shmat() to bind to the same address, but I suspect
that might fail in some cases.
Post by Jan Wieck
Anyhow, looks as if I can get that fork() vs. fork()+exec()
feature done pretty soon. It'll be controlled by another
Postmaster commandline switch. After cleaning up the mess I
did to get it working quick, I'll provide a patch for
discussion.
Yes, very little impact. We then need someone to do some Win32 timings
to see if things have improved. As Tom mentioned, we need some hard
numbers for these things. In fact, I would like a Win32 test that takes
our code and compares fork(), then exit(), with CreateProcess(), exit().
It doesn't have create a db session, but I would like to see some
timings to know what we are gaining. Heck, time CreateThread too and
let's see what that shows.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly
Jon Franz
2000-09-28 03:53:44 UTC
Permalink
Yeah: ST is designed for network apps, and its for network bound apps that
you
gain the most performance - but by using it to allow
a child process to hold multiple connections and accept/return data to
those connections simultaneously, I forsaw a potential performance
improvement...
*shrug* Most connections remain idle most of thier life... yes?

the SGI folks developed this library as part of a project to make apache
faster (http://aap.sourceforge.net/) - multiple child
processes as normal, but allowed multiple connections per child.

And although the performance improvements they got were greatest on irix,
performance was improved upto 70% on linux. Some of this was from QSC
(http://aap.sourceforge.net/mod_qsc.html) , however...

just some food for thought.


----- Original Message -----
From: "Neil Conway" <***@klamath.dyndns.org>
To: "Jon Franz" <***@one.net>
Cc: <pgsql-***@postgresql.org>
Sent: Wednesday, June 05, 2002 8:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Neil Conway
On Wed, 5 Jun 2002 18:50:46 -0400
Post by Jon Franz
One note: SGI developers discovered they could get amazing performance using
as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
utilizing thier 'state threads' model.
I think ST is designed for network I/O-bound apps -- last I checked,
disk I/O will still block an entire ST process. While you can get around
that by using another process to do disk I/O, it sounds like ST won't be
that useful.
However, Chris KL. (I believe) raised the idea of using POSIX AIO for
PostgreSQL. Without having looked into it extensively, this technique
sounds promising. Perhaps someone who has looked into this further
(e.g. someone from Redhat) can comment?
Cheers,
Neil
--
PGP Key ID: DB3C29FC
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Nicolas Bazin
2002-06-06 00:50:09 UTC
Permalink
Yes I proposed to use the GNU Pth library instead. It's an event
demultiplexer just like the sgi library, but has a posix thread interface.
This architecture is actually the more robust and also the more scalable. On
a single processor server, you don't have the multi-thread synchronization
and context switching overhead and you also take full advantage of
multi-processor servers when you create several processes. Plus you have
much less concern about global variables.

Also for those concerned about the licence of this library here is an
abstract of it:
"The author places this library under the LGPL to make sure that it
can be used both commercially and non-commercially provided that
modifications to the code base are always donated back to the official
code base under the same license conditions. Please keep in mind that
especially using this library in code not staying under the GPL or
the LGPL _is_ allowed and that any taint or license creap into code
that uses the library is not the authors intention. It is just the
case that _including_ this library into the source tree of other
applications is a little bit more inconvinient because of the LGPL.
But it has to be this way for good reasons. And keep in mind that
inconvinient doesn't mean not allowed or even impossible."

So it can be used in both commercial and non commercial project.


----- Original Message -----
From: "Jon Franz" <***@one.net>
To: <pgsql-***@postgresql.org>
Sent: Thursday, June 06, 2002 8:50 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Jon Franz
One note: SGI developers discovered they could get amazing performance using
as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
http://state-threads.sourceforge.net/docs/st.html
Thus, on platforms where it can be supported, we should keep in mind that a
hybrid multiprocess/multithreaded postgresql might be the fastest
solution...
----- Original Message -----
Sent: Wednesday, June 05, 2002 4:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Bruce Momjian
Post by Igor Kovalenko
I might be naive here, but would not proper threading model remove the
need
Post by Bruce Momjian
Post by Igor Kovalenko
for fork() altogether? On both Unix and Win32? Should not be too hard to
come up with abstraction which encapsulates POSIX, BeOS and Win32
threads...
Post by Bruce Momjian
Post by Igor Kovalenko
I am not sure how universal POSIX threads are by now. Any important Unix
platforms which don't support them yet?
This has downside of letting any bug to kill the whole thing. On the
bright
Post by Bruce Momjian
Post by Igor Kovalenko
side, performance should be better on some platforms (note however,
Apache
Post by Bruce Momjian
Post by Igor Kovalenko
group still can't come up with implementation of threaded model which
would
Post by Bruce Momjian
Post by Igor Kovalenko
provide better performance than forked or other models). The need to
deal
Post by Bruce Momjian
Post by Igor Kovalenko
with possibility of 'alien' postmaster running along with orphaned
backends
Post by Bruce Momjian
Post by Igor Kovalenko
would also be removed since there would be only one process.
Issue of thread safety of code will come up undoubtedly and some
things
Post by Jon Franz
will
Post by Bruce Momjian
Post by Igor Kovalenko
probably have to be revamped. But in long term this is probably best
way
Post by Jon Franz
if
Post by Bruce Momjian
Post by Igor Kovalenko
you want to have efficient and uniform Unix AND Win32 implementations.
I am not too familiar with Win32. Speaking about POSIX threads, it
would
Post by Jon Franz
be
Post by Bruce Momjian
Post by Igor Kovalenko
something like a thread pool with low & high watermarks. Main thread
would
Post by Bruce Momjian
Post by Igor Kovalenko
handle thread pool and hand over requests to worker threads (blocked on
condvar). How does that sound?
Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.
--
Bruce Momjian | http://candle.pha.pa.us
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026
---------------------------(end of broadcast)---------------------------
---------------------------(end of broadcast)---------------------------
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Nicolas Bazin
2002-06-06 01:22:40 UTC
Permalink
Gnu Pth also supports AIO
----- Original Message -----
From: "Nicolas Bazin" <***@ingenico.com.au>
To: "Jon Franz" <***@one.net>; <pgsql-***@postgresql.org>
Sent: Thursday, June 06, 2002 10:50 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Nicolas Bazin
Yes I proposed to use the GNU Pth library instead. It's an event
demultiplexer just like the sgi library, but has a posix thread interface.
This architecture is actually the more robust and also the more scalable. On
a single processor server, you don't have the multi-thread synchronization
and context switching overhead and you also take full advantage of
multi-processor servers when you create several processes. Plus you have
much less concern about global variables.
Also for those concerned about the licence of this library here is an
"The author places this library under the LGPL to make sure that it
can be used both commercially and non-commercially provided that
modifications to the code base are always donated back to the official
code base under the same license conditions. Please keep in mind that
especially using this library in code not staying under the GPL or
the LGPL _is_ allowed and that any taint or license creap into code
that uses the library is not the authors intention. It is just the
case that _including_ this library into the source tree of other
applications is a little bit more inconvinient because of the LGPL.
But it has to be this way for good reasons. And keep in mind that
inconvinient doesn't mean not allowed or even impossible."
So it can be used in both commercial and non commercial project.
----- Original Message -----
Sent: Thursday, June 06, 2002 8:50 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Jon Franz
One note: SGI developers discovered they could get amazing performance
using
Post by Jon Franz
as hybrid threaded and forked-process model with apache - we might want to
look into this. They even have a library for network-communication
http://state-threads.sourceforge.net/docs/st.html
Thus, on platforms where it can be supported, we should keep in mind
that
Post by Nicolas Bazin
a
Post by Jon Franz
hybrid multiprocess/multithreaded postgresql might be the fastest
solution...
----- Original Message -----
Sent: Wednesday, June 05, 2002 4:05 PM
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Bruce Momjian
Post by Igor Kovalenko
I might be naive here, but would not proper threading model remove the
need
Post by Bruce Momjian
Post by Igor Kovalenko
for fork() altogether? On both Unix and Win32? Should not be too
hard
Post by Nicolas Bazin
to
Post by Jon Franz
Post by Bruce Momjian
Post by Igor Kovalenko
come up with abstraction which encapsulates POSIX, BeOS and Win32
threads...
Post by Bruce Momjian
Post by Igor Kovalenko
I am not sure how universal POSIX threads are by now. Any important
Unix
Post by Jon Franz
Post by Bruce Momjian
Post by Igor Kovalenko
platforms which don't support them yet?
This has downside of letting any bug to kill the whole thing. On the
bright
Post by Bruce Momjian
Post by Igor Kovalenko
side, performance should be better on some platforms (note however,
Apache
Post by Bruce Momjian
Post by Igor Kovalenko
group still can't come up with implementation of threaded model which
would
Post by Bruce Momjian
Post by Igor Kovalenko
provide better performance than forked or other models). The need to
deal
Post by Bruce Momjian
Post by Igor Kovalenko
with possibility of 'alien' postmaster running along with orphaned
backends
Post by Bruce Momjian
Post by Igor Kovalenko
would also be removed since there would be only one process.
Issue of thread safety of code will come up undoubtedly and some
things
Post by Jon Franz
will
Post by Bruce Momjian
Post by Igor Kovalenko
probably have to be revamped. But in long term this is probably best
way
Post by Jon Franz
if
Post by Bruce Momjian
Post by Igor Kovalenko
you want to have efficient and uniform Unix AND Win32
implementations.
Post by Nicolas Bazin
Post by Jon Franz
Post by Bruce Momjian
Post by Igor Kovalenko
I am not too familiar with Win32. Speaking about POSIX threads, it
would
Post by Jon Franz
be
Post by Bruce Momjian
Post by Igor Kovalenko
something like a thread pool with low & high watermarks. Main thread
would
Post by Bruce Momjian
Post by Igor Kovalenko
handle thread pool and hand over requests to worker threads (blocked
on
Post by Jon Franz
Post by Bruce Momjian
Post by Igor Kovalenko
condvar). How does that sound?
Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a web backend
where many sessions are a single query, people may want to give up the
stability of fork() and go with threads, even on Unix.
--
Bruce Momjian | http://candle.pha.pa.us
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania
19026
Post by Jon Franz
Post by Bruce Momjian
---------------------------(end of
broadcast)---------------------------
Post by Nicolas Bazin
Post by Jon Franz
Post by Bruce Momjian
TIP 1: subscribe and unsubscribe commands go to
---------------------------(end of broadcast)---------------------------
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly
Steve Howe
2002-06-06 01:37:17 UTC
Permalink
Hello Bruce,

Wednesday, June 5, 2002, 1:33:44 AM, you wrote:

BM> INSTALLER
BM> ---------

BM> We clearly need an installer that is zero-hassle for users. We need to
BM> decide on a direction for this.
I suggest Nullsoft install system
(http://www.nullsoft.com/free/nsis/). It's real good and very simple
to use. I can help on this if you want.

BM> ENVIRONMENT

BM> Lots of our code requires a unix shell and utilities. Will we continue
BM> using cygwin for this?
There are other ports ( http://unxutils.sourceforge.net/ ) that won't
require Cygwin but they won't provide an environment so complete as
Cygwin does.

I also would like to empathize that probably a small GUI for
controlling the PostgreSQL service/application would be nice. I think
about something sitting in the system tray like MSSQL, Oracle,
Interbase, etc. does.
I could code this in Delphi if you like. I don't have experience in
writing GUI apps in C. There is an open source versions of Delphi so
it won't be a problem compiling it.
Also coming with this, a code for starting the PostgreSQL as a service
would be really nice. For those from UNIX world that don't know what a
service is, think about it as a daemon for Windows. A service can be
automatically started when the machine boots up.

-------------
Best regards,
Steve Howe mailto:***@carcass.dhs.org


---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Bruce Momjian
2002-06-06 02:57:00 UTC
Permalink
Here is a summary of the responses to my Win32 roadmap. I hope this
will allow further discussion.

---------------------------------------------------------------------------

INSTALLER
---------
Cygwin Setup.exe http://cygwin.com
Nullsoft http://www.nullsoft.com/free/nsis/

GUI
---
pgAdmin2 http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1
pgaccess http://pgaccess.org/
Java admin (to be written)
Dev-C++ admin (to be written) http://sourceforge.net/projects/dev-cpp/

BINARY
------


FORK()

cygwin fork() http://cygwin.com
CreateProcess() and copy global area

THREADING

Posix threads
Gnu pth http://www.gnu.org/software/pth/
ST http://state-threads.sourceforge.net/docs/st.html
(single-session multi-threading possible)
(Posix AIO is possible)

IPC

Cygwin http://cygwin.com
MinGW http://www.mingw.org/
ACE http://www.cs.wustl.edu/~schmidt/ACE.html
APR http://apr.apache.org/
Our own

ENVIRONMENT

Cygwin http://cygwin.com
UnxUtils http://unxutils.sourceforge.net/
Write own initdb


IMPLEMENTATIONS
---------------
PostgreSQLe http://hp.vector.co.jp/authors/VA023283/PostgreSQLe.html
Dbexperts http://www.dbexperts.net/postgresql
Connx http://www.connx.com/
gborg http://gborg.postgresql.org/project/winpackage/projdisplay.php
Interbase http://community.borland.com/article/0,1410,23217,00.html
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Robert Schrem
2002-06-06 09:54:36 UTC
Permalink
Hi Bruce,

You obviosuly missed my recent posting advertising the homepage
of Konstantin Knizhnik?

Make sure to have a look: http://www.garret.ru/~knizhnik/

You find there -everything- concerning multiplatform IPC,
threading and even some extraordinary, complete database
backends that are superior to the database backends
previously available as open source (including PostgreSQL,
I'm afraid...). The licensing of all of this stuff is -public domain-.
I think this should really be worth a look/discussion/mentioning.

Here an excerpt of my last email, describing the furios list
of features abailable in GOODS:

Some core features of the GOODS backend (as they come to my mind):
-> full ACID transaction support, incl. distributed transactions
-> Multiple stoarge servers distributed over a TCP/ID network
-> multible reader/single writer (MVCC)
-> dual client side object cache
-> online backup (snapshot backup AND permanent backup)
-> nested transactions on object level
-> transaction isolation levels on object level
-> object level shared and exclusive locks
-> excellent C++ programming interface
-> WAL
-> garbage collection for no longer reference database objects (online VACUUM)
-> fully thread safe client interface
-> JAVA client API
-> very high performance as a result of a lot of fine tuning (better
perfomance than berkeley db in my benchmarks!!!)
-> asyncrous event notification on object instance modification
-> extremly high code quality
-> a one person effort, hence a very clean design
-> the most relevant platforms are supported out of the box
-> complete build is done in less than a minute on my machine
-> it's documented
-> it's tested and found to be working for a while now
...

kind regards,
Robert Schrem
Post by Bruce Momjian
Here is a summary of the responses to my Win32 roadmap. I hope this
will allow further discussion.
---------------------------------------------------------------------------
INSTALLER
---------
Cygwin Setup.exe http://cygwin.com
Nullsoft http://www.nullsoft.com/free/nsis/
GUI
---
pgAdmin2
http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1 pgaccess
http://pgaccess.org/
Java admin (to be written)
Dev-C++ admin (to be written)
http://sourceforge.net/projects/dev-cpp/
BINARY
------
FORK()
cygwin fork() http://cygwin.com
CreateProcess() and copy global area
THREADING
Posix threads
Gnu pth http://www.gnu.org/software/pth/
ST
http://state-threads.sourceforge.net/docs/st.html (single-session
multi-threading possible)
(Posix AIO is possible)
IPC
Cygwin http://cygwin.com
MinGW http://www.mingw.org/
ACE
http://www.cs.wustl.edu/~schmidt/ACE.html APR
http://apr.apache.org/
Our own
ENVIRONMENT
Cygwin http://cygwin.com
UnxUtils http://unxutils.sourceforge.net/
Write own initdb
IMPLEMENTATIONS
---------------
PostgreSQLe
http://hp.vector.co.jp/authors/VA023283/PostgreSQLe.html Dbexperts
http://www.dbexperts.net/postgresql Connx
http://www.connx.com/
gborg
http://gborg.postgresql.org/project/winpackage/projdisplay.php Interbase
http://community.borland.com/article/0,1410,23217,00.html
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly
Bruce Momjian
2002-06-06 14:26:02 UTC
Permalink
Added to the list. Thanks.

---------------------------------------------------------------------------
Post by Robert Schrem
Hi Bruce,
You obviosuly missed my recent posting advertising the homepage
of Konstantin Knizhnik?
Make sure to have a look: http://www.garret.ru/~knizhnik/
You find there -everything- concerning multiplatform IPC,
threading and even some extraordinary, complete database
backends that are superior to the database backends
previously available as open source (including PostgreSQL,
I'm afraid...). The licensing of all of this stuff is -public domain-.
I think this should really be worth a look/discussion/mentioning.
Here an excerpt of my last email, describing the furios list
-> full ACID transaction support, incl. distributed transactions
-> Multiple stoarge servers distributed over a TCP/ID network
-> multible reader/single writer (MVCC)
-> dual client side object cache
-> online backup (snapshot backup AND permanent backup)
-> nested transactions on object level
-> transaction isolation levels on object level
-> object level shared and exclusive locks
-> excellent C++ programming interface
-> WAL
-> garbage collection for no longer reference database objects (online VACUUM)
-> fully thread safe client interface
-> JAVA client API
-> very high performance as a result of a lot of fine tuning (better
perfomance than berkeley db in my benchmarks!!!)
-> asyncrous event notification on object instance modification
-> extremly high code quality
-> a one person effort, hence a very clean design
-> the most relevant platforms are supported out of the box
-> complete build is done in less than a minute on my machine
-> it's documented
-> it's tested and found to be working for a while now
...
kind regards,
Robert Schrem
Post by Bruce Momjian
Here is a summary of the responses to my Win32 roadmap. I hope this
will allow further discussion.
---------------------------------------------------------------------------
INSTALLER
---------
Cygwin Setup.exe http://cygwin.com
Nullsoft http://www.nullsoft.com/free/nsis/
GUI
---
pgAdmin2
http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1 pgaccess
http://pgaccess.org/
Java admin (to be written)
Dev-C++ admin (to be written)
http://sourceforge.net/projects/dev-cpp/
BINARY
------
FORK()
cygwin fork() http://cygwin.com
CreateProcess() and copy global area
THREADING
Posix threads
Gnu pth http://www.gnu.org/software/pth/
ST
http://state-threads.sourceforge.net/docs/st.html (single-session
multi-threading possible)
(Posix AIO is possible)
IPC
Cygwin http://cygwin.com
MinGW http://www.mingw.org/
ACE
http://www.cs.wustl.edu/~schmidt/ACE.html APR
http://apr.apache.org/
Our own
ENVIRONMENT
Cygwin http://cygwin.com
UnxUtils http://unxutils.sourceforge.net/
Write own initdb
IMPLEMENTATIONS
---------------
PostgreSQLe
http://hp.vector.co.jp/authors/VA023283/PostgreSQLe.html Dbexperts
http://www.dbexperts.net/postgresql Connx
http://www.connx.com/
gborg
http://gborg.postgresql.org/project/winpackage/projdisplay.php Interbase
http://community.borland.com/article/0,1410,23217,00.html
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Peter Eisentraut
2002-06-07 17:42:11 UTC
Permalink
Post by Bruce Momjian
GUI
---
pgAdmin2 http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1
pgaccess http://pgaccess.org/
Java admin (to be written)
Dev-C++ admin (to be written) http://sourceforge.net/projects/dev-cpp/
Surely Unix folks would like a GUI as well?
--
Peter Eisentraut ***@gmx.net


---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Scott Shattuck
2002-06-07 22:05:58 UTC
Permalink
How about a SOAP interface and a web-based front end that provides the cross
platform support? My company's TIBET framework would provide a solid
foundation for this kind of admin suite. In fact, we're already in the
planning stages on doing just that.

ss

Scott Shattuck
Technical Pursuit Inc.


----- Original Message -----
From: "Peter Eisentraut" <***@gmx.net>
To: "Bruce Momjian" <***@candle.pha.pa.us>
Cc: "PostgreSQL-development" <pgsql-***@postgreSQL.org>
Sent: Friday, June 07, 2002 11:42 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Peter Eisentraut
Post by Bruce Momjian
GUI
---
pgAdmin2
http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1
Post by Peter Eisentraut
Post by Bruce Momjian
pgaccess http://pgaccess.org/
Java admin (to be written)
Dev-C++ admin (to be written)
http://sourceforge.net/projects/dev-cpp/
Post by Peter Eisentraut
Surely Unix folks would like a GUI as well?
--
---------------------------(end of broadcast)---------------------------
---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Dave Cramer
2002-06-10 16:21:42 UTC
Permalink
Scott,

I just started a java admin tool project on sf called
www.sf.net/projects/jpgadmin, which should be able to handle web based
interfaces, the idea being to seperate the model and view so that we can
support a swing or web interface.

Dave
Post by Scott Shattuck
How about a SOAP interface and a web-based front end that provides the cross
platform support? My company's TIBET framework would provide a solid
foundation for this kind of admin suite. In fact, we're already in the
planning stages on doing just that.
ss
Scott Shattuck
Technical Pursuit Inc.
----- Original Message -----
Sent: Friday, June 07, 2002 11:42 AM
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Peter Eisentraut
Post by Bruce Momjian
GUI
---
pgAdmin2
http://pgadmin.postgresql.org/pgadmin2.php?ContentID=1
Post by Peter Eisentraut
Post by Bruce Momjian
pgaccess http://pgaccess.org/
Java admin (to be written)
Dev-C++ admin (to be written)
http://sourceforge.net/projects/dev-cpp/
Post by Peter Eisentraut
Surely Unix folks would like a GUI as well?
--
---------------------------(end of broadcast)---------------------------
---------------------------(end of broadcast)---------------------------
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Peter Eisentraut
2002-06-06 17:11:12 UTC
Permalink
Post by Bruce Momjian
Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?
We should probably get rid of using shell scripts for application programs
altogether, for a number of reasons besides this one, such as the
inability to properly handle input values with spaces, commas, etc. (we
probably don't handle very long values either on some platforms), the
inability to maintain open database connections so that createlang needs
to prompt for the same password thrice, general portable scripting
headaches, and the lack of internationalization facilities.

I'd even volunteer to do this. Comments?
--
Peter Eisentraut ***@gmx.net


---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Bruce Momjian
2002-06-06 17:57:05 UTC
Permalink
Post by Peter Eisentraut
Post by Bruce Momjian
Lots of our code requires a unix shell and utilities. Will we continue
using cygwin for this?
We should probably get rid of using shell scripts for application programs
altogether, for a number of reasons besides this one, such as the
inability to properly handle input values with spaces, commas, etc. (we
probably don't handle very long values either on some platforms), the
inability to maintain open database connections so that createlang needs
to prompt for the same password thrice, general portable scripting
headaches, and the lack of internationalization facilities.
I'd even volunteer to do this. Comments?
I know I have discouraged it because I think shell script language has a
good toolset for those applications. I have fixed all the spacing
issues.

What language where you thinking of using? C?

Also, it seems Win32 doesn't need these scripts, except initdb.
PostgreSQLe didn't use the, it just did initdb, and the rest were done
using a GUI. However, initdb would remain a problem. PostgreSQLe wrote
its own.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Peter Eisentraut
2002-06-07 22:27:59 UTC
Permalink
Post by Bruce Momjian
I know I have discouraged it because I think shell script language has a
good toolset for those applications. I have fixed all the spacing
issues.
My point is that it is not, for the reasons that I listed. Handling
spaces is a small part of one of the several problems, there are problems
with newlines, tabs, commas, slashes, quotes -- everytime you call sed or
read you lose one character.
Post by Bruce Momjian
What language where you thinking of using? C?
Yes, that way we can share code (pg_dumpall<->pg_dump, initdb<->postgres),
use the established internationalization facilities, and use libpq
directly in create* and drop*.
Post by Bruce Momjian
Also, it seems Win32 doesn't need these scripts, except initdb.
The utility of these programs is independent of the platform. If we think
pg_dumpall is not useful, then let's remove it.
--
Peter Eisentraut ***@gmx.net


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Tom Lane
2002-06-08 15:48:20 UTC
Permalink
Post by Peter Eisentraut
Post by Bruce Momjian
Also, it seems Win32 doesn't need these scripts, except initdb.
The utility of these programs is independent of the platform. If we think
pg_dumpall is not useful, then let's remove it.
I have been seriously considering converting pg_dumpall to C anyway,
because it's already *very* messy, and I don't see any reasonable
way to make it support dumping per-database and per-user config
settings. (Do you really want to try to parse array values in a
shell script?)

(I'd actually consider making pg_dumpall a part of the pg_dump
executable; then it could invoke pg_dump as a subroutine call...)

If Peter's got the time/energy to convert 'em all, I'm for it.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Bruce Momjian
2002-06-08 15:57:07 UTC
Permalink
Post by Tom Lane
Post by Peter Eisentraut
Post by Bruce Momjian
Also, it seems Win32 doesn't need these scripts, except initdb.
The utility of these programs is independent of the platform. If we think
pg_dumpall is not useful, then let's remove it.
I have been seriously considering converting pg_dumpall to C anyway,
because it's already *very* messy, and I don't see any reasonable
way to make it support dumping per-database and per-user config
settings. (Do you really want to try to parse array values in a
shell script?)
(I'd actually consider making pg_dumpall a part of the pg_dump
executable; then it could invoke pg_dump as a subroutine call...)
If Peter's got the time/energy to convert 'em all, I'm for it.
Yea, shame it will now take 15 lines of C code to do what we could do in
1 line of shell script but I don't see any other option.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Tom Lane
2002-06-08 16:05:37 UTC
Permalink
Post by Bruce Momjian
Yea, shame it will now take 15 lines of C code to do what we could do in
1 line of shell script but I don't see any other option.
In places we are using 15 lines of shell to do what would take 1 line
in C ;-). Yes, it'll probably be bigger overall, but I think you are
overstating the penalty.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Bruce Momjian
2002-06-08 21:48:12 UTC
Permalink
Post by Peter Eisentraut
Post by Bruce Momjian
I know I have discouraged it because I think shell script language has a
good toolset for those applications. I have fixed all the spacing
issues.
My point is that it is not, for the reasons that I listed. Handling
spaces is a small part of one of the several problems, there are problems
with newlines, tabs, commas, slashes, quotes -- everytime you call sed or
read you lose one character.
Post by Bruce Momjian
What language where you thinking of using? C?
Yes, that way we can share code (pg_dumpall<->pg_dump, initdb<->postgres),
use the established internationalization facilities, and use libpq
directly in create* and drop*.
Post by Bruce Momjian
Also, it seems Win32 doesn't need these scripts, except initdb.
The utility of these programs is independent of the platform. If we think
pg_dumpall is not useful, then let's remove it.
I think the first two targets for C-ification would be pg_dumpall and
initdb. The others have SQL equivalents. Maybe pg_ctl too.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Peter Eisentraut
2002-06-17 21:19:33 UTC
Permalink
Post by Bruce Momjian
I think the first two targets for C-ification would be pg_dumpall and
initdb. The others have SQL equivalents. Maybe pg_ctl too.
I think eventually pg_ctl should be folded into the postmaster executable.
This would remove a great amount of possible misunderstandings between the
two programs.
--
Peter Eisentraut ***@gmx.net


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Bruce Momjian
2002-06-17 21:31:01 UTC
Permalink
Post by Peter Eisentraut
Post by Bruce Momjian
I think the first two targets for C-ification would be pg_dumpall and
initdb. The others have SQL equivalents. Maybe pg_ctl too.
I think eventually pg_ctl should be folded into the postmaster executable.
This would remove a great amount of possible misunderstandings between the
two programs.
And pg_ctl will be run with a symlink to postmaster like postgres,
right? Makes sense.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly
Jan Wieck
2002-06-18 01:05:34 UTC
Permalink
Post by Bruce Momjian
Post by Peter Eisentraut
Post by Bruce Momjian
I think the first two targets for C-ification would be pg_dumpall and
initdb. The others have SQL equivalents. Maybe pg_ctl too.
I think eventually pg_ctl should be folded into the postmaster executable.
This would remove a great amount of possible misunderstandings between the
two programs.
And pg_ctl will be run with a symlink to postmaster like postgres,
right? Makes sense.
No symlink. Windows doesn't have symlinks, the "link" stuff you
see is just some file with a special meaning for the Windows
explorer. There is absolutely no support built into the OS. They
really haven't learned alot since the DOS times, when they added
"." and ".." entries to directories to "look" similar to UNIX.
Actually, they never really understood what a hardlink is in the
first place, so why do you expect them to know how to implement
symbolic ones?

It will be at least another copy of the postmaster (dot exe).


Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being
right. #
# Let's break this rule - forgive
me. #
#==================================================
***@Yahoo.com #

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Bruce Momjian
2002-06-18 01:00:46 UTC
Permalink
Post by Jan Wieck
Post by Bruce Momjian
And pg_ctl will be run with a symlink to postmaster like postgres,
right? Makes sense.
No symlink. Windows doesn't have symlinks, the "link" stuff you
see is just some file with a special meaning for the Windows
explorer. There is absolutely no support built into the OS. They
really haven't learned alot since the DOS times, when they added
"." and ".." entries to directories to "look" similar to UNIX.
Actually, they never really understood what a hardlink is in the
first place, so why do you expect them to know how to implement
symbolic ones?
It will be at least another copy of the postmaster (dot exe).
Yea, I just liked the idea of the postmaster binary somehow reporting
the postmaster status. Seems it is in a better position to do that than
a shell script.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Tom Lane
2002-06-18 14:52:53 UTC
Permalink
Post by Peter Eisentraut
I think eventually pg_ctl should be folded into the postmaster executable.
This would remove a great amount of possible misunderstandings between the
two programs.
Like what?

The thing pg_ctl needs to know is where PGDATA is, and that
unfortunately isn't going to be known any better just by sharing
executables.

I don't object to C-ifying pg_ctl, but I don't see that it will
automatically improve pg_ctl's robustness materially.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Peter Eisentraut
2002-06-18 21:55:20 UTC
Permalink
Post by Tom Lane
Post by Peter Eisentraut
I think eventually pg_ctl should be folded into the postmaster executable.
This would remove a great amount of possible misunderstandings between the
two programs.
Like what?
The biggie is that pg_ctl reports the postmaster to have started
successfully without ever checking. And the "wait" option is broken and
not trivial to fix.

Other problems are the matching of the port numbers and the requirement
that admins should be able to enter a password when the server starts (for
SSL).

The luring prerequisite here is that the postmaster would have to be able
to log directly to a file, which now that all communication is guaranteed
to go through elog() should be less complicated, at least compared to
fixing the "wait" option. In fact I'm hoping that the Windows porters
will run into this same requirement just about pretty soon.
--
Peter Eisentraut ***@gmx.net


---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Tom Lane
2002-06-19 12:45:43 UTC
Permalink
Post by Peter Eisentraut
Post by Tom Lane
Like what?
The biggie is that pg_ctl reports the postmaster to have started
successfully without ever checking. And the "wait" option is broken and
not trivial to fix.
Indeed, but how will it help to merge the two executables into one?
I don't think you can simply postpone the fork() until all setup
is complete --- that would mean you don't know the final postmaster
PID until much too late.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Hannu Krosing
2002-05-16 14:57:14 UTC
Permalink
Post by Joerg Hessdoerfer
So, my route would be to get it to run *somehow* without paying attention to
speed and not to change much of the existing code, THEN see how we could get
rid of fork() on windows.
Getting it to compile and then "somehow" run on MinGW seems a good first
step on road to full native Win32 PG.

----------
Hannu



---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Magnus Naeslund(f)
2002-05-16 20:10:03 UTC
Permalink
Post by Joerg Hessdoerfer
So, my route would be to get it to run *somehow* without paying
attention to speed and not to change much of the existing code,
THEN see how we could get rid of fork() on windows.
What is the biggest problem here?
The Shmem/IPC stuff, or the fork() stuff?
I'm think that we could do a fork() implementation in usermode by copying the memory allocations.
How fast that would be regarding the context switches, i don't know, but i'm willing to experiment some to see how feesible this is...

Anyone tried this before?

Magnus


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postm
Joerg Hessdoerfer
2002-05-16 20:35:58 UTC
Permalink
On Thursday 16 May 2002 22:10, you wrote:
[...]
Post by Magnus Naeslund(f)
What is the biggest problem here?
The Shmem/IPC stuff, or the fork() stuff?
I'm think that we could do a fork() implementation in usermode by copying
the memory allocations. How fast that would be regarding the context
switches, i don't know, but i'm willing to experiment some to see how
feesible this is...
Anyone tried this before?
Magnus
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
The problem is not the fork() call itself, this has been done (MinGW and
cygwin I know of, possibly others) but the speed of fork() on windows, it's
creepingly slow (due to usermode copy, I assume ;-).

IPC needs to be done, I'm just about to start...

Greetings,
Joerg
--
Leading SW developer - S.E.A GmbH
Mail: ***@sea-gmbh.com
WWW: http://www.sea-gmbh.com

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Magnus Naeslund(f)
2002-05-17 00:16:02 UTC
Permalink
Joerg Hessdoerfer <***@sea-gmbh.com> wrote:
[snip]
Post by Joerg Hessdoerfer
The problem is not the fork() call itself, this has been done (MinGW
and cygwin I know of, possibly others) but the speed of fork() on
windows, it's creepingly slow (due to usermode copy, I assume ;-).
IPC needs to be done, I'm just about to start...
I'm not so familiar with the win32 kernel mode stuff.
But i've seen programs using .vxd (kernelmode, ring X ?) helpers for getting more privileges, maybe cross process ones.
Well, i'll look into this sometime if it's possible to reduce the context switches by going vxd.
There must be some way to read protection of the pages and map them as COW or RO in the new process to get rid of much of the copy, but then again, we're talking microsoft here :)
I once did a .exe loader that used the MapViewOfFile (win32 mmap) of the .exe itself to accomplish shared loadable modules that worked on x86 linux and win32 without recompile (might be something like the XFree86 binary gfx card drivers).

Good luck on the IPC work!

Magnus

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Programmer/Networker [|] Magnus Naeslund
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?
Christopher Kings-Lynne
2002-05-17 02:25:38 UTC
Permalink
Maybe Vince could set up a Win32 porting project page, and since we now seem
to have a few interested parties willing to code on a native Win32 version,
they should have their own project page. This could make communication
easier for them and make sure the project doesn't die...

Chris
-----Original Message-----
Hessdoerfer
Sent: Friday, 17 May 2002 4:36 AM
To: Magnus Naeslund(f)
Subject: Re: [HACKERS] WIN32 native ... lets start?!?
[...]
Post by Magnus Naeslund(f)
What is the biggest problem here?
The Shmem/IPC stuff, or the fork() stuff?
I'm think that we could do a fork() implementation in usermode
by copying
Post by Magnus Naeslund(f)
the memory allocations. How fast that would be regarding the context
switches, i don't know, but i'm willing to experiment some to see how
feesible this is...
Anyone tried this before?
Magnus
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
The problem is not the fork() call itself, this has been done (MinGW and
cygwin I know of, possibly others) but the speed of fork() on
windows, it's
creepingly slow (due to usermode copy, I assume ;-).
IPC needs to be done, I'm just about to start...
Greetings,
Joerg
--
Leading SW developer - S.E.A GmbH
WWW: http://www.sea-gmbh.com
---------------------------(end of broadcast)---------------------------
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Marc G. Fournier
2002-05-17 03:17:49 UTC
Permalink
Post by Christopher Kings-Lynne
Maybe Vince could set up a Win32 porting project page, and since we now seem
to have a few interested parties willing to code on a native Win32 version,
they should have their own project page. This could make communication
easier for them and make sure the project doesn't die...
Might be an idea to create a pgsql-hackers-win32 list also? Or just
pgsql-win32?
Post by Christopher Kings-Lynne
Chris
-----Original Message-----
Hessdoerfer
Sent: Friday, 17 May 2002 4:36 AM
To: Magnus Naeslund(f)
Subject: Re: [HACKERS] WIN32 native ... lets start?!?
[...]
Post by Magnus Naeslund(f)
What is the biggest problem here?
The Shmem/IPC stuff, or the fork() stuff?
I'm think that we could do a fork() implementation in usermode
by copying
Post by Magnus Naeslund(f)
the memory allocations. How fast that would be regarding the context
switches, i don't know, but i'm willing to experiment some to see how
feesible this is...
Anyone tried this before?
Magnus
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
The problem is not the fork() call itself, this has been done (MinGW and
cygwin I know of, possibly others) but the speed of fork() on
windows, it's
creepingly slow (due to usermode copy, I assume ;-).
IPC needs to be done, I'm just about to start...
Greetings,
Joerg
--
Leading SW developer - S.E.A GmbH
WWW: http://www.sea-gmbh.com
---------------------------(end of broadcast)---------------------------
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Tom Lane
2002-05-17 20:16:18 UTC
Permalink
Post by Marc G. Fournier
Might be an idea to create a pgsql-hackers-win32 list also? Or just
pgsql-win32?
Actually, I think that'd be a bad idea. The very last thing we need is
for these discussions to get fragmented. The issues affect the whole
backend AFAICS.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Joerg Hessdoerfer
2002-05-18 13:57:42 UTC
Permalink
Post by Tom Lane
Post by Marc G. Fournier
Might be an idea to create a pgsql-hackers-win32 list also? Or just
pgsql-win32?
Actually, I think that'd be a bad idea. The very last thing we need is
for these discussions to get fragmented. The issues affect the whole
backend AFAICS.
regards, tom lane
Yes, indeed. I would also like to discuss matters on this list, as one get's
a 'heads up' from people in the know much easier.

BTW, I'm in the process of doing the 'really only what is necessary for pg'
ipc-stuff, and was wondering if anybody already did some configuration of the
source tree towards MinGW?? How should we go about that? I would rather like
not using cygwin's sh for that ;-), and we have no 'ln' !!

Greetings,
Joerg
--
Leading SW developer - S.E.A GmbH
Mail: ***@sea-gmbh.com
WWW: http://www.sea-gmbh.com

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Dann Corbit
2002-06-05 05:02:14 UTC
Permalink
-----Original Message-----
Sent: Tuesday, June 04, 2002 9:34 PM
To: PostgreSQL-development
Subject: [HACKERS] Roadmap for a Win32 port
OK, I think I am now caught up on the Win32/cygwin
discussion, and would
like to make some remarks.
First, are we doing enough to support the Win32 platform? I think the
answer is clearly "no". There are 3-5 groups/companies
working on Win32
ports of PostgreSQL. We always said there would not be
PostgreSQL forks
if we were doing our job to meet user needs. Well,
obviously, a number
of groups see a need for a better Win32 port and we aren't
meeting that
need, so they are. I believe this is one of the few cases
where groups
are going out on their own because we are falling behind.
So, there is no question in my mind we need to do more to encourage
Win32 ports. Now, on to the details.
INSTALLER
---------
We clearly need an installer that is zero-hassle for users.
We need to
decide on a direction for this.
GUI
---
We need a slick GUI. pgadmin2 seems to be everyone's favorite, with
pgaccess on Win32 also an option. What else do we need here?
Nothing else. It is better than any commercial tools in current use.
An excellent piece of work.
BINARY
------
FORK()
How do we handle fork()? Do we use the cygwin method that copies the
whole data segment, or put the global data in shared memory and copy
that small part manually after we create a new process?
Do not try to do a fork() on Win32. The one at PW32 is better, but
still awful. Win32 just does not have fascilities for fork().

If you use Cygwin, it will kill the project for commercial use (at least
for many institutions). That's fine, but it will become an academic
exercise instead of a viable commercial tool. If they are comfortable
in that [Cygwin] environment, it makes no sense to use Cygwin instead of
Redhat. The Redhat version will fork() 100 times faster. After all, if
they are going to use unix tools in a unix interface with Unix scripts
you might as well use UNIX. And Cygwin requires a license for
commercial use.
http://cygwin.com/licensing.html
THREADING
Related to fork(), do we implement an optionally threaded postmaster,
which eliminates CreateProcess() entirely? I don't think we will have
superior performance on Win32 without it. (This would greatly help
Solaris as well.)
CreateProcess() works well for Win32. That is the approach that we used
and also the approach used by the Japanese team.
It is very simple. Simply do a create process call and then perform the
same operations that were done up to that point. It isn't difficult.
Threading is another possibility. I think create process is better,
because you can clone the rights of the one who attaches for the spawned
server (if you want to do that).
IPC
We can use Cygwin, MinGW, Apache, or our own code for this. Are there
other options?
We wrote our own from scratch. Cygwin will kill it. If there is a
MinGW version it might be OK, but if MinGW is GPL, that will kill it.
Have a look at ACE:
http://www.cs.wustl.edu/~schmidt/ACE.html
Their license is on the same level as a BSD license. Now, they use C++,
but you can always write:
extern "C" {
}
wrappers for stuff and keep PostgreSQL itself in pure, vanilla C. GCC
does come with a C++ compiler, so it isn't going to cut anyone off.
ENVIRONMENT
Lots of our code requires a Unix shell and utilities. Will
we continue
using cygwin for this?
We wrote our own utilities from scratch (e.g. initdb). The Japanese
group that did the port did the same thing.
--------------------------------------------------------------
-------------
As a roadmap, it would be good to get consensus on as many of these
items as possible so people can start working in these areas. We can
keep a web page of decisions we have made to help rally developers to
the project.
If you want a roadmap, the Japanese group laid it out for you. They
did the exact same steps as we did. Now, I don't know if we will be
able to contribute or not (it is very much up in the air). And we had
to do a lot of hacking of the source, so you might not want it if we
volunteered.

Suggestion:
Ask the Japanese group if they would like to post their changes back or
expose them so that the programming team can get ideas form it.

I actually like what they did better than what we did (A giant DLL and
all the binaries are microscopic -- it was how I suggested to do it here
but it was vetoed).

Anyway, here is a roadmap laid out for you exactly. Just do what it
says and you will be fine:
http://hp.vector.co.jp/authors/VA023283/PostgreSQLe.html

Look at where it says "Gists for patch" and do that.

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly
Jason Tishler
2002-06-05 12:07:06 UTC
Permalink
Dan,

The following is to help keep the archives accurate and should not be
construed as an argument against the native Win32 port.
And Cygwin requires a license for commercial use.
http://cygwin.com/licensing.html
The above is not necessarily true:

Red Hat sells a special Cygwin License for customers who are unable
to provide their application in open source code form.

Note that the above only comes into play if your application links
with the Cygwin DLL. This is easily avoidable by using JDBC, ODBC,
Win32 libpq, etc. Hence, most people will not be required to purchase
this license from Red Hat.

Jason

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Bruce Momjian
2002-06-05 15:17:09 UTC
Permalink
Post by Jason Tishler
Dan,
The following is to help keep the archives accurate and should not be
construed as an argument against the native Win32 port.
And Cygwin requires a license for commercial use.
http://cygwin.com/licensing.html
Red Hat sells a special Cygwin License for customers who are unable
to provide their application in open source code form.
Note that the above only comes into play if your application links
with the Cygwin DLL. This is easily avoidable by using JDBC, ODBC,
Win32 libpq, etc. Hence, most people will not be required to purchase
this license from Red Hat.
So apps written using client libraries are BSD, while server-side
changes would have to release source. Makes sense, though we have never
had this distinction before. I assume plpgsql stored procedures would
have be open source, but of course those are stored in plaintext on the
server so that isn't a problem. If companies created custom C stored
procedures, those would have to be open source if using cygwin.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Dann Corbit
2002-06-05 22:09:29 UTC
Permalink
-----Original Message-----
Sent: Wednesday, June 05, 2002 3:03 PM
To: Bruce Momjian
Cc: Igor Kovalenko; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port
...
Post by Bruce Momjian
Good summary. I think we would support both threaded and fork()
operation, and users can control which they prefer. For a
web backend
Post by Bruce Momjian
where many sessions are a single query, people may want to
give up the
Post by Bruce Momjian
stability of fork() and go with threads, even on Unix.
I would think that we would build on our strengths of having
a fork/exec
model for separate clients. A threaded model *could* benefit
individual
clients who are doing queries on multiprocessor servers, and
I would be
supportive of efforts to enable that.
But the requirements for that may be less severe than for managing
multiple clients within the same process, and imho there is not strong
requirement to enable the latter for our current crop of well
supported
targets. If it came for free then great, but if it came with
a high cost
then the choice is not as obvious. It is also not a
*requirement* if we
were instead able to do the multiple threads for a single client
scenerio first.
Notion:
Have one version do both. Your server can fork(), and your sever can
thread. It can fork() and thread, it can fork() or thread.

That gives the best of all worlds. One client who has his attachments
to a database all setup might want to do a bunch of similar queries.
Hence a threaded model is nice.

A server may be set up to clone the rights of the attaching process for
security reasons. Then you launch a new server with fork().

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Thomas Lockhart
2002-06-05 22:21:22 UTC
Permalink
...
Post by Dann Corbit
Have one version do both. Your server can fork(), and your sever can
thread. It can fork() and thread, it can fork() or thread.
That gives the best of all worlds. One client who has his attachments
to a database all setup might want to do a bunch of similar queries.
Hence a threaded model is nice.
A server may be set up to clone the rights of the attaching process for
security reasons. Then you launch a new server with fork().
Right. If/when that is possible then let's do it, as long as the cost is
not too high. But the intermediate steps are a possibility also, and are
not precluded from discussion.

This will all work out as a *convergence* of interests imho. And there
is no great identifiable benefit for our current crop of platforms for
going to a threaded model *unless* that enables queries for a single
client to execute in parallel (all imho of course ;).

So our convergence of interests for all platforms is in enabling
threading for these two purposes, and focusing on enabling the
multithreaded single client *first* means that the current crop of
clients don't have to accept all negatives while we start on the road to
better support of Win32 machines.

- Thomas

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Dave Page
2002-06-07 07:42:33 UTC
Permalink
-----Original Message-----
Sent: 06 June 2002 02:37
To: Bruce Momjian
Cc: PostgreSQL-development
Subject: Re: Roadmap for a Win32 port
Hello Bruce,
BM> INSTALLER
BM> ---------
BM> We clearly need an installer that is zero-hassle for
users. We need
BM> to decide on a direction for this.
I suggest Nullsoft install system
(http://www.nullsoft.com/free/nsis/). It's > real good and very
simple to use. I can help on this if you want.
I think that a Windows Installer compatible package would be better as
it would allow us to build the package as a merge module which others
could use in their installers for their PostgreSQL based apps, allowing
one installation to install everything they require easily and (more
importantly) correctly. An example of this can be found in the psqlODBC
installer.

I can handle this if required.
BM> ENVIRONMENT
I also would like to empathize that probably a small GUI for
controlling the PostgreSQL service/application would be nice.
I'm happy to add such code to pgAdmin - seems like the natural thing to
do (to me at least!).

Regards, Dave.

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Dave Page
2002-06-09 10:38:26 UTC
Permalink
-----Original Message-----
Sent: 08 June 2002 22:48
To: Peter Eisentraut
Cc: PostgreSQL-development
Subject: Re: Roadmap for a Win32 port
Post by Peter Eisentraut
Post by Bruce Momjian
Also, it seems Win32 doesn't need these scripts, except initdb.
The utility of these programs is independent of the
platform. If we
Post by Peter Eisentraut
think pg_dumpall is not useful, then let's remove it.
I think the first two targets for C-ification would be
pg_dumpall and initdb. The others have SQL equivalents.
Maybe pg_ctl too.
I looked at this issue some time ago & came to the conclusion that the
only scripts that Win32 really needed were pg_dumpall, initdb &
initlocation.

The others have SQL equivalents as you say, apart from pg_ctl which
under Windows should probably (and generally is) be replaced by the SCM
(Service Control Manager). The only thing that comes to mind that the
SCM can't do is a reload.

Regards, Dave.

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Dann Corbit
2002-06-18 01:17:51 UTC
Permalink
-----Original Message-----
Sent: Monday, June 17, 2002 6:01 PM
To: Jan Wieck
Cc: Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Jan Wieck
Post by Bruce Momjian
And pg_ctl will be run with a symlink to postmaster like postgres,
right? Makes sense.
No symlink. Windows doesn't have symlinks, the "link" stuff you
see is just some file with a special meaning for the Windows
explorer. There is absolutely no support built into the OS. They
really haven't learned alot since the DOS times, when they added
"." and ".." entries to directories to "look" similar to UNIX.
Actually, they never really understood what a hardlink is in the
first place, so why do you expect them to know how to implement
symbolic ones?
It will be at least another copy of the postmaster (dot exe).
Yea, I just liked the idea of the postmaster binary somehow reporting
the postmaster status. Seems it is in a better position to
do that than
a shell script.
Architectural notion:
The Postmaster is about 100x bigger than it needs to be.

The Postmaster needs to set up shared memory and launch servers. It
does not need to know anything about SQL grammar or any of that
rigamarole.

It could be a 15K executable.

Why not have an itty-bitty Postmaster that does nothing but a spawn or a
create process to create threaded Postgres instances?

Just a notion.

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Bruce Momjian
2002-06-18 01:19:30 UTC
Permalink
Post by Dann Corbit
Post by Bruce Momjian
Post by Jan Wieck
It will be at least another copy of the postmaster (dot exe).
Yea, I just liked the idea of the postmaster binary somehow reporting
the postmaster status. Seems it is in a better position to
do that than
a shell script.
The Postmaster is about 100x bigger than it needs to be.
The Postmaster needs to set up shared memory and launch servers. It
does not need to know anything about SQL grammar or any of that
rigamarole.
It could be a 15K executable.
Why not have an itty-bitty Postmaster that does nothing but a spawn or a
create process to create threaded Postgres instances?
Can't. postmaster/postgres are symlinks to the same file, and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Dann Corbit
2002-06-18 01:25:48 UTC
Permalink
-----Original Message-----
Sent: Monday, June 17, 2002 6:20 PM
To: Dann Corbit
Cc: Jan Wieck; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Dann Corbit
Post by Bruce Momjian
Post by Jan Wieck
It will be at least another copy of the postmaster (dot exe).
Yea, I just liked the idea of the postmaster binary
somehow reporting
Post by Dann Corbit
Post by Bruce Momjian
the postmaster status. Seems it is in a better position to
do that than
a shell script.
The Postmaster is about 100x bigger than it needs to be.
The Postmaster needs to set up shared memory and launch servers. It
does not need to know anything about SQL grammar or any of that
rigamarole.
It could be a 15K executable.
Why not have an itty-bitty Postmaster that does nothing but
a spawn or a
Post by Dann Corbit
create process to create threaded Postgres instances?
Can't. postmaster/postgres are symlinks to the same file,
and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.
Is fork() faster than creation of a new process via exec()? After the
creation of the shared memory, the information needed to use it could be
passed to the Postgres servers on the command line.

The startup stuff for PostgreSQL is just a few files. It does not seem
insurmountable to change it. But it is none of my business. If it is a
major hassle (for reasons which I am not aware) then I see no driving
reason to change it.

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Bruce Momjian
2002-06-18 01:27:02 UTC
Permalink
Post by Dann Corbit
Post by Bruce Momjian
Can't. postmaster/postgres are symlinks to the same file,
and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.
Is fork() faster than creation of a new process via exec()? After the
creation of the shared memory, the information needed to use it could be
passed to the Postgres servers on the command line.
The startup stuff for PostgreSQL is just a few files. It does not seem
insurmountable to change it. But it is none of my business. If it is a
major hassle (for reasons which I am not aware) then I see no driving
reason to change it.
We used to fork() and exec(), but that was slow. Now we preload stuff
in the postmaster for each backend. It is faster.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Jan Wieck
2002-06-18 14:07:26 UTC
Permalink
Post by Dann Corbit
-----Original Message-----
Sent: Monday, June 17, 2002 6:20 PM
To: Dann Corbit
Cc: Jan Wieck; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Dann Corbit
Post by Bruce Momjian
Post by Jan Wieck
It will be at least another copy of the postmaster (dot exe).
Yea, I just liked the idea of the postmaster binary
somehow reporting
Post by Dann Corbit
Post by Bruce Momjian
the postmaster status. Seems it is in a better position to
do that than
a shell script.
The Postmaster is about 100x bigger than it needs to be.
The Postmaster needs to set up shared memory and launch servers. It
does not need to know anything about SQL grammar or any of that
rigamarole.
It could be a 15K executable.
Why not have an itty-bitty Postmaster that does nothing but
a spawn or a
Post by Dann Corbit
create process to create threaded Postgres instances?
Can't. postmaster/postgres are symlinks to the same file,
and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.
Is fork() faster than creation of a new process via exec()? After the
creation of the shared memory, the information needed to use it could be
passed to the Postgres servers on the command line.
exec() does NOT create new processes. It loads another executable file
into the existing, calling process.

fork() duplicates the calling process. In modern unix variants, this is
done in a very efficient way, so that the text segment (program code) is
shared readonly and everything else (data and stack segments) are shared
copy on write. Thus, fork() itself doesn't even cause memory copying.
That happens later when one of the now two processes writes to a memory
page the first time.

Windows does not have these two separate steps. It wants the full blown
expensive "create process and load executable", or the "let's all muck
around with the same handles" modell, called threading.
Post by Dann Corbit
The startup stuff for PostgreSQL is just a few files. It does not seem
insurmountable to change it. But it is none of my business. If it is a
major hassle (for reasons which I am not aware) then I see no driving
reason to change it.
It has to be changed for Windows, it is a major hassle for reasons I
wasn't aware of, and I am half way through ;-)


Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== ***@Yahoo.com #

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Greg Copeland
2002-06-18 15:56:38 UTC
Permalink
Post by Jan Wieck
Post by Dann Corbit
The startup stuff for PostgreSQL is just a few files. It does not seem
insurmountable to change it. But it is none of my business. If it is a
major hassle (for reasons which I am not aware) then I see no driving
reason to change it.
It has to be changed for Windows, it is a major hassle for reasons I
wasn't aware of, and I am half way through ;-)
Well, if you're going to go through the trouble of rewriting postmaster
to be win32 specific, you might as well make it hook into the services
control panel.

If I recall, shared memory is "owned" by a process in Win32 (please
correct as needed...as I've slept since I last looked). That means that
the postmaster process not only owns the shared memory but needs to make
sure that it's persists as the rest of postgres is expecting.

Please provide more details as to the nature of your Win32 changes. I'm
certainly curious. If you've already covered them, just say so...I have
no problem going back to the archives! :)

Greg
Serge Adda
2002-06-18 16:42:33 UTC
Permalink
Hello,

I am new to PostgreSQL, but I am interested in the Win32 port.
I have studied the architecture of other databases like Oracle.

They have had to turn their multi-process model used on Unix into a fully
multi-threaded one on Win32. I have the feeling that they have had the same
debate that the one you have.

The CreateProcess() syscall is very costly on Windows. Some improvements
have been done in Windows XP but it is still far more costly than a Unix
fork().

I have been programming with threads on NT for a long time now.
They are quiet robust and efficient. I fear that it is the only successful
way to port PostgreSQL.

Sorry for this interruption,
Serge

-----Original Message-----
From: Jan Wieck [mailto:***@Yahoo.com]
Sent: Tuesday, June 18, 2002 16:07
To: Dann Corbit
Cc: Bruce Momjian; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Dann Corbit
-----Original Message-----
Sent: Monday, June 17, 2002 6:20 PM
To: Dann Corbit
Cc: Jan Wieck; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Dann Corbit
Post by Bruce Momjian
Post by Jan Wieck
It will be at least another copy of the postmaster (dot exe).
Yea, I just liked the idea of the postmaster binary
somehow reporting
Post by Dann Corbit
Post by Bruce Momjian
the postmaster status. Seems it is in a better position to
do that than
a shell script.
The Postmaster is about 100x bigger than it needs to be.
The Postmaster needs to set up shared memory and launch servers. It
does not need to know anything about SQL grammar or any of that
rigamarole.
It could be a 15K executable.
Why not have an itty-bitty Postmaster that does nothing but
a spawn or a
Post by Dann Corbit
create process to create threaded Postgres instances?
Can't. postmaster/postgres are symlinks to the same file,
and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.
Is fork() faster than creation of a new process via exec()? After the
creation of the shared memory, the information needed to use it could be
passed to the Postgres servers on the command line.
exec() does NOT create new processes. It loads another executable file
into the existing, calling process.

fork() duplicates the calling process. In modern unix variants, this is
done in a very efficient way, so that the text segment (program code) is
shared readonly and everything else (data and stack segments) are shared
copy on write. Thus, fork() itself doesn't even cause memory copying.
That happens later when one of the now two processes writes to a memory
page the first time.

Windows does not have these two separate steps. It wants the full blown
expensive "create process and load executable", or the "let's all muck
around with the same handles" modell, called threading.
Post by Dann Corbit
The startup stuff for PostgreSQL is just a few files. It does not seem
insurmountable to change it. But it is none of my business. If it is a
major hassle (for reasons which I am not aware) then I see no driving
reason to change it.
It has to be changed for Windows, it is a major hassle for reasons I
wasn't aware of, and I am half way through ;-)


Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== ***@Yahoo.com #

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Marek Mosiewicz
2002-06-18 20:50:16 UTC
Permalink
I know that Apache Group created special library to handle difference
between different platforms (including win32). They had similar problems
porting Apache to Windows. They build very portable threads api (win32,
POSIX, native Linux thread and more) There is also all IPC stuff (mutex,
signals mmap etc.) and many more. This functions work both on unix and
windows and use most effective implementation (e.g. POSIX functions on
Winodws are slow compared to native).

http://apr.apache.org/



-----Original Message-----
From: pgsql-hackers-***@postgresql.org
[mailto:pgsql-hackers-***@postgresql.org] On Behalf Of Serge Adda
Sent: Tuesday, June 18, 2002 6:43 PM
To: 'Jan Wieck'; 'Dann Corbit'
Cc: 'Bruce Momjian'; 'Peter Eisentraut'; 'PostgreSQL-development'
Subject: Re: [HACKERS] Roadmap for a Win32 port

Hello,

I am new to PostgreSQL, but I am interested in the Win32 port.
I have studied the architecture of other databases like Oracle.

They have had to turn their multi-process model used on Unix into a
fully
multi-threaded one on Win32. I have the feeling that they have had the
same
debate that the one you have.

The CreateProcess() syscall is very costly on Windows. Some improvements
have been done in Windows XP but it is still far more costly than a Unix
fork().

I have been programming with threads on NT for a long time now.
They are quiet robust and efficient. I fear that it is the only
successful
way to port PostgreSQL.

Sorry for this interruption,
Serge

-----Original Message-----
From: Jan Wieck [mailto:***@Yahoo.com]
Sent: Tuesday, June 18, 2002 16:07
To: Dann Corbit
Cc: Bruce Momjian; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Dann Corbit
-----Original Message-----
Sent: Monday, June 17, 2002 6:20 PM
To: Dann Corbit
Cc: Jan Wieck; Peter Eisentraut; PostgreSQL-development
Subject: Re: [HACKERS] Roadmap for a Win32 port
Post by Dann Corbit
Post by Bruce Momjian
Post by Jan Wieck
It will be at least another copy of the postmaster (dot exe).
Yea, I just liked the idea of the postmaster binary
somehow reporting
Post by Dann Corbit
Post by Bruce Momjian
the postmaster status. Seems it is in a better position to
do that than
a shell script.
The Postmaster is about 100x bigger than it needs to be.
The Postmaster needs to set up shared memory and launch servers.
It
Post by Dann Corbit
Post by Dann Corbit
does not need to know anything about SQL grammar or any of that
rigamarole.
It could be a 15K executable.
Why not have an itty-bitty Postmaster that does nothing but
a spawn or a
Post by Dann Corbit
create process to create threaded Postgres instances?
Can't. postmaster/postgres are symlinks to the same file,
and we fork()
from postmaster to create backends. All the code has to be in the
postmaster so the fork works.
Is fork() faster than creation of a new process via exec()? After the
creation of the shared memory, the information needed to use it could be
passed to the Postgres servers on the command line.
exec() does NOT create new processes. It loads another executable file
into the existing, calling process.

fork() duplicates the calling process. In modern unix variants, this is
done in a very efficient way, so that the text segment (program code) is
shared readonly and everything else (data and stack segments) are shared
copy on write. Thus, fork() itself doesn't even cause memory copying.
That happens later when one of the now two processes writes to a memory
page the first time.

Windows does not have these two separate steps. It wants the full blown
expensive "create process and load executable", or the "let's all muck
around with the same handles" modell, called threading.
Post by Dann Corbit
The startup stuff for PostgreSQL is just a few files. It does not seem
insurmountable to change it. But it is none of my business. If it is a
major hassle (for reasons which I am not aware) then I see no driving
reason to change it.
It has to be changed for Windows, it is a major hassle for reasons I
wasn't aware of, and I am half way through ;-)


Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== ***@Yahoo.com #

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html


---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Jean-Michel POURE
2002-06-20 10:01:53 UTC
Permalink
Post by Serge Adda
I am new to PostgreSQL, but I am interested in the Win32 port.
I have studied the architecture of other databases like Oracle.
Hello,

It seems clear that several teams are working without central point management
and contact:

- W32 port: at least a Japase team, a PostgreSQL hacker team and a company are
working separately. This makes three separate efforts for the most important
project this year.

- Replication: development is slow although a lot of people would be
interested in helping. But there is no central organization apart from the
hackers-list.

- Web site: a www list is working on several issues. Is there a central design
for all PostgreSQL site, like PHP does?

- Marketing: MySQL sucks and has a team of marketing sending junk technical
emails and writing false benchmarks. Who is in charge of marketing at
PostgreSQL? Where can I find a list of PostgreSQL features?

Personnaly, I agree with someone who wrote "Remember Betamax. It was the best
technical standard, but it died for commercial reasons". I don't want
PostgreSQL to be the next Betamax.

Some projects, like Debian, have a democratic organisation. The team leader is
elected for a year. Why not settle a similar organization? This would help
take decisions ... and not loose time on important issues.

PostgreSQL is a software but it is also a community. If we believe in
democracy, I suggest we should organize in a democratic way and elect a
leader for a year.

Just my 2 cents.
Cheers.
Jean-Michel POURE

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly
Karel Zak
2002-06-20 11:39:08 UTC
Permalink
Post by Jean-Michel POURE
Some projects, like Debian, have a democratic organisation. The team leader is
elected for a year. Why not settle a similar organization? This would help
IMHO there is not problem with organization -- I don't know what do
you want to organize on actual number of developers / contributors
:-)

The PostgreSQL is not project like Debian -- if you want to work on Debian
you must know package system and Debian standards only. But if you want to
work on PostgreSQL you must be at the least average programmer (means several
years experience) and you must know a great many of current PostgreSQL code.
Post by Jean-Michel POURE
PostgreSQL is a software but it is also a community. If we believe in
democracy, I suggest we should organize in a democratic way and elect a
leader for a year.
What is non-democratic now?

Karel
--
Karel Zak <***@zf.jcu.cz>
http://home.zf.jcu.cz/~zakkr/

C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Jean-Michel POURE
2002-06-20 12:33:04 UTC
Permalink
Post by Karel Zak
IMHO there is not problem with organization -- I don't know what do
you want to organize on actual number of developers / contributors
Dear Karel,

My previous e-mail points out several projects where, IMHO, a leadership would
benefit the community at large :
- replication,
- W32 port,
- marketing (read the post "Read this and puke").
Post by Karel Zak
What is non-democratic now?
The current processes are based on discussion, and therefore are democratic.
My proposal does not intend to change discussion processes between
pgsql-hackers.

But, in order to face companies like MySQL AB, Oracle or Micro$oft, the
community needs to take important decisions that will help team work. A
clarified organization would help.

Please note I am not a PostgreSQL hacker myself, as I do not contribute code
to PostgreSQL main sources. But, as an outside spectator, I would only like
to point out that some efforts need coordination.

Debian is a very interesting example of Open-Source organization, as for all
aspects linked to "decision making". Usually, at Debian, when a discussion is
driven, a clear choice arizes after a limited time. Projects are sometimes
slow, but always reach their goals.

As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code? Can someone publish a
schedule for replication availability? Who is in charge of explaining newbees
that MySQL InnoDB is just a marketing lie? What is the current PostgreSQL
market share?

In other words, we should ask ourselves the question of PostgreSQL future
organization. We come to point where PostgreSQL has equal chances to become
the #1 database or die like Betamax.

Best regards to you all,
Jean-Michel

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Karel Zak
2002-06-20 13:05:59 UTC
Permalink
Post by Jean-Michel POURE
Post by Karel Zak
IMHO there is not problem with organization -- I don't know what do
you want to organize on actual number of developers / contributors
My previous e-mail points out several projects where, IMHO, a leadership would
- replication,
- W32 port,
- marketing (read the post "Read this and puke").
I understend you. I saw a lot of project and ideas, but it
_always_ depend on people and their time. You can organize, you can
prepare cool planns, but don't forget -- in finale must somebody
implement it. Who? I don't know method how clone Tom Lane or get
money for others developers who can't full time work on PostgreSQL
now.
Post by Jean-Michel POURE
As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code? Can someone publish a
If nobody -- you can test it, make it better and write something
about it. If nobody works on some theme it means this theme is not
important for now. BUT everybody can change it and everybody can
start work on arbitrary TODO item. It seems hard, but it's right :-)

Karel
--
Karel Zak <***@zf.jcu.cz>
http://home.zf.jcu.cz/~zakkr/

C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Jan Wieck
2002-06-20 13:08:20 UTC
Permalink
Post by Jean-Michel POURE
[...]
As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code? Can someone publish a
schedule for replication availability? Who is in charge of explaining newbees
that MySQL InnoDB is just a marketing lie? What is the current PostgreSQL
market share?
I think the first native Win32 port that gets contributet will make it.
So far I have seen a couple of people claiming the "have" a native Win32
port, not using CygWIN. But none of them was willing to contribute their
work.

There is no official "schedule" for any of the features. PostgreSQL is a
100% volunteer project, so people work on it if and when they feel the
need to AND have the time. Even if some of us do work on PostgreSQL at
their job, a few even fulltime, you will not see such commitments.

And why do you think InnoDB is a marketing lie? It bought MySQL a good
number of features, including row level locking, transactions and
limited foreign key support. Actually it stopped MySQL from yelling out
the FUD they told people far too long, like "transactions are useless
overhead" and "referential integrity is bad".


Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== ***@Yahoo.com #

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Bruce Momjian
2002-06-20 16:31:46 UTC
Permalink
Post by Jan Wieck
Post by Jean-Michel POURE
[...]
As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code? Can someone publish a
schedule for replication availability? Who is in charge of explaining newbees
that MySQL InnoDB is just a marketing lie? What is the current PostgreSQL
market share?
I think the first native Win32 port that gets contributet will make it.
This line bothers me. With multiple people working on Win32, I would
like us to decide how we would _like_ such a port to be implemented. I
think this will assist those working on the project to _know_ that their
work will be accepted if submitted.

Also, I encourage people working on Win32 ports to contribute their work
_as_ the complete each module, rather than as one huge patch. That way,
other Win32 porters can benefit from their work and maybe we can get
this done in 1/2 the time.

What I don't want to happen is two Win32 projects contributing duplicate
code at the same time. It is a waste when they could have combined
their efforts.

So, my message to Win32 porters is to communicate what you are doing,
including implementation details, and contribute back as soon as you
can. If you don't, someone else may beat you to it, and your patch may
be rejected, leaving you to either scrap your work or continually port
your changes to every future PostgreSQL release.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Marc G. Fournier
2002-06-20 17:23:02 UTC
Permalink
Post by Bruce Momjian
Post by Jan Wieck
Post by Jean-Michel POURE
[...]
As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code? Can someone publish a
schedule for replication availability? Who is in charge of explaining newbees
that MySQL InnoDB is just a marketing lie? What is the current PostgreSQL
market share?
I think the first native Win32 port that gets contributet will make it.
This line bothers me. With multiple people working on Win32, I would
like us to decide how we would _like_ such a port to be implemented. I
think this will assist those working on the project to _know_ that their
work will be accepted if submitted.
1. that is not accurate, as it won't necessarily be accepted if submitted
... its not like years ago when are standards were lax ... Jan is
perfectly accurate in his assessment of which will make it in, except for
omitting one point ... it has to meet our standards ... so, its more "the
first to contribute *and* meet our standards" ...
Post by Bruce Momjian
Also, I encourage people working on Win32 ports to contribute their work
_as_ the complete each module, rather than as one huge patch. That way,
other Win32 porters can benefit from their work and maybe we can get
this done in 1/2 the time.
This is what should be done anyway ... I wouldn't even be so much worried
abou t"other Win32 porters" as much as trying to integrate that 'hugh
patch' at a later date, consiedering all the changes that go into our code
:)
Post by Bruce Momjian
What I don't want to happen is two Win32 projects contributing duplicate
code at the same time. It is a waste when they could have combined
their efforts.
IMHO, that is actually their problem ... without meaning to sound crass
about it, but its not like we haven't discussed it extensively here, and
openly ... hell, we've even tried to break down the whole project into
smaller components to make the whole easier to merge in :)
Post by Bruce Momjian
So, my message to Win32 porters is to communicate what you are doing,
including implementation details, and contribute back as soon as you
can. If you don't, someone else may beat you to it, and your patch may
be rejected, leaving you to either scrap your work or continually port
your changes to every future PostgreSQL release.
Agreed here ... and this doesn't just apply to Win32 porters ... there
have been *alot* of "big projects" worked on over the years that have been
implemented in pieces so that code-base doesn't diverge too much, making
patching difficult ... the last thing anyone should want is to work for a
few months to create a nice big patch to find out that what they have
"fixed" got yanked out of the code already :)



---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Jan Wieck
2002-06-20 18:57:20 UTC
Permalink
Post by Marc G. Fournier
Post by Bruce Momjian
This line bothers me. With multiple people working on Win32, I would
like us to decide how we would _like_ such a port to be implemented. I
think this will assist those working on the project to _know_ that their
work will be accepted if submitted.
1. that is not accurate, as it won't necessarily be accepted if submitted
... its not like years ago when are standards were lax ... Jan is
perfectly accurate in his assessment of which will make it in, except for
omitting one point ... it has to meet our standards ... so, its more "the
first to contribute *and* meet our standards" ...
Omitted point taken :-)
Post by Marc G. Fournier
Post by Bruce Momjian
What I don't want to happen is two Win32 projects contributing duplicate
code at the same time. It is a waste when they could have combined
their efforts.
IMHO, that is actually their problem ... without meaning to sound crass
about it, but its not like we haven't discussed it extensively here, and
openly ... hell, we've even tried to break down the whole project into
smaller components to make the whole easier to merge in :)
The problem with this kind of project is that you have a big stumbling
block at the beginning, which has to be done before you can rollout and
integrate the help of developers scattered around the globe. This was
the case with the foreign key project, where the trigger queue and one
set of triggers where working, and then Stephan did all the others and I
forgot who else helped to do the utility commands and CREATE TABLE
syntax and tried to decrypt the SQL definitions? In the Windows port
case it is to get it as far that you at least can fire up a postmaster,
get past the startup process, connect to the database and do a few
queries before the thing blows up. Before this everybody has exactly the
same problem, "It doesn't startup", so the likelyhood of everyone
stomping over each others work every single night is about 99.9%!


Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== ***@Yahoo.com #

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Bruce Momjian
2002-06-20 19:37:14 UTC
Permalink
Post by Jan Wieck
Post by Marc G. Fournier
Post by Bruce Momjian
What I don't want to happen is two Win32 projects contributing duplicate
code at the same time. It is a waste when they could have combined
their efforts.
IMHO, that is actually their problem ... without meaning to sound crass
about it, but its not like we haven't discussed it extensively here, and
openly ... hell, we've even tried to break down the whole project into
smaller components to make the whole easier to merge in :)
The problem with this kind of project is that you have a big stumbling
block at the beginning, which has to be done before you can rollout and
integrate the help of developers scattered around the globe. This was
the case with the foreign key project, where the trigger queue and one
set of triggers where working, and then Stephan did all the others and I
forgot who else helped to do the utility commands and CREATE TABLE
syntax and tried to decrypt the SQL definitions? In the Windows port
case it is to get it as far that you at least can fire up a postmaster,
get past the startup process, connect to the database and do a few
queries before the thing blows up. Before this everybody has exactly the
same problem, "It doesn't startup", so the likelyhood of everyone
stomping over each others work every single night is about 99.9%!
Yes, but it doesn't prevent discussion. I think open implementation
discussion will help. I am suggesting this to everyone, not just Jan.
I have been in private discussion with others too.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Jon Franz
2002-06-21 00:22:27 UTC
Permalink
It could be helpful to create a mailing list just for this project,
since not all members of pg-hackers will/shall participate, and we
would probably flood this list quite a bit trying to figure out what
is the best way to implement a win32 port. Just like the
pg-replication list, this new list would be project specific.

However, as an aside, I think the 'first best fit shall be commited'
approach is a _bad_ idea. Everyone (whos interested in the port)
agrees with the basic goals, and we will get a working system much
faster if we all work on a single solution: And not try to race each
other.
If the main pg developers do not want to bless a specific method/project
for the port, then the people interested should hash it out, before
hundreds of man-hours are wasted developing something that ends up not
being used. Debuging-into existence is a bad idea, as the single-night
example hints at (wether intentionaly or not) - with a proper plan we
should be able to create unit tests that can prove whether the methods
choosen are functioning well before we ever get a fully working
postmaster.

~Jon Franz
Post by Bruce Momjian
Post by Jan Wieck
Post by Marc G. Fournier
Post by Bruce Momjian
What I don't want to happen is two Win32 projects contributing duplicate
code at the same time. It is a waste when they could have combined
their efforts.
IMHO, that is actually their problem ... without meaning to sound crass
about it, but its not like we haven't discussed it extensively here, and
openly ... hell, we've even tried to break down the whole project into
smaller components to make the whole easier to merge in :)
The problem with this kind of project is that you have a big stumbling
block at the beginning, which has to be done before you can rollout and
integrate the help of developers scattered around the globe. This was
the case with the foreign key project, where the trigger queue and one
set of triggers where working, and then Stephan did all the others and I
forgot who else helped to do the utility commands and CREATE TABLE
syntax and tried to decrypt the SQL definitions? In the Windows port
case it is to get it as far that you at least can fire up a postmaster,
get past the startup process, connect to the database and do a few
queries before the thing blows up. Before this everybody has exactly the
same problem, "It doesn't startup", so the likelyhood of everyone
stomping over each others work every single night is about 99.9%!
Yes, but it doesn't prevent discussion. I think open implementation
discussion will help. I am suggesting this to everyone, not just Jan.
I have been in private discussion with others too.
---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Bruce Momjian
2002-06-21 00:44:19 UTC
Permalink
Post by Jon Franz
It could be helpful to create a mailing list just for this project,
since not all members of pg-hackers will/shall participate, and we
would probably flood this list quite a bit trying to figure out what
is the best way to implement a win32 port. Just like the
pg-replication list, this new list would be project specific.
However, as an aside, I think the 'first best fit shall be commited'
approach is a _bad_ idea. Everyone (whos interested in the port)
agrees with the basic goals, and we will get a working system much
faster if we all work on a single solution: And not try to race each
other.
I think we have to be involved to prevent chaos when those patches
arrive.
Post by Jon Franz
If the main pg developers do not want to bless a specific method/project
for the port, then the people interested should hash it out, before
hundreds of man-hours are wasted developing something that ends up not
being used. Debuging-into existence is a bad idea, as the single-night
example hints at (wether intentionaly or not) - with a proper plan we
should be able to create unit tests that can prove whether the methods
choosen are functioning well before we ever get a fully working
postmaster.
Actually, don't we have a cygwin mailing list? Seems that would be a
great location, except for the name. Maybe Marc can close the list and
migrate them all to a new 'win32' list.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly
Marc G. Fournier
2002-06-21 01:47:32 UTC
Permalink
Post by Bruce Momjian
Post by Jon Franz
It could be helpful to create a mailing list just for this project,
since not all members of pg-hackers will/shall participate, and we
would probably flood this list quite a bit trying to figure out what
is the best way to implement a win32 port. Just like the
pg-replication list, this new list would be project specific.
However, as an aside, I think the 'first best fit shall be commited'
approach is a _bad_ idea. Everyone (whos interested in the port)
agrees with the basic goals, and we will get a working system much
faster if we all work on a single solution: And not try to race each
other.
I think we have to be involved to prevent chaos when those patches
arrive.
Post by Jon Franz
If the main pg developers do not want to bless a specific method/project
for the port, then the people interested should hash it out, before
hundreds of man-hours are wasted developing something that ends up not
being used. Debuging-into existence is a bad idea, as the single-night
example hints at (wether intentionaly or not) - with a proper plan we
should be able to create unit tests that can prove whether the methods
choosen are functioning well before we ever get a fully working
postmaster.
Actually, don't we have a cygwin mailing list? Seems that would be a
great location, except for the name. Maybe Marc can close the list and
migrate them all to a new 'win32' list.
Two different issues here ... a win32 port is different then the cygwin
issues ... also, I had suggested ages back creatin ga seperate list, but
was told that it would frustrate the effort ... so unless those that
disagreed with it in the first place have changed their mind, the
discussion of any porting effort to win32 should remain here ...



---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Lamar Owen
2002-06-20 19:56:53 UTC
Permalink
Post by Jan Wieck
set of triggers where working, and then Stephan did all the others and I
forgot who else helped to do the utility commands and CREATE TABLE
syntax and tried to decrypt the SQL definitions?
Don Baccus?
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Mike Mascari
2002-06-20 19:58:17 UTC
Permalink
...
Post by Jan Wieck
Post by Marc G. Fournier
IMHO, that is actually their problem ... without meaning to sound crass
about it, but its not like we haven't discussed it extensively here, and
openly ... hell, we've even tried to break down the whole project into
smaller components to make the whole easier to merge in :)
The problem with this kind of project is that you have a big stumbling
block at the beginning, which has to be done before you can rollout and
integrate the help of developers scattered around the globe. This was
the case with the foreign key project, where the trigger queue and one
set of triggers where working, and then Stephan did all the others and I
forgot who else helped to do the utility commands and CREATE TABLE
syntax and tried to decrypt the SQL definitions? In the Windows port
case it is to get it as far that you at least can fire up a postmaster,
get past the startup process, connect to the database and do a few
queries before the thing blows up. Before this everybody has exactly the
same problem, "It doesn't startup", so the likelyhood of everyone
stomping over each others work every single night is about 99.9%!
It would be nice to also have it "fire up" under Windows CE as well ;-)

Mike Mascari
***@mascari.com

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Stephan Szabo
2002-06-20 21:58:54 UTC
Permalink
Post by Jan Wieck
Post by Marc G. Fournier
Post by Bruce Momjian
This line bothers me. With multiple people working on Win32, I would
like us to decide how we would _like_ such a port to be implemented. I
think this will assist those working on the project to _know_ that their
work will be accepted if submitted.
1. that is not accurate, as it won't necessarily be accepted if submitted
... its not like years ago when are standards were lax ... Jan is
perfectly accurate in his assessment of which will make it in, except for
omitting one point ... it has to meet our standards ... so, its more "the
first to contribute *and* meet our standards" ...
Omitted point taken :-)
Post by Marc G. Fournier
Post by Bruce Momjian
What I don't want to happen is two Win32 projects contributing duplicate
code at the same time. It is a waste when they could have combined
their efforts.
IMHO, that is actually their problem ... without meaning to sound crass
about it, but its not like we haven't discussed it extensively here, and
openly ... hell, we've even tried to break down the whole project into
smaller components to make the whole easier to merge in :)
The problem with this kind of project is that you have a big stumbling
block at the beginning, which has to be done before you can rollout and
integrate the help of developers scattered around the globe. This was
the case with the foreign key project, where the trigger queue and one
set of triggers where working, and then Stephan did all the others and I
forgot who else helped to do the utility commands and CREATE TABLE
syntax and tried to decrypt the SQL definitions? In the Windows port
Actually, IIRC Don did the triggers, and I did the utility commands/create
stuff, but the point is still the same. (Made in the point of
historical accuracy since I don't want someone else's work to end up
getting attributed to me since that's unfair to them. :) )


---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Jan Wieck
2002-06-21 00:17:41 UTC
Permalink
Post by Stephan Szabo
Actually, IIRC Don did the triggers, and I did the utility commands/create
stuff, but the point is still the same. (Made in the point of
historical accuracy since I don't want someone else's work to end up
getting attributed to me since that's unfair to them. :) )
This is why I love this PostgreSQL Project. The honesty and
humbleness everyone is practicing. Who ever did what, together we
did a good job! Which other open source database has the full set
of referential actions and deferrability?


Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being
right. #
# Let's break this rule - forgive
me. #
#==================================================
***@Yahoo.com #

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Gavin Sherry
2002-06-20 13:17:16 UTC
Permalink
Post by Jean-Michel POURE
Post by Karel Zak
IMHO there is not problem with organization -- I don't know what do
you want to organize on actual number of developers / contributors
=20
Dear Karel,
=20
My previous e-mail points out several projects where, IMHO, a leadership =
would=20
Post by Jean-Michel POURE
- replication,
- W32 port,
- marketing (read the post "Read this and puke").
=20
Post by Karel Zak
What is non-democratic now?
=20
The current processes are based on discussion, and therefore are democrat=
ic.=20
Post by Jean-Michel POURE
My proposal does not intend to change discussion processes between=20
pgsql-hackers.
=20
But, in order to face companies like MySQL AB, Oracle or Micro$oft, the=
=20
Post by Jean-Michel POURE
community needs to take important decisions that will help team work. A=
=20
Post by Jean-Michel POURE
clarified organization would help.
Jean,

Why on earth does this matter? Postgres will continue to be a good
database as long as developers and users cut code. It is not a
sufficiently complicated project to warrant too much concern about things
like this. Besides, a significant amount of the code committed to
Postgres is inspired by personal interest not obligation.

Gavin



---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Tom Lane
2002-06-20 13:22:53 UTC
Permalink
Post by Jean-Michel POURE
As for current PostgreSQL organization, can someone explain me which
W32 port will make its way to PostgreSQL main source code? Can someone
publish a schedule for replication availability? Who is in charge of
explaining newbees that MySQL InnoDB is just a marketing lie? What is
the current PostgreSQL market share?
And an "elected leader" will make all this stuff clear exactly how?

If someone would like to step up and actually *do* that marketing work,
that'd be great. Complaining that it's not being done is a waste of
list bandwidth. Electing someone isn't going to magically make it
happen, either.

It does concern me that there seem to be several W32 porting efforts
going on without contact with each other or with the wider pghackers
community; but if they want to work that way, I don't think there's
much we can do to force them to come out in the open.

BTW, we do already have a recognized leadership group: the core
committee. The committee members mostly prefer to lead by example
and by consensus, rather than trying to impose their will on others,
which is maybe why you hadn't noticed.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Jean-Michel POURE
2002-06-20 14:00:04 UTC
Permalink
Post by Tom Lane
BTW, we do already have a recognized leadership group: the core
committee. The committee members mostly prefer to lead by example
and by consensus, rather than trying to impose their will on others,
which is maybe why you hadn't noticed.
You are right. Leading by example and consensus is the way things work. If the
committee can publish clear goals, there is no need to elect a leader.

Now I realize my concern is about project management. Do you think the
committee could publish a project web page showing who works on what?

Someone willing to help on a projet would view the page and contact
developpers more easilly. Presently, there is no clear knowledge who is
working on the W32 port or on replication. Using a project page, there would
be less chances to see new forks of particular projects (W32, replication).

This page would be accessible from the to-do-list. Conversly, the to-do-list
would lead to the projects page.

Just my 2 cents.
Thanks to all of you who replied my mail.

Cheers,
Jean-Michel

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Marc G. Fournier
2002-06-20 17:30:50 UTC
Permalink
Post by Jean-Michel POURE
Post by Tom Lane
BTW, we do already have a recognized leadership group: the core
committee. The committee members mostly prefer to lead by example
and by consensus, rather than trying to impose their will on others,
which is maybe why you hadn't noticed.
You are right. Leading by example and consensus is the way things work. If the
committee can publish clear goals, there is no need to elect a leader.
Now I realize my concern is about project management. Do you think the
committee could publish a project web page showing who works on what?
No ... cause nobody works on anything specific ... they work on what
appeals to them, as it appeals to them ...
Post by Jean-Michel POURE
Someone willing to help on a projet would view the page and contact
developpers more easilly. Presently, there is no clear knowledge who is
working on the W32 port or on replication. Using a project page, there
would be less chances to see new forks of particular projects (W32,
replication).
If someone workign on the w32 port isn't willing to announce such on
-hackers, how do you think we'd get that information for a web page? The
easiest way to 'contact developers' is to post to this list, and then you
don't restrict yourself to anyone one person, but everyone involved in teh
project ...

I think you are confusing a project on the scale of an operating system
with PgSQL ... an OS has to have someone at the top to make sure each
sub-package integrates well into the overall OS ... we *are* the
sub-package, and our "leaders" are essentially "our developers" ... we
have a core committee that consists of several of us that brought PgSQL
out of Berkeley and into the real world, but if you saw the archives for
that list, you woudln't see much, since we tend to try and keep all
discussions *on* -hackers ...


---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Marc G. Fournier
2002-06-20 17:15:31 UTC
Permalink
Post by Jean-Michel POURE
As for current PostgreSQL organization, can someone explain me which W32
port will make its way to PostgreSQL main source code?
Whichever one actually submits patches for review first that is deemed
acceptable for inclusion ... as its always been ...
Post by Jean-Michel POURE
Can someone publish a schedule for replication availability?
Its available now, and has been for several months *shrug* There is a
project called PgReplication that is Open Source (hopefully someone from
that camp will pop up) ... and PgSQL, Inc has several deployments of their
commercial replication out there now, with several more pending ...

In fact, all of our work has been based on the rserv code that is in
contrib, and that we released over (almost?) a year ago ... but I've seen
nobody actually try to build on it, altho its what we've extended and are
using sucessfully in production environments ...
Post by Jean-Michel POURE
Who is in charge of explaining newbees that MySQL InnoDB is just a
marketing lie?
You are ... and anyone else that asks about it / mentions it ...
Post by Jean-Michel POURE
What is the current PostgreSQL market share?
If you can think of a method of calculating this, as there is no
'commercial licensing' involved, please let us know ... would love to find
out ... there have been several surveys about it, but, quite frankly,
without having some sort of 'licensing' required to use, there is zero way
of getting any *real* numbers on this ...
Post by Jean-Michel POURE
In other words, we should ask ourselves the question of PostgreSQL
future organization. We come to point where PostgreSQL has equal chances
to become the #1 database or die like Betamax.
At this point in time, the current organization == future organization ...
you are putting a 'marketing effort' as being the responsibility of the
open source project, and using MySQL AB as a comparison ... MySQL AB is a
*commercial company*, as is PostgreSQL, Inc -and- SRA and several other
newcomers, all of whom are doing marketing in their own way, based on
budget and requirements for growth ...

The "developmental organization", which we are, has been successful for
the past 7 years now ... flame wars are minimal, as are disagreements ...
there are some patches that get rejected that those generating them are
disappointed about, but most of *them* just bounce back with improvements
based on what has been told to them as being unacceptable ... about the
only thing that *has* changed over the past 7 years is that our standards
are tightened up as we move from mainly fixing bugs/stability to improving
the server itself ...




---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Bruce Momjian
2002-06-20 16:09:35 UTC
Permalink
Post by Serge Adda
Post by Serge Adda
I am new to PostgreSQL, but I am interested in the Win32 port.
I have studied the architecture of other databases like Oracle.
Hello,
It seems clear that several teams are working without central point management
- W32 port: at least a Japanese team, a PostgreSQL hacker team and a company are
working separately. This makes three separate efforts for the most important
project this year.
Funny you should mention that. I talked with three people on the phone
just yesterday in an attempt to get them together on the Win32 port. I
have been hesitant to publicly try and join them together because they
haven't publicly stated if they are going to contribute their code
back to the project, or haven't been clear on _when_ they would
contribute it back.

My goal is to take the roadmap I posted on June 5, make a web page, and
get some agreement from the community on how to address each item:

http://candle.pha.pa.us/cgi-bin/pgtodo?win32
.
Then, the groups can know how we prefer to have things done, and if they
contribute back quickly, the other projects can benefit from their work,
and we _don't_ get conflicting Win32 implementations contributed back to
the project.
Post by Serge Adda
- Replication: development is slow although a lot of people would be
interested in helping. But there is no central organization apart from the
hackers-list.
I am on the replication mailing list:

http://gborg.postgresql.org/project/pgreplication/projdisplay.php

That project is moving along, and hopefully they can merge their code
into the main tree so others can assist them. Not sure what else we can
do.
Post by Serge Adda
- Web site: a www list is working on several issues. Is there a central design
for all PostgreSQL site, like PHP does?
- Marketing: MySQL sucks and has a team of marketing sending junk technical
emails and writing false benchmarks. Who is in charge of marketing at
PostgreSQL? Where can I find a list of PostgreSQL features?
I skipped commenting on the marketing discussion because I was working
on a patch. Probably our biggest problem compared to MySQL is that
MySQL is a single company behind a single database, while we have >4
companies behind PostgreSQL, and their thrust is promoting their company
rather than PostgreSQL itself.

While we have an excellent development team, we don't have a funded team
to go around to trade shows and write articles promoting PostgreSQL. We
only have a very few people employed full-time with PostgreSQL, and it
usually takes a few full-time people to do just marketing. Great Bridge
was really the only one to push PostgreSQL, and they are gone.

MySQL has such a team, and so does Oracle, and it helps. Linux was in a
similar boat, with multiple companies behind Linux, and every one
promoting its own company rather than Linux itself. We need large
PostgreSQL companies that promote themselves, and PostgreSQL along with
it. Linux is in the same boat, with distributors promoting themselves
and Linux along with it.
Post by Serge Adda
Personnaly, I agree with someone who wrote "Remember Betamax. It was the best
technical standard, but it died for commercial reasons". I don't want
PostgreSQL to be the next Betamax.
I thought the problem with Betamax was that Sony controlled the
standard, and other companies didn't like that.


I am always looking for more ideas in these areas.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Thomas Lockhart
2002-06-20 16:23:18 UTC
Permalink
...
Post by Bruce Momjian
MySQL has such a team, and so does Oracle, and it helps. Linux was in a
similar boat, with multiple companies behind Linux, and every one
promoting its own company rather than Linux itself. We need large
PostgreSQL companies that promote themselves, and PostgreSQL along with
it. Linux is in the same boat, with distributors promoting themselves
and Linux along with it.
Right. That may have slowed down the trade press recognition of Linux,
but in the end Linux has risen into view by popular demand and acclaim.

PostgreSQL has a similar opportunity and can succeed on a similar path.
We do have a superior technical solution, we have a superior open-source
support system, and we have (imho :) a superior development model and
development team open to new contributions and innovation.

The basis is there for success into the future, and at some point we
will get a lot for a little effort on marketing. Other products
bootstrap themselves by marketing early and often. And to the detriment
of the technical side.

- Thomas

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Neil Conway
2002-06-20 18:22:39 UTC
Permalink
On Thu, 20 Jun 2002 12:09:35 -0400 (EDT)
Post by Bruce Momjian
Post by Jean-Michel POURE
- Replication: development is slow although a lot of people would be
interested in helping. But there is no central organization apart from the
hackers-list.
Replication development is co-ordinated on the pgreplication-general
list which Bruce mentions below, not on -hackers. Any interested
developers should subscribe to it, read the relevant research papers
on Postgres-R, and contribute code.

As for the speed of development, I've started to contribute code recently,
and Darren Johnson (the main replication developer) says he should have some
free time soon -- there are also some other new developers interested in
the project. So while there was a period of inactivity, I think that progress
is now being made.
Post by Bruce Momjian
http://gborg.postgresql.org/project/pgreplication/projdisplay.php
That project is moving along, and hopefully they can merge their code
into the main tree so others can assist them. Not sure what else we can
do.
I can't make any claims about a schedule -- since everyone working on the
project is a volunteer, the only release goals I'd like to set are "when it's
ready". The code right now is pretty unstable, so I don't think it's appropriate
for the main CVS tree until we work it into better shape.

Cheers,

Neil
--
Neil Conway <***@rogers.com>
PGP Key ID: DB3C29FC

---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Hannu Krosing
2002-06-20 14:06:24 UTC
Permalink
Post by Jean-Michel POURE
Post by Karel Zak
IMHO there is not problem with organization -- I don't know what do
you want to organize on actual number of developers / contributors
Dear Karel,
My previous e-mail points out several projects where, IMHO, a leadership would
- replication,
- W32 port,
- marketing (read the post "Read this and puke").
Post by Karel Zak
What is non-democratic now?
The current processes are based on discussion, and therefore are democratic.
My proposal does not intend to change discussion processes between
pgsql-hackers.
But, in order to face companies like MySQL AB, Oracle or Micro$oft, the
community needs to take important decisions that will help team work. A
clarified organization would help.
In what way ?

Do you really think that if we would elect Tom or Bruce or someone else
"The President of the PostgreSQL Community" then their word would weight
more in mainstream press ?
Post by Jean-Michel POURE
Please note I am not a PostgreSQL hacker myself, as I do not contribute code
to PostgreSQL main sources. But, as an outside spectator, I would only like
to point out that some efforts need coordination.
Debian is a very interesting example of Open-Source organization, as for all
aspects linked to "decision making". Usually, at Debian, when a discussion is
driven, a clear choice arizes after a limited time. Projects are sometimes
slow, but always reach their goals.
From an "outside spactator" perspective Debian seems to be really,
really slow.
Post by Jean-Michel POURE
As for current PostgreSQL organization, can someone explain me which W32 port
will make its way to PostgreSQL main source code?
There is already one W32 port in main source (the one that uses cygwin).

IMHO the first native port that
a) works
b) does not make *nix version slower/harder to maintain
and
c) is submitted for inclusion will make it into main source.
Post by Jean-Michel POURE
Can someone publish a schedule for replication availability?
I guess any of the teams working on different ways to replicate could do
it. Another question is if they can stick to it.
Post by Jean-Michel POURE
Who is in charge of explaining newbees that MySQL InnoDB is just a
marketing lie?
Nobody is "in charge", but everybody is welcome to do it, even without
being "elected" or "nominated ";)

Still, having a "success stories" or "advocacy" section on
www.postgresq.org seems like a good idea.
Post by Jean-Michel POURE
What is the current PostgreSQL market share?
In other words, we should ask ourselves the question of PostgreSQL future
organization.
The current organization is "a loosely knit team" which seems to work
quite well.
Post by Jean-Michel POURE
We come to point where PostgreSQL has equal chances to become
the #1 database or die like Betamax.
Open-source will probably work differently, i.e postgres will probably
not die even if it will not be #1.

----------------
Hannu

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Marc G. Fournier
2002-06-20 17:32:28 UTC
Permalink
Post by Hannu Krosing
Nobody is "in charge", but everybody is welcome to do it, even without
being "elected" or "nominated ";)
Still, having a "success stories" or "advocacy" section on
www.postgresq.org seems like a good idea.
Being worked on ... we are actually working on totally revamping and
cleaning up the web site(s) ...



---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Josh Berkus
2002-06-24 21:44:37 UTC
Permalink
Jean-Michel,
Post by Jean-Michel POURE
It seems clear that several teams are working without central point
management
<snip>
Post by Jean-Michel POURE
- Marketing: MySQL sucks and has a team of marketing sending junk technical
emails and writing false benchmarks. Who is in charge of marketing at
PostgreSQL? Where can I find a list of PostgreSQL features?
<snip>
ome projects, like Debian, have a democratic organisation. The team leader is
Post by Jean-Michel POURE
elected for a year. Why not settle a similar organization? This would help
take decisions ... and not loose time on important issues.
PostgreSQL is a software but it is also a community. If we believe in
democracy, I suggest we should organize in a democratic way and elect a
leader for a year.
Let me introduce myself. In addition to being a contributor of supplimentary
documentation and the occasional spec to the PostgreSQL project, I am
volunteer marketing lead and the primary motivator for governance overhaul in
the OpenOffice.org project.

And frankly, I think you're way off base here. We have leaders: Tom, Bruce,
Jan, Stephan, Thomas, Marc and Oliver (did I miss anybody?). Frankly, if
OpenOffice.org had the kind of widely trusted, committed, involved in the
community core developers that PostgreSQL already has, I wouldn't be on my
fourth draft of an OpenOffice.org Community Council charter. OpenOffice.org
will have an election process because we are too big and too dispersed for a
simple trust network, not because we want one for its own sake.

PostgreSQL is, quite possibly, the smoothest-running Open Source project with
worldwide adoption. I find myself saying, at least once a week, "if only
project X were as well-organized as PostgreSQL!" It is perhaps not
coincidental that Postgres is one of the 15 or 20 oldest Open Source projects
(older than Linux, I believe).

How would a "democratic election" improve this? And why would we want an
elected person or body who was not a core developer? And if we elected a
core developer, why bother? They aready run things.

Regarding your marketing angle: Feel free to nominate yourself "PostgreSQL
Marketing Czar." Write articles. Contact journalists. Generate press
releases for each new Postgres version. Apply for a dot-org booth at
LinuxWorld. Nobody voted for me (actually, I got stuck with the job by not
protesting hard enough <grin>).

Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors. As often as not, the database engine choice is made by the
DBA, and they will choose PostgreSQL on its merits, not because of some
Washington Post article.

OpenOffice.org is a different story, as an end-user application. So we have
a Marketing Project.
--
-Josh Berkus
Porject Lead, OpenOffice.org Marketing



---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Tom Lane
2002-06-25 05:21:06 UTC
Permalink
Post by Josh Berkus
Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.
Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.

So far I have not worried about that scenario too much, because Monty
has always treated the MySQL sources as his personal preserve; if he
hadn't written it or closely reviewed it, it didn't get in, and if it
didn't hew closely to his opinion of what's important, it didn't get in.
But I get the impression that he's loosened up of late. If MySQL stops
being limited by what one guy can do or review, their rate of progress
could improve dramatically.

In short: we could use an organized marketing effort. I really
feel the lack of Great Bridge these days; there isn't anyone with
comparable willingness to expend marketing talent and dollars on
promoting Postgres as such. Not sure what to do about it. We've
sort of dismissed Jean-Michel's comments (and those of others in
the past) with "sure, step right up and do the marketing" responses.
But the truth of the matter is that a few amateurs with no budget
won't make much of an impression. We really need some professionals
with actual dollars to spend, and I don't know where to find 'em.

regards, tom lane



---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Marc G. Fournier
2002-06-25 07:55:52 UTC
Permalink
Post by Tom Lane
Post by Josh Berkus
Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.
Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.
Actually, I'm not sure how "viable" MySQL *is* in a commercial environment
... personally, I think that they just shot themselves in the foot with
their recent 'law suit' with NuSphere, no? Other then MySQL AB
themselves, how many are going to jump onto that bandwagon if nobody is
allowed *some* sort of competitive advantage?
Post by Tom Lane
So far I have not worried about that scenario too much, because Monty
has always treated the MySQL sources as his personal preserve; if he
hadn't written it or closely reviewed it, it didn't get in, and if it
didn't hew closely to his opinion of what's important, it didn't get in.
But I get the impression that he's loosened up of late. If MySQL stops
being limited by what one guy can do or review, their rate of progress
could improve dramatically.
I don't know ... again, my view of Monty is extended to "if it doesn't get
submitted for review, whether it gets in or not, we'll sue you for breach
of license" ... really gives those considering jumping onto MySQL a warm,
fuzzy feeling, no? :(




---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Tycho Fruru
2002-06-25 14:25:24 UTC
Permalink
Post by Tom Lane
Post by Josh Berkus
Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.
Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.
#pragma no-flames-please

When/if MySQL becomes better than PostgreSQL, we'd simply have a still
better open-source/free software database than we have now.

In what way exactly is this bad ?

Perhaps (hypothetically speaking) the "hot breath" of MySQL becoming
hotter and hotter will also induce more ideas and performance/enterprise
optimisations for postgres ...

If MySQL were better than PostgreSQL, you could really do two things :

(1) divert your development effort to MySQL, injecting it with the good
stuff known from the postgres effort
(2) make sure postgres becomes even better !
or
(3) turn away from computers and programming in disgust (not
recommended)

Of course, as things stand now, MySQL has still a long way to run before
it's up to par with PostgreSQL on enterprise-level database features.
But *both* are not yet at 100% !

I do agree we need more publicity for PostgreSQL, I mention it every
time I explain a database backend system to our prospects. I really am
a great fan of postgres and use it exclusively (except to look how
poorly other db's compare with it :-).

[snip]
Post by Tom Lane
But I get the impression that he's loosened up of late. If MySQL stops
being limited by what one guy can do or review, their rate of progress
could improve dramatically.
which in itself is not bad at all.

Look at it this way and perhaps the situation doesn't appear too
negative:

A guesstimate of 90% of all open-source/free software systems and
development is GNU/Linux. Nonetheless, the remaining 10% is *very*
active and are in some ways even ahead of the Linux track. In other
ways they are different. In still other ways, they're behind Linux.
For specific tasks, the *BSD family is vastly superior to Linux.
Everyone should use and support the tools that fit the bill.


For the marketing stuff, what about asking some big company's IT dept
for a statement, sort of "FooBarBank chooses/switches to PostgreSQL open
source database"? Then it's just a matter of making a press release
(wording is very important, anyone proficient in making press releases
here ?) and time them adequately.

I'll ask around here to see whether we can publicize some cases.

Cheers,
Tycho

/* this mail protected by No-Flam(tm) fire retardant asbestos
underwear (owwww itchy itchy) */
--
Tycho Fruru ***@fruru.com
"Prediction is extremely difficult. Especially about the future."
- Niels Bohr
Andrew Sullivan
2002-06-25 15:38:03 UTC
Permalink
Post by Tycho Fruru
Everyone should use and support the tools that fit the bill.
I've mentioned before, however, that "the tools that fit the bill" is
partly a function of network effects. The *BSD guys have the same
problem when facing the Linux juggernaut: as one system begins to
dominate the minds of certain types of people who happen to make a
lot of decisions, that system knocks other things out of the running
just by virtue of its PH quotient [1].
Post by Tycho Fruru
For the marketing stuff, what about asking some big company's IT dept
for a statement, sort of "FooBarBank chooses/switches to PostgreSQL open
source database"? Then it's just a matter of making a press release
(wording is very important, anyone proficient in making press releases
here ?) and time them adequately.
Best of luck. Here's the dirty secret about PostgreSQL: _lots_ of
big-ish companies are using it, and using it in important, central
functions of their organisations. But they're not willing to admit
it. What you always get is something like, "Yes, we're using an
enterprise-class system with good ANSI SQL 99 compliance, WAL, hot
backup, triggers, rules, an advanced, extensible datatypes system,
and excellent scalability to high concurrency. The system we're
using, **mumble PostmmuumblehandinfrontofmouthgrSQmllL **mumble**, is
very similar to ORACLE in a lot of respects. We have looked
carefully at ORACLE, and are always aware of the constantly-changing
database marketplace. We have a history of strong relationships with
vendors. . . ." You can substitute your favourite big-name RDBMS.
The point of such utterances seems mostly to be to get the name brand
inserted as often as possible, as though some sort of reflected glory
is the answer.

I don't know why this is. I am, to put it mildly, unbelievably
frustrated (not to say embarrassed) by at least one instance of it.
But it's nevertheless true.


[1] Pointy-hair quotient: the tendency of a given product name to
elicit recognition from a technical manager of dubious technical
ability.

A
--
----
Andrew Sullivan 87 Mowat Avenue
Liberty RMS Toronto, Ontario Canada
<***@libertyrms.info> M6K 3E3
+1 416 646 3304 x110




---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
Dave Cramer
2002-06-25 15:12:57 UTC
Permalink
IMO One of the big reasons that MySQL is viewed as being better is it's
percieved simplicity. It has a large following because of this, and many
of them are not experienced database users, in fact just the opposite.

This large user base is perhaps the best marketing that an open source
project can hope for. So I think that if we want to attract more users
we should try to make postgres easier to use. The hard part is how to do
this without sacrificing the integrity of the project. I think for
starters when evaluating the next feature we want to work on we ask the
following questions:

1) Does it make it easier to use for a non-dba ?
2) Does it facilitate making web-applications easier ( assuming that
this is the largest user base ) ?
3) I'm sure there are others, but at the moment I can't come up with
them.

Then if faced with a choice of implementing something which is going to
make postgres more technically complete or something which is going to
appeal to more users we lean towards more users. Note I said lean!


Dave
Post by Tom Lane
Post by Josh Berkus
Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.
Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.
So far I have not worried about that scenario too much, because Monty
has always treated the MySQL sources as his personal preserve; if he
hadn't written it or closely reviewed it, it didn't get in, and if it
didn't hew closely to his opinion of what's important, it didn't get in.
But I get the impression that he's loosened up of late. If MySQL stops
being limited by what one guy can do or review, their rate of progress
could improve dramatically.
In short: we could use an organized marketing effort. I really
feel the lack of Great Bridge these days; there isn't anyone with
comparable willingness to expend marketing talent and dollars on
promoting Postgres as such. Not sure what to do about it. We've
sort of dismissed Jean-Michel's comments (and those of others in
the past) with "sure, step right up and do the marketing" responses.
But the truth of the matter is that a few amateurs with no budget
won't make much of an impression. We really need some professionals
with actual dollars to spend, and I don't know where to find 'em.
regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
James Hubbard
2002-06-25 16:08:53 UTC
Permalink
I don't normally post to this list, but have a crazy suggestion that is a
little farfetched.

Suggestion:
Fix the portability problems so that there is a Windows native version of
PostgreSQL. Then offer the Open Office organization PostgreSQL as the
project's database. This would increase the user base my leaps and bounds.

The problem is that using and administrating PostgreSQL can be complex. Also,
some people may automatically assume that PostgreSQL is a low end database not
capable of doing more than being used as a backend for a free office app. Of
course we all know better.

Maybe a PostgreSQL-Lite would be a better idea. One that condenses the main
code down to something easy, that a desktop user could use, but maintain the
strength of the core code. I suppose that means creating another project.

Here are just a few links that I've come across recently:
How-to for using Open Office and unixODBC
http://www.unixodbc.org/doc/OOoMySQL.pdf

Others are considering MySQL.
http://dba.openoffice.org/proposals/MySQL_OOo.html

James Hubbard
Post by Dave Cramer
IMO One of the big reasons that MySQL is viewed as being better is it's
percieved simplicity. It has a large following because of this, and many
of them are not experienced database users, in fact just the opposite.
This large user base is perhaps the best marketing that an open source
project can hope for. So I think that if we want to attract more users
we should try to make postgres easier to use. The hard part is how to do
this without sacrificing the integrity of the project. I think for
starters when evaluating the next feature we want to work on we ask the
1) Does it make it easier to use for a non-dba ?
2) Does it facilitate making web-applications easier ( assuming that
this is the largest user base ) ?
3) I'm sure there are others, but at the moment I can't come up with
them.
Then if faced with a choice of implementing something which is going to
make postgres more technically complete or something which is going to
appeal to more users we lean towards more users. Note I said lean!
Dave
Post by Tom Lane
Post by Josh Berkus
Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.
Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.
So far I have not worried about that scenario too much, because Monty
has always treated the MySQL sources as his personal preserve; if he
hadn't written it or closely reviewed it, it didn't get in, and if it
didn't hew closely to his opinion of what's important, it didn't get in.
But I get the impression that he's loosened up of late. If MySQL stops
being limited by what one guy can do or review, their rate of progress
could improve dramatically.
In short: we could use an organized marketing effort. I really
feel the lack of Great Bridge these days; there isn't anyone with
comparable willingness to expend marketing talent and dollars on
promoting Postgres as such. Not sure what to do about it. We've
sort of dismissed Jean-Michel's comments (and those of others in
the past) with "sure, step right up and do the marketing" responses.
But the truth of the matter is that a few amateurs with no budget
won't make much of an impression. We really need some professionals
with actual dollars to spend, and I don't know where to find 'em.
regards, tom lane
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to ***@postgresql.org so that your
message can get through to the mailing list cleanly
Bruce Momjian
2002-06-25 16:23:36 UTC
Permalink
Post by James Hubbard
I don't normally post to this list, but have a crazy suggestion that is a
little farfetched.
Fix the portability problems so that there is a Windows native version of
PostgreSQL. Then offer the Open Office organization PostgreSQL as the
project's database. This would increase the user base my leaps and bounds.
OK, we just started heavy discussion on this and I believe a few people
are actively working on this. The timeframe is 3-6 months, though we
have the Cygwin solution right now.
Post by James Hubbard
The problem is that using and administrating PostgreSQL can be complex. Also,
some people may automatically assume that PostgreSQL is a low end database not
capable of doing more than being used as a backend for a free office app. Of
course we all know better.
OK, I have heard this before, but I would like to know specifically how
is PostgreSQL harder to administer than MySQL. Knowing that will help
us address the issue.
Post by James Hubbard
Maybe a PostgreSQL-Lite would be a better idea. One that condenses the main
code down to something easy, that a desktop user could use, but maintain the
strength of the core code. I suppose that means creating another project.
Perhaps a config utility that asked you questions and modified template1
and the config files. How about that?
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026



---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org
Josh Berkus
2002-06-25 15:42:13 UTC
Permalink
Tom,
Post by Tom Lane
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.
We also don't have a couple of other things that MySQL has: A
viciously divided community, a bobby-trapped licensing situation, and a
flagrant disredard for the SQL standard and cumulative wisdom of 25
years of database knowledge. (Nested tables! Sheesh!) These things
handicap the MySQL project quite effectively, and are not likely to be
straightened out in the next year.

BTW, PLEASE DO NOT QUOTE the above. It's ok for the hacker's list,
but I do not want to fuel the MySQL/Postgres "debate" anywhere more
public. This "debate" does not benefit either project.

Also, I am concerned about the focus on MySQL as our "only competitor".
Frankly, I have never regarded MySQL as our primary competitor; that
spot is reserved for Microsoft SQL Server. Especially with the death
of SQL Anywhere, Postgres and MS SQL are the two major databases in the
transaction/vertical application space for the budget-minded business
(although MS SQL is considerably less budget-minded than it was a year
ago).

When we've crushed MS SQL, then it's time to take on Oracle and DB2.

I think there's plenty of room in the RDBMS market for both MySQL and
PostgreSQL. If there's a marketing need, it's to educate DBA's on the
different strengths of the two databases. You think MySQL would
cooperate in this, or do they see themselves as competing head-on with
us?
Post by Tom Lane
In short: we could use an organized marketing effort. I really
feel the lack of Great Bridge these days; there isn't anyone with
comparable willingness to expend marketing talent and dollars on
promoting Postgres as such. Not sure what to do about it. We've
sort of dismissed Jean-Michel's comments (and those of others in
the past) with "sure, step right up and do the marketing" responses.
But the truth of the matter is that a few amateurs with no budget
won't make much of an impression. We really need some professionals
with actual dollars to spend, and I don't know where to find 'em.
I disagree pretty strongly, Tom. OpenOffice.org Marketing has no
cash, and is an all-volunteer effort. To quote journalist Amy Wohl
"[OpenOffice.org] have managed to put together a better buunch of
volunteer marketers than Sun is able to hire." Frankly, of the various
marketing techniques, only going to trade shows costs money; the rest
is all labor which can be done by volunteers and donors.

Of course, this requires somebody pretty inspired to organize it. I
already have my hands full with OpenOffice.org. Volunteers?

As Great Bridge should have taught you, corporate money for mmarketing
comes with expectations and deadlines attached. Landmark gave you one
shot at "making it", and then yanked the carpet when that didn't pan
out immediately. Other companies are going to be the same. One of
the greatest things about Postgres is that we have been able to outlast
the death of half a dozen companies that supported us, and replace them
with new.

And isn't Red Hat doing anything to promote us?

Finally, thanks to you guys, we are still advancing our project faster
than most commercial software. How many RDBMSs out there have DOMAIN
support? How many have advanced data types that really work? How
many support 5 procedural languages and subselects just abotu
everywhere?

-Josh Berkus




---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to ***@postgresql.org
Bruce Momjian
2002-06-25 16:33:07 UTC
Permalink
Post by Josh Berkus
Also, I am concerned about the focus on MySQL as our "only competitor".
Frankly, I have never regarded MySQL as our primary competitor; that
spot is reserved for Microsoft SQL Server. Especially with the death
of SQL Anywhere, Postgres and MS SQL are the two major databases in the
transaction/vertical application space for the budget-minded business
(although MS SQL is considerably less budget-minded than it was a year
ago).
When we've crushed MS SQL, then it's time to take on Oracle and DB2.
I think Oracle is our main competitor. We seem to get more people
porting from Oracle than any other database, and our feature set matches
there's most closely.
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026



---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to ***@postgresql.org)
Bruce Momjian
2002-06-25 16:29:25 UTC
Permalink
Post by Tom Lane
Post by Josh Berkus
Frankly, my feeling is, as a "geek-to-geek" product, PostgreSQL is already
adequately marketed through our huge network of DBA users and code
contributors.
Well, mumble ... it seems to me that we are definitely suffering from
a "buzz gap" (cf missile gap, Dr Strangelove, etc) compared to MySQL.
That doesn't bother me in itself, but the long-term implications are
scary. If MySQL manages to attract a larger development community as
a consequence of more usage or better marketing, then eventually they
will be ahead of us on features and every other measure that counts.
Once we're number two with no prayer of catching up, how long will our
project remain viable? So, no matter how silly you might think
"MySQL is better" is today, you've got to consider the prospect that
it will become a self-fulfilling prophecy.
OK, I want to know, does anyone see MySQL gaining in market share in
comparison to PostgreSQL, or is MySQL gaining against other databases?
Is MySQL gaining sites faster than we are gaining sites?

Every indication I can see is that PostgreSQL is gaining on MySQL.

The Linux/FreeBSD comparison is potent. Does PostgreSQL remain a niche
player? Does *BSD remain a niche player?
--
Bruce Momjian | http://candle.pha.pa.us
***@candle.pha.pa.us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026



---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/users-lounge/docs/faq.html
Loading...