Discussion:
[Bitcoin-development] Change to multiple executables?
John Smith
2011-08-10 09:36:05 UTC
Permalink
All,

In the current mainline client everything is lugged into one executable
(with an optional daemon-only one). I think this is a bad idea for various
reasons, and would propose something like:

- bitcoind: bitcoin daemon
- bitcoin(-qt): bitcoin GUI executable
- bitcoincl: bitcoin RPC command line

By default, all three would be built. In non-GUI mode, only bitcoind and
bitcoincl are built (the names are obviously open for discussion).

Advantages:

- It is more clear to the user. One command, one function.
- It simplifies the main functions.
- The UI would no longer double-function as daemon. It is a waste of
memory to link the UI libs if you only want to run a background process.
- The UI and daemon would no longer double-function as RPC call. Why load
the code for UI and network if you just want to send a single command over
JSONRPC? This would also prevent accidentally launching the daemon/UI
locally if you just want to send a command and forget to give an argument.

JS
Matt Corallo
2011-08-10 10:14:49 UTC
Permalink
Post by John Smith
All,
In the current mainline client everything is lugged into one
executable (with an optional daemon-only one). I think this is a bad
* bitcoind: bitcoin daemon
* bitcoin(-qt): bitcoin GUI executable
* bitcoincl: bitcoin RPC command line
By default, all three would be built. In non-GUI mode, only bitcoind
and bitcoincl are built (the names are obviously open for
discussion).
* It is more clear to the user. One command, one function.
I would argue its less clear for the user. Instead of opening either
bitcoind or bitcoin to get RPC or GUI, now you have to open bitcoin and
bitcoind or bitcoincl and bitcoind. Now, obviously bitcoin and
bitcoincl can open bitcoind for you, but I think adding more executables
complicates things for little clear advantage.
Post by John Smith
* It simplifies the main functions.
* The UI would no longer double-function as daemon. It is a
waste of memory to link the UI libs if you only want to run a
background process.
As you pointed out, we have bitcoind for just this reason.
Post by John Smith
* The UI and daemon would no longer double-function as RPC call.
Why load the code for UI and network if you just want to send
a single command over JSONRPC? This would also prevent
accidentally launching the daemon/UI locally if you just want
to send a command and forget to give an argument.
Making RPC optional for GUI users would be an interesting addition.
Post by John Smith
JS
All this said, I totally agree with the more clear split of the source
into separate library-ish components (I'm working on part of that now).
However, I don't like the idea of splitting into more executables.

If you are suggesting this so that bitcoin-qt can be distributed being
built off of bitcoind, I would say go ahead and pull-request bitcoin-qt.
I'm of the opinion that it should be merged whether we have autotools or
not (we already have 5 makefiles, whats a few more options in those?)
and jgarzik seemed to indicate that he would agree (Gavin?, sipa?
tcatm?).

Matt
John Smith
2011-08-10 10:26:30 UTC
Permalink
Post by Matt Corallo
I would argue its less clear for the user. Instead of opening either
bitcoind or bitcoin to get RPC or GUI, now you have to open bitcoin and
bitcoind or bitcoincl and bitcoind. Now, obviously bitcoin and
bitcoincl can open bitcoind for you, but I think adding more executables
complicates things for little clear advantage.
UI would obviously still have RPC functionality with -server. I don't mean
dropping that. The UI links both the UI and the network code (for now, until
this is separated out and the preferred UI<->core communication method is
through RPC).

I just mean that the *headless* daemon is separate from the UI executable,
which is the case for any other sane client/server-based program in
existence, from bittorrent nodes to game servers.

It would also make it possible to build the command line RPC client
(bitcoin-cl) *without* building the server or UI. Useful if you want to
remotely control a Bitcoin daemon but not want to build it locally.

JS
Pieter Wuille
2011-08-10 10:43:36 UTC
Permalink
Post by Matt Corallo
Post by John Smith
All,
In the current mainline client everything is lugged into one
executable (with an optional daemon-only one). I think this is a bad
* bitcoind: bitcoin daemon
* bitcoin(-qt): bitcoin GUI executable
* bitcoincl: bitcoin RPC command line
By default, all three would be built. In non-GUI mode, only bitcoind
and bitcoincl are built (the names are obviously open for
discussion).
All this said, I totally agree with the more clear split of the source
into separate library-ish components (I'm working on part of that now).
However, I don't like the idea of splitting into more executables.
I do agree about splitting off bitcoincl - it's kinda confusing now how
the client behaves as a rpc daemon or UI when no RPC command-line
parameters are present, and as a command-line client otherwise.

I am less sure UI and RPC should be split (though being able to select
both independently from eachother at compile time would be nice). I
often run the UI and switch to RPC calls to inspect some details.
Not sure how common this usage pattern is, though.
Post by Matt Corallo
If you are suggesting this so that bitcoin-qt can be distributed being
built off of bitcoind, I would say go ahead and pull-request bitcoin-qt.
I'm of the opinion that it should be merged whether we have autotools or
not (we already have 5 makefiles, whats a few more options in those?)
and jgarzik seemed to indicate that he would agree (Gavin?, sipa?
tcatm?).
The problem is that bitcoin-qt is built using qmake, and the rest using
makefiles... so it's more than just adding an additional makefile.

That said, it seems bitcoin-qt is mature enough to replace wxbitcoin
to me, and would definitely like to see it in mainline. How streamlined
is the process of building bitcoin-qt on windows and osx? Maybe we can
switch everything to qmake (for now, as long as no maintained autotools
is present)?
--
Pieter
John Smith
2011-08-10 13:18:38 UTC
Permalink
Post by Pieter Wuille
I do agree about splitting off bitcoincl - it's kinda confusing now how
the client behaves as a rpc daemon or UI when no RPC command-line
parameters are present, and as a command-line client otherwise.
I am less sure UI and RPC should be split (though being able to select
both independently from eachother at compile time would be nice). I
often run the UI and switch to RPC calls to inspect some details.
Not sure how common this usage pattern is, though.
No no no I never stated that the UI should no longer support RPC. If you
want the UI, with RPC, you can still run the UI executable with -server.
There are many usecases in which you want to access the UI bitcoin client
using RPC...

I only meant that it would also build the *headless* daemon by default, as
separate "bitcoind" executable. So you cannot run the UI exectuable as
headless server anymore. The -daemon option would go away. It would make the
setup a lot easier: The UI can connect to X and display a splash screen
immediately without first looking at the command line arguments, whereas the
headless daemon can ignore all that stuff and get straight to work.
Post by Pieter Wuille
Post by Matt Corallo
If you are suggesting this so that bitcoin-qt can be distributed being
built off of bitcoind, I would say go ahead and pull-request bitcoin-qt.
I'm of the opinion that it should be merged whether we have autotools or
not (we already have 5 makefiles, whats a few more options in those?)
and jgarzik seemed to indicate that he would agree (Gavin?, sipa?
tcatm?).
That said, it seems bitcoin-qt is mature enough to replace wxbitcoin
to me, and would definitely like to see it in mainline. How streamlined
is the process of building bitcoin-qt on windows and osx? Maybe we can
switch everything to qmake (for now, as long as no maintained autotools
is present)?
It's easy on windows: just install Qt Creator (comes with the Qt SDK),
install the extra dependencies (build instructions are in README.rst), and
hit build.

On MacOSX I'm not sure. I think it's similar, as a few people built it for
MacOSX and contributed settings for the .pro file...

It can build the GUI fine for every platform, however it can only build the
GUI, not the headless daemon or the command line client. You'd still need
old fashioned makefiles for those.

Cmake or autotools would be better, especially as those are intelligent
enough to auto-detect the name of libraries such as boost, and detect
presence of optional dependencies (miniupnp).

JS
Gavin Andresen
2011-08-10 16:49:21 UTC
Permalink
RE: splitting off the "send commands to a running bitcoin" :

I'm mildly against it. It would be less confusing for newbies, at the
cost of forcing everybody who has already written backup scripts or
other interact-with-running-bitcoin tools to tweak their code. The
coding will be easy, but do you really want to spend the time to
answer all the "I installed Bitcoin X.Y and now my backup script
doesn't work" questions and modify the wiki pages and ...

I'd rather that time be spent working on any remaining build issues so
we can switch to bitcoin-qt. I don't care if it is autotools or qmake
or QT creator, I just care that it works on Windows and Linux under
gitian and has clear instructions so I can build it on my Mac.
--
--
Gavin Andresen
John Smith
2011-08-10 17:45:42 UTC
Permalink
Post by Gavin Andresen
I'm mildly against it. It would be less confusing for newbies, at the
cost of forcing everybody who has already written backup scripts or
other interact-with-running-bitcoin tools to tweak their code. The
coding will be easy, but do you really want to spend the time to
answer all the "I installed Bitcoin X.Y and now my backup script
doesn't work" questions and modify the wiki pages and ...
As the project is still in "experimental" phase I suppose people can expect
changes like this? And the change is pretty much trivial, and it makes sense
for a future direction (UI<->Wallet in separate processes for security
concerns).

To be honest I feel a bit like every change that I (and I've also heard this
from others) propose is shot down, no matter how well formulated. This is
actively discouraging developers from joining this project.

Of course it makes sense to be a careful, but the project is not on life
support is it? Satoshi did a great job making the program, but his work was
not perfect, and it makes sense to look ahead a bit.

I think it would be better to switch to two branches, like most other open
source projects I've worked with:

0.3.x -> small, compatible changes, bugfixes, like now
0.4.x -> trunk, more impactful changes, refactorings, eventual major release

Both will obviously be fully compatible on the P2P-level.
Post by Gavin Andresen
I'd rather that time be spent working on any remaining build issues so
we can switch to bitcoin-qt. I don't care if it is autotools or qmake
or QT creator, I just care that it works on Windows and Linux under
gitian and has clear instructions so I can build it on my Mac.
I could do the Gitian stuff but not the Mac instructions, as I don't have a
Mac...

JS
Gavin Andresen
2011-08-10 18:41:51 UTC
Permalink
Post by John Smith
To be honest I feel a bit like every change that I (and I've also heard this
from others) propose is shot down, no matter how well formulated.  This is
actively discouraging developers from joining this project.
Well, to be honest I don't think more developers adding new features
are needed right now-- I think the project's critical needs are more
people testing and helping to fix bugs and scalability issues.

In this particular case, I said I was mildly against it-- if you want
me to switch to supporting it, then reassure me you're willing to do
ALL the work to make it happen. Send me a list of wiki pages you'll
edit to document the change and tell me that you'll be around to help
people rewrite their backup scripts.
Post by John Smith
I think it would be better to switch to two branches, like most other open
source projects I've worked with.
I don't see how dividing efforts between a 'bug fix' and 'development'
branch will help fix the project's critical needs. If we did, I think
there would be less pressure to help with the boring bug-fixing and
testing of the bug-fix branch, which I think would be bad.
--
--
Gavin Andresen
Andy Parkins
2011-08-10 19:32:00 UTC
Permalink
Post by Gavin Andresen
Post by John Smith
To be honest I feel a bit like every change that I (and I've also heard
this from others) propose is shot down, no matter how well
formulated. This is actively discouraging developers from joining
this project.
Well, to be honest I don't think more developers adding new features
are needed right now-- I think the project's critical needs are more
people testing and helping to fix bugs and scalability issues.
(Rant follows; stop reading now)

That paragraph reveals a gross misunderstanding of how open source works.

People get itches and they want to scratch them. They aren't paid, so they
don't necessarilly want to turn up and be told which part they _should_ be
working on. The choice is not "bug fix that Gavin wants" or "new feature
that New Developer wants", it is "New Feature" or nothing.

Of course, nothing forces existing developers to accept these new features;
but the incredibly negative attitude on display when any new feature is
suggested is not the way to grow a community. The correct way is a
mentoring attitude -- offering opinions on how a new developer can get their
idea in rather than telling them why it will never happen.
Post by Gavin Andresen
I don't see how dividing efforts between a 'bug fix' and 'development'
branch will help fix the project's critical needs. If we did, I think
Again: that's not your call. People will work on what interests them. I've
suggested a couple of features both here and on the forum and been shot down
in varying degrees every time. Fine, but don't expect that I'm thinking
"well I'll become an unpaid bug fixing grunt instead".

I don't expect to be appointed head developer because I suggest an idea. I
don't even expect anyone else to implement my idea for me. But why should I
spend time on my own idea when the feedback is "no", "no", "we've already
thought of that", "not needed", "go away", "why not fix some bugs instead"?

I'm amazed that John Smith is as polite and persistent as he is looking at
the amount of effort he's put in putting a pretty face on the train crash
that existed before hand and seems to get no benefit of the doubt for his
work.
Post by Gavin Andresen
there would be less pressure to help with the boring bug-fixing and
testing of the bug-fix branch, which I think would be bad.
That pressure might be relieved if the community were able to grow a bit,
and people felt they had a personal investment. That means loosening the
reigns a bit; and perhaps a development branch would be the way to do that
while not compromising code quality.

I suggest a look at the way git itself is developed; it has the following
branches:

- master: the latest release + newly accepted features
- maint: the latest release + bug fixes only
- next: new features planned for inclusion, actively being worked on.
Often created by merging "topic" branches from individual developers
working on their current itch
- pu: crazy stuff; not planned for inclusion, but acting as a staging
area for people to show what they're working on



Andy
--
Dr Andy Parkins
***@gmail.com
Jeff Garzik
2011-08-10 19:57:29 UTC
Permalink
People get itches and they want to scratch them.  They aren't paid, so they
don't necessarilly want to turn up and be told which part they _should_ be
working on.  The choice is not "bug fix that Gavin wants" or "new feature
that New Developer wants", it is "New Feature" or nothing.
This is true -- though there is value to having a list of "things we
think people should focus on" for the motivated, and for new people
interested in tackling a project, but not sure what project to tackle.
Post by Gavin Andresen
there would be less pressure to help with the boring bug-fixing and
testing of the bug-fix branch, which I think would be bad.
That pressure might be relieved if the community were able to grow a bit,
and people felt they had a personal investment.  That means loosening the
reigns a bit; and perhaps a development branch would be the way to do that
while not compromising code quality.
A centrally managed development branch on bitcoin/bitcoin.git is not
the way to do it, however. See the description of linux-next, in my
previous email, for a more distributed method which can easily be
layered on top of the existing bitcoin dev structure by any motivated
volunteer(s).

Think distributed. :) The community does not need Linus's help
(linux-next) or Gavin's help (bitcoin-next) to do this. linux-next
became so widely used and useful that Linus requires almost all
changes to be first staged in linux-next.
--
Jeff Garzik
exMULTI, Inc.
***@exmulti.com
Andy Parkins
2011-08-10 21:13:09 UTC
Permalink
Post by Jeff Garzik
Post by Andy Parkins
People get itches and they want to scratch them. They aren't paid, so
they don't necessarilly want to turn up and be told which part they
_should_ be working on. The choice is not "bug fix that Gavin wants"
or "new feature that New Developer wants", it is "New Feature" or
nothing.
This is true -- though there is value to having a list of "things we
think people should focus on" for the motivated, and for new people
interested in tackling a project, but not sure what project to tackle.
My objection is not that such a list exists, it is that potential new
developers are, essentially, shouted down unless they are working on that
list. I cannot imagine that many new developers arrive under those
circumstances.
Post by Jeff Garzik
A centrally managed development branch on bitcoin/bitcoin.git is not
the way to do it, however. See the description of linux-next, in my
previous email, for a more distributed method which can easily be
layered on top of the existing bitcoin dev structure by any motivated
volunteer(s).
I don't think I said anything about it being centrally managed. git lets us
store these branches anywhere of course. The fact is that such a branch
exists somewhere.
Post by Jeff Garzik
Think distributed. :) The community does not need Linus's help
(linux-next) or Gavin's help (bitcoin-next) to do this. linux-next
I didn't say that it required anybody's help; but it does require a bit of
willingess on the part of the master-branch-owning developers to import from
that branch.
Post by Jeff Garzik
became so widely used and useful that Linus requires almost all
changes to be first staged in linux-next.
They key thing with linux-next is that work done on it _does_ make it into
the kernel. Tell me -- how many feature branches for bitcoin are just
sitting as a pull request on github, and are now months old and abandoned
out of disgust by their original authors? Here's another question: why is
it that so many projects have "specially compiled" versions of bitcoin?
Rhetorical question... it's because the official client doesn't do what they
need, and won't accept their patches to add it (even optionally).

I've only been watching this list for a few weeks (since the forum turned
into an echo chamber); but I'm completely depressed by the agressive
rejections of every new idea anyone raises.

Don't believe me? Here's a list of ideas I've had "no, no, no"d so far; not
one of which would have any financial implication at all. Only some of
which would break backward compatibility.

- Extra bits in the service field of the version message to allow nodes
to indicate if they are mining; if they are willing to be seed nodes;
if they relay transactions; if they want relayed transactions.
- getblocks in reverse chronological order so clients can start up quicker
while downloading the blocks in the backround. Ironically I was told
"patches welcome" by someone who didn't reject this one instantly.
- Remove verack, as it's completely unnecessary.
- Query miners for pending transactions
- Application version separate from client version
- A way of requesting block bodies without headers (saving a lot of traffic
for a thin client upgrading)
- Double SHA-256 for a packet checksum? Seriously?
- Sequence number as part of TxIn instead of part of the whole transaction
- Script parameters should be stored outside the script, and reference by
the script. All that ridiculous filtering of the scripts in OP_CHECKSIG
would then go away.
- MSG_DOUBLESPEND... nope
- getblocks to accept MSG_TX and do something sensible

Every single one of those has been shot down by one or more of the main
developers. I'm not a genius, and not arrogant enough to assume that
everything I say is right, but _nothing_? Really? There is no problem that
one of the above addresses?

Given that, what do I do? Hang around and get battered some more, or go
away to my own little corner and work on my own implementation?

You can imagine then that when I read moans about there not being enough new
developers fixing bugs, that I am unsurprised and unsympathetic. I like
bitcoin enough to hover on this list; and offer a view of your world from a
potential developer who was chased away.



Andy
--
Dr Andy Parkins
***@gmail.com
Jeff Garzik
2011-08-10 21:35:01 UTC
Permalink
Don't believe me?  Here's a list of ideas I've had "no, no, no"d so far; not
one of which would have any financial implication at all.  Only some of
which would break backward compatibility.
Breaking backwards compatibility means breaking people's access to
their own money.

If you remove an "unnecessary" step that existing nodes expect, then
the cost of disrupting monetary access seems higher than the value of
that breaking change.
 - Extra bits in the service field of the version message to allow nodes
  to indicate if they are mining; if they are willing to be seed nodes;
  if they relay transactions; if they want relayed transactions.
My own 'supernode' proposal also includes using the nServices bits.
There's nothing fundamentally incompatible or wrong about that.
 - Remove verack, as it's completely unnecessary.
Compatibility issues?
 - Query miners for pending transactions
I could see value in querying a bitcoind node over JSON-RPC for
pending transactions... and by extension, supporting that as an RPC on
various miners' pool servers. Having a local dump of pending TX's
would be useful.

As an optional bitcoin P2P protocol command, available to anyone,
seems to negatively impact privacy.
 - Application version separate from client version
Consensus has already approved this one, AFAIK.
 - A way of requesting block bodies without headers (saving a lot of traffic
  for a thin client upgrading)
Do you mean headers without bodies? Gavin wants to work on
headers-only, from what I've read, but others are welcome to
contribute patches.
 - Double SHA-256 for a packet checksum?  Seriously?
Compatibility issues?
 - Sequence number as part of TxIn instead of part of the whole transaction
Compatibility issues?
 - Script parameters should be stored outside the script, and reference by
  the script.  All that ridiculous filtering of the scripts in OP_CHECKSIG
  would then go away.
Compatibility issues?
 - MSG_DOUBLESPEND... nope
Does consensus want this?
 - getblocks to accept MSG_TX and do something sensible
Link to elaboration of use case and need?
You can imagine then that when I read moans about there not being enough new
developers fixing bugs, that I am unsurprised and unsympathetic.  I like
bitcoin enough to hover on this list; and offer a view of your world from a
potential developer who was chased away.
Well, one unfortunate current aspect of bitcoin is... there seem to
be problems aplenty right now :)

However demotivating it may be, keeping the current system running
must take priority over new features.

I also heartily encourage others to do something I always want to do,
but for lack of time: work on the design for bitcoin v2 ("theme: any
breaking change is acceptable, it is a new block chain") There you
may improve the protocol, get rid of the patent-cloudy ECDSA, use
google's protocol buffers for encoding, make the proof-of-work
algorithm memory-intensive, and other excellent, thoughtful
breaking-change suggestions that have been made.

Securing the integrity of money means that a lot of implementation
decisions have been cemented into stone, however much we may
personally dislike them. Backwards compatibility is paramount.
--
Jeff Garzik
exMULTI, Inc.
***@exmulti.com
Andy Parkins
2011-08-10 22:38:21 UTC
Permalink
Post by Jeff Garzik
Post by Andy Parkins
Don't believe me? Here's a list of ideas I've had "no, no, no"d so
far; not one of which would have any financial implication at all.
Only some of which would break backward compatibility.
Breaking backwards compatibility means breaking people's access to
their own money.
I wasn't actually giving a full explanation of how these things could be
done, I was providing a list of "negatively received ideas"; imagine my
surprise that they have been negatively received by you.

However... The version number field combined with the massive complexity of:

if( blockNumber > 500000 )
new_process();
else
old_process();

Would sort all of your "compatibility" objections out, and would give nodes
time to upgrade.
Post by Jeff Garzik
If you remove an "unnecessary" step that existing nodes expect, then
the cost of disrupting monetary access seems higher than the value of
that breaking change.
If only there were some way of sending different things to different nodes,
based on some sort of version number field.
Post by Jeff Garzik
Post by Andy Parkins
- Remove verack, as it's completely unnecessary.
Compatibility issues?
if( Version < VERSION_INTRODUCED )
sendVerack();

My point is that you are a clever guy; you are perfectly capable of coming
up with these answers, but you don't want to. Nor does any other bitcoin
developer. The protocol is perfect and there is no way of changing it.
Post by Jeff Garzik
Post by Andy Parkins
- Query miners for pending transactions
I could see value in querying a bitcoind node over JSON-RPC for
pending transactions... and by extension, supporting that as an RPC on
various miners' pool servers. Having a local dump of pending TX's
would be useful.
As an optional bitcoin P2P protocol command, available to anyone,
seems to negatively impact privacy.
Eh? The transaction list is available on bitcoincharts. If my node had
been connected it would have received that list anyway when each one was
broadcast. What possible privacy loss could there be by making it possible
to request it be repeated?

Again though: the detail isn't the point. It's another half-hearted
objection.
Post by Jeff Garzik
Post by Andy Parkins
- A way of requesting block bodies without headers (saving a lot of
traffic for a thin client upgrading)
Do you mean headers without bodies? Gavin wants to work on
headers-only, from what I've read, but others are welcome to
contribute patches.
No; I mean being able to ask for just the block without the header. The
reason being that a thin client might request blocks on demand... it's
already got the header and doesn't need it again.

The response: "it's only 80 bytes, blah, blah". 80*150000*N is a non-
trivial amount of traffic.
Post by Jeff Garzik
Post by Andy Parkins
- Double SHA-256 for a packet checksum? Seriously?
Compatibility issues?
Only for the version message. But it would be trivial to do both types of
checksum on the version message, and if either is true to accept the version
message. After which the version is known and a much simpler checksum could
be used for subsequent messages. Eventually the network would be upgraded
enough that the old way can be dropped.

Besides... hasn't TCP already got checksumming? Let's just stop checking
the checksum. Or better still, stop calculating it and sending it. Double
SHA-256 on every single message on every single node to create four checksum
bytes is an enormous waste of CPU.
Post by Jeff Garzik
Post by Andy Parkins
- Sequence number as part of TxIn instead of part of the whole transaction
Compatibility issues?
If only there were a version field in the transaction and block structures.

Again; casual rejection.
Post by Jeff Garzik
Post by Andy Parkins
- Script parameters should be stored outside the script, and reference
by the script. All that ridiculous filtering of the scripts in
OP_CHECKSIG would then go away.
Compatibility issues?
See above.
Post by Jeff Garzik
Post by Andy Parkins
- MSG_DOUBLESPEND... nope
Does consensus want this?
No, "consensus" doesn't. I was simply listing all the ideas that got
rejected out of hand. The reason "consensus" doesn't think this one is
necessary is because "we can already detect double spends by being widely
connected"; ignoring the fact that a light or intermittently connected
client would not be widely connected. But that's okay because "eventually
payment processors will appear". Yep, my idea for fixing bitcoin is stupid
because eventually someone else will mitigate it.
Post by Jeff Garzik
Post by Andy Parkins
- getblocks to accept MSG_TX and do something sensible
Link to elaboration of use case and need?
It was a few weeks ago; and it was an email from me about getblocks
enhancements. It was patronisingly laughed off as being something that all
you newbie "alternative client" writers go through.

The use case is an on-demand thin client that wants to find the block that
contains a particular transaction ID without downloading and indexing every
single block in the chain. Additionally, _I_ plan to separate the block
chain and wallet executables, so much so that the wallet executable doesn't
necessarily need a local blockchain node and relies on a partially trusted
remote -- it still wants to be able to do spot checks on that remote, and
confirm whatever it's told. I would like to be able to do that using only
commands that are in the official protocol; but I'm rapidly coming to accept
that nothing I ask for will ever go in because there is no "use case".
Post by Jeff Garzik
Well, one unfortunate current aspect of bitcoin is... there seem to
be problems aplenty right now :)
As with every project.

However, the protocol is being treated as if it is some kind of holy scroll,
and must not be touched. Bitcoin's ideas are revolutionary, its
implementation is not. If we started again today, it would be done
differently. Shouldn't we be trying to move the current protocol toward
_that_ "done differently" as much as possible while bitcoin is still
relatively small? Rhetorical again... I know the answer, it's "no".

What exactly do the developers mean when they keep talking about bitcoin as
"experimental"? It seems to me they mean "incredibly conservative, with no
changes for the rest of time".
Post by Jeff Garzik
However demotivating it may be, keeping the current system running
must take priority over new features.
Nothing I've suggested was to "stop the current system". I'm not even
asking for developers to prioritise my ideas. I would just like mine, or
anyone's ideas to not be instantly rejected out of hand. I mean for
goodness sake, even "splitting into multiple executables" has been stomped
on in this very thread. If something as trivial as that is "impossible"
what chance is there that I would ever get "Change the 64-bit timestamp
field to be microseconds since the epoch instead of seconds" in?
Post by Jeff Garzik
I also heartily encourage others to do something I always want to do,
but for lack of time: work on the design for bitcoin v2 ("theme: any
breaking change is acceptable, it is a new block chain") There you
may improve the protocol, get rid of the patent-cloudy ECDSA, use
google's protocol buffers for encoding, make the proof-of-work
algorithm memory-intensive, and other excellent, thoughtful
breaking-change suggestions that have been made.
There is a popular idea that some other cryptocurrency will come along and
displace bitcoin. It's not going to happen. Networking effects mean that
there is no reason for people to change. I can just see the queue around
the block now for bitcoin.2; identical in function to bitcoin except it
"doesn't use ECDSA and the it uses protocol buffers on the wire, and uses
more memory". Wow; there's a set of unique selling points. I'll get signs
made.

Let's be practical: technical-only improvements _have_ to be to bitcoin.1.
Bitcoin's financial features are already complete or in progress; and it is
financial features that would make people migrate to a competitor. Nobody
is going to move to bitcoin.v2 because the source code has better comments.
Post by Jeff Garzik
Securing the integrity of money means that a lot of implementation
decisions have been cemented into stone, however much we may
personally dislike them. Backwards compatibility is paramount.
I disagree about how set in stone these things are; but yeah; I've accepted
that I'm on a loser. My list was to demonstrate how negative the community
is; and you have confirmed that for me admirably. Bear that in mind the
next time you're discussing the lack of manpower for bug fixes.



Andy
--
Dr Andy Parkins
***@gmail.com
Jeff Garzik
2011-08-11 03:20:25 UTC
Permalink
Post by Andy Parkins
Post by Jeff Garzik
Don't believe me?  Here's a list of ideas I've had "no, no, no"d so
far; not one of which would have any financial implication at all.
 Only some of which would break backward compatibility.
Breaking backwards compatibility means breaking people's access to
their own money.
I wasn't actually giving a full explanation of how these things could be
done, I was providing a list of "negatively received ideas"; imagine my
surprise that they have been negatively received by you.
 if( blockNumber > 500000 )
  new_process();
 else
  old_process();
Would sort all of your "compatibility" objections out, and would give nodes
time to upgrade.
The above has been discussed on the forums. The community seems to
consider that option one of last resort, because the consequences of
-not- upgrading immediately become "I cannot access my money." The
community also seems rather hard-wired against breaking changes like
that, because it implies that we lowly dev peons are daring to mess
with the Blessed Satoshi Design that has received extensive study, and
100% communal agreement.

If the community changes its mind, and there are strong calls to make
a breaking change, we can certainly do that. Bitcoin is not only open
source but very much democratic -- people vote by choosing which
client software to use.
Post by Andy Parkins
However, the protocol is being treated as if it is some kind of holy scroll,
and must not be touched. Bitcoin's ideas are revolutionary, its
implementation is not. If we started again today, it would be done
differently. Shouldn't we be trying to move the current protocol toward
_that_ "done differently" as much as possible while bitcoin is still
relatively small? Rhetorical again... I know the answer, it's "no".
Historically speaking, the protocol has had minor tweaks, if you check
the patch history. Adding new protocol commands is pretty easy, for
example. Removing commands tends to be high cost for low benefit
("protocol removes a harmless redundancy"), if you're messing with the
initial negotiation sequence. IMO verack is not redundant, anyway.

But the answer is "what do the users want" not "no" At the end of the
day we're here to adequately reflect the needs of our userbase at all.
And so far, the userbase seems highly conservative when it comes to
incompatible changes. Correct me if I'm wrong...
Post by Andy Parkins
I disagree about how set in stone these things are; but yeah; I've accepted
that I'm on a loser.  My list was to demonstrate how negative the community
is; and you have confirmed that for me admirably.  Bear that in mind the
next time you're discussing the lack of manpower for bug fixes.
It's negative to weight costs vs. benefits? That is what I expect any
good engineer to do.

In any case, our -users- are not clamoring for breaking changes of the
type you describe above, as far as I can see. Just the opposite. So
if you want to deploy a breaking change, the burden is on you to
convince the bitcoin users and miners that it's a good idea.

If the bitcoin dev team is not accurately reflecting the desire of its
users, then that should be corrected, and we want to hear feedback.
--
Jeff Garzik
exMULTI, Inc.
***@exmulti.com
Andy Parkins
2011-08-11 05:47:34 UTC
Permalink
Post by Jeff Garzik
Post by Andy Parkins
if( blockNumber > 500000 )
new_process();
else
old_process();
Would sort all of your "compatibility" objections out, and would give
nodes time to upgrade.
The above has been discussed on the forums. The community seems to
consider that option one of last resort, because the consequences of
-not- upgrading immediately become "I cannot access my money." The
Did you even read what I wrote? "if( blockNumber > 5000000 )" is about as
far from immediate as you can get. I'm not an idiot; I understand we can't
lock people out of their money simply because of a software upgrade. It's
not unreasonable to expect people will have upgraded by block 500000 though
(or whatever number the community decided upon).

Again you're missing my point... you are still shooting ideas down.
Post by Jeff Garzik
community also seems rather hard-wired against breaking changes like
that, because it implies that we lowly dev peons are daring to mess
with the Blessed Satoshi Design that has received extensive study, and
100% communal agreement.
Well the community had better unhardwire itself or its going to end up with
five developers and no more.
Post by Jeff Garzik
If the community changes its mind, and there are strong calls to make
a breaking change, we can certainly do that. Bitcoin is not only open
source but very much democratic -- people vote by choosing which
client software to use.
Voting with ones feet should be a last resort. Wouldn't it be better not to
end up with incompatible clients out there?
Post by Jeff Garzik
Historically speaking, the protocol has had minor tweaks, if you check
the patch history. Adding new protocol commands is pretty easy, for
example. Removing commands tends to be high cost for low benefit
("protocol removes a harmless redundancy"), if you're messing with the
initial negotiation sequence. IMO verack is not redundant, anyway.
Client: I speak version 10
Server: hmmm, I don't speak version 10, I only speak version 5
Client: I am willing to lower to version 5 so I shall continue

or

Client: I speak version 10
Server: hmmm, I don't speak version 10, I only speak version 5
Client: I am unwilling to lower to version 5 so I shall hang up

or

Client: I speak version 5
Server: hmmm, I speak version 10, but I am willing to speak version 5

or

Client: I speak version 5
Server: hmmm, I speak version 10, and I am unwilling to speak version 5
so I shall hang up

'verack' is redundant. It sends no information and merely says that the
other end is willing to continue. Willing to continue is easily determined
when the remote continues. Handling 'verack' is an annoyance, and adds
nothing.
Post by Jeff Garzik
But the answer is "what do the users want" not "no" At the end of the
day we're here to adequately reflect the needs of our userbase at all.
And so far, the userbase seems highly conservative when it comes to
incompatible changes. Correct me if I'm wrong...
Please point me at a single incompatible change that has been rejected by
the userbase.

Further: I'm not suggesting incompatible changes alone; that would be
insane. I'm suggesting upgrade paths that delay incompatible changes until
the change has propagated.
Post by Jeff Garzik
It's negative to weight costs vs. benefits? That is what I expect any
good engineer to do.
I don't think that's what's happening. Not once have I seen the "benefits"
side of that equation. What I have seen is plenty of "I can't see a use
case for that"; when the key word in that sentence is "I".
Post by Jeff Garzik
In any case, our -users- are not clamoring for breaking changes of the
type you describe above, as far as I can see. Just the opposite. So
if you want to deploy a breaking change, the burden is on you to
convince the bitcoin users and miners that it's a good idea.
The users aren't typically going to be familiar enough with the internals of
bitcoin to care about many of the changes I suggested. I have repeatedly
said I don't want to break anything, I want to transition in an orderly
fashion (and the majority of my suggestions were backward compatible). But
of course, I don't actually want to do anything with bitcoind itself, it's
been made repeatedly clear to me that anything I might ask for is not going
to happen -- and of course what I was pointing out, _not_ asking for, was
that you can't expect to get new developers on board if they aren't going to
be allowed to scratch their itches.
Post by Jeff Garzik
If the bitcoin dev team is not accurately reflecting the desire of its
users, then that should be corrected, and we want to hear feedback.
You've just had some. The response was "you're wrong".



Andy
--
Dr Andy Parkins
***@gmail.com
Joel Joonatan Kaartinen
2011-08-11 11:45:30 UTC
Permalink
Post by Andy Parkins
Again you're missing my point... you are still shooting ideas down.
And you're only shooting his actions down without indicating clearly
what you think ought to be done instead. What do you want him to say
instead?
Post by Andy Parkins
Post by Jeff Garzik
community also seems rather hard-wired against breaking changes like
that, because it implies that we lowly dev peons are daring to mess
with the Blessed Satoshi Design that has received extensive study, and
100% communal agreement.
Well the community had better unhardwire itself or its going to end up with
five developers and no more.
No way that will happen. A fork is going to happen sooner rather than
later if this continues. It'd be great if it could be done within this
project and named bitcoin-dev or bitcoin-next or similar.

If this is not done, I wouldn't be surprised with the network splitting
into 2 camps with different protocols but still working on the same
blockchain.
Post by Andy Parkins
Post by Jeff Garzik
If the community changes its mind, and there are strong calls to make
a breaking change, we can certainly do that. Bitcoin is not only open
source but very much democratic -- people vote by choosing which
client software to use.
Voting with ones feet should be a last resort. Wouldn't it be better not to
end up with incompatible clients out there?
There's no way to get the majority to vote with their feet to move to an
incompatible client. Not immediately anyway. It can happen gradually
though.

As in: 1) alternative client is published that is compatible but
extended. 2) this client gets the majority share of users/miners 3) they
see this and decide to break compatibility. 4) original bitcoin client
is now incompatible/history.
Post by Andy Parkins
Post by Jeff Garzik
It's negative to weight costs vs. benefits? That is what I expect any
good engineer to do.
I don't think that's what's happening. Not once have I seen the "benefits"
side of that equation. What I have seen is plenty of "I can't see a use
case for that"; when the key word in that sentence is "I".
What is happening here is that most suggestions you point at have been
discussed about before and so the "early adopter" developers remember
that a decision about that was made already. However, the problem here
lies with the fact that it's difficult to find the previous
conversations.

If there was a section in the wiki for recording past suggestions, there
would be no need to say 'no'. You could instead say "We have discussed
this before, please read..." and give them a link to the page with the
relevant discussion. Of course, this would require actively forwarding
people to the wiki for discussions and having them there. I think this
would be a good idea.

That would leave this list for discussing and coordinating the
implementation of the changes that have been agreed on.

For things that have already been discussed, you could try to find the
previous discussion and add it there. But I expect for some things, new
discussion needs to be had to get it on the wiki.

- Joel
Christian Decker
2011-08-11 12:01:33 UTC
Permalink
On Thu, Aug 11, 2011 at 1:45 PM, Joel Joonatan Kaartinen <
Post by Joel Joonatan Kaartinen
Post by Andy Parkins
Again you're missing my point... you are still shooting ideas down.
And you're only shooting his actions down without indicating clearly
what you think ought to be done instead. What do you want him to say
instead?
Post by Andy Parkins
Post by Jeff Garzik
community also seems rather hard-wired against breaking changes like
that, because it implies that we lowly dev peons are daring to mess
with the Blessed Satoshi Design that has received extensive study, and
100% communal agreement.
Well the community had better unhardwire itself or its going to end up
with
Post by Andy Parkins
five developers and no more.
No way that will happen. A fork is going to happen sooner rather than
later if this continues. It'd be great if it could be done within this
project and named bitcoin-dev or bitcoin-next or similar.
I personally would welcome alternative clients as a vulnerability in the
main client right now has the potential to kill the entire network.
Post by Joel Joonatan Kaartinen
If this is not done, I wouldn't be surprised with the network splitting
into 2 camps with different protocols but still working on the same
blockchain.
Changes to the protocol are hard, mainly because hashes of packets are used
to identify transactions and blocks, and even the target hash is a hash of a
packet.
As for your proposal to eliminate some parts of the protocol, I have to
agree (the magic bytes seem an ugly hack by satoshi as I discussed with
Mike, and the double SHA256 hashes as checksums are incredibly wasteful, and
seem to have been chosen simply because a double hashing was already
implemented).
Post by Joel Joonatan Kaartinen
Post by Andy Parkins
Post by Jeff Garzik
If the community changes its mind, and there are strong calls to make
a breaking change, we can certainly do that. Bitcoin is not only open
source but very much democratic -- people vote by choosing which
client software to use.
Voting with ones feet should be a last resort. Wouldn't it be better not
to
Post by Andy Parkins
end up with incompatible clients out there?
There's no way to get the majority to vote with their feet to move to an
incompatible client. Not immediately anyway. It can happen gradually
though.
As in: 1) alternative client is published that is compatible but
extended. 2) this client gets the majority share of users/miners 3) they
see this and decide to break compatibility. 4) original bitcoin client
is now incompatible/history.
Changes should be implemented with backward compatibility in mind, even if
it restricts the freedom of what can be changed.
Post by Joel Joonatan Kaartinen
Post by Andy Parkins
Post by Jeff Garzik
It's negative to weight costs vs. benefits? That is what I expect any
good engineer to do.
I don't think that's what's happening. Not once have I seen the
"benefits"
Post by Andy Parkins
side of that equation. What I have seen is plenty of "I can't see a use
case for that"; when the key word in that sentence is "I".
What is happening here is that most suggestions you point at have been
discussed about before and so the "early adopter" developers remember
that a decision about that was made already. However, the problem here
lies with the fact that it's difficult to find the previous
conversations.
If there was a section in the wiki for recording past suggestions, there
would be no need to say 'no'. You could instead say "We have discussed
this before, please read..." and give them a link to the page with the
relevant discussion. Of course, this would require actively forwarding
people to the wiki for discussions and having them there. I think this
would be a good idea.
Having a Wiki or a single Wikipage to list proposed changes, with all pro
and cons, maybe pointing back to the original discussion would be nice. But
don't forget that situations change, and features that have been shot down
way back might become reachable/desirable at a later time, so please don't
just use it as a method to shoot down ideas, but as a way to bring people up
to speed and, if necessary, continue the discussion where it left.
Post by Joel Joonatan Kaartinen
That would leave this list for discussing and coordinating the
implementation of the changes that have been agreed on.
For things that have already been discussed, you could try to find the
previous discussion and add it there. But I expect for some things, new
discussion needs to be had to get it on the wiki.
- Joel
- cdecker
Andy Parkins
2011-08-11 14:04:22 UTC
Permalink
Post by Joel Joonatan Kaartinen
Post by Andy Parkins
Again you're missing my point... you are still shooting ideas down.
And you're only shooting his actions down without indicating clearly
Yeah, shooting down a shooting down, which you've just shot down. Where will
it end?
Post by Joel Joonatan Kaartinen
what you think ought to be done instead. What do you want him to say
instead?
How about:

"This is a good idea, but we don't want to break backward compatibility a
little piece at a time. Instead we'd like to collect all such changes into
one single compatibility breaking release. Here's the wiki page you should
update; and here's the git branch you should push changes like this to."
Post by Joel Joonatan Kaartinen
most suggestions you point at have been discussed about before
I know the application/protocol version split has been discussed before, but
please point me to the relevant discussion on: loading the block chain in
reverse; transaction only requests; checksumming removal; verack removal;
storing script parameters outside the script; and requesting blocks by
transaction hash instead of block hash.

If I've missed all of these discussions and their inevitable logically
indisputable rejection, I apologise.



Andy
--
Dr Andy Parkins
***@gmail.com
Mike Hearn
2011-08-11 12:11:29 UTC
Permalink
I don't think Gavin misunderstands how open source works at all. It's
completely normal for project maintainers to say "no" more often than
they say "yes". When I worked on open source for a living this was
part of the natural flow of things.

It's important to understand that ideas which receive "maybe" or "yes
but later" or "no unless you convince me" or "perhaps in a different
way" are not being shot down. These answers are requests for more work
to be done. You *cannot* get emotional about open source contributions
and any veteran will tell you this. Open source maintainers cannot and
do not say yes to every patch or idea that is proposed. I would be
very worried if Gavin did.
Don't believe me?  Here's a list of ideas
 - Extra bits in the service field of the version message to allow nodes
  to indicate if they are mining; if they are willing to be seed nodes;
  if they relay transactions; if they want relayed transactions.
I think the concept is reasonable but service flags might not be the
best way to do it, for instance, asking for a filtered transaction
feed is useful for lightweight clients so you'd want more precision
that can be fit into service bits.
 - getblocks in reverse chronological order so clients can start up quicker
  while downloading the blocks in the backround.  Ironically I was told
  "patches welcome" by someone who didn't reject this one instantly.
I already told you this won't help startup time because you have to
connect blocks together in sequence. You can't build up the block
chain backwards unless you don't care about validation at all.
 - Query miners for pending transactions
Or just have them send an inv containing them after connect. I don't
remember this one being "shot down".
 - Application version separate from client version
You mean separate from protocol version, right?
 - A way of requesting block bodies without headers (saving a lot of traffic
  for a thin client upgrading)
The cost/benefit ratio of this one isn't obvious at all. The resource
requirements for running a full node are large enough that
re-downloading 80 bytes per block is the least of your worries if
you're upgrading.
 - Double SHA-256 for a packet checksum?  Seriously?
Feel free to submit a patch to disable checksum validation and see if
Gavin accepts it. It needs to still be calculated at send time for
other implementations.
 - Sequence number as part of TxIn instead of part of the whole transaction
Sequence numbers are already part of the tx inputs. Or do you mean
they should be part of the whole transaction? If the latter then this
is indeed an idea that will be shot down, it's deliberate that seqnums
are part of the txinputs and it needs to be that way for contracts. It
can't be changed without forking the protocol anyway.
Every single one of those has been shot down by one or more of the main
developers.  I'm not a genius, and not arrogant enough to assume that
everything I say is right, but _nothing_?  Really?  There is no problem that
one of the above addresses?
Some of your proposals address problems that need to be solved, but
it's not clear that way is the right way to solve them. Others reflect
either lack of understanding of the system or the fact that you don't
value backwards compatibility whereas other people do.
Andy Parkins
2011-08-11 13:51:04 UTC
Permalink
Post by Mike Hearn
I don't think Gavin misunderstands how open source works at all. It's
completely normal for project maintainers to say "no" more often than
they say "yes". When I worked on open source for a living this was
part of the natural flow of things.
That wasn't the part I said he didn't understand. It was assuming that you
can just declare that people should work on bug fixes and not features was a
misunderstanding. People work on open source (at least at first) to get a
feature they want. They aren't just going to show up and cry "command me
lord".
Post by Mike Hearn
It's important to understand that ideas which receive "maybe" or "yes
but later" or "no unless you convince me" or "perhaps in a different
way" are not being shot down. These answers are requests for more work
to be done. You *cannot* get emotional about open source contributions
and any veteran will tell you this. Open source maintainers cannot and
do not say yes to every patch or idea that is proposed. I would be
very worried if Gavin did.
I don't expect them to; as I said, I'm not after everything I say being
accepted out of hand, certainly as I haven't even turned up with patches. And
you are absolutely correct that that would be worrying if it were so. What I
object to is no guidance is offered to get the suggester what they want, a
"you could have this if you did it like this", or "perhaps if you explained a
bit more". It's just "no, your idea is based on your weak understanding of
bitcoin," perhaps I'm being overly arrogant, but I think I understand it a lot
more than you presume I do.

I do try not to get emotional about these things; and email is not the best
medium for conveying level of distress -- I'm certainly not banging on my
keyboard, close to a heart attack. My motivation is only that I would like to
see bitcoin do well, and I do see that the treatment of potential new people,
while not offensive (nobody says f*ck off), is not encouraging.
Honestly you needn't have bothered. They've been reviewed to death at this
point; and I'm not that interested in fighting to get them into a project that
doesn't want them. I'll just play with my bricks over in the corner if that's
okay? I offered the list as a demonstration that ideas don't get constructive
help as to how progress can be made on them (i.e. how to make them
acceptable), they just get rejected.

Anyway; as you've put the time in, I'll do the same and respond.
Post by Mike Hearn
Post by Andy Parkins
Don't believe me? Here's a list of ideas
- Extra bits in the service field of the version message to allow nodes
to indicate if they are mining; if they are willing to be seed nodes;
if they relay transactions; if they want relayed transactions.
I think the concept is reasonable but service flags might not be the
best way to do it, for instance, asking for a filtered transaction
feed is useful for lightweight clients so you'd want more precision
that can be fit into service bits.
The service bits just seemed like the "bitcoin way" as the field already
existed. Personally I would prefer an additional "capabilities" request with
a variable number of ASCII strings in it, each indicating a capability, and if
that's good with all of you -- excellent.
Post by Mike Hearn
Post by Andy Parkins
- getblocks in reverse chronological order so clients can start up
quicker while downloading the blocks in the backround. Ironically I was
told "patches welcome" by someone who didn't reject this one instantly.
I already told you this won't help startup time because you have to
connect blocks together in sequence. You can't build up the block
chain backwards unless you don't care about validation at all.
I know you "told me this", but I think you are wrong. This is an example of
the problem I'm trying to get across -- I see things differently; but rather
than try and either fix my misunderstanding or see what I'm trying to achieve,
it's rejected.

I've already got it well on its way to being implemented is how I know you are
wrong. It's perfectly possible to validate backwards because you are
constructing a coherent chain based on an unvalidated start point. You then
request the parent block and either (a) you finally reach the genesis block,
you have reached a hard-coded valid point and the entire chain is therefore
instantly validated or (b) you have a new start block, floating but validated
to be part of the chain, if not absolutely validated. Further, with some
checkpoints hard coded you don't even need to reach the genesis block to get a
validated chain. The body of a block obviously can't be faked because of the
Merkle hash.

And finally... who says I care about validation? Perhaps I plan a situation
where I implicitly trust the peer I'm talking to (which is exactly what I do
plan). "There are more things in heaven and earth, Horatio, than are dreamt
of in your philosophy".
Post by Mike Hearn
Post by Andy Parkins
- Query miners for pending transactions
Or just have them send an inv containing them after connect. I don't
remember this one being "shot down".
I was told it had severe privacy implications; and you told me that it would
be better to wait for some sort of filtering system that was planned, which
I'd not heard of. I admit it wasn't exactly clear to me how what you
described helped with my suggestion. Your suggestion here is a good
alternative; but wouldn't it waste bandwidth? After all a receving node has
no idea whether I have been connected to another node for 24 hours before I
connect to it, and hence wouldn't need the list.
Post by Mike Hearn
Post by Andy Parkins
- Application version separate from client version
You mean separate from protocol version, right?
Yep. I can well imagine that when alternative clients start appearing, some
will have bugs. It will be very handy to either work around those bugs or
simply deny version 1.4.17 of "Andy's Sexy Bitcoin Client" from connecting.
Even just for monitoring network state it's useful. There is already talk, I
see, of establishing how much of the network runs each released bitcoin
version.
Post by Mike Hearn
Post by Andy Parkins
- A way of requesting block bodies without headers (saving a lot of
traffic for a thin client upgrading)
The cost/benefit ratio of this one isn't obvious at all. The resource
requirements for running a full node are large enough that
re-downloading 80 bytes per block is the least of your worries if
you're upgrading.
The benefit I'm aiming at is to imagine a thin client that has done a fast
startup and only downloaded the headers. Then, it has a finite number of
addresses it's interested in and wants to grab only the relevant bodies from
the full chain. Or, fast startup is to grab all the headers, and then slowly
grab the transactions from the blocks.

The cost is

if( !bodyOnly )
sendHeader();
sendBody();

I can't say I'm that invested in it; but it was another one for the list of
"well I don't see what use that is" responses.
Post by Mike Hearn
Post by Andy Parkins
- Double SHA-256 for a packet checksum? Seriously?
Feel free to submit a patch to disable checksum validation and see if
Gavin accepts it. It needs to still be calculated at send time for
other implementations.
I do feel free to write any patch I like. It's such a trivial patch though,
that I feel certain you are being faceitous, knowing full well that it
wouldn't be accepted. I'm trying to look five years in the future. I'm not
suggesting it be turned off now -- that's impossible and I'm not an idiot.
I'm trying to think of what the protocol should be and have a way of moving to
that.

The patch that is needed then is the one that makes the change gracefully.
Post by Mike Hearn
Post by Andy Parkins
- Sequence number as part of TxIn instead of part of the whole transaction
Sequence numbers are already part of the tx inputs. Or do you mean
they should be part of the whole transaction? If the latter then this
is indeed an idea that will be shot down, it's deliberate that seqnums
are part of the txinputs and it needs to be that way for contracts. It
can't be changed without forking the protocol anyway.
The sequence number (and perhaps I've misunderstood) allows me to replace a
transaction I've already submitted. I can't replace just one of the inputs, I
have to replace the whole transaction. It's therefore the transaction that
should have the sequence number. A signed transaction received with a higher
sequence number should displace a lower one.

I'm happy to accept that I have missed the use of the current sequence numbers
in contracts. (To be fair, the wiki says "Transaction version as defined by
the sender. Intended for "replacement" of transactions when information is
updated before inclusion into a block.")

Perhaps putting it in TxIn was because no one thought of having
OP_PUSH_SEQUENCENUMBER as a script operator.
Post by Mike Hearn
Post by Andy Parkins
Every single one of those has been shot down by one or more of the main
developers. I'm not a genius, and not arrogant enough to assume that
everything I say is right, but _nothing_? Really? There is no problem
that one of the above addresses?
Some of your proposals address problems that need to be solved, but
it's not clear that way is the right way to solve them. Others reflect
either lack of understanding of the system or the fact that you don't
value backwards compatibility whereas other people do.
Of the above, only one could be lack of understanding (txIn).

As to not valuing backward compatibility -- I certainly do. That shouldn't be
used as an excuse to freeze the protocol forever. There are version fields in
there, sensibly so; they should be used to fix problems. As I said a few
times, the incompatible changes don't have to activate straight away, they can
be delayed using the block number. Make it a block number four years away if
you want, but the sooner those changes go in (whatever they may be), the more
likely it is you'll get the majority of the network to change over. And once
the alternative clients start appearing, the opportunity is gone -- if it's
hard to get one client to change, imagine how hard it will be to change five.

As I said above though, I don't want these fights. I know full well that what
I want is not what you all want as far as client ideas go. I only started
this response because I thought Gavin's "we don't want new developers for new
features, we want bug fixes" was a bit of a foolish thing to say.




Andy
--
Dr Andy Parkins
***@gmail.com
John Smith
2011-08-11 12:19:45 UTC
Permalink
Post by Gavin Andresen
Well, to be honest I don't think more developers adding new features
are needed right now-- I think the project's critical needs are more
people testing and helping to fix bugs and scalability issues.
I do think to be an successful open source project we need more developers
and more features. Activity is very important, it is kind of the lifeblood.
Yes, a project will generally become more stable and slow in time as it
nears "perfection" (or at least, a local optimum) but the current source is
*far* from that.

Yes -- bugs and scalability issues need to be addressed, but this will be a
lot easier if some underlying problems are solved. And if we ignore the end
users, they may run away and it becomes a pointless exercise.

Taking the "long view" this includes build system, handling of
threads/concurrency, modularization, pluggable DB and storage back-ends,
separating the system into multiple "locked-down" processes, and so on.
This can all be done while remaining P2P-compatible for as long as possible
(we have versioning, don't we?).

My proposal with multiple branches was about looking at the long view as
well as the immediate firefighting. Yes, some changes might be riskier than
others, but we can't just cargo-cult Satoshi's work forever... so with
multiple branches, people can choose whether they have the balls to try
something newer or just want to run the older version with the issues they
know and love.

It's better to be open. Look at Open Office, it only started to un-stagnate
when it was forked out of Oracle's stranglehold. People want to work on
these things, so why not?

Until this is addressed, developers will prefer creating their own fork or
even alternative client. After this UI stuff is handled I'll probably join
up with one of them.
Post by Gavin Andresen
In this particular case, I said I was mildly against it-- if you want
me to switch to supporting it, then reassure me you're willing to do
ALL the work to make it happen. Send me a list of wiki pages you'll
edit to document the change and tell me that you'll be around to help
people rewrite their backup scripts.
IMO this should have been your first reply, instead of first discourgaging
me from doing it. Just make a list of what needs to be done.

But I won't bother anymore... Let's just keep lumping everything in one
executable. It's the Satoshi way.
Post by Gavin Andresen
Of course, nothing forces existing developers to accept these new
features;
Post by Gavin Andresen
but the incredibly negative attitude on display when any new feature is
suggested is not the way to grow a community. The correct way is a
mentoring attitude -- offering opinions on how a new developer can get
their
Post by Gavin Andresen
idea in rather than telling them why it will never happen.
Exactly. My gripe is more about the negative attitude then anything else.
The focus is always on the negative sides of every proposal, a bit of a
climate of fear.

I've had an employer that worked in the same way. Eternally hammering on
"stability" the codebase, hiring 100's of extra developers, all firefighting
and fixing immediate issues with "priority", the code became a minefield.
Even with 8 hours of testcases, the overall structure of the code caused so
many issues that customers feared every new release more. A classic negative
feedback loop.

I understand where it is coming from, many people just come and dump their
"ideas" and never implement a line of code. But if people are actually
proposing to implement something, or implemented it, they should IMO be
given the benefit of the doubt. Not all outside ideas are bad.

JS
Pieter Wuille
2011-08-11 13:08:45 UTC
Permalink
Post by John Smith
Post by Gavin Andresen
In this particular case, I said I was mildly against it-- if you want
me to switch to supporting it, then reassure me you're willing to do
ALL the work to make it happen. Send me a list of wiki pages you'll
edit to document the change and tell me that you'll be around to help
people rewrite their backup scripts.
IMO this should have been your first reply, instead of first discourgaging
me from doing it. Just make a list of what needs to be done.
But I won't bother anymore... Let's just keep lumping everything in one
executable. It's the Satoshi way.
I think there are a lot of misunderstandings here. Given your clarification
that you simply wanted to remove the RPC client from the gui/daemon executable,
I'm all for it. If I reread the answers, there is only "mild against" and a "no"
that was based on a partial misunderstanding.

Somehow it seems that many discussions in which some negative remarks were made
just die out, and the person originally suggesting it takes this as a "shot
down". Maybe (and that includes myself) we should be more outspoken about ideas
we do like.

As for the rest of this thread: i think some dev branch (either something
linux-next like, or a separate official branch, or something else) is indeed
very needed. The main tree should definitely be dealt with in a conservative
way, but it's hard to make progress if you know that every patch that does
some internal changes will need many rebasings and maintainance before it's
actually merged and finally tested by some larger user base.

Considering the issue of backward incompatible changes to the protocol: there is
no denying that there are some serious deficiencies now (double sha256 checksums,
the handling of the data being signed) and redundant things (magic bytes, verack).

Yes, it is true that we could change these in the future with a (nBlocks >= X)
test, but that would still mean you carry around both the old and the new code
until at least block height X. Additionally, if you get another (better) idea
that supercedes it somewhere between now and block X, you're still forced to
first switch to the intermediate one, as some clients may not have upgraded...

This is not to say this isn't an option we should consider, but for now, it just
doesn't seem worth the hassle to me. There may come a day where we absolutely
need a change to the protocol, and when we do, maybe it is time to fix all these
"known and agreed upon defficiencies". It's definitely useful to discuss these,
and in the context of "for when we do an incompatible change", no "breaks backward
compatibility!" argument is valid. I'm in favor of wiki page where these are listed,
together with pro and cons.
--
Pieter
Luke-Jr
2011-08-10 18:43:14 UTC
Permalink
Post by John Smith
0.3.x -> small, compatible changes, bugfixes, like now
0.4.x -> trunk, more impactful changes, refactorings, eventual major release
It seems there's room for some kind of "experimental" branch as well,
including features that might not make it into any stable release (due to lack
of use/interest or whatever).
Jeff Garzik
2011-08-10 19:48:10 UTC
Permalink
Post by Luke-Jr
Post by John Smith
0.3.x -> small, compatible changes, bugfixes, like now
0.4.x -> trunk, more impactful changes, refactorings, eventual major release
It seems there's room for some kind of "experimental" branch as well,
including features that might not make it into any stable release (due to lack
of use/interest or whatever).
In kernel land there exists "linux-next" Stephen Rothwell maintains a
tree that is linux -tip, plus a list of trees & branches to pull from
various individual developers. For example, linux-next pulls my SATA
tree from libata-dev.git branch NEXT.

Each developer is expected to publish changes they feel are ready for
upstream. Developers are expected to "play nicely" and coordinate
amongst themselves when two trees include conflicting changes.
Trivial merge conflicts are handled by Stephen Rothwell, who does
merging, build testing and such of the final set-of-N-trees result.
More difficult merge conflicts are coordinated by the developers
themselves, who work together to create a temporary "merge tree" that
is then pulled by the linux-next maintainer.

linux-next is the always moving, regenerated daily target where
developers stage [in their opinion] upstream-ready changes.

Thus Linus's linux.git development process really looks like the
following, when linux-next is included in the picture:

1. Version X-1 is released, on day 0.
2. Merge window for version X opens, on day 0.
3. Linus pulls all changes that have seen testing in linux-next, over
the -rc window (step #6, below)
4. Merge window closes, on day 7.
5. Version X-rc1 is released, on day 7.
6. Only bug fixes are accepted now (hopefully seen at least 24 hours
of testing in linux-next, unless urgency demands otherwise). All new
development is done in developer trees and branches, and is
automatically published nightly in linux-next.
7. Version X is released, on day 90.

Thus "upstream" stays almost constantly stable, except for the short
1-week merge window period, and linux-next comprises the rolling
"development version" where new changes are staged.

Note the subtle but important distinction between this and maintaining
a strict 'bugfix' and 'development' branch system like John Smith
described. The underlying linux-next dependent trees may be rebased
at any time, and so linux-next is constantly regenerated, rather than
being a cumulative history of choatic development. Major changes can
and will be staged, de-staged, and re-staged during development, and
maintaining a strict "official development branch" methodology is less
flexible.

Here is an example linux-next report. Stephen sends one, daily, with
each linux-next tree generated:
http://marc.info/?l=linux-next&m=131295044704945&w=2

As it applies to bitcoin, this "bitcoin-next" approach may simply be
layered on top of the current methodology. All it requires is a
volunteer who maintains this tree-of-trees, and wha-la: bitcoin has a
development branch.
--
Jeff Garzik
exMULTI, Inc.
***@exmulti.com
Jeff Garzik
2011-08-10 21:37:41 UTC
Permalink
Post by John Smith
In the current mainline client everything is lugged into one executable
(with an optional daemon-only one). I think this is a bad idea for various
bitcoind: bitcoin daemon
bitcoin(-qt): bitcoin GUI executable
bitcoincl: bitcoin RPC command line
By default, all three would be built. In non-GUI mode, only bitcoind and
bitcoincl are built (the names are obviously open for discussion).
Seems reasonable to me.
--
Jeff Garzik
exMULTI, Inc.
***@exmulti.com
Pieter Wuille
2011-08-11 13:50:54 UTC
Permalink
Post by John Smith
All,
In the current mainline client everything is lugged into one executable
(with an optional daemon-only one). I think this is a bad idea for various
bitcoind: bitcoin daemon
bitcoin(-qt): bitcoin GUI executable
bitcoincl: bitcoin RPC command line
Back on topic:

I initially misunderstood your proposal. Let me reformulate, and
suggest some names:
* bitcoin-gui (or bitcoin-qt): always starts GUI, optionally starts
RPC server, no RPC client
* bitcoin-server: always starts RPC server, no RPC client, no GUI
* bitcoin-client: always runs RPC client, no RPC server, no GUI

Additionally, we could offer a script or symlinked executable with
names "bitcoin" and
"bitcoind" that detect whether RPC commands are present on the command line, and
based on this invoke either bitcoin-server/bitcoin-gui or
bitcoin-client (for backward
compatibility).
--
Pieter
Loading...