Discussion:
is there any value in completely rewriting unix from scratch
(too old to reply)
G G
2020-07-19 00:01:35 UTC
Permalink
would there be any value in rewriting unix from scratch in 2020?


completely leaving behind the ideal of back compatibility.

I wonder how clean that would. I wonder too if it would result in
small source.

It would be so cool to see versions written in all Rust, all C, all C++,
all Swift, all Ada, all Java, using the latest defs of the respective
languages. yes and all new libraries.

Can the holes in Security be designed out? hmm?
Keith Thompson
2020-07-19 00:04:11 UTC
Permalink
Post by G G
would there be any value in rewriting unix from scratch in 2020?
completely leaving behind the ideal of back compatibility.
If you leave behind the idea of backwards compatibility, in what sense
is the result Unix?
Post by G G
I wonder how clean that would. I wonder too if it would result in
small source.
It would be so cool to see versions written in all Rust, all C, all C++,
all Swift, all Ada, all Java, using the latest defs of the respective
languages. yes and all new libraries.
Can the holes in Security be designed out? hmm?
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+***@gmail.com
Working, but not speaking, for Philips Healthcare
void Void(void) { Void(); } /* The recursive call of the void */
G G
2020-07-19 01:28:25 UTC
Permalink
Post by Keith Thompson
If you leave behind the idea of backwards compatibility, in what sense
is the result Unix?
the thought would be to write to the specification of Unix in 2020, 2021,
and if old software ran, ok, if not, a rewrite of that software would be great.
I figure programmers love solving problems. the research in algorithm
design has to afford some clearer, speedier, safer opportunities overall.
Kaz Kylheku
2020-07-19 01:46:27 UTC
Permalink
Post by Keith Thompson
Post by G G
would there be any value in rewriting unix from scratch in 2020?
completely leaving behind the ideal of back compatibility.
If you leave behind the idea of backwards compatibility, in what sense
is the result Unix?
One way would be to be backward compatible to something earlier,
like say 1990 vintage POSIX. From that vantage point, for instance,
better designed threads could be added.
Boris Dorestand
2020-07-20 00:38:52 UTC
Permalink
Post by Kaz Kylheku
Post by Keith Thompson
Post by G G
would there be any value in rewriting unix from scratch in 2020?
completely leaving behind the ideal of back compatibility.
If you leave behind the idea of backwards compatibility, in what sense
is the result Unix?
One way would be to be backward compatible to something earlier,
like say 1990 vintage POSIX. From that vantage point, for instance,
better designed threads could be added.
It's a major job. When people undertake something like that, it seems
they decide to do something that might end up being a lot better. Is
Plan9 an example?
Siri Cruise
2020-07-20 02:12:34 UTC
Permalink
Post by Boris Dorestand
Post by Kaz Kylheku
Post by Keith Thompson
Post by G G
would there be any value in rewriting unix from scratch in 2020?
completely leaving behind the ideal of back compatibility.
If you leave behind the idea of backwards compatibility, in what sense
is the result Unix?
One way would be to be backward compatible to something earlier,
like say 1990 vintage POSIX. From that vantage point, for instance,
better designed threads could be added.
It's a major job. When people undertake something like that, it seems
they decide to do something that might end up being a lot better. Is
Plan9 an example?
A tactic is to take chunks of Linux, recreate them, and sneak
them in. Do enough that in accord to secret treaties and
eventually you can discuss don't fear Theseus's ship.
--
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted. @
'I desire mercy, not sacrifice.' /|\
The first law of discordiamism: The more energy This post / \
to make order is nore energy made into entropy. insults Islam. Mohammed
James K. Lowden
2020-07-20 19:31:24 UTC
Permalink
On Sun, 19 Jul 2020 21:38:52 -0300
Post by Boris Dorestand
Post by G G
would there be any value in rewriting unix from scratch in 2020?
completely leaving behind the ideal of back compatibility.
It's a major job. When people undertake something like that, it seems
they decide to do something that might end up being a lot better. Is
Plan9 an example?
Plan9 is *the* example, and proof progress has become impossible.

In the BSD community particularly, but also in Linux, there is a
conservatism -- I would say fetishism -- toward the OS interfaces that
prevents new paradigms from emerging. It's possible only to fiddle
around the edges.

Plan9 is very unix-like in many ways; one might say more unixy than
Unix Because it was developed after bitmapped displays and networking
became ubiquitous, they're part of Plan9, not added on in the form of
rpc, ssh, Kerberos, and X11. Everything is under /dev again, and one
protocol unifies the distributed parts.

Plan9 isn't the only example of good ideas rejected. IPv6 and IPsec
never made it past the embryo stage. Instead of Plan9's
kernel-integrated, exportable window system, we're looking forward (so
to speak) to Wayland, still a userspace system, but with remote display
reluctantly, if at all. Then there's my favorite, troff, a markup
language that improves on each of its successors, bar none.

The fundamental problem, as I see it, is that there's no consensus on
"better". Each one of the things I mentioned is controversial even
here, where most readers have at least a passing familiarity with
them. In the wider world, not only is the cruftiness and dysfunction
of the OS not well understood, it's being reproduced in Javascript,
cargo-cult style, to run in the browser.

--jkl
Scott Lurndal
2020-07-20 21:44:16 UTC
Permalink
Post by James K. Lowden
Plan9 isn't the only example of good ideas rejected. IPv6 and IPsec
never made it past the embryo stage.
On this, we must disagree. IPsec and IPv6 are widely used.
James K. Lowden
2020-07-21 16:59:23 UTC
Permalink
On Mon, 20 Jul 2020 21:44:16 GMT
Post by James K. Lowden
IPv6 and IPsec
Post by James K. Lowden
never made it past the embryo stage.
On this, we must disagree. IPsec and IPv6 are widely used.
Are they? I would bet for every IPv6 address in use, there's at least
one IPv4 NAT router. I've never worked anywhere where the local
network was IPv6.

I really thought IPsec died on the vine. If it was ubitquitous, SMTP
traffic would be immune to network snooping.

Could you explain what you mean by "widely"?

--jkl
Miquel van Smoorenburg
2020-07-23 12:41:02 UTC
Permalink
Post by James K. Lowden
On Mon, 20 Jul 2020 21:44:16 GMT
Post by James K. Lowden
IPv6 and IPsec
Post by James K. Lowden
never made it past the embryo stage.
On this, we must disagree. IPsec and IPv6 are widely used.
Are they? I would bet for every IPv6 address in use, there's at least
one IPv4 NAT router. I've never worked anywhere where the local
network was IPv6.
I work at an ISP in Europe. All our subscribers have IPv6. More than
half our traffic is IPv6.

Not surprising, given that Google / Youtube / Facebook / Netflix /
Cloudflare / etc all serve their content over IPv6 (and v4, obviously).

Most of the work of implementing IPv6 is convincing people that it
needs to be done. Then just do it.

Also, in the mobile world IPv6 is everywhere. For example, the T-Mobile
network in the USA is IPv6-only. They NAT IPv4 to IPv6 at the edge.

Mike.
Boris Dorestand
2020-07-23 18:10:27 UTC
Permalink
Post by Miquel van Smoorenburg
Post by James K. Lowden
On Mon, 20 Jul 2020 21:44:16 GMT
Post by James K. Lowden
IPv6 and IPsec
Post by James K. Lowden
never made it past the embryo stage.
On this, we must disagree. IPsec and IPv6 are widely used.
Are they? I would bet for every IPv6 address in use, there's at least
one IPv4 NAT router. I've never worked anywhere where the local
network was IPv6.
I work at an ISP in Europe. All our subscribers have IPv6. More than
half our traffic is IPv6.
Not surprising, given that Google / Youtube / Facebook / Netflix /
Cloudflare / etc all serve their content over IPv6 (and v4, obviously).
When will they turn off their IPv4 support?
Post by Miquel van Smoorenburg
Most of the work of implementing IPv6 is convincing people that it
needs to be done. Then just do it.
Who are these people to be convinved?
Post by Miquel van Smoorenburg
Also, in the mobile world IPv6 is everywhere. For example, the T-Mobile
network in the USA is IPv6-only. They NAT IPv4 to IPv6 at the edge.
When will they turn off their NAT?

Thanks!
Kees Nuyt
2020-07-23 22:24:59 UTC
Permalink
On Thu, 23 Jul 2020 15:10:27 -0300, Boris Dorestand
[...]
Post by Boris Dorestand
Post by Miquel van Smoorenburg
I work at an ISP in Europe. All our subscribers have IPv6. More than
half our traffic is IPv6.
Not surprising, given that Google / Youtube / Facebook / Netflix /
Cloudflare / etc all serve their content over IPv6 (and v4, obviously).
When will they turn off their IPv4 support?
They don't have to. IPv6 is designed to co-exist with IPv4.
In theory, when all relevant user devices and all Customer
Premises Equipment and all ISP's support IPv6.
Post by Boris Dorestand
Post by Miquel van Smoorenburg
Most of the work of implementing IPv6 is convincing people that it
needs to be done. Then just do it.
Who are these people to be convinved?
Many small content providers. Some ISP's. Some providers of
proxy servers, VPN's, firewalls.
Of those, mostly owners and management. Tech staff is usually
convinced, but often lacks resources and time.
Post by Boris Dorestand
Post by Miquel van Smoorenburg
Also, in the mobile world IPv6 is everywhere. For example, the T-Mobile
network in the USA is IPv6-only. They NAT IPv4 to IPv6 at the edge.
When will they turn off their NAT?
As soon as example.com doesn't have an IPv4 address anymore, in
other words, all ISP's and all content providers (including all
the tiny personal websites) have switched over to IPv6,
so probably not in the next decade.
--
Regards,
Kees Nuyt
Boris Dorestand
2020-07-24 00:13:41 UTC
Permalink
Post by Kees Nuyt
On Thu, 23 Jul 2020 15:10:27 -0300, Boris Dorestand
[...]
Post by Boris Dorestand
Post by Miquel van Smoorenburg
I work at an ISP in Europe. All our subscribers have IPv6. More than
half our traffic is IPv6.
Not surprising, given that Google / Youtube / Facebook / Netflix /
Cloudflare / etc all serve their content over IPv6 (and v4, obviously).
When will they turn off their IPv4 support?
They don't have to. IPv6 is designed to co-exist with IPv4.
In theory, when all relevant user devices and all Customer
Premises Equipment and all ISP's support IPv6.
I believe this might never happen. Say I'm a programmer. I got a
useful software somewhere that doesn't support IPv6. Should I stop and
go update it? I guess not: I gotta earn a living. IPv4 will always be
there, so I'm good.
Post by Kees Nuyt
Post by Boris Dorestand
Post by Miquel van Smoorenburg
Most of the work of implementing IPv6 is convincing people that it
needs to be done. Then just do it.
Who are these people to be convinved?
Many small content providers. Some ISP's. Some providers of
proxy servers, VPN's, firewalls.
Of those, mostly owners and management. Tech staff is usually
convinced, but often lacks resources and time.
I think somehow IPv6 should have managed to coexist with IPv4. (Not
saying this is easy or even possible.) Take UTF-8. It's a superset of
ASCII. That's a solution. IPv6 should somehow be a superset of IPv4.
That would have been a solution.
Post by Kees Nuyt
Post by Boris Dorestand
Post by Miquel van Smoorenburg
Also, in the mobile world IPv6 is everywhere. For example, the T-Mobile
network in the USA is IPv6-only. They NAT IPv4 to IPv6 at the edge.
When will they turn off their NAT?
As soon as example.com doesn't have an IPv4 address anymore, in
other words, all ISP's and all content providers (including all
the tiny personal websites) have switched over to IPv6,
so probably not in the next decade.
Do you see it happening eventually?
b***@nowhere.co.uk
2020-07-24 07:43:46 UTC
Permalink
On Thu, 23 Jul 2020 21:13:41 -0300
Post by Boris Dorestand
I think somehow IPv6 should have managed to coexist with IPv4. (Not
saying this is easy or even possible.) Take UTF-8. It's a superset of
ASCII. That's a solution. IPv6 should somehow be a superset of IPv4.
That would have been a solution.
Too complicated because (assuming there are any free flags left in the
header) IP4 stacks wouldn't realise its IP6 because they wouldn't
recognise the flag and would try and parse the packet anyway. Its much
easy to set the IP version field in the header so it gets sent to the
correct stack. What they did do wrong was the unwieldy 128bit addresses
which is way too much (yes, 640K enough for anyone etc etc) but in this
case 64 bits really would have been enough for now and the forseable
future and would have made manual setup a lot simpler. And yes, that
does still have to be done in a lot of places with air gapped networks
and just relying on the link IP6 local address won't work.
Miquel van Smoorenburg
2020-07-25 01:06:32 UTC
Permalink
Post by Boris Dorestand
Post by Kees Nuyt
On Thu, 23 Jul 2020 15:10:27 -0300, Boris Dorestand
They don't have to. IPv6 is designed to co-exist with IPv4.
In theory, when all relevant user devices and all Customer
Premises Equipment and all ISP's support IPv6.
I believe this might never happen. Say I'm a programmer. I got a
useful software somewhere that doesn't support IPv6. Should I stop and
go update it? I guess not: I gotta earn a living. IPv4 will always be
there, so I'm good.
Well, unless you're the author of an iOS app. If your app is
dependent on network access in any way, it has to do dual-stack,
or Apple will not allow it in the app store.

BTW, the protocol independent versions of gethostbyname/gethostbyaddr
are getaddrinfo/getnameinfo and they already existed in the
previous century.

Mike.
Jorgen Grahn
2020-07-25 06:36:31 UTC
Permalink
Post by Boris Dorestand
Post by Kees Nuyt
On Thu, 23 Jul 2020 15:10:27 -0300, Boris Dorestand
[...]
Post by Boris Dorestand
Post by Miquel van Smoorenburg
I work at an ISP in Europe. All our subscribers have IPv6. More than
half our traffic is IPv6.
Ah, I would love to have such an ISP.
Post by Boris Dorestand
Post by Kees Nuyt
Post by Boris Dorestand
Post by Miquel van Smoorenburg
Not surprising, given that Google / Youtube / Facebook / Netflix /
Cloudflare / etc all serve their content over IPv6 (and v4, obviously).
When will they turn off their IPv4 support?
They don't have to. IPv6 is designed to co-exist with IPv4.
In theory, when all relevant user devices and all Customer
Premises Equipment and all ISP's support IPv6.
I believe this might never happen. Say I'm a programmer. I got a
useful software somewhere that doesn't support IPv6. Should I stop and
go update it? I guess not: I gotta earn a living. IPv4 will always be
there, so I'm good.
Note that supporting IPv6 isn't hard. If you use the right APIs
(getaddrinfo(3) and friends) you probably support it already. If you
use a higher-level API you probably have to do something explicit to
disable it.

/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
Boris Dorestand
2020-07-22 00:24:49 UTC
Permalink
Post by Scott Lurndal
Post by James K. Lowden
Plan9 isn't the only example of good ideas rejected. IPv6 and IPsec
never made it past the embryo stage.
On this, we must disagree. IPsec and IPv6 are widely used.
I'm not sure IPv6 is a good idea. :-) It seems to me IPv6 fails to find
an effective means to implement itself. I think someone down this this
thread, in message-id

<***@speakeasy.net>

makes the interesting comment that ``[he has] never worked anywhere
where the local network was IPv6.'' That's my observation too.
b***@nowhere.co.uk
2020-07-22 08:00:52 UTC
Permalink
On Tue, 21 Jul 2020 21:24:49 -0300
Post by Boris Dorestand
Post by Scott Lurndal
Post by James K. Lowden
Plan9 isn't the only example of good ideas rejected. IPv6 and IPsec
never made it past the embryo stage.
On this, we must disagree. IPsec and IPv6 are widely used.
I'm not sure IPv6 is a good idea. :-) It seems to me IPv6 fails to find
an effective means to implement itself. I think someone down this this
thread, in message-id
makes the interesting comment that ``[he has] never worked anywhere
where the local network was IPv6.'' That's my observation too.
After 20 years things seemed to have settled into a state where IP6 is mostly
used for network backbone data transfer and IP4 is used internally in companies
and home routers where IP6 is overkill. The company I work for produces gear
that is networked to control industrial equipment and it only runs IP4 which is
on an airgapped network. There's no need for IP6 and the IP4 addresses and
subnets are static and have to be set manually which is much easier with IP4
than IP6.
b***@nowhere.co.uk
2020-07-20 07:51:32 UTC
Permalink
On Sat, 18 Jul 2020 17:01:35 -0700 (PDT)
Post by G G
would there be any value in rewriting unix from scratch in 2020?
GNU tried that with Plan 9. It wasn't a stellar success. Not sure if its even
still being developed.
Post by G G
I wonder how clean that would. I wonder too if it would result in
small source.
Unlikely. The core kernel only takes up a small percentage of the code, most
of it is contained in drivers.
Post by G G
It would be so cool to see versions written in all Rust, all C, all C++,
all Swift, all Ada, all Java, using the latest defs of the respective
languages. yes and all new libraries.
Java? Good luck with that. How will it control the hardware when it has little
to no to-the-metal abilities and what will run its JVM if its running the OS?
Keith Thompson
2020-07-20 08:54:03 UTC
Permalink
Post by b***@nowhere.co.uk
On Sat, 18 Jul 2020 17:01:35 -0700 (PDT)
Post by G G
would there be any value in rewriting unix from scratch in 2020?
GNU tried that with Plan 9. It wasn't a stellar success. Not sure if its even
still being developed.
Plan 9 was from Bell Labs, not GNU.

[...]
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+***@gmail.com
Working, but not speaking, for Philips Healthcare
void Void(void) { Void(); } /* The recursive call of the void */
b***@nowhere.org
2020-07-20 11:17:19 UTC
Permalink
On Mon, 20 Jul 2020 01:54:03 -0700
Post by Keith Thompson
Post by b***@nowhere.co.uk
On Sat, 18 Jul 2020 17:01:35 -0700 (PDT)
Post by G G
would there be any value in rewriting unix from scratch in 2020?
GNU tried that with Plan 9. It wasn't a stellar success. Not sure if its
even
Post by b***@nowhere.co.uk
still being developed.
Plan 9 was from Bell Labs, not GNU.
Yes, my mistake. GNUs effort was called Hurd. Point still stands though.
Boris Dorestand
2020-07-20 14:20:10 UTC
Permalink
Post by b***@nowhere.org
On Mon, 20 Jul 2020 01:54:03 -0700
Post by Keith Thompson
Post by b***@nowhere.co.uk
On Sat, 18 Jul 2020 17:01:35 -0700 (PDT)
Post by G G
would there be any value in rewriting unix from scratch in 2020?
GNU tried that with Plan 9. It wasn't a stellar success. Not sure if its
even
Post by b***@nowhere.co.uk
still being developed.
Plan 9 was from Bell Labs, not GNU.
Yes, my mistake. GNUs effort was called Hurd. Point still stands though.
GNU seems to be doing well with kFreeBSD.

https://www.debian.org/ports/kfreebsd-gnu/
b***@nowhere.co.uk
2020-07-20 14:43:33 UTC
Permalink
On Mon, 20 Jul 2020 11:20:10 -0300
Post by Boris Dorestand
Post by b***@nowhere.org
On Mon, 20 Jul 2020 01:54:03 -0700
Post by Keith Thompson
Post by b***@nowhere.co.uk
On Sat, 18 Jul 2020 17:01:35 -0700 (PDT)
Post by G G
would there be any value in rewriting unix from scratch in 2020?
GNU tried that with Plan 9. It wasn't a stellar success. Not sure if its
even
Post by b***@nowhere.co.uk
still being developed.
Plan 9 was from Bell Labs, not GNU.
Yes, my mistake. GNUs effort was called Hurd. Point still stands though.
GNU seems to be doing well with kFreeBSD.
https://www.debian.org/ports/kfreebsd-gnu/
I don't quite see the point of that. If you want to run a BSD just run one.
I'd rather there were more Linux distros without the systemd abomination.
Rainer Weikusat
2020-07-20 15:00:42 UTC
Permalink
Post by Boris Dorestand
Post by b***@nowhere.org
On Mon, 20 Jul 2020 01:54:03 -0700
Post by Keith Thompson
Post by b***@nowhere.co.uk
On Sat, 18 Jul 2020 17:01:35 -0700 (PDT)
Post by G G
would there be any value in rewriting unix from scratch in 2020?
GNU tried that with Plan 9. It wasn't a stellar success. Not sure if its
even
Post by b***@nowhere.co.uk
still being developed.
Plan 9 was from Bell Labs, not GNU.
Yes, my mistake. GNUs effort was called Hurd. Point still stands though.
GNU seems to be doing well with kFreeBSD.
https://www.debian.org/ports/kfreebsd-gnu/
DING! (Debian Is Not Gnu)

Apart from that, the port to a BSD-kernel is dead-in-water: Since Debian
has set forth towards becoming a rebranded version of Fedora (ie,
another distribution of systemd slowly converging towards "the" systemd
distribution), the "default installation" can no longer run on top of
anything but (certain versions of) Linux because systemd needs that.
Boris Dorestand
2020-07-20 23:22:23 UTC
Permalink
Post by Rainer Weikusat
Post by Boris Dorestand
Post by b***@nowhere.org
On Mon, 20 Jul 2020 01:54:03 -0700
Post by Keith Thompson
Post by b***@nowhere.co.uk
On Sat, 18 Jul 2020 17:01:35 -0700 (PDT)
Post by G G
would there be any value in rewriting unix from scratch in 2020?
GNU tried that with Plan 9. It wasn't a stellar success. Not sure if its
even
Post by b***@nowhere.co.uk
still being developed.
Plan 9 was from Bell Labs, not GNU.
Yes, my mistake. GNUs effort was called Hurd. Point still stands though.
GNU seems to be doing well with kFreeBSD.
https://www.debian.org/ports/kfreebsd-gnu/
DING! (Debian Is Not Gnu)
That's right, I guess. Good catch.
Post by Rainer Weikusat
Apart from that, the port to a BSD-kernel is dead-in-water: Since Debian
has set forth towards becoming a rebranded version of Fedora (ie,
another distribution of systemd slowly converging towards "the" systemd
distribution), the "default installation" can no longer run on top of
anything but (certain versions of) Linux because systemd needs that.
That sucks. Speaking of which, how do you explain the popularity of
systemd? Why does ``everyone'' love it?
b***@nowhere.co.uk
2020-07-21 07:56:29 UTC
Permalink
On Mon, 20 Jul 2020 20:22:23 -0300
Post by Boris Dorestand
Post by Rainer Weikusat
Post by Boris Dorestand
Post by b***@nowhere.org
On Mon, 20 Jul 2020 01:54:03 -0700
Post by Keith Thompson
Post by b***@nowhere.co.uk
On Sat, 18 Jul 2020 17:01:35 -0700 (PDT)
Post by G G
would there be any value in rewriting unix from scratch in 2020?
GNU tried that with Plan 9. It wasn't a stellar success. Not sure if its
even
Post by b***@nowhere.co.uk
still being developed.
Plan 9 was from Bell Labs, not GNU.
Yes, my mistake. GNUs effort was called Hurd. Point still stands though.
GNU seems to be doing well with kFreeBSD.
https://www.debian.org/ports/kfreebsd-gnu/
DING! (Debian Is Not Gnu)
That's right, I guess. Good catch.
Post by Rainer Weikusat
Apart from that, the port to a BSD-kernel is dead-in-water: Since Debian
has set forth towards becoming a rebranded version of Fedora (ie,
another distribution of systemd slowly converging towards "the" systemd
distribution), the "default installation" can no longer run on top of
anything but (certain versions of) Linux because systemd needs that.
That sucks. Speaking of which, how do you explain the popularity of
systemd? Why does ``everyone'' love it?
There have been rumours that it was a deliberate ploy by DeadRat to own most
of the non-kernel linux space. If you own init and persuade enough idiots to
use your replacement then you're effectively leading them by the nose and
you can dictate where Linux user space goes in the future. Which suits IBM
who now own RedHat down to the ground.
Nicolas George
2020-07-21 08:26:01 UTC
Permalink
Post by b***@nowhere.co.uk
If you own init and persuade enough idiots to
The arrogance of calling a majority of developers of many excellent
distributions "idiots"...
b***@nowhere.co.uk
2020-07-21 14:11:52 UTC
Permalink
On 21 Jul 2020 08:26:01 GMT
Post by Nicolas George
Post by b***@nowhere.co.uk
If you own init and persuade enough idiots to
The arrogance of calling a majority of developers of many excellent
distributions "idiots"...
Most of the distro developers don't have any say in it.
Nicolas George
2020-07-21 14:36:49 UTC
Permalink
Post by b***@nowhere.co.uk
Most of the distro developers don't have any say in it.
That's not true, as you could see if you were more interested in learning
the truth than spreading conspiracy theories.
b***@nowhere.co.uk
2020-07-21 15:05:52 UTC
Permalink
On 21 Jul 2020 14:36:49 GMT
Post by Nicolas George
Post by b***@nowhere.co.uk
Most of the distro developers don't have any say in it.
That's not true, as you could see if you were more interested in learning
the truth than spreading conspiracy theories.
You have a strange definition of conspiracy theories.
Nicolas George
2020-07-21 15:46:37 UTC
Permalink
Post by b***@nowhere.co.uk
You have a strange definition of conspiracy theories.
Who wrote "There have been rumours that it was a deliberate ploy by"?
Jorgen Grahn
2020-07-21 16:44:29 UTC
Permalink
Post by Nicolas George
Post by b***@nowhere.co.uk
You have a strange definition of conspiracy theories.
Who wrote "There have been rumours that it was a deliberate ploy by"?
Rumors about secret business plans are not necessarily conspiracy
theories. (They may still be false, of course.)

/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
Nicolas George
2020-07-21 17:16:24 UTC
Permalink
Jorgen Grahn , dans le message
Post by Jorgen Grahn
Rumors about secret business plans are not necessarily conspiracy
theories. (They may still be false, of course.)
When "business" involve community distributions, they are.
Jorgen Grahn
2020-07-24 08:35:51 UTC
Permalink
Post by Nicolas George
Jorgen Grahn , dans le message
Post by Jorgen Grahn
Rumors about secret business plans are not necessarily conspiracy
theories. (They may still be false, of course.)
When "business" involve community distributions, they are.
Yes, I agree. At least at the time I wrote that, the claim seemed to
be about RedHat/IBM only, with the others as unsuspecting victims.

/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
Nicolas George
2020-07-24 08:45:44 UTC
Permalink
Jorgen Grahn , dans le message
Post by Jorgen Grahn
Yes, I agree. At least at the time I wrote that, the claim seemed to
be about RedHat/IBM only, with the others as unsuspecting victims.
Post by Nicolas George
The arrogance of calling a majority of developers of many excellent
distributions "idiots"...
Because conspiracy theorists are nothing if not arrogant: everybody is wrong
except them.
b***@nowhere.co.uk
2020-07-24 09:24:47 UTC
Permalink
On 24 Jul 2020 08:45:44 GMT
Post by Nicolas George
Jorgen Grahn , dans le message
Post by Jorgen Grahn
Yes, I agree. At least at the time I wrote that, the claim seemed to
be about RedHat/IBM only, with the others as unsuspecting victims.
Post by Nicolas George
The arrogance of calling a majority of developers of many excellent
distributions "idiots"...
Because conspiracy theorists are nothing if not arrogant: everybody is wrong
except them.
Given the shear amount of pushback against systemd , going ahead with it
anyway demonstrates the arrogance that is within the distro maintainers. It
even caused a fork of Debian.
Nicolas George
2020-07-24 09:28:59 UTC
Permalink
Post by b***@nowhere.co.uk
even caused a fork of Debian.
Yes, a failing one. That should tell you something.
b***@nowhere.co.uk
2020-07-24 12:18:41 UTC
Permalink
On 24 Jul 2020 09:28:59 GMT
Post by Nicolas George
Post by b***@nowhere.co.uk
even caused a fork of Debian.
Yes, a failing one. That should tell you something.
I suppose thats why it already has 18 distros based on it.
Rainer Weikusat
2020-07-24 13:55:38 UTC
Permalink
Post by b***@nowhere.co.uk
Post by Nicolas George
Post by b***@nowhere.co.uk
even caused a fork of Debian.
Yes, a failing one. That should tell you something.
I suppose thats why it already has 18 distros based on it.
It's also running my main work computer because I'm kind-of old
fashionend in this respect: I want something which works and enables me
to get useful stuff done without consuming and absurd amount of my time
with itself. I don't care about buzzword-compliance and fancyness :-).

NB: To block one of the usual non-arguments. I'm also involved in the
care and feeding of systemd. But that's because I get paid to do so, not
because I think it's particularly useful.
b***@nowhere.co.uk
2020-07-22 07:55:25 UTC
Permalink
On 21 Jul 2020 15:46:37 GMT
Post by Nicolas George
Post by b***@nowhere.co.uk
You have a strange definition of conspiracy theories.
Who wrote "There have been rumours that it was a deliberate ploy by"?
Its hardly a conspiracy theory to assume IBM wants control of as much of linux
as it can get is it? Its hardly out of character for that company.
Nicolas George
2020-07-22 07:58:34 UTC
Permalink
Post by b***@nowhere.co.uk
Its hardly a conspiracy theory to assume IBM wants control of as much of linux
as it can get is it? Its hardly out of character for that company.
Conspiracy theorists rarely admit they are. Enjoy your bubble.
b***@nowhere.co.uk
2020-07-22 08:08:22 UTC
Permalink
On 22 Jul 2020 07:58:34 GMT
Post by Nicolas George
Post by b***@nowhere.co.uk
Its hardly a conspiracy theory to assume IBM wants control of as much of
linux
Post by b***@nowhere.co.uk
as it can get is it? Its hardly out of character for that company.
Conspiracy theorists rarely admit they are. Enjoy your bubble.
Ok, sure, IBM only have the best interests of the Linux community at heart.
They paid $34 BILLION for RedHat just so they could be charitable for everyone
who doesn't like Windows much, nothing to do with wanting to control a market.

In other news Santa really does exist and Elvis has a job as one of his elves.
Rainer Weikusat
2020-07-21 14:25:14 UTC
Permalink
[...]
Post by Boris Dorestand
Post by Rainer Weikusat
Apart from that, the port to a BSD-kernel is dead-in-water: Since Debian
has set forth towards becoming a rebranded version of Fedora (ie,
another distribution of systemd slowly converging towards "the" systemd
distribution), the "default installation" can no longer run on top of
anything but (certain versions of) Linux because systemd needs that.
That sucks. Speaking of which, how do you explain the popularity of
systemd? Why does ``everyone'' love it?
It's an improvement over the 20 years of largely function-free
historical[*] mess accumulated in distribution init scripts. For most
parts, it's absolutely conventional (ie, uses topological sorting based
on declared dependencies and actually even still used pid files). It's
written based on the wrong assumption that "systems" would be static
after startup despite half-assedly trying to work around the problem
that they're not (by automatically restarting processes) a lot of people
strongly believe in (this belief forming the base for nonsense ideas like
applying the make-algorithm to system booting or "readiness
notifications"[**]). It's pleasingly complicated. It was marketed to all
target groups in a skillful way. And it got a lot of underhand political
support by people willing to twist and bend[***] rules in order to
achieve a certain end.

It also seems to be less chaotic and buggy than upstart[****].

[*] I still unfondly remember a comment in an old SuSE init script
complaining about the need to work around bugs in another init
script. Apparently, the idea to fix the other script instead never came
to anybody's mind.

[**] Of what use is a "readiness notification" when the service could be
come unready a microseond later?

[***] The Debian committee responsible for making technical steering
decisions is (or at least used to be) composed of an even number of
people. In order to resolve the problem of draws, the head of the
committee is allowed to vote twice in order to break one. At the time
when this was current, the then-head of the committee (a systemd
supporter) intentionally called for a vote between upstart and systemd
he knew would end in a draw and used his second vote to decide the issue
in favour of his preferred option.

[****] upstart supports joining conditions with and and or but the
documentation cautions against using these features as they don't work
in the way people would expect them to work: Conditions are events and
not states, hence, it's (for instance) not possible that a dynamic
network initialization service (ie, one which configures interfaces as
they appear or disappear) waits until all local filesystems have been
mounted: "Local filesystems mounted" will be true once for the first
interface which is detected and then never again.
b***@nowhere.co.uk
2020-07-21 15:10:35 UTC
Permalink
On Tue, 21 Jul 2020 15:25:14 +0100
Post by Rainer Weikusat
[...]
Post by Boris Dorestand
Post by Rainer Weikusat
Apart from that, the port to a BSD-kernel is dead-in-water: Since Debian
has set forth towards becoming a rebranded version of Fedora (ie,
another distribution of systemd slowly converging towards "the" systemd
distribution), the "default installation" can no longer run on top of
anything but (certain versions of) Linux because systemd needs that.
That sucks. Speaking of which, how do you explain the popularity of
systemd? Why does ``everyone'' love it?
It's an improvement over the 20 years of largely function-free
historical[*] mess accumulated in distribution init scripts. For most
Since when are shell scripts function free?

A lot of the new inits seem to shoehorn logic functionality into their config
scripts using some convoluted new syntax made up on the back of a cigarette
packet when it was already available in the shell. I'm not arguing that init
is perfect , but using shell scripts in the boot process was and is a very good
idea and it seems to me the only reason a lot of the new inits got rid of it
was Not Invented Here syndrome.
Rainer Weikusat
2020-07-21 15:24:13 UTC
Permalink
Post by b***@nowhere.co.uk
Post by Rainer Weikusat
[...]
Post by Boris Dorestand
Post by Rainer Weikusat
Apart from that, the port to a BSD-kernel is dead-in-water: Since Debian
has set forth towards becoming a rebranded version of Fedora (ie,
another distribution of systemd slowly converging towards "the" systemd
distribution), the "default installation" can no longer run on top of
anything but (certain versions of) Linux because systemd needs that.
That sucks. Speaking of which, how do you explain the popularity of
systemd? Why does ``everyone'' love it?
It's an improvement over the 20 years of largely function-free
historical[*] mess accumulated in distribution init scripts. For most
Since when are shell scripts function free?
A lot of the new inits seem to shoehorn logic functionality into their config
scripts using some convoluted new syntax made up on the back of a cigarette
packet when it was already available in the shell. I'm not arguing that init
is perfect , but using shell scripts in the boot process was and is a very good
idea and it seems to me the only reason a lot of the new inits got rid of it
was Not Invented Here syndrome.
I think using shell scripts for this is a good idea. But the way this
used to be implemented in distributions using sysv init was nevertheless
horrible.

Eg - I'm sorry of this seems silly but it was an actual
example - someone pointed out that systemd could start sendmail much
more quickly than a certain, existing init script. While this was
observably true, the reason for this was not "secret systemd magic" but
that the script hard-coded a multisecond sleep to wait for a certain
filesystem name becoming available. After replacing that (as "quick
hack") with something based on inotifywait, the script was faster
than systemd.
Scott Lurndal
2020-07-20 16:29:08 UTC
Permalink
Post by Keith Thompson
Post by b***@nowhere.co.uk
On Sat, 18 Jul 2020 17:01:35 -0700 (PDT)
Post by G G
would there be any value in rewriting unix from scratch in 2020?
GNU tried that with Plan 9. It wasn't a stellar success. Not sure if its even
still being developed.
Plan 9 was from Bell Labs, not GNU.
Indeed. The GNU version was called Hurd, IIRC.
Keith Thompson
2020-07-20 18:18:35 UTC
Permalink
Post by Scott Lurndal
Post by Keith Thompson
Post by b***@nowhere.co.uk
On Sat, 18 Jul 2020 17:01:35 -0700 (PDT)
Post by G G
would there be any value in rewriting unix from scratch in 2020?
GNU tried that with Plan 9. It wasn't a stellar success. Not sure if its even
still being developed.
Plan 9 was from Bell Labs, not GNU.
Indeed. The GNU version was called Hurd, IIRC.
Hurd is a kernel. Plan 9 is an operating system.
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+***@gmail.com
Working, but not speaking, for Philips Healthcare
void Void(void) { Void(); } /* The recursive call of the void */
G G
2020-07-20 19:36:49 UTC
Permalink
Post by Keith Thompson
Hurd is a kernel. Plan 9 is an operating system.
Hurd runs on top of Mach microkernel. There is no active
work being done on Mach. Mach's web page says last updated
1994.

Unix has stood the text of time.
G G
2020-07-20 19:59:56 UTC
Permalink
Post by G G
Post by Keith Thompson
Hurd is a kernel. Plan 9 is an operating system.
Hurd runs on top of Mach microkernel. There is no active
work being done on Mach. Mach's web page says last updated
1994.
Unix has stood the test of time.
Lew Pitcher
2020-07-20 20:11:09 UTC
Permalink
Post by G G
Post by Keith Thompson
Hurd is a kernel. Plan 9 is an operating system.
Hurd runs on top of Mach microkernel.
Hurd runs on top of the GNU Mach microkernel, "It is maintained by the Hurd
developers for the GNU project and remains compatible with Mach 3.0."[1]
Post by G G
There is no active work being done on Mach.
Mach's web page says last updated 1994.
GNU Mach was last updated in 2016.[2]
Post by G G
Unix has stood the test of time.
[1] https://www.gnu.org/software/hurd/microkernel/mach/gnumach.html
[2] https://www.gnu.org/software/hurd/news/2016-12-18-releases.html
--
Lew Pitcher
"In Skills, We Trust"
Keith Thompson
2020-07-20 20:07:28 UTC
Permalink
Post by G G
Post by Keith Thompson
Hurd is a kernel. Plan 9 is an operating system.
Hurd runs on top of Mach microkernel. There is no active
work being done on Mach. Mach's web page says last updated
1994.
Unix has stood the text of time.
As I understand it, Hurd does not depend on the original Mach
implementation from CMU. It's based on it, but Hurd runs on
GNU Mach.

CMU Mach has not been updated in a long time, and Hurd development
is delayed, but there is no direct correlation between those facts.
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+***@gmail.com
Working, but not speaking, for Philips Healthcare
void Void(void) { Void(); } /* The recursive call of the void */
G G
2020-07-20 16:22:17 UTC
Permalink
Post by b***@nowhere.co.uk
Java? Good luck with that. How will it control the hardware when it has little
to no to-the-metal abilities and what will run its JVM if its running the OS?
There is no reason java can't produce machine
specific code is there? Java does run in it's VM,
but the language could produce processor specific
if so situated right? I mean running in the VM
is what makes it extremely portable, but the VMs
have to be specific, generically so, but still
specific to the processor... I thought.
Lew Pitcher
2020-07-20 16:42:38 UTC
Permalink
Post by G G
Post by b***@nowhere.co.uk
Java? Good luck with that. How will it control the hardware when it has
little to no to-the-metal abilities and what will run its JVM if its
running the OS?
There is no reason java can't produce machine
specific code is there? Java does run in it's VM,
but the language could produce processor specific
if so situated right? I mean running in the VM
is what makes it extremely portable, but the VMs
have to be specific, generically so, but still
specific to the processor... I thought.
Java is a Turing-complete language, and someone likely /could/ write an
operating system in Java, despite the complexities of a bare-metal JVM, or
compilation of Java to bare-metal code. But, then again, COBOL is also
Turing complete, and you /could/ write an OS in it as well. But, why would
you?

The overriding questions are:
Do the benefits of the chosen implementation language justify the
effort of writing such an OS in it?
and
Would such an operating system perform at acceptable levels?

If "no" to either question, then a Java-implemented OS would not likely have
a wide audience.
--
Lew Pitcher
"In Skills, We Trust"
G G
2020-07-20 17:03:58 UTC
Permalink
Post by Lew Pitcher
Post by G G
Post by b***@nowhere.co.uk
Java? Good luck with that. How will it control the hardware when it has
little to no to-the-metal abilities and what will run its JVM if its
running the OS?
There is no reason java can't produce machine
specific code is there? Java does run in it's VM,
but the language could produce processor specific
if so situated right? I mean running in the VM
is what makes it extremely portable, but the VMs
have to be specific, generically so, but still
specific to the processor... I thought.
Java is a Turing-complete language, and someone likely /could/ write an
operating system in Java, despite the complexities of a bare-metal JVM, or
compilation of Java to bare-metal code. But, then again, COBOL is also
Turing complete, and you /could/ write an OS in it as well. But, why would
you?
well, cobol is wordy as all get out to start with. :-), lol.

https://www.jopdesign.com/doc/hwobj.pdf

why Java? some programmers love it. They are well versed in it.
It is under consistent and current development and it may
do things in a way that is more intuitive than
another language. Wait, wait, I'm just answering the why
question, not demanding or even recommending Java base
on features in Java language. Who knows what could come
about in the hands of some really talented Java engineers.
Rainer Weikusat
2020-07-20 17:56:38 UTC
Permalink
Post by G G
Post by Lew Pitcher
Post by G G
Post by b***@nowhere.co.uk
Java? Good luck with that. How will it control the hardware when it has
little to no to-the-metal abilities and what will run its JVM if its
running the OS?
There is no reason java can't produce machine
specific code is there? Java does run in it's VM,
but the language could produce processor specific
if so situated right? I mean running in the VM
is what makes it extremely portable, but the VMs
have to be specific, generically so, but still
specific to the processor... I thought.
Java is a Turing-complete language, and someone likely /could/ write an
operating system in Java, despite the complexities of a bare-metal JVM, or
compilation of Java to bare-metal code. But, then again, COBOL is also
Turing complete, and you /could/ write an OS in it as well. But, why would
you?
well, cobol is wordy as all get out to start with. :-), lol.
https://www.jopdesign.com/doc/hwobj.pdf
why Java? some programmers love it. They are well versed in it.
It is under consistent and current development and it may
do things in a way that is more intuitive than
another language.
[...]
Post by G G
Who knows what could come about in the hands of some really talented
Java engineers.
Java developers - at least the minority who can be regarded as
intelligent beings and not just copy-and-paste-automatons :-> - usually
evolve into language designers ...
G G
2020-07-20 18:18:05 UTC
Permalink
Post by Rainer Weikusat
Java developers - at least the minority who can be regarded as
intelligent beings and not just copy-and-paste-automatons :-> - usually
evolve into language designers ...
smh, lol

now, now, lol
Scott Lurndal
2020-07-20 17:11:23 UTC
Permalink
Post by Lew Pitcher
Post by G G
Post by b***@nowhere.co.uk
Java? Good luck with that. How will it control the hardware when it has
little to no to-the-metal abilities and what will run its JVM if its
running the OS?
There is no reason java can't produce machine
specific code is there? Java does run in it's VM,
but the language could produce processor specific
if so situated right? I mean running in the VM
is what makes it extremely portable, but the VMs
have to be specific, generically so, but still
specific to the processor... I thought.
GCJ will compile java. There's a runtime component to JAVA
(e.g. garbage collection) that becomes more complicated in a
compiled environment.

Of course, the entire point of java was "write once, run everywhere",
which is predicated on the existence of a JVM "everywhere".
Post by Lew Pitcher
Java is a Turing-complete language, and someone likely /could/ write an
operating system in Java, despite the complexities of a bare-metal JVM, or
compilation of Java to bare-metal code. But, then again, COBOL is also
Turing complete, and you /could/ write an OS in it as well. But, why would
you?
There was a company in the mid 2000s which built a JVM processor (which
treated the JVM directives as an "instruction set"). Required an
intel-coprocessor to handle the I/O and managment side of things.
Commercially unsuccessful and the company failed.

Likewise, ARM included the Jazelle instruction set in ARMv7 to
accelerate the JVM. They subsequently removed it from ARMv8.
Lew Pitcher
2020-07-21 16:39:21 UTC
Permalink
Post by Scott Lurndal
Post by Lew Pitcher
Post by G G
Post by b***@nowhere.co.uk
Java? Good luck with that. How will it control the hardware when it has
little to no to-the-metal abilities and what will run its JVM if its
running the OS?
There is no reason java can't produce machine
specific code is there? Java does run in it's VM,
but the language could produce processor specific
if so situated right? I mean running in the VM
is what makes it extremely portable, but the VMs
have to be specific, generically so, but still
specific to the processor... I thought.
GCJ will compile java. There's a runtime component to JAVA
(e.g. garbage collection) that becomes more complicated in a
compiled environment.
Of course, the entire point of java was "write once, run everywhere",
which is predicated on the existence of a JVM "everywhere".
Post by Lew Pitcher
Java is a Turing-complete language, and someone likely /could/ write an
operating system in Java, despite the complexities of a bare-metal JVM, or
compilation of Java to bare-metal code. But, then again, COBOL is also
Turing complete, and you /could/ write an OS in it as well. But, why would
you?
There was a company in the mid 2000s which built a JVM processor (which
treated the JVM directives as an "instruction set"). Required an
intel-coprocessor to handle the I/O and managment side of things.
Commercially unsuccessful and the company failed.
Likewise, ARM included the Jazelle instruction set in ARMv7 to
accelerate the JVM. They subsequently removed it from ARMv8.
I would venture to say that there are a number of "operating sytems" (for
limited values of "operating system") that have been written in Java.
Embedded device manufacturers, especially those in the entertainment-
delivery business, tend to imbed Java deep into their devices.
--
Lew Pitcher
"In Skills, We Trust"
luser droog
2020-07-21 17:01:40 UTC
Permalink
Post by Scott Lurndal
Post by G G
Post by b***@nowhere.co.uk
Java? Good luck with that. How will it control the hardware when it has
little to no to-the-metal abilities and what will run its JVM if its
running the OS?
There is no reason java can't produce machine
specific code is there? Java does run in it's VM,
but the language could produce processor specific
if so situated right? I mean running in the VM
is what makes it extremely portable, but the VMs
have to be specific, generically so, but still
specific to the processor... I thought.
GCJ will compile java. There's a runtime component to JAVA
(e.g. garbage collection) that becomes more complicated in a
compiled environment.
GCJ has been deprecated and removed from current GCC.
G G
2020-07-20 16:04:03 UTC
Permalink
Is there a free Open Specification online? One that
does not require some much information in order to
read the spec. I found The Open Group,
www.opengroup.org, but good grief to read the spec
seems to require a login and to get an account they
want a lot of information. Are there licensing fees
to produce a Unix? Or fees just for testing comformance
to the standard? or fees for calling the OS Unix?
Scott Lurndal
2020-07-20 16:30:22 UTC
Permalink
Post by G G
Is there a free Open Specification online?
https://pubs.opengroup.org/onlinepubs/9699919799/
Geoff Clare
2020-07-21 12:46:26 UTC
Permalink
Are there licensing fees to produce a Unix?
The standard is open for anyone to use. If a system is developed
which behaves as per the standard but is not claimed to be a UNIX
system, then there would be no fee.

However, if the developer wants to say the system is a UNIX system,
then there would be a licensing fee for the UNIX trademark.
Or fees just for testing conformance to the standard?
Yes.
or fees for calling the OS Unix?
See first answer.
--
Geoff Clare <***@gclare.org.uk>
James K. Lowden
2020-07-20 19:31:20 UTC
Permalink
On Sat, 18 Jul 2020 17:01:35 -0700 (PDT)
Post by G G
It would be so cool to see versions written in all Rust, all C, all C+
+, all Swift, all Ada, all Java, using the latest defs of the
respective languages. yes and all new libraries.
I'm not sure I know what you mean. Do you "the OS" would be written in
many languages? Or do you mean the OS services would be accessible to
many languages through their standard libraries? Because the latter
already exists....
Post by G G
Can the holes in Security be designed out? hmm?
Are you suggesting security in general is a function of the
programming language used?

--jkl
G G
2020-07-20 21:24:53 UTC
Permalink
Thanks Lew, Thanks Keith
Miquel van Smoorenburg
2020-07-23 12:52:09 UTC
Permalink
Post by G G
would there be any value in rewriting unix from scratch in 2020?
completely leaving behind the ideal of back compatibility.
I wonder how clean that would. I wonder too if it would result in
small source.
It would be so cool to see versions written in all Rust, all C, all C++,
all Swift, all Ada, all Java, using the latest defs of the respective
languages. yes and all new libraries.
You can see that now. People are still writing new OSes.

Redox, https://www.redox-os.org/

Has a micro-kernel. Completely written in Rust. Based on plan-9 ideas,
but instead of "everything is a file", "everything is a URL".
Has a posix campatibility API (relibc).

Fuchsia, https://fuchsia.dev/fuchsia-src/concepts

A new OS by Google, possible for Android/ChromeOS. Kernel is in C/C++.
Userland code is in C/C++/Dart/Go/Rust/Java.

There are more.

Mike.
b***@nowhere.co.uk
2020-07-23 14:27:05 UTC
Permalink
On 23 Jul 2020 12:52:09 GMT
Post by Miquel van Smoorenburg
Redox, https://www.redox-os.org/
Has a micro-kernel. Completely written in Rust. Based on plan-9 ideas,
but instead of "everything is a file", "everything is a URL".
Sounds like Project Xanadu by Ted Nelson. If a heavyweight like Nelson
can't get traction on that concept in 50 years of trying then it ain't
gonna happen.
G G
2020-07-24 09:10:54 UTC
Permalink
Post by Miquel van Smoorenburg
Post by G G
would there be any value in rewriting unix from scratch in 2020?
completely leaving behind the ideal of back compatibility.
I wonder how clean that would. I wonder too if it would result in
small source.
It would be so cool to see versions written in all Rust, all C, all C++,
all Swift, all Ada, all Java, using the latest defs of the respective
languages. yes and all new libraries.
You can see that now. People are still writing new OSes.
Redox, https://www.redox-os.org/
Has a micro-kernel. Completely written in Rust. Based on plan-9 ideas,
but instead of "everything is a file", "everything is a URL".
Has a posix campatibility API (relibc).
Miguel :

So what do you think of the OS, redox-os?

And how odd or beneficial do you find "everything is a URL"?

did they offer a rationale for everything is a url vs. a file?
G G
2020-07-24 13:01:27 UTC
Permalink
Post by G G
Post by Miquel van Smoorenburg
Post by G G
would there be any value in rewriting unix from scratch in 2020?
completely leaving behind the ideal of back compatibility.
I wonder how clean that would. I wonder too if it would result in
small source.
It would be so cool to see versions written in all Rust, all C, all C++,
all Swift, all Ada, all Java, using the latest defs of the respective
languages. yes and all new libraries.
You can see that now. People are still writing new OSes.
Redox, https://www.redox-os.org/
Has a micro-kernel. Completely written in Rust. Based on plan-9 ideas,
but instead of "everything is a file", "everything is a URL".
Has a posix campatibility API (relibc).
So what do you think of the OS, redox-os?
And how odd or beneficial do you find "everything is a URL"
did they offer a rationale for everything is a url vs. a file?
and looking at their source code it must load really fast?
Miquel van Smoorenburg
2020-07-25 01:13:29 UTC
Permalink
Post by G G
Post by Miquel van Smoorenburg
Post by G G
would there be any value in rewriting unix from scratch in 2020?
completely leaving behind the ideal of back compatibility.
I wonder how clean that would. I wonder too if it would result in
small source.
It would be so cool to see versions written in all Rust, all C, all C++,
all Swift, all Ada, all Java, using the latest defs of the respective
languages. yes and all new libraries.
You can see that now. People are still writing new OSes.
Redox, https://www.redox-os.org/
Has a micro-kernel. Completely written in Rust. Based on plan-9 ideas,
but instead of "everything is a file", "everything is a URL".
Has a posix campatibility API (relibc).
So what do you think of the OS, redox-os?
I think it's interesting. Chances are it's not going to be the next
big thing, but then again, in 1991, could anyone have predicted what
the next big thing was going to be - Hurd, BSD, windows, solaris,
minix, freax (the original name linus gave linux, yes really).
Post by G G
And how odd or beneficial do you find "everything is a URL"?
It's not radical. MSDOS had this :) A: vs C: vs CON: . It's just
namespaces so you don't have to special case path names
(like /dev, /proc, /sys etc).
Post by G G
did they offer a rationale for everything is a url vs. a file?
See their website.

Mike.
Rainer Weikusat
2020-07-25 13:15:21 UTC
Permalink
[...]
Post by Miquel van Smoorenburg
Post by G G
And how odd or beneficial do you find "everything is a URL"?
It's not radical. MSDOS had this :) A: vs C: vs CON: . It's just
namespaces so you don't have to special case path names
(like /dev, /proc, /sys etc).
There's nothting special about /proc and /sys, they're just
conventionally used as mount points for certain filesystems (/dev in
principle as well). And MS-DOG didn't support "namespaces for pathnames"
but wasn't sophisticated enough to provide any kind of general,
abstract filesystem namespace. Hence, a physcial device ID had to be
prefixed to absolute paths.
G G
2020-07-25 04:14:02 UTC
Permalink
" Complex algorithms are used only if their complexity can be local-
ized. "

Taken from :https://cscie28.dce.harvard.edu/~lib215/reference/history/unix-thompson.pdf (introduction)

what would that mean?
Siri Cruise
2020-07-25 07:21:39 UTC
Permalink
In article
Post by G G
" Complex algorithms are used only if their complexity can be local-
ized. "
Taken from
:https://cscie28.dce.harvard.edu/~lib215/reference/history/unix-thompson.pdf
(introduction)
what would that mean?
Divide the program into pieces that can be understood on their
own passing minimal information between the pieces.

For example the file descriptor is an important concept. Ideally
you only need to pass the file desciptor alone around the kernel
and not everything associated with the descriptor like socket
buffers, inode disk maps, serial port signals, etc. You can pass
the fd to the network module to functions to operate on the
socket, and to return only such network information that is
relevant to rest of the kernel. You wouldn't drag around a block
about the socket to the disc i/o, especially with a translucent
tag that anyone can modify.
--
:-<> Siri Seal of Disavowal #000-001. Disavowed. Denied. Deleted. @
'I desire mercy, not sacrifice.' /|\
The first law of discordiamism: The more energy This post / \
to make order is nore energy made into entropy. insults Islam. Mohammed
b***@nowhere.co.uk
2020-07-25 14:16:58 UTC
Permalink
On Sat, 25 Jul 2020 00:21:39 -0700
Post by Siri Cruise
For example the file descriptor is an important concept. Ideally
you only need to pass the file desciptor alone around the kernel
and not everything associated with the descriptor like socket
buffers, inode disk maps, serial port signals, etc. You can pass
Having a simple integer descriptor that can be a handle for a lot
of different system object types and be used in many functions
regardless of the object type was a radical concept back in the 70s
and even today Windows still doesn't manage it at the win32 level.
Eg Windows sockets are still stored in a special handle type, you
can't use file descriptors, making functions such as select()
virtually useless.
G G
2020-07-27 05:26:53 UTC
Permalink
curious as to whether:

creating a process in unix takes longer than in windows?
Kaz Kylheku
2020-07-27 07:48:56 UTC
Permalink
Post by G G
creating a process in unix takes longer than in windows?
There aren't enough different implementations of Windows to make
interesting empirical results.

First of all, Windows does not offer fork (in its regular Win32
subsystems). So the following operation is very expensive:

Creating a process which is a replica of the invoking "parent"
process, executing the same point in the same function.

The Cygwin project implements this fork operation in plain Win32.
It's neither fast nor pretty.

If we have the following requirement:

Creating a process which runs a new process image specified
via a nominated executable program in the filesystem space,
which receives a new address space.

then for that, Windows has the CreateProcess family of functions,
whereas traditional unix has fork combined with an exec function.

CreateProcess is, overall, saddled with fewer expensive requirements;
since there is no fork, there is no address space copying, and since
there is no exec-in-place, there is no clean-up of an existing space.

Unix systems have historically optimized fork with lazy semantics
to make it faster for all its intended uses, including the exec
case, where laziness wins by avoiding up-front duplication of an address
space only to then replace it. Further optimized variants of fork also
appeared (vfork) which so optimized toward the exec case as to be
incorrect if used for any other purpose.

exec in Unix has to clean up various parent resources. An
exec-optimized fork can help here by minimizing the clean-up.
If the exec is done out of a vfork, it knows not to clean up
most of the resources, which would be wrong, since they are still shared
with the parent.

Furthermore, certain resources are passed to the new process image,
such as signal hangling dispositions (which signals are SIG_IGN or
SIG_DFL) and open file descriptors not marked close-on-exec.
An exec-optimized fork cannot do much here.

None of this is an issue for CreateProcess which makes a new address
space, not requiring any clean-up. There is a BOOL flag for whether open
handles are inherited by the new process, which we would set to FALSE
if we were benchmarking process creation. Signals don't exist.

Complicating the comparison is that POSIX now has a function called
posix_spawn, which in many ways resembles CreateProcess. It creates
a new process without the involvement of fork semantics, over a
specified executable program, and has arguments for controlling
inheritance of resources.

If we were benchmarking process creation in a modern Unix implementation
against Win32, of course it would behoove us to use posix_spawn.
--
TXR Programming Lanuage: http://nongnu.org/txr
Music DIY Mailing List: http://www.kylheku.com/diy
ADA MP-1 Mailing List: http://www.kylheku.com/mp1
b***@nowhere.co.uk
2020-07-27 08:21:49 UTC
Permalink
On Mon, 27 Jul 2020 07:48:56 +0000 (UTC)
Post by Kaz Kylheku
Post by G G
creating a process in unix takes longer than in windows?
There aren't enough different implementations of Windows to make
interesting empirical results.
First of all, Windows does not offer fork (in its regular Win32
I really don't understand why MS doesn't offer fork() as a standard
API function since their kernels are quite capable of doing it.
The lousy process control in Windows is one reason why multithreading
became so popular on that platform and that spilled over into the
unix world where arguably the advantages of multi threading over
multi process are equivocal.
Post by Kaz Kylheku
space only to then replace it. Further optimized variants of fork also
appeared (vfork) which so optimized toward the exec case as to be
incorrect if used for any other purpose.
The problem with vfork() is that the parent is suspended until the
child exits or calls exec*().
Post by Kaz Kylheku
posix_spawn, which in many ways resembles CreateProcess. It creates
I hadn't heard of posix_spawn, will have to check it out.
Rainer Weikusat
2020-07-27 14:12:38 UTC
Permalink
[...]
Post by b***@nowhere.co.uk
Post by Kaz Kylheku
posix_spawn, which in many ways resembles CreateProcess. It creates
I hadn't heard of posix_spawn, will have to check it out.
The main raison d'etre of posix_spawn is as a workaound for the fact
that running Java (and - to a degree - als running C++) programs usually
create insanely huge address spaces and hence, would make a fork/ exec
disproportionally expensive for such programs.

It's also a psychological band aid for people suffering from incurable
BTNHMDI![*]-disease.

[*] But That's Not How Microsoft Does It!
Scott Lurndal
2020-07-27 17:22:37 UTC
Permalink
Post by Rainer Weikusat
[...]
Post by b***@nowhere.co.uk
Post by Kaz Kylheku
posix_spawn, which in many ways resembles CreateProcess. It creates
I hadn't heard of posix_spawn, will have to check it out.
The main raison d'etre of posix_spawn is as a workaound for the fact
that running Java (and - to a degree - als running C++) programs usually
create insanely huge address spaces and hence, would make a fork/ exec
disproportionally expensive for such programs.
Which problem was obviated 20 years ago by using COW page tables
in the newly created child. If the child calls exec immediately, there
is no adverse impact on allocated memory when forking a process with a
large address space (other than the very small amount of time
setting up the new page tables). Even after over two decades,
posix_spawn is not widely used (linux clone is probably more widely
used).
Rainer Weikusat
2020-07-27 20:35:01 UTC
Permalink
Post by Scott Lurndal
Post by Rainer Weikusat
[...]
Post by b***@nowhere.co.uk
Post by Kaz Kylheku
posix_spawn, which in many ways resembles CreateProcess. It creates
I hadn't heard of posix_spawn, will have to check it out.
The main raison d'etre of posix_spawn is as a workaound for the fact
that running Java (and - to a degree - als running C++) programs usually
create insanely huge address spaces and hence, would make a fork/ exec
disproportionally expensive for such programs.
Which problem was obviated 20 years ago by using COW page tables
in the newly created child. If the child calls exec immediately, there
is no adverse impact on allocated memory when forking a process with a
large address space (other than the very small amount of time
setting up the new page tables).
Some people believe that's already "huge enough", cf

http://www.netbsd.org/docs/kernel/vfork.html

and then, there's no problem with this on Linux due to memory
overcommittment. But that's not true on other systems, most notably the
BSDs and Solaris, which have to reserve enough swap space for all of the
inherited address space to be copied eventually.
Rainer Weikusat
2020-07-27 14:15:10 UTC
Permalink
[...]
Post by b***@nowhere.co.uk
Post by Kaz Kylheku
posix_spawn, which in many ways resembles CreateProcess. It creates
I hadn't heard of posix_spawn, will have to check it out.
The main raison d'etre of posix_spawn is as a workaound for the fact
that running Java (and - to a degree - als running C++) programs usually
create insanely huge address spaces and hence, fork/ exec being
disproportionally expensive for such programs.

It's also a psychological band aid for people suffering from incurable
BTNHMDI![*]-disease.

[*] But That's Not How Microsoft Does It!
Keith Thompson
2020-07-27 16:11:40 UTC
Permalink
Post by Rainer Weikusat
[...]
Post by b***@nowhere.co.uk
Post by Kaz Kylheku
posix_spawn, which in many ways resembles CreateProcess. It creates
I hadn't heard of posix_spawn, will have to check it out.
The main raison d'etre of posix_spawn is as a workaound for the fact
that running Java (and - to a degree - als running C++) programs usually
create insanely huge address spaces and hence, fork/ exec being
disproportionally expensive for such programs.
Is fork/exec really that expensive, given that copy-on-write means the
huge address space doesn't actually have to be duplicated? Is fork
expensive, or is it *perceived* as expensive?

If Wikipedia is correct (yes, *if*), posix_spawn was intended for
(embedded) systems where fork isn't available.
Post by Rainer Weikusat
It's also a psychological band aid for people suffering from incurable
BTNHMDI![*]-disease.
[*] But That's Not How Microsoft Does It!
--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+***@gmail.com
Working, but not speaking, for Philips Healthcare
void Void(void) { Void(); } /* The recursive call of the void */
Kaz Kylheku
2020-07-27 17:44:38 UTC
Permalink
Post by Keith Thompson
Post by Rainer Weikusat
[...]
Post by b***@nowhere.co.uk
Post by Kaz Kylheku
posix_spawn, which in many ways resembles CreateProcess. It creates
I hadn't heard of posix_spawn, will have to check it out.
The main raison d'etre of posix_spawn is as a workaound for the fact
that running Java (and - to a degree - als running C++) programs usually
create insanely huge address spaces and hence, fork/ exec being
disproportionally expensive for such programs.
Is fork/exec really that expensive, given that copy-on-write means the
huge address space doesn't actually have to be duplicated? Is fork
expensive, or is it *perceived* as expensive?
Sharing the address space with a new process for read access with
copy-on-write involves some work proportional to the size of the address
space. My recollection of this stuff isn't great, but basically the
child process has to have enough of the page table structure in place so
that it can take the page faults to populate the rest of it. Also, the
elements of the page directory structure have to be tweaked to enable
copy-on-write.
Post by Keith Thompson
If Wikipedia is correct (yes, *if*), posix_spawn was intended for
(embedded) systems where fork isn't available.
That's more or less what the rationale notes say right in POSIX under
the description of posix_spawn.
Scott Lurndal
2020-07-27 18:54:22 UTC
Permalink
Post by Kaz Kylheku
Post by Keith Thompson
Post by Rainer Weikusat
[...]
Post by b***@nowhere.co.uk
Post by Kaz Kylheku
posix_spawn, which in many ways resembles CreateProcess. It creates
I hadn't heard of posix_spawn, will have to check it out.
The main raison d'etre of posix_spawn is as a workaound for the fact
that running Java (and - to a degree - als running C++) programs usually
create insanely huge address spaces and hence, fork/ exec being
disproportionally expensive for such programs.
Is fork/exec really that expensive, given that copy-on-write means the
huge address space doesn't actually have to be duplicated? Is fork
expensive, or is it *perceived* as expensive?
Sharing the address space with a new process for read access with
copy-on-write involves some work proportional to the size of the address
space. My recollection of this stuff isn't great, but basically the
child process has to have enough of the page table structure in place so
that it can take the page faults to populate the rest of it. Also, the
elements of the page directory structure have to be tweaked to enable
copy-on-write.
Post by Keith Thompson
If Wikipedia is correct (yes, *if*), posix_spawn was intended for
(embedded) systems where fork isn't available.
That's more or less what the rationale notes say right in POSIX under
the description of posix_spawn.
To be fair, at the time we were also dealing with fork v. vfork and
posix_spawn was designed with that in mind. Moreover, we added a
bunch of stuff to posix_spawn to eliminate many of the system calls that
happened between fork and exec (such as marking file descriptors close-on-exec,
setting schedular group/policy, signal masks, process group, dup2, etc).

Therd are something like 22 different posix_spawn related interfaces.
b***@nowhere.co.uk
2020-07-28 07:37:31 UTC
Permalink
On Mon, 27 Jul 2020 18:54:22 GMT
Post by Scott Lurndal
Post by Kaz Kylheku
That's more or less what the rationale notes say right in POSIX under
the description of posix_spawn.
To be fair, at the time we were also dealing with fork v. vfork and
posix_spawn was designed with that in mind. Moreover, we added a
bunch of stuff to posix_spawn to eliminate many of the system calls that
happened between fork and exec (such as marking file descriptors close-on-exec,
If you were involved in posix_spawn I'd be interested in knowing why you
decided not to set errno but return an error code instead. There's nothing
wrong with doing that but it does rather break convention and caught me out
when I wrote a small test program with it yesterday.
Scott Lurndal
2020-07-28 16:04:13 UTC
Permalink
Post by b***@nowhere.co.uk
On Mon, 27 Jul 2020 18:54:22 GMT
Post by Scott Lurndal
Post by Kaz Kylheku
That's more or less what the rationale notes say right in POSIX under
the description of posix_spawn.
To be fair, at the time we were also dealing with fork v. vfork and
posix_spawn was designed with that in mind. Moreover, we added a
bunch of stuff to posix_spawn to eliminate many of the system calls that
happened between fork and exec (such as marking file descriptors close-on-exec,
If you were involved in posix_spawn I'd be interested in knowing why you
decided not to set errno but return an error code instead. There's nothing
wrong with doing that but it does rather break convention and caught me out
when I wrote a small test program with it yesterday.
That was a result of the POSIX 1003.4 changes (that added pthreads).

Using 'errno' in threaded applications is problematic since it was,
at the time, a process global variable. So the convention for all
system calls added with (and post) 1003.4 is to return the error code
rather than setting a global variable as a side-effect.

Meanwhile, errno was subsequently defined as a thread-specific variable
to make it safe for multithreaded code to issue legacy system calls without
overt synchronization (albeit with a potential performance impact due to
the implementation defined thread-local access mechanisms).
Rainer Weikusat
2020-07-27 20:04:55 UTC
Permalink
Post by Keith Thompson
Post by Rainer Weikusat
[...]
Post by b***@nowhere.co.uk
Post by Kaz Kylheku
posix_spawn, which in many ways resembles CreateProcess. It creates
I hadn't heard of posix_spawn, will have to check it out.
The main raison d'etre of posix_spawn is as a workaound for the fact
that running Java (and - to a degree - als running C++) programs usually
create insanely huge address spaces and hence, fork/ exec being
disproportionally expensive for such programs.
Is fork/exec really that expensive, given that copy-on-write means the
huge address space doesn't actually have to be duplicated? Is fork
expensive, or is it *perceived* as expensive?
This depends on the implementation: On Linux, it isn't very expensive,
not even for a huge process, as only the MMU management structures need
to be duplicated (for the address space).

But that's different for systems implementing UNIX V7 semantics for
fork, ie, only allow the fork to succeed if it can be guaranteed that
the forked process will be able to write to every page of its address
space by reserving a sufficient amount of "swap space" to back it.
James K. Lowden
2020-07-28 14:46:19 UTC
Permalink
On Mon, 27 Jul 2020 21:04:55 +0100
Post by Rainer Weikusat
Post by Keith Thompson
Is fork/exec really that expensive, given that copy-on-write means
the huge address space doesn't actually have to be duplicated? Is
fork expensive, or is it *perceived* as expensive?
This depends on the implementation: On Linux, it isn't very expensive,
[...]
Post by Rainer Weikusat
But that's different for systems implementing UNIX V7 semantics for
fork, ie, only allow the fork to succeed if it can be guaranteed that
the forked process will be able to write to every page of its address
space by reserving a sufficient amount of "swap space" to back it.
But is that so very expensive? There's no I/O, just a bookkeeping
adjustment to the kernel's filesystem image, a few tables in memory.

--jkl
Rainer Weikusat
2020-07-28 19:08:50 UTC
Permalink
Post by James K. Lowden
Post by Rainer Weikusat
Post by Keith Thompson
Is fork/exec really that expensive, given that copy-on-write means
the huge address space doesn't actually have to be duplicated? Is
fork expensive, or is it *perceived* as expensive?
This depends on the implementation: On Linux, it isn't very expensive,
[...]
Post by Rainer Weikusat
But that's different for systems implementing UNIX V7 semantics for
fork, ie, only allow the fork to succeed if it can be guaranteed that
the forked process will be able to write to every page of its address
space by reserving a sufficient amount of "swap space" to back it.
But is that so very expensive? There's no I/O, just a bookkeeping
adjustment to the kernel's filesystem image, a few tables in memory.
It requires that the system (at least for a short period) has enough
resources to support two processes of the virtual size of the original
one not sharing any originally writable memory mappings with each other.

That's reportedly "too expensive" for "unknown real-world scenarios"[*]

[*] This may be a somewhat dubious opinion, as in the "Why vfork" text I
linked to in another posting. An experiment is mentioned there to
justify vfork:

"It shaves several seconds off a build of libc on a 200MHz PPro."

Next to the kernel, the C library is probably the most complex
individual source tree of "some OS". Hence, compiling it with one of the
multipass code generators usually used on UNIX involves and
extraordinary large number of program executions. Hence, this is an
unrealistic example selected to maximize the effect of the vfork
implementation.

Further, "build of libc on 200Mhz PPro" will - at that time, certainly
have taken more then 10 minutes and probably, more than 20
minutes. Saving "several seconds" of 10 - 20 minutes isn't an
"optimization" people will even notice.
b***@nowhere.co.uk
2020-07-29 07:38:30 UTC
Permalink
On Tue, 28 Jul 2020 20:08:50 +0100
Post by Rainer Weikusat
[*] This may be a somewhat dubious opinion, as in the "Why vfork" text I
linked to in another posting. An experiment is mentioned there to
"It shaves several seconds off a build of libc on a 200MHz PPro."
I'm surprised using a multiprocess build on a single core chip such as
that didn't ADD to the build time because of all the extra swap time.
Kaz Kylheku
2020-07-29 18:02:35 UTC
Permalink
Post by b***@nowhere.co.uk
On Tue, 28 Jul 2020 20:08:50 +0100
Post by Rainer Weikusat
[*] This may be a somewhat dubious opinion, as in the "Why vfork" text I
linked to in another posting. An experiment is mentioned there to
"It shaves several seconds off a build of libc on a 200MHz PPro."
I'm surprised using a multiprocess build on a single core chip such as
that didn't ADD to the build time because of all the extra swap time.
Changing some fork calls to vfork is not going to cause extra swapping.
Rainer Weikusat
2020-07-29 20:14:48 UTC
Permalink
Post by b***@nowhere.co.uk
On Tue, 28 Jul 2020 20:08:50 +0100
Post by Rainer Weikusat
[*] This may be a somewhat dubious opinion, as in the "Why vfork" text I
linked to in another posting. An experiment is mentioned there to
"It shaves several seconds off a build of libc on a 200MHz PPro."
I'm surprised using a multiprocess build on a single core chip such as
that didn't ADD to the build time because of all the extra swap time.
Building something fairly large involves lots of I/O and multiple
processes can wait for disk I/O while others are actually working on
something.

Kaz Kylheku
2020-07-29 17:59:18 UTC
Permalink
Post by Rainer Weikusat
Post by James K. Lowden
Post by Rainer Weikusat
Post by Keith Thompson
Is fork/exec really that expensive, given that copy-on-write means
the huge address space doesn't actually have to be duplicated? Is
fork expensive, or is it *perceived* as expensive?
This depends on the implementation: On Linux, it isn't very expensive,
[...]
Post by Rainer Weikusat
But that's different for systems implementing UNIX V7 semantics for
fork, ie, only allow the fork to succeed if it can be guaranteed that
the forked process will be able to write to every page of its address
space by reserving a sufficient amount of "swap space" to back it.
But is that so very expensive? There's no I/O, just a bookkeeping
adjustment to the kernel's filesystem image, a few tables in memory.
It requires that the system (at least for a short period) has enough
resources to support two processes of the virtual size of the original
one not sharing any originally writable memory mappings with each other.
That's reportedly "too expensive" for "unknown real-world scenarios"[*]
[*] This may be a somewhat dubious opinion, as in the "Why vfork" text I
linked to in another posting. An experiment is mentioned there to
"It shaves several seconds off a build of libc on a 200MHz PPro."
World records get been beaten by smaller margins.
Post by Rainer Weikusat
Next to the kernel, the C library is probably the most complex
individual source tree of "some OS". Hence, compiling it with one of the
multipass code generators usually used on UNIX involves and
extraordinary large number of program executions. Hence, this is an
unrealistic example selected to maximize the effect of the vfork
implementation.
If we want to discover how much faster it is to spawn a process with
vfork, we in fact need a test case that does a lot of forking and
execing. "Most programs don't do that much forking and execing so who
cares" doesn't answer the question.

Would you disparage an improved, faster floating point divide
instruction using the fact that some regular expression searches haven't
gotten any faster?

From the point of view of someone optimizing a specific application,
the question "how often are we forking and execing" is a critical piece
of information for deciding whether to bother with some mechanism like
vfork or not.

From the point of view of someone providing the low-level building
blocks in a general-purpose system: machine instructions, library
functions, services, utility programs, ... the performance of everything
and anything potentially matters, pretty much as a default view. Almost
anything could end up used such that its performance is a big chunk of
the performance of whatever it is used in. You need a solid argument
that this isn't the case before you brush aside the need for some given
building block to perform well.
Post by Rainer Weikusat
Further, "build of libc on 200Mhz PPro" will - at that time, certainly
have taken more then 10 minutes and probably, more than 20
minutes. Saving "several seconds" of 10 - 20 minutes isn't an
"optimization" people will even notice.
Compiling actually doesn't seem like such a great test for isolating the
cost of launching a process, because the toolchain components can
actually do quite a bit of computation work.

I just rebuilt a program containing some 43 .o files. It took 40
seconds. According to strace, there are 10 successful execve operations
done per file, 8 clone operations, and 4 vforks. That's not a lot of
these operations for a span of 40 seconds.

The fact that seconds were shaved off a libc build on a 200 Mhz
Pentium Pro is pretty significant, since the number of vforks in the
build is probably only into the several thousand.
William Ahern
2020-07-29 05:26:24 UTC
Permalink
Post by Rainer Weikusat
Post by Keith Thompson
Post by Rainer Weikusat
[...]
Post by b***@nowhere.co.uk
Post by Kaz Kylheku
posix_spawn, which in many ways resembles CreateProcess. It creates
I hadn't heard of posix_spawn, will have to check it out.
The main raison d'etre of posix_spawn is as a workaound for the fact
that running Java (and - to a degree - als running C++) programs usually
create insanely huge address spaces and hence, fork/ exec being
disproportionally expensive for such programs.
Is fork/exec really that expensive, given that copy-on-write means the
huge address space doesn't actually have to be duplicated? Is fork
expensive, or is it *perceived* as expensive?
This depends on the implementation: On Linux, it isn't very expensive,
not even for a huge process, as only the MMU management structures need
to be duplicated (for the address space).
But that's different for systems implementing UNIX V7 semantics for
fork, ie, only allow the fork to succeed if it can be guaranteed that
the forked process will be able to write to every page of its address
space by reserving a sufficient amount of "swap space" to back it.
Which can nonetheless require significant work for large working sets:

https://about.gitlab.com/blog/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/
G G
2020-07-29 07:49:39 UTC
Permalink
Post by William Ahern
Post by Rainer Weikusat
Post by Keith Thompson
Post by Rainer Weikusat
[...]
Post by b***@nowhere.co.uk
Post by Kaz Kylheku
posix_spawn, which in many ways resembles CreateProcess. It creates
I hadn't heard of posix_spawn, will have to check it out.
The main raison d'etre of posix_spawn is as a workaound for the fact
that running Java (and - to a degree - als running C++) programs usually
create insanely huge address spaces and hence, fork/ exec being
disproportionally expensive for such programs.
Is fork/exec really that expensive, given that copy-on-write means the
huge address space doesn't actually have to be duplicated? Is fork
expensive, or is it *perceived* as expensive?
This depends on the implementation: On Linux, it isn't very expensive,
not even for a huge process, as only the MMU management structures need
to be duplicated (for the address space).
But that's different for systems implementing UNIX V7 semantics for
fork, ie, only allow the fork to succeed if it can be guaranteed that
the forked process will be able to write to every page of its address
space by reserving a sufficient amount of "swap space" to back it.
https://about.gitlab.com/blog/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/
WOW! that seems to be a pretty big find.
Rainer Weikusat
2020-07-29 14:38:29 UTC
Permalink
[...]
Post by G G
Post by William Ahern
Post by Rainer Weikusat
This depends on the implementation: On Linux, it isn't very expensive,
not even for a huge process, as only the MMU management structures need
to be duplicated (for the address space).
[...]
Post by G G
Post by William Ahern
https://about.gitlab.com/blog/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/
WOW! that seems to be a pretty big find.
"Performance impact of seriously contended global lock used to fight OS
semantics found to be awful?"

"Go breaks down under load as designed?"

or - alternatively -

"Despite hypothetical improvements, tracing GC still shit in practice?"

None of this is exactly new.
Rainer Weikusat
2020-07-29 14:52:53 UTC
Permalink
Post by Rainer Weikusat
[...]
Post by G G
Post by William Ahern
Post by Rainer Weikusat
This depends on the implementation: On Linux, it isn't very expensive,
not even for a huge process, as only the MMU management structures need
to be duplicated (for the address space).
[...]
Post by G G
Post by William Ahern
https://about.gitlab.com/blog/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/
WOW! that seems to be a pretty big find.
"Performance impact of seriously contended global lock used to fight OS
semantics found to be awful?"
"Go breaks down under load as designed?"
https://github.com/golang/go/blob/release-branch.go1.8/src/syscall/exec_unix.go#L17

Serializes all forks in a process and all operations
creating a file descriptor block forks so that "each newly created
descriptor can be marked close-on-exec with the forked child unmarking
those supposed to be inherited", this nonense even being employed when
native system features, eg, O_CLOEXEC and friends on Linux, could be
used to avoid it.

Not to mention that the forked child could just close file descriptors
not supposed to be inherited across an exec instead of removing CLOEXEC
flags from file descriptors supposed to be inherited. Any number of
child processes could do so in parallell without any non-kernel
synchronization.

Extremely clever people could even come up with the idea to use a
specialized program for that to avoid keeping the obese processes around
...
Kaz Kylheku
2020-07-29 18:13:57 UTC
Permalink
Post by Rainer Weikusat
Post by Rainer Weikusat
[...]
Post by G G
Post by William Ahern
Post by Rainer Weikusat
This depends on the implementation: On Linux, it isn't very expensive,
not even for a huge process, as only the MMU management structures need
to be duplicated (for the address space).
[...]
Post by G G
Post by William Ahern
https://about.gitlab.com/blog/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/
WOW! that seems to be a pretty big find.
"Performance impact of seriously contended global lock used to fight OS
semantics found to be awful?"
"Go breaks down under load as designed?"
https://github.com/golang/go/blob/release-branch.go1.8/src/syscall/exec_unix.go#L17
Serializes all forks in a process and all operations
creating a file descriptor block forks so that "each newly created
descriptor can be marked close-on-exec with the forked child unmarking
those supposed to be inherited", this nonense even being employed when
native system features, eg, O_CLOEXEC and friends on Linux, could be
used to avoid it.
Not to mention that the forked child could just close file descriptors
not supposed to be inherited across an exec instead of removing CLOEXEC
flags from file descriptors supposed to be inherited. Any number of
child processes could do so in parallell without any non-kernel
synchronization.
That does seem pretty stupid. If you control all the fork calls and
file descriptor manipulations, why mess around with flags that are
specifically dedicated to influencing fork operations not under your
control?

Fork calls not under your control will not observe the lock; they are
subject to the race between opening the descriptor and setting the flag
(that are addressed by Linux's extensions in open, and by dup3).

File descritors not created under your control won't get the
close-on-exec flags, and you have to care about those.

Fork under your control can take care of closing unwanted file
descriptors created by threads not under your control.
Rainer Weikusat
2020-07-29 18:15:03 UTC
Permalink
[...]
Post by William Ahern
Post by Rainer Weikusat
Post by Keith Thompson
Is fork/exec really that expensive, given that copy-on-write means the
huge address space doesn't actually have to be duplicated? Is fork
expensive, or is it *perceived* as expensive?
This depends on the implementation: On Linux, it isn't very expensive,
not even for a huge process, as only the MMU management structures need
to be duplicated (for the address space).
[...]
Post by William Ahern
https://about.gitlab.com/blog/2018/01/23/how-a-fix-in-go-19-sped-up-our-gitaly-service-by-30x/
Something doesn't quite hang together about that: The insane algorithm
described in the forklock documentation linked from the text can't still
be in use when using vfork instead of fork as this would require
modifying the file descriptors without writing to any memory. But
nevertheless, fork does become pretty expensive for 'large' processes:

--------
#include <signal.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>

enum {
N = 1000
};

static void burn_a_G(void)
{
char *p, *r;

p = malloc(1 << 30);
r = p;
do *r++ = 1; while (r - p < (1<<30));
}

int main(void)
{
unsigned n;

signal(SIGCHLD, SIG_IGN);

burn_a_G();
burn_a_G();
burn_a_G();
burn_a_G();

n = N;
do {
if (fork() == 0) _exit(0);
} while (--n);

return 0;
}
--------
[needs to be compiled without optimization]

It's possible to verify that fork is really O(n) regarding the adress
space size by changing the number of burn_a_G() calls. With four,
thousand forks take abour 42 seconds on the computer where I tested
this, ie, one roughly 0.04s.

This still throws some doubt on the text where it's claimed that

it [forking] can take several hundred milliseconds in a memory-intensive
process.

"Several hundred milliseconds" would mean > 0.2s. Considering the linear
change in fork performance, this would mean address space sizes >20G,
roughly 10G per 0.1s additional run time for fork (assuming that Gitlab
hardware is at least as powerful as my "underdesktower" :-).
G G
2020-07-27 08:22:36 UTC
Permalink
Post by Kaz Kylheku
Post by G G
creating a process in unix takes longer than in windows?
There aren't enough different implementations of Windows to make
interesting empirical results.
First of all, Windows does not offer fork (in its regular Win32
Creating a process which is a replica of the invoking "parent"
process, executing the same point in the same function.
The Cygwin project implements this fork operation in plain Win32.
It's neither fast nor pretty.
Creating a process which runs a new process image specified
via a nominated executable program in the filesystem space,
which receives a new address space.
then for that, Windows has the CreateProcess family of functions,
whereas traditional unix has fork combined with an exec function.
CreateProcess is, overall, saddled with fewer expensive requirements;
since there is no fork, there is no address space copying, and since
there is no exec-in-place, there is no clean-up of an existing space.
Unix systems have historically optimized fork with lazy semantics
to make it faster for all its intended uses, including the exec
case, where laziness wins by avoiding up-front duplication of an address
space only to then replace it. Further optimized variants of fork also
appeared (vfork) which so optimized toward the exec case as to be
incorrect if used for any other purpose.
exec in Unix has to clean up various parent resources. An
exec-optimized fork can help here by minimizing the clean-up.
If the exec is done out of a vfork, it knows not to clean up
most of the resources, which would be wrong, since they are still shared
with the parent.
Furthermore, certain resources are passed to the new process image,
such as signal hangling dispositions (which signals are SIG_IGN or
SIG_DFL) and open file descriptors not marked close-on-exec.
An exec-optimized fork cannot do much here.
None of this is an issue for CreateProcess which makes a new address
space, not requiring any clean-up. There is a BOOL flag for whether open
handles are inherited by the new process, which we would set to FALSE
if we were benchmarking process creation. Signals don't exist.
Complicating the comparison is that POSIX now has a function called
posix_spawn, which in many ways resembles CreateProcess. It creates
a new process without the involvement of fork semantics, over a
specified executable program, and has arguments for controlling
inheritance of resources.
If we were benchmarking process creation in a modern Unix implementation
against Win32, of course it would behoove us to use posix_spawn.
--
TXR Programming Lanuage: http://nongnu.org/txr
Music DIY Mailing List: http://www.kylheku.com/diy
ADA MP-1 Mailing List: http://www.kylheku.com/mp1
thanks Kaz Kylheku, that was real interesting and exactly
what i was looking for, though the question was, i guess,
vague in nature. i'm reading "Operating System Concepts,
10th edition, and the author explains creating a process,
using fork() in unix and Windows' CreateProcess. it seemed
to me that the forking was much more involved, so to
speak/write, lending itself to be somewhat slower.

thanks again.
Jorgen Grahn
2020-07-28 15:02:24 UTC
Permalink
Post by G G
creating a process in unix takes longer than in windows?
That was one of Eric Raymond's main propositions in "The Art of Unix
Programming", available online. See that one.

(Personally I don't care what Windows does.)

/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
Loading...