Discussion:
[Toybox] Patch submission process
Luis Felipe Strano Moraes
2012-02-03 13:20:38 UTC
Permalink
Hey guys,

just did a small patch removing an unnecessary allocation I spotted in
toybox, what would be the best way to submit it?

It is available in bitbucket here:
https://bitbucket.org/lfelipe/toybox/changeset/84f7fc53c69f

Best regards,
--
Lu?s Felipe Strano Moraes
http://www.strano.org
Rob Landley
2012-02-04 04:33:08 UTC
Permalink
Post by Luis Felipe Strano Moraes
Hey guys,
just did a small patch removing an unnecessary allocation I spotted in
toybox, what would be the best way to submit it?
https://bitbucket.org/lfelipe/toybox/changeset/84f7fc53c69f
Best regards,
Good catch. Applied.

Thanks,

Rob
Frank Bergmann
2012-02-04 10:32:52 UTC
Permalink
Dear List,

as a new subscriber I want to introduce myself.
I'm a systems administrator and software engineer from Germany. Started
thirty years before with 8080, 6502 and Z80 assembler I still like to
write small and fast programs and tools - embedded or raw at linux system
level (of course mainly in C and just sometimes in assembler).
I dislike the overhead of busybox and heared about toybox. If there's
something which I can contribute I would be glad to do so.
Some of my own tools are published at www.tuxad.com.

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-04 17:09:25 UTC
Permalink
Post by Frank Bergmann
Dear List,
as a new subscriber I want to introduce myself.
Welcome to the list!
Post by Frank Bergmann
I'm a systems administrator and software engineer from Germany. Started
thirty years before with 8080, 6502 and Z80 assembler
I started on a commodore 64 myself. (38911 basic bytes free. Not
counting c000, or the tendency for Blitz! to compress things.)
Post by Frank Bergmann
I still like to
write small and fast programs and tools - embedded or raw at linux system
level (of course mainly in C and just sometimes in assembler).
I like small and fast, but primarily what I like is _simple_.

That's what attracted me to busybox in the first place, and why I
couldn't go back to it when I got over the reason I left: it stopped
being simple. I like code that is as clear, straightforward, and easy
to understand as it can be for what it's doing.
Post by Frank Bergmann
I dislike the overhead of busybox and heared about toybox. If there's
something which I can contribute I would be glad to do so.
http://landley.net/toybox/design.html and
http://landley.net/toybox/code.html are the first two things to read.
That second page has an "adding a new command section.

The todo lists are in two places:

My "slush pile" notes are at http://landley.net/toybox/todos and there's
a wiki at http://elinux.org/index.php?title=Busybox_replacement

Feel free to edit the wiki. I note that xargs has since been
implemented and I haven't updated the status on the wiki yet.

The wiki lays out the various use cases I find important. One of the
big ones is I'm trying to steer Android into becoming a generic PC
replacement, beating iPhone while staying reasonably open. To do that
it not only needs a full posix command line, but it needs to become self
hosting. (It's not a full-fledged software platform until it leaves the
nest and can rebuild itself. Right now, you need a PC to do that, but
we can _fix_that.)

I should start an "Android Self Hosting Project" page on the wiki for
all this info, shouldn't I?

Anyway, I already worked out the smallest and simplest self-hosting
Linux platform I could (Aboriginal Linux). It started with an automated
Linux From Scratch (3.x) and then I replaced glibc with uClibc and all
the gnu command line utilities with BusyBox. (For a while I was trying
to get it down to four packages: "linux, tinycc, busybox, uClibc". But
my tinycc fork is mothballed like toybox was, and if I restart it I need
to do qcc... Tangent.)

All the busybox work I did was a spin-off of that tiny bootstrapping
project. When I started, busybox could _not_ be used by itself to
replace all the Linux From Scratch command line packages: now it can.

And now I'm using Aboriginal Linux as a test environment for toolbox,
and replacing the _busybox_ commands one at a time:

http://landley.net/notes-2011.html#29-12-2011

(The boxen.sh script mentioned there is attached.)

However, that's only half the story. The other half is replacing
Android's "toolbox" in a bionic build environment. (Bionic is android's
libc, and it's crap, thus hard to build real programs against.) I need
to set up an android build environment, and use the existing toolchain
to build toybox, and make sure it's a drop-in replacement for toolbox.
This involves a few commands that don't exist anywhere else (some of
which are really stupid; android ported the windows registry to Linux, I
don't know what they were smoking), but I just added build-time probes
to scripts/genconfig.sh and can extend those for bionic support for that
stuff.

Beyond that: POSIX-2008/SUSv4 support is good on general principles, and
it's a free spec on the web, so I'm trying to get as close to that as is
reasonable, and document where we diverge. (Internationalization, for
example: being 8-bit clean is obvious and some UTF-8 support may be
reasonable in places, but most of this belongs at the GUI level, not at
the command line level. Whatever language you're working in, the "for"
keyword in C remains the same.)

I've documented some of this in http://landley.net/toybox/design.html
but possibly there should be a wiki page on that too...
Post by Frank Bergmann
Some of my own tools are published at www.tuxad.com.
Bookmarked. I'll try to look at it later today, but I need to move away
from the free caffeine refills to regain net access. :)

Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: boxen.sh
Type: application/x-sh
Size: 361 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20120204/0b999a74/attachment.sh>
Rob Landley
2012-02-05 18:07:57 UTC
Permalink
Hello Rob,
I read some list postings in the archive.
I write as PM because I don't want to "scare" one of the authors (Tim Bird).
Trust me, if the lwn.net coverage didn't scare him, I doubt honest code
review would make him blink. :)

I'll cc: this to the list but chop out your email, reply there if you
want to decloak. :)
- Does your goal "be simple" mean that the code must be very easy
readable? id.c IS very easy readable but in case of e.g. flags "u" and
"r" set you will make 4 syscalls when only 1 is needed. Shouldn't it
be "optimized"?
I tend to go through and do optimization passes on submissions. I
haven't gotten to id yet.

My "quick glance" when he sent it said "would putting a for(;;) loop
around only two printf calls actually improve matters, or is the
overhead of the loop going to make it larger? I need to bench it."

What I didn't do in the quick glance was notice that the options are
exclusive. As usual, the gnu/dammit version has a special error message
for every occasion:

$ id -ug
id: cannot print "only" of more than one choice

Or that this id implementation is just a stub that is _not_ compliant
The following formats shall be used...
"uid=%u(%s) gid=%u(%s)\n", <real user ID>, <user-name>,
<real group ID>, <group-name>
So yeah, it needs a second pass in a big way. I should default it to
"n" in the meantime because it's not really there right now...

Instead I've been doing release cleanup. Specificially regression
testing the existing commands in my aboriginal linux setup, and trying
to figure out when I swap in each toybox command individually the build
works for all of them, but when I swap in all of them uClibc dies with
an internal compiler error and I get segfaults from "sort" and "xargs"
in dmesg.

Plus I'd like to finish the C rewrite of scripts/config2help.py so
python isn't a build prerequisite if "help" is enabled. And I had to
teach scripts/genconfig.sh to do a compile probe against the containers
infrastructure and disable unshare if the target toolchain headers
haven't got the appropriate #defines...
- Many coders who extremely dislike overhead (including myself) only use
libcall printf() for debugging purposes. It's some kind of
"deprecated".
A valid concern, and I did think about this. But
printf/asprintf/vsprintf/fprintf are pretty much all the same engine,
once it's pulled in for one use the overhead of using it in a second
place is trivial.

Something _like_ it would be unavoidable for perror_exit() and friends
(attempts to move complexity from the library to the callsite are a net
increase in overhead when you scale to many calls). For example, the
above output sequence for "id" with no arguments would be 9 separate
output statements without printf. (Or 9 statements assembling a string,
of who knows what length, and then a tenth to write it out.)

Keep in mind that my primary design goal is _simplicity_, then size,
speed, and features. You have to trade these off against each other when
implementing, but simplicity is the one I weight the highest, because in
the long run it's the one that makes the implementation understandable,
and thus maintainable (and security auditable). That means I tend to
approach commands with "what's the simplest way to implement this
feature", "what's the simplest way to speed this up", and so on.

Leveraging code people already wrote makes _toybox_ simple, otherwise
you spend eight lines of code doing what one line of code can do, and
you wind up repeating that all through your program. This code is
required to be there by c99, and it's required to have certain behavior.
There are _sucky_implementations_ of this code, but there are also
reasonable ones.

The linux kernel contains a reasonable printf() implementation, and I
feel comfortable writing my own if necessary. The uClibc implementation
of printf() is about 10k, based on binary size growth for a static i586
compile between "puts("Hello\n")" and "printf("Hello%d\n",0)". (Note
that printf() with a just constant string and no arguments gets
downshifted to a puts equivalent in uClibc, identical binary size.)

Yes, I'm balancing several definitions of "small" here. I want to make
toybox as a whole, statically linked, take up as little disk space as
possible. And I want to make it take up as little ram as possible at
runtime.

And I want to let you build individual commands and have _those_ be as
small as possible, but in doing so I assume those will be dynamically
linked. I am not optimizing for the "multiple individual executables,
each statically linked" case because it's _crazy_ and means I should
never leverage shared infrastructure, which is how you make the one big
statically linked executable small.

If your objection is that most people don't know how to _use_ these
tools in a reasonable manner: Yes, I'm aware of the newbie mistake the
sudo guys did recently. (Welcome to C, where we cook with knives and
fire.) I've got linked lists containing pointers from different
allocation contexts. (The struct arg_list is malloced from the heap,
the values it points to live in environment space. That's the normal
use case of lib/args.c, because I don't want to copy memory
unnecessarily. There's also the global toybuf[] and the stack without
even bringing mmap() into it. When you write in C, you have to know
what you're _doing_.)

Over on xargs I was at first trying to make a version that fit in
toybuf[] and upshifted to a malloc() when it outgrew the 4k static
buffer, but as long as I had to write the malloc() version anyway making
it the _only_ code path simplified things, so that's what I checked in.
I do a lot of that, weigh multiple approaches and then wind up doing
the _less_ microoptimized version because it's simpler and scales better
and performance is reasonable and yes I'm aware of nommu systems but
this mostly still runs on them, and trying to make a nommu system
reliable in low memory situations isn't worth complicating the common case.
Libs like libowfat or libdjb will reduce overhead while
not making the source hard to read. What's your opinion?
Never heard of either, but in general:

The external dependencies of BusyBox (under my tenure):

libc

The external dependencies of toybox:

libc

I'm all for finding a better libc, but I don't even depend on stuff like
zlib or ncurses. (I'm ok shelling out to an external gzip binary and
piping stuff through it; the only sane way to add https support to wget
and such would be via stunnel.)

(Part of my reasoning here is that at the lowest levels of the system
the dependnecies go circular, which makes bootstrapping hard, and you
wind up building stuff twice, which is nuts and one configuration
inevitably gets way less testing. Avoiding external dependencies in
these packages other than "I need a compiler with libc" means you don't
have to build toybox twice to bootstrap a system.)

Note how glibc had a "stage 1 compiler", but toybox didn't. (Sheesh,
perl's probably the silliest one: building microperl so perl's build
system can be written _in_perl_. The C compiler has a good excuse for
that, perl does not.)
Frank
/* show effective, unless user specifies real */
uid = geteuid();
gid = getegid();
if (flags & FLAG_r) {
uid = getuid();
gid = getgid();
}
if (flags & FLAG_u) {
printf("%d\n", uid);
return;
}
if (flags & FLAG_g) {
printf("%d\n", gid);
return;
}
printf("%d %d\n", uid, gid);
if (flags & (FLAG_u|FLAG_g))
printf("%d\n", (flags & FLAG_u) ? uid | gid);
else printf("%d %d\n", uid, gid);

Except the standard _also_ requires -G and -n. And a far more thorough
reading. (Yeah, it's a todo item.)

Rob
Rob Landley
2012-02-05 20:02:59 UTC
Permalink
I suck at this bcc: thing.
Post by Rob Landley
Hello Rob,
I read some list postings in the archive.
I write as PM because I don't want to "scare" one of the authors (Tim Bird).
Trust me, if the lwn.net coverage didn't scare him, I doubt honest code
review would make him blink. :)
I'll cc: this to the list but chop out your email, reply there if you
want to decloak. :)
- Does your goal "be simple" mean that the code must be very easy
readable? id.c IS very easy readable but in case of e.g. flags "u" and
"r" set you will make 4 syscalls when only 1 is needed. Shouldn't it
be "optimized"?
I tend to go through and do optimization passes on submissions. I
haven't gotten to id yet.
My "quick glance" when he sent it said "would putting a for(;;) loop
around only two printf calls actually improve matters, or is the
overhead of the loop going to make it larger? I need to bench it."
What I didn't do in the quick glance was notice that the options are
exclusive. As usual, the gnu/dammit version has a special error message
$ id -ug
id: cannot print "only" of more than one choice
Or that this id implementation is just a stub that is _not_ compliant
The following formats shall be used...
"uid=%u(%s) gid=%u(%s)\n", <real user ID>, <user-name>,
<real group ID>, <group-name>
So yeah, it needs a second pass in a big way. I should default it to
"n" in the meantime because it's not really there right now...
Instead I've been doing release cleanup. Specificially regression
testing the existing commands in my aboriginal linux setup, and trying
to figure out when I swap in each toybox command individually the build
works for all of them, but when I swap in all of them uClibc dies with
an internal compiler error and I get segfaults from "sort" and "xargs"
in dmesg.
Plus I'd like to finish the C rewrite of scripts/config2help.py so
python isn't a build prerequisite if "help" is enabled. And I had to
teach scripts/genconfig.sh to do a compile probe against the containers
infrastructure and disable unshare if the target toolchain headers
haven't got the appropriate #defines...
- Many coders who extremely dislike overhead (including myself) only use
libcall printf() for debugging purposes. It's some kind of
"deprecated".
A valid concern, and I did think about this. But
printf/asprintf/vsprintf/fprintf are pretty much all the same engine,
once it's pulled in for one use the overhead of using it in a second
place is trivial.
Something _like_ it would be unavoidable for perror_exit() and friends
(attempts to move complexity from the library to the callsite are a net
increase in overhead when you scale to many calls). For example, the
above output sequence for "id" with no arguments would be 9 separate
output statements without printf. (Or 9 statements assembling a string,
of who knows what length, and then a tenth to write it out.)
Keep in mind that my primary design goal is _simplicity_, then size,
speed, and features. You have to trade these off against each other when
implementing, but simplicity is the one I weight the highest, because in
the long run it's the one that makes the implementation understandable,
and thus maintainable (and security auditable). That means I tend to
approach commands with "what's the simplest way to implement this
feature", "what's the simplest way to speed this up", and so on.
Leveraging code people already wrote makes _toybox_ simple, otherwise
you spend eight lines of code doing what one line of code can do, and
you wind up repeating that all through your program. This code is
required to be there by c99, and it's required to have certain behavior.
There are _sucky_implementations_ of this code, but there are also
reasonable ones.
The linux kernel contains a reasonable printf() implementation, and I
feel comfortable writing my own if necessary. The uClibc implementation
of printf() is about 10k, based on binary size growth for a static i586
compile between "puts("Hello\n")" and "printf("Hello%d\n",0)". (Note
that printf() with a just constant string and no arguments gets
downshifted to a puts equivalent in uClibc, identical binary size.)
Yes, I'm balancing several definitions of "small" here. I want to make
toybox as a whole, statically linked, take up as little disk space as
possible. And I want to make it take up as little ram as possible at
runtime.
And I want to let you build individual commands and have _those_ be as
small as possible, but in doing so I assume those will be dynamically
linked. I am not optimizing for the "multiple individual executables,
each statically linked" case because it's _crazy_ and means I should
never leverage shared infrastructure, which is how you make the one big
statically linked executable small.
If your objection is that most people don't know how to _use_ these
tools in a reasonable manner: Yes, I'm aware of the newbie mistake the
sudo guys did recently. (Welcome to C, where we cook with knives and
fire.) I've got linked lists containing pointers from different
allocation contexts. (The struct arg_list is malloced from the heap,
the values it points to live in environment space. That's the normal
use case of lib/args.c, because I don't want to copy memory
unnecessarily. There's also the global toybuf[] and the stack without
even bringing mmap() into it. When you write in C, you have to know
what you're _doing_.)
Over on xargs I was at first trying to make a version that fit in
toybuf[] and upshifted to a malloc() when it outgrew the 4k static
buffer, but as long as I had to write the malloc() version anyway making
it the _only_ code path simplified things, so that's what I checked in.
I do a lot of that, weigh multiple approaches and then wind up doing
the _less_ microoptimized version because it's simpler and scales better
and performance is reasonable and yes I'm aware of nommu systems but
this mostly still runs on them, and trying to make a nommu system
reliable in low memory situations isn't worth complicating the common case.
Libs like libowfat or libdjb will reduce overhead while
not making the source hard to read. What's your opinion?
libc
libc
I'm all for finding a better libc, but I don't even depend on stuff like
zlib or ncurses. (I'm ok shelling out to an external gzip binary and
piping stuff through it; the only sane way to add https support to wget
and such would be via stunnel.)
(Part of my reasoning here is that at the lowest levels of the system
the dependnecies go circular, which makes bootstrapping hard, and you
wind up building stuff twice, which is nuts and one configuration
inevitably gets way less testing. Avoiding external dependencies in
these packages other than "I need a compiler with libc" means you don't
have to build toybox twice to bootstrap a system.)
Note how glibc had a "stage 1 compiler", but toybox didn't. (Sheesh,
perl's probably the silliest one: building microperl so perl's build
system can be written _in_perl_. The C compiler has a good excuse for
that, perl does not.)
Frank
/* show effective, unless user specifies real */
uid = geteuid();
gid = getegid();
if (flags & FLAG_r) {
uid = getuid();
gid = getgid();
}
if (flags & FLAG_u) {
printf("%d\n", uid);
return;
}
if (flags & FLAG_g) {
printf("%d\n", gid);
return;
}
printf("%d %d\n", uid, gid);
if (flags & (FLAG_u|FLAG_g))
printf("%d\n", (flags & FLAG_u) ? uid | gid);
else printf("%d %d\n", uid, gid);
Except the standard _also_ requires -G and -n. And a far more thorough
reading. (Yeah, it's a todo item.)
Rob
_______________________________________________
Toybox mailing list
Toybox at lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net
Frank Bergmann
2012-02-05 22:31:03 UTC
Permalink
er... forwarded to the list.
Hi Rob,
Post by Rob Landley
"uid=%u(%s) gid=%u(%s)\n", <real user ID>, <user-name>,
<real group ID>, <group-name>
er... I remember now. :-(
Post by Rob Landley
above output sequence for "id" with no arguments would be 9 separate
output statements without printf. (Or 9 statements assembling a string,
of who knows what length, and then a tenth to write it out.)
Yes, this is a source code blow-up. But using your own calls it can also
be easily readable. I remember such sequences of statements in the source
buffer_puts(buffer_1,"http://");
buffer_puts(buffer_1,host);
buffer_puts(buffer_1,"/");
buffer_puts(buffer_1,url);
buffer_puts(buffer_1,"/\r\n\r\n");
You know what it means and you get just one syscall. Using printf() it is
very hard to guess, how many syscalls one statement will use.
Post by Rob Landley
Keep in mind that my primary design goal is _simplicity_, then size,
speed, and features. You have to trade these off against each other when
Yes, that was my question: Which kind of simplicity. ;-)
Post by Rob Landley
And I want to let you build individual commands and have _those_ be as
small as possible, but in doing so I assume those will be dynamically
linked. I am not optimizing for the "multiple individual executables,
Even if you replace some "evil" libcalls of stdio with your own (internal)
lib you can still make code more small. I always test my binaries after
big changes with different clibs and statically/dynamically and screw them
up with strace and/or ltrace. That's not every day business but doing so
gives you sometimes surprising results. :-)
Post by Rob Landley
libc
libc
These libs are meant for internal usage but they are meant only as
examples. In my tools I replaced much stdio-stuff with own "lib" routines
and still have only libc as dependency. This lowers side effects when
people use very different c-libs.
Post by Rob Landley
if (flags & (FLAG_u|FLAG_g))
printf("%d\n", (flags & FLAG_u) ? uid | gid);
else printf("%d %d\n", uid, gid);
What about the syscalls? ;-)
Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-06 12:29:02 UTC
Permalink
Post by Frank Bergmann
er... forwarded to the list.
Hi Rob,
Post by Rob Landley
"uid=%u(%s) gid=%u(%s)\n", <real user ID>, <user-name>,
<real group ID>, <group-name>
er... I remember now. :-(
That the real world has complex output formats, or that id's output
looks like that?
Post by Frank Bergmann
Post by Rob Landley
above output sequence for "id" with no arguments would be 9 separate
output statements without printf. (Or 9 statements assembling a string,
of who knows what length, and then a tenth to write it out.)
Yes, this is a source code blow-up. But using your own calls it can also
be easily readable. I remember such sequences of statements in the source
buffer_puts(buffer_1,"http://");
buffer_puts(buffer_1,host);
buffer_puts(buffer_1,"/");
buffer_puts(buffer_1,url);
buffer_puts(buffer_1,"/\r\n\r\n");
I fail to see how that's an improvement.

Replacing one function call, which fits on maybe two lines, with half a
screen full of function calls (at 80x25 terminal size) means less code
fits on the screen at once so you've got a lot more scrolling to see the
same amount of program in front of you. When the function call in
question is part of the C standard and the replacement isn't...

And if you really really care how many write function calls:

sprintf(buffer_1, "http://%s/%s/\r\n\r\n", host, url);

Or xmsprintf() in lib/lib.c will xmalloc() the appropriate amount for you.
Post by Frank Bergmann
You know what it means and you get just one syscall. Using printf() it is
very hard to guess, how many syscalls one statement will use.
And that's a "performance over simplicity" mis-optimization. I only
care how many system calls it is when it's a hot path, and the ascii
FILE structure has a built-in buffer to batch that stuff up a bit.
Post by Frank Bergmann
Post by Rob Landley
Keep in mind that my primary design goal is _simplicity_, then size,
speed, and features. You have to trade these off against each other when
Yes, that was my question: Which kind of simplicity. ;-)
Well, for one thing the above functions aren't in the standard C
library, and the ones I'm using are.

I've been wrapping a number of standard library calls (heck,there's an
xprintf()) but using simple rules like "anything that starts with an x
will perror_exit() if it does not succeed, so you never have to check
the return value for failure". My largest divergence there is ditching
getopt(), but that's because my new infrastructure does all the work for
each command before it even runs, and getopt() required the command to
have a switch statement.

With your calls above, presumably you have a reason for not using
"strcat()" instead. Your buffer has a length but I don't know what it
is. Does it automatically flush itself or is there an implicit "now
write all this out" that you didn't include, and if so is filling the
buffer an error or an auto-flush that's going to do an implicit system
call anyway?

What's handling write failures, both from the other end prematurely
hanging up and from "sigstop" causing short writes, possibly zero
length, which you have to check errno to distinguish the sort write from
EOF? And if you don't get the short writes correct then suddenly "tar c
blah | gzip | ssh" suddenly corrupts the tarball if you ctrl-z and then
fg the pipeline...

Yes, toybox has a lot of implicit knowledge like that too, such as
toybuf being page sized (4k, although not necessarily page aligned, but
that's still an amount of data in a single transaction that caches tend
to be tuned for), and us relying on sigaction(SA_RESTART) to avoid zero
length read/write problems having to check errno everywhere (which isn't
fully implemented yet but is the goal). I know what it all gets us, and
have tried to document most of it. I'm not sure what this "I don't like
printf, let me open-code every occurrence of it" gets you. Why not
write a better printf?

(P.S. if you look at echo, xprintf(), xputc(), and xflush() all exist
because "echo > /dev/full" is supposed to return with a non-zero error
code. This is the sort of detail I'm trying to get right.)
Post by Frank Bergmann
Post by Rob Landley
And I want to let you build individual commands and have _those_ be as
small as possible, but in doing so I assume those will be dynamically
linked. I am not optimizing for the "multiple individual executables,
Even if you replace some "evil" libcalls of stdio with your own (internal)
lib you can still make code more small.
You mean like the xblah() wrappers so I avoid testing return codes, or
the way xsmprintf() is doing mostly the same thing asprintf() is but
that's one of the gnu/dammit extensions that I didn't want to rely on?
(And really, I mean to replace getline() too but that brings us to the
"must write line editing with command history for both the shell and vi,
and yes SUSv4 specifies vi".)
Post by Frank Bergmann
I always test my binaries after
big changes with different clibs and statically/dynamically and screw them
up with strace and/or ltrace. That's not every day business but doing so
gives you sometimes surprising results. :-)
Notice the prebuilt strace binaries for a bunch of different hardware
targets at http;//landley.net/aboriginal/bin because yes, strace is
extremely useful. I don't like to look at the gnu tool source to see how
they implemented anything (*shudder*), but I do occasionally run the
command my distro comes with under strace.

But really, "minimal number of syscalls" is speed to me, not simplicity.
Hello world is simple code, if the libc it's linked against chooses to
do a separate syscall for every character of output that's libc's
problem. If performance sucks I'll look into it, but I expect some
variation from libc to libc and if it sucks _fix_your_libc_ or use a
better one.
Post by Frank Bergmann
Post by Rob Landley
libc
libc
These libs are meant for internal usage but they are meant only as
examples. In my tools I replaced much stdio-stuff with own "lib" routines
and still have only libc as dependency. This lowers side effects when
people use very different c-libs.
I'm coding for Linux, and now android. On Linux, klibc is laughable,
dietlibc is broken, uClibc I'm testing against, the musl developers
presumably fix their stuff when they find something wrong with it, and
bionic I expect to have to work around but that's with some sort of
lib/makeitstop.c that implements things like "printf" according to the
darn standard.

I have a vague academic interest in MacOS X but I've already got mdev
and unshare and last time I wrote mount it had --bind and --move and
such, and used /proc/mounts instead of the obsolete /etc/mtab, all of
which are very linux specific.
Post by Frank Bergmann
Post by Rob Landley
if (flags & (FLAG_u|FLAG_g))
printf("%d\n", (flags & FLAG_u) ? uid | gid);
else printf("%d %d\n", uid, gid);
What about the syscalls? ;-)
Presumably it will make them for me.

Syscalls under linux are way way way cheaper than under crap like
Slowaris. The whole kernel's backed by a hugepage with a permanent TLB
entry so the transition's basically free from a cache standpoint, they
upshifted from int 80 to SYSENTER to save a couple clock cycles, and
this is 10 years _after_ doing optimizations like
http://cryptnet.net/mirrors/texts/kissedagirl.html that other operating
systems just ignored...

This is the kind of performance optimization work they were doing 10
years ago:

http://kerneltrap.org/node/384

And these days, most Linux system calls don't even cause a context
switch (I.E. cache flush and page table walk):

http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html

That has gettid() at 100 nanoseconds or less...

Rob
Frank Bergmann
2012-02-06 15:31:08 UTC
Permalink
Hello,
Post by Rob Landley
That the real world has complex output formats, or that id's output
looks like that?
... that id's output should look like that (regarding POSIX). I know the
differences between POSIX and i.e. GNU only as described in man-page
sections 2 and 3. :-)
Post by Rob Landley
I fail to see how that's an improvement.
- small code size
- faster execution due to reduced overhead
- often reduced stack size
- sometimes reduce amount of syscalls

We should not start a "printf-flame-war". I just wanted to talk about some
experiences I've made (and Fefe and some others, too).
IMHO printf() is actually a wonder of existence because it breaks with
Doug McIlroy's great recommendation. ;-)

[...]

Guessing the number of syscalls a libcall produces at runtime should not
be done in a "released package" but while development. Remember your links
and the article about the tumb programmer? :-) I don't know what printf()
is actually doing on c-lib abc or c-lib xyz. To get a little bit more
control there are libcalls like setvbuf() which I rarely found in code
using stdio out in the wild. Funny thing, isn't it? ;-)
Post by Rob Landley
And that's a "performance over simplicity" mis-optimization. I only
care how many system calls it is when it's a hot path, and the ascii
FILE structure has a built-in buffer to batch that stuff up a bit.
Of course, taking printf as example. But you should always keep it in mind.
The worst thing I've seen was a trace of rrd-update on a Debian system.
More than 300 Syscalls due to dynamic loading and searching paths. After
that one or two fadvise/madvise-calls and then the write() itself. I
wonder if this binary was ever measured. :-)
Post by Rob Landley
Well, for one thing the above functions aren't in the standard C
library, and the ones I'm using are.
[...]
Post by Rob Landley
the return value for failure". My largest divergence there is ditching
getopt(), but that's because my new infrastructure does all the work for
each command before it even runs, and getopt() required the command to
have a switch statement.
This is the thing I was talking / asking about: Which functions are
provided by the internal lib? Do you have any plans to use a special
and license-compatible library like libowfat (this is one I remember) or
do you write your own basic functions (like I do for some years)?
Post by Rob Landley
With your calls above, presumably you have a reason for not using
"strcat()" instead. Your buffer has a length but I don't know what it
is. Does it automatically flush itself or is there an implicit "now
write all this out" that you didn't include, and if so is filling the
buffer an error or an auto-flush that's going to do an implicit system
call anyway?
er... actually I don't know what fnord's buffer routines do. IMHO you
specify the size of the buffer (comparing setvbuf) and you also got a
buffer_flush (comparing fflush).
Writing something like cat you should use a buffer and libcalls managing
it. Writing something like id you may use a small linebuffer within your
main function. It should be measured.
Post by Rob Landley
What's handling write failures, both from the other end prematurely
hanging up and from "sigstop" causing short writes, possibly zero
length, which you have to check errno to distinguish the sort write from
EOF? And if you don't get the short writes correct then suddenly "tar c
blah | gzip | ssh" suddenly corrupts the tarball if you ctrl-z and then
fg the pipeline...
I know how to handle these cases if it should be necessary but I don't
know how printf will react - even in glibc! Nice question, I should
investigate now. :-)
(BTW - that's one reason why I re-implmented some basic functions for
myself.)
Post by Rob Landley
have tried to document most of it. I'm not sure what this "I don't like
printf, let me open-code every occurrence of it" gets you. Why not
write a better printf?
That was one of the questions you may find between my lines. ;-)
Do you have one or a license-compatible "foreign" implementation?
Is it worth to be done?
Post by Rob Landley
You mean like the xblah() wrappers so I avoid testing return codes, or
the way xsmprintf() is doing mostly the same thing asprintf() is but
that's one of the gnu/dammit extensions that I didn't want to rely on?
Yes, some kind of. Another extensions which I often use is stpcpy (shame
on me). This leads to the question which interface standard should be
used? POSIX? 1996 or 2001? What about functions which are "deprecated"
(i.e. gethostbyname) or "rejected" (i.e. clearenv)?
Post by Rob Landley
But really, "minimal number of syscalls" is speed to me, not simplicity.
Hello world is simple code, if the libc it's linked against chooses to
do a separate syscall for every character of output that's libc's
problem. If performance sucks I'll look into it, but I expect some
variation from libc to libc and if it sucks _fix_your_libc_ or use a
better one.
OK. Some foreign lib bashing in source code comments is always nice to
read. ;-)
Post by Rob Landley
I have a vague academic interest in MacOS X but I've already got mdev
It's the only OS I know where you can switch off ptrace! ;-)
Its BSD-API is sometimes frustrating.
Post by Rob Landley
upshifted from int 80 to SYSENTER to save a couple clock cycles, and
Don't talk about Slowlaris. They NEED threads because only threads are
actually lightweight as processes in Linux are. But even with sysenter
which is IMHO used on every linux system today you should still keep in
mind that syscalls are the most expensive calls.
Post by Rob Landley
http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html
That has gettid() at 100 nanoseconds or less...
Wow! This is really low. But out in the wild you found crappy software
(IMHO more than 90%) which does hundreds of time()-calls a second. And
sometimes when Nagios warns about raising contextswitches on a specific
host you detect software with an strace you would never dream of - even if
you dream of Freddy Kruger all night long. ;-)

Rob, you're making me hungry. I should co/clone the current toybox stuff.
:-)

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-08 01:26:22 UTC
Permalink
Post by Frank Bergmann
Hello,
Post by Rob Landley
That the real world has complex output formats, or that id's output
looks like that?
... that id's output should look like that (regarding POSIX). I know the
differences between POSIX and i.e. GNU only as described in man-page
sections 2 and 3. :-)
I'm happy to not implement gnu extensions unless somebody's actually
using them, but if it's in susv4 I need to document deviances and
explain why. I.E. that needs a good _excuse_ not to implement stuff.

(I've been reading the bionic source a bit, I sympathize with not
implementing internationalization, and blocking C++ makes me smile, but
I'm still going to need to supplement it a bit.)
Post by Frank Bergmann
Post by Rob Landley
I fail to see how that's an improvement.
- small code size
- faster execution due to reduced overhead
- often reduced stack size
- sometimes reduce amount of syscalls
We should not start a "printf-flame-war".
I totally sympathize with your goals, and I'm all for pushing the limits
and asking "what can we get away with not doing". I do that all the
time, I regularly have people point out things I missed, and I
constantly need to reevaluate stuff.

But in this case, I plan to continue using the printf() family because
doing without it wouldn't actually simplify the code, I'd just wind up
writing something just as bad.

C and string handling are not a good mix. String handling is the thing
C is weakest at, because string handling actually turns out to be a hard
problem to get right at the hardware level.
Post by Frank Bergmann
I just wanted to talk about some
experiences I've made (and Fefe and some others, too).
IMHO printf() is actually a wonder of existence because it breaks with
Doug McIlroy's great recommendation. ;-)
Yeah, but after 40 years of being grandfathered in, it's still useful
enough to stick around.
Post by Frank Bergmann
[...]
Guessing the number of syscalls a libcall produces at runtime should not
be done in a "released package" but while development.
I run things under strace rather a lot, but even if it boiled down to
doing a for() loop around write(1, &char, 1) I'd tell you to fix your
libc rather than change what I was doing.
Post by Frank Bergmann
Remember your links
and the article about the tumb programmer? :-) I don't know what printf()
is actually doing on c-lib abc or c-lib xyz. To get a little bit more
control there are libcalls like setvbuf() which I rarely found in code
using stdio out in the wild. Funny thing, isn't it? ;-)
I'm not that interested in micromanaging something that most likely
stays in L1 cache either way.
Post by Frank Bergmann
Post by Rob Landley
And that's a "performance over simplicity" mis-optimization. I only
care how many system calls it is when it's a hot path, and the ascii
FILE structure has a built-in buffer to batch that stuff up a bit.
Of course, taking printf as example. But you should always keep it in mind.
The worst thing I've seen was a trace of rrd-update on a Debian system.
More than 300 Syscalls due to dynamic loading and searching paths.
That's the worst you've seen? Never run strace on gcc. Certainly not
right after eating:

http://landley.net/notes-2006.html#03-11-2006

(This is why I wrote ccwrap.c in aboriginal linux. The pathological
path logic in gcc is not salvageable, and piling more crap on top
ever-higher in hopes the bad foundations will go away is how we got
Windows and microkernels.)
Post by Frank Bergmann
After
that one or two fadvise/madvise-calls and then the write() itself. I
wonder if this binary was ever measured. :-)
Post by Rob Landley
Well, for one thing the above functions aren't in the standard C
library, and the ones I'm using are.
[...]
Post by Rob Landley
the return value for failure". My largest divergence there is ditching
getopt(), but that's because my new infrastructure does all the work for
each command before it even runs, and getopt() required the command to
have a switch statement.
This is the thing I was talking / asking about: Which functions are
provided by the internal lib?
See lib/lib.h

I brought up get_optflags() in response to printf() because they're
conceptually similar: both bundle a lot of commonly used functionality
together into one place that does it right. Open-coding printf strikes
me as no different than having a for(;;) loop in every command to parse
command line stuff. It's only a win if you never have to do it more
than once.
Post by Frank Bergmann
Do you have any plans to use a special
and license-compatible library like libowfat (this is one I remember) or
do you write your own basic functions (like I do for some years)?
I write my own basic functions. I'd only use an external library if I
was willing to fork it, merge a copy in, and maintain it myself. Which
is almost guaranteed to be infrastructure in search of a user.

Minimal system bootstrapping is theoretically four things:

A) kernel
B) libc
C) POSIX command line
D) compiler

None of those should depend on anything outside that group. Way back
when I was hoping for "linux, uClibc, busybox, tinycc" for the four.
That didn't pan out, but I'm still working on keeping it simple.

Environmental dependencies are a form of complexity we're bad at
tracking, and which most people forget about, but that doesn't mean
they're any less important. (I've got code in sort that avoids pulling
in libm, which is supplied by libc, because it wasn't necessary in that
case.)
Post by Frank Bergmann
Post by Rob Landley
With your calls above, presumably you have a reason for not using
"strcat()" instead. Your buffer has a length but I don't know what it
is. Does it automatically flush itself or is there an implicit "now
write all this out" that you didn't include, and if so is filling the
buffer an error or an auto-flush that's going to do an implicit system
call anyway?
er... actually I don't know what fnord's buffer routines do. IMHO you
specify the size of the buffer (comparing setvbuf) and you also got a
buffer_flush (comparing fflush).
Writing something like cat you should use a buffer and libcalls managing
it.
Or you could do toys/cat.c using the global toybuf[4096] which is part
of the bss and only gets its page faulted in if it's actually dirtied.
(Modulo alignment considerations I haven't bothered about.)
Post by Frank Bergmann
Writing something like id you may use a small linebuffer within your
main function. It should be measured.
I put toybuf[] in so every command has access to a global 4k buffer they
don't have to malloc (on top of the DEFINE_GLOBALS()), so I avoid some
heap walks that way.

I note I'm not happy with the lib/dirtree.c stuff (which is the only
current lib/* thing that actualy depends on toybuf, everything else
leaves it for the command to use) because it's got a 4k PATH_MAX
limitation, which the linux kernel doesn't currently have. But at the
same time, it made the code noticeably simpler, and the correct fix is
to use openat() and similar anyway. I think I blogged about this...
Post by Frank Bergmann
Post by Rob Landley
What's handling write failures, both from the other end prematurely
hanging up and from "sigstop" causing short writes, possibly zero
length, which you have to check errno to distinguish the sort write from
EOF? And if you don't get the short writes correct then suddenly "tar c
blah | gzip | ssh" suddenly corrupts the tarball if you ctrl-z and then
fg the pipeline...
I know how to handle these cases if it should be necessary but I don't
know how printf will react - even in glibc! Nice question, I should
investigate now. :-)
I already did. See lib/lib.c function xprintf().
Post by Frank Bergmann
(BTW - that's one reason why I re-implmented some basic functions for
myself.)
Or you could read the spec and see what behavior is _required_ from a
conformant implementation. Back when the c99 committee was doing its
thing they posted drafts of the spec for public review and then demanded
money for the final version, so everybody kept the last draft version
and called it good. I have a copy on my website, and it says:

http://landley.net/c99-draft.html#7.19.6
Post by Frank Bergmann
The fprintf function returns the number of characters transmitted, or
a negative value if an output or encoding error occurred.
...
Post by Frank Bergmann
The printf function is equivalent to fprintf with the argument stdout
interposed before the arguments to printf.
(The man 3 printf page says it too, but that documents what glibc did
rather than what the standard requires.)

If there is a standard requiring behavior, I choose the right to require
conformance with that standard in the environments I build in. I'm
doing doing one of those insane ./configure stages with 8000 tests to
try to work around every possible kind of broken out there (which can't
be done). I'm letting the build break and telling you to fix your
environment.

I may tolerate specific environments that diverge from that standard
(I.E. bionic, and maybe macosx), but in that case I'm humoring the bugs
of a specific implementation while requiring everything else to conform
to the darn standard.

I do not allow UNKNOWN environments to violate the standard and still
expect to work. So if your argument is "this printf() is horrible" then
don't use it, and if your argument is "some unknown printf() might be
horrible someday" I can honestly say it's not my problem.

As I think I said in design.html, I'm replying on c99, posix-2008, and
LP64. (If I wasn't clear enough there tell me and I'll go fix it.)
Post by Frank Bergmann
Post by Rob Landley
have tried to document most of it. I'm not sure what this "I don't like
printf, let me open-code every occurrence of it" gets you. Why not
write a better printf?
That was one of the questions you may find between my lines. ;-)
Because the c99 standard supplies one for me and I use rather a lot of
it, including the printf("% *ld", len, val) stuff in df which was one of
the first commands I wrote for toybox. (Actually I started it for
busybox but left that project before it was finished, so never submitted
it.)
Post by Frank Bergmann
Do you have one or a license-compatible "foreign" implementation?
Is it worth to be done?
I'm not here to reimplement c99 unless I'm writing a C library or a
compiler. Toybox is neither.

The environments that are in front of me are uClibc (which I've patched
before and can push patches for upstream), glibc (which I'm building
against in the first pass), and bionic (which I expect to supplement
heavily and we'll see if Google wants patches or if I need a
libtrionic.c or something).
Post by Frank Bergmann
Post by Rob Landley
You mean like the xblah() wrappers so I avoid testing return codes, or
the way xsmprintf() is doing mostly the same thing asprintf() is but
that's one of the gnu/dammit extensions that I didn't want to rely on?
Yes, some kind of. Another extensions which I often use is stpcpy (shame
on me).
Never heard of it. I've got a strlcpy() but everybody does since
strncpy() isn't guaranteed to null terminate the output.)

Huh, apparently it's not an extension, it's in SUSv4:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/stpcpy.html

That's really cool. Thanks. I wonder where that's needed in lib/* or
toys/* (or main.c), and I wonder if uClibc or bionic have caught up yet?
Post by Frank Bergmann
This leads to the question which interface standard should be
used? POSIX? 1996 or 2001? What about functions which are "deprecated"
(i.e. gethostbyname) or "rejected" (i.e. clearenv)?
Did you read http://landley.net/toybox/design.html yet? Do I need to
fluff that out a bit?

(At a quick glance I see it says SUSv3 because when I wrote it, SUSv4
wasn't out yet. Yeah, could use some updating. And explaining how
POSIX-2008 and SUSv4 and the Open Group Base Specifications Issue 7 are
all the same darn thing just like ANSI/ISO C is the same standard
approved by two standards bodies...)
Post by Frank Bergmann
Post by Rob Landley
But really, "minimal number of syscalls" is speed to me, not simplicity.
Hello world is simple code, if the libc it's linked against chooses to
do a separate syscall for every character of output that's libc's
problem. If performance sucks I'll look into it, but I expect some
variation from libc to libc and if it sucks _fix_your_libc_ or use a
better one.
OK. Some foreign lib bashing in source code comments is always nice to
read. ;-)
I note that I pay no attention to dietlibc, and am unlikely to start.
Post by Frank Bergmann
Post by Rob Landley
I have a vague academic interest in MacOS X but I've already got mdev
It's the only OS I know where you can switch off ptrace! ;-)
Its BSD-API is sometimes frustrating.
Interesting,there doesn't seem to be a kernel CONFIG symbol for ptrace...

There should be. Possibly only with CONFIG_EMBEDDED, but there really
should be a way to configure that out...
Post by Frank Bergmann
Post by Rob Landley
upshifted from int 80 to SYSENTER to save a couple clock cycles, and
Don't talk about Slowlaris. They NEED threads because only threads are
actually lightweight as processes in Linux are. But even with sysenter
which is IMHO used on every linux system today
It's an x86 instruction and we're targeting ARM android devices, so "not
so much". Although I believe android has something similar.
Post by Frank Bergmann
you should still keep in
mind that syscalls are the most expensive calls.
Over on my other major hobby project, Aboriginal Linux, I analyzed the
performance of ./configure and determined that statically linking
busybox would speed up ./configure by around 20% due to the lower page
translation overhead from eliminating the self-modifying-code aspect of
dynamic linking that forced executable pages to be retranslated.

Trust me: I know how to profile stuff, and how to understand the
performance of things, and I am not worried about the overhead of a
dozen or so nonblocking syscalls where we're faulting in maybe three
cache lines from L2 to L1.

Implement, _then_ optimize. I'd like to worry about where the
bottlenecks actually _are_ rather than borrowing trouble.
Post by Frank Bergmann
Post by Rob Landley
http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html
That has gettid() at 100 nanoseconds or less...
Wow! This is really low. But out in the wild you found crappy software
(IMHO more than 90%) which does hundreds of time()-calls a second.
Which is why they changed it so gettimeofday() can just read an atomic
variable out of the vsyscall page:

http://www.win.tue.nl/~aeb/linux/lk/lk-4.html

(They were discussing it, anyway. I don't remember if that change went
through...)

Yeah, fixing the code not to do that is a good thign too, but for things
like video playback software you need short waits and to be woken up
often because it's what the program _does_...
Post by Frank Bergmann
And
sometimes when Nagios warns about raising contextswitches on a specific
host you detect software with an strace you would never dream of - even if
you dream of Freddy Kruger all night long. ;-)
I'm totally aware that most existing userspace software is crap:

http://lwn.net/Articles/192214/

However, you can go too far in over-optimizing (which is actually part
of the _reason_ so much of it's crap).

Software needs to remain flexible. If you don't understand why it's
doing things the way it's doing, you can't change it later and it gets
brittle and you do workarounds instead of fixes and it all falls apart.

Sometimes, doing a nice tight implementation winds up with knots that
are hard to undo. (Both of the current patch and sort implementations
are in that category, for example.) Sometimes, that's the local peak.
Making it simpler or easier to understand results in significantly more
code or signficantly more runtime. (I still try anyway, and at least
comment the house-of-cards bit.)

But complexity is a cost, I want to make sure we get the best bang for
the bug, and figuring out what simple _is_ can be really hard. (Simple
is elegant. I can't always come up with an elegant solution. Doesn't
mean I don't go back and polish the the dorodango until it gleams, and
sometimes I _do_ figure out how to make an actual gemstone out of it.)
Post by Frank Bergmann
Rob, you're making me hungry. I should co/clone the current toybox stuff.
:-)
The more eyeballs the merrier. I'm stretched a bit thin at present, but
trying to keep up... :)
Post by Frank Bergmann
Frank
Rob
David Seikel
2012-02-08 02:19:48 UTC
Permalink
Post by Rob Landley
The more eyeballs the merrier. I'm stretched a bit thin at present,
but trying to keep up... :)
Stretched a bit thin myself at the moment, which is why I've not gotten
around to actually helping out yet with toybox. And now my last
contract has run out of money still owing me for Decembers work. Need
a new contract. sigh
--
A big old stinking pile of genius that no one wants
coz there are too many silver coated monkeys in the world.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20120208/0c123897/attachment-0001.pgp>
Frank Bergmann
2012-02-08 08:00:10 UTC
Permalink
Hi.
Post by Rob Landley
But in this case, I plan to continue using the printf() family because
doing without it wouldn't actually simplify the code, I'd just wind up
writing something just as bad.
In this case you may use more control to use printf as desired. I'd
made a clone pf toybox and busybox and issued one command to compare:

$ grep -r setvbuf busybox|wc -l
9
$ grep -r setvbuf toybox-cloned-1328632507|wc -l
0
Post by Rob Landley
C and string handling are not a good mix. String handling is the thing
C is weakest at, because string handling actually turns out to be a hard
problem to get right at the hardware level.
We all know that C actually doesn't know anything about "strings". ;-)
Writing "bigger" software it *may* be worth implementing strings as a
class in C (er... struct) like Wietse Venema does. But this means to
rewrite all code.
Post by Rob Landley
Yeah, but after 40 years of being grandfathered in, it's still useful
enough to stick around.
Yeah, but 99% of software out in the wild breaks the basic rules KISS and
YAGNI and introduce (sometimes many) bugs and holes with this.
Post by Rob Landley
I run things under strace rather a lot, but even if it boiled down to
doing a for() loop around write(1, &char, 1) I'd tell you to fix your
libc rather than change what I was doing.
OK, this self-written printf-implementation you are stracing is not very
well optimized. ;-)
Post by Rob Landley
I'm not that interested in micromanaging something that most likely
stays in L1 cache either way.
Hmmm... if your code fits in L1-cache right before doing a sysenter then
the cache will be dirty when doing the call of sysenter, isn't it?
I never measured applications touching this "problem" mostly because of
that causes:
- hard to measure L1 cache running an OS with many tasks and not many
cores
- Loops already fit in L1 cache and did not call code "outside", running
so fast that you can't measure
- reducing the amount of syscalls brought most speedup, other changes were
only ambiguous measurable
- too many syscalls which can't be reduced let the advantage of the cache
vanish (mostly showing big i/o waits in top at the core the application
runs)
- some causes where further optimizations didn't make any sense (due to
e.g. network latencies)
As you wrote: You'll have to measure it (all). Until then you must keep
caches in mind.
Post by Rob Landley
That's the worst you've seen?
Yes, I didn't expect it on rrdtool. But after stracing it I understand the
cause why it is actually running slow even on big hosts with many
rrd-updates even though fadvise/madvise should catch these cases.
Post by Rob Landley
Never run strace on gcc. Certainly not
gcc is one of the tools I never *wanted* to strace because I already
expected a nightmare (other tools are e.g. php). ;-)
Post by Rob Landley
See lib/lib.h
clone done. Patches submit to the list?
My first make did throw the nasty "dereferencing type-punned pointer will
break strict-aliasing rules". In sort.c you use TT.lines as char* and not
char**.
Post by Rob Landley
command line stuff. It's only a win if you never have to do it more
than once.
That's why I often used small internal output buffers and the nasty
stpcpy.
BTW - I've read that pivot_root doesn't have a high priority in your
TODO-list. It's very easy to implement cause Linux offers a syscall. Older
glibc didn't offer a wrapping but this is also easy to check (and to
implement if necessary). I want to write it as my first toy-code if no one
else is working on it.
Next thing could be mount even though it's not that easy.
Post by Rob Landley
Or you could do toys/cat.c using the global toybuf[4096] which is part
of the bss and only gets its page faulted in if it's actually dirtied.
(Modulo alignment considerations I haven't bothered about.)
I've read your docs but know I did also the clone and read some code. :-)
Post by Rob Landley
As I think I said in design.html, I'm replying on c99, posix-2008, and
LP64. (If I wasn't clear enough there tell me and I'll go fix it.)
No, it is clear. It was just a big bunch of docs. I yet don't know POSIX
2008, only 2001. I think there is much more "deprecated".
Post by Rob Landley
the first commands I wrote for toybox. (Actually I started it for
busybox but left that project before it was finished, so never submitted
it.)
Maybe they will backport some day. ;-)
Post by Rob Landley
Never heard of it. I've got a strlcpy() but everybody does since
strncpy() isn't guaranteed to null terminate the output.)
Better you forget this nasty thing. The man-page says that it is a
GNU-extension and that it maybe goes back to the old msdos times...
Post by Rob Landley
http://pubs.opengroup.org/onlinepubs/9699919799/functions/stpcpy.html
er... maybe this is a april fools day joke
Post by Rob Landley
That's really cool. Thanks. I wonder where that's needed in lib/* or
Don't forget the name "msdos" in its history. ;-)
Post by Rob Landley
Did you read http://landley.net/toybox/design.html yet? Do I need to
fluff that out a bit?
Sorry, it was a bunch of docs. First I read the stuff easily linked and
then the urls you posted.
Post by Rob Landley
all the same darn thing just like ANSI/ISO C is the same standard
approved by two standards bodies...)
Yes, we should be glad there is not a DIN standard yet. ;-)
Post by Rob Landley
Trust me: I know how to profile stuff, and how to understand the
I do. Before I read your opinion about tumb programmers I already tried to
think so.

My experiences are mainly the results of writing some monitoring tools
which sometimes can cause i/o wait, or writing a fast fgrep where I
measured that the size of the buffer is a great killer if you want to
speed it up. :-)
Post by Rob Landley
Which is why they changed it so gettimeofday() can just read an atomic
Yes, I know this. But even if gettimeofday is not a "young" call there are
many, many projects which didn't recognize.
I still use it in some of my tools but only one time and not many times
and even more not many times a second.
Post by Rob Landley
http://lwn.net/Articles/192214/
Bookmarked. And - yes! - stat calls are the next "evil" calls which are
way too many called. Like times it is also a problem in the tool mentioned
above (not gcc but the other ;-) ).

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Frank Bergmann
2012-02-08 10:36:26 UTC
Permalink
Post by Frank Bergmann
clone done. Patches submit to the list?
My first make did throw the nasty "dereferencing type-punned pointer will
break strict-aliasing rules". In sort.c you use TT.lines as char* and not
char**.
Here's the stuff:

Repository:
Make generated/config.h from .config.
Compile toybox...
toys/sort.c: In function ?sort_read?:
toys/sort.c:311: warning: dereferencing type-punned pointer will break strict-aliasing rules
[fwb at vdr toybox-cloned-1328632507]$ cd ../toybox

Local:
Make generated/config.h from .config.
Compile toybox...
[fwb at vdr toybox]$ ln -s toybox sort
[fwb at vdr toybox]$ echo -e '1\n2\n4\n3' >1
[fwb at vdr toybox]$ ./sort -c 1
sort: 1: Check line 3

[fwb at vdr toybox]$ diff -u ../toybox-cloned-1328632507/toys/sort.c toys/sort.c
--- ../toybox-cloned-1328632507/toys/sort.c 2012-02-07 17:35:07.000000000 +0100
+++ toys/sort.c 2012-02-07 17:37:41.000000000 +0100
@@ -308,7 +308,7 @@
if (CFG_SORT_BIG && (toys.optflags&FLAG_c)) {
int j = (toys.optflags&FLAG_u) ? -1 : 0;

- if (TT.lines && compare_keys((char **)&TT.lines, &line)>j)
+ if (TT.lines && compare_keys((void *)&TT.lines, &line)>j)
error_exit("%s: Check line %d\n", name, TT.linecount);
free(TT.lines);
TT.lines = (char **)line;

Your last patch which maybe introduced the warning:
diff -r 87edfe8ae99e -r 9e7aaecf0683 toys/sort.c
--- a/toys/sort.c Mon Feb 06 21:14:22 2012 -0600
+++ b/toys/sort.c Tue Feb 07 00:31:37 2012 -0600
@@ -308,11 +308,9 @@
if (CFG_SORT_BIG && (toys.optflags&FLAG_c)) {
int j = (toys.optflags&FLAG_u) ? -1 : 0;

- if (TT.linecount && compare_keys((char *)TT.lines,line)>j)
+ if (TT.lines && compare_keys((char **)&TT.lines, &line)>j)
error_exit("%s: Check line %d\n", name, TT.linecount);
-
- if (TT.lines) free(TT.lines);
- else TT.linecount = 0;
+ free(TT.lines);
TT.lines = (char **)line;
} else {
if (!(TT.linecount&63))
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-09 01:34:10 UTC
Permalink
Post by Frank Bergmann
clone done. Patches submit to the list?
My first make did throw the nasty "dereferencing type-punned pointer will
break strict-aliasing rules". In sort.c you use TT.lines as char* and not
char**.
I fixed the sort bug yesterday morning:

http://landley.net/hg/toybox/rev/433

It conflicts with your patch. If there's still an aliasing warning for
you, can you send me a patch on top of the current repository?

Thanks,

Rob
Frank Bergmann
2012-02-09 09:37:05 UTC
Permalink
Hi,
Post by Rob Landley
http://landley.net/hg/toybox/rev/433
It conflicts with your patch. If there's still an aliasing warning for
you, can you send me a patch on top of the current repository?
maybe my hg is broken but I don't think it is. I did a update and then
applied the patch to the current rev.

Doing update again it does nothing:
$ hg update
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg log toys/sort.c
changeset: 433:9e7aaecf0683
tag: tip
user: Rob Landley <rob at landley.net>
date: Tue Feb 07 00:31:37 2012 -0600
summary: Fix sort -uc (pointer vs pointer to pointer confusion, covered by typecast).

(I'm new to mercurial. I did also search a way of recovering the
timestamps. It took me some time until I got a line doing the job. rcs,
cvs and svn I used many years (seldom git) but I'm a newbie regarding
mercurial.)

I still got the same diff:
$ diff -u ../toybox-cloned-1328632507/toys/sort.c
toys/sort.c
--- ../toybox-cloned-1328632507/toys/sort.c 2012-02-07
17:35:07.000000000 +0100
+++ toys/sort.c 2012-02-07 17:37:41.000000000 +0100
@@ -308,7 +308,7 @@
if (CFG_SORT_BIG && (toys.optflags&FLAG_c)) {
int j = (toys.optflags&FLAG_u) ? -1 : 0;

- if (TT.lines && compare_keys((char **)&TT.lines, &line)>j)
+ if (TT.lines && compare_keys((void *)&TT.lines, &line)>j)

TT.lines is char** and used as char*, char is 8 bit / void "no bit" and
void* must be interchangeable with xxxxxx* - IMHO this must be valid c99 but
I didn't test it.

BTW - the last value of TT.lines is free()ed before assigning line as new
value. I guess you use malloc of the C-lib, aren't you? For tools with
low running times and small memory usage it could be worth using an
allocation function as wrapper like alloc() by DJB.
Instructing your authors to use this is far more easy than using a string
class: vim/sed does the job. ;-)
My local copy:
http://www.tuxad.com/ngtx/ngtx-current/lib/alloc.c
IMHO you know it already because DJB's daemontools package was mentioned
on the bb mailing list (Denis Vlasenko and runit).

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-14 04:00:13 UTC
Permalink
Post by Frank Bergmann
Hi,
Post by Rob Landley
http://landley.net/hg/toybox/rev/433
It conflicts with your patch. If there's still an aliasing warning for
you, can you send me a patch on top of the current repository?
maybe my hg is broken but I don't think it is. I did a update and then
applied the patch to the current rev.
I missed the fact you'd quoted _my_ previous patch, which was the
conflict. :)

My bad.
Post by Frank Bergmann
TT.lines is char** and used as char*, char is 8 bit / void "no bit" and
void* must be interchangeable with xxxxxx* - IMHO this must be valid c99 but
I didn't test it.
void * automatically get promoted in C. (In C++ they must be explicitly
typecast, but luckily we're not doing C++.)
Post by Frank Bergmann
BTW - the last value of TT.lines is free()ed before assigning line as new
value.
Yes: leaking memory on potentially infinite input is bad.
Post by Frank Bergmann
I guess you use malloc of the C-lib, aren't you?
We're writing in C, it comes with the territory.
Post by Frank Bergmann
For tools with
low running times and small memory usage it could be worth using an
allocation function as wrapper like alloc() by DJB.
Here are toolchains that link against uClibc:

http://landley.net/aboriginal/bin

Last I checked it had something like 3 different allocators you could
pick in menuconfig when you built it.

I'm also working on putting together a toolchain that links against
bionic, and there's the musl guys, and...

A) This is "build environment", I.E. "not my problem". My job there is
to stay out of your way. If you want to swap out malloc() for a
different malloc(), be my guest. You can do so as part of your build
environment. ($CFLAGS should work.)

B) I personally find DJB annoying, and try to ignore him rather than
ranting about him.
Post by Frank Bergmann
Instructing your authors to use this is far more easy than using a string
class: vim/sed does the job. ;-)
Should I instruct people on how to use valgrind, strace, and operf as
well? (What's wrong with malloc?)
Post by Frank Bergmann
http://www.tuxad.com/ngtx/ngtx-current/lib/alloc.c
IMHO you know it already because DJB's daemontools package was mentioned
on the bb mailing list (Denis Vlasenko and runit).
You're assuming I've been paying waaaay more attention to the busybox
list than I actually have since I left. But I was vaguely aware of
Denys adding his pet toolset to busybox, which I felt those had no
business being in busybox since we already had an init and they weren't
present in any system I was familiar with, but he was maintainer now and
I wasn't going to criticize his decisions...

(And I followed the "what license is qmail under", "go away and stop
bothering me" decade of people using software the guy refused to
clarify. And which nobody else could patch because the had no terms...)

(And despite what ESR said it's not "bernstein chaining" if the same
darn technique was already used by nice, detach, strace, sudo, and a
gazillion other things...)

Sigh. Sore spot with me...

Rob
Rob Landley
2012-02-14 04:00:13 UTC
Permalink
Post by Frank Bergmann
Hi,
Post by Rob Landley
http://landley.net/hg/toybox/rev/433
It conflicts with your patch. If there's still an aliasing warning for
you, can you send me a patch on top of the current repository?
maybe my hg is broken but I don't think it is. I did a update and then
applied the patch to the current rev.
I missed the fact you'd quoted _my_ previous patch, which was the
conflict. :)

My bad.
Post by Frank Bergmann
TT.lines is char** and used as char*, char is 8 bit / void "no bit" and
void* must be interchangeable with xxxxxx* - IMHO this must be valid c99 but
I didn't test it.
void * automatically get promoted in C. (In C++ they must be explicitly
typecast, but luckily we're not doing C++.)
Post by Frank Bergmann
BTW - the last value of TT.lines is free()ed before assigning line as new
value.
Yes: leaking memory on potentially infinite input is bad.
Post by Frank Bergmann
I guess you use malloc of the C-lib, aren't you?
We're writing in C, it comes with the territory.
Post by Frank Bergmann
For tools with
low running times and small memory usage it could be worth using an
allocation function as wrapper like alloc() by DJB.
Here are toolchains that link against uClibc:

http://landley.net/aboriginal/bin

Last I checked it had something like 3 different allocators you could
pick in menuconfig when you built it.

I'm also working on putting together a toolchain that links against
bionic, and there's the musl guys, and...

A) This is "build environment", I.E. "not my problem". My job there is
to stay out of your way. If you want to swap out malloc() for a
different malloc(), be my guest. You can do so as part of your build
environment. ($CFLAGS should work.)

B) I personally find DJB annoying, and try to ignore him rather than
ranting about him.
Post by Frank Bergmann
Instructing your authors to use this is far more easy than using a string
class: vim/sed does the job. ;-)
Should I instruct people on how to use valgrind, strace, and operf as
well? (What's wrong with malloc?)
Post by Frank Bergmann
http://www.tuxad.com/ngtx/ngtx-current/lib/alloc.c
IMHO you know it already because DJB's daemontools package was mentioned
on the bb mailing list (Denis Vlasenko and runit).
You're assuming I've been paying waaaay more attention to the busybox
list than I actually have since I left. But I was vaguely aware of
Denys adding his pet toolset to busybox, which I felt those had no
business being in busybox since we already had an init and they weren't
present in any system I was familiar with, but he was maintainer now and
I wasn't going to criticize his decisions...

(And I followed the "what license is qmail under", "go away and stop
bothering me" decade of people using software the guy refused to
clarify. And which nobody else could patch because the had no terms...)

(And despite what ESR said it's not "bernstein chaining" if the same
darn technique was already used by nice, detach, strace, sudo, and a
gazillion other things...)

Sigh. Sore spot with me...

Rob
Frank Bergmann
2012-02-09 09:37:05 UTC
Permalink
Hi,
Post by Rob Landley
http://landley.net/hg/toybox/rev/433
It conflicts with your patch. If there's still an aliasing warning for
you, can you send me a patch on top of the current repository?
maybe my hg is broken but I don't think it is. I did a update and then
applied the patch to the current rev.

Doing update again it does nothing:
$ hg update
0 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg log toys/sort.c
changeset: 433:9e7aaecf0683
tag: tip
user: Rob Landley <rob at landley.net>
date: Tue Feb 07 00:31:37 2012 -0600
summary: Fix sort -uc (pointer vs pointer to pointer confusion, covered by typecast).

(I'm new to mercurial. I did also search a way of recovering the
timestamps. It took me some time until I got a line doing the job. rcs,
cvs and svn I used many years (seldom git) but I'm a newbie regarding
mercurial.)

I still got the same diff:
$ diff -u ../toybox-cloned-1328632507/toys/sort.c
toys/sort.c
--- ../toybox-cloned-1328632507/toys/sort.c 2012-02-07
17:35:07.000000000 +0100
+++ toys/sort.c 2012-02-07 17:37:41.000000000 +0100
@@ -308,7 +308,7 @@
if (CFG_SORT_BIG && (toys.optflags&FLAG_c)) {
int j = (toys.optflags&FLAG_u) ? -1 : 0;

- if (TT.lines && compare_keys((char **)&TT.lines, &line)>j)
+ if (TT.lines && compare_keys((void *)&TT.lines, &line)>j)

TT.lines is char** and used as char*, char is 8 bit / void "no bit" and
void* must be interchangeable with xxxxxx* - IMHO this must be valid c99 but
I didn't test it.

BTW - the last value of TT.lines is free()ed before assigning line as new
value. I guess you use malloc of the C-lib, aren't you? For tools with
low running times and small memory usage it could be worth using an
allocation function as wrapper like alloc() by DJB.
Instructing your authors to use this is far more easy than using a string
class: vim/sed does the job. ;-)
My local copy:
http://www.tuxad.com/ngtx/ngtx-current/lib/alloc.c
IMHO you know it already because DJB's daemontools package was mentioned
on the bb mailing list (Denis Vlasenko and runit).

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-14 03:45:10 UTC
Permalink
Post by Frank Bergmann
clone done. Patches submit to the list?
My first make did throw the nasty "dereferencing type-punned pointer will
break strict-aliasing rules". In sort.c you use TT.lines as char* and not
char**.
Unfortunately, when I cut and paste the patch you sent, the result is
whitespace damaged (all tabs expanded to spaces) so doesn't apply
cleanly, and when I save the email I get:

Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Errors-To: toybox-bounces at lists.landley.net
Sender: toybox-bounces at lists.landley.net

T24gV2VkLCBGZWIgMDgsIDIwMTIgYXQgMDk6MDA6MTBBTSArMDEwMCwgRnJhbmsgQmVyZ21hbm4g
d3JvdGU6Cj4gY2xvbmUgZG9uZS4gUGF0Y2hlcyBzdWJtaXQgdG8gdGhlIGxpc3Q/Cj4gTXkgZmly
c3QgbWFrZSBkaWQgdGhyb3cgdGhlIG5hc3R5ICJkZXJlZmVyZW5jaW5nIHR5cGUtcHVubmVkIHBv
aW50ZXIgd2lsbAo+IGJyZWFrIHN0cmljdC1hbGlhc2luZyBydWxlcyIuIEluIHNvcnQuYyB5b3Ug
dXNlIFRULmxpbmVzIGFzIGNoYXIqIGFuZCBub3QKPiBjaGFyKiouCgpIZXJlJ3MgdGhlIHN0dWZm

And so on...

Luckily it's simple enough to apply by hand, but we might want to
resolve this for future patches...

Rob
David Seikel
2012-02-14 03:50:45 UTC
Permalink
Post by Rob Landley
Post by Frank Bergmann
clone done. Patches submit to the list?
My first make did throw the nasty "dereferencing type-punned
pointer will break strict-aliasing rules". In sort.c you use
TT.lines as char* and not char**.
Unfortunately, when I cut and paste the patch you sent, the result is
whitespace damaged (all tabs expanded to spaces) so doesn't apply
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Errors-To: toybox-bounces at lists.landley.net
Sender: toybox-bounces at lists.landley.net
T24gV2VkLCBGZWIgMDgsIDIwMTIgYXQgMDk6MDA6MTBBTSArMDEwMCwgRnJhbmsgQmVyZ21hbm4g
d3JvdGU6Cj4gY2xvbmUgZG9uZS4gUGF0Y2hlcyBzdWJtaXQgdG8gdGhlIGxpc3Q/Cj4gTXkgZmly
c3QgbWFrZSBkaWQgdGhyb3cgdGhlIG5hc3R5ICJkZXJlZmVyZW5jaW5nIHR5cGUtcHVubmVkIHBv
aW50ZXIgd2lsbAo+IGJyZWFrIHN0cmljdC1hbGlhc2luZyBydWxlcyIuIEluIHNvcnQuYyB5b3Ug
dXNlIFRULmxpbmVzIGFzIGNoYXIqIGFuZCBub3QKPiBjaGFyKiouCgpIZXJlJ3MgdGhlIHN0dWZm
And so on...
Luckily it's simple enough to apply by hand, but we might want to
resolve this for future patches...
Patches to mailing lists sent as attached .txt files, as type text/plain
usually works the best I've found.
--
A big old stinking pile of genius that no one wants
coz there are too many silver coated monkeys in the world.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20120214/86a3a5aa/attachment.pgp>
Frank Bergmann
2012-02-14 07:25:26 UTC
Permalink
Hi,
Post by Rob Landley
Unfortunately, when I cut and paste the patch you sent, the result is
whitespace damaged (all tabs expanded to spaces) so doesn't apply
it is always possible that an editor has a bad default config but in this
case it is not the cause:

[fwb at vdr toybox]$ diff -u toybox/toys/sort.c toybox-cloned-1329201307/toys/sort.c
[fwb at vdr toybox]$ diff -u toybox/toys/sort.c toybox-cloned-1328632507/toys/sort.c
--- toybox/toys/sort.c 2012-02-07 17:33:27.000000000 +0100
+++ toybox-cloned-1328632507/toys/sort.c 2012-02-07
17:35:07.000000000 +0100
@@ -308,7 +308,7 @@
if (CFG_SORT_BIG && (toys.optflags&FLAG_c)) {
int j = (toys.optflags&FLAG_u) ? -1 : 0;

- if (TT.lines && compare_keys((void *)&TT.lines, &line)>j)
+ if (TT.lines && compare_keys((char **)&TT.lines, &line)>j)
error_exit("%s: Check line %d\n", name, TT.linecount);
free(TT.lines);
TT.lines = (char **)line;

(This was C&P and has of course space indents.)

I will check my mail config.

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
David Seikel
2012-02-14 03:50:45 UTC
Permalink
Post by Rob Landley
Post by Frank Bergmann
clone done. Patches submit to the list?
My first make did throw the nasty "dereferencing type-punned
pointer will break strict-aliasing rules". In sort.c you use
TT.lines as char* and not char**.
Unfortunately, when I cut and paste the patch you sent, the result is
whitespace damaged (all tabs expanded to spaces) so doesn't apply
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Errors-To: toybox-bounces at lists.landley.net
Sender: toybox-bounces at lists.landley.net
T24gV2VkLCBGZWIgMDgsIDIwMTIgYXQgMDk6MDA6MTBBTSArMDEwMCwgRnJhbmsgQmVyZ21hbm4g
d3JvdGU6Cj4gY2xvbmUgZG9uZS4gUGF0Y2hlcyBzdWJtaXQgdG8gdGhlIGxpc3Q/Cj4gTXkgZmly
c3QgbWFrZSBkaWQgdGhyb3cgdGhlIG5hc3R5ICJkZXJlZmVyZW5jaW5nIHR5cGUtcHVubmVkIHBv
aW50ZXIgd2lsbAo+IGJyZWFrIHN0cmljdC1hbGlhc2luZyBydWxlcyIuIEluIHNvcnQuYyB5b3Ug
dXNlIFRULmxpbmVzIGFzIGNoYXIqIGFuZCBub3QKPiBjaGFyKiouCgpIZXJlJ3MgdGhlIHN0dWZm
And so on...
Luckily it's simple enough to apply by hand, but we might want to
resolve this for future patches...
Patches to mailing lists sent as attached .txt files, as type text/plain
usually works the best I've found.
--
A big old stinking pile of genius that no one wants
coz there are too many silver coated monkeys in the world.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20120214/86a3a5aa/attachment-0002.pgp>
Frank Bergmann
2012-02-14 07:25:26 UTC
Permalink
Hi,
Post by Rob Landley
Unfortunately, when I cut and paste the patch you sent, the result is
whitespace damaged (all tabs expanded to spaces) so doesn't apply
it is always possible that an editor has a bad default config but in this
case it is not the cause:

[fwb at vdr toybox]$ diff -u toybox/toys/sort.c toybox-cloned-1329201307/toys/sort.c
[fwb at vdr toybox]$ diff -u toybox/toys/sort.c toybox-cloned-1328632507/toys/sort.c
--- toybox/toys/sort.c 2012-02-07 17:33:27.000000000 +0100
+++ toybox-cloned-1328632507/toys/sort.c 2012-02-07
17:35:07.000000000 +0100
@@ -308,7 +308,7 @@
if (CFG_SORT_BIG && (toys.optflags&FLAG_c)) {
int j = (toys.optflags&FLAG_u) ? -1 : 0;

- if (TT.lines && compare_keys((void *)&TT.lines, &line)>j)
+ if (TT.lines && compare_keys((char **)&TT.lines, &line)>j)
error_exit("%s: Check line %d\n", name, TT.linecount);
free(TT.lines);
TT.lines = (char **)line;

(This was C&P and has of course space indents.)

I will check my mail config.

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-09 01:34:10 UTC
Permalink
Post by Frank Bergmann
clone done. Patches submit to the list?
My first make did throw the nasty "dereferencing type-punned pointer will
break strict-aliasing rules". In sort.c you use TT.lines as char* and not
char**.
I fixed the sort bug yesterday morning:

http://landley.net/hg/toybox/rev/433

It conflicts with your patch. If there's still an aliasing warning for
you, can you send me a patch on top of the current repository?

Thanks,

Rob
Rob Landley
2012-02-14 03:45:10 UTC
Permalink
Post by Frank Bergmann
clone done. Patches submit to the list?
My first make did throw the nasty "dereferencing type-punned pointer will
break strict-aliasing rules". In sort.c you use TT.lines as char* and not
char**.
Unfortunately, when I cut and paste the patch you sent, the result is
whitespace damaged (all tabs expanded to spaces) so doesn't apply
cleanly, and when I save the email I get:

Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Errors-To: toybox-bounces at lists.landley.net
Sender: toybox-bounces at lists.landley.net

T24gV2VkLCBGZWIgMDgsIDIwMTIgYXQgMDk6MDA6MTBBTSArMDEwMCwgRnJhbmsgQmVyZ21hbm4g
d3JvdGU6Cj4gY2xvbmUgZG9uZS4gUGF0Y2hlcyBzdWJtaXQgdG8gdGhlIGxpc3Q/Cj4gTXkgZmly
c3QgbWFrZSBkaWQgdGhyb3cgdGhlIG5hc3R5ICJkZXJlZmVyZW5jaW5nIHR5cGUtcHVubmVkIHBv
aW50ZXIgd2lsbAo+IGJyZWFrIHN0cmljdC1hbGlhc2luZyBydWxlcyIuIEluIHNvcnQuYyB5b3Ug
dXNlIFRULmxpbmVzIGFzIGNoYXIqIGFuZCBub3QKPiBjaGFyKiouCgpIZXJlJ3MgdGhlIHN0dWZm

And so on...

Luckily it's simple enough to apply by hand, but we might want to
resolve this for future patches...

Rob
Rob Landley
2012-02-08 17:59:43 UTC
Permalink
Post by Frank Bergmann
Hi.
Post by Rob Landley
But in this case, I plan to continue using the printf() family because
doing without it wouldn't actually simplify the code, I'd just wind up
writing something just as bad.
In this case you may use more control to use printf as desired. I'd
$ grep -r setvbuf busybox|wc -l
9
$ grep -r setvbuf toybox-cloned-1328632507|wc -l
0
I have no idea what that means. I pulled up the man page on setvbuf and
it says to me "micromanaging the behavior of stdio", which there's could
be a reason for but it wouldn't be my first choice. However, a
defconfig busybox is 348 commands last time I checked, and defconfig
toybox is 45. It's quite possible some of those busybox commands
legitimately need to do setvbuf, dunno.

(You have to define boundaries for your project. Beyond this point is
Somebody Else's Problem. I'd rather not have "libc's stdio sucks" on my
sideof that line just now. Ask Dmitriy. :)
Post by Frank Bergmann
Post by Rob Landley
C and string handling are not a good mix. String handling is the thing
C is weakest at, because string handling actually turns out to be a hard
problem to get right at the hardware level.
We all know that C actually doesn't know anything about "strings". ;-)
Writing "bigger" software it *may* be worth implementing strings as a
class in C (er... struct) like Wietse Venema does. But this means to
rewrite all code.
And it means teach all your developers about the new semantics as a
condition of them contributing to your program, which is a big cost.
I've got enough of that already, and I'm explicitly leveraging the large
pool of existing knowledge of things like Kconfig syntax so the actual
cost isn't quite as high as it could be.

I want to make the code easy to _read_, which means I've got to be good
enough to make it look easy. I regularly fail at this, but am trying...

P.S. According to Joel Spolsky, the history of C++ Standard Template
Library (and C++ template support in general) can be explained by a long
protracted fight with "making strings work". Every change is due to yet
another way they found their String class du jour was broken. It's
still broken.
Post by Frank Bergmann
Post by Rob Landley
Yeah, but after 40 years of being grandfathered in, it's still useful
enough to stick around.
Yeah, but 99% of software out in the wild breaks the basic rules KISS and
YAGNI and introduce (sometimes many) bugs and holes with this.
Sturgeon's Law is universal.
Post by Frank Bergmann
Post by Rob Landley
I run things under strace rather a lot, but even if it boiled down to
doing a for() loop around write(1, &char, 1) I'd tell you to fix your
libc rather than change what I was doing.
OK, this self-written printf-implementation you are stracing is not very
well optimized. ;-)
lib/libc.c get_line() which I have a pending question about which I hope
to answer this evening. (No time right now...)
Post by Frank Bergmann
Post by Rob Landley
I'm not that interested in micromanaging something that most likely
stays in L1 cache either way.
Hmmm... if your code fits in L1-cache right before doing a sysenter then
the cache will be dirty when doing the call of sysenter, isn't it?
What do you mean "dirty"?

The Linux kernel has a hugepage TLB entry covering the whole kernel
executable code, which is permanently there, which is one reason going
into the kernel is so cheap on Linux: no page table tree walk to switch
between kernel/user space, thanks to the pinned kernel TLB entries.
(Yeah,that leaves less for userspace but it's a good tradeoff.) Yeah,
you flip permission bits in those entries but that's cheap. Note that
doing this is why you have the 2G/2G split on 32 bit systems: the
kernel's virtual address space is always mapped when you're in user
space, it's just inaccessable permission-wise. So you can run out of
virtual address space (or have to go to "high memory" which the kernel
has to create a temporary mapping for in order to access it), but it
makes system calls really cheap.

The actual cache lines faulted in are another matter, but if you make a
lot of syscalls at least the entry point tends to stay in L1, and beyond
that it's no different than function calls.
Post by Frank Bergmann
I never measured applications touching this "problem" mostly because of
- hard to measure L1 cache running an OS with many tasks and not many
cores
- Loops already fit in L1 cache and did not call code "outside", running
so fast that you can't measure
CPU cycle counter. Run it on a quiescent system and see how many cycles
it took. (The phrase to google for is "linux microbenchmark" or
something like that.)


There was an awful lot of discussion of this when the P4 came out
because it turned out the pentium 4 SUCKED, in a lot of non-obvious
ways. Also the "to prefetch or not to prefetch" wars, which lwn.net
covered.
Post by Frank Bergmann
- reducing the amount of syscalls brought most speedup, other changes were
only ambiguous measurable
- too many syscalls which can't be reduced let the advantage of the cache
vanish (mostly showing big i/o waits in top at the core the application
runs)
- some causes where further optimizations didn't make any sense (due to
e.g. network latencies)
As you wrote: You'll have to measure it (all). Until then you must keep
caches in mind.
You have to understand what the system is doing. You also have to
realize that different hardware works in different ways.

However "system calls" are an unavoidable hot path everywhere, and have
been optimized within an inch of their lives. If you can reduce them by
an order of magnitude maybe you've got something, but if you're fighting
to reduce them 20% it's probably not worth complicating your code.
Post by Frank Bergmann
Post by Rob Landley
That's the worst you've seen?
Yes, I didn't expect it on rrdtool. But after stracing it I understand the
cause why it is actually running slow even on big hosts with many
rrd-updates even though fadvise/madvise should catch these cases.
Post by Rob Landley
Never run strace on gcc. Certainly not
gcc is one of the tools I never *wanted* to strace because I already
expected a nightmare (other tools are e.g. php). ;-)
The horror is indescribable. But I tried in my blog...
Post by Frank Bergmann
Post by Rob Landley
See lib/lib.h
clone done. Patches submit to the list?
Yup. :)
Post by Frank Bergmann
My first make did throw the nasty "dereferencing type-punned pointer will
break strict-aliasing rules". In sort.c you use TT.lines as char* and not
char**.
Sigh. I suspected some compilers would do that. The one I'm doing here
isn't warning about it, but the hexagon target would actually _break_
when you do this.

I hate that error. There is nothing WRONG with type-punning a pointer,
your darn optimizer is being too clever for its own good, and if there's
a -fstoptryingtodothat I would happily add it to CFLAGS and move on with
my life. (I don't care about the performance change, IT'S VALID C!)
Post by Frank Bergmann
Post by Rob Landley
command line stuff. It's only a win if you never have to do it more
than once.
That's why I often used small internal output buffers and the nasty
stpcpy.
What's nasty? It's in POSIX 2008.
Post by Frank Bergmann
BTW - I've read that pivot_root doesn't have a high priority in your
TODO-list. It's very easy to implement cause Linux offers a syscall.
The internal implementation of that syscall is disgusting, because it
has to examine and potentially modify the state of every process on the
system.

Basically, I need to patch the kernel to make chroot() actually do
something different, and integrate the patch upstream in order to
obsolete switch_root():

http://landley.net/notes-2011.html#02-06-2011

Note that adjusting the process-local mount tree wasn't possible until
A) there was a process-local mount tree, B) --bind mounts had been
invented so you can split a mount point.

But once you _have_ got it, reference counting should automatically
unmount orphaned filesystems, and a reparent_to_init() variant can move
kernel threads into initramfs. (Which is pretty much where they should
_always_ be anyway, that's why we've got initramfs.)
Post by Frank Bergmann
Older
glibc didn't offer a wrapping but this is also easy to check (and to
implement if necessary).
The one and only test along those lines I've implemented so far (just
added last week) adds a CONFIG symbol for a capability, and makes other
things depend on that symbol. So "unshare" drops out when the compile
probe fails to find the relevant constants in the header.

I'm pretty happy doing that for pivot_root as well: if your libc hasn't
got the syscall, fix your libc.
Post by Frank Bergmann
I want to write it as my first toy-code if no one
else is working on it.
Next thing could be mount even though it's not that easy.
I already wrote the busybox mount command, which had BUCKETS of corner
case behavior I care about getting right. My old implementation wasn't
a clean from-scratch rewrite (instead I replaced all the exisiting code
in about three passes) so I can't just port it and have to redo it. But
I have plans for that one and would like to do it myself.
Post by Frank Bergmann
Post by Rob Landley
Or you could do toys/cat.c using the global toybuf[4096] which is part
of the bss and only gets its page faulted in if it's actually dirtied.
(Modulo alignment considerations I haven't bothered about.)
I've read your docs but know I did also the clone and read some code. :-)
Post by Rob Landley
As I think I said in design.html, I'm replying on c99, posix-2008, and
LP64. (If I wasn't clear enough there tell me and I'll go fix it.)
No, it is clear. It was just a big bunch of docs. I yet don't know POSIX
2008, only 2001. I think there is much more "deprecated".
I'm more interested in the new capabilities that got added, and as long
as I'm implementing standardized command line utilities I might as well
implement (a defined subset of) the most recent standard.

Note that the common thing about all three of those? Available free on
the web. If it's not available free on the web, IT ISN'T A STANDARD.
Post by Frank Bergmann
Post by Rob Landley
the first commands I wrote for toybox. (Actually I started it for
busybox but left that project before it was finished, so never submitted
it.)
Maybe they will backport some day. ;-)
Eh, I tried to push stuff upstream into busybox back when I mothballed
toybox. Got a bit of it up, but:

http://landley.net/notes-2010.html#11-03-2010

(The "chirping crickets" paragraph.)

I met Denys in person at CELF in April and _explained_ to him how toybox
puts everything in one darn file and busybox needed you to touch five
files, and he sort of got it, and this eventually resulted in:

http://lists.busybox.net/pipermail/busybox/2010-May/072386.html

But I really like my syntax better, and "pick up that shirt" is not the
same as "clean your room" when they can't tell it's dirty...

I got toybox's "patch" implementation upstream because Aboriginal Linux
_needed_ that (the one in busybox was a joke), but my enthusiasm beyond
that gradually waned again:

http://landley.net/notes-2010.html#05-01-2010
http://landley.net/notes-2011.html#08-06-2011

The problem is busybox became a tool lots of people depend on, and those
guys happily monkey-patch it to get it to work for them, and they don't
care if it's clean they care if it _works_, and Denys has sort of gone
over to their side rather than fighting for "clean code" as the primary
objective. He's let simplicity become about job 4, after features and
speed and executable size.

I consider fast and small to mostly be _functions_ of the simplest
implementation you can manage, and treat complexity as a cost you spend
to get features, with some features not being worth the cost. (Yes, this
means I wind up pushing back on the userbase a bit, but as Alan Cox
said, "a maintainer's job is to say no".)
Post by Frank Bergmann
Post by Rob Landley
Never heard of it. I've got a strlcpy() but everybody does since
strncpy() isn't guaranteed to null terminate the output.)
Better you forget this nasty thing. The man-page says that it is a
GNU-extension and that it maybe goes back to the old msdos times...
Post by Rob Landley
http://pubs.opengroup.org/onlinepubs/9699919799/functions/stpcpy.html
er... maybe this is a april fools day joke
Nope, it makes sense. The implementation is trivial:

char *stpcpy(char *to, char *from)
{
while (*to = *from++) to++;
return to;
}
Post by Frank Bergmann
Post by Rob Landley
That's really cool. Thanks. I wonder where that's needed in lib/* or
Don't forget the name "msdos" in its history. ;-)
Up through DOS 3 Paul Allen was in charge so it wasn't so bad. Keep in
mind DOS was about the best you could do on the original IBM PC with
16-64k of ram. (That's kilobytes, not megabytes. Paul Allen wanted to
do xenix but it just didn't have the memory.)

Paul Allen came down with Hodgkins Lymphoma and quit the company in
1983, leaving Gates and his laptog Ballmer in charge, at which point the
techies were reporting to marketing. But before that there was actual
technical merit in the company, viewed through the context they were
operating in.

This is also the reason Dos 4, 5, and 6 were essentially interchangeable
with 3. No new useful innovation came out of Microsoft for the next 20
years with exactly one exception:

http://blogs.msdn.com/larryosterman/archive/2005/02/02/365635.aspx

Did I mention comptuer history is a hobby of mine?

http://landley.net/history/mirror
Post by Frank Bergmann
Post by Rob Landley
Did you read http://landley.net/toybox/design.html yet? Do I need to
fluff that out a bit?
Sorry, it was a bunch of docs. First I read the stuff easily linked and
then the urls you posted.
I have been working on toybox for a number of years, there's an awful
lot of context. The code that's there is what I managed to distill all
the design work _down_ to.
Post by Frank Bergmann
Post by Rob Landley
all the same darn thing just like ANSI/ISO C is the same standard
approved by two standards bodies...)
Yes, we should be glad there is not a DIN standard yet. ;-)
I can presumably avoid caring. (Just because a standard body emits
something doesn't mean I have to pay attention. SUSv4, C99, and LP64
are standards I _like_.)
Post by Frank Bergmann
Post by Rob Landley
Trust me: I know how to profile stuff, and how to understand the
I do. Before I read your opinion about tumb programmers I already tried to
think so.
My experiences are mainly the results of writing some monitoring tools
which sometimes can cause i/o wait, or writing a fast fgrep where I
measured that the size of the buffer is a great killer if you want to
speed it up. :-)
Proper batch sizes have been one of the main performance knobs for 60
years. There was a marvelous talk Maddog gave at LinuxWorld Expo in
2001 about speeding up an old reel-to-reel tape backup job from taking
all day to being done in minutes (and from requiring many tapes to
fitting on one) by changing single byte writes into kilobyte block
writes, because for each write the tapes would spin up to speed (leaving
a gap), write a start of transaction, write the data byte, write an end
of transaction, and then spin to a stop (more gap). Writing a kilobyte
at a time this didn't dominate, writing a byte at a time it did.

But most optimization is a moving target. Precalculating trigonometry
tables for video games (so you could point your ship any direction in
360 degrees) was great... until the processors started getting a CPU
cache and the calculation that stayed in cache was faster than the table
lookup that had to fault in a cache line. Then the cache grew large
enough that the whole table fit in memory again and the table was once
again faster! Until machines got floating point coprocessors, and then
the table was slower again...

My response? Do the simple thing. That particular ping/pong cycle I
noticed was 15 years ago, that's 10 iterations of moore's law, I.E. 10
DOUBLINGS of processor speed, so everything they were optimizing for is
utterly irrelevant and people have since implemented an emulator IN
FLASH that runs the old binaries...

Fabrice Bellard (the creator of tinycc and qemu) wrote i386 emulator,
booting Linux, ENTIRELY IN JAVASCRIPT:

http://bellard.org/jslinux

On a modern PC, it's reasonably snappy. (It would look snappier if he'd
compiled his kernel with EARLY_PRINTK, but oh well. Not his area.)

Back up and think about that for a moment. What is the _goal_ here?
Post by Frank Bergmann
Post by Rob Landley
Which is why they changed it so gettimeofday() can just read an atomic
Yes, I know this. But even if gettimeofday is not a "young" call there are
many, many projects which didn't recognize.
I still use it in some of my tools but only one time and not many times
and even more not many times a second.
I'm writing new code. I care about "given the current context I'm
writing it in, what's the best way to write it now?" I don't really
care about optimizing for obsolete stuff or trying to predict what
changes will happen in future. I'm trying to write it so it can be
easily _changed_ in future, by keeping it simple and easy to read/modify.

You said that printf() violated an early unix maxim, but there's another
one: "When in doubt, use brute force". Implement, _then_ optimize.

Rob
Frank Bergmann
2012-02-09 08:06:21 UTC
Permalink
Hi.
Post by Rob Landley
Somebody Else's Problem. I'd rather not have "libc's stdio sucks" on my
Just in case that you maybe misunderstood me: I'm trying to say that you
don't use the full power of stdio. Maybe you could enable at least line
buffering and read whole lines with scanf. Maybe it can substitue reading
one char at a time. I don't know, I'll have to read the stdio docs again.
I also don't know current implementations in C-libs.
Post by Rob Landley
The actual cache lines faulted in are another matter, but if you make a
lot of syscalls at least the entry point tends to stay in L1, and beyond
Ah! This is the answer. I thought that at least the entry point would be
seldom in l1 cache.
Post by Rob Landley
CPU cycle counter. Run it on a quiescent system and see how many cycles
it took. (The phrase to google for is "linux microbenchmark" or
something like that.)
I know.
On all projects running on 32 but Intel I have this line in the central .h
file:
#define rdtscl(low) __asm__ __volatile__ ("rdtsc" : "=a" (low) : : "edx")
But you still have to do many test runs to get useful results on a system
with many tasks.
Post by Rob Landley
You have to understand what the system is doing. You also have to
realize that different hardware works in different ways.
Running my software on different embedded platforms and getting
immediately errors like segfaults (i.e. on Linux/atmel) teached me some
experiences. ;-)
Post by Rob Landley
The horror is indescribable. But I tried in my blog...
I already thought of you as a fan of horror movies after the last post. ;-)
Post by Rob Landley
I hate that error. There is nothing WRONG with type-punning a pointer,
One of the things programmers like is to satisfy compilers. ;-)
Post by Rob Landley
my life. (I don't care about the performance change, IT'S VALID C!)
Are you sure? For K&R I'm sure that it is but IMHO c99 "requires" the
usage of void in this case (using char** as char*).
Post by Rob Landley
Post by Frank Bergmann
Post by Rob Landley
command line stuff. It's only a win if you never have to do it more
than once.
That's why I often used small internal output buffers and the nasty
stpcpy.
What's nasty? It's in POSIX 2008.
You're not the only one who was surprised because of that.
("nasty" because I thought it was a GNU extension of the lib with msdos in
history.)
Post by Rob Landley
The internal implementation of that syscall is disgusting, because it
has to examine and potentially modify the state of every process on the
system.
Sure but in i.e. initrd it means mostly two or three processes. Of course
it is disgusting and only useful in rare cirumstances but we have a
syscall and we are writing userspace software. :-)
Post by Rob Landley
http://landley.net/notes-2011.html#02-06-2011
http://www.tuxad.com/ngtx/ngtx-current/tools/breakout.asm
:-)
(Of course you need the capability to break out.)
Post by Rob Landley
Note that adjusting the process-local mount tree wasn't possible until
A) there was a process-local mount tree, B) --bind mounts had been
invented so you can split a mount point.
*shudder* some weeks ago I had to deal with some big servers with dozens
of containers and even more bindmounts and kernel of the early 2.6.2x
series with no full bindmount support.

But what I doesn't understand right now: Are you fighting against
pivot_root? I just mentioned it as a starting point for me.
Post by Rob Landley
I have plans for that one and would like to do it myself.
unmount is left. ;-) I'll peek into the list again.
Post by Rob Landley
Note that the common thing about all three of those? Available free on
the web. If it's not available free on the web, IT ISN'T A STANDARD.
... and it is expensive. ;-)
Of course. Before I "detected" it I'd written my own function. But like
you I searched for a "standard function".
Post by Rob Landley
http://landley.net/history/mirror
Bookmarked. I'm glad to see that you did mention "Space Travel" (not Space
Wars like Linus wrote). :-)
Post by Rob Landley
Fabrice Bellard (the creator of tinycc and qemu) wrote i386 emulator,
He is already carved in stone in the history of computers. :-)
Post by Rob Landley
You said that printf() violated an early unix maxim, but there's another
one: "When in doubt, use brute force". Implement, _then_ optimize.
It was MAYBE breaking a rule by Doug McIlroy! ;-)

BTW - if you're interested in computer history: Do you have a (valid)
source for this Kernighan citate? In the Bell Labs documents I did not
found it.

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-11 20:28:15 UTC
Permalink
Post by Frank Bergmann
Hi.
Post by Rob Landley
Somebody Else's Problem. I'd rather not have "libc's stdio sucks" on my
Just in case that you maybe misunderstood me: I'm trying to say that you
don't use the full power of stdio. Maybe you could enable at least line
buffering and read whole lines with scanf.
Line input is something I mean to revisit.

One of the problems is the terminator isn't always \n, when you do "find
. -print0 | xargs 0" the terminator is null bytes. This is actually
extremely useful: it avoids all the horrible whitespace parsing issues
where filenames can have newlines in them. (There are only two
characters filenames can't include: NUL and / (the directory separator),
anything else is fair game in Linux, and I think in posix.)
Post by Frank Bergmann
Maybe it can substitue reading
one char at a time. I don't know, I'll have to read the stdio docs again.
I also don't know current implementations in C-libs.
The downside of reading one char at a time is it's slow. One of the
biggest issues of getting good performance is having reasonable
transaction sizes, so you amortize the per-transaction overhead. (What
counts as "reasonable" is a big performance tuning issue, but page size
is a good sweet spot processors are tuned for.)
Post by Frank Bergmann
Post by Rob Landley
The actual cache lines faulted in are another matter, but if you make a
lot of syscalls at least the entry point tends to stay in L1, and beyond
Ah! This is the answer. I thought that at least the entry point would be
seldom in l1 cache.
It's as hot as hot paths get, it tends to stick around unless you're
going out of your way to avoid it.
Post by Frank Bergmann
Post by Rob Landley
CPU cycle counter. Run it on a quiescent system and see how many cycles
it took. (The phrase to google for is "linux microbenchmark" or
something like that.)
I know.
On all projects running on 32 but Intel I have this line in the central .h
#define rdtscl(low) __asm__ __volatile__ ("rdtsc" : "=a" (low) : : "edx")
But you still have to do many test runs to get useful results on a system
with many tasks.
There was a marvelous thread on linux-kernel a few years ago about
microbenchmarks _shifting_ between intel processor revisions. This was
before the 64 bit stuff came out, they were looking at Pentium, Pentium
Pro, PII, PIII, P4, and Athlon. (And the P4 was a giant 'island of
suck' outlier in pretty much everything.)

My takeaway from it was "don't microoptimize". Keep things like cache
behavior and L1 vs L2 vs DRAM in mind on a general level, know what the
_options_ are and try not to trip over anything obvious.

Here's another one of those yin/yang cycle things: processors doing
speculative execution and register renaming and such to keep multiple
execution units busy are much happier with:

x = 42;
if (blah) x = 37;

Than with:

x = blah ? 42 : 37.

Because _with_ multiple register profiles they can copy the register
profile, speculatively perform the assignment _this_ clock cycle in one
copy and leave the other alone, and then either save or discard the
results when they get the results back from the test. This lets them do
more stuff in parallel, and thus go faster.

It also eats power performing computations they ain't gonna keep, so the
processors that are worried about the power to performance ratio instead
of the price to performance ratio (best bang for the watt instead of for
the dollar) get better performance by not wasting energy on work they
don't know if they need yet.

So if you optimize too much for x86-64, you may actually slow _down_ arm.

In reality, your compiler's optimizer will happily turn "x = blah ? a :
b" into "x = a; if (blah) x = b" if that's what's best for this
architecture. I try not to get in its way if it feels like doing that.
Post by Frank Bergmann
Post by Rob Landley
You have to understand what the system is doing. You also have to
realize that different hardware works in different ways.
Running my software on different embedded platforms and getting
immediately errors like segfaults (i.e. on Linux/atmel) teached me some
experiences. ;-)
Ah, alignment issues.

This really should have been called "why cross compiling sucks":

http://landley.net/writing/docs/cross-compiling.html
Post by Frank Bergmann
Post by Rob Landley
The horror is indescribable. But I tried in my blog...
I already thought of you as a fan of horror movies after the last post. ;-)
Not really. I get enough and feel no need to see it _out_.

I _am_ a fan of movies like "aliens" and "terminator II" though, which
support my theory that the main difference between an action movie and a
horror movie is how heavily armed the protagonist is.
Post by Frank Bergmann
Post by Rob Landley
I hate that error. There is nothing WRONG with type-punning a pointer,
One of the things programmers like is to satisfy compilers. ;-)
Nah, I occasionally humor them at best.

But if the compiler is beign stupid,the compiler is being stupid.
Post by Frank Bergmann
Post by Rob Landley
my life. (I don't care about the performance change, IT'S VALID C!)
Are you sure? For K&R I'm sure that it is but IMHO c99 "requires" the
usage of void in this case (using char** as char*).
Can you point me to where in the spec?

http://landley.net/c99-draft.html

I note that the latest variant is available here:

http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf

Apparently they're still plinking away at it. But A) if they changed
anything my compiler won't do it, I started caring about c99 around 2005
when compiler vendors had a chance to catch up, B) it's PDF which isn't
very searchable.
Post by Frank Bergmann
Post by Rob Landley
Post by Frank Bergmann
Post by Rob Landley
command line stuff. It's only a win if you never have to do it more
than once.
That's why I often used small internal output buffers and the nasty
stpcpy.
What's nasty? It's in POSIX 2008.
You're not the only one who was surprised because of that.
("nasty" because I thought it was a GNU extension of the lib with msdos in
history.)
And bzero() is obsolete but does what I need. I have a portability.h
header that may turn into a portability.c file that implements things in
various environments that don't have them.
Post by Frank Bergmann
Post by Rob Landley
The internal implementation of that syscall is disgusting, because it
has to examine and potentially modify the state of every process on the
system.
Sure but in i.e. initrd it means mostly two or three processes.
The vanilla containers control package lxc uses pivot_root() to do its
chroot because the chroot() syscall is flawed:

http://yarchive.net/comp/linux/pivot_root.html

Which is where:

http://landley.net/notes-2011.html#02-06-2011

Came from. (Yes, Linus Torvalds personally had to explain how the guts
of chroot() worked when I was writing switch_root for busybox, because
apparently nobody else _knew_.)
Post by Frank Bergmann
Of course
it is disgusting and only useful in rare cirumstances but we have a
syscall and we are writing userspace software. :-)
Or if you're working around flaws in chroot() and don't feel like
properly _fixing_ it. (Which is still on my todo list...)
Post by Frank Bergmann
Post by Rob Landley
http://landley.net/notes-2011.html#02-06-2011
http://www.tuxad.com/ngtx/ngtx-current/tools/breakout.asm
:-)
(Of course you need the capability to break out.)
Post by Rob Landley
Note that adjusting the process-local mount tree wasn't possible until
A) there was a process-local mount tree, B) --bind mounts had been
invented so you can split a mount point.
*shudder* some weeks ago I had to deal with some big servers with dozens
of containers and even more bindmounts and kernel of the early 2.6.2x
series with no full bindmount support.
But what I doesn't understand right now: Are you fighting against
pivot_root? I just mentioned it as a starting point for me.
I'm not fighting against it, I'm saying I want to _fix_ chroot now that
we have the infrastructure for it to really do what it seems like it
would do.
Post by Frank Bergmann
Post by Rob Landley
I have plans for that one and would like to do it myself.
unmount is left. ;-) I'll peek into the list again.
I need to update http://elinux.org/Busybox_replacement but what I
_really_ need to do is take the big master todo list at
http://elinux.org/Busybox_replacement#Command_List and break it up the
way I did http://landley.net/toybox/todos/susv4.txt into "low hanging
fruit", "medium hanging fruit", and "fiddly" commands.

With more explanation of _why_ each medium/fiddly command isn't
low-hanging. (Also, the low/medium ones are what it would take _me_ to
do them: mount and sed are easier for me because I already implemented
those before. Each one took about three months to figure out what I was
doing...)
Post by Frank Bergmann
Post by Rob Landley
Note that the common thing about all three of those? Available free on
the web. If it's not available free on the web, IT ISN'T A STANDARD.
... and it is expensive. ;-)
Of course. Before I "detected" it I'd written my own function. But like
you I searched for a "standard function".
Eh, when I first learned C I wrote my own strlcpy() because I needed it.
This was on Turbo C for DOS, _years_ before I'd heard of BSD. (Whether
the "l" stood for "length" or "landley", I never had to come down on one
side or the other of...)
Post by Frank Bergmann
Post by Rob Landley
http://landley.net/history/mirror
Bookmarked. I'm glad to see that you did mention "Space Travel" (not Space
Wars like Linus wrote). :-)
He was conflating "sky" and the PDP-1 thing Slug Russell wrote. Both of
which are important in their own way...
Post by Frank Bergmann
Post by Rob Landley
Fabrice Bellard (the creator of tinycc and qemu) wrote i386 emulator,
He is already carved in stone in the history of computers. :-)
Post by Rob Landley
You said that printf() violated an early unix maxim, but there's another
one: "When in doubt, use brute force". Implement, _then_ optimize.
It was MAYBE breaking a rule by Doug McIlroy! ;-)
BTW - if you're interested in computer history: Do you have a (valid)
source for this Kernighan citate? In the Bell Labs documents I did not
found it.
That's because it was Ken Thompson, not Kernighan.

http://catb.org/~esr/writings/taoup/html/ch01s06.html#id2877917

(It was also mentioned in The Unix Philosophy by Mike Gancarz, but I was
there when Eric Raymond _opened_ the mail package containing the draft
manuscript of The Art of Unix Programming which Ken Thompson had
handwritten his review notes on. We didn't have incense or anything, but
it was definitely treated as a holy relic...)

That bit about "One of my most productive days was throwing away 1000
lines of code." was ken's hand-written citation, in red pen, as a margin
comment to the section that now quotes it.

Rob
Frank Bergmann
2012-02-13 00:07:04 UTC
Permalink
Hi,

On Sat, Feb 11, 2012 at 02:28:15PM -0600, Rob Landley wrote:
[...]
Post by Rob Landley
x = 42;
if (blah) x = 37;
x = blah ? 42 : 37.
... at least it is better readable if you use this nice kernel macros
__likely() and unlikely(). ;-)
Post by Rob Landley
Ah, alignment issues.
No! This was also my first thought. But I have alignments where they are
needed. A pre-allocated buffer of 1 MB seems to be too big for an AP7000
processor. :-)
Post by Rob Landley
Post by Frank Bergmann
Are you sure? For K&R I'm sure that it is but IMHO c99 "requires" the
usage of void in this case (using char** as char*).
Can you point me to where in the spec?
Did I wrote "IMHO"? ;-) K&R IMHO doesn't know anything about void. A
pointer char* was the universal pointer which may be "converted" (without
explicit casting) to any other pointer and vice versa. C99 (and the
standard before it) IMHO introduced void at least for this role. And IMHO
a non-explicit cast is only allowed for void* and not for char* anymore.
Post by Rob Landley
http://landley.net/c99-draft.html
6.3.2.3 #1 describes the first part.
Maybe #7 may be interpreted as char* is not the "universal" pointer any
more.
sigh...
Post by Rob Landley
He was conflating "sky" and the PDP-1 thing Slug Russell wrote. Both of
which are important in their own way...
Of course I know it. But Linus' research actually wasn't correct. :-)

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-15 12:10:38 UTC
Permalink
Post by Frank Bergmann
Hi,
[...]
Post by Rob Landley
x = 42;
if (blah) x = 37;
x = blah ? 42 : 37.
... at least it is better readable if you use this nice kernel macros
__likely() and unlikely(). ;-)
The kernel is always running when any program does, so
micro-optimizations they do affect every single load you ever run. The
kernel guys have a huge incentive to microoptimize... and then undo it
again when their microoptimizations bitrot:

http://lwn.net/Articles/419102/
http://lwn.net/Articles/404103/
http://lwn.net/Articles/444336/
http://lwn.net/Articles/166172/

I like to be aware of this sort of thing so I don't gratuitously screw
it up, but I don't want to try to micromanage the compiler because
different builds (for different chips with different compiler versions)
need different things.
Post by Frank Bergmann
Post by Rob Landley
Post by Frank Bergmann
Are you sure? For K&R I'm sure that it is but IMHO c99 "requires" the
usage of void in this case (using char** as char*).
Can you point me to where in the spec?
Did I wrote "IMHO"? ;-) K&R IMHO doesn't know anything about void. A
pointer char* was the universal pointer which may be "converted" (without
explicit casting) to any other pointer and vice versa. C99 (and the
standard before it) IMHO introduced void at least for this role. And IMHO
a non-explicit cast is only allowed for void* and not for char* anymore.
This is the _compiler_ goign overboard with microoptimizations, if you
ask me. I treat it the same way I treat the "may be used uninitialized,
even though it isn't in this case" warning. (Which you can apparently
disable with "int x=x;" in your declarations, which is a syntax
specifically to tell it to shut up about that, but still way too magic
for my tastes.)

Rob
Frank Bergmann
2012-02-15 12:45:14 UTC
Permalink
Hi Rob,
Post by Rob Landley
Post by Frank Bergmann
... at least it is better readable if you use this nice kernel macros
__likely() and unlikely(). ;-)
[...]
Post by Rob Landley
http://lwn.net/Articles/419102/
http://lwn.net/Articles/404103/
http://lwn.net/Articles/444336/
http://lwn.net/Articles/166172/
- I guess my smilies are far too small ;-)
- I just mentioned the kernel macro which uses the gcc-function
__builtin_expect which is useful if you don't use gcc-profiling and you
are pretty sure about your optimization (reduce branches by falling
through a conditional branch in most cases)
- your references are about prefetch, manually prefetches shouldn't be
done, of course I fully agree
Post by Rob Landley
Post by Frank Bergmann
Did I wrote "IMHO"? ;-) K&R IMHO doesn't know anything about void. A
pointer char* was the universal pointer which may be "converted" (without
explicit casting) to any other pointer and vice versa. C99 (and the
standard before it) IMHO introduced void at least for this role. And IMHO
a non-explicit cast is only allowed for void* and not for char* anymore.
This is the _compiler_ goign overboard with microoptimizations, if you
ask me. I treat it the same way I treat the "may be used uninitialized,
even though it isn't in this case" warning. (Which you can apparently
disable with "int x=x;" in your declarations, which is a syntax
specifically to tell it to shut up about that, but still way too magic
for my tastes.)
Full ACK, but you mixed up two points.
You asked me about sort.c, char**, char* and void*. I only said that IMHO
void* may be converted without casting to any other pointer. The
old-fashioned K&R C doesn't know void and uses char* for this "role"
(IMHO).

Frank
Frank Bergmann
2012-02-15 12:45:14 UTC
Permalink
Hi Rob,
Post by Rob Landley
Post by Frank Bergmann
... at least it is better readable if you use this nice kernel macros
__likely() and unlikely(). ;-)
[...]
Post by Rob Landley
http://lwn.net/Articles/419102/
http://lwn.net/Articles/404103/
http://lwn.net/Articles/444336/
http://lwn.net/Articles/166172/
- I guess my smilies are far too small ;-)
- I just mentioned the kernel macro which uses the gcc-function
__builtin_expect which is useful if you don't use gcc-profiling and you
are pretty sure about your optimization (reduce branches by falling
through a conditional branch in most cases)
- your references are about prefetch, manually prefetches shouldn't be
done, of course I fully agree
Post by Rob Landley
Post by Frank Bergmann
Did I wrote "IMHO"? ;-) K&R IMHO doesn't know anything about void. A
pointer char* was the universal pointer which may be "converted" (without
explicit casting) to any other pointer and vice versa. C99 (and the
standard before it) IMHO introduced void at least for this role. And IMHO
a non-explicit cast is only allowed for void* and not for char* anymore.
This is the _compiler_ goign overboard with microoptimizations, if you
ask me. I treat it the same way I treat the "may be used uninitialized,
even though it isn't in this case" warning. (Which you can apparently
disable with "int x=x;" in your declarations, which is a syntax
specifically to tell it to shut up about that, but still way too magic
for my tastes.)
Full ACK, but you mixed up two points.
You asked me about sort.c, char**, char* and void*. I only said that IMHO
void* may be converted without casting to any other pointer. The
old-fashioned K&R C doesn't know void and uses char* for this "role"
(IMHO).

Frank

Rob Landley
2012-02-15 12:10:38 UTC
Permalink
Post by Frank Bergmann
Hi,
[...]
Post by Rob Landley
x = 42;
if (blah) x = 37;
x = blah ? 42 : 37.
... at least it is better readable if you use this nice kernel macros
__likely() and unlikely(). ;-)
The kernel is always running when any program does, so
micro-optimizations they do affect every single load you ever run. The
kernel guys have a huge incentive to microoptimize... and then undo it
again when their microoptimizations bitrot:

http://lwn.net/Articles/419102/
http://lwn.net/Articles/404103/
http://lwn.net/Articles/444336/
http://lwn.net/Articles/166172/

I like to be aware of this sort of thing so I don't gratuitously screw
it up, but I don't want to try to micromanage the compiler because
different builds (for different chips with different compiler versions)
need different things.
Post by Frank Bergmann
Post by Rob Landley
Post by Frank Bergmann
Are you sure? For K&R I'm sure that it is but IMHO c99 "requires" the
usage of void in this case (using char** as char*).
Can you point me to where in the spec?
Did I wrote "IMHO"? ;-) K&R IMHO doesn't know anything about void. A
pointer char* was the universal pointer which may be "converted" (without
explicit casting) to any other pointer and vice versa. C99 (and the
standard before it) IMHO introduced void at least for this role. And IMHO
a non-explicit cast is only allowed for void* and not for char* anymore.
This is the _compiler_ goign overboard with microoptimizations, if you
ask me. I treat it the same way I treat the "may be used uninitialized,
even though it isn't in this case" warning. (Which you can apparently
disable with "int x=x;" in your declarations, which is a syntax
specifically to tell it to shut up about that, but still way too magic
for my tastes.)

Rob
Frank Bergmann
2012-02-13 00:07:04 UTC
Permalink
Hi,

On Sat, Feb 11, 2012 at 02:28:15PM -0600, Rob Landley wrote:
[...]
Post by Rob Landley
x = 42;
if (blah) x = 37;
x = blah ? 42 : 37.
... at least it is better readable if you use this nice kernel macros
__likely() and unlikely(). ;-)
Post by Rob Landley
Ah, alignment issues.
No! This was also my first thought. But I have alignments where they are
needed. A pre-allocated buffer of 1 MB seems to be too big for an AP7000
processor. :-)
Post by Rob Landley
Post by Frank Bergmann
Are you sure? For K&R I'm sure that it is but IMHO c99 "requires" the
usage of void in this case (using char** as char*).
Can you point me to where in the spec?
Did I wrote "IMHO"? ;-) K&R IMHO doesn't know anything about void. A
pointer char* was the universal pointer which may be "converted" (without
explicit casting) to any other pointer and vice versa. C99 (and the
standard before it) IMHO introduced void at least for this role. And IMHO
a non-explicit cast is only allowed for void* and not for char* anymore.
Post by Rob Landley
http://landley.net/c99-draft.html
6.3.2.3 #1 describes the first part.
Maybe #7 may be interpreted as char* is not the "universal" pointer any
more.
sigh...
Post by Rob Landley
He was conflating "sky" and the PDP-1 thing Slug Russell wrote. Both of
which are important in their own way...
Of course I know it. But Linus' research actually wasn't correct. :-)

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-11 20:28:15 UTC
Permalink
Post by Frank Bergmann
Hi.
Post by Rob Landley
Somebody Else's Problem. I'd rather not have "libc's stdio sucks" on my
Just in case that you maybe misunderstood me: I'm trying to say that you
don't use the full power of stdio. Maybe you could enable at least line
buffering and read whole lines with scanf.
Line input is something I mean to revisit.

One of the problems is the terminator isn't always \n, when you do "find
. -print0 | xargs 0" the terminator is null bytes. This is actually
extremely useful: it avoids all the horrible whitespace parsing issues
where filenames can have newlines in them. (There are only two
characters filenames can't include: NUL and / (the directory separator),
anything else is fair game in Linux, and I think in posix.)
Post by Frank Bergmann
Maybe it can substitue reading
one char at a time. I don't know, I'll have to read the stdio docs again.
I also don't know current implementations in C-libs.
The downside of reading one char at a time is it's slow. One of the
biggest issues of getting good performance is having reasonable
transaction sizes, so you amortize the per-transaction overhead. (What
counts as "reasonable" is a big performance tuning issue, but page size
is a good sweet spot processors are tuned for.)
Post by Frank Bergmann
Post by Rob Landley
The actual cache lines faulted in are another matter, but if you make a
lot of syscalls at least the entry point tends to stay in L1, and beyond
Ah! This is the answer. I thought that at least the entry point would be
seldom in l1 cache.
It's as hot as hot paths get, it tends to stick around unless you're
going out of your way to avoid it.
Post by Frank Bergmann
Post by Rob Landley
CPU cycle counter. Run it on a quiescent system and see how many cycles
it took. (The phrase to google for is "linux microbenchmark" or
something like that.)
I know.
On all projects running on 32 but Intel I have this line in the central .h
#define rdtscl(low) __asm__ __volatile__ ("rdtsc" : "=a" (low) : : "edx")
But you still have to do many test runs to get useful results on a system
with many tasks.
There was a marvelous thread on linux-kernel a few years ago about
microbenchmarks _shifting_ between intel processor revisions. This was
before the 64 bit stuff came out, they were looking at Pentium, Pentium
Pro, PII, PIII, P4, and Athlon. (And the P4 was a giant 'island of
suck' outlier in pretty much everything.)

My takeaway from it was "don't microoptimize". Keep things like cache
behavior and L1 vs L2 vs DRAM in mind on a general level, know what the
_options_ are and try not to trip over anything obvious.

Here's another one of those yin/yang cycle things: processors doing
speculative execution and register renaming and such to keep multiple
execution units busy are much happier with:

x = 42;
if (blah) x = 37;

Than with:

x = blah ? 42 : 37.

Because _with_ multiple register profiles they can copy the register
profile, speculatively perform the assignment _this_ clock cycle in one
copy and leave the other alone, and then either save or discard the
results when they get the results back from the test. This lets them do
more stuff in parallel, and thus go faster.

It also eats power performing computations they ain't gonna keep, so the
processors that are worried about the power to performance ratio instead
of the price to performance ratio (best bang for the watt instead of for
the dollar) get better performance by not wasting energy on work they
don't know if they need yet.

So if you optimize too much for x86-64, you may actually slow _down_ arm.

In reality, your compiler's optimizer will happily turn "x = blah ? a :
b" into "x = a; if (blah) x = b" if that's what's best for this
architecture. I try not to get in its way if it feels like doing that.
Post by Frank Bergmann
Post by Rob Landley
You have to understand what the system is doing. You also have to
realize that different hardware works in different ways.
Running my software on different embedded platforms and getting
immediately errors like segfaults (i.e. on Linux/atmel) teached me some
experiences. ;-)
Ah, alignment issues.

This really should have been called "why cross compiling sucks":

http://landley.net/writing/docs/cross-compiling.html
Post by Frank Bergmann
Post by Rob Landley
The horror is indescribable. But I tried in my blog...
I already thought of you as a fan of horror movies after the last post. ;-)
Not really. I get enough and feel no need to see it _out_.

I _am_ a fan of movies like "aliens" and "terminator II" though, which
support my theory that the main difference between an action movie and a
horror movie is how heavily armed the protagonist is.
Post by Frank Bergmann
Post by Rob Landley
I hate that error. There is nothing WRONG with type-punning a pointer,
One of the things programmers like is to satisfy compilers. ;-)
Nah, I occasionally humor them at best.

But if the compiler is beign stupid,the compiler is being stupid.
Post by Frank Bergmann
Post by Rob Landley
my life. (I don't care about the performance change, IT'S VALID C!)
Are you sure? For K&R I'm sure that it is but IMHO c99 "requires" the
usage of void in this case (using char** as char*).
Can you point me to where in the spec?

http://landley.net/c99-draft.html

I note that the latest variant is available here:

http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf

Apparently they're still plinking away at it. But A) if they changed
anything my compiler won't do it, I started caring about c99 around 2005
when compiler vendors had a chance to catch up, B) it's PDF which isn't
very searchable.
Post by Frank Bergmann
Post by Rob Landley
Post by Frank Bergmann
Post by Rob Landley
command line stuff. It's only a win if you never have to do it more
than once.
That's why I often used small internal output buffers and the nasty
stpcpy.
What's nasty? It's in POSIX 2008.
You're not the only one who was surprised because of that.
("nasty" because I thought it was a GNU extension of the lib with msdos in
history.)
And bzero() is obsolete but does what I need. I have a portability.h
header that may turn into a portability.c file that implements things in
various environments that don't have them.
Post by Frank Bergmann
Post by Rob Landley
The internal implementation of that syscall is disgusting, because it
has to examine and potentially modify the state of every process on the
system.
Sure but in i.e. initrd it means mostly two or three processes.
The vanilla containers control package lxc uses pivot_root() to do its
chroot because the chroot() syscall is flawed:

http://yarchive.net/comp/linux/pivot_root.html

Which is where:

http://landley.net/notes-2011.html#02-06-2011

Came from. (Yes, Linus Torvalds personally had to explain how the guts
of chroot() worked when I was writing switch_root for busybox, because
apparently nobody else _knew_.)
Post by Frank Bergmann
Of course
it is disgusting and only useful in rare cirumstances but we have a
syscall and we are writing userspace software. :-)
Or if you're working around flaws in chroot() and don't feel like
properly _fixing_ it. (Which is still on my todo list...)
Post by Frank Bergmann
Post by Rob Landley
http://landley.net/notes-2011.html#02-06-2011
http://www.tuxad.com/ngtx/ngtx-current/tools/breakout.asm
:-)
(Of course you need the capability to break out.)
Post by Rob Landley
Note that adjusting the process-local mount tree wasn't possible until
A) there was a process-local mount tree, B) --bind mounts had been
invented so you can split a mount point.
*shudder* some weeks ago I had to deal with some big servers with dozens
of containers and even more bindmounts and kernel of the early 2.6.2x
series with no full bindmount support.
But what I doesn't understand right now: Are you fighting against
pivot_root? I just mentioned it as a starting point for me.
I'm not fighting against it, I'm saying I want to _fix_ chroot now that
we have the infrastructure for it to really do what it seems like it
would do.
Post by Frank Bergmann
Post by Rob Landley
I have plans for that one and would like to do it myself.
unmount is left. ;-) I'll peek into the list again.
I need to update http://elinux.org/Busybox_replacement but what I
_really_ need to do is take the big master todo list at
http://elinux.org/Busybox_replacement#Command_List and break it up the
way I did http://landley.net/toybox/todos/susv4.txt into "low hanging
fruit", "medium hanging fruit", and "fiddly" commands.

With more explanation of _why_ each medium/fiddly command isn't
low-hanging. (Also, the low/medium ones are what it would take _me_ to
do them: mount and sed are easier for me because I already implemented
those before. Each one took about three months to figure out what I was
doing...)
Post by Frank Bergmann
Post by Rob Landley
Note that the common thing about all three of those? Available free on
the web. If it's not available free on the web, IT ISN'T A STANDARD.
... and it is expensive. ;-)
Of course. Before I "detected" it I'd written my own function. But like
you I searched for a "standard function".
Eh, when I first learned C I wrote my own strlcpy() because I needed it.
This was on Turbo C for DOS, _years_ before I'd heard of BSD. (Whether
the "l" stood for "length" or "landley", I never had to come down on one
side or the other of...)
Post by Frank Bergmann
Post by Rob Landley
http://landley.net/history/mirror
Bookmarked. I'm glad to see that you did mention "Space Travel" (not Space
Wars like Linus wrote). :-)
He was conflating "sky" and the PDP-1 thing Slug Russell wrote. Both of
which are important in their own way...
Post by Frank Bergmann
Post by Rob Landley
Fabrice Bellard (the creator of tinycc and qemu) wrote i386 emulator,
He is already carved in stone in the history of computers. :-)
Post by Rob Landley
You said that printf() violated an early unix maxim, but there's another
one: "When in doubt, use brute force". Implement, _then_ optimize.
It was MAYBE breaking a rule by Doug McIlroy! ;-)
BTW - if you're interested in computer history: Do you have a (valid)
source for this Kernighan citate? In the Bell Labs documents I did not
found it.
That's because it was Ken Thompson, not Kernighan.

http://catb.org/~esr/writings/taoup/html/ch01s06.html#id2877917

(It was also mentioned in The Unix Philosophy by Mike Gancarz, but I was
there when Eric Raymond _opened_ the mail package containing the draft
manuscript of The Art of Unix Programming which Ken Thompson had
handwritten his review notes on. We didn't have incense or anything, but
it was definitely treated as a holy relic...)

That bit about "One of my most productive days was throwing away 1000
lines of code." was ken's hand-written citation, in red pen, as a margin
comment to the section that now quotes it.

Rob
Frank Bergmann
2012-02-09 08:06:21 UTC
Permalink
Hi.
Post by Rob Landley
Somebody Else's Problem. I'd rather not have "libc's stdio sucks" on my
Just in case that you maybe misunderstood me: I'm trying to say that you
don't use the full power of stdio. Maybe you could enable at least line
buffering and read whole lines with scanf. Maybe it can substitue reading
one char at a time. I don't know, I'll have to read the stdio docs again.
I also don't know current implementations in C-libs.
Post by Rob Landley
The actual cache lines faulted in are another matter, but if you make a
lot of syscalls at least the entry point tends to stay in L1, and beyond
Ah! This is the answer. I thought that at least the entry point would be
seldom in l1 cache.
Post by Rob Landley
CPU cycle counter. Run it on a quiescent system and see how many cycles
it took. (The phrase to google for is "linux microbenchmark" or
something like that.)
I know.
On all projects running on 32 but Intel I have this line in the central .h
file:
#define rdtscl(low) __asm__ __volatile__ ("rdtsc" : "=a" (low) : : "edx")
But you still have to do many test runs to get useful results on a system
with many tasks.
Post by Rob Landley
You have to understand what the system is doing. You also have to
realize that different hardware works in different ways.
Running my software on different embedded platforms and getting
immediately errors like segfaults (i.e. on Linux/atmel) teached me some
experiences. ;-)
Post by Rob Landley
The horror is indescribable. But I tried in my blog...
I already thought of you as a fan of horror movies after the last post. ;-)
Post by Rob Landley
I hate that error. There is nothing WRONG with type-punning a pointer,
One of the things programmers like is to satisfy compilers. ;-)
Post by Rob Landley
my life. (I don't care about the performance change, IT'S VALID C!)
Are you sure? For K&R I'm sure that it is but IMHO c99 "requires" the
usage of void in this case (using char** as char*).
Post by Rob Landley
Post by Frank Bergmann
Post by Rob Landley
command line stuff. It's only a win if you never have to do it more
than once.
That's why I often used small internal output buffers and the nasty
stpcpy.
What's nasty? It's in POSIX 2008.
You're not the only one who was surprised because of that.
("nasty" because I thought it was a GNU extension of the lib with msdos in
history.)
Post by Rob Landley
The internal implementation of that syscall is disgusting, because it
has to examine and potentially modify the state of every process on the
system.
Sure but in i.e. initrd it means mostly two or three processes. Of course
it is disgusting and only useful in rare cirumstances but we have a
syscall and we are writing userspace software. :-)
Post by Rob Landley
http://landley.net/notes-2011.html#02-06-2011
http://www.tuxad.com/ngtx/ngtx-current/tools/breakout.asm
:-)
(Of course you need the capability to break out.)
Post by Rob Landley
Note that adjusting the process-local mount tree wasn't possible until
A) there was a process-local mount tree, B) --bind mounts had been
invented so you can split a mount point.
*shudder* some weeks ago I had to deal with some big servers with dozens
of containers and even more bindmounts and kernel of the early 2.6.2x
series with no full bindmount support.

But what I doesn't understand right now: Are you fighting against
pivot_root? I just mentioned it as a starting point for me.
Post by Rob Landley
I have plans for that one and would like to do it myself.
unmount is left. ;-) I'll peek into the list again.
Post by Rob Landley
Note that the common thing about all three of those? Available free on
the web. If it's not available free on the web, IT ISN'T A STANDARD.
... and it is expensive. ;-)
Of course. Before I "detected" it I'd written my own function. But like
you I searched for a "standard function".
Post by Rob Landley
http://landley.net/history/mirror
Bookmarked. I'm glad to see that you did mention "Space Travel" (not Space
Wars like Linus wrote). :-)
Post by Rob Landley
Fabrice Bellard (the creator of tinycc and qemu) wrote i386 emulator,
He is already carved in stone in the history of computers. :-)
Post by Rob Landley
You said that printf() violated an early unix maxim, but there's another
one: "When in doubt, use brute force". Implement, _then_ optimize.
It was MAYBE breaking a rule by Doug McIlroy! ;-)

BTW - if you're interested in computer history: Do you have a (valid)
source for this Kernighan citate? In the Bell Labs documents I did not
found it.

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Frank Bergmann
2012-02-08 10:36:26 UTC
Permalink
Post by Frank Bergmann
clone done. Patches submit to the list?
My first make did throw the nasty "dereferencing type-punned pointer will
break strict-aliasing rules". In sort.c you use TT.lines as char* and not
char**.
Here's the stuff:

Repository:
Make generated/config.h from .config.
Compile toybox...
toys/sort.c: In function ?sort_read?:
toys/sort.c:311: warning: dereferencing type-punned pointer will break strict-aliasing rules
[fwb at vdr toybox-cloned-1328632507]$ cd ../toybox

Local:
Make generated/config.h from .config.
Compile toybox...
[fwb at vdr toybox]$ ln -s toybox sort
[fwb at vdr toybox]$ echo -e '1\n2\n4\n3' >1
[fwb at vdr toybox]$ ./sort -c 1
sort: 1: Check line 3

[fwb at vdr toybox]$ diff -u ../toybox-cloned-1328632507/toys/sort.c toys/sort.c
--- ../toybox-cloned-1328632507/toys/sort.c 2012-02-07 17:35:07.000000000 +0100
+++ toys/sort.c 2012-02-07 17:37:41.000000000 +0100
@@ -308,7 +308,7 @@
if (CFG_SORT_BIG && (toys.optflags&FLAG_c)) {
int j = (toys.optflags&FLAG_u) ? -1 : 0;

- if (TT.lines && compare_keys((char **)&TT.lines, &line)>j)
+ if (TT.lines && compare_keys((void *)&TT.lines, &line)>j)
error_exit("%s: Check line %d\n", name, TT.linecount);
free(TT.lines);
TT.lines = (char **)line;

Your last patch which maybe introduced the warning:
diff -r 87edfe8ae99e -r 9e7aaecf0683 toys/sort.c
--- a/toys/sort.c Mon Feb 06 21:14:22 2012 -0600
+++ b/toys/sort.c Tue Feb 07 00:31:37 2012 -0600
@@ -308,11 +308,9 @@
if (CFG_SORT_BIG && (toys.optflags&FLAG_c)) {
int j = (toys.optflags&FLAG_u) ? -1 : 0;

- if (TT.linecount && compare_keys((char *)TT.lines,line)>j)
+ if (TT.lines && compare_keys((char **)&TT.lines, &line)>j)
error_exit("%s: Check line %d\n", name, TT.linecount);
-
- if (TT.lines) free(TT.lines);
- else TT.linecount = 0;
+ free(TT.lines);
TT.lines = (char **)line;
} else {
if (!(TT.linecount&63))
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-08 17:59:43 UTC
Permalink
Post by Frank Bergmann
Hi.
Post by Rob Landley
But in this case, I plan to continue using the printf() family because
doing without it wouldn't actually simplify the code, I'd just wind up
writing something just as bad.
In this case you may use more control to use printf as desired. I'd
$ grep -r setvbuf busybox|wc -l
9
$ grep -r setvbuf toybox-cloned-1328632507|wc -l
0
I have no idea what that means. I pulled up the man page on setvbuf and
it says to me "micromanaging the behavior of stdio", which there's could
be a reason for but it wouldn't be my first choice. However, a
defconfig busybox is 348 commands last time I checked, and defconfig
toybox is 45. It's quite possible some of those busybox commands
legitimately need to do setvbuf, dunno.

(You have to define boundaries for your project. Beyond this point is
Somebody Else's Problem. I'd rather not have "libc's stdio sucks" on my
sideof that line just now. Ask Dmitriy. :)
Post by Frank Bergmann
Post by Rob Landley
C and string handling are not a good mix. String handling is the thing
C is weakest at, because string handling actually turns out to be a hard
problem to get right at the hardware level.
We all know that C actually doesn't know anything about "strings". ;-)
Writing "bigger" software it *may* be worth implementing strings as a
class in C (er... struct) like Wietse Venema does. But this means to
rewrite all code.
And it means teach all your developers about the new semantics as a
condition of them contributing to your program, which is a big cost.
I've got enough of that already, and I'm explicitly leveraging the large
pool of existing knowledge of things like Kconfig syntax so the actual
cost isn't quite as high as it could be.

I want to make the code easy to _read_, which means I've got to be good
enough to make it look easy. I regularly fail at this, but am trying...

P.S. According to Joel Spolsky, the history of C++ Standard Template
Library (and C++ template support in general) can be explained by a long
protracted fight with "making strings work". Every change is due to yet
another way they found their String class du jour was broken. It's
still broken.
Post by Frank Bergmann
Post by Rob Landley
Yeah, but after 40 years of being grandfathered in, it's still useful
enough to stick around.
Yeah, but 99% of software out in the wild breaks the basic rules KISS and
YAGNI and introduce (sometimes many) bugs and holes with this.
Sturgeon's Law is universal.
Post by Frank Bergmann
Post by Rob Landley
I run things under strace rather a lot, but even if it boiled down to
doing a for() loop around write(1, &char, 1) I'd tell you to fix your
libc rather than change what I was doing.
OK, this self-written printf-implementation you are stracing is not very
well optimized. ;-)
lib/libc.c get_line() which I have a pending question about which I hope
to answer this evening. (No time right now...)
Post by Frank Bergmann
Post by Rob Landley
I'm not that interested in micromanaging something that most likely
stays in L1 cache either way.
Hmmm... if your code fits in L1-cache right before doing a sysenter then
the cache will be dirty when doing the call of sysenter, isn't it?
What do you mean "dirty"?

The Linux kernel has a hugepage TLB entry covering the whole kernel
executable code, which is permanently there, which is one reason going
into the kernel is so cheap on Linux: no page table tree walk to switch
between kernel/user space, thanks to the pinned kernel TLB entries.
(Yeah,that leaves less for userspace but it's a good tradeoff.) Yeah,
you flip permission bits in those entries but that's cheap. Note that
doing this is why you have the 2G/2G split on 32 bit systems: the
kernel's virtual address space is always mapped when you're in user
space, it's just inaccessable permission-wise. So you can run out of
virtual address space (or have to go to "high memory" which the kernel
has to create a temporary mapping for in order to access it), but it
makes system calls really cheap.

The actual cache lines faulted in are another matter, but if you make a
lot of syscalls at least the entry point tends to stay in L1, and beyond
that it's no different than function calls.
Post by Frank Bergmann
I never measured applications touching this "problem" mostly because of
- hard to measure L1 cache running an OS with many tasks and not many
cores
- Loops already fit in L1 cache and did not call code "outside", running
so fast that you can't measure
CPU cycle counter. Run it on a quiescent system and see how many cycles
it took. (The phrase to google for is "linux microbenchmark" or
something like that.)


There was an awful lot of discussion of this when the P4 came out
because it turned out the pentium 4 SUCKED, in a lot of non-obvious
ways. Also the "to prefetch or not to prefetch" wars, which lwn.net
covered.
Post by Frank Bergmann
- reducing the amount of syscalls brought most speedup, other changes were
only ambiguous measurable
- too many syscalls which can't be reduced let the advantage of the cache
vanish (mostly showing big i/o waits in top at the core the application
runs)
- some causes where further optimizations didn't make any sense (due to
e.g. network latencies)
As you wrote: You'll have to measure it (all). Until then you must keep
caches in mind.
You have to understand what the system is doing. You also have to
realize that different hardware works in different ways.

However "system calls" are an unavoidable hot path everywhere, and have
been optimized within an inch of their lives. If you can reduce them by
an order of magnitude maybe you've got something, but if you're fighting
to reduce them 20% it's probably not worth complicating your code.
Post by Frank Bergmann
Post by Rob Landley
That's the worst you've seen?
Yes, I didn't expect it on rrdtool. But after stracing it I understand the
cause why it is actually running slow even on big hosts with many
rrd-updates even though fadvise/madvise should catch these cases.
Post by Rob Landley
Never run strace on gcc. Certainly not
gcc is one of the tools I never *wanted* to strace because I already
expected a nightmare (other tools are e.g. php). ;-)
The horror is indescribable. But I tried in my blog...
Post by Frank Bergmann
Post by Rob Landley
See lib/lib.h
clone done. Patches submit to the list?
Yup. :)
Post by Frank Bergmann
My first make did throw the nasty "dereferencing type-punned pointer will
break strict-aliasing rules". In sort.c you use TT.lines as char* and not
char**.
Sigh. I suspected some compilers would do that. The one I'm doing here
isn't warning about it, but the hexagon target would actually _break_
when you do this.

I hate that error. There is nothing WRONG with type-punning a pointer,
your darn optimizer is being too clever for its own good, and if there's
a -fstoptryingtodothat I would happily add it to CFLAGS and move on with
my life. (I don't care about the performance change, IT'S VALID C!)
Post by Frank Bergmann
Post by Rob Landley
command line stuff. It's only a win if you never have to do it more
than once.
That's why I often used small internal output buffers and the nasty
stpcpy.
What's nasty? It's in POSIX 2008.
Post by Frank Bergmann
BTW - I've read that pivot_root doesn't have a high priority in your
TODO-list. It's very easy to implement cause Linux offers a syscall.
The internal implementation of that syscall is disgusting, because it
has to examine and potentially modify the state of every process on the
system.

Basically, I need to patch the kernel to make chroot() actually do
something different, and integrate the patch upstream in order to
obsolete switch_root():

http://landley.net/notes-2011.html#02-06-2011

Note that adjusting the process-local mount tree wasn't possible until
A) there was a process-local mount tree, B) --bind mounts had been
invented so you can split a mount point.

But once you _have_ got it, reference counting should automatically
unmount orphaned filesystems, and a reparent_to_init() variant can move
kernel threads into initramfs. (Which is pretty much where they should
_always_ be anyway, that's why we've got initramfs.)
Post by Frank Bergmann
Older
glibc didn't offer a wrapping but this is also easy to check (and to
implement if necessary).
The one and only test along those lines I've implemented so far (just
added last week) adds a CONFIG symbol for a capability, and makes other
things depend on that symbol. So "unshare" drops out when the compile
probe fails to find the relevant constants in the header.

I'm pretty happy doing that for pivot_root as well: if your libc hasn't
got the syscall, fix your libc.
Post by Frank Bergmann
I want to write it as my first toy-code if no one
else is working on it.
Next thing could be mount even though it's not that easy.
I already wrote the busybox mount command, which had BUCKETS of corner
case behavior I care about getting right. My old implementation wasn't
a clean from-scratch rewrite (instead I replaced all the exisiting code
in about three passes) so I can't just port it and have to redo it. But
I have plans for that one and would like to do it myself.
Post by Frank Bergmann
Post by Rob Landley
Or you could do toys/cat.c using the global toybuf[4096] which is part
of the bss and only gets its page faulted in if it's actually dirtied.
(Modulo alignment considerations I haven't bothered about.)
I've read your docs but know I did also the clone and read some code. :-)
Post by Rob Landley
As I think I said in design.html, I'm replying on c99, posix-2008, and
LP64. (If I wasn't clear enough there tell me and I'll go fix it.)
No, it is clear. It was just a big bunch of docs. I yet don't know POSIX
2008, only 2001. I think there is much more "deprecated".
I'm more interested in the new capabilities that got added, and as long
as I'm implementing standardized command line utilities I might as well
implement (a defined subset of) the most recent standard.

Note that the common thing about all three of those? Available free on
the web. If it's not available free on the web, IT ISN'T A STANDARD.
Post by Frank Bergmann
Post by Rob Landley
the first commands I wrote for toybox. (Actually I started it for
busybox but left that project before it was finished, so never submitted
it.)
Maybe they will backport some day. ;-)
Eh, I tried to push stuff upstream into busybox back when I mothballed
toybox. Got a bit of it up, but:

http://landley.net/notes-2010.html#11-03-2010

(The "chirping crickets" paragraph.)

I met Denys in person at CELF in April and _explained_ to him how toybox
puts everything in one darn file and busybox needed you to touch five
files, and he sort of got it, and this eventually resulted in:

http://lists.busybox.net/pipermail/busybox/2010-May/072386.html

But I really like my syntax better, and "pick up that shirt" is not the
same as "clean your room" when they can't tell it's dirty...

I got toybox's "patch" implementation upstream because Aboriginal Linux
_needed_ that (the one in busybox was a joke), but my enthusiasm beyond
that gradually waned again:

http://landley.net/notes-2010.html#05-01-2010
http://landley.net/notes-2011.html#08-06-2011

The problem is busybox became a tool lots of people depend on, and those
guys happily monkey-patch it to get it to work for them, and they don't
care if it's clean they care if it _works_, and Denys has sort of gone
over to their side rather than fighting for "clean code" as the primary
objective. He's let simplicity become about job 4, after features and
speed and executable size.

I consider fast and small to mostly be _functions_ of the simplest
implementation you can manage, and treat complexity as a cost you spend
to get features, with some features not being worth the cost. (Yes, this
means I wind up pushing back on the userbase a bit, but as Alan Cox
said, "a maintainer's job is to say no".)
Post by Frank Bergmann
Post by Rob Landley
Never heard of it. I've got a strlcpy() but everybody does since
strncpy() isn't guaranteed to null terminate the output.)
Better you forget this nasty thing. The man-page says that it is a
GNU-extension and that it maybe goes back to the old msdos times...
Post by Rob Landley
http://pubs.opengroup.org/onlinepubs/9699919799/functions/stpcpy.html
er... maybe this is a april fools day joke
Nope, it makes sense. The implementation is trivial:

char *stpcpy(char *to, char *from)
{
while (*to = *from++) to++;
return to;
}
Post by Frank Bergmann
Post by Rob Landley
That's really cool. Thanks. I wonder where that's needed in lib/* or
Don't forget the name "msdos" in its history. ;-)
Up through DOS 3 Paul Allen was in charge so it wasn't so bad. Keep in
mind DOS was about the best you could do on the original IBM PC with
16-64k of ram. (That's kilobytes, not megabytes. Paul Allen wanted to
do xenix but it just didn't have the memory.)

Paul Allen came down with Hodgkins Lymphoma and quit the company in
1983, leaving Gates and his laptog Ballmer in charge, at which point the
techies were reporting to marketing. But before that there was actual
technical merit in the company, viewed through the context they were
operating in.

This is also the reason Dos 4, 5, and 6 were essentially interchangeable
with 3. No new useful innovation came out of Microsoft for the next 20
years with exactly one exception:

http://blogs.msdn.com/larryosterman/archive/2005/02/02/365635.aspx

Did I mention comptuer history is a hobby of mine?

http://landley.net/history/mirror
Post by Frank Bergmann
Post by Rob Landley
Did you read http://landley.net/toybox/design.html yet? Do I need to
fluff that out a bit?
Sorry, it was a bunch of docs. First I read the stuff easily linked and
then the urls you posted.
I have been working on toybox for a number of years, there's an awful
lot of context. The code that's there is what I managed to distill all
the design work _down_ to.
Post by Frank Bergmann
Post by Rob Landley
all the same darn thing just like ANSI/ISO C is the same standard
approved by two standards bodies...)
Yes, we should be glad there is not a DIN standard yet. ;-)
I can presumably avoid caring. (Just because a standard body emits
something doesn't mean I have to pay attention. SUSv4, C99, and LP64
are standards I _like_.)
Post by Frank Bergmann
Post by Rob Landley
Trust me: I know how to profile stuff, and how to understand the
I do. Before I read your opinion about tumb programmers I already tried to
think so.
My experiences are mainly the results of writing some monitoring tools
which sometimes can cause i/o wait, or writing a fast fgrep where I
measured that the size of the buffer is a great killer if you want to
speed it up. :-)
Proper batch sizes have been one of the main performance knobs for 60
years. There was a marvelous talk Maddog gave at LinuxWorld Expo in
2001 about speeding up an old reel-to-reel tape backup job from taking
all day to being done in minutes (and from requiring many tapes to
fitting on one) by changing single byte writes into kilobyte block
writes, because for each write the tapes would spin up to speed (leaving
a gap), write a start of transaction, write the data byte, write an end
of transaction, and then spin to a stop (more gap). Writing a kilobyte
at a time this didn't dominate, writing a byte at a time it did.

But most optimization is a moving target. Precalculating trigonometry
tables for video games (so you could point your ship any direction in
360 degrees) was great... until the processors started getting a CPU
cache and the calculation that stayed in cache was faster than the table
lookup that had to fault in a cache line. Then the cache grew large
enough that the whole table fit in memory again and the table was once
again faster! Until machines got floating point coprocessors, and then
the table was slower again...

My response? Do the simple thing. That particular ping/pong cycle I
noticed was 15 years ago, that's 10 iterations of moore's law, I.E. 10
DOUBLINGS of processor speed, so everything they were optimizing for is
utterly irrelevant and people have since implemented an emulator IN
FLASH that runs the old binaries...

Fabrice Bellard (the creator of tinycc and qemu) wrote i386 emulator,
booting Linux, ENTIRELY IN JAVASCRIPT:

http://bellard.org/jslinux

On a modern PC, it's reasonably snappy. (It would look snappier if he'd
compiled his kernel with EARLY_PRINTK, but oh well. Not his area.)

Back up and think about that for a moment. What is the _goal_ here?
Post by Frank Bergmann
Post by Rob Landley
Which is why they changed it so gettimeofday() can just read an atomic
Yes, I know this. But even if gettimeofday is not a "young" call there are
many, many projects which didn't recognize.
I still use it in some of my tools but only one time and not many times
and even more not many times a second.
I'm writing new code. I care about "given the current context I'm
writing it in, what's the best way to write it now?" I don't really
care about optimizing for obsolete stuff or trying to predict what
changes will happen in future. I'm trying to write it so it can be
easily _changed_ in future, by keeping it simple and easy to read/modify.

You said that printf() violated an early unix maxim, but there's another
one: "When in doubt, use brute force". Implement, _then_ optimize.

Rob
David Seikel
2012-02-08 02:19:48 UTC
Permalink
Post by Rob Landley
The more eyeballs the merrier. I'm stretched a bit thin at present,
but trying to keep up... :)
Stretched a bit thin myself at the moment, which is why I've not gotten
around to actually helping out yet with toybox. And now my last
contract has run out of money still owing me for Decembers work. Need
a new contract. sigh
--
A big old stinking pile of genius that no one wants
coz there are too many silver coated monkeys in the world.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20120208/0c123897/attachment-0002.pgp>
Frank Bergmann
2012-02-08 08:00:10 UTC
Permalink
Hi.
Post by Rob Landley
But in this case, I plan to continue using the printf() family because
doing without it wouldn't actually simplify the code, I'd just wind up
writing something just as bad.
In this case you may use more control to use printf as desired. I'd
made a clone pf toybox and busybox and issued one command to compare:

$ grep -r setvbuf busybox|wc -l
9
$ grep -r setvbuf toybox-cloned-1328632507|wc -l
0
Post by Rob Landley
C and string handling are not a good mix. String handling is the thing
C is weakest at, because string handling actually turns out to be a hard
problem to get right at the hardware level.
We all know that C actually doesn't know anything about "strings". ;-)
Writing "bigger" software it *may* be worth implementing strings as a
class in C (er... struct) like Wietse Venema does. But this means to
rewrite all code.
Post by Rob Landley
Yeah, but after 40 years of being grandfathered in, it's still useful
enough to stick around.
Yeah, but 99% of software out in the wild breaks the basic rules KISS and
YAGNI and introduce (sometimes many) bugs and holes with this.
Post by Rob Landley
I run things under strace rather a lot, but even if it boiled down to
doing a for() loop around write(1, &char, 1) I'd tell you to fix your
libc rather than change what I was doing.
OK, this self-written printf-implementation you are stracing is not very
well optimized. ;-)
Post by Rob Landley
I'm not that interested in micromanaging something that most likely
stays in L1 cache either way.
Hmmm... if your code fits in L1-cache right before doing a sysenter then
the cache will be dirty when doing the call of sysenter, isn't it?
I never measured applications touching this "problem" mostly because of
that causes:
- hard to measure L1 cache running an OS with many tasks and not many
cores
- Loops already fit in L1 cache and did not call code "outside", running
so fast that you can't measure
- reducing the amount of syscalls brought most speedup, other changes were
only ambiguous measurable
- too many syscalls which can't be reduced let the advantage of the cache
vanish (mostly showing big i/o waits in top at the core the application
runs)
- some causes where further optimizations didn't make any sense (due to
e.g. network latencies)
As you wrote: You'll have to measure it (all). Until then you must keep
caches in mind.
Post by Rob Landley
That's the worst you've seen?
Yes, I didn't expect it on rrdtool. But after stracing it I understand the
cause why it is actually running slow even on big hosts with many
rrd-updates even though fadvise/madvise should catch these cases.
Post by Rob Landley
Never run strace on gcc. Certainly not
gcc is one of the tools I never *wanted* to strace because I already
expected a nightmare (other tools are e.g. php). ;-)
Post by Rob Landley
See lib/lib.h
clone done. Patches submit to the list?
My first make did throw the nasty "dereferencing type-punned pointer will
break strict-aliasing rules". In sort.c you use TT.lines as char* and not
char**.
Post by Rob Landley
command line stuff. It's only a win if you never have to do it more
than once.
That's why I often used small internal output buffers and the nasty
stpcpy.
BTW - I've read that pivot_root doesn't have a high priority in your
TODO-list. It's very easy to implement cause Linux offers a syscall. Older
glibc didn't offer a wrapping but this is also easy to check (and to
implement if necessary). I want to write it as my first toy-code if no one
else is working on it.
Next thing could be mount even though it's not that easy.
Post by Rob Landley
Or you could do toys/cat.c using the global toybuf[4096] which is part
of the bss and only gets its page faulted in if it's actually dirtied.
(Modulo alignment considerations I haven't bothered about.)
I've read your docs but know I did also the clone and read some code. :-)
Post by Rob Landley
As I think I said in design.html, I'm replying on c99, posix-2008, and
LP64. (If I wasn't clear enough there tell me and I'll go fix it.)
No, it is clear. It was just a big bunch of docs. I yet don't know POSIX
2008, only 2001. I think there is much more "deprecated".
Post by Rob Landley
the first commands I wrote for toybox. (Actually I started it for
busybox but left that project before it was finished, so never submitted
it.)
Maybe they will backport some day. ;-)
Post by Rob Landley
Never heard of it. I've got a strlcpy() but everybody does since
strncpy() isn't guaranteed to null terminate the output.)
Better you forget this nasty thing. The man-page says that it is a
GNU-extension and that it maybe goes back to the old msdos times...
Post by Rob Landley
http://pubs.opengroup.org/onlinepubs/9699919799/functions/stpcpy.html
er... maybe this is a april fools day joke
Post by Rob Landley
That's really cool. Thanks. I wonder where that's needed in lib/* or
Don't forget the name "msdos" in its history. ;-)
Post by Rob Landley
Did you read http://landley.net/toybox/design.html yet? Do I need to
fluff that out a bit?
Sorry, it was a bunch of docs. First I read the stuff easily linked and
then the urls you posted.
Post by Rob Landley
all the same darn thing just like ANSI/ISO C is the same standard
approved by two standards bodies...)
Yes, we should be glad there is not a DIN standard yet. ;-)
Post by Rob Landley
Trust me: I know how to profile stuff, and how to understand the
I do. Before I read your opinion about tumb programmers I already tried to
think so.

My experiences are mainly the results of writing some monitoring tools
which sometimes can cause i/o wait, or writing a fast fgrep where I
measured that the size of the buffer is a great killer if you want to
speed it up. :-)
Post by Rob Landley
Which is why they changed it so gettimeofday() can just read an atomic
Yes, I know this. But even if gettimeofday is not a "young" call there are
many, many projects which didn't recognize.
I still use it in some of my tools but only one time and not many times
and even more not many times a second.
Post by Rob Landley
http://lwn.net/Articles/192214/
Bookmarked. And - yes! - stat calls are the next "evil" calls which are
way too many called. Like times it is also a problem in the tool mentioned
above (not gcc but the other ;-) ).

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-08 01:26:22 UTC
Permalink
Post by Frank Bergmann
Hello,
Post by Rob Landley
That the real world has complex output formats, or that id's output
looks like that?
... that id's output should look like that (regarding POSIX). I know the
differences between POSIX and i.e. GNU only as described in man-page
sections 2 and 3. :-)
I'm happy to not implement gnu extensions unless somebody's actually
using them, but if it's in susv4 I need to document deviances and
explain why. I.E. that needs a good _excuse_ not to implement stuff.

(I've been reading the bionic source a bit, I sympathize with not
implementing internationalization, and blocking C++ makes me smile, but
I'm still going to need to supplement it a bit.)
Post by Frank Bergmann
Post by Rob Landley
I fail to see how that's an improvement.
- small code size
- faster execution due to reduced overhead
- often reduced stack size
- sometimes reduce amount of syscalls
We should not start a "printf-flame-war".
I totally sympathize with your goals, and I'm all for pushing the limits
and asking "what can we get away with not doing". I do that all the
time, I regularly have people point out things I missed, and I
constantly need to reevaluate stuff.

But in this case, I plan to continue using the printf() family because
doing without it wouldn't actually simplify the code, I'd just wind up
writing something just as bad.

C and string handling are not a good mix. String handling is the thing
C is weakest at, because string handling actually turns out to be a hard
problem to get right at the hardware level.
Post by Frank Bergmann
I just wanted to talk about some
experiences I've made (and Fefe and some others, too).
IMHO printf() is actually a wonder of existence because it breaks with
Doug McIlroy's great recommendation. ;-)
Yeah, but after 40 years of being grandfathered in, it's still useful
enough to stick around.
Post by Frank Bergmann
[...]
Guessing the number of syscalls a libcall produces at runtime should not
be done in a "released package" but while development.
I run things under strace rather a lot, but even if it boiled down to
doing a for() loop around write(1, &char, 1) I'd tell you to fix your
libc rather than change what I was doing.
Post by Frank Bergmann
Remember your links
and the article about the tumb programmer? :-) I don't know what printf()
is actually doing on c-lib abc or c-lib xyz. To get a little bit more
control there are libcalls like setvbuf() which I rarely found in code
using stdio out in the wild. Funny thing, isn't it? ;-)
I'm not that interested in micromanaging something that most likely
stays in L1 cache either way.
Post by Frank Bergmann
Post by Rob Landley
And that's a "performance over simplicity" mis-optimization. I only
care how many system calls it is when it's a hot path, and the ascii
FILE structure has a built-in buffer to batch that stuff up a bit.
Of course, taking printf as example. But you should always keep it in mind.
The worst thing I've seen was a trace of rrd-update on a Debian system.
More than 300 Syscalls due to dynamic loading and searching paths.
That's the worst you've seen? Never run strace on gcc. Certainly not
right after eating:

http://landley.net/notes-2006.html#03-11-2006

(This is why I wrote ccwrap.c in aboriginal linux. The pathological
path logic in gcc is not salvageable, and piling more crap on top
ever-higher in hopes the bad foundations will go away is how we got
Windows and microkernels.)
Post by Frank Bergmann
After
that one or two fadvise/madvise-calls and then the write() itself. I
wonder if this binary was ever measured. :-)
Post by Rob Landley
Well, for one thing the above functions aren't in the standard C
library, and the ones I'm using are.
[...]
Post by Rob Landley
the return value for failure". My largest divergence there is ditching
getopt(), but that's because my new infrastructure does all the work for
each command before it even runs, and getopt() required the command to
have a switch statement.
This is the thing I was talking / asking about: Which functions are
provided by the internal lib?
See lib/lib.h

I brought up get_optflags() in response to printf() because they're
conceptually similar: both bundle a lot of commonly used functionality
together into one place that does it right. Open-coding printf strikes
me as no different than having a for(;;) loop in every command to parse
command line stuff. It's only a win if you never have to do it more
than once.
Post by Frank Bergmann
Do you have any plans to use a special
and license-compatible library like libowfat (this is one I remember) or
do you write your own basic functions (like I do for some years)?
I write my own basic functions. I'd only use an external library if I
was willing to fork it, merge a copy in, and maintain it myself. Which
is almost guaranteed to be infrastructure in search of a user.

Minimal system bootstrapping is theoretically four things:

A) kernel
B) libc
C) POSIX command line
D) compiler

None of those should depend on anything outside that group. Way back
when I was hoping for "linux, uClibc, busybox, tinycc" for the four.
That didn't pan out, but I'm still working on keeping it simple.

Environmental dependencies are a form of complexity we're bad at
tracking, and which most people forget about, but that doesn't mean
they're any less important. (I've got code in sort that avoids pulling
in libm, which is supplied by libc, because it wasn't necessary in that
case.)
Post by Frank Bergmann
Post by Rob Landley
With your calls above, presumably you have a reason for not using
"strcat()" instead. Your buffer has a length but I don't know what it
is. Does it automatically flush itself or is there an implicit "now
write all this out" that you didn't include, and if so is filling the
buffer an error or an auto-flush that's going to do an implicit system
call anyway?
er... actually I don't know what fnord's buffer routines do. IMHO you
specify the size of the buffer (comparing setvbuf) and you also got a
buffer_flush (comparing fflush).
Writing something like cat you should use a buffer and libcalls managing
it.
Or you could do toys/cat.c using the global toybuf[4096] which is part
of the bss and only gets its page faulted in if it's actually dirtied.
(Modulo alignment considerations I haven't bothered about.)
Post by Frank Bergmann
Writing something like id you may use a small linebuffer within your
main function. It should be measured.
I put toybuf[] in so every command has access to a global 4k buffer they
don't have to malloc (on top of the DEFINE_GLOBALS()), so I avoid some
heap walks that way.

I note I'm not happy with the lib/dirtree.c stuff (which is the only
current lib/* thing that actualy depends on toybuf, everything else
leaves it for the command to use) because it's got a 4k PATH_MAX
limitation, which the linux kernel doesn't currently have. But at the
same time, it made the code noticeably simpler, and the correct fix is
to use openat() and similar anyway. I think I blogged about this...
Post by Frank Bergmann
Post by Rob Landley
What's handling write failures, both from the other end prematurely
hanging up and from "sigstop" causing short writes, possibly zero
length, which you have to check errno to distinguish the sort write from
EOF? And if you don't get the short writes correct then suddenly "tar c
blah | gzip | ssh" suddenly corrupts the tarball if you ctrl-z and then
fg the pipeline...
I know how to handle these cases if it should be necessary but I don't
know how printf will react - even in glibc! Nice question, I should
investigate now. :-)
I already did. See lib/lib.c function xprintf().
Post by Frank Bergmann
(BTW - that's one reason why I re-implmented some basic functions for
myself.)
Or you could read the spec and see what behavior is _required_ from a
conformant implementation. Back when the c99 committee was doing its
thing they posted drafts of the spec for public review and then demanded
money for the final version, so everybody kept the last draft version
and called it good. I have a copy on my website, and it says:

http://landley.net/c99-draft.html#7.19.6
Post by Frank Bergmann
The fprintf function returns the number of characters transmitted, or
a negative value if an output or encoding error occurred.
...
Post by Frank Bergmann
The printf function is equivalent to fprintf with the argument stdout
interposed before the arguments to printf.
(The man 3 printf page says it too, but that documents what glibc did
rather than what the standard requires.)

If there is a standard requiring behavior, I choose the right to require
conformance with that standard in the environments I build in. I'm
doing doing one of those insane ./configure stages with 8000 tests to
try to work around every possible kind of broken out there (which can't
be done). I'm letting the build break and telling you to fix your
environment.

I may tolerate specific environments that diverge from that standard
(I.E. bionic, and maybe macosx), but in that case I'm humoring the bugs
of a specific implementation while requiring everything else to conform
to the darn standard.

I do not allow UNKNOWN environments to violate the standard and still
expect to work. So if your argument is "this printf() is horrible" then
don't use it, and if your argument is "some unknown printf() might be
horrible someday" I can honestly say it's not my problem.

As I think I said in design.html, I'm replying on c99, posix-2008, and
LP64. (If I wasn't clear enough there tell me and I'll go fix it.)
Post by Frank Bergmann
Post by Rob Landley
have tried to document most of it. I'm not sure what this "I don't like
printf, let me open-code every occurrence of it" gets you. Why not
write a better printf?
That was one of the questions you may find between my lines. ;-)
Because the c99 standard supplies one for me and I use rather a lot of
it, including the printf("% *ld", len, val) stuff in df which was one of
the first commands I wrote for toybox. (Actually I started it for
busybox but left that project before it was finished, so never submitted
it.)
Post by Frank Bergmann
Do you have one or a license-compatible "foreign" implementation?
Is it worth to be done?
I'm not here to reimplement c99 unless I'm writing a C library or a
compiler. Toybox is neither.

The environments that are in front of me are uClibc (which I've patched
before and can push patches for upstream), glibc (which I'm building
against in the first pass), and bionic (which I expect to supplement
heavily and we'll see if Google wants patches or if I need a
libtrionic.c or something).
Post by Frank Bergmann
Post by Rob Landley
You mean like the xblah() wrappers so I avoid testing return codes, or
the way xsmprintf() is doing mostly the same thing asprintf() is but
that's one of the gnu/dammit extensions that I didn't want to rely on?
Yes, some kind of. Another extensions which I often use is stpcpy (shame
on me).
Never heard of it. I've got a strlcpy() but everybody does since
strncpy() isn't guaranteed to null terminate the output.)

Huh, apparently it's not an extension, it's in SUSv4:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/stpcpy.html

That's really cool. Thanks. I wonder where that's needed in lib/* or
toys/* (or main.c), and I wonder if uClibc or bionic have caught up yet?
Post by Frank Bergmann
This leads to the question which interface standard should be
used? POSIX? 1996 or 2001? What about functions which are "deprecated"
(i.e. gethostbyname) or "rejected" (i.e. clearenv)?
Did you read http://landley.net/toybox/design.html yet? Do I need to
fluff that out a bit?

(At a quick glance I see it says SUSv3 because when I wrote it, SUSv4
wasn't out yet. Yeah, could use some updating. And explaining how
POSIX-2008 and SUSv4 and the Open Group Base Specifications Issue 7 are
all the same darn thing just like ANSI/ISO C is the same standard
approved by two standards bodies...)
Post by Frank Bergmann
Post by Rob Landley
But really, "minimal number of syscalls" is speed to me, not simplicity.
Hello world is simple code, if the libc it's linked against chooses to
do a separate syscall for every character of output that's libc's
problem. If performance sucks I'll look into it, but I expect some
variation from libc to libc and if it sucks _fix_your_libc_ or use a
better one.
OK. Some foreign lib bashing in source code comments is always nice to
read. ;-)
I note that I pay no attention to dietlibc, and am unlikely to start.
Post by Frank Bergmann
Post by Rob Landley
I have a vague academic interest in MacOS X but I've already got mdev
It's the only OS I know where you can switch off ptrace! ;-)
Its BSD-API is sometimes frustrating.
Interesting,there doesn't seem to be a kernel CONFIG symbol for ptrace...

There should be. Possibly only with CONFIG_EMBEDDED, but there really
should be a way to configure that out...
Post by Frank Bergmann
Post by Rob Landley
upshifted from int 80 to SYSENTER to save a couple clock cycles, and
Don't talk about Slowlaris. They NEED threads because only threads are
actually lightweight as processes in Linux are. But even with sysenter
which is IMHO used on every linux system today
It's an x86 instruction and we're targeting ARM android devices, so "not
so much". Although I believe android has something similar.
Post by Frank Bergmann
you should still keep in
mind that syscalls are the most expensive calls.
Over on my other major hobby project, Aboriginal Linux, I analyzed the
performance of ./configure and determined that statically linking
busybox would speed up ./configure by around 20% due to the lower page
translation overhead from eliminating the self-modifying-code aspect of
dynamic linking that forced executable pages to be retranslated.

Trust me: I know how to profile stuff, and how to understand the
performance of things, and I am not worried about the overhead of a
dozen or so nonblocking syscalls where we're faulting in maybe three
cache lines from L2 to L1.

Implement, _then_ optimize. I'd like to worry about where the
bottlenecks actually _are_ rather than borrowing trouble.
Post by Frank Bergmann
Post by Rob Landley
http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html
That has gettid() at 100 nanoseconds or less...
Wow! This is really low. But out in the wild you found crappy software
(IMHO more than 90%) which does hundreds of time()-calls a second.
Which is why they changed it so gettimeofday() can just read an atomic
variable out of the vsyscall page:

http://www.win.tue.nl/~aeb/linux/lk/lk-4.html

(They were discussing it, anyway. I don't remember if that change went
through...)

Yeah, fixing the code not to do that is a good thign too, but for things
like video playback software you need short waits and to be woken up
often because it's what the program _does_...
Post by Frank Bergmann
And
sometimes when Nagios warns about raising contextswitches on a specific
host you detect software with an strace you would never dream of - even if
you dream of Freddy Kruger all night long. ;-)
I'm totally aware that most existing userspace software is crap:

http://lwn.net/Articles/192214/

However, you can go too far in over-optimizing (which is actually part
of the _reason_ so much of it's crap).

Software needs to remain flexible. If you don't understand why it's
doing things the way it's doing, you can't change it later and it gets
brittle and you do workarounds instead of fixes and it all falls apart.

Sometimes, doing a nice tight implementation winds up with knots that
are hard to undo. (Both of the current patch and sort implementations
are in that category, for example.) Sometimes, that's the local peak.
Making it simpler or easier to understand results in significantly more
code or signficantly more runtime. (I still try anyway, and at least
comment the house-of-cards bit.)

But complexity is a cost, I want to make sure we get the best bang for
the bug, and figuring out what simple _is_ can be really hard. (Simple
is elegant. I can't always come up with an elegant solution. Doesn't
mean I don't go back and polish the the dorodango until it gleams, and
sometimes I _do_ figure out how to make an actual gemstone out of it.)
Post by Frank Bergmann
Rob, you're making me hungry. I should co/clone the current toybox stuff.
:-)
The more eyeballs the merrier. I'm stretched a bit thin at present, but
trying to keep up... :)
Post by Frank Bergmann
Frank
Rob
Frank Bergmann
2012-02-06 15:31:08 UTC
Permalink
Hello,
Post by Rob Landley
That the real world has complex output formats, or that id's output
looks like that?
... that id's output should look like that (regarding POSIX). I know the
differences between POSIX and i.e. GNU only as described in man-page
sections 2 and 3. :-)
Post by Rob Landley
I fail to see how that's an improvement.
- small code size
- faster execution due to reduced overhead
- often reduced stack size
- sometimes reduce amount of syscalls

We should not start a "printf-flame-war". I just wanted to talk about some
experiences I've made (and Fefe and some others, too).
IMHO printf() is actually a wonder of existence because it breaks with
Doug McIlroy's great recommendation. ;-)

[...]

Guessing the number of syscalls a libcall produces at runtime should not
be done in a "released package" but while development. Remember your links
and the article about the tumb programmer? :-) I don't know what printf()
is actually doing on c-lib abc or c-lib xyz. To get a little bit more
control there are libcalls like setvbuf() which I rarely found in code
using stdio out in the wild. Funny thing, isn't it? ;-)
Post by Rob Landley
And that's a "performance over simplicity" mis-optimization. I only
care how many system calls it is when it's a hot path, and the ascii
FILE structure has a built-in buffer to batch that stuff up a bit.
Of course, taking printf as example. But you should always keep it in mind.
The worst thing I've seen was a trace of rrd-update on a Debian system.
More than 300 Syscalls due to dynamic loading and searching paths. After
that one or two fadvise/madvise-calls and then the write() itself. I
wonder if this binary was ever measured. :-)
Post by Rob Landley
Well, for one thing the above functions aren't in the standard C
library, and the ones I'm using are.
[...]
Post by Rob Landley
the return value for failure". My largest divergence there is ditching
getopt(), but that's because my new infrastructure does all the work for
each command before it even runs, and getopt() required the command to
have a switch statement.
This is the thing I was talking / asking about: Which functions are
provided by the internal lib? Do you have any plans to use a special
and license-compatible library like libowfat (this is one I remember) or
do you write your own basic functions (like I do for some years)?
Post by Rob Landley
With your calls above, presumably you have a reason for not using
"strcat()" instead. Your buffer has a length but I don't know what it
is. Does it automatically flush itself or is there an implicit "now
write all this out" that you didn't include, and if so is filling the
buffer an error or an auto-flush that's going to do an implicit system
call anyway?
er... actually I don't know what fnord's buffer routines do. IMHO you
specify the size of the buffer (comparing setvbuf) and you also got a
buffer_flush (comparing fflush).
Writing something like cat you should use a buffer and libcalls managing
it. Writing something like id you may use a small linebuffer within your
main function. It should be measured.
Post by Rob Landley
What's handling write failures, both from the other end prematurely
hanging up and from "sigstop" causing short writes, possibly zero
length, which you have to check errno to distinguish the sort write from
EOF? And if you don't get the short writes correct then suddenly "tar c
blah | gzip | ssh" suddenly corrupts the tarball if you ctrl-z and then
fg the pipeline...
I know how to handle these cases if it should be necessary but I don't
know how printf will react - even in glibc! Nice question, I should
investigate now. :-)
(BTW - that's one reason why I re-implmented some basic functions for
myself.)
Post by Rob Landley
have tried to document most of it. I'm not sure what this "I don't like
printf, let me open-code every occurrence of it" gets you. Why not
write a better printf?
That was one of the questions you may find between my lines. ;-)
Do you have one or a license-compatible "foreign" implementation?
Is it worth to be done?
Post by Rob Landley
You mean like the xblah() wrappers so I avoid testing return codes, or
the way xsmprintf() is doing mostly the same thing asprintf() is but
that's one of the gnu/dammit extensions that I didn't want to rely on?
Yes, some kind of. Another extensions which I often use is stpcpy (shame
on me). This leads to the question which interface standard should be
used? POSIX? 1996 or 2001? What about functions which are "deprecated"
(i.e. gethostbyname) or "rejected" (i.e. clearenv)?
Post by Rob Landley
But really, "minimal number of syscalls" is speed to me, not simplicity.
Hello world is simple code, if the libc it's linked against chooses to
do a separate syscall for every character of output that's libc's
problem. If performance sucks I'll look into it, but I expect some
variation from libc to libc and if it sucks _fix_your_libc_ or use a
better one.
OK. Some foreign lib bashing in source code comments is always nice to
read. ;-)
Post by Rob Landley
I have a vague academic interest in MacOS X but I've already got mdev
It's the only OS I know where you can switch off ptrace! ;-)
Its BSD-API is sometimes frustrating.
Post by Rob Landley
upshifted from int 80 to SYSENTER to save a couple clock cycles, and
Don't talk about Slowlaris. They NEED threads because only threads are
actually lightweight as processes in Linux are. But even with sysenter
which is IMHO used on every linux system today you should still keep in
mind that syscalls are the most expensive calls.
Post by Rob Landley
http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html
That has gettid() at 100 nanoseconds or less...
Wow! This is really low. But out in the wild you found crappy software
(IMHO more than 90%) which does hundreds of time()-calls a second. And
sometimes when Nagios warns about raising contextswitches on a specific
host you detect software with an strace you would never dream of - even if
you dream of Freddy Kruger all night long. ;-)

Rob, you're making me hungry. I should co/clone the current toybox stuff.
:-)

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-06 12:29:02 UTC
Permalink
Post by Frank Bergmann
er... forwarded to the list.
Hi Rob,
Post by Rob Landley
"uid=%u(%s) gid=%u(%s)\n", <real user ID>, <user-name>,
<real group ID>, <group-name>
er... I remember now. :-(
That the real world has complex output formats, or that id's output
looks like that?
Post by Frank Bergmann
Post by Rob Landley
above output sequence for "id" with no arguments would be 9 separate
output statements without printf. (Or 9 statements assembling a string,
of who knows what length, and then a tenth to write it out.)
Yes, this is a source code blow-up. But using your own calls it can also
be easily readable. I remember such sequences of statements in the source
buffer_puts(buffer_1,"http://");
buffer_puts(buffer_1,host);
buffer_puts(buffer_1,"/");
buffer_puts(buffer_1,url);
buffer_puts(buffer_1,"/\r\n\r\n");
I fail to see how that's an improvement.

Replacing one function call, which fits on maybe two lines, with half a
screen full of function calls (at 80x25 terminal size) means less code
fits on the screen at once so you've got a lot more scrolling to see the
same amount of program in front of you. When the function call in
question is part of the C standard and the replacement isn't...

And if you really really care how many write function calls:

sprintf(buffer_1, "http://%s/%s/\r\n\r\n", host, url);

Or xmsprintf() in lib/lib.c will xmalloc() the appropriate amount for you.
Post by Frank Bergmann
You know what it means and you get just one syscall. Using printf() it is
very hard to guess, how many syscalls one statement will use.
And that's a "performance over simplicity" mis-optimization. I only
care how many system calls it is when it's a hot path, and the ascii
FILE structure has a built-in buffer to batch that stuff up a bit.
Post by Frank Bergmann
Post by Rob Landley
Keep in mind that my primary design goal is _simplicity_, then size,
speed, and features. You have to trade these off against each other when
Yes, that was my question: Which kind of simplicity. ;-)
Well, for one thing the above functions aren't in the standard C
library, and the ones I'm using are.

I've been wrapping a number of standard library calls (heck,there's an
xprintf()) but using simple rules like "anything that starts with an x
will perror_exit() if it does not succeed, so you never have to check
the return value for failure". My largest divergence there is ditching
getopt(), but that's because my new infrastructure does all the work for
each command before it even runs, and getopt() required the command to
have a switch statement.

With your calls above, presumably you have a reason for not using
"strcat()" instead. Your buffer has a length but I don't know what it
is. Does it automatically flush itself or is there an implicit "now
write all this out" that you didn't include, and if so is filling the
buffer an error or an auto-flush that's going to do an implicit system
call anyway?

What's handling write failures, both from the other end prematurely
hanging up and from "sigstop" causing short writes, possibly zero
length, which you have to check errno to distinguish the sort write from
EOF? And if you don't get the short writes correct then suddenly "tar c
blah | gzip | ssh" suddenly corrupts the tarball if you ctrl-z and then
fg the pipeline...

Yes, toybox has a lot of implicit knowledge like that too, such as
toybuf being page sized (4k, although not necessarily page aligned, but
that's still an amount of data in a single transaction that caches tend
to be tuned for), and us relying on sigaction(SA_RESTART) to avoid zero
length read/write problems having to check errno everywhere (which isn't
fully implemented yet but is the goal). I know what it all gets us, and
have tried to document most of it. I'm not sure what this "I don't like
printf, let me open-code every occurrence of it" gets you. Why not
write a better printf?

(P.S. if you look at echo, xprintf(), xputc(), and xflush() all exist
because "echo > /dev/full" is supposed to return with a non-zero error
code. This is the sort of detail I'm trying to get right.)
Post by Frank Bergmann
Post by Rob Landley
And I want to let you build individual commands and have _those_ be as
small as possible, but in doing so I assume those will be dynamically
linked. I am not optimizing for the "multiple individual executables,
Even if you replace some "evil" libcalls of stdio with your own (internal)
lib you can still make code more small.
You mean like the xblah() wrappers so I avoid testing return codes, or
the way xsmprintf() is doing mostly the same thing asprintf() is but
that's one of the gnu/dammit extensions that I didn't want to rely on?
(And really, I mean to replace getline() too but that brings us to the
"must write line editing with command history for both the shell and vi,
and yes SUSv4 specifies vi".)
Post by Frank Bergmann
I always test my binaries after
big changes with different clibs and statically/dynamically and screw them
up with strace and/or ltrace. That's not every day business but doing so
gives you sometimes surprising results. :-)
Notice the prebuilt strace binaries for a bunch of different hardware
targets at http;//landley.net/aboriginal/bin because yes, strace is
extremely useful. I don't like to look at the gnu tool source to see how
they implemented anything (*shudder*), but I do occasionally run the
command my distro comes with under strace.

But really, "minimal number of syscalls" is speed to me, not simplicity.
Hello world is simple code, if the libc it's linked against chooses to
do a separate syscall for every character of output that's libc's
problem. If performance sucks I'll look into it, but I expect some
variation from libc to libc and if it sucks _fix_your_libc_ or use a
better one.
Post by Frank Bergmann
Post by Rob Landley
libc
libc
These libs are meant for internal usage but they are meant only as
examples. In my tools I replaced much stdio-stuff with own "lib" routines
and still have only libc as dependency. This lowers side effects when
people use very different c-libs.
I'm coding for Linux, and now android. On Linux, klibc is laughable,
dietlibc is broken, uClibc I'm testing against, the musl developers
presumably fix their stuff when they find something wrong with it, and
bionic I expect to have to work around but that's with some sort of
lib/makeitstop.c that implements things like "printf" according to the
darn standard.

I have a vague academic interest in MacOS X but I've already got mdev
and unshare and last time I wrote mount it had --bind and --move and
such, and used /proc/mounts instead of the obsolete /etc/mtab, all of
which are very linux specific.
Post by Frank Bergmann
Post by Rob Landley
if (flags & (FLAG_u|FLAG_g))
printf("%d\n", (flags & FLAG_u) ? uid | gid);
else printf("%d %d\n", uid, gid);
What about the syscalls? ;-)
Presumably it will make them for me.

Syscalls under linux are way way way cheaper than under crap like
Slowaris. The whole kernel's backed by a hugepage with a permanent TLB
entry so the transition's basically free from a cache standpoint, they
upshifted from int 80 to SYSENTER to save a couple clock cycles, and
this is 10 years _after_ doing optimizations like
http://cryptnet.net/mirrors/texts/kissedagirl.html that other operating
systems just ignored...

This is the kind of performance optimization work they were doing 10
years ago:

http://kerneltrap.org/node/384

And these days, most Linux system calls don't even cause a context
switch (I.E. cache flush and page table walk):

http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html

That has gettid() at 100 nanoseconds or less...

Rob
Rob Landley
2012-02-05 20:02:59 UTC
Permalink
I suck at this bcc: thing.
Post by Rob Landley
Hello Rob,
I read some list postings in the archive.
I write as PM because I don't want to "scare" one of the authors (Tim Bird).
Trust me, if the lwn.net coverage didn't scare him, I doubt honest code
review would make him blink. :)
I'll cc: this to the list but chop out your email, reply there if you
want to decloak. :)
- Does your goal "be simple" mean that the code must be very easy
readable? id.c IS very easy readable but in case of e.g. flags "u" and
"r" set you will make 4 syscalls when only 1 is needed. Shouldn't it
be "optimized"?
I tend to go through and do optimization passes on submissions. I
haven't gotten to id yet.
My "quick glance" when he sent it said "would putting a for(;;) loop
around only two printf calls actually improve matters, or is the
overhead of the loop going to make it larger? I need to bench it."
What I didn't do in the quick glance was notice that the options are
exclusive. As usual, the gnu/dammit version has a special error message
$ id -ug
id: cannot print "only" of more than one choice
Or that this id implementation is just a stub that is _not_ compliant
The following formats shall be used...
"uid=%u(%s) gid=%u(%s)\n", <real user ID>, <user-name>,
<real group ID>, <group-name>
So yeah, it needs a second pass in a big way. I should default it to
"n" in the meantime because it's not really there right now...
Instead I've been doing release cleanup. Specificially regression
testing the existing commands in my aboriginal linux setup, and trying
to figure out when I swap in each toybox command individually the build
works for all of them, but when I swap in all of them uClibc dies with
an internal compiler error and I get segfaults from "sort" and "xargs"
in dmesg.
Plus I'd like to finish the C rewrite of scripts/config2help.py so
python isn't a build prerequisite if "help" is enabled. And I had to
teach scripts/genconfig.sh to do a compile probe against the containers
infrastructure and disable unshare if the target toolchain headers
haven't got the appropriate #defines...
- Many coders who extremely dislike overhead (including myself) only use
libcall printf() for debugging purposes. It's some kind of
"deprecated".
A valid concern, and I did think about this. But
printf/asprintf/vsprintf/fprintf are pretty much all the same engine,
once it's pulled in for one use the overhead of using it in a second
place is trivial.
Something _like_ it would be unavoidable for perror_exit() and friends
(attempts to move complexity from the library to the callsite are a net
increase in overhead when you scale to many calls). For example, the
above output sequence for "id" with no arguments would be 9 separate
output statements without printf. (Or 9 statements assembling a string,
of who knows what length, and then a tenth to write it out.)
Keep in mind that my primary design goal is _simplicity_, then size,
speed, and features. You have to trade these off against each other when
implementing, but simplicity is the one I weight the highest, because in
the long run it's the one that makes the implementation understandable,
and thus maintainable (and security auditable). That means I tend to
approach commands with "what's the simplest way to implement this
feature", "what's the simplest way to speed this up", and so on.
Leveraging code people already wrote makes _toybox_ simple, otherwise
you spend eight lines of code doing what one line of code can do, and
you wind up repeating that all through your program. This code is
required to be there by c99, and it's required to have certain behavior.
There are _sucky_implementations_ of this code, but there are also
reasonable ones.
The linux kernel contains a reasonable printf() implementation, and I
feel comfortable writing my own if necessary. The uClibc implementation
of printf() is about 10k, based on binary size growth for a static i586
compile between "puts("Hello\n")" and "printf("Hello%d\n",0)". (Note
that printf() with a just constant string and no arguments gets
downshifted to a puts equivalent in uClibc, identical binary size.)
Yes, I'm balancing several definitions of "small" here. I want to make
toybox as a whole, statically linked, take up as little disk space as
possible. And I want to make it take up as little ram as possible at
runtime.
And I want to let you build individual commands and have _those_ be as
small as possible, but in doing so I assume those will be dynamically
linked. I am not optimizing for the "multiple individual executables,
each statically linked" case because it's _crazy_ and means I should
never leverage shared infrastructure, which is how you make the one big
statically linked executable small.
If your objection is that most people don't know how to _use_ these
tools in a reasonable manner: Yes, I'm aware of the newbie mistake the
sudo guys did recently. (Welcome to C, where we cook with knives and
fire.) I've got linked lists containing pointers from different
allocation contexts. (The struct arg_list is malloced from the heap,
the values it points to live in environment space. That's the normal
use case of lib/args.c, because I don't want to copy memory
unnecessarily. There's also the global toybuf[] and the stack without
even bringing mmap() into it. When you write in C, you have to know
what you're _doing_.)
Over on xargs I was at first trying to make a version that fit in
toybuf[] and upshifted to a malloc() when it outgrew the 4k static
buffer, but as long as I had to write the malloc() version anyway making
it the _only_ code path simplified things, so that's what I checked in.
I do a lot of that, weigh multiple approaches and then wind up doing
the _less_ microoptimized version because it's simpler and scales better
and performance is reasonable and yes I'm aware of nommu systems but
this mostly still runs on them, and trying to make a nommu system
reliable in low memory situations isn't worth complicating the common case.
Libs like libowfat or libdjb will reduce overhead while
not making the source hard to read. What's your opinion?
libc
libc
I'm all for finding a better libc, but I don't even depend on stuff like
zlib or ncurses. (I'm ok shelling out to an external gzip binary and
piping stuff through it; the only sane way to add https support to wget
and such would be via stunnel.)
(Part of my reasoning here is that at the lowest levels of the system
the dependnecies go circular, which makes bootstrapping hard, and you
wind up building stuff twice, which is nuts and one configuration
inevitably gets way less testing. Avoiding external dependencies in
these packages other than "I need a compiler with libc" means you don't
have to build toybox twice to bootstrap a system.)
Note how glibc had a "stage 1 compiler", but toybox didn't. (Sheesh,
perl's probably the silliest one: building microperl so perl's build
system can be written _in_perl_. The C compiler has a good excuse for
that, perl does not.)
Frank
/* show effective, unless user specifies real */
uid = geteuid();
gid = getegid();
if (flags & FLAG_r) {
uid = getuid();
gid = getgid();
}
if (flags & FLAG_u) {
printf("%d\n", uid);
return;
}
if (flags & FLAG_g) {
printf("%d\n", gid);
return;
}
printf("%d %d\n", uid, gid);
if (flags & (FLAG_u|FLAG_g))
printf("%d\n", (flags & FLAG_u) ? uid | gid);
else printf("%d %d\n", uid, gid);
Except the standard _also_ requires -G and -n. And a far more thorough
reading. (Yeah, it's a todo item.)
Rob
_______________________________________________
Toybox mailing list
Toybox at lists.landley.net
http://lists.landley.net/listinfo.cgi/toybox-landley.net
Frank Bergmann
2012-02-05 22:31:03 UTC
Permalink
er... forwarded to the list.
Hi Rob,
Post by Rob Landley
"uid=%u(%s) gid=%u(%s)\n", <real user ID>, <user-name>,
<real group ID>, <group-name>
er... I remember now. :-(
Post by Rob Landley
above output sequence for "id" with no arguments would be 9 separate
output statements without printf. (Or 9 statements assembling a string,
of who knows what length, and then a tenth to write it out.)
Yes, this is a source code blow-up. But using your own calls it can also
be easily readable. I remember such sequences of statements in the source
buffer_puts(buffer_1,"http://");
buffer_puts(buffer_1,host);
buffer_puts(buffer_1,"/");
buffer_puts(buffer_1,url);
buffer_puts(buffer_1,"/\r\n\r\n");
You know what it means and you get just one syscall. Using printf() it is
very hard to guess, how many syscalls one statement will use.
Post by Rob Landley
Keep in mind that my primary design goal is _simplicity_, then size,
speed, and features. You have to trade these off against each other when
Yes, that was my question: Which kind of simplicity. ;-)
Post by Rob Landley
And I want to let you build individual commands and have _those_ be as
small as possible, but in doing so I assume those will be dynamically
linked. I am not optimizing for the "multiple individual executables,
Even if you replace some "evil" libcalls of stdio with your own (internal)
lib you can still make code more small. I always test my binaries after
big changes with different clibs and statically/dynamically and screw them
up with strace and/or ltrace. That's not every day business but doing so
gives you sometimes surprising results. :-)
Post by Rob Landley
libc
libc
These libs are meant for internal usage but they are meant only as
examples. In my tools I replaced much stdio-stuff with own "lib" routines
and still have only libc as dependency. This lowers side effects when
people use very different c-libs.
Post by Rob Landley
if (flags & (FLAG_u|FLAG_g))
printf("%d\n", (flags & FLAG_u) ? uid | gid);
else printf("%d %d\n", uid, gid);
What about the syscalls? ;-)
Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Rob Landley
2012-02-07 13:32:34 UTC
Permalink
2012/2/5 Rob Landley <rob at landley.net <mailto:rob at landley.net>>
Thank you for your detailed response!
May be better create some small and
fast libc implementation to make you ab linux more small and easy
to use?
I've been pondering it. That's why I was looking at musl. But doing so
is not necessarily the same project as toybox. Toybox is replacing
toolbox. Replacing or supplementing _bionic_ is a separate issue.
I'm all for working with people who are trying, but the only guys I know
starting a new technically interesting libc right now are wasting their
time fighting for a fraction of uClibc's market share which is fighting
for a fraction of glibc's market share which is fighting for a fraction
of Windows' market share back on the PC, which is a waste of time.
Well, I encounteredthe same problem when was looking for bsd-licensed
libc separated from OS and less compiler depended. Sad, but true.
Toolbox didn't have anything worth extending (and toybox actually
predates its release, anyway). But at first glance bionic might
actually be worth extending.

The python syscall thing is the wrong approach, there should be some way
to annotate the upstream kernel system call list and extract automatic
stuff from it. I encountered a similar problem adding hexagon support
to strace, you can get system call name and id from the headers, but not
argument list. It's in the kernel source, I just need to figure out how
to extract it cleanly...

And they need a generic architecture. And they need a lot of stubs
filled out...

But on the whole, it more or less works. Not doing internationalization
and not contaminating the thing wtih C++ are both defensible decisions...
I was hoping that in the toybox have something like that.
I _need_ a way to supplement bionic into a full libc, and if I can't get
it I'll have to write it. But I'll have to burn that bridge when I come
to it...
Good that you're interested in it.There is early alpha of it, but may be
later I will public some researches on this topic.
Cool,

Rob
Rob Landley
2012-02-05 18:07:57 UTC
Permalink
Hello Rob,
I read some list postings in the archive.
I write as PM because I don't want to "scare" one of the authors (Tim Bird).
Trust me, if the lwn.net coverage didn't scare him, I doubt honest code
review would make him blink. :)

I'll cc: this to the list but chop out your email, reply there if you
want to decloak. :)
- Does your goal "be simple" mean that the code must be very easy
readable? id.c IS very easy readable but in case of e.g. flags "u" and
"r" set you will make 4 syscalls when only 1 is needed. Shouldn't it
be "optimized"?
I tend to go through and do optimization passes on submissions. I
haven't gotten to id yet.

My "quick glance" when he sent it said "would putting a for(;;) loop
around only two printf calls actually improve matters, or is the
overhead of the loop going to make it larger? I need to bench it."

What I didn't do in the quick glance was notice that the options are
exclusive. As usual, the gnu/dammit version has a special error message
for every occasion:

$ id -ug
id: cannot print "only" of more than one choice

Or that this id implementation is just a stub that is _not_ compliant
The following formats shall be used...
"uid=%u(%s) gid=%u(%s)\n", <real user ID>, <user-name>,
<real group ID>, <group-name>
So yeah, it needs a second pass in a big way. I should default it to
"n" in the meantime because it's not really there right now...

Instead I've been doing release cleanup. Specificially regression
testing the existing commands in my aboriginal linux setup, and trying
to figure out when I swap in each toybox command individually the build
works for all of them, but when I swap in all of them uClibc dies with
an internal compiler error and I get segfaults from "sort" and "xargs"
in dmesg.

Plus I'd like to finish the C rewrite of scripts/config2help.py so
python isn't a build prerequisite if "help" is enabled. And I had to
teach scripts/genconfig.sh to do a compile probe against the containers
infrastructure and disable unshare if the target toolchain headers
haven't got the appropriate #defines...
- Many coders who extremely dislike overhead (including myself) only use
libcall printf() for debugging purposes. It's some kind of
"deprecated".
A valid concern, and I did think about this. But
printf/asprintf/vsprintf/fprintf are pretty much all the same engine,
once it's pulled in for one use the overhead of using it in a second
place is trivial.

Something _like_ it would be unavoidable for perror_exit() and friends
(attempts to move complexity from the library to the callsite are a net
increase in overhead when you scale to many calls). For example, the
above output sequence for "id" with no arguments would be 9 separate
output statements without printf. (Or 9 statements assembling a string,
of who knows what length, and then a tenth to write it out.)

Keep in mind that my primary design goal is _simplicity_, then size,
speed, and features. You have to trade these off against each other when
implementing, but simplicity is the one I weight the highest, because in
the long run it's the one that makes the implementation understandable,
and thus maintainable (and security auditable). That means I tend to
approach commands with "what's the simplest way to implement this
feature", "what's the simplest way to speed this up", and so on.

Leveraging code people already wrote makes _toybox_ simple, otherwise
you spend eight lines of code doing what one line of code can do, and
you wind up repeating that all through your program. This code is
required to be there by c99, and it's required to have certain behavior.
There are _sucky_implementations_ of this code, but there are also
reasonable ones.

The linux kernel contains a reasonable printf() implementation, and I
feel comfortable writing my own if necessary. The uClibc implementation
of printf() is about 10k, based on binary size growth for a static i586
compile between "puts("Hello\n")" and "printf("Hello%d\n",0)". (Note
that printf() with a just constant string and no arguments gets
downshifted to a puts equivalent in uClibc, identical binary size.)

Yes, I'm balancing several definitions of "small" here. I want to make
toybox as a whole, statically linked, take up as little disk space as
possible. And I want to make it take up as little ram as possible at
runtime.

And I want to let you build individual commands and have _those_ be as
small as possible, but in doing so I assume those will be dynamically
linked. I am not optimizing for the "multiple individual executables,
each statically linked" case because it's _crazy_ and means I should
never leverage shared infrastructure, which is how you make the one big
statically linked executable small.

If your objection is that most people don't know how to _use_ these
tools in a reasonable manner: Yes, I'm aware of the newbie mistake the
sudo guys did recently. (Welcome to C, where we cook with knives and
fire.) I've got linked lists containing pointers from different
allocation contexts. (The struct arg_list is malloced from the heap,
the values it points to live in environment space. That's the normal
use case of lib/args.c, because I don't want to copy memory
unnecessarily. There's also the global toybuf[] and the stack without
even bringing mmap() into it. When you write in C, you have to know
what you're _doing_.)

Over on xargs I was at first trying to make a version that fit in
toybuf[] and upshifted to a malloc() when it outgrew the 4k static
buffer, but as long as I had to write the malloc() version anyway making
it the _only_ code path simplified things, so that's what I checked in.
I do a lot of that, weigh multiple approaches and then wind up doing
the _less_ microoptimized version because it's simpler and scales better
and performance is reasonable and yes I'm aware of nommu systems but
this mostly still runs on them, and trying to make a nommu system
reliable in low memory situations isn't worth complicating the common case.
Libs like libowfat or libdjb will reduce overhead while
not making the source hard to read. What's your opinion?
Never heard of either, but in general:

The external dependencies of BusyBox (under my tenure):

libc

The external dependencies of toybox:

libc

I'm all for finding a better libc, but I don't even depend on stuff like
zlib or ncurses. (I'm ok shelling out to an external gzip binary and
piping stuff through it; the only sane way to add https support to wget
and such would be via stunnel.)

(Part of my reasoning here is that at the lowest levels of the system
the dependnecies go circular, which makes bootstrapping hard, and you
wind up building stuff twice, which is nuts and one configuration
inevitably gets way less testing. Avoiding external dependencies in
these packages other than "I need a compiler with libc" means you don't
have to build toybox twice to bootstrap a system.)

Note how glibc had a "stage 1 compiler", but toybox didn't. (Sheesh,
perl's probably the silliest one: building microperl so perl's build
system can be written _in_perl_. The C compiler has a good excuse for
that, perl does not.)
Frank
/* show effective, unless user specifies real */
uid = geteuid();
gid = getegid();
if (flags & FLAG_r) {
uid = getuid();
gid = getgid();
}
if (flags & FLAG_u) {
printf("%d\n", uid);
return;
}
if (flags & FLAG_g) {
printf("%d\n", gid);
return;
}
printf("%d %d\n", uid, gid);
if (flags & (FLAG_u|FLAG_g))
printf("%d\n", (flags & FLAG_u) ? uid | gid);
else printf("%d %d\n", uid, gid);

Except the standard _also_ requires -G and -n. And a far more thorough
reading. (Yeah, it's a todo item.)

Rob
Rob Landley
2012-02-07 13:32:34 UTC
Permalink
2012/2/5 Rob Landley <rob at landley.net <mailto:rob at landley.net>>
Thank you for your detailed response!
May be better create some small and
fast libc implementation to make you ab linux more small and easy
to use?
I've been pondering it. That's why I was looking at musl. But doing so
is not necessarily the same project as toybox. Toybox is replacing
toolbox. Replacing or supplementing _bionic_ is a separate issue.
I'm all for working with people who are trying, but the only guys I know
starting a new technically interesting libc right now are wasting their
time fighting for a fraction of uClibc's market share which is fighting
for a fraction of glibc's market share which is fighting for a fraction
of Windows' market share back on the PC, which is a waste of time.
Well, I encounteredthe same problem when was looking for bsd-licensed
libc separated from OS and less compiler depended. Sad, but true.
Toolbox didn't have anything worth extending (and toybox actually
predates its release, anyway). But at first glance bionic might
actually be worth extending.

The python syscall thing is the wrong approach, there should be some way
to annotate the upstream kernel system call list and extract automatic
stuff from it. I encountered a similar problem adding hexagon support
to strace, you can get system call name and id from the headers, but not
argument list. It's in the kernel source, I just need to figure out how
to extract it cleanly...

And they need a generic architecture. And they need a lot of stubs
filled out...

But on the whole, it more or less works. Not doing internationalization
and not contaminating the thing wtih C++ are both defensible decisions...
I was hoping that in the toybox have something like that.
I _need_ a way to supplement bionic into a full libc, and if I can't get
it I'll have to write it. But I'll have to burn that bridge when I come
to it...
Good that you're interested in it.There is early alpha of it, but may be
later I will public some researches on this topic.
Cool,

Rob
Rob Landley
2012-02-04 17:09:25 UTC
Permalink
Post by Frank Bergmann
Dear List,
as a new subscriber I want to introduce myself.
Welcome to the list!
Post by Frank Bergmann
I'm a systems administrator and software engineer from Germany. Started
thirty years before with 8080, 6502 and Z80 assembler
I started on a commodore 64 myself. (38911 basic bytes free. Not
counting c000, or the tendency for Blitz! to compress things.)
Post by Frank Bergmann
I still like to
write small and fast programs and tools - embedded or raw at linux system
level (of course mainly in C and just sometimes in assembler).
I like small and fast, but primarily what I like is _simple_.

That's what attracted me to busybox in the first place, and why I
couldn't go back to it when I got over the reason I left: it stopped
being simple. I like code that is as clear, straightforward, and easy
to understand as it can be for what it's doing.
Post by Frank Bergmann
I dislike the overhead of busybox and heared about toybox. If there's
something which I can contribute I would be glad to do so.
http://landley.net/toybox/design.html and
http://landley.net/toybox/code.html are the first two things to read.
That second page has an "adding a new command section.

The todo lists are in two places:

My "slush pile" notes are at http://landley.net/toybox/todos and there's
a wiki at http://elinux.org/index.php?title=Busybox_replacement

Feel free to edit the wiki. I note that xargs has since been
implemented and I haven't updated the status on the wiki yet.

The wiki lays out the various use cases I find important. One of the
big ones is I'm trying to steer Android into becoming a generic PC
replacement, beating iPhone while staying reasonably open. To do that
it not only needs a full posix command line, but it needs to become self
hosting. (It's not a full-fledged software platform until it leaves the
nest and can rebuild itself. Right now, you need a PC to do that, but
we can _fix_that.)

I should start an "Android Self Hosting Project" page on the wiki for
all this info, shouldn't I?

Anyway, I already worked out the smallest and simplest self-hosting
Linux platform I could (Aboriginal Linux). It started with an automated
Linux From Scratch (3.x) and then I replaced glibc with uClibc and all
the gnu command line utilities with BusyBox. (For a while I was trying
to get it down to four packages: "linux, tinycc, busybox, uClibc". But
my tinycc fork is mothballed like toybox was, and if I restart it I need
to do qcc... Tangent.)

All the busybox work I did was a spin-off of that tiny bootstrapping
project. When I started, busybox could _not_ be used by itself to
replace all the Linux From Scratch command line packages: now it can.

And now I'm using Aboriginal Linux as a test environment for toolbox,
and replacing the _busybox_ commands one at a time:

http://landley.net/notes-2011.html#29-12-2011

(The boxen.sh script mentioned there is attached.)

However, that's only half the story. The other half is replacing
Android's "toolbox" in a bionic build environment. (Bionic is android's
libc, and it's crap, thus hard to build real programs against.) I need
to set up an android build environment, and use the existing toolchain
to build toybox, and make sure it's a drop-in replacement for toolbox.
This involves a few commands that don't exist anywhere else (some of
which are really stupid; android ported the windows registry to Linux, I
don't know what they were smoking), but I just added build-time probes
to scripts/genconfig.sh and can extend those for bionic support for that
stuff.

Beyond that: POSIX-2008/SUSv4 support is good on general principles, and
it's a free spec on the web, so I'm trying to get as close to that as is
reasonable, and document where we diverge. (Internationalization, for
example: being 8-bit clean is obvious and some UTF-8 support may be
reasonable in places, but most of this belongs at the GUI level, not at
the command line level. Whatever language you're working in, the "for"
keyword in C remains the same.)

I've documented some of this in http://landley.net/toybox/design.html
but possibly there should be a wiki page on that too...
Post by Frank Bergmann
Some of my own tools are published at www.tuxad.com.
Bookmarked. I'll try to look at it later today, but I need to move away
from the free caffeine refills to regain net access. :)

Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: boxen.sh
Type: application/x-sh
Size: 361 bytes
Desc: not available
URL: <http://lists.landley.net/pipermail/toybox-landley.net/attachments/20120204/0b999a74/attachment-0002.sh>
Luis Felipe Strano Moraes
2012-02-04 13:39:36 UTC
Permalink
By the way, as we talked on IRC, I've setup a Git clone of the repository here:
https://github.com/lfelipe/toybox

I honestly find git much easier to work with, but that is basically
because it is the one I work with daily. It seems there are mercurial
plugins that make dealing with a git repository quite easy as well,
but I'm pretty much OK with submitting stuff via mercurial repository
if need be. I'm not sure if the way I created the git repository is
the best one for this kind of work, but if it is not I can just
recreate it later.

Best regards,
Luis Felipe
Post by Luis Felipe Strano Moraes
Hey guys,
just did a small patch removing an unnecessary allocation I spotted in
toybox, what would be the best way to submit it?
https://bitbucket.org/lfelipe/toybox/changeset/84f7fc53c69f
Best regards,
Good catch. ?Applied.
Thanks,
Rob
--
Lu?s Felipe Strano Moraes
http://www.strano.org
Luis Felipe Strano Moraes
2012-02-04 13:39:55 UTC
Permalink
By the way, this is the script I used for the import:
https://github.com/alexgorbatchev/hg-to-git

--lf


On Sat, Feb 4, 2012 at 11:39 AM, Luis Felipe Strano Moraes
Post by Luis Felipe Strano Moraes
https://github.com/lfelipe/toybox
I honestly find git much easier to work with, but that is basically
because it is the one I work with daily. It seems there are mercurial
plugins that make dealing with a git repository quite easy as well,
but I'm pretty much OK with submitting stuff via mercurial repository
if need be. I'm not sure if the way I created the git repository is
the best one for this kind of work, but if it is not I can just
recreate it later.
Best regards,
Luis Felipe
Post by Luis Felipe Strano Moraes
Hey guys,
just did a small patch removing an unnecessary allocation I spotted in
toybox, what would be the best way to submit it?
https://bitbucket.org/lfelipe/toybox/changeset/84f7fc53c69f
Best regards,
Good catch. ?Applied.
Thanks,
Rob
--
Lu?s Felipe Strano Moraes
http://www.strano.org
--
Lu?s Felipe Strano Moraes
http://www.strano.org
Rob Landley
2012-02-04 16:45:35 UTC
Permalink
Post by Luis Felipe Strano Moraes
https://github.com/alexgorbatchev/hg-to-git
I was actually looking at hg-git.github.com which seems to be the
"official" solution. For all I know the above is the same thing, I
can't look at it right now due to the web here being flaky. (This reply
is going into my email outbox to get sent when I next download my email.)

Rob
Luis Felipe Strano Moraes
2012-02-06 22:42:33 UTC
Permalink
I've changed the git repository to use the hg-git script. I had
previously seen it, but hg-to-git seemed more direct. Unfortunately,
it also seems targeted at making a one-way-only transition, and since
this is not what we want, I started over with hg-git.

It seems to work really ok, even providing a nice git repository
inside your mercurial tree in order to easy maintenance.

I've sent one commit there already (fixing a warning), please take a
look and see if you think integration will be easy. If you prefer, I
can keep both this and the hg repo at bitbucket uptodate with one
another.

--lf
Post by Rob Landley
Post by Luis Felipe Strano Moraes
https://github.com/alexgorbatchev/hg-to-git
I was actually looking at hg-git.github.com which seems to be the
"official" solution. ?For all I know the above is the same thing, I
can't look at it right now due to the web here being flaky. ?(This reply
is going into my email outbox to get sent when I next download my email.)
Rob
--
Lu?s Felipe Strano Moraes
http://www.strano.org
Rob Landley
2012-02-07 05:22:49 UTC
Permalink
Post by Luis Felipe Strano Moraes
I've changed the git repository to use the hg-git script. I had
previously seen it, but hg-to-git seemed more direct. Unfortunately,
it also seems targeted at making a one-way-only transition, and since
this is not what we want, I started over with hg-git.
It seems to work really ok, even providing a nice git repository
inside your mercurial tree in order to easy maintenance.
I've sent one commit there already (fixing a warning), please take a
look and see if you think integration will be easy. If you prefer, I
can keep both this and the hg repo at bitbucket uptodate with one
another.
--lf
You never actually said where the git repository was,but I guessed
http://github.com/lfelipe/toybox and that looks like it.

The commit's good. I need to set up hg-git myself...

pushing to git+ssh://github.com/landley/toybox.git
exporting hg objects to git
creating and sending data
abort: could not import module SubprocessWrapper!

It's not happy. Need to figure out why...

https://github.com/schacon/hg-git/issues/173

Looks like Ubuntu broke Dulwich. Thanks Ubuntu...

It's 11:30 and I have work in the morning, I'll try again tomorrow...

Rob
Post by Luis Felipe Strano Moraes
Post by Rob Landley
Post by Luis Felipe Strano Moraes
https://github.com/alexgorbatchev/hg-to-git
I was actually looking at hg-git.github.com which seems to be the
"official" solution. For all I know the above is the same thing, I
can't look at it right now due to the web here being flaky. (This reply
is going into my email outbox to get sent when I next download my email.)
Rob
Luis Felipe Strano Moraes
2012-02-07 12:09:31 UTC
Permalink
Post by Rob Landley
You never actually said where the git repository was,but I guessed
http://github.com/lfelipe/toybox and that looks like it.
Yep, it is there (I had mentioned it on previous e-mail, forgot to
mention it again).
Post by Rob Landley
The commit's good. ?I need to set up hg-git myself...
pushing to git+ssh://github.com/landley/toybox.git
exporting hg objects to git
creating and sending data
abort: could not import module SubprocessWrapper!
It's not happy. ?Need to figure out why...
https://github.com/schacon/hg-git/issues/173
Looks like Ubuntu broke Dulwich. ?Thanks Ubuntu...
It's 11:30 and I have work in the morning, I'll try again tomorrow...
Actually, I've been toying with hg-git and so far it doesn't seem to
work completely right. Basically I've been able to make it work only
by starting afresh everytime, but the interaction with existing git
repo does not seem to go just fine. Pulling also didn't quite work as
expected, so unless it works for you, I'm considering just giving this
up for now and focusing on toybox itself.

I've pushed both this commit (and the other initial one for the who
command) to my mercurial repository on bitbucket:
https://bitbucket.org/lfelipe/toybox

They are both on top of your latest commits, I should probably have
made them on a separate branch, but I don't know Mercurial yet, will
be going over some docs for it later today.

Best regards,
Luis Felipe
Post by Rob Landley
Rob
Post by Rob Landley
Post by Luis Felipe Strano Moraes
https://github.com/alexgorbatchev/hg-to-git
I was actually looking at hg-git.github.com which seems to be the
"official" solution. ?For all I know the above is the same thing, I
can't look at it right now due to the web here being flaky. ?(This reply
is going into my email outbox to get sent when I next download my email.)
Rob
--
Lu?s Felipe Strano Moraes
http://www.strano.org
Rob Landley
2012-02-07 13:39:22 UTC
Permalink
Post by Luis Felipe Strano Moraes
Post by Rob Landley
Looks like Ubuntu broke Dulwich. Thanks Ubuntu...
It's 11:30 and I have work in the morning, I'll try again tomorrow...
Actually, I've been toying with hg-git and so far it doesn't seem to
work completely right. Basically I've been able to make it work only
by starting afresh everytime, but the interaction with existing git
repo does not seem to go just fine. Pulling also didn't quite work as
expected, so unless it works for you, I'm considering just giving this
up for now and focusing on toybox itself.
Focusing on toybox is what I've been doing. Mercurial-git integration is
a todo item of mine, but not a high priority one, and not really _my_
problem (it should work upstream).

Submit a bug report and move on, I guess...
Post by Luis Felipe Strano Moraes
I've pushed both this commit (and the other initial one for the who
https://bitbucket.org/lfelipe/toybox
I've re-cloned it but I'll have to look at it tomorrow. Time to go to
work, and this evening I've got stuff...
Post by Luis Felipe Strano Moraes
They are both on top of your latest commits, I should probably have
made them on a separate branch, but I don't know Mercurial yet, will
be going over some docs for it later today.
I don't make extensive use of the features myself. (I'm probably going
to have to learn now that there are more committers. :)

Thanks,

Rob
Tim Elliott
2012-02-07 22:46:41 UTC
Permalink
(I'm probably going to have to learn now that there are more committers. :)
The LWN articles and recent blog posts have drawn attention to toybox,
and ironically may bring more contributors.

It certainly got me interested, though mostly for selfish reasons -
this as an opportunity to write simple C programs that will be
scrutinized by some experienced and picky eyeballs. This is the best
way to learn, and it's free!

Cheers,
Tim
Tim Elliott
2012-02-07 22:46:41 UTC
Permalink
(I'm probably going to have to learn now that there are more committers. :)
The LWN articles and recent blog posts have drawn attention to toybox,
and ironically may bring more contributors.

It certainly got me interested, though mostly for selfish reasons -
this as an opportunity to write simple C programs that will be
scrutinized by some experienced and picky eyeballs. This is the best
way to learn, and it's free!

Cheers,
Tim
Rob Landley
2012-02-07 13:39:22 UTC
Permalink
Post by Luis Felipe Strano Moraes
Post by Rob Landley
Looks like Ubuntu broke Dulwich. Thanks Ubuntu...
It's 11:30 and I have work in the morning, I'll try again tomorrow...
Actually, I've been toying with hg-git and so far it doesn't seem to
work completely right. Basically I've been able to make it work only
by starting afresh everytime, but the interaction with existing git
repo does not seem to go just fine. Pulling also didn't quite work as
expected, so unless it works for you, I'm considering just giving this
up for now and focusing on toybox itself.
Focusing on toybox is what I've been doing. Mercurial-git integration is
a todo item of mine, but not a high priority one, and not really _my_
problem (it should work upstream).

Submit a bug report and move on, I guess...
Post by Luis Felipe Strano Moraes
I've pushed both this commit (and the other initial one for the who
https://bitbucket.org/lfelipe/toybox
I've re-cloned it but I'll have to look at it tomorrow. Time to go to
work, and this evening I've got stuff...
Post by Luis Felipe Strano Moraes
They are both on top of your latest commits, I should probably have
made them on a separate branch, but I don't know Mercurial yet, will
be going over some docs for it later today.
I don't make extensive use of the features myself. (I'm probably going
to have to learn now that there are more committers. :)

Thanks,

Rob
Luis Felipe Strano Moraes
2012-02-07 12:09:31 UTC
Permalink
Post by Rob Landley
You never actually said where the git repository was,but I guessed
http://github.com/lfelipe/toybox and that looks like it.
Yep, it is there (I had mentioned it on previous e-mail, forgot to
mention it again).
Post by Rob Landley
The commit's good. ?I need to set up hg-git myself...
pushing to git+ssh://github.com/landley/toybox.git
exporting hg objects to git
creating and sending data
abort: could not import module SubprocessWrapper!
It's not happy. ?Need to figure out why...
https://github.com/schacon/hg-git/issues/173
Looks like Ubuntu broke Dulwich. ?Thanks Ubuntu...
It's 11:30 and I have work in the morning, I'll try again tomorrow...
Actually, I've been toying with hg-git and so far it doesn't seem to
work completely right. Basically I've been able to make it work only
by starting afresh everytime, but the interaction with existing git
repo does not seem to go just fine. Pulling also didn't quite work as
expected, so unless it works for you, I'm considering just giving this
up for now and focusing on toybox itself.

I've pushed both this commit (and the other initial one for the who
command) to my mercurial repository on bitbucket:
https://bitbucket.org/lfelipe/toybox

They are both on top of your latest commits, I should probably have
made them on a separate branch, but I don't know Mercurial yet, will
be going over some docs for it later today.

Best regards,
Luis Felipe
Post by Rob Landley
Rob
Post by Rob Landley
Post by Luis Felipe Strano Moraes
https://github.com/alexgorbatchev/hg-to-git
I was actually looking at hg-git.github.com which seems to be the
"official" solution. ?For all I know the above is the same thing, I
can't look at it right now due to the web here being flaky. ?(This reply
is going into my email outbox to get sent when I next download my email.)
Rob
--
Lu?s Felipe Strano Moraes
http://www.strano.org
Rob Landley
2012-02-07 05:22:49 UTC
Permalink
Post by Luis Felipe Strano Moraes
I've changed the git repository to use the hg-git script. I had
previously seen it, but hg-to-git seemed more direct. Unfortunately,
it also seems targeted at making a one-way-only transition, and since
this is not what we want, I started over with hg-git.
It seems to work really ok, even providing a nice git repository
inside your mercurial tree in order to easy maintenance.
I've sent one commit there already (fixing a warning), please take a
look and see if you think integration will be easy. If you prefer, I
can keep both this and the hg repo at bitbucket uptodate with one
another.
--lf
You never actually said where the git repository was,but I guessed
http://github.com/lfelipe/toybox and that looks like it.

The commit's good. I need to set up hg-git myself...

pushing to git+ssh://github.com/landley/toybox.git
exporting hg objects to git
creating and sending data
abort: could not import module SubprocessWrapper!

It's not happy. Need to figure out why...

https://github.com/schacon/hg-git/issues/173

Looks like Ubuntu broke Dulwich. Thanks Ubuntu...

It's 11:30 and I have work in the morning, I'll try again tomorrow...

Rob
Post by Luis Felipe Strano Moraes
Post by Rob Landley
Post by Luis Felipe Strano Moraes
https://github.com/alexgorbatchev/hg-to-git
I was actually looking at hg-git.github.com which seems to be the
"official" solution. For all I know the above is the same thing, I
can't look at it right now due to the web here being flaky. (This reply
is going into my email outbox to get sent when I next download my email.)
Rob
Luis Felipe Strano Moraes
2012-02-06 22:42:33 UTC
Permalink
I've changed the git repository to use the hg-git script. I had
previously seen it, but hg-to-git seemed more direct. Unfortunately,
it also seems targeted at making a one-way-only transition, and since
this is not what we want, I started over with hg-git.

It seems to work really ok, even providing a nice git repository
inside your mercurial tree in order to easy maintenance.

I've sent one commit there already (fixing a warning), please take a
look and see if you think integration will be easy. If you prefer, I
can keep both this and the hg repo at bitbucket uptodate with one
another.

--lf
Post by Rob Landley
Post by Luis Felipe Strano Moraes
https://github.com/alexgorbatchev/hg-to-git
I was actually looking at hg-git.github.com which seems to be the
"official" solution. ?For all I know the above is the same thing, I
can't look at it right now due to the web here being flaky. ?(This reply
is going into my email outbox to get sent when I next download my email.)
Rob
--
Lu?s Felipe Strano Moraes
http://www.strano.org
Rob Landley
2012-02-04 16:45:35 UTC
Permalink
Post by Luis Felipe Strano Moraes
https://github.com/alexgorbatchev/hg-to-git
I was actually looking at hg-git.github.com which seems to be the
"official" solution. For all I know the above is the same thing, I
can't look at it right now due to the web here being flaky. (This reply
is going into my email outbox to get sent when I next download my email.)

Rob
Rob Landley
2012-02-04 16:43:41 UTC
Permalink
Post by Luis Felipe Strano Moraes
https://github.com/lfelipe/toybox
I cloned it, and then did an hg export/import
Post by Luis Felipe Strano Moraes
I honestly find git much easier to work with, but that is basically
because it is the one I work with daily. It seems there are mercurial
plugins that make dealing with a git repository quite easy as well,
I mean to set up a git mirror using those plugins. Possibly this weekend.
Post by Luis Felipe Strano Moraes
but I'm pretty much OK with submitting stuff via mercurial repository
if need be. I'm not sure if the way I created the git repository is
the best one for this kind of work, but if it is not I can just
recreate it later.
Worst case scenario I can get a patch from git and cut and paste the
commit comment, but I'd rather have the tools do it so the attribution's
clearer and the metadata's right for future pulls.

If you did it right, I can hg pull from that git tree, but I've still
got to set the plugins up properly over here to try that. (My net
access is intermittent from the table I've snagged at Einstein's bagels,
but the soda refills are unlimited and there's an electrical outlet, so
I'll have to poke at repo stuff later. :)
Post by Luis Felipe Strano Moraes
Best regards,
Luis Felipe
Rob
Post by Luis Felipe Strano Moraes
Post by Rob Landley
Post by Luis Felipe Strano Moraes
Hey guys,
just did a small patch removing an unnecessary allocation I spotted in
toybox, what would be the best way to submit it?
https://bitbucket.org/lfelipe/toybox/changeset/84f7fc53c69f
Best regards,
Good catch. Applied.
Thanks,
Rob
Luis Felipe Strano Moraes
2012-02-04 13:39:55 UTC
Permalink
By the way, this is the script I used for the import:
https://github.com/alexgorbatchev/hg-to-git

--lf


On Sat, Feb 4, 2012 at 11:39 AM, Luis Felipe Strano Moraes
Post by Luis Felipe Strano Moraes
https://github.com/lfelipe/toybox
I honestly find git much easier to work with, but that is basically
because it is the one I work with daily. It seems there are mercurial
plugins that make dealing with a git repository quite easy as well,
but I'm pretty much OK with submitting stuff via mercurial repository
if need be. I'm not sure if the way I created the git repository is
the best one for this kind of work, but if it is not I can just
recreate it later.
Best regards,
Luis Felipe
Post by Luis Felipe Strano Moraes
Hey guys,
just did a small patch removing an unnecessary allocation I spotted in
toybox, what would be the best way to submit it?
https://bitbucket.org/lfelipe/toybox/changeset/84f7fc53c69f
Best regards,
Good catch. ?Applied.
Thanks,
Rob
--
Lu?s Felipe Strano Moraes
http://www.strano.org
--
Lu?s Felipe Strano Moraes
http://www.strano.org
Rob Landley
2012-02-04 16:43:41 UTC
Permalink
Post by Luis Felipe Strano Moraes
https://github.com/lfelipe/toybox
I cloned it, and then did an hg export/import
Post by Luis Felipe Strano Moraes
I honestly find git much easier to work with, but that is basically
because it is the one I work with daily. It seems there are mercurial
plugins that make dealing with a git repository quite easy as well,
I mean to set up a git mirror using those plugins. Possibly this weekend.
Post by Luis Felipe Strano Moraes
but I'm pretty much OK with submitting stuff via mercurial repository
if need be. I'm not sure if the way I created the git repository is
the best one for this kind of work, but if it is not I can just
recreate it later.
Worst case scenario I can get a patch from git and cut and paste the
commit comment, but I'd rather have the tools do it so the attribution's
clearer and the metadata's right for future pulls.

If you did it right, I can hg pull from that git tree, but I've still
got to set the plugins up properly over here to try that. (My net
access is intermittent from the table I've snagged at Einstein's bagels,
but the soda refills are unlimited and there's an electrical outlet, so
I'll have to poke at repo stuff later. :)
Post by Luis Felipe Strano Moraes
Best regards,
Luis Felipe
Rob
Post by Luis Felipe Strano Moraes
Post by Rob Landley
Post by Luis Felipe Strano Moraes
Hey guys,
just did a small patch removing an unnecessary allocation I spotted in
toybox, what would be the best way to submit it?
https://bitbucket.org/lfelipe/toybox/changeset/84f7fc53c69f
Best regards,
Good catch. Applied.
Thanks,
Rob
Frank Bergmann
2012-02-04 10:32:52 UTC
Permalink
Dear List,

as a new subscriber I want to introduce myself.
I'm a systems administrator and software engineer from Germany. Started
thirty years before with 8080, 6502 and Z80 assembler I still like to
write small and fast programs and tools - embedded or raw at linux system
level (of course mainly in C and just sometimes in assembler).
I dislike the overhead of busybox and heared about toybox. If there's
something which I can contribute I would be glad to do so.
Some of my own tools are published at www.tuxad.com.

Frank
--
EDV Frank Bergmann Tel. 05221-9249753
LPIC-3 Linux Professional Fax 05221-9249754
P?dinghauser Str. 5 email iservice at tuxad.com
32051 Herford USt-IdNr DE237314606
Luis Felipe Strano Moraes
2012-02-04 13:39:36 UTC
Permalink
By the way, as we talked on IRC, I've setup a Git clone of the repository here:
https://github.com/lfelipe/toybox

I honestly find git much easier to work with, but that is basically
because it is the one I work with daily. It seems there are mercurial
plugins that make dealing with a git repository quite easy as well,
but I'm pretty much OK with submitting stuff via mercurial repository
if need be. I'm not sure if the way I created the git repository is
the best one for this kind of work, but if it is not I can just
recreate it later.

Best regards,
Luis Felipe
Post by Luis Felipe Strano Moraes
Hey guys,
just did a small patch removing an unnecessary allocation I spotted in
toybox, what would be the best way to submit it?
https://bitbucket.org/lfelipe/toybox/changeset/84f7fc53c69f
Best regards,
Good catch. ?Applied.
Thanks,
Rob
--
Lu?s Felipe Strano Moraes
http://www.strano.org
Luis Felipe Strano Moraes
2012-02-03 13:20:38 UTC
Permalink
Hey guys,

just did a small patch removing an unnecessary allocation I spotted in
toybox, what would be the best way to submit it?

It is available in bitbucket here:
https://bitbucket.org/lfelipe/toybox/changeset/84f7fc53c69f

Best regards,
--
Lu?s Felipe Strano Moraes
http://www.strano.org
Rob Landley
2012-02-04 04:33:08 UTC
Permalink
Post by Luis Felipe Strano Moraes
Hey guys,
just did a small patch removing an unnecessary allocation I spotted in
toybox, what would be the best way to submit it?
https://bitbucket.org/lfelipe/toybox/changeset/84f7fc53c69f
Best regards,
Good catch. Applied.

Thanks,

Rob
Loading...