PATCH [0/3]: Simplify the kernel build by removing perl.

[Rob Landley]

Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 )
building a Linux kernel never required perl to be installed on the build
system. (Various development and debugging scripts were written in perl and
python and such, but they weren't involved in actually building a kernel.)
Building a kernel before 2.6.25 could be done with a minimal system built from
gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel, and nothing
else. (Embedded developers creating clean cross compile environments that
won't leak bits of the host system into the target, or bootstrapping native
development environments to run on target hardware or under emulators, tend to
care about this sort of thing.)

The perl checkin for 2.6.25 was the camel's nose under the tent flap, and
since then two more instances of perl have shown up in the core kernel build.
This patch series removes the three required to build a basic kernel for qemu
for the targets I've tested (arm, mips, powerpc, sparc, m68k, sh4, and of
course x86 and x86-64), replacing them with shell scripts.

Historically the kernel has gone out of its way to minimize environmental
dependencies for the build. For example, the plethora of *_shipped files mean
we don't need lex and yacc to build the kernel. The kconfig infrastructure
once required curses to run "make oldconfig", but that requirement was
restricted to just menuconfig so the kernel could build on systems without
ncurses.

That was a very nice policy. Kernel development already requires an in-depth
knowledge of C, make, and shell, plus things like the kconfig language. A
similarly in-depth knowledge of perl is bigger than all that combined (even
Larry Wall probably doesn't know all of Perl anymore), and then what's the
excuse _not_ to add Python to the core build? And after that why not java, or
lua? Where does it end? What's the criteria to say "no" here?

This patch series saves time and says no now.

[Rob Landley]

From: Rob Landley <***@landley.net>

Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell script
is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003.

Peter Anvin added this perl to 2.6.25. Before that, the kernel had never
required perl to build.

Signed-off-by: Rob Landley <***@landley.net>
---

kernel/Makefile | 4
kernel/timeconst.pl | 378 ------------------------------------------
kernel/timeconst.sh | 91 ++++++++++
3 files changed, 93 insertions(+), 380 deletions(-)

diff -r d0c7611dcfd6 kernel/Makefile
--- a/kernel/Makefile Tue Dec 30 17:48:25 2008 -0800
+++ b/kernel/Makefile Fri Jan 02 00:10:44 2009 -0600
@@ -115,7 +115,7 @@
$(obj)/time.o: $(obj)/timeconst.h

quiet_cmd_timeconst = TIMEC $@
- cmd_timeconst = $(PERL) $< $(CONFIG_HZ) > $@
+ cmd_timeconst = $(CONFIG_SHELL) $< $(CONFIG_HZ) > $@
targets += timeconst.h
-$(obj)/timeconst.h: $(src)/timeconst.pl FORCE
+$(obj)/timeconst.h: $(src)/timeconst.sh FORCE
$(call if_changed,timeconst)
--- /dev/null 1969-12-31 00:00:00.000000000 -0600
+++ hg/kernel/timeconst.sh 2009-01-01 23:53:17.000000000 -0600
@@ -0,0 +1,91 @@
+#!/bin/bash
+
+if [ $# -ne 1 ]
+then
+ echo "Usage: timeconst.sh HZ"
+ exit 1
+fi
+
+HZ=$1
+
+# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit math.
+
+[ $((1<<32)) -lt 0 ] && exit 1
+
+# Output start of header file
+
+cat << EOF
+/* Automatically generated by kernel/timeconst.sh */
+/* Conversion constants for HZ == $HZ */
+
+#ifndef KERNEL_TIMECONST_H
+#define KERNEL_TIMECONST_H
+
+#include <linux/param.h>
+#include <linux/types.h>
+
+#if HZ != $HZ
+#error "kernel/timeconst.h has the wrong HZ value!"
+#endif
+
+EOF
+
+# For both Miliseconds and Microseconds
+
+for i in "MSEC 1000" "USEC 1000000"
+do
+ NAME=$(echo $i | awk '{print $1}')
+ PERIOD=$(echo $i | awk '{print $2}')
+
+ # Find greatest common denominator (using Euclid's algorithm)
+
+ A=$HZ
+ B=$PERIOD
+
+ while [ $B -ne 0 ]
+ do
+ C=$(($A%$B))
+ A=$B
+ B=$C
+ done
+
+ GCD=$A
+
+ # Do this for each direction (HZ_TO_PERIOD and PERIOD_TO_HZ)
+
+ for DIRECTION in 0 1
+ do
+ if [ $DIRECTION -eq 0 ]
+ then
+ CONVERT="HZ_TO_${NAME}"
+ FROM=$HZ
+ TO=$PERIOD
+ else
+ CONVERT="${NAME}_TO_HZ"
+ FROM=$PERIOD
+ TO=$HZ
+ fi
+
+ # How many shift bits give 32 bits of significant data?
+
+ SHIFT=0
+ while [ $(( (($TO<<$SHIFT)+$FROM-1)/$FROM )) -lt $((1<<31)) ]
+ do
+ SHIFT=$(( $SHIFT+1 ))
+ done
+
+ MUL32=$(( (($TO<<$SHIFT)+$FROM-1)/$FROM ))
+ MUL32=$(printf %x $MUL32)
+ echo "#define ${CONVERT}_MUL32 U64_C(0x$MUL32)"
+ ADJ32=$(($FROM/$GCD))
+ ADJ32=$(( (($ADJ32-1)<<$SHIFT)/$ADJ32 ))
+ ADJ32=$(printf %x $ADJ32)
+ echo "#define ${CONVERT}_ADJ32 U64_C(0x$ADJ32)"
+ echo "#define ${CONVERT}_SHR32 $SHIFT"
+ echo "#define ${CONVERT}_NUM U64_C($(($TO/$GCD)))"
+ echo "#define ${CONVERT}_DEN U64_C($(($FROM/$GCD)))"
+ done
+done
+
+echo
+echo "#endif /* KERNEL_TIMECHONST_H */"
--- hg/kernel/timeconst.pl 2008-11-22 19:09:13.000000000 -0600
+++ /dev/null 1969-12-31 00:00:00.000000000 -0600
@@ -1,378 +0,0 @@
-#!/usr/bin/perl
-# -----------------------------------------------------------------------
-#
-# Copyright 2007-2008 rPath, Inc. - All Rights Reserved
-#
-# This file is part of the Linux kernel, and is made available under
-# the terms of the GNU General Public License version 2 or (at your
-# option) any later version; incorporated herein by reference.
-#
-# -----------------------------------------------------------------------
-#
-
-#
-# Usage: timeconst.pl HZ > timeconst.h
-#
-
-# Precomputed values for systems without Math::BigInt
-# Generated by:
-# timeconst.pl --can 24 32 48 64 100 122 128 200 250 256 300 512 1000 1024 1200
-%canned_values = (
- 24 => [
- '0xa6aaaaab','0x2aaaaaa',26,
- 125,3,
- '0xc49ba5e4','0x1fbe76c8b4',37,
- 3,125,
- '0xa2c2aaab','0xaaaa',16,
- 125000,3,
- '0xc9539b89','0x7fffbce4217d',47,
- 3,125000,
- ], 32 => [
- '0xfa000000','0x6000000',27,
- 125,4,
- '0x83126e98','0xfdf3b645a',36,
- 4,125,
- '0xf4240000','0x0',17,
- 31250,1,
- '0x8637bd06','0x3fff79c842fa',46,
- 1,31250,
- ], 48 => [
- '0xa6aaaaab','0x6aaaaaa',27,
- 125,6,
- '0xc49ba5e4','0xfdf3b645a',36,
- 6,125,
- '0xa2c2aaab','0x15555',17,
- 62500,3,
- '0xc9539b89','0x3fffbce4217d',46,
- 3,62500,
- ], 64 => [
- '0xfa000000','0xe000000',28,
- 125,8,
- '0x83126e98','0x7ef9db22d',35,
- 8,125,
- '0xf4240000','0x0',18,
- 15625,1,
- '0x8637bd06','0x1fff79c842fa',45,
- 1,15625,
- ], 100 => [
- '0xa0000000','0x0',28,
- 10,1,
- '0xcccccccd','0x733333333',35,
- 1,10,
- '0x9c400000','0x0',18,
- 10000,1,
- '0xd1b71759','0x1fff2e48e8a7',45,
- 1,10000,
- ], 122 => [
- '0x8325c53f','0xfbcda3a',28,
- 500,61,
- '0xf9db22d1','0x7fbe76c8b',35,
- 61,500,
- '0x8012e2a0','0x3ef36',18,
- 500000,61,
- '0xffda4053','0x1ffffbce4217',45,
- 61,500000,
- ], 128 => [
- '0xfa000000','0x1e000000',29,
- 125,16,
- '0x83126e98','0x3f7ced916',34,
- 16,125,
- '0xf4240000','0x40000',19,
- 15625,2,
- '0x8637bd06','0xfffbce4217d',44,
- 2,15625,
- ], 200 => [
- '0xa0000000','0x0',29,
- 5,1,
- '0xcccccccd','0x333333333',34,
- 1,5,
- '0x9c400000','0x0',19,
- 5000,1,
- '0xd1b71759','0xfff2e48e8a7',44,
- 1,5000,
- ], 250 => [
- '0x80000000','0x0',29,
- 4,1,
- '0x80000000','0x180000000',33,
- 1,4,
- '0xfa000000','0x0',20,
- 4000,1,
- '0x83126e98','0x7ff7ced9168',43,
- 1,4000,
- ], 256 => [
- '0xfa000000','0x3e000000',30,
- 125,32,
- '0x83126e98','0x1fbe76c8b',33,
- 32,125,
- '0xf4240000','0xc0000',20,
- 15625,4,
- '0x8637bd06','0x7ffde7210be',43,
- 4,15625,
- ], 300 => [
- '0xd5555556','0x2aaaaaaa',30,
- 10,3,
- '0x9999999a','0x1cccccccc',33,
- 3,10,
- '0xd0555556','0xaaaaa',20,
- 10000,3,
- '0x9d495183','0x7ffcb923a29',43,
- 3,10000,
- ], 512 => [
- '0xfa000000','0x7e000000',31,
- 125,64,
- '0x83126e98','0xfdf3b645',32,
- 64,125,
- '0xf4240000','0x1c0000',21,
- 15625,8,
- '0x8637bd06','0x3ffef39085f',42,
- 8,15625,
- ], 1000 => [
- '0x80000000','0x0',31,
- 1,1,
- '0x80000000','0x0',31,
- 1,1,
- '0xfa000000','0x0',22,
- 1000,1,
- '0x83126e98','0x1ff7ced9168',41,
- 1,1000,
- ], 1024 => [
- '0xfa000000','0xfe000000',32,
- 125,128,
- '0x83126e98','0x7ef9db22',31,
- 128,125,
- '0xf4240000','0x3c0000',22,
- 15625,16,
- '0x8637bd06','0x1fff79c842f',41,
- 16,15625,
- ], 1200 => [
- '0xd5555556','0xd5555555',32,
- 5,6,
- '0x9999999a','0x66666666',31,
- 6,5,
- '0xd0555556','0x2aaaaa',22,
- 2500,3,
- '0x9d495183','0x1ffcb923a29',41,
- 3,2500,
- ]
-);
-
-$has_bigint = eval 'use Math::BigInt qw(bgcd); 1;';
-
-sub bint($)
-{
- my($x) = @_;
- return Math::BigInt->new($x);
-}
-
-#
-# Constants for division by reciprocal multiplication.
-# (bits, numerator, denominator)
-#
-sub fmul($$$)
-{
- my ($b,$n,$d) = @_;
-
- $n = bint($n);
- $d = bint($d);
-
- return scalar (($n << $b)+$d-bint(1))/$d;
-}
-
-sub fadj($$$)
-{
- my($b,$n,$d) = @_;
-
- $n = bint($n);
- $d = bint($d);
-
- $d = $d/bgcd($n, $d);
- return scalar (($d-bint(1)) << $b)/$d;
-}
-
-sub fmuls($$$) {
- my($b,$n,$d) = @_;
- my($s,$m);
- my($thres) = bint(1) << ($b-1);
-
- $n = bint($n);
- $d = bint($d);
-
- for ($s = 0; 1; $s++) {
- $m = fmul($s,$n,$d);
- return $s if ($m >= $thres);
- }
- return 0;
-}
-
-# Generate a hex value if the result fits in 64 bits;
-# otherwise skip.
-sub bignum_hex($) {
- my($x) = @_;
- my $s = $x->as_hex();
-
- return (length($s) > 18) ? undef : $s;
-}
-
-# Provides mul, adj, and shr factors for a specific
-# (bit, time, hz) combination
-sub muladj($$$) {
- my($b, $t, $hz) = @_;
- my $s = fmuls($b, $t, $hz);
- my $m = fmul($s, $t, $hz);
- my $a = fadj($s, $t, $hz);
- return (bignum_hex($m), bignum_hex($a), $s);
-}
-
-# Provides numerator, denominator values
-sub numden($$) {
- my($n, $d) = @_;
- my $g = bgcd($n, $d);
- return ($n/$g, $d/$g);
-}
-
-# All values for a specific (time, hz) combo
-sub conversions($$) {
- my ($t, $hz) = @_;
- my @val = ();
-
- # HZ_TO_xx
- push(@val, muladj(32, $t, $hz));
- push(@val, numden($t, $hz));
-
- # xx_TO_HZ
- push(@val, muladj(32, $hz, $t));
- push(@val, numden($hz, $t));
-
- return @val;
-}
-
-sub compute_values($) {
- my($hz) = @_;
- my @val = ();
- my $s, $m, $a, $g;
-
- if (!$has_bigint) {
- die "$0: HZ == $hz not canned and ".
- "Math::BigInt not available\n";
- }
-
- # MSEC conversions
- push(@val, conversions(1000, $hz));
-
- # USEC conversions
- push(@val, conversions(1000000, $hz));
-
- return @val;
-}
-
-sub outputval($$)
-{
- my($name, $val) = @_;
- my $csuf;
-
- if (defined($val)) {
- if ($name !~ /SHR/) {
- $val = "U64_C($val)";
- }
- printf "#define %-23s %s\n", $name.$csuf, $val.$csuf;
- }
-}
-
-sub output($@)
-{
- my($hz, @val) = @_;
- my $pfx, $bit, $suf, $s, $m, $a;
-
- print "/* Automatically generated by kernel/timeconst.pl */\n";
- print "/* Conversion constants for HZ == $hz */\n";
- print "\n";
- print "#ifndef KERNEL_TIMECONST_H\n";
- print "#define KERNEL_TIMECONST_H\n";
- print "\n";
-
- print "#include <linux/param.h>\n";
- print "#include <linux/types.h>\n";
-
- print "\n";
- print "#if HZ != $hz\n";
- print "#error \"kernel/timeconst.h has the wrong HZ value!\"\n";
- print "#endif\n";
- print "\n";
-
- foreach $pfx ('HZ_TO_MSEC','MSEC_TO_HZ',
- 'HZ_TO_USEC','USEC_TO_HZ') {
- foreach $bit (32) {
- foreach $suf ('MUL', 'ADJ', 'SHR') {
- outputval("${pfx}_$suf$bit", shift(@val));
- }
- }
- foreach $suf ('NUM', 'DEN') {
- outputval("${pfx}_$suf", shift(@val));
- }
- }
-
- print "\n";
- print "#endif /* KERNEL_TIMECONST_H */\n";
-}
-
-# Pretty-print Perl values
-sub perlvals(@) {
- my $v;
- my @l = ();
-
- foreach $v (@_) {
- if (!defined($v)) {
- push(@l, 'undef');
- } elsif ($v =~ /^0x/) {
- push(@l, "\'".$v."\'");
- } else {
- push(@l, $v.'');
- }
- }
- return join(',', @l);
-}
-
-($hz) = @ARGV;
-
-# Use this to generate the %canned_values structure
-if ($hz eq '--can') {
- shift(@ARGV);
- @hzlist = sort {$a <=> $b} (@ARGV);
-
- print "# Precomputed values for systems without Math::BigInt\n";
- print "# Generated by:\n";
- print "# timeconst.pl --can ", join(' ', @hzlist), "\n";
- print "\%canned_values = (\n";
- my $pf = "\t";
- foreach $hz (@hzlist) {
- my @values = compute_values($hz);
- print "$pf$hz => [\n";
- while (scalar(@values)) {
- my $bit;
- foreach $bit (32) {
- my $m = shift(@values);
- my $a = shift(@values);
- my $s = shift(@values);
- print "\t\t", perlvals($m,$a,$s), ",\n";
- }
- my $n = shift(@values);
- my $d = shift(@values);
- print "\t\t", perlvals($n,$d), ",\n";
- }
- print "\t]";
- $pf = ', ';
- }
- print "\n);\n";
-} else {
- $hz += 0; # Force to number
- if ($hz < 1) {
- die "Usage: $0 HZ\n";
- }
-
- @val = @{$canned_values{$hz}};
- if (!defined(@val)) {
- @val = compute_values($hz)

[Sam Ravnborg]

Hi Rob.

Post by Rob Landley
Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell script
is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003.

This part of the changelog is OK except that is fails to
address why we want to get away from perl.

Please drop the remining part of the changelog (but not the s-o-b).

Post by Rob Landley
Peter Anvin added this perl to 2.6.25. Before that, the kernel had never
required perl to build.
---
kernel/Makefile | 4
kernel/timeconst.pl | 378 ------------------------------------------
kernel/timeconst.sh | 91 ++++++++++
3 files changed, 93 insertions(+), 380 deletions(-)
diff -r d0c7611dcfd6 kernel/Makefile
--- a/kernel/Makefile Tue Dec 30 17:48:25 2008 -0800
+++ b/kernel/Makefile Fri Jan 02 00:10:44 2009 -0600
@@ -115,7 +115,7 @@
$(obj)/time.o: $(obj)/timeconst.h
targets += timeconst.h
-$(obj)/timeconst.h: $(src)/timeconst.pl FORCE
+$(obj)/timeconst.h: $(src)/timeconst.sh FORCE
$(call if_changed,timeconst)

OK

Post by Rob Landley
--- /dev/null 1969-12-31 00:00:00.000000000 -0600
+++ hg/kernel/timeconst.sh 2009-01-01 23:53:17.000000000 -0600
@@ -0,0 +1,91 @@
+#!/bin/bash
+
+if [ $# -ne 1 ]
+then
+ echo "Usage: timeconst.sh HZ"
+ exit 1
+fi

That usage is useless. Either extend it or spend a few lines
in the shell script explainign what the shell script is supposed to do.

Post by Rob Landley
+
+HZ=$1
+
+# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit math.
+
+[ $((1<<32)) -lt 0 ] && exit 1
+

If it fails then add an error message explaining why. So if we get reports that this
fails then we at least can see something like:
"timeconst noticed that the shell did not support 64 bit math - stop"

Post by Rob Landley
+# Output start of header file
+
+cat << EOF
+/* Automatically generated by kernel/timeconst.sh */
+/* Conversion constants for HZ == $HZ */
+
+#ifndef KERNEL_TIMECONST_H
+#define KERNEL_TIMECONST_H

Please use __KERNEL_TIMECONST_H. The two underscores are standard.

Post by Rob Landley
+
+#include <linux/param.h>
+#include <linux/types.h>
+
+#if HZ != $HZ
+#error "kernel/timeconst.h has the wrong HZ value!"
+#endif
+
+EOF
+
+# For both Miliseconds and Microseconds
+
+for i in "MSEC 1000" "USEC 1000000"
+do
+ NAME=$(echo $i | awk '{print $1}')
+ PERIOD=$(echo $i | awk '{print $2}')
+
+ # Find greatest common denominator (using Euclid's algorithm)
+
+ A=$HZ
+ B=$PERIOD
+
+ while [ $B -ne 0 ]
+ do
+ C=$(($A%$B))
+ A=$B
+ B=$C
+ done
+
+ GCD=$A
+
+ # Do this for each direction (HZ_TO_PERIOD and PERIOD_TO_HZ)
+
+ for DIRECTION in 0 1
+ do
+ if [ $DIRECTION -eq 0 ]
+ then
+ CONVERT="HZ_TO_${NAME}"
+ FROM=$HZ
+ TO=$PERIOD
+ else
+ CONVERT="${NAME}_TO_HZ"
+ FROM=$PERIOD
+ TO=$HZ
+ fi
+
+ # How many shift bits give 32 bits of significant data?
+
+ SHIFT=0
+ while [ $(( (($TO<<$SHIFT)+$FROM-1)/$FROM )) -lt $((1<<31)) ]
+ do
+ SHIFT=$(( $SHIFT+1 ))
+ done
+
+ MUL32=$(( (($TO<<$SHIFT)+$FROM-1)/$FROM ))
+ MUL32=$(printf %x $MUL32)
+ echo "#define ${CONVERT}_MUL32 U64_C(0x$MUL32)"
+ ADJ32=$(($FROM/$GCD))
+ ADJ32=$(( (($ADJ32-1)<<$SHIFT)/$ADJ32 ))
+ ADJ32=$(printf %x $ADJ32)
+ echo "#define ${CONVERT}_ADJ32 U64_C(0x$ADJ32)"
+ echo "#define ${CONVERT}_SHR32 $SHIFT"
+ echo "#define ${CONVERT}_NUM U64_C($(($TO/$GCD)))"
+ echo "#define ${CONVERT}_DEN U64_C($(($FROM/$GCD)))"
+ done
+done

Is it a shell limitation that all spaces around operators are missing?
Makes it hard to read..

You should use trap in your shell script so we do not end up with a
partially generated file.
Or you should change the rule in the Makefile to use
a temperary file and then rename it.

We have similar bugs in other places - but no need to add
in more places.


Sam

[Rob Landley]

Post by Sam Ravnborg
Hi Rob.

Post by Rob Landley
Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell
script is much simpler, about 1/4 the size, and runs on Red Hat 9 from
2003.

This part of the changelog is OK except that is fails to
address why we want to get away from perl.

You mean "The new shell script is much simpler, about 1/4 the size, runs on
Red Hat 9 from 2003, and isn't perl?" :)

Post by Sam Ravnborg
Please drop the remining part of the changelog (but not the s-o-b).

ok.

Post by Sam Ravnborg

Post by Rob Landley
Peter Anvin added this perl to 2.6.25. Before that, the kernel had never
required perl to build.
---
kernel/Makefile | 4
kernel/timeconst.pl | 378 ------------------------------------------
kernel/timeconst.sh | 91 ++++++++++
3 files changed, 93 insertions(+), 380 deletions(-)
diff -r d0c7611dcfd6 kernel/Makefile
--- a/kernel/Makefile Tue Dec 30 17:48:25 2008 -0800
+++ b/kernel/Makefile Fri Jan 02 00:10:44 2009 -0600
@@ -115,7 +115,7 @@
$(obj)/time.o: $(obj)/timeconst.h
targets += timeconst.h
-$(obj)/timeconst.h: $(src)/timeconst.pl FORCE
+$(obj)/timeconst.h: $(src)/timeconst.sh FORCE
$(call if_changed,timeconst)

OK

Post by Rob Landley
--- /dev/null 1969-12-31 00:00:00.000000000 -0600
+++ hg/kernel/timeconst.sh 2009-01-01 23:53:17.000000000 -0600
@@ -0,0 +1,91 @@
+#!/bin/bash
+
+if [ $# -ne 1 ]
+then
+ echo "Usage: timeconst.sh HZ"
+ exit 1
+fi

That usage is useless. Either extend it or spend a few lines
in the shell script explainign what the shell script is supposed to do.

Do you mean something more like:

echo "Usage: timeconst.sh HZ"
echo
echo "Generates a header file of constants for converting between decimal"
echo "HZ timer ticks and milisecond or microsecond delays"

I'm happy turning it into a comment instead, I just found a quick check that
I'd remembered to type an argument useful during debugging. :)

Post by Sam Ravnborg

Post by Rob Landley
+
+HZ=$1
+
+# Sanity test: even the shell in Red Hat 9 (circa 2003) supported 64 bit
math. +
+[ $((1<<32)) -lt 0 ] && exit 1
+

If it fails then add an error message explaining why. So if we get reports
"timeconst noticed that the shell did not support 64 bit math - stop"

Ok.

Post by Sam Ravnborg

Post by Rob Landley
+# Output start of header file
+
+cat << EOF
+/* Automatically generated by kernel/timeconst.sh */
+/* Conversion constants for HZ == $HZ */
+
+#ifndef KERNEL_TIMECONST_H
+#define KERNEL_TIMECONST_H

Please use __KERNEL_TIMECONST_H. The two underscores are standard.

Sure thing. (I was just copying the perl there, I'll post an updated patch
after I get some sleep.)

Post by Sam Ravnborg

Post by Rob Landley
+
+#include <linux/param.h>
+#include <linux/types.h>
+
+#if HZ != $HZ
+#error "kernel/timeconst.h has the wrong HZ value!"
+#endif
+
+EOF
+
+# For both Miliseconds and Microseconds
+
+for i in "MSEC 1000" "USEC 1000000"
+do
+ NAME=$(echo $i | awk '{print $1}')
+ PERIOD=$(echo $i | awk '{print $2}')
+
+ # Find greatest common denominator (using Euclid's algorithm)
+
+ A=$HZ
+ B=$PERIOD
+
+ while [ $B -ne 0 ]
+ do
+ C=$(($A%$B))
+ A=$B
+ B=$C
+ done
+
+ GCD=$A
+
+ # Do this for each direction (HZ_TO_PERIOD and PERIOD_TO_HZ)
+
+ for DIRECTION in 0 1
+ do
+ if [ $DIRECTION -eq 0 ]
+ then
+ CONVERT="HZ_TO_${NAME}"
+ FROM=$HZ
+ TO=$PERIOD
+ else
+ CONVERT="${NAME}_TO_HZ"
+ FROM=$PERIOD
+ TO=$HZ
+ fi
+
+ # How many shift bits give 32 bits of significant data?
+
+ SHIFT=0
+ while [ $(( (($TO<<$SHIFT)+$FROM-1)/$FROM )) -lt $((1<<31)) ]
+ do
+ SHIFT=$(( $SHIFT+1 ))
+ done
+
+ MUL32=$(( (($TO<<$SHIFT)+$FROM-1)/$FROM ))
+ MUL32=$(printf %x $MUL32)
+ echo "#define ${CONVERT}_MUL32 U64_C(0x$MUL32)"
+ ADJ32=$(($FROM/$GCD))
+ ADJ32=$(( (($ADJ32-1)<<$SHIFT)/$ADJ32 ))
+ ADJ32=$(printf %x $ADJ32)
+ echo "#define ${CONVERT}_ADJ32 U64_C(0x$ADJ32)"
+ echo "#define ${CONVERT}_SHR32 $SHIFT"
+ echo "#define ${CONVERT}_NUM U64_C($(($TO/$GCD)))"
+ echo "#define ${CONVERT}_DEN U64_C($(($FROM/$GCD)))"
+ done
+done

Is it a shell limitation that all spaces around operators are missing?
Makes it hard to read..

No, I was just trying to make sure I didn't go over the 80 char limit.
(Several temporary variables are there primarily because the style guide says
indentation should be via 8 space tabs.)

I'll add some more spaces on the respin.

Post by Sam Ravnborg
You should use trap in your shell script so we do not end up with a
partially generated file.
Or you should change the rule in the Makefile to use
a temperary file and then rename it.
We have similar bugs in other places - but no need to add
in more places.

Well, preserve in this case. :)

I can add an "EXIT" trap and then disarm it later, but the output is
redirected in the makefile, so generating an error won't remove the output if
you run again. Perhaps the file to create should be fed in as a second
argument to the script, so the trap can "rm" the output file if it triggers?

I'll poke at it in the morning.

Thanks for the feedback,

Rob

[Alan Cox]

I note that sed and printf and such are all susv3. I have an explicit test
for 32 bit math in the script that cares, and this worked in Red Hat 9 circa
2003.

If you are trying to do arbitary precision maths using standard posix
tools just use dc. That way the standard is explicit about what you will
get.

[H. Peter Anvin]

Post by Alan Cox

I note that sed and printf and such are all susv3. I have an explicit test
for 32 bit math in the script that cares, and this worked in Red Hat 9 circa
2003.

If you are trying to do arbitary precision maths using standard posix
tools just use dc. That way the standard is explicit about what you will
get.

The original patch used bc. Unfortunately, it ran into bc bugs -- on
some platforms like some of akpm's older test machines it would just hang.

-hpa

[Rob Landley]

Post by Alan Cox

I note that sed and printf and such are all susv3. I have an explicit
test for 32 bit math in the script that cares, and this worked in Red Hat
9 circa 2003.

If you are trying to do arbitary precision maths using standard posix
tools just use dc. That way the standard is explicit about what you will
get.

I looked at that, but:

A) the Open Group Base Specifications (which I normally go by, since they're
roughly synonymous with SUSv3 and Posix and available free on the web; they
just released version 7 a week or so back) don't list dc as one of their
utilities. (They mention bc, but not dc.)

B) The busybox implementation of dc is crap. I got 'em to fix the bug where
the output defaulted to binary instead of decimal, but the math is all done as
floating point rather than arbitrary precision, and they don't implement the
<< operator.

C) The only calculation which can overflow 64 bits (the ADJ32 one) turns out
not to need arbitrary precision math, just 72 bits, and if it ever uses more
than 32 then bottom 32 are all zero before the divide so you can do it in
three lines.

Essentially, the ADJ32 calculation is "(($NUMBER-1)<<$SHIFT)/$NUMBER".

$SHIFT maxes out at 51 and the largest interesting $NUMBER is 1000000.
(That's the pathological case of HZ=1, calculating the USEC_TO_HZ direction.
A larger $HZ results in a smaller $SHIFT by the number of bits needed to store
$HZ, by the way, so a $HZ of 17 would have a shift of 47. So even a HZ bigger
than a million should have a small enough $SHIFT not to cause trouble here,
although that's probably an _insane_ input to this script.)

1 million needs 20 bits to store, so the above calculation has to cope with an
intermediate value of 999999<<51 which takes a little over 70 bits to store,
hence the potential to overflow 63 bits of signed math.

But this calculation has two special properties:

1) The number you start with before the shift is divided back out at the end
(more or less), so the _result_ has to be less than 1<<$SHIFT and thus only
takes $SHIFT bits to store. With a maximum $SHIFT of 51 it has to fit in a 64
bit result with about a dozen bits to spare.

2) The bottom $SHIFT many bits are all zero before the divide.

We can use these two properties to easily break the math into chunks that
can't overflow by:

a) Chopping off the bottom X bits and dividing what's left by $NUMBER, keeping
both the dividend and the remainder. Choose any X that's big enough that this
step won't overflow. (I chose X=32, leaving at most 40-ish bits here).
b) Shift that dividend X bits to the left. This can't overflow because of
special property 1 above.
c) Shift the remainder X bits to the left. The remainder can't be larger than
the $NUMBER you started with, so if X+bits($NUMBER)<64 this has to fit too.
With X=32 and bits=20 we again have a dozen bits to spare.
d) Add the results of (b) and (c) together. Since the bottom X bits were all
zero, this is equivalent to having done the full divide. (Easy enough to mask
those bottom bits off and add them to the remainder before the divide if they
weren't, but we didn't need to do that because we know they were zero.)

So no arbitrary precision math is actually required here, and yes there's a
comment in the source about this. :)

Rob

[H. Peter Anvin]

Post by Rob Landley
C) The only calculation which can overflow 64 bits (the ADJ32 one) turns out
not to need arbitrary precision math, just 72 bits, and if it ever uses more
than 32 then bottom 32 are all zero before the divide so you can do it in
three lines.

... for the current code (32 bits). When we get an overflow-less 64-bit
implementation, this code will have to be redone, which is not true for
a properly done implementation.

-hpa

[Rob Landley]

Post by H. Peter Anvin

Post by Rob Landley
C) The only calculation which can overflow 64 bits (the ADJ32 one) turns
out not to need arbitrary precision math, just 72 bits, and if it ever
uses more than 32 then bottom 32 are all zero before the divide so you
can do it in three lines.

... for the current code (32 bits). When we get an overflow-less 64-bit
implementation, this code will have to be redone, which is not true for
a properly done implementation.

One extra mask and add is a strange definition of "redone", but I can add it
now if you like. (I'd personally prefer to wait for something to actually
need it, but...)

Post by H. Peter Anvin
-hpa

Rob

[Alejandro Mery]

Close, but no cee-gar. cut does something counter-intuitive with multiple
% echo 'a b' | awk '{print $2}'
b
% echo 'a b' | cut -d' ' -f2
% echo 'a b' | sed -r 's/[ ]+/ /g' | cut -d' ' -f2
b
Unfortunately, 'sed -r' isn't in the opengroup.org list of required options,
and sed 's/ / /g' doesn't DTRT for 3 or more blanks (as it won't recursively
apply the change to a *new* double blank formed by the previous change).

echo 'a b' | tr -s ' ' | cut -d' ' -f2
b

that is the light way ;-)

Alejandro Mery

[Ray Lee]

Post by Rob Landley
Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell script
is much simpler, about 1/4 the size, and runs on Red Hat 9 from 2003.
Peter Anvin added this perl to 2.6.25. Before that, the kernel had never
required perl to build.

Nice work. As the computations can all be done in 64-bit precision
now, and there have been concerns expressed about some shells not
supporting 64 bit integers, is there any reason this can't be done
using long longs in C?

Other than ruining a good bike shed argument, anyway.

[Rob Landley]

Post by Ray Lee

Post by Rob Landley
Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell
script is much simpler, about 1/4 the size, and runs on Red Hat 9 from
2003.
Peter Anvin added this perl to 2.6.25. Before that, the kernel had never
required perl to build.

Nice work.

Thanks. You'll definitely want to look at the _second_ version of that patch
rather than the first, though. :)

Post by Ray Lee
As the computations can all be done in 64-bit precision
now, and there have been concerns expressed about some shells not
supporting 64 bit integers, is there any reason this can't be done
using long longs in C?

Nope. Any of this could be done in C. (And that's the approach Sam Ravnborg
prefers to take for the second patch in the series, upgrading unifdef.c to do
everything itself.)

I tend to lean towards scripts that create header files rather than programs
that create header files, but as long as you remember to use HOSTCC it's
fairly straightforward. :)

Post by Ray Lee
Other than ruining a good bike shed argument, anyway.

Oh pile on. It beats being dismissed as the only one on the planet who cares
about the issue (again). :)

Rob

[Rob Landley]

From: Rob Landley <***@landley.net>

Remove perl from make headers_install by replacing a perl script (doing
a simple regex search and replace) with a smaller and faster shell script
implementation. The new shell script is a single for loop calling sed and
piping its output through unifdef to produce the target file.

Sam Ravnborg added this perl to 2.6.27.

Signed-off-by: Rob Landley <***@landley.net>
---

scripts/Makefile.headersinst | 6 ++--
scripts/headers_install.pl | 46 ---------------------------------
scripts/headers_install.sh | 23 ++++++++++++++++
3 files changed, 26 insertions(+), 49 deletions(-)

--- /dev/null 2008-11-21 04:46:41.000000000 -0600
+++ b/scripts/headers_install.sh 2008-12-15 22:09:45.000000000 -0600
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+# Grab arguments
+
+INDIR="$1"
+shift
+OUTDIR="$1"
+shift
+ARCH="$1"
+shift
+
+# Iterate through files listed on command line
+
+for i in "$@"
+do
+ sed -r \
+ -e 's/([ \t(])(__user|__force|__iomem)[ \t]/\1/g' \
+ -e 's/__attribute_const__([ \t]|$)/\1/g' \
+ -e 's@^#include <linux/compiler.h>@@' "$INDIR/$i" |
+ scripts/unifdef -U__KERNEL__ - > "$OUTDIR/$i"
+done
+
+exit 0
diff -r d9b501c91442 scripts/Makefile.headersinst
--- a/scripts/Makefile.headersinst Sun Dec 14 16:25:19 2008 -0800
+++ b/scripts/Makefile.headersinst Mon Dec 15 23:30:15 2008 -0600
@@ -44,8 +44,8 @@
quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
file$(if $(word 2, $(all-files)),s))
cmd_install = \
- $(PERL) $< $(srctree)/$(obj) $(install) $(SRCARCH) $(header-y); \
- $(PERL) $< $(objtree)/$(obj) $(install) $(SRCARCH) $(objhdr-y); \
+ $(CONFIG_SHELL) $< $(srctree)/$(obj) $(install) $(SRCARCH) $(header-y); \
+ $(CONFIG_SHELL) $< $(objtree)/$(obj) $(install) $(SRCARCH) $(objhdr-y); \
touch $@

quiet_cmd_remove = REMOVE $(unwanted)
@@ -64,7 +64,7 @@
@:

targets += $(install-file)
-$(install-file): scripts/headers_install.pl $(input-files) FORCE
+$(install-file): scripts/headers_install.sh $(input-files) FORCE
$(if $(unwanted),$(call cmd,remove),)
$(if $(wildcard $(dir $@)),,$(shell mkdir -p $(dir $@)))
$(call if_changed,install)
--- hg/scripts/headers_install.pl 2008-11-22 19:09:21.000000000 -0600
+++ /dev/null 1970-01-01 00:00:00 -0600
@@ -1,46 +0,0 @@
-#!/usr/bin/perl -w
-#
-# headers_install prepare the listed header files for use in
-# user space and copy the files to their destination.
-#
-# Usage: headers_install.pl readdir installdir arch [files...]
-# readdir: dir to open files
-# installdir: dir to install the files
-# arch: current architecture
-# arch is used to force a reinstallation when the arch
-# changes because kbuild then detect a command line change.
-# files: list of files to check
-#
-# Step in preparation for users space:
-# 1) Drop all use of compiler.h definitions
-# 2) Drop include of compiler.h
-# 3) Drop all sections defined out by __KERNEL__ (using unifdef)
-
-use strict;
-
-my ($readdir, $installdir, $arch, @files) = @ARGV;
-
-my $unifdef = "scripts/unifdef -U__KERNEL__";
-
-foreach my $file (@files) {
- local *INFILE;
- local *OUTFILE;
- my $tmpfile = "$installdir/$file.tmp";
- open(INFILE, "<$readdir/$file")
- or die "$readdir/$file: $!\n";
- open(OUTFILE, ">$tmpfile") or die "$tmpfile: $!\n";
- while (my $line = <INFILE>) {
- $line =~ s/([\s(])__user\s/$1/g;
- $line =~ s/([\s(])__force\s/$1/g;
- $line =~ s/([\s(])__iomem\s/$1/g;
- $line =~ s/\s__attribute_const__\s/ /g;
- $line =~ s/\s__attribute_const__$//g;
- $line =~ s/^#include <linux\/compiler.h>//;
- printf OUTFILE "%s", $line;
- }
- close OUTFILE;
- close INFILE;
- system $unifdef . " $tmpfile > $installdir/$file";
- unlink $tmpfile;

[Sam Ravnborg]

Post by Rob Landley
Remove perl from make headers_install by replacing a perl script (doing
a simple regex search and replace) with a smaller and faster shell script
implementation. The new shell script is a single for loop calling sed and
piping its output through unifdef to produce the target file.

OK

Post by Rob Landley
Sam Ravnborg added this perl to 2.6.27.

Drop this part - this is just to make you happy and for no use for others.

Post by Rob Landley
---
scripts/Makefile.headersinst | 6 ++--
scripts/headers_install.pl | 46 ---------------------------------
scripts/headers_install.sh | 23 ++++++++++++++++
3 files changed, 26 insertions(+), 49 deletions(-)
--- /dev/null 2008-11-21 04:46:41.000000000 -0600
+++ b/scripts/headers_install.sh 2008-12-15 22:09:45.000000000 -0600
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+# Grab arguments
+
+INDIR="$1"
+shift
+OUTDIR="$1"
+shift
+ARCH="$1"
+shift

Please add a short explanation what this file is used for
and what it does.
You can take more or less a direct copy from headers_install.pl

Post by Rob Landley
+
+# Iterate through files listed on command line
+
+do
+ sed -r \
+ -e 's/([ \t(])(__user|__force|__iomem)[ \t]/\1/g' \
+ -e 's/__attribute_const__([ \t]|$)/\1/g' \
+ scripts/unifdef -U__KERNEL__ - > "$OUTDIR/$i"
+done
+
+exit 0
diff -r d9b501c91442 scripts/Makefile.headersinst
--- a/scripts/Makefile.headersinst Sun Dec 14 16:25:19 2008 -0800
+++ b/scripts/Makefile.headersinst Mon Dec 15 23:30:15 2008 -0600
@@ -44,8 +44,8 @@
quiet_cmd_install = INSTALL $(printdir) ($(words $(all-files))\
file$(if $(word 2, $(all-files)),s))
cmd_install = \
- $(PERL) $< $(srctree)/$(obj) $(install) $(SRCARCH) $(header-y); \
- $(PERL) $< $(objtree)/$(obj) $(install) $(SRCARCH) $(objhdr-y); \
+ $(CONFIG_SHELL) $< $(srctree)/$(obj) $(install) $(SRCARCH) $(header-y); \
+ $(CONFIG_SHELL) $< $(objtree)/$(obj) $(install) $(SRCARCH) $(objhdr-y); \

Stick to the coding style i the file and do not add your own.
Makefile.headersinst uses tabs for commands and not for indent.



Sam

[Rob Landley]

From: Rob Landley <***@landley.net>

Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh.

This script generates kernel/cpu/capflags.c from include/asm/cpufeature.h.

Peter Anvin added this perl to 2.6.28.

Signed-off-by: Rob Landley <***@landley.net>
---

arch/x86/kernel/cpu/Makefile | 4 +--
arch/x86/kernel/cpu/mkcapflags.pl | 32 ----------------------------
arch/x86/kernel/cpu/mkcapflags.sh | 28 ++++++++++++++++++++++++
3 files changed, 30 insertions(+), 34 deletions(-)

diff -ruN alt-linux/arch/x86/kernel/cpu/Makefile alt-linux2/arch/x86/kernel/cpu/Makefile
--- alt-linux/arch/x86/kernel/cpu/Makefile 2008-12-24 17:26:37.000000000 -0600
+++ alt-linux2/arch/x86/kernel/cpu/Makefile 2008-12-28 18:10:51.000000000 -0600
@@ -23,10 +23,10 @@
obj-$(CONFIG_X86_LOCAL_APIC) += perfctr-watchdog.o

quiet_cmd_mkcapflags = MKCAP $@
- cmd_mkcapflags = $(PERL) $(srctree)/$(src)/mkcapflags.pl $< $@
+ cmd_mkcapflags = $(CONFIG_SHELL) $(srctree)/$(src)/mkcapflags.sh $< $@

cpufeature = $(src)/../../include/asm/cpufeature.h

targets += capflags.c
-$(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.pl FORCE
+$(obj)/capflags.c: $(cpufeature) $(src)/mkcapflags.sh FORCE
$(call if_changed,mkcapflags)
diff -ruN alt-linux/arch/x86/kernel/cpu/mkcapflags.pl alt-linux2/arch/x86/kernel/cpu/mkcapflags.pl
--- alt-linux/arch/x86/kernel/cpu/mkcapflags.pl 2008-12-24 17:26:37.000000000 -0600
+++ alt-linux2/arch/x86/kernel/cpu/mkcapflags.pl 1969-12-31 18:00:00.000000000 -0600
@@ -1,32 +0,0 @@
-#!/usr/bin/perl
-#
-# Generate the x86_cap_flags[] array from include/asm-x86/cpufeature.h
-#
-
-($in, $out) = @ARGV;
-
-open(IN, "< $in\0") or die "$0: cannot open: $in: $!\n";
-open(OUT, "> $out\0") or die "$0: cannot create: $out: $!\n";
-
-print OUT "#include <asm/cpufeature.h>\n\n";
-print OUT "const char * const x86_cap_flags[NCAPINTS*32] = {\n";
-
-while (defined($line = <IN>)) {
- if ($line =~ /^\s*\#\s*define\s+(X86_FEATURE_(\S+))\s+(.*)$/) {
- $macro = $1;
- $feature = $2;
- $tail = $3;
- if ($tail =~ /\/\*\s*\"([^"]*)\".*\*\//) {
- $feature = $1;
- }
-
- if ($feature ne '') {
- printf OUT "\t%-32s = \"%s\",\n",
- "[$macro]", "\L$feature";
- }
- }
-}
-print OUT "};\n";
-
-close(IN);
-close(OUT);
diff -ruN alt-linux/arch/x86/kernel/cpu/mkcapflags.sh alt-linux2/arch/x86/kernel/cpu/mkcapflags.sh
--- alt-linux/arch/x86/kernel/cpu/mkcapflags.sh 1969-12-31 18:00:00.000000000 -0600
+++ alt-linux2/arch/x86/kernel/cpu/mkcapflags.sh 2008-12-28 18:08:50.000000000 -0600
@@ -0,0 +1,28 @@
+#!/bin/bash
+#
+# Generate the x86_cap_flags[] array from include/asm/cpufeature.h
+#
+
+IN=$1
+OUT=$2
+
+(
+ echo "#include <asm/cpufeature.h>"
+ echo ""
+ echo "const char * const x86_cap_flags[NCAPINTS*32] = {"
+
+ # Iterate through any input lines starting with #define X86_FEATURE_
+ sed -n -e 's/\t/ /g' -e 's/^ *# *define *X86_FEATURE_//p' $IN |
+ while read i
+ do
+ # Name is everything up to the first whitespace
+ NAME="$(echo "$i" | sed 's/ .*//')"
+
+ # If the /* comment */ starts with a quote string, grab that.
+ VALUE="$(echo "$i" | sed -n '***@.*/\* *\("[^"]*"\).*\*/@\***@p')"
+ [ -z "$VALUE" ] && VALUE="\"$(echo "$NAME" | tr A-Z a-z)\""
+
+ [ "$VALUE" != '""' ] && echo " [X86_FEATU

[Sam Ravnborg]

Post by Rob Landley
Convert kernel/cpu/mkcapflags.pl to kernel/cpu/mkcapflags.sh.
This script generates kernel/cpu/capflags.c from include/asm/cpufeature.h.

OK

Post by Rob Landley
Peter Anvin added this perl to 2.6.28.

Drop it.

Implementation looks OK.

Sam

[Arkadiusz Miskiewicz]

Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0=

dc2 )

building a Linux kernel never required perl to be installed on the bu=

ild

system. (Various development and debugging scripts were written in p=

erl

and python and such, but they weren't involved in actually building a
kernel.) Building a kernel before 2.6.25 could be done with a minimal
system built from gcc, binutils, bash, make, busybox, uClibc, and the=

Linux

kernel, and nothing else.

And now bash is going to be required... while some distros don't need/h=
ave=20
bash. /bin/sh should be enough.

Heh,
--=20
Arkadiusz Mi=C5=9Bkiewicz PLD/Linux Team
arekm / maven.pl http://ftp.pld-linux.org/

[Christoph Hellwig]

Post by Rob Landley
Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 )
building a Linux kernel never required perl to be installed on the build
system. (Various development and debugging scripts were written in perl
and python and such, but they weren't involved in actually building a
kernel.) Building a kernel before 2.6.25 could be done with a minimal
system built from gcc, binutils, bash, make, busybox, uClibc, and the Linux
kernel, and nothing else.

And now bash is going to be required... while some distros don't need/have
bash. /bin/sh should be enough.

*nod* bash is in many ways a worse requirement than perl. strict posix
/bin/sh + awk + sed would be nicest, but if that's too much work perl
seems reasonable.

[Alejandro Mery]

Post by Christoph Hellwig

Post by Rob Landley
Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 )
building a Linux kernel never required perl to be installed on the build
system. (Various development and debugging scripts were written in perl
and python and such, but they weren't involved in actually building a
kernel.) Building a kernel before 2.6.25 could be done with a minimal
system built from gcc, binutils, bash, make, busybox, uClibc, and the Linux
kernel, and nothing else.

And now bash is going to be required... while some distros don't need/have
bash. /bin/sh should be enough.

*nod* bash is in many ways a worse requirement than perl. strict posix
/bin/sh + awk + sed would be nicest, but if that's too much work perl
seems reasonable.

well, bash is not worse as bash is trivial to cross-compile to run on a
constrained sandbox and perl is a nightmare, but I agree bash should be
avoided too.

I think the $(( ... )) bash-ism can be replaced with a simple .c helper toy.

Thank Rob for reopening the topic.

Alejandro Mery

[Mark Miller]

On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote=

Before 2.6.25 (specifically git =20
bdc807871d58285737d50dc6163d0feb72cb0dc2 )
building a Linux kernel never required perl to be installed on =20
the build
system. (Various development and debugging scripts were written =20
in perl
and python and such, but they weren't involved in actually =20
building a
kernel.) Building a kernel before 2.6.25 could be done with a =20
minimal
system built from gcc, binutils, bash, make, busybox, uClibc, and =

=20

the Linux
kernel, and nothing else.

And now bash is going to be required... while some distros don't =20
need/have
bash. /bin/sh should be enough.

*nod* bash is in many ways a worse requirement than perl. strict =20
posix
/bin/sh + awk + sed would be nicest, but if that's too much work per=

l

seems reasonable.

well, bash is not worse as bash is trivial to cross-compile to run =20
on a
constrained sandbox and perl is a nightmare, but I agree bash should =

=20

be
avoided too.
I think the $(( ... )) bash-ism can be replaced with a simple .c =20
helper toy.
Thank Rob for reopening the topic.
Alejandro Mery

And actually, one of the things that I just recalled, is that several =20
of the Perl configure scripts in order to actually build itself, rely =20
on Bourne shell calls. So the argument to require a strict POSIX+sed=20
+awk implementation rather than Perl to build the kernel, fails, since =
=20
you already require some variant of shell greater than strict POSIX /=20
bin/sh to build Perl. So this is one less dependency.

Also, attempting to cross-compile Perl, is indeed a nightmare.

--
Mark Miller
***@mirell.org

[Matt Keenan]

=20

On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wro=

Before 2.6.25 (specifically git =20
bdc807871d58285737d50dc6163d0feb72cb0dc2 )
building a Linux kernel never required perl to be installed on =20
the build
system. (Various development and debugging scripts were written=

=20

in perl
and python and such, but they weren't involved in actually =20
building a
kernel.) Building a kernel before 2.6.25 could be done with a =20
minimal
system built from gcc, binutils, bash, make, busybox, uClibc, an=

d =20

the Linux
kernel, and nothing else.

And now bash is going to be required... while some distros don't =

=20

need/have
bash. /bin/sh should be enough.

*nod* bash is in many ways a worse requirement than perl. strict=

=20

posix
/bin/sh + awk + sed would be nicest, but if that's too much work p=

erl

seems reasonable.

=20

well, bash is not worse as bash is trivial to cross-compile to run =

=20

on a
constrained sandbox and perl is a nightmare, but I agree bash shoul=

d =20

be
avoided too.
I think the $(( ... )) bash-ism can be replaced with a simple .c =20
helper toy.
Thank Rob for reopening the topic.
Alejandro Mery

=20
And actually, one of the things that I just recalled, is that several=

=20

of the Perl configure scripts in order to actually build itself, rely=

=20

on Bourne shell calls. So the argument to require a strict POSIX+sed=20
+awk implementation rather than Perl to build the kernel, fails, sinc=

e =20

you already require some variant of shell greater than strict POSIX /=

=20

bin/sh to build Perl. So this is one less dependency.
=20
Also, attempting to cross-compile Perl, is indeed a nightmare.
=20

Having cross compiled Perl, on to a Unix with a brain dead third party
TCP/IP no less, only 4 years after starting to use Unix / Linux I can
attest that it is not that difficult to cross compile. Heck it even run=
s
on those weird Crays that don't know which byte sex they are.

Matt

[Måns Rullgård]

On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrote=
=20

=20

Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72=

cb0dc2 )

building a Linux kernel never required perl to be installed on the=

build

system. (Various development and debugging scripts were written i=

n perl

and python and such, but they weren't involved in actually buildin=

g a

kernel.) Building a kernel before 2.6.25 could be done with a mini=

mal

system built from gcc, binutils, bash, make, busybox, uClibc, and =

the Linux

kernel, and nothing else.
=20

And now bash is going to be required... while some distros don't ne=

ed/have=20

bash. /bin/sh should be enough.
=20

*nod* bash is in many ways a worse requirement than perl. strict p=

osix

/bin/sh + awk + sed would be nicest, but if that's too much work per=

l

seems reasonable.

well, bash is not worse as bash is trivial to cross-compile to run on=

a

constrained sandbox and perl is a nightmare, but I agree bash should =

be

avoided too.
I think the $(( ... )) bash-ism can be replaced with a simple .c help=

er toy.

The $(( ... )) construct is standard POSIX shell syntax, see
http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.html=
#tag_02_06_04

Bash supports $[ ... ] as an alternate syntax for the same thing.
Perhaps you were thinking of that.

--=20
M=E5ns Rullg=E5rd
***@mansr.com

[Pádraig Brady]

Post by Måns Rullgård
=20

I think the $(( ... )) bash-ism can be replaced with a simple .c hel=

per toy.

Post by Måns Rullgård
=20
The $(( ... )) construct is standard POSIX shell syntax, see
http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.ht=

ml#tag_02_06_04

Post by Måns Rullgård
=20
Bash supports $[ ... ] as an alternate syntax for the same thing.
Perhaps you were thinking of that.

I think the misconception that $(( ... )) is a bashism is caused by
the wrong highlighting defaults chosen by vim. To fix this add this to =
~/.vimrc

let g:is_posix =3D 1

That will also allow you to use the $(command) POSIX construct.
BTW, the vim syntax maintainers don't agree with changing this default:
http://groups.google.com/group/vim_dev/browse_thread/thread/41139a32772=
b2f5f

cheers,
P=E1draig.

[Jamie Lokier]

Post by Pádraig Brady

Post by Måns Rullgård
The $(( ... )) construct is standard POSIX shell syntax, see
http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02.=

html#tag_02_06_04

Post by Pádraig Brady

Post by Måns Rullgård
=20
Bash supports $[ ... ] as an alternate syntax for the same thing.
Perhaps you were thinking of that.

=20
I think the misconception that $(( ... )) is a bashism is caused by
the wrong highlighting defaults chosen by vim.

I think the misconception is because traditional unix bourne shells
don't implement that construct. I just tried it on a few machines,
and it failed on 4 of them. Admittedly, the only up to date one is
running Solaris 10; the others are older unixes that you're unlikely
to build Linux on.

-- Jamie

[Måns Rullgård]

Post by Jamie Lokier

Post by Pádraig Brady

Post by Måns Rullgård
The $(( ... )) construct is standard POSIX shell syntax, see
http://www.opengroup.org/onlinepubs/000095399/utilities/xcu_chap02=

=2Ehtml#tag_02_06_04

Post by Jamie Lokier

Post by Pádraig Brady

Post by Måns Rullgård
=20
Bash supports $[ ... ] as an alternate syntax for the same thing.
Perhaps you were thinking of that.

=20
I think the misconception that $(( ... )) is a bashism is caused by
the wrong highlighting defaults chosen by vim.

I think the misconception is because traditional unix bourne shells
don't implement that construct. I just tried it on a few machines,
and it failed on 4 of them. Admittedly, the only up to date one is
running Solaris 10; the others are older unixes that you're unlikely
to build Linux on.

On Solaris, /usr/xpg4/bin/sh is usually a POSIX-compliant shell. I
don't know how long it has been there.

--=20
M=E5ns Rullg=E5rd
***@mansr.com

[Rob Landley]

On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wrot=

Post by Arkadiusz Miskiewicz

Post by Rob Landley
Before 2.6.25 (specifically git
bdc807871d58285737d50dc6163d0feb72cb0dc2 ) building a Linux kerne=

l

Post by Arkadiusz Miskiewicz

Post by Rob Landley
never required perl to be installed on the build system. (Variou=

s

Post by Arkadiusz Miskiewicz

Post by Rob Landley
development and debugging scripts were written in perl and python=

and

Post by Arkadiusz Miskiewicz

Post by Rob Landley
such, but they weren't involved in actually building a kernel.)
Building a kernel before 2.6.25 could be done with a minimal syst=

em

Post by Arkadiusz Miskiewicz

Post by Rob Landley
built from gcc, binutils, bash, make, busybox, uClibc, and the Li=

nux

Post by Arkadiusz Miskiewicz

Post by Rob Landley
kernel, and nothing else.

And now bash is going to be required... while some distros don't
need/have bash. /bin/sh should be enough.

*nod* bash is in many ways a worse requirement than perl. strict =

posix

/bin/sh + awk + sed would be nicest, but if that's too much work pe=

rl

seems reasonable.

well, bash is not worse as bash is trivial to cross-compile to run on=

a

constrained sandbox and perl is a nightmare, but I agree bash should =

be

avoided too.
I think the $(( ... )) bash-ism can be replaced with a simple .c help=

er

toy.

No, $[ ] is the bashism, $(( )) is susv3:
http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html=
#tag_18_06_04

I intentionally switched from $[ ] to $(( )) to make dash work.

Rob

[Sam Ravnborg]

On Fri, Jan 02, 2009 at 10:26:37AM +0100, Arkadiusz Miskiewicz wr=

Post by Arkadiusz Miskiewicz

Post by Rob Landley
Before 2.6.25 (specifically git
bdc807871d58285737d50dc6163d0feb72cb0dc2 ) building a Linux ker=

nel

Post by Arkadiusz Miskiewicz

Post by Rob Landley
never required perl to be installed on the build system. (Vari=

ous

Post by Arkadiusz Miskiewicz

Post by Rob Landley
development and debugging scripts were written in perl and pyth=

on and

Post by Arkadiusz Miskiewicz

Post by Rob Landley
such, but they weren't involved in actually building a kernel.)
Building a kernel before 2.6.25 could be done with a minimal sy=

stem

Post by Arkadiusz Miskiewicz

Post by Rob Landley
built from gcc, binutils, bash, make, busybox, uClibc, and the =

Linux

Post by Arkadiusz Miskiewicz

Post by Rob Landley
kernel, and nothing else.

And now bash is going to be required... while some distros don't
need/have bash. /bin/sh should be enough.

*nod* bash is in many ways a worse requirement than perl. stric=

t posix

/bin/sh + awk + sed would be nicest, but if that's too much work =

perl

seems reasonable.

well, bash is not worse as bash is trivial to cross-compile to run =

on a

constrained sandbox and perl is a nightmare, but I agree bash shoul=

d be

avoided too.
I think the $(( ... )) bash-ism can be replaced with a simple .c he=

lper

toy.

=20
http://www.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.ht=

ml#tag_18_06_04

=20
I intentionally switched from $[ ] to $(( )) to make dash work.

The focus on this patch is to create a minimal set of
dependencies so please document what dependencies / compatibility
this patch introduces.

It is not obvious for me for example if the script
requires sh, bash or dash or whatever.

The shebang is so often wrong that this is not docuemnting such
things.

Sam

[Rob Landley]

Post by Christoph Hellwig

Post by Arkadiusz Miskiewicz

Post by Rob Landley
Before 2.6.25 (specifically git
bdc807871d58285737d50dc6163d0feb72cb0dc2 ) building a Linux kernel
never required perl to be installed on the build system. (Various
development and debugging scripts were written in perl and python and
such, but they weren't involved in actually building a kernel.)
Building a kernel before 2.6.25 could be done with a minimal system
built from gcc, binutils, bash, make, busybox, uClibc, and the Linux
kernel, and nothing else.

And now bash is going to be required... while some distros don't
need/have bash. /bin/sh should be enough.

*nod* bash is in many ways a worse requirement than perl. strict posix
/bin/sh + awk + sed would be nicest, but if that's too much work perl
seems reasonable.

The scripts should work with dash (modulo the one that needs 64 bit math,
which dash only provides on a 64 bit host), or with busybox ash (which can
provide 64 bit math on 32 bit hosts just like bash can). I'll explicitly
retest both of those when I respin the patches in the <strike>morning</strike>
afternoon.

(And yes I thought about writing my own arbitrary precision arithmetic shell
functions, but it really didn't seem worth the complexity since the only 32
bit Linux distros I've seen that install dash also install bash by default. I
just put in a test for 32 bit math so it can spot it and fail, on the off
chance you're running a 32 bit host with dash after explicitly uninstalling
bash. All the embedded 32 bit ones that try to act as development
environments use busybox ash, or more often just install bash.)

That said, how is bash _worse_ than perl? (Where's the second implementation
of perl? Even Python had jython, but perl has... what? The attempt to rebase
on Parrot went down in flames...)

If the argument is that "depending on a specific shell implementation is as
bad as depending on the one and only implementation of perl", that argument I
can at least follow, even if it doesn't actually apply in this case. But
where does "worse" come in?

Rob

[Theodore Tso]

Post by Rob Landley
That said, how is bash _worse_ than perl? (Where's the second
implementation of perl? Even Python had jython, but perl
has... what? The attempt to rebase on Parrot went down in
flames...)

(1) bash implies POSIX extensions; perl is actually quite portable.

(2) There are distributions that install with perl by default but not
bash; they use dash for speed reasons.

Sounds like though modulo dealing with 64-bit arithmetic, your patches
are mostly dash/POSIX.2 comformant, so you're probably mostly good on
that front once you address the 32/64-bit issues. I'd also suggest
explicitly add a reminder to the shell scripts' comments to avoid
bashisms for maximum portability, to remind developers in the future
who might try to change the shell scripts to watch out for portability
issues.

- Ted

[Mark Miller]

Before 2.6.25 (specifically git =20
bdc807871d58285737d50dc6163d0feb72cb0dc2 )
building a Linux kernel never required perl to be installed on the =20
build
system. (Various development and debugging scripts were written in =

=20

perl
and python and such, but they weren't involved in actually building =

a

kernel.) Building a kernel before 2.6.25 could be done with a minima=

l

system built from gcc, binutils, bash, make, busybox, uClibc, and =20
the Linux
kernel, and nothing else.

And now bash is going to be required... while some distros don't =20
need/have
bash. /bin/sh should be enough.

Which distros only have /bin/sh which do not have Perl? I'm honestly =20
curious.

--=20
Arkadiusz Mi=C5=9Bkiewicz PLD/Linux Team
arekm / maven.pl http://ftp.pld-linux.org/

--
Mark Miller
***@mirell.org

[Mark Miller]

Post by Mark Miller

Before 2.6.25 (specifically git =20
bdc807871d58285737d50dc6163d0feb72cb0dc2 )
building a Linux kernel never required perl to be installed on the =

=20

Post by Mark Miller

build
system. (Various development and debugging scripts were written =20
in perl
and python and such, but they weren't involved in actually =20
building a
kernel.) Building a kernel before 2.6.25 could be done with a =20
minimal
system built from gcc, binutils, bash, make, busybox, uClibc, and =20
the Linux
kernel, and nothing else.

And now bash is going to be required... while some distros don't =20
need/have
bash. /bin/sh should be enough.

Which distros only have /bin/sh which do not have Perl? I'm honestly =

=20

Post by Mark Miller
curious.

That is, *do* have Perl. Typo there.

Post by Mark Miller

--=20
Arkadiusz Mi=C5=9Bkiewicz PLD/Linux Team
arekm / maven.pl http://ftp.pld-linux.org/

--
Mark Miller
***@mirell.org

[Rob Landley]

Post by Rob Landley
Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2
) building a Linux kernel never required perl to be installed on the
build system. (Various development and debugging scripts were written in
perl and python and such, but they weren't involved in actually building
a kernel.) Building a kernel before 2.6.25 could be done with a minimal
system built from gcc, binutils, bash, make, busybox, uClibc, and the
Linux kernel, and nothing else.

And now bash is going to be required... while some distros don't need/have
bash. /bin/sh should be enough.
Heh,

I believe all three scripts run under dash and busybox ash. (The timeconst.sh
one needs 64 bit math which dash only provides on 64 bit hosts, which is a
regression from Red Hat 9 in 2003 by the way. Busybox ash can also provide 64
bit math on 32 bit hosts, and the script should run with that just fine if you
haven't got bash and that's what your "sh" in the path is.)

The makefiles execute those scripts via CONFIG_SHELL, not via the #!/blah line
at the start, so it's largely irrelevant what gets put there anyway. If you
haven't got bash installed it'll use "sh", which should work with dash on a 64
bit host or with busybox ash. (That's why that one file has a test to make
sure 64 bit math _does_ work. The only Linux development environment I'm
aware of where that test would trigger is if you use a 32 bit ubuntu and go
out of your way to _uninstall_ bash. Even Cygwin uses bash.)

Beyond that, "find linux -type f | xargs grep bin/bash | wc" comes up with 38
instances (admittedly half of 'em in Documentation, but lots in scripts, and
makefiles, and defconfigs, at least one hardwired in C code.) So this would
not a _new_ dependency.

By the way, what Linux distros install a compiler toolchain but not bash? I'm
curious. (Even after Ubuntu moved #!/bin/sh to point to dash, it still
installs bash as part of the default environment, even if you don't install
development tools.) You've built the kernel on this system before?

Rob

[Matthieu CASTET]

Heh,

=20
I believe all three scripts run under dash and busybox ash. (The tim=

econst.sh=20

one needs 64 bit math which dash only provides on 64 bit hosts, which=

is a=20

regression from Red Hat 9 in 2003 by the way.

With dash 0.5.4-12 (from debian sid), I seems I got the 64 bit math for
32 bit hosts :
$ uname -m
i686
$ dash -c 'echo $((1<<32))'
4294967296


Matthieu

[Paul Mundt]

Post by Rob Landley
Before 2.6.25 (specifically git bdc807871d58285737d50dc6163d0feb72cb0dc2 )
building a Linux kernel never required perl to be installed on the build
system. (Various development and debugging scripts were written in perl and
python and such, but they weren't involved in actually building a kernel.)
Building a kernel before 2.6.25 could be done with a minimal system built from
gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel, and nothing
else. (Embedded developers creating clean cross compile environments that
won't leak bits of the host system into the target, or bootstrapping native
development environments to run on target hardware or under emulators, tend to
care about this sort of thing.)
The perl checkin for 2.6.25 was the camel's nose under the tent flap, and
since then two more instances of perl have shown up in the core kernel build.
This patch series removes the three required to build a basic kernel for qemu
for the targets I've tested (arm, mips, powerpc, sparc, m68k, sh4, and of
course x86 and x86-64), replacing them with shell scripts.

Misguided rhetoric aside, what does this actually accomplish? If folks
add meaningful tools in to the kernel that require python, and it is
generally regarded as being fairly ubiquitous, I can't imagine there
being any substantiated objections against it.

Your main reasons against inclusion of perl seem to be that there is no
realistic expectation for target systems that will be self-hosting will
have perl included, or the inherent complexity in maintaining a coherent
cross compiling environment. Both of these things are issues with your
own environment, and in no way are these representative of the embedded
development community at large.

Now with that out of the way, this entire series fundamentally fails to
convert half of the perl scripts shipped with the kernel today, some that
are required for build depending on Kconfig options, and others that are
simply useful tools for self-hosted environments. Simply converting a
couple of scripts over you find objectionable is certainly fine if there
is a real benefit in doing so, but this doesn't actually accomplish your
goal of removing the perl dependency.

Ignoring the compile-time dependencies that you overlooked, what you
define as "development and debugging" scripts are still an integral part
of the system, unless you are trying to argue that embedded developers
have no interest in things like checkstack due to the trouble of trying
to get perl built.

Until you can post a series that converts all of scripts/*.pl in its
entirety, you have failed to address the fundamental reason why perl is
used in the first place. Trying to toss bizarre policy statements around
regarding things you personally find objectionable without any coherent
technical argument to the contrary is of no constructive use whatsoever.

The kernel is and always has been about using the right tool for the job,
not a matter of dictating what tools you must use in order to accomplish
a task you are interested in. Common sense does apply here though, so
this might be a more daunting task for some than others.

[Mark Miller]

Post by Paul Mundt

Post by Rob Landley
Before 2.6.25 (specifically git
bdc807871d58285737d50dc6163d0feb72cb0dc2 )
building a Linux kernel never required perl to be installed on the build
system. (Various development and debugging scripts were written in perl and
python and such, but they weren't involved in actually building a kernel.)
Building a kernel before 2.6.25 could be done with a minimal system built from
gcc, binutils, bash, make, busybox, uClibc, and the Linux kernel, and nothing
else. (Embedded developers creating clean cross compile
environments that
won't leak bits of the host system into the target, or
bootstrapping native
development environments to run on target hardware or under
emulators, tend to
care about this sort of thing.)
The perl checkin for 2.6.25 was the camel's nose under the tent flap, and
since then two more instances of perl have shown up in the core kernel build.
This patch series removes the three required to build a basic
kernel for qemu
for the targets I've tested (arm, mips, powerpc, sparc, m68k, sh4, and of
course x86 and x86-64), replacing them with shell scripts.

Misguided rhetoric aside, what does this actually accomplish? If folks
add meaningful tools in to the kernel that require python, and it is
generally regarded as being fairly ubiquitous, I can't imagine there
being any substantiated objections against it.

And if the said meaningful tools introduce complex dependencies, then
there should be an open discussion as to why exactly we need those
tools as opposed to a simpler implementation.

Post by Paul Mundt
Your main reasons against inclusion of perl seem to be that there is no
realistic expectation for target systems that will be self-hosting will
have perl included, or the inherent complexity in maintaining a coherent
cross compiling environment. Both of these things are issues with your
own environment, and in no way are these representative of the
embedded
development community at large.

I feel that if you attempted to look for discussions on "cross-
compiling perl", you will meet with a variety of complaints on what a
nightmare it is to do so in a sandboxed environment.

Post by Paul Mundt
Now with that out of the way, this entire series fundamentally fails to
convert half of the perl scripts shipped with the kernel today, some that
are required for build depending on Kconfig options, and others that are
simply useful tools for self-hosted environments. Simply converting a
couple of scripts over you find objectionable is certainly fine if there
is a real benefit in doing so, but this doesn't actually accomplish your
goal of removing the perl dependency.

From what I can tell, it allows one to fully build the Linux kernel
without Perl. Without these series of patches, to actually *build* the
kernel, one requires Perl. Unless there is an alternate way to do
"make headers_install" that I am not seeing, that appears to now be
done in Perl. All the other Perl scripts were merely nice to have, not
necessary to build a Linux kernel.

Post by Paul Mundt
Ignoring the compile-time dependencies that you overlooked, what you
define as "development and debugging" scripts are still an integral part
of the system, unless you are trying to argue that embedded developers
have no interest in things like checkstack due to the trouble of trying
to get perl built.

Do I need that to compile a kernel? No.

Post by Paul Mundt
Until you can post a series that converts all of scripts/*.pl in its
entirety, you have failed to address the fundamental reason why perl is
used in the first place. Trying to toss bizarre policy statements around
regarding things you personally find objectionable without any
coherent
technical argument to the contrary is of no constructive use
whatsoever.

Perl was not required to build the Linux kernel. Now it is. It adds
another dependency to the Linux kernel. Requiring bash is far less a
dependency that Perl is.

Post by Paul Mundt
The kernel is and always has been about using the right tool for the job,
not a matter of dictating what tools you must use in order to
accomplish
a task you are interested in. Common sense does apply here though, so
this might be a more daunting task for some than others.

How is Perl a better tool for the job than what currently bash and
other standard utilities already offer?

--
Mark Miller
***@mirell.org

[Paul Mundt]

Post by Mark Miller

Post by Paul Mundt
Misguided rhetoric aside, what does this actually accomplish? If folks
add meaningful tools in to the kernel that require python, and it is
generally regarded as being fairly ubiquitous, I can't imagine there
being any substantiated objections against it.

And if the said meaningful tools introduce complex dependencies, then
there should be an open discussion as to why exactly we need those
tools as opposed to a simpler implementation.

Complex is relative, something that is fairly ubiquitious can hardly be
labelled as complex, regardless of whether historically people have had
issues with that dependency in certain spaces. In any event, simplifying
things is always good, but this open discussion thing is pure fancy,
since when was the kernel a democracy?

Post by Mark Miller

Post by Paul Mundt
Your main reasons against inclusion of perl seem to be that there is
no realistic expectation for target systems that will be self-hosting
will have perl included, or the inherent complexity in maintaining a
coherent cross compiling environment. Both of these things are issues
with your own environment, and in no way are these representative of
the embedded development community at large.

I feel that if you attempted to look for discussions on "cross-
compiling perl", you will meet with a variety of complaints on what a
nightmare it is to do so in a sandboxed environment.

I've had to deal with cross compiling perl for over a decade, in all of
its various forms, in all manner of embedded applications, so please tell
someone who cares. There are various options around this headache today,
as there have been for ages (though the options these days are rather
less painful), and if you felt like attempting to look for discussions on
those rather than trying to push this silly matter perhaps you might come
up with something.

The key thing you hit on is that there are a variety of complaints, and
that is all they have ever been. If your system is so constrained, you
shouldn't be doing builds on it in the first place. You certainly won't
be able to build anything in your crippled environment outside of some
simple applications anyways, this isn't a valid technical reason for
keeping the kernel a policy stranglehold.

Post by Mark Miller

Post by Paul Mundt
Now with that out of the way, this entire series fundamentally fails
to convert half of the perl scripts shipped with the kernel today,
some that are required for build depending on Kconfig options,

[snip]

Post by Mark Miller
From what I can tell, it allows one to fully build the Linux kernel
without Perl.

Wrong, re-read what I said and try again.

Post by Mark Miller

Post by Paul Mundt
Ignoring the compile-time dependencies that you overlooked, what you
define as "development and debugging" scripts are still an integral
part of the system, unless you are trying to argue that embedded
developers have no interest in things like checkstack due to the
trouble of trying to get perl built.

Do I need that to compile a kernel? No.

Compile-time dependencies you do, yes.

Post by Mark Miller
Perl was not required to build the Linux kernel. Now it is. It adds
another dependency to the Linux kernel. Requiring bash is far less a
dependency that Perl is.

Perhaps so, but that is largely irrelevant. Moving off of the bash
dependency is a lot more trivial than moving off of the perl one. The
kernel bashisms are fairly marginal, and if someone were to do the leg
work for that, it might even be accepted. Getting rid of perl is an
uphill battle for no visible gain. People will continue to write and add
scripts both in bash and perl on a routine basis regardless.

Post by Mark Miller

Post by Paul Mundt
The kernel is and always has been about using the right tool for the
job, not a matter of dictating what tools you must use in order to
accomplish a task you are interested in. Common sense does apply here
though, so this might be a more daunting task for some than others.

How is Perl a better tool for the job than what currently bash and
other standard utilities already offer?

How is bash a better tool for than job than what sed and posix shell
already offer? Yes, we can reduce our dependencies to the bare minimum,
but that is not constructive for the folks that are actually writing the
scripts in the first place.

Likewise, this is not even a real problem in the embedded developer
demographic, the only place this is a problem is in specially stripped
distributions or people that don't want to go through the pain of cross
compiling perl. None of which is of any concern.

If people are going to write useful things that are reasonably expected
to be standard on build machines, there is no reason to restrict what
tools they are permitted to use. If you have a personal vendetta against
something that is fairly standard, that is entirely your own personal
problem, you can choose to deal with it or patch out of tree for your own
crippled environment (assuming patch isn't too much of an obscure
dependency).

[Mark Miller]

Post by Paul Mundt

Post by Mark Miller

Post by Paul Mundt
Misguided rhetoric aside, what does this actually accomplish? If folks
add meaningful tools in to the kernel that require python, and it is
generally regarded as being fairly ubiquitous, I can't imagine there
being any substantiated objections against it.

And if the said meaningful tools introduce complex dependencies, then
there should be an open discussion as to why exactly we need those
tools as opposed to a simpler implementation.

Complex is relative, something that is fairly ubiquitious can hardly be
labelled as complex, regardless of whether historically people have had
issues with that dependency in certain spaces. In any event,
simplifying
things is always good, but this open discussion thing is pure fancy,
since when was the kernel a democracy?

I'm ignoring the bait.

Post by Paul Mundt

Post by Mark Miller

Post by Paul Mundt
Your main reasons against inclusion of perl seem to be that there is
no realistic expectation for target systems that will be self-
hosting
will have perl included, or the inherent complexity in maintaining a
coherent cross compiling environment. Both of these things are issues
with your own environment, and in no way are these representative of
the embedded development community at large.

I feel that if you attempted to look for discussions on "cross-
compiling perl", you will meet with a variety of complaints on what a
nightmare it is to do so in a sandboxed environment.

I've had to deal with cross compiling perl for over a decade, in all of
its various forms, in all manner of embedded applications, so please tell
someone who cares.

Ignoring this as well.

Post by Paul Mundt
There are various options around this headache today,
as there have been for ages (though the options these days are rather
less painful), and if you felt like attempting to look for
discussions on
those rather than trying to push this silly matter perhaps you might come
up with something.

And ignoring this too.

Post by Paul Mundt
The key thing you hit on is that there are a variety of complaints, and
that is all they have ever been. If your system is so constrained, you
shouldn't be doing builds on it in the first place. You certainly won't
be able to build anything in your crippled environment outside of some
simple applications anyways, this isn't a valid technical reason for
keeping the kernel a policy stranglehold.

I merely don't like seeing another dependency added when there's no
logical reason to do it, other than "why not". And there have been
reasons given for the "not". Your reply, seems merely to be, "Because."

Post by Paul Mundt

Post by Mark Miller

Post by Paul Mundt
Now with that out of the way, this entire series fundamentally fails
to convert half of the perl scripts shipped with the kernel today,
some that are required for build depending on Kconfig options,

[snip]

Post by Mark Miller
From what I can tell, it allows one to fully build the Linux kernel
without Perl.

Wrong, re-read what I said and try again.

I have done so. And I restate, without these patches, Perl is required
*for any install*. Your logic seems to be, if Perl is required for any
part of doing something with the Linux kernel, then there is no reason
it should not be allowed in all parts.

With this logic, soon the Linux kernel will require so many
dependencies, that even a minimal system to rebuild itself under
itself will comprise 200MB for stripped binaries. Right now, it's down
to about 20ish.

With that, you also require more expertise in maintaing these variety
of tools. You have given no logical reason on why we need to add more
complexity to the kernel, when previous tools have managed to deal
with this task for over ten years without requiring Perl.

Post by Paul Mundt

Post by Mark Miller

Post by Paul Mundt
Ignoring the compile-time dependencies that you overlooked, what you
define as "development and debugging" scripts are still an integral
part of the system, unless you are trying to argue that embedded
developers have no interest in things like checkstack due to the
trouble of trying to get perl built.

Do I need that to compile a kernel? No.

Compile-time dependencies you do, yes.

Post by Mark Miller
Perl was not required to build the Linux kernel. Now it is. It adds
another dependency to the Linux kernel. Requiring bash is far less a
dependency that Perl is.

Perhaps so, but that is largely irrelevant. Moving off of the bash
dependency is a lot more trivial than moving off of the perl one. The
kernel bashisms are fairly marginal, and if someone were to do the leg
work for that, it might even be accepted. Getting rid of perl is an
uphill battle for no visible gain. People will continue to write and add
scripts both in bash and perl on a routine basis regardless.

Post by Mark Miller

Post by Paul Mundt
The kernel is and always has been about using the right tool for the
job, not a matter of dictating what tools you must use in order to
accomplish a task you are interested in. Common sense does apply here
though, so this might be a more daunting task for some than others.

How is Perl a better tool for the job than what currently bash and
other standard utilities already offer?

How is bash a better tool for than job than what sed and posix shell
already offer? Yes, we can reduce our dependencies to the bare
minimum,
but that is not constructive for the folks that are actually writing the
scripts in the first place.
Likewise, this is not even a real problem in the embedded developer
demographic, the only place this is a problem is in specially stripped
distributions or people that don't want to go through the pain of cross
compiling perl. None of which is of any concern.

Yet again, you have a habit of writing off a large segment of the
embedded development population that you seem to think doesn't exist.

Post by Paul Mundt
If people are going to write useful things that are reasonably
expected
to be standard on build machines, there is no reason to restrict what
tools they are permitted to use.
If you have a personal vendetta against
something that is fairly standard, that is entirely your own personal
problem, you can choose to deal with it or patch out of tree for your own
crippled environment (assuming patch isn't too much of an obscure
dependency).

Last I checked, patch was a part of busybox. Or is that yet another
realm of embedded development that you are casting off into the depths
of nowhere?

Honestly, it seems you have a personal vendetta against those not
wanting to add another dependency.

--
Mark Miller
***@mirell.org

[Rob Landley]

Post by Paul Mundt

Post by Rob Landley
The perl checkin for 2.6.25 was the camel's nose under the tent flap, and
since then two more instances of perl have shown up in the core kernel
build. This patch series removes the three required to build a basic
kernel for qemu for the targets I've tested (arm, mips, powerpc, sparc,
m68k, sh4, and of course x86 and x86-64), replacing them with shell
scripts.

Misguided rhetoric aside, what does this actually accomplish? If folks
add meaningful tools in to the kernel that require python, and it is
generally regarded as being fairly ubiquitous, I can't imagine there
being any substantiated objections against it.

I think bloat-o-meter is a marvelous tool, and I'm a big fan of python. But I
don't think you shouldn't have to run that to compile a kernel either, largely
because not needing it for the first 17 years or so implied living without
this requirement could be done, sustainably even.

There's a difference between a development workstation and a dedicated build
system. Requiring you to install X11 plus qt on the headless build server
cranking out nightly snapshots in order to run the configure stage of the
kernel build would be silly. But this is not an argument for ripping out
"make xconfig" from the kernel.

Spot the difference?

Post by Paul Mundt
Your main reasons against inclusion of perl seem to be that there is no
realistic expectation for target systems that will be self-hosting will
have perl included, or the inherent complexity in maintaining a coherent
cross compiling environment.

I'm saying it's a major new environmental dependency that went in fairly
recently and largely without comment, and it causes real world headaches for
real people, of which I am only one.

If you don't think environmental dependencies are a problem, I welcome you to
attempt to build open office. (Even the gentoo guys gave up on that one and
just started shipping a prebuilt binary.)

I think large amounts of complexity start with small amounts of complexity
that grow. Complexity is inevitable, but there should be a _reason_ for
increases in it.

Post by Paul Mundt
Both of these things are issues with your
own environment, and in no way are these representative of the embedded
development community at large.
Now with that out of the way, this entire series fundamentally fails to
convert half of the perl scripts shipped with the kernel today, some that
are required for build depending on Kconfig options, and others that are
simply useful tools for self-hosted environments.

I didn't say the job was finished. These are just the ones I've already
personally hit, and thus A) needed to rewrite to build the kernel in my build
environment, B) have a handy test case for.

Post by Paul Mundt
Simply converting a
couple of scripts over you find objectionable is certainly fine if there
is a real benefit in doing so, but this doesn't actually accomplish your
goal of removing the perl dependency.

A) It's a start.

B) It works for me, and builds the .configs I've personally needed so far.

Post by Paul Mundt
Ignoring the compile-time dependencies that you overlooked, what you
define as "development and debugging" scripts are still an integral part
of the system, unless you are trying to argue that embedded developers
have no interest in things like checkstack due to the trouble of trying
to get perl built.

Coming up with new patches and modifying the source is a different use for
source code than going "./configure; make; make install". This is true for
most open source software, I'd expect.

Or are you implying that eclipse or Emacs are such great IDEs that being able
to build outside of a GUI isn't interesting? The ability to build within an
IDE should be allowed to preclude the ability to build without one?

Post by Paul Mundt
Until you can post a series that converts all of scripts/*.pl in its
entirety, you have failed to address the fundamental reason why perl is
used in the first place.

Never start anything unless you can finish it all in one go, eh?

Last I heard the kernel guys tend to frown on people who wander off in their
own corner for a year and then dump a massive rewrite on them. They seem to
prefer the incremental "trail of breadcrumbs" approach. Release early,
release often, incorporate feedback, keep at it.

Or am I wrong?

Post by Paul Mundt
Trying to toss bizarre policy statements around
regarding things you personally find objectionable without any coherent
technical argument to the contrary is of no constructive use whatsoever.

Complexity is a cost, environmental dependencies are a form of complexity, if
the benefit isn't worth the cost (or you can get the benefit without the cost)
then you shouldn't pay the cost.

I was unaware this was a controversial attitude?

Post by Paul Mundt
The kernel is and always has been about using the right tool for the job,
not a matter of dictating what tools you must use in order to accomplish
a task you are interested in.

And yet despite "unifdef" being an existing tool, the kernel developers didn't
require you to install it on your host, but instead built their own version
from source. (They didn't even require you to install a version of cpio on
the host that understands -H newc, but instead built their own initramfs
packer in C.) More recently, they integrated dtc (the device tree compiler).

In each case, they made a balancing decision, and decided that integrating and
maintaining a small C tool was better than having an environmental dependency.

Post by Paul Mundt
Common sense does apply here though, so
this might be a more daunting task for some than others.

Apparently so.

I don't care if checkstack.pl is written in perl for the same reason I don't
care if eclipse is written in java. They're not build dependencies.

If I had to install gdb as a precondition of compiling the Linux kernel I'd
think something was funny with that, which isn't saying I object to the
existence of gdb or the ability to hook it up to kgdb or qemu or UML to poke
at kernels. It's two different categories.

It seems clear to _me_, anyway. But it is coming up on 7am and I haven't
gotten to bed yet, so take that for what it's worth...

Rob

[Wookey]

Post by Paul Mundt
Your main reasons against inclusion of perl seem to be that there is no
realistic expectation for target systems that will be self-hosting will
have perl included, or the inherent complexity in maintaining a coherent
cross compiling environment. Both of these things are issues with your
own environment, and in no way are these representative of the embedded
development community at large.

It may well be true that most embedded people cross-build kernels and
use native perl on a fat build box, but there are plenty of situations
where being able to build kernels without perl is useful. Given the
simplicitly of these patches I can't see any reason not to put them
in, and appreciate Rob's work on this.

And if cross-building perl is really easy, as some in this thread
claim, can someone fix the Debian packages to do it, because that
would be really useful (it appears to be of similar complexity to
cross-building gcc, requiring a 2-stage self-referential build).

Wookey

[Sam Ravnborg]

Hi Wookey.

Post by Wookey
Given the
simplicitly of these patches I can't see any reason not to put them
in

Please do NOT do the mistake and think this the same thing.

Rob's patch simplyfy the timecost stuff - and will be applied on
this merit alone assuming comments will be addressed.

But the serie rased anohter topic: shall we ban use of perl
for generating a kernel.
And this is what primary is discussed and the outcome of
that discussion will not prevent patches that stands on their
own to be applied.

Sam

[Vladimir Dronnikov]

1 - Being able to build the kernel natively on a constrained
target is useful, regardless of whether it is being used for
regression/stress testing or for headers installation or whatever
else.
2 - Cross-compiling perl is hard.
3 - Some oddly constrained target distributions manage to ship
with an entire toolchain yet fail to provide any implementation
of perl.
4 - It wasn't required before.

OK, let's see.

If you have enough space and CPU power and
complete build environment to crunch away at the kernel for stress
testing natively, you can do the same with building perl and negating
point #2.

It seems you meant "If you have enough space ... for stress testing
natively, you _MUST ALSO_ do the same with building perl _TO JUST_
negate point #2". From this point of view your further arguments are
obvious.

#2 is another byproduct of your environment and generally a non-issue.

So you put a constraint on environment to build kernel. Not on a
specific version of shell or gcc. But require SANE environment in
rhetoric sence. In the absence of clear specification of sane
environment it _IS_ an issue. Or simply: "sane" now reads "perlful
enough"?

If there is anything I missed, feel free to add it to the list.

The rationale of changes being proposed is to keep people able to make
their choice on how to build and use their environment. If the code,
which now has to be generated by perl scripts, was shipped with
linux*.tar.bz2, I'd nullify the majority of my objections. Please, DO
NOT require this code to be BUILT, and many would again be happy. You
see, the total question is then reduces to some changes in makefiles
(eliminating FORCE prerequisites).

Otherwise you just force the number of people to invent and maintain
"regress" patches, which is counter-progressive in all sences.

Best regards,
--
Vladimir V. Dronnikov

[Rob Landley]

Since you're turning down an existing patch in favor of a theoretical
patch, I assume you have plans to do this yourself?

If noone else beats me I will do so - yes.

Ok.

And this must be in a single program that can process
all headers in one go so the install process becomes so fast
that we do not worry about if it was done before or not.
Then we can avoid all the .* files in the directory
where we isntall the headers.

What if they run out of disk space halfway through writing a file and
thus it creates a short file (or a 0 length file where the dentry was
created but no blocks could be allocated for the write)?

Then they fail and make will know. Then may leave a file or 100
but it still failed. At next run everything will be done right
assuming the culprint has been fixed.

Ok, so the important thing is propagating failures up to the exit code, then?

When making this patch I hit a problem that the exit code of "unifdef" seems
to depend on whether it found anything to remove within the file it was
processing, so when I changed the caller to actually care about its exit code
it spontaneously aborted.

Fixing that probably does require changing unifdef.c.

I can try to make the shell version more readable, and more powerful.
It's already noticeably faster than the perl version. I have no
objections to making unifdef do all of this, I just haven't got any
interest either.

I have no interest in merging a shell version.

*shrug* Ok. I await your C version, and have a workable patch meeting my own
needs in the meantime.

Thanks,

Rob

[Rob Landley]

Post by Rob Landley
Replace kernel/timeconst.pl with kernel/timeconst.sh. The new shell
script is much simpler, about 1/4 the size, and runs on Red Hat 9 from
2003.
It requires a shell which can do 64 bit math, such as bash, busybox ash,
or dash running on a 64 bit host.

I use Ubuntu (hence dash) on 32 bit systems so I think this needs to
work with dash on 32 bit hosts.

I have a qemu/images directory full of various OS images for testing purposes.

I just fired up my jeos 7.10 image to make sure that even the most stripped-
down version of Ubuntu ("just enough operating system) still installs bash by
default, and it does. (It doesn't install a development toolchain, but it
does install bash.)

I also installed a 32 bit xubuntu 8.10 image (which took 4 hours for some
reason, and which also has bash), and explicitly tested its 32-bit
"/bin/dash", and that did 64-bit math too. So current versions of dash do
offer 64 bit math on 32 bit platforms.

David

Rob

[Jamie Lokier]

In a private email, Bernd Petrovitsch suggested "set -- $i" and then
using NAME=$1; PERIOD=$2. (I keep getting private email responses
to these sort of threads, and then getting dismissed as the only one
who cares about the issue. Less so this time around, but still...)
This apparently works all the way back to the bourne shell.

If you're going "all the way back to the bourne shell", don't use "set
-- $i"; use "set x $i" instead, and don't expect to do any arithmetic
in the shell; use "expr" or "awk" for arithmetic.

(Not relevant to kernel scripts, imho, since you can always assume
something a bit more modern and not too stripped down).

(I have 850 Linux boxes on my network with a bourne shell which
doesn't do $((...)). I won't be building kernels on them though :-)

-- Jamie

[Bernd Petrovitsch]

Post by Jamie Lokier

In a private email, Bernd Petrovitsch suggested "set -- $i" and then
using NAME=$1; PERIOD=$2. (I keep getting private email responses
to these sort of threads, and then getting dismissed as the only one
who cares about the issue. Less so this time around, but still...)
This apparently works all the way back to the bourne shell.

If you're going "all the way back to the bourne shell", don't use "set

"Going back to a Bourne shell" was neither the intention nor makes it
sense IMHO.
I mentioned it to point out that the `set -- ' (or `set x `) is nothing
new or even a bash-ism.

Post by Jamie Lokier
-- $i"; use "set x $i" instead, and don't expect to do any arithmetic
in the shell; use "expr" or "awk" for arithmetic.
(Not relevant to kernel scripts, imho, since you can always assume
something a bit more modern and not too stripped down).

ACK. A bash can IMHO be expected. Even going for `dash` is IMHO somewhat
too extreme.

Post by Jamie Lokier
(I have 850 Linux boxes on my network with a bourne shell which
doesn't do $((...)). I won't be building kernels on them though :-)

Believe it or not, but there are folks out there who build the firmware
on ARM 200 MHz NFS-mounted systems natively (and not simply
cross-compile it on a 2GHz PC .....).

Bernd

[Jamie Lokier]

Post by Bernd Petrovitsch

Post by Jamie Lokier
(I have 850 Linux boxes on my network with a bourne shell which
doesn't do $((...)). I won't be building kernels on them though :-)

Believe it or not, but there are folks out there who build the firmware
on ARM 200 MHz NFS-mounted systems natively (and not simply
cross-compile it on a 2GHz PC .....).

Really?

My 850 Linux boxes are 166MHz ARMs and occasionally NFS-mounted.
Their /bin/sh does not do $((...)), and Bash is not there at all.

If I were installing GCC natively on them, I'd install GNU Make and a
proper shell while I were at it. But I don't know if Bash works
properly without fork()* - or even if GCC does :-)

Perl might be hard, as shared libraries aren't supported by the
toolchain which targets my ARMs* and Perl likes its loadable modules.

I'm not sure why I would want to build a kernel on these devices.

But I see why people with mobile ARM devices like gphones might
want to, when they're out travelling.

-- Jamie

(* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add
proper shared libs. Feel free to fund this :-)

[Bernd Petrovitsch]

Post by Jamie Lokier

Post by Bernd Petrovitsch

Post by Jamie Lokier
(I have 850 Linux boxes on my network with a bourne shell which
doesn't do $((...)). I won't be building kernels on them though :-)

Believe it or not, but there are folks out there who build the firmware
on ARM 200 MHz NFS-mounted systems natively (and not simply
cross-compile it on a 2GHz PC .....).

Really?
My 850 Linux boxes are 166MHz ARMs and occasionally NFS-mounted.
Their /bin/sh does not do $((...)), and Bash is not there at all.

I assume that the NFS-mounted root filesystem is a real distribution.
And on the local flash is a usual busybox based firmware.

Post by Jamie Lokier
If I were installing GCC natively on them, I'd install GNU Make and a
proper shell while I were at it. But I don't know if Bash works

ACK.

Post by Jamie Lokier
properly without fork()* - or even if GCC does :-)
Perl might be hard, as shared libraries aren't supported by the
toolchain which targets my ARMs* and Perl likes its loadable modules.

The simplest way to go is probably to use CentOS or Debian or another
ready binary distribution on ARM (or MIPS or PPC or whatever core the
embedded system has) possibly on a custom build kernel (if necessary).

[...]

Post by Jamie Lokier
(* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add
proper shared libs. Feel free to fund this :-)

The above mentioned ARMs have a MMU. Without MMU, it would be truly
insane IMHO.

Bernd

[Jamie Lokier]

Post by Bernd Petrovitsch
I assume that the NFS-mounted root filesystem is a real distribution.

Not unless you call uClinux (MMU-less) a real distribution, no.

Post by Bernd Petrovitsch

Post by Jamie Lokier
(* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add
proper shared libs. Feel free to fund this :-)

The above mentioned ARMs have a MMU. Without MMU, it would be truly
insane IMHO.

We have similar cross-build issues without MMUs... I.e. that a lot of
useful packages don't cross-build properly (including many which use
Autoconf), and it might be easier to make a native build environment
than to debug and patch all the broken-for-cross-build packages.
Especially as sometimes they build, but fail at run-time in some
conditions.

But you're right it's probably insane to try. I haven't dared as I
suspect GCC and/or Binutils would break too :-)

I'm sticking instead with "oh well cross-build a few packages by hand
and just don't even _try_ to use most of the handy software out there".

You mentioned ARM Debian. According to
http://wiki.debian.org/ArmEabiPort one recommended method of
bootstrapping it is building natively on an emulated ARM, because
cross-building is fragile.

-- Jamie

[Bernd Petrovitsch]

Post by Jamie Lokier

Post by Bernd Petrovitsch
I assume that the NFS-mounted root filesystem is a real distribution.

Not unless you call uClinux (MMU-less) a real distribution, no.

Not really.

Post by Jamie Lokier

Post by Bernd Petrovitsch

Post by Jamie Lokier
(* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add
proper shared libs. Feel free to fund this :-)

The above mentioned ARMs have a MMU. Without MMU, it would be truly
insane IMHO.

We have similar cross-build issues without MMUs... I.e. that a lot of

Of course.

Post by Jamie Lokier
useful packages don't cross-build properly (including many which use
Autoconf), and it might be easier to make a native build environment

Tell me about it - AC_TRY_RUN() is the culprit.
And `pkg-config` supports cross-compilation only since 18 months or so.
Before one had to rewrite the generated .pc files.

[...]

Post by Jamie Lokier
You mentioned ARM Debian. According to
http://wiki.debian.org/ArmEabiPort one recommended method of
bootstrapping it is building natively on an emulated ARM, because
cross-building is fragile.

That's of course the other solution - if qemu supports your
$EMBEDDED_CPU good enough.

Bernd

[Rob Landley]

Post by Jamie Lokier

Post by Bernd Petrovitsch
I assume that the NFS-mounted root filesystem is a real distribution.

Not unless you call uClinux (MMU-less) a real distribution, no.

I want things to be orthogonal. The following should be completely separate
steps:

1) Creating a cross compiler
2) building a native development environment
3) booting a native development environment (on real hardware or under and
emulator)
4) natively building your target system.

You should be able to mix and match. Crosstool for #1, go download "fedora
for arm" instead of #2, qemu or real hardware is your choice for #3, and then
you should be able to natively build gentoo under an ubuntu host or vice
versa. (How is not currently properly documented, but I'm working on that.)

My objection to build systems like buildroot or uClinux is that they bundle
all this together into a big hairball. They create their own cross compiler,
build their own root file system, use their own packaging system, and you have
to take it all or nothing.

My build system is ruthlessly orthogonal. I try not to make it depend on
other bits of _itself_ more than necessary.

Post by Jamie Lokier

Post by Bernd Petrovitsch

Post by Jamie Lokier
(* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add
proper shared libs. Feel free to fund this :-)

The above mentioned ARMs have a MMU. Without MMU, it would be truly
insane IMHO.

We have similar cross-build issues without MMUs... I.e. that a lot of
useful packages don't cross-build properly (including many which use
Autoconf), and it might be easier to make a native build environment
than to debug and patch all the broken-for-cross-build packages.
Especially as sometimes they build, but fail at run-time in some
conditions.

If you can get a version of the same architecture with an mmu you can actually
build natively on that. It's not ideal (it's a bit like trying to build i486
code on an i686; the fact it runs on the host is no guarantee it'll run on the
target), but it's better than cross compiling. And most things have a broad
enough compatible "base architecture" that you can mostly get away with it.

Post by Jamie Lokier
But you're right it's probably insane to try. I haven't dared as I
suspect GCC and/or Binutils would break too :-)

Oh it does, but you can fix it. :)

Post by Jamie Lokier
I'm sticking instead with "oh well cross-build a few packages by hand
and just don't even _try_ to use most of the handy software out there".

Cross compiling doesn't scale, and it bit-rots insanely quickly.

Post by Jamie Lokier
You mentioned ARM Debian. According to
http://wiki.debian.org/ArmEabiPort one recommended method of
bootstrapping it is building natively on an emulated ARM, because
cross-building is fragile.

That's what my firmware linux project does too. (I believe I was one of the
first doing this back in 2006, but there are three or four others out there
doing it now.)

Native compiling under emulation is an idea whose time has come. Emulators on
cheap x86-64 laptops today are about as powerful as high end tricked out build
servers circa 2001, and Moore's Law continues to advance. More memory, more
CPU (maybe via SMP but distcc can take advantage of that today and qemu will
develop threading someday). You can throw engineering time at the problem
(making cross compiling work) or you can throw hardware at the problem (build
natively and buy fast native or emulator-hosting hardware). The balance used
to be in favor of the former; not so much anymore.

That said, my drive for reproducibility and orthogonality says that your
native development environment must be something you can reproduce entirely
from source on an arbitrary host. You can't make cross compiling go away
entirely, the best you can do is limit it to bootstrapping the native
environment. But I want to keep the parts I have to cross compile as small
and simple as possible, and then run a native build script to get a richer
environment. For the past 5+ years my definition has been "an environment
that can rebuild itself under itself is powerful enough, that's all I need to
cross compile", and from the first time I tried this (late 2002) up until
2.6.25 that was 7 packages. That's why I responded to the addition of perl as
a regression, because for my use case it was.

Post by Jamie Lokier
-- Jamie

Rob

[Rob Landley]

Post by Bernd Petrovitsch

Post by Jamie Lokier
My 850 Linux boxes are 166MHz ARMs and occasionally NFS-mounted.
Their /bin/sh does not do $((...)), and Bash is not there at all.

I assume that the NFS-mounted root filesystem is a real distribution.
And on the local flash is a usual busybox based firmware.

Building on an nfs mount is evil. Make cares greatly about timestamp
accuracy, and NFS's dentry cacheing doesn't really, especially when it
discards cached copies and re-fetches them, and the server and client's clocks
are a half-second off from each other.

Sometimes you haven't got a choice, but I hate having to debug the build
problems this intermittently causes. If you never do anything except "make
all" it should suck less.

Post by Bernd Petrovitsch

Post by Jamie Lokier
If I were installing GCC natively on them, I'd install GNU Make and a
proper shell while I were at it. But I don't know if Bash works

ACK.

Post by Jamie Lokier
properly without fork()* - or even if GCC does :-)
Perl might be hard, as shared libraries aren't supported by the
toolchain which targets my ARMs* and Perl likes its loadable modules.

The simplest way to go is probably to use CentOS or Debian or another
ready binary distribution on ARM (or MIPS or PPC or whatever core the
embedded system has) possibly on a custom build kernel (if necessary).

Building natively on target hardware or under QEMU is growing in popularity.
That's how the non-x86 versions of major distros build, and they even have
policy documents about it.

Here's Fedora's:
http://fedoraproject.org/wiki/Architectures/ARM#Native_Compilation

And here are the guys who opened the door for Ubuntu's official Arm port:
http://mojo.handhelds.org/files/HandheldsMojo_ELC2008.pdf

Of course hobbyists like myself haven't got the budget to buy a cluster of
high-end arm systems and they're not always even _available_ for things like
cris, and for new architectures (Xylinx microblaze anyone?) you'll always have
to cross compile to bootstrap the first development environment on 'em anyway,
and it's nice for your environment to be _reproducible_...

So a more flexible approach is to cross compile just enough to get a working
native development environment on the target, and then continue the build
natively (whether it's under qemu or on a sufficiently powerful piece of
target hardware). That's what my "art piece" Firmware Linux project does, and
there's a scratchbox rewrite (sbox2,
http://www.freedesktop.org/wiki/Software/sbox2 ) that does the same sort of
thing, and there are others out there in various states of development. With
x86 hardware so cheap and powerful, building under emulation for less powerful
targets starts to make sense.

Building natively under emulation (QEMU) is available to hobbyists like me and
avoids most of the fun cross compiling issues you don't find out about until
after you've shipped the system and somebody tries to do something with it you
didn't test. So far the record for diagnosing one of these is the two full-
time weeks my friend Garrett spent back at TimeSys tracking down why perl
signal handling wasn't working on mips; turned out it was using x86 signal
numbers rather which don't match the mips ones. The BSP had been shipping for
over a year at that point, but nobody had ever tried to do signal handling in
perl on mips before, and since the perl ./configure step is written in perl
finding the broken part took some doing. This was back in the mists of early
2007 so it's ancient history by now, of course...

If you have set up a cross compiler, you can configure QEMU to use distcc to
call out through its virtual network to the cross compiler running on the
host, which gives you a speed boost without reintroducing most of the horrible
cross compiling issues: there's still only a native toolchain so your build
doesn't have to keep two contexts (hostcc/targetcc) straight, ./configure
still runs natively so any binaries it builds can run and any questions it
asks about the host it's building on should give the right answers for the
target it's building for (including uname -m and friends), headers are
#included natively and libraries are linked natively (that's just how distcc
works, preprocessing and linking happen on the local machine) and there's only
one set so they can't accidentally mix and the cross compiler isn't even
_involved_ in that, make runs natively so it won't get confused by strange
environment variables (yeah, seen that one)...) Only the heavy lifting of
compiling preprocessed .c files to .o files gets exported, which is the one
thing the cross compiler can't really screw up.

But bootstraping a native build environment to run under the emulator is
something you want to keep down to as few packages as possible, because if
you're trying to get the same behavior across half a dozen boards, cross
compiling breaks every time you upgrade _anything_.

Post by Bernd Petrovitsch
[...]

Post by Jamie Lokier
(* - No MMU on some ARMs, but I'm working on ARM FDPIC-ELF to add
proper shared libs. Feel free to fund this :-)

The above mentioned ARMs have a MMU. Without MMU, it would be truly
insane IMHO.

Without an mmu you have a restricted set of packages that run anyway. No
variable length stacks, you have to use vfork() instead of fork() (no copy on
write), memory fragmentation is a big problem so malloc() fails way more
often...

So toolchain problems aren't a "hump" to get past on nommu systems: the area
past that it isn't necessarily any easier.

Post by Bernd Petrovitsch
Bernd

Rob

[Rob Landley]

Post by Bernd Petrovitsch

Post by Jamie Lokier

In a private email, Bernd Petrovitsch suggested "set -- $i" and then
using NAME=$1; PERIOD=$2. (I keep getting private email responses
to these sort of threads, and then getting dismissed as the only one
who cares about the issue. Less so this time around, but still...)
This apparently works all the way back to the bourne shell.

If you're going "all the way back to the bourne shell", don't use "set

"Going back to a Bourne shell" was neither the intention nor makes it
sense IMHO.
I mentioned it to point out that the `set -- ' (or `set x `) is nothing
new or even a bash-ism.

Post by Jamie Lokier
-- $i"; use "set x $i" instead, and don't expect to do any arithmetic
in the shell; use "expr" or "awk" for arithmetic.
(Not relevant to kernel scripts, imho, since you can always assume
something a bit more modern and not too stripped down).

ACK. A bash can IMHO be expected. Even going for `dash` is IMHO somewhat
too extreme.

I have yet to encounter a system that uses dash _without_ bash. (All ubuntu
variants, even jeos, install bash by default. They moved the /bin/sh symlink
but they didn't stop installing bash, and the kernel will preferentially use
bash if it finds it.) People keep telling me they exist. I suppose you could
uninstall bash. You could also uninstall gcc. Not sure what that proves.
(And nobody's shown me this mythical second implementation of perl that all
these perl scripts are supposed to be portable to...)

Busybox ash is a more interesting case, but that implements lots of bash
extensions.

That said, it's easy enough the scripts to work with current versions of dash.
The whole shell portability issue mostly seems to be a stand-in for other
objections (Peter Anvin didn't change syslinux and klibc to require perl to
build this year because of dash), but it's easy enough to just address the
proxy objection and move on rather than arguing about it...

Post by Bernd Petrovitsch

Post by Jamie Lokier
(I have 850 Linux boxes on my network with a bourne shell which
doesn't do $((...)). I won't be building kernels on them though :-)

Believe it or not, but there are folks out there who build the firmware
on ARM 200 MHz NFS-mounted systems natively (and not simply
cross-compile it on a 2GHz PC .....).

Yeah, but according to Changes if they do it with the current kernel they do
it with at least gcc 3.2 (August 2002) and make 3.79.1 (June 2000), so trying
to make it work on software released pre-Y2K probably isn't that high a
priority. :)

Post by Bernd Petrovitsch
Bernd

Rob

[Bernd Petrovitsch]

[...]

Post by Rob Landley

Post by Bernd Petrovitsch
ACK. A bash can IMHO be expected. Even going for `dash` is IMHO somewhat
too extreme.

I have yet to encounter a system that uses dash _without_ bash. (All ubuntu

Hmm, should be doable with a chroot environment quite cheap and simple.

Post by Rob Landley
variants, even jeos, install bash by default. They moved the /bin/sh symlink

Yes, I know (small) embedded systems that have a bash (and not "only"
one of busybox shells). It eases writing somewhat fast shell scripts
without the need for lots of fork()s+exec()s too .....

Bernd

[Mike Frysinger]

2 - Cross-compiling perl is hard.

2 is not hard.

i dont know where you're getting this mythical version of perl, but
anything beyond micro-perl is a pita. and micro-perl is practically
worthless. maybe for kernel/
-mike

[Bernd Petrovitsch]

On Son, 2009-01-04 at 11:23 +0100, Leon Woestenberg wrote:
[...]
[...]

I'm ignoring the "cross-compile perl" issue - haven't tried it for
years.

5. Tool *version* dependency is hard to get right. When cross-building
30 software packages all requiring native perl, we probably need to
build a few versions of perl (native), one for each set of packages.

perl is IMHO special (and quite different to others - including
especially autotools): perl5 is used widely enough so that "one somewhat
recent version" should cover all of 30 software packages.
The hard part are the CPAN modules and their versions which are really a
PITA.
As long as you don't use modules from CPAN, "perl5" should be specific
enough.

Bernd

[Mark A. Miller]

Post by Bernd Petrovitsch
[...]
[...]
I'm ignoring the "cross-compile perl" issue - haven't tried it for
years.

5. Tool *version* dependency is hard to get right. When cross-building
30 software packages all requiring native perl, we probably need to
build a few versions of perl (native), one for each set of packages.

perl is IMHO special (and quite different to others - including
especially autotools): perl5 is used widely enough so that "one somewhat
recent version" should cover all of 30 software packages.
The hard part are the CPAN modules and their versions which are really a
PITA.
As long as you don't use modules from CPAN, "perl5" should be specific
enough.
Bernd
--
Firmix Software GmbH http://www.firmix.at/
mobil: [phone] fax: [phone]
Embedded Linux Development and Services

Actually, something that has amused me during this discussion, is that
right now, the latest stable Perl (5.8.8) does not compile correctly
on a uclibc host, which is typically what you want for embedded
systems, which is why you'd bother to cross compile. (Albeit I was
doing this natively under QEMU with a gcc/uclibc toolchain).

I'll have a patch submitted upstream shortly, but basically the
hints/linux.sh assumes *obviously* you're linking to glibc. I've made
that less libc dependent, looking for either glibc or uclibc.

So without patching Perl, by adding it to the kernel configuration,
it's broken being able to compile the kernel on most embedded
architectures.

And for H. Peter Anvin, before you refer to such uses as compiling the
kernel under a native environment as a "piece of art", please be aware
that the mainstream embedded development environment, buildroot, is
also attempting to utilize QEMU for a "sanity check" on the
environment.

There are several other packages which are broken for embedded
architectures, which I will hopefully attempt to fix by submitting
patches upstream. But this is why we should be cautious about
including new tools for compiling the kernel. Sam Ravnborg was correct
in that a C program to do the work would be the proper way. But by not
addressing a currently existing problem with an adequate replacement
with something that does not exist currently, seems faulty.
--
Mark A. Miller
***@mirell.org

[H. Peter Anvin]

Post by Mark A. Miller
Actually, something that has amused me during this discussion, is that
right now, the latest stable Perl (5.8.8) does not compile correctly
on a uclibc host...

The latest stable Perl is 5.10.0, and the latest of the 5.8 series is 5.8.9.

-hpa

[Mark A. Miller]

Post by H. Peter Anvin

Post by Mark A. Miller
Actually, something that has amused me during this discussion, is that
right now, the latest stable Perl (5.8.8) does not compile correctly
on a uclibc host...

The latest stable Perl is 5.10.0, and the latest of the 5.8 series is 5.8.9.
-hpa
--
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel. I don't speak on their behalf.

My mistake. However,
http://perl5.git.perl.org/perl.git/blob_plain/HEAD:/hints/linux.sh
still has the issue, specifically:

if test -L /lib/libc.so.6; then
libc=`ls -l /lib/libc.so.6 | awk '{print $NF}'`
libc=/lib/$libc
fi

So, my version was incorrect, yet the problem still exists. I've got a
patch, need to submit it, yet Perl *does not compile* on a uclibc
target *as is*.

And this is why we should avoid adding new tools to build the kernel,
because they introduce yet more break points, as such.

Thanks.

[Paul Mundt]

Post by Mark A. Miller
Actually, something that has amused me during this discussion, is that
right now, the latest stable Perl (5.8.8) does not compile correctly
on a uclibc host, which is typically what you want for embedded
systems, which is why you'd bother to cross compile. (Albeit I was
doing this natively under QEMU with a gcc/uclibc toolchain).
I'll have a patch submitted upstream shortly, but basically the
hints/linux.sh assumes *obviously* you're linking to glibc. I've made
that less libc dependent, looking for either glibc or uclibc.

There are plenty that ship with glibc, too. What you "want" for embedded
systems depends entirely on the application for the device, not general
hand-wavy assertions. We (Renesas) ship BSPs on both precisely because of
this reason, and eglibc will probably factor in at some point later on
too.

Post by Mark A. Miller
So without patching Perl, by adding it to the kernel configuration,
it's broken being able to compile the kernel on most embedded
architectures.

This again has nothing to do with the kernel and everything to do with
your distribution. I use perl on uclibc natively just fine, it is
possible there are patches that have not been merged upstream, but this
is again an entirely separate issue.

You seem to be confusing the fact that people who build distributions and
people who use them are one and the same, whereas "most" embedded
developers are going to be using pre-built distributions provided with
their reference hardware, and locking it down during productization. The
fact you are doing a distribution aimed at embedded devices is nice, but
do not try to push off problems you run in to that have a reasonable
expectation of working everywhere else on to the kernel community as
something we ought to care about.

If you need to use a different libc on your platform, yes, you will have
to update packages for this. This used to be true for gcc and other
packages as well, but those were all fixed over time. The fact perl still
stands out despite there being patches available is simply an indicator
that folks working in that area haven't been very proactive in getting
their changes merged upstream. Tough.

This is now entirely off-topic and has nothing to do with the kernel any
more. Please take this to the uclibc or perl lists instead.

[Mark A. Miller]

Post by Paul Mundt

Post by Mark A. Miller
Actually, something that has amused me during this discussion, is that
right now, the latest stable Perl (5.8.8) does not compile correctly
on a uclibc host, which is typically what you want for embedded
systems, which is why you'd bother to cross compile. (Albeit I was
doing this natively under QEMU with a gcc/uclibc toolchain).
I'll have a patch submitted upstream shortly, but basically the
hints/linux.sh assumes *obviously* you're linking to glibc. I've made
that less libc dependent, looking for either glibc or uclibc.

There are plenty that ship with glibc, too. What you "want" for embedded
systems depends entirely on the application for the device, not general
hand-wavy assertions. We (Renesas) ship BSPs on both precisely because of
this reason, and eglibc will probably factor in at some point later on
too.

Post by Mark A. Miller
So without patching Perl, by adding it to the kernel configuration,
it's broken being able to compile the kernel on most embedded
architectures.

This again has nothing to do with the kernel and everything to do with
your distribution. I use perl on uclibc natively just fine, it is
possible there are patches that have not been merged upstream, but this
is again an entirely separate issue.
You seem to be confusing the fact that people who build distributions and
people who use them are one and the same, whereas "most" embedded
developers are going to be using pre-built distributions provided with
their reference hardware, and locking it down during productization. The
fact you are doing a distribution aimed at embedded devices is nice, but
do not try to push off problems you run in to that have a reasonable
expectation of working everywhere else on to the kernel community as
something we ought to care about.
If you need to use a different libc on your platform, yes, you will have
to update packages for this. This used to be true for gcc and other
packages as well, but those were all fixed over time. The fact perl still
stands out despite there being patches available is simply an indicator
that folks working in that area haven't been very proactive in getting
their changes merged upstream. Tough.
This is now entirely off-topic and has nothing to do with the kernel any
more. Please take this to the uclibc or perl lists instead.

Paul:
I initially wrote a rather details response to your e-mail. But

Post by Paul Mundt
I will repeat again that no one has provided a
_single_ reason for why they are unable to provide perl within their
constrained environment. Until that happens, this entire thread is a
joke.

And I did so. And you have disregarded it. That makes me question the
logic of your fervent vehemence against such "Perl is perhaps not a
good idea in the kernel build infrastructure" people like myself.

Thanks.

[Paul Mundt]

Post by Mark A. Miller
I initially wrote a rather details response to your e-mail. But

Post by Paul Mundt
I will repeat again that no one has provided a
_single_ reason for why they are unable to provide perl within their
constrained environment. Until that happens, this entire thread is a
joke.

And I did so. And you have disregarded it. That makes me question the
logic of your fervent vehemence against such "Perl is perhaps not a
good idea in the kernel build infrastructure" people like myself.

You have done no such thing. You have provided an example as to why you
personally find perl objectionable, and in your previous mail you even
noted that you have patches for perl to fix the build issues, so there is
no fundamental reason why you are _unable_ to provide perl in your
environment. It all comes down to the fact you don't want to be bothered
to put the effort in to getting perl setup in your environment, which
quite frankly is no one's problem but your own.

Personally I think perl (and python for that matter) is a terrible
language and I wouldn't use it for anything of consequence, but again,
that is my personal opinion and has nothing to do with regards to the
build system and whether it was the better tool for the job as perceived
by the people that elected to implemented infrastructure using it. I
choose not to use it for my own projects, but I have no qualms with those
that do.

The kernel does not need to provide justification for every change that
goes on as long as there is a reasonable attempt not to break things for
other people. The onus is (and always has been) on you to demonstrate why
perl is an unreasonable dependency to push on embedded developers, and
you have failed utterly at demonstrating this in any sort of coherent
fashion.

I will repeat, there has not been a single coherent argument against what
makes perl inherently incapable of being supported. Every single thing
you have presented as a rebuttal has been your personal preference, which
in this case is simply irrelevant.

[Mark A. Miller]

Post by Paul Mundt
I will repeat, there has not been a single coherent argument against what
makes perl inherently incapable of being supported.

You're right, this thread is worthless with that particular mindset,
Paul. Not, perhaps that the tool in question is brittle, and prone to
potentially break on more architectures than the current make and C
code infrastructure, no, your stance is, unless Perl *cannot* run on
that particular architecture and environment, it has a valid place in
the kernel because it was chosen by certain developers.

And you're right, I did patch around Perl in order to get it to build
under a MIPSEL uclibc environment.

But yes, this particular thread with you *is* worthless, because it's
an argument who's stance is not worth fighting against because of it's
flawed premise.

[Paul Mundt]

Post by Mark A. Miller

Post by Paul Mundt
I will repeat, there has not been a single coherent argument against what
makes perl inherently incapable of being supported.

You're right, this thread is worthless with that particular mindset,
Paul. Not, perhaps that the tool in question is brittle, and prone to
potentially break on more architectures than the current make and C
code infrastructure, no, your stance is, unless Perl *cannot* run on
that particular architecture and environment, it has a valid place in
the kernel because it was chosen by certain developers.

Nonsense. I singled out that point because that was the one you were
replying to in the first place. I itemized the objections in this thread
earlier on and attempted to indicate why they were not applicable in this
context, and asked people to add to it if anything had been overlooked.
If you want to play semantics, do it somewhere else.

If the tool is brittle and constantly breaking, we will see bug reports,
and re-evaluate the support position. This hasn't happened to date in the
context of the kernel build system, so there is no point in even bringing
this up.

Anyways, given that you haven't contributed anything to the kernel and
are therefore perhaps unfamiliar with how things work, I attempted to
show you why the kernel made the decision it did and what it would take
to change that. You have from the beginning only wanted to focus on idle
semantics and refused to re-evaluate your own position on what precisely
it is you find to be problematic in the first place.

So, with that, I am done with this thread, and it seems the key takeaways
from this entire thing has only been a few new lines in my killfile.
It's regrettable you didn't get anything else out of this thread, though
I think both the kernel and embedded linux will survive.

[Rob Landley]

Post by Paul Mundt
Personally I think perl (and python for that matter) is a terrible
language and I wouldn't use it for anything of consequence, but again,
that is my personal opinion and has nothing to do with regards to the
build system and whether it was the better tool for the job as perceived
by the people that elected to implemented infrastructure using it. I
choose not to use it for my own projects, but I have no qualms with those
that do.

Apparently you have qualms with people who chose to reimplement the perl bits
in one of the languages kernel developers already needed to know this time
last year (shell, C, make).

Post by Paul Mundt
The kernel does not need to provide justification for every change that
goes on as long as there is a reasonable attempt not to break things for
other people.

I submitted a change, you insisted that I justify it to your satisfaction.
That pretty much summarizes your participation in this thread.

Post by Paul Mundt
The onus is (and always has been) on you to demonstrate why
perl is an unreasonable dependency to push on embedded developers, and
you have failed utterly at demonstrating this in any sort of coherent
fashion.

Large additional dependencies without benefit are unreasonable. My primary
objection to perl is that it happens to be an additional large dependency
without a correspondingly large benefit.

Your position is not internally consistent. There was no need to scrutinize
it when it went in, but there is a need to scrutinize patches reimplementing
those bits without it. You don't need the word "perl" in that sentence for
your position to be a touch unbalanced.

Post by Paul Mundt
I will repeat, there has not been a single coherent argument against what
makes perl inherently incapable of being supported.

I didn't say it was incapable of being supported. We're _capable_ of
reimplementing the entire kernel in perl except for a microkernel interpreter
running on the bare metal. Or cobol. Sun spent some time trying to do one in
Java a few years back.

"It can be done" and "It's a good idea" are two completely different criteria.

Post by Paul Mundt
Every single thing
you have presented as a rebuttal has been your personal preference, which
in this case is simply irrelevant.

Stop getting so hung up on the word "perl". Did you ever notice the _shipped
files in the kernel so you don't have to have lex or yacc installed? That's
been kernel policy for how many years now? The arguments about "dash vs bash"
when reviewing the shell versions of these scripts are a similar impulse:
trying to reduce unnecessary dependencies.

My first version of the timeconst patch didn't remove the perl script that
generated the file, it simply shipped the pregenerated .h file so it was
possible to _build_ without perl. That was not sufficient for technical
reasons (due to the two architectures that allow you to enter arbitrary
values), so I redid the patch.

Rob

[Alan Cox]

Post by Rob Landley
I didn't say it was incapable of being supported. We're _capable_ of
reimplementing the entire kernel in perl

Which perl. What minor release, what day of the week syntax.

Ask anyone in the distribution business about the joy of perl and you can
listen to the screams for hours.

Perl5 has no formal grammar and you cannot tell what perl of the week
does and perl of last week doesn't do.

That makes it a bad candidate for our toolchain dependencies.


Alan
--
"I don't want world domination if it means I have to deal with more
people like this" - Mike Wangsmo

[Peter Korsgaard]

Mark> And for H. Peter Anvin, before you refer to such uses as compiling the
Mark> kernel under a native environment as a "piece of art", please be aware
Mark> that the mainstream embedded development environment, buildroot, is
Mark> also attempting to utilize QEMU for a "sanity check" on the
Mark> environment.

That's for verifying that the rootfs'es actually work, not for
building stuff.

[Rob Landley]

Post by Peter Korsgaard
Mark> And for H. Peter Anvin, before you refer to such uses as compiling
the Mark> kernel under a native environment as a "piece of art", please be
aware Mark> that the mainstream embedded development environment,
buildroot, is Mark> also attempting to utilize QEMU for a "sanity check" on
the Mark> environment.
That's for verifying that the rootfs'es actually work, not for
building stuff.

Not in my case.

Rob

[Sam Ravnborg]

Post by Mark A. Miller
There are several other packages which are broken for embedded
architectures, which I will hopefully attempt to fix by submitting patches
upstream. But this is why we should be cautious about including new tools
for compiling the kernel. Sam Ravnborg was correct in that a C program to do
the work would be the proper way. But by not addressing a currently existing
problem with an adequate replacement with something that does not exist
currently, seems faulty.

Why are "make headers_install" such a crucial thing for your
embedded environmnet?

I would assume that if this of such improtance then there is also
someone to step up and contribute a C version of it.

Sam

[Mark A. Miller]

Post by Sam Ravnborg

Post by Mark A. Miller
There are several other packages which are broken for embedded
architectures, which I will hopefully attempt to fix by submitting patches
upstream. But this is why we should be cautious about including new tools
for compiling the kernel. Sam Ravnborg was correct in that a C program to do
the work would be the proper way. But by not addressing a currently existing
problem with an adequate replacement with something that does not exist
currently, seems faulty.

Why are "make headers_install" such a crucial thing for your
embedded environmnet?

Sanity check. If the environment cannot replicate itself, then
something has been faulty in the cross-compiling stage, that was used
to propagate a native environment for the target architecture.

Post by Sam Ravnborg
I would assume that if this of such improtance then there is also
someone to step up and contribute a C version of it.

You've already dismissed a shell version to correct the issue, in
hopes of a "possible" C version. It would be nice, I'm not capable of
doing it personally, but a solution already exists to "unbreak" the
kernel build. If you're unwilling to merge the current patches, then
feel free to claim that this doesn't break anything on current
architectures, but it's incorrect, due to Perl not even compiling as
is currently on a native uclibc environment.

I look forward to what other tools will be introduced to break yet
more architectures until the kernel cannot be compiled unless on an
i686+glibc architecture.

Post by Sam Ravnborg
Sam

[Sam Ravnborg]

Post by Mark A. Miller

Post by Sam Ravnborg

Post by Mark A. Miller
There are several other packages which are broken for embedded
architectures, which I will hopefully attempt to fix by submitting patches
upstream. But this is why we should be cautious about including new tools
for compiling the kernel. Sam Ravnborg was correct in that a C program to do
the work would be the proper way. But by not addressing a currently existing
problem with an adequate replacement with something that does not exist
currently, seems faulty.

Why are "make headers_install" such a crucial thing for your
embedded environmnet?

Sanity check. If the environment cannot replicate itself, then
something has been faulty in the cross-compiling stage, that was used
to propagate a native environment for the target architecture.

So you actually build your target toolchain on your target?

Sam

[Mark A. Miller]

Post by Sam Ravnborg

Post by Mark A. Miller

Post by Sam Ravnborg

Post by Mark A. Miller
There are several other packages which are broken for embedded
architectures, which I will hopefully attempt to fix by submitting patches
upstream. But this is why we should be cautious about including new tools
for compiling the kernel. Sam Ravnborg was correct in that a C program to do
the work would be the proper way. But by not addressing a currently existing
problem with an adequate replacement with something that does not exist
currently, seems faulty.

Why are "make headers_install" such a crucial thing for your
embedded environmnet?

Sanity check. If the environment cannot replicate itself, then
something has been faulty in the cross-compiling stage, that was used
to propagate a native environment for the target architecture.

So you actually build your target toolchain on your target?
Sam

Correct, albeit under emulation, such as QEMU. Obviously the target
architecture, such as an embedded MIPSEL device with only 8MB of flash
and 64MB of RAM, is not going to (particularly well) re-compile its
entire environment, QEMU allows it nicely with distcc at a reasonable
speed. (Albeit there is no distconfigure, but that's entirely an
unrelated tanget of muck and despair and rants against configure, but
we're not going there...)

[Paul Mundt]

Post by Sam Ravnborg

Post by Mark A. Miller

Post by Sam Ravnborg

Post by Mark A. Miller
There are several other packages which are broken for embedded
architectures, which I will hopefully attempt to fix by submitting patches
upstream. But this is why we should be cautious about including new tools
for compiling the kernel. Sam Ravnborg was correct in that a C program to do
the work would be the proper way. But by not addressing a currently existing
problem with an adequate replacement with something that does not exist
currently, seems faulty.

Why are "make headers_install" such a crucial thing for your
embedded environmnet?

Sanity check. If the environment cannot replicate itself, then
something has been faulty in the cross-compiling stage, that was used
to propagate a native environment for the target architecture.

So you actually build your target toolchain on your target?

This happens in a lot of places, like embedded gentoo ports, where almost
all of the work is sent across distcc to a cross-compilation machine. In
systems that use package management, it is done on the host through
emulation, or painfully cross-compiled.

[Jamie Lokier]

Post by Paul Mundt
This happens in a lot of places, like embedded gentoo ports, where almost
all of the work is sent across distcc to a cross-compilation machine. In
systems that use package management, it is done on the host through
emulation, or painfully cross-compiled.

Ah yes, I remember using embedded Gentoo.

95% of the time in ./configure scripts, 5% in compilations.

And this is on x86! I dread to think how slow it gets on something
slow.

-- Jamie

[Rob Landley]

Post by Jamie Lokier

This happens in a lot of places, like embedded gentoo ports, where =

almost

Post by Jamie Lokier

all of the work is sent across distcc to a cross-compilation machin=

e. In

Post by Jamie Lokier

systems that use package management, it is done on the host through
emulation, or painfully cross-compiled.

Ah yes, I remember using embedded Gentoo.
95% of the time in ./configure scripts, 5% in compilations.

With SMP becoming commonplace, expect this to become the norm everywher=
e. =20
Once you get to around quad processor, any C program with a ./configure=
step=20
is probably going to take longer to configure than to compile. (Of cou=
rse C++=20
manages to remain slow enough that autoconf isn't so obvious a bottlene=
ck.)

Post by Jamie Lokier
And this is on x86! =A0I dread to think how slow it gets on something
slow.

My friend Mark's been experimenting with the amazon "cloud" thing, feed=
ing in=20
an image with a qemu instance and distcc+cross-compiler, and running bu=
ilds=20
under that. Renting an 8-way ~2.5 ghz server with 7 gigabytes of ram a=
nd 1.6=20
terabytes of disk is 80 cents/hour through them plus another few cents/=
day for=20
bandwidth and persistent storage and such. That's likely to get cheape=
r as=20
time goes on.

We're still planning to buy a build server of our own to have something=
in-
house, but for running nightly builds it's almost to the point where=20
depreciation on the hardware is more than buying time from a server far=
m. =20
Just _one_ of those 8-way servers is enough hardware to build an entire=
distro=20
in an hour or so.

What this really allows us to do is experiment with "how parallel can w=
e get=20
our build"? Because renting ten 8-way servers in a cluster is $8/hour,=
and=20
distcc already scales trivially over that. Down the road what Firmware=
Linux=20
is working towards is multiple qemu instances running in parallel with =
a=20
central instance distributing builds to each one, so each can do its ow=
n=20
=2E/configure in parallel, distribute compilation to the distccd instan=
ces as it=20
has stuff to compile, and then package up the resulting binary into one=
of=20
those portage tarballs and send it back to the central node to install =
on a=20
network mount that the lot of 'em can mount as build context, so the pa=
ckages=20
can get their dependencies right. (You don't want your build taking pl=
ace in=20
a network mount, but your OS being on one you never write to isn't so b=
ad as=20
long as you have local storage to build in.)

We'll probably leverage the heck out of Portage for this, and might win=
d up=20
modifying it heavily. Dunno yet. (We can even force dependencies on p=
ortage=20
so it doesn't need to calculate 'em, the central node can do that and t=
hen say=20
"you have these packages, _build_"...)

But yeah, hobbyists with a laptop, network access, and a monthly budget=
of $20=20
can do cluster builds these days.

Rob

P.S. I still hope autoconf dies off and the world wakes up and moves a=
way=20
from that. And from makefiles for that matter. But in the meantime, I=
can=20
work around it with enough effort.

[V***@vt.edu]

P.S. I still hope autoconf dies off and the world wakes up and moves away
from that. And from makefiles for that matter. But in the meantime, I can
work around it with enough effort.

What do you propose autoconf and makefiles get replaced by?

% wc pidgin/configure*
34287 118303 1004074 pidgin/configure
2499 7684 81532 pidgin/configure.ac

Which you rather code, 2.5K lines of autoconf or 35K lines of configure
script? As long as there's enough diversity to require configure scripts,
there's going to be a demand for an autoconf-ish feature to ease writing them.

[Rob Landley]

Post by V***@vt.edu

Post by Rob Landley
P.S. I still hope autoconf dies off and the world wakes up and moves
away from that. And from makefiles for that matter. But in the
meantime, I can work around it with enough effort.

What do you propose autoconf and makefiles get replaced by?

At a first guess, meaningful standards and an acknowledgement that Linux is a
platform in its own right? Between the two of them, that's 90% of your
problem right there.

Not a new issue, here's a link from 2002:
http://news.cnet.com/2100-1001-950180.html

And here's a multi-vendor effort at a standards group (the 86open project to
definite a Unix binary standard for x86 processors) dissolving itself way back
in 1999 with a declaration that the Linux binary format already was an open
standard and other unixes should just use things like lxrun to support that:
http://www.telly.org/86open/

Also, it would be nice if configure steps could stop entangling 1) users
providing preferences (command line options like --prefix or most of the --
enable and --disable flags) for which things like kconfig might make more
sense, 2) probe for installed packages like zlib and enabling optional support
if found, 3) questions like "does my build environment provide strlcpy()".
Turning that into a big hairball does not help keep things simple.

Post by V***@vt.edu
% wc pidgin/configure*
34287 118303 1004074 pidgin/configure
2499 7684 81532 pidgin/configure.ac
Which you rather code, 2.5K lines of autoconf or 35K lines of configure
script?

I consider it a false dichotomy. I prefer "neither", and have seen it done
successfully many times.

I've never built pidgin from source, but I've got the output of the binutils
build in a log file. How many of these tests are actually necessary on any
Linux system:

checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking for suffix of object files... o
checking whether gcc accepts -g... yes
checking for library containing strerror... none required
checking how to run the C preprocessor... gcc -E
checking whether build environment is sane... yes
checking whether gcc and cc understand -c and -o together... yes
checking for an ANSI C-conforming const... yes
checking for inline... inline

It just goes on and on and on like this. Tests like "checking whether byte
ordering is bigendian... no" means "Either I didn't know endian.h existed, or
I don't trust it to be there". How about the long stretches checking for the
existence of header files specified by posix? Or this gem:

checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk

(If you can use awk, when is it _not_ there? Probable answer: there was some
broken version on irix back in 1992 with zero remaining seats today, and they
never went back to clean anything out of the makefile because "simplifying the
build" and "autoconf" do not go together, and because you can't sanely
regression test against build environments nobody has anymore.)

This argument is a can of worms, and the linux-kernel list probably isn't the
best place for it. However, "install mingw on windows instead of trying to
get your thing to build under Visual Studio" is a decent support strategy.
Tinycc and Intel's icc both implemented gcc extensions to build the kernel,
and the kernel's been removing such extensions where c99 versions are
available since then. Things like uClibc implement standards and copy glibc
extensions that are actually used.

In the era of open source, it's now a viable strategy to specify, document,
and require a standardized build environment. The way to make that build
environment portable is to keep it simple and standardized, not to create a
tower of #ifdefs testing the output of a huge environment probing shell
script.

Rob

[Jamie Lokier]

Post by Rob Landley

Post by V***@vt.edu

Post by Rob Landley
P.S. I still hope autoconf dies off and the world wakes up and moves
away from that. And from makefiles for that matter. But in the
meantime, I can work around it with enough effort.

What do you propose autoconf and makefiles get replaced by?

I've never built pidgin from source, but I've got the output of the binutils
build in a log file.

None, but then it's not a Linux-only program that you're compiling.
(Nor is it Linux-in-2009-only).

If you _know_ you're running on Linux from a particular era, you can
provide a config.cache file with the correct answers already filled in.

I agree that Autoconf sucks (I've written enough sucking Autoconf
macros myself, I hate it), but the tough part is providing a suitable
replacement when you still want portable source code.

Post by Rob Landley
It just goes on and on and on like this. Tests like "checking
whether byte ordering is bigendian... no" means "Either I didn't
know endian.h existed, or I don't trust it to be there". How about
the long stretches checking for the existence of header files
specified by posix?

You seem to be arguing for "let's make all our programs Linux-specific
(and Glibc-specific in many cases)". Given all the problems you've
seen with cross-compiling, let alone compiling for different OS
platforms, that seems a little odd.

-- Jamie

[Rob Landley]

Post by Jamie Lokier

Post by Rob Landley

Post by V***@vt.edu

Post by Rob Landley
P.S. I still hope autoconf dies off and the world wakes up and moves
away from that. And from makefiles for that matter. But in the
meantime, I can work around it with enough effort.

What do you propose autoconf and makefiles get replaced by?

I've never built pidgin from source, but I've got the output of the
binutils build in a log file.

None, but then it's not a Linux-only program that you're compiling.
(Nor is it Linux-in-2009-only).

Yeah, I noticed. It's not quite as bad as OpenSSL (where Linux support is
intentionally an afterthought), but things like "Libtool" are supposed to be a
NOP on ELF Linux and yet regularly screw up builds. (It's supposed to do
nothing, and can't manage to do it correctly. That's sad.)

Post by Jamie Lokier
If you _know_ you're running on Linux from a particular era, you can
provide a config.cache file with the correct answers already filled in.

And yet very few projects actually do.

As for "from a particular era", just for fun I fired up the Red Hat 9 qemu
image I keep around for this sort of thing, downloaded glibc 2.7 (the most
recent one they bothered to cut a tarball for on ftp.gnu.org and one of the
big autoconf offenders), and ran its ./configure. It died with:

configure: error:
*** These critical programs are missing or too old: gcc
*** Check the INSTALL file for required versions.

So you can't build a 2 year old version of glibc under a 6 year old version of
Linux (which was the most popular Linux version in the world when it shipped,
with about 50% market share among Linux seats). And yet glibc (one of the
FSF's flagship projects) bothers doing extensive autoconf probes. Why?
Autoconf isn't really _helping_ here...

The bottom line is that if your assumption is that you have an open source
application targeting an open source operating system, lots of the hoops you
used to have to jump through just aren't very interesting anymore.

Post by Jamie Lokier
I agree that Autoconf sucks (I've written enough sucking Autoconf
macros myself, I hate it), but the tough part is providing a suitable
replacement when you still want portable source code.

Depends on your definition of "portable". The unix wars of the 80's are over;
they're pretty much all dead. Even the surviving legacy deployments of
Solaris and AIX provide Linux emulation environments. And of course FreeBSD's
done so for years:
http://www.onlamp.com/pub/a/bsd/2006/01/12/Big_Scary_Daemons.html

MacOS X and windows are still very much alive, but if you want to target those
you can either A) treat them as Posix/SUSv3 (which both _claim_ to support),
B) use cross platform libraries like SDL and opengl and program to their APIs,
C) bother to do a proper port of the thing ala
http://porting.openoffice.org/mac/ or http://www.kju-app.org/ or the way khtml
wound up in Safari.

For Windows there's Cygwin, running windows programs on Linux has Wine. Or
just qemu/kvm in either direction.

Basically, pick a standard to write to. If you want to write to posix and
SUSv4, do it. If you want to add in LSB and the Linux man pages, go for it.
But autoconf was designed for portability between Irix and HP-UX, which just
doesn't come up much anymore.

Post by Jamie Lokier

Post by Rob Landley
It just goes on and on and on like this. Tests like "checking
whether byte ordering is bigendian... no" means "Either I didn't
know endian.h existed, or I don't trust it to be there". How about
the long stretches checking for the existence of header files
specified by posix?

You seem to be arguing for "let's make all our programs Linux-specific
(and Glibc-specific in many cases)".

Checking for the existence of posix header files is Linux-specific?

I'm saying there are many standards, and you can choose to adhere to standards
like LP64 (which MacOSX, Linux, and the BSDs all support) and _say_ you do so
and achieve portability that way.

You also have "#if defined __linux__" and "#if defined __APPLE__" and so on,
so header files can do a lot of the tests that people wind up doing with
autoconf for some reason.

And there will always be platforms you're NOT portable to. (Game consoles
come to mind: your average autoconf recipe isn't going to make your program
run on a PS3, XBox 360, or Wii. Unless you load Linux on those systems first
and program for Linux.)

Post by Jamie Lokier
Given all the problems you've
seen with cross-compiling, let alone compiling for different OS
platforms, that seems a little odd.

If I can't get Linux running on the hardware (which is seldom an interesting
case anymore; it's on everything from cell phones to the S390), and I can't
get a Linux emulation environment like Solaris' lxrun, aix5L, or cygwin, then
I probably want a rigidly posix environment. (Heck, even wince has celib and
gnuwince, not that I really care.)

The extra "portability" isn't even going to buy you 5% more seats these days.
It's really not that interesting. It's not that the world is homogenous now,
it's that between open source and open standards we don't need giant if/else
ladders with hundreds of tests to cover the interesting cases. And trying to
achieve portability by relying on standards is a superior approach to trying
to achieve portability via an infinite series of special case checks.

Post by Jamie Lokier
-- Jamie

Rob

[Pádraig Brady]

Implementing something by hand isn't _always_ a good alternative, sur=

e. That=20

would be the "thinking about the problem" part. In this instance, av=

oiding=20

overflow is trivial. (If 1<<-1 didn't wrap around, it wouldn't even =

need the=20

if statement.)

I don't think this affects your script but it's worth noting
that both bash and ksh use arithmetic rather than logical shift
for the >> operator.

Now arithmetic shift is not useful on 2's compliment machines,
and moreover it's compiler dependent as to whether arithmetic
or logical shift is done for >>. Therefore to increase usefulness
and decrease ambiguity, shells really should only shift unsigned
variables internally.

I know the opengroup spec says to use signed ints, but I think
that is intended to disambiguate input and output, rather than defining
internal operations. This is correct I think since the POSIX spec says
you can even use floating point internally if you like.

I asked the bash maintainer who said he would need clarification
from the austin group (CC'd) before changing anything.

cheers,
P=C3=A1draig.