Discussion:
--create-testdir eats CPU and does nothing
Tim Rühsen
2017-08-05 12:15:32 UTC
Permalink
Hi,

with a fresh git clone, this command doesn't come back:

$ ./gnulib-tool --create-testdir --dir=../testdir1
gnulib-tool: warning: module euidaccess depends on a module with an
incompatible license: group-member

Looks like it eats CPU of one core.

./testdir1 has been created but still empty after 15 minutes.

What did I miss ?

Regards, Tim
Tim Rühsen
2017-08-05 12:26:53 UTC
Permalink
Post by Tim Rühsen
Hi,
$ ./gnulib-tool --create-testdir --dir=../testdir1
gnulib-tool: warning: module euidaccess depends on a module with an
incompatible license: group-member
Looks like it eats CPU of one core.
./testdir1 has been created but still empty after 15 minutes.
What did I miss ?
Correction, it came back after ~20 minutes.

This is a 8 Core, 16 Threads, 16GB, m.2 SSD (3Gb/s) machine.
Is there any chance to parallelize the work of ./gnulib-tool ? The machine was
just 6-7% idle.

Regards, Tim
Paul Eggert
2017-08-05 15:46:22 UTC
Permalink
Post by Tim Rühsen
Is there any chance to parallelize the work of ./gnulib-tool ? The machine was
just 6-7% idle.
There was a GNU project to improve gnulib-tool's performance a while ago. I
don't know what came of it. That, plus 'configure', is a lot of why single-core
performance still matters when building GNU software.
Tim Rühsen
2017-08-05 16:56:03 UTC
Permalink
Post by Paul Eggert
Post by Tim Rühsen
Is there any chance to parallelize the work of ./gnulib-tool ? The machine
was just 6-7% idle.
There was a GNU project to improve gnulib-tool's performance a while ago. I
don't know what came of it. That, plus 'configure', is a lot of why
single-core performance still matters when building GNU software.
There was a thread in gnu-prog-discuss with some ideas how to improve the
speed of configure runs. But I guess, nobody works on it consequently.
I put some work at it: https://gitlab.com/gnuwget/wget2/wikis/Developer-hints:-Increasing-speed-of-GNU-toolchain

Regards, Tim
Bruno Haible
2017-08-05 16:34:58 UTC
Permalink
Post by Tim Rühsen
Post by Tim Rühsen
$ ./gnulib-tool --create-testdir --dir=../testdir1
gnulib-tool: warning: module euidaccess depends on a module with an
incompatible license: group-member
Correction, it came back after ~20 minutes.
You have a fast machine.
Post by Tim Rühsen
Is there any chance to parallelize the work of ./gnulib-tool ? The machine was
just 6-7% idle.
Not parallelize. But rewrite in a decent programming language. IMO the way to go
would be to merge Dmitry Selyutin's rewrite
https://github.com/ghostmansd/gnulib-python
into gnulib.

Bruno
Tim Rühsen
2017-08-05 17:01:51 UTC
Permalink
Post by Bruno Haible
Post by Tim Rühsen
Post by Tim Rühsen
$ ./gnulib-tool --create-testdir --dir=../testdir1
gnulib-tool: warning: module euidaccess depends on a module with an
incompatible license: group-member
Correction, it came back after ~20 minutes.
You have a fast machine.
Post by Tim Rühsen
Is there any chance to parallelize the work of ./gnulib-tool ? The machine
was just 6-7% idle.
Not parallelize. But rewrite in a decent programming language. IMO the way
to go would be to merge Dmitry Selyutin's rewrite
https://github.com/ghostmansd/gnulib-python
into gnulib.
But I guess there is a good reason for not doing it !? It dates 2012.

check-copyright seems to be a good candidate for parallelizing. I tested that
with the 'parallel' tool (with some modifications of course).

A speedup only when certain tools / shell are/is available would be fine, IMO.

But that's just a random idea, I am absolutely not enough into gnulib nor
shell programming.

Regards, Tim
Dmitry Selyutin
2017-08-06 04:09:23 UTC
Permalink
Hi everyone!

I'm amazed that someone still remembers this project. Looking at its
sources, I see that there is room for improvement, but still the overall
project structure may still be integrated into gnulib. I would be glad to
continue working on the project in my free time, though my Python skills
are a bit rusty. As for parallelization, I think it may be a good idea to
use Python 3 since it now has some form of Go-like coroutines IIRC.

Do you think it's a good idea to revive the old project? It seems that it
may still be useful.
Bruno Haible
2017-08-06 09:11:42 UTC
Permalink
Hi Dmitry, all,

First of all, my deepest apologies for not having pulled your gnulib-tool
rewrite into gnulib once you finished it. The reason was simply that the
project went from April to September 2012, and I stopped having time for GNU
in July 2012. I only "came back" recently.

If there's one thing I learned in that time, it is that I get nowhere if I
don't set priorities (which projects to work on) for myself. My priorities
are set for the next year or so, and they include only _occasional_
contributions to gnulib. Therefore I can not do this integration myself,
but I can help out with guidance.

In any case, I'm in favour of integrating it into gnulib.
Post by Dmitry Selyutin
I'm amazed that someone still remembers this project. Looking at its
sources, I see that there is room for improvement, but still the overall
project structure may still be integrated into gnulib. I would be glad to
continue working on the project in my free time,
This would be ideal!
Post by Dmitry Selyutin
though my Python skills are a bit rusty.
The good thing about Python is that even with superficial knowledge of the
language, you can read and maintain code.
Post by Dmitry Selyutin
As for parallelization, I think it may be a good idea to
use Python 3 since it now has some form of Go-like coroutines IIRC.
Huh? I thought I insisted that we use Python 3 from the beginning. Anyway,
I think no one will be asking for parallelization once they use
gnulib-tool.py.
Post by Dmitry Selyutin
Do you think it's a good idea to revive the old project? It seems that it
may still be useful.
Yes, sure. I think the biggest lack, currently, is that there is no test
suite that would guarantee behavioural compatibility between the shell script
and the python program. But this is not a blocker; it can be mitigated by
having the gnulib users test one versus the other. And once enough users are
satisfied with the python program's results, it can be declared the "official"
one.

So, if you (or someone else) volunteers to work on this integration, I would
see the sequence of actions as:
1. Merge https://github.com/ghostmansd/gnulib-python into gnulib.
2. There were 52 modifications to gnulib-tool since April 2012. Do the
same modifications to gnulib-tool.py, for consistency. (Most of these
changes were quite small. The biggest one is the one from 2015-11-21.)
3. Document how to use gnulib-tool.py, but mark it experimental.
4. For several months, answer support requests from users.
5. Once a sufficient number of users have used it and are satisfied with it,
document it as recommended and make it the default.

Bruno
Tim Rühsen
2017-08-12 21:04:53 UTC
Permalink
Post by Dmitry Selyutin
Hi everyone!
I'm amazed that someone still remembers this project. Looking at its
sources, I see that there is room for improvement, but still the overall
project structure may still be integrated into gnulib. I would be glad to
continue working on the project in my free time, though my Python skills
are a bit rusty. As for parallelization, I think it may be a good idea to
use Python 3 since it now has some form of Go-like coroutines IIRC.
Do you think it's a good idea to revive the old project? It seems that it
may still be useful.
It sounds like you have been pretty far with that project.

I would love to see it working with the current gnulib.
Please let me know when it is ready to be tested.

Regards, Tim

Loading...