[john-dev] Linux x86

Hi,

Post by a***@lx42.de
I am new to this mailinglist, because I compiled jtr on a few
platforms and ran into various problems.

Your message would be more appropriate for the john-users list than for
john-dev. You're reporting a problem rather than intend to participate
in JtR development (as far as I can tell from your message).

Post by a***@lx42.de
make clean && make

[...]

Post by a***@lx42.de
make[1]: Entering directory
`/nfs__share/ae/cc/189/JohnTheRipper/src/escrypt'
/bin/rm -f tests crypto_scrypt-best.o crypto_scrypt-common.o sha256.o
tests.o crypto_scrypt-*.o
make[1]: Leaving directory `/nfs__share/ae/cc/189/JohnTheRipper/src/escrypt'
make find_version
make[1]: Entering directory `/nfs__share/ae/cc/189/JohnTheRipper/src'
/bin/mkdir -p ../run/kernels
cp ../run/kernels ../run/kernels
cp: omitting directory `../run/kernels'
make[1]: *** [opencl/7z_kernel.cl] Error 1

I can't reproduce the problem. For me, this looks as follows:

make[1]: Entering directory `/home/solar/j/bleeding-jumbo-20171026/src/escrypt'
/bin/rm -f tests crypto_scrypt-best.o crypto_scrypt-common.o sha256.o tests.o crypto_scrypt-*.o
make[1]: Leaving directory `/home/solar/j/bleeding-jumbo-20171026/src/escrypt'
make find_version
make[1]: Entering directory `/home/solar/j/bleeding-jumbo-20171026/src'
/bin/mkdir -p ../run/kernels
cp opencl_device_info.h ../run/kernels
cp opencl_misc.h ../run/kernels

... and so on.

In your case, the "cp ../run/kernels ../run/kernels" command is clearly
wrong, but it is unclear why it became so wrong.

magnum, on a related note, maybe we should have configure suggest "-j"
in the second make, e.g. replace:

Configure finished. Now 'make clean && make -s' to compile.

with:

Configure finished. Now 'make clean && make -sj8' to compile.

given that almost everyone is on a multi-core machine these days.
Ideally, we'd determine and include the logical CPU count in there, but
even a generally reasonable constant like 8 would do.

Alexander

Solar Designer

2017-10-26 13:15:03 UTC

Post by a***@lx42.de
/nfs__share/ae/cc/189/JohnTheRipper/src>CFLAGS="$CFLAGS $cflags
-I$asx_prefix/include" LDFLAGS="$LDFLAGS $ldflags -L$asx_prefix/lib"
OPENSSL_LIBS="$asx_pref
ix/lib/libssl.a $asx_prefix/lib/libcrypto.a"
OPENSSL_CFLAGS="-I$asx_prefix/include -L$asx_prefix/lib" ./configure
--prefix="$asx_prefix"
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu

So this looks like a custom or at least unusual Linux distro (or at
least build of gcc), and you use custom build options.

Post by a***@lx42.de
configure: Testing build host's native CPU features
checking for Hyperthreading... yes
checking for MMX... yes
checking for SSE2... yes
checking for SSSE3... no

It is rather unusual to have a CPU supporting SSE2 yet not SSSE3 these
days. I'd guess this non-detection of SSSE3 is a side-effect of the
overridden CFLAGS, but I can't reproduce it. So are you actually on a
currently rare early x86-64 CPU from prior to 2006 or so? What is it?

Post by a***@lx42.de
configure: WARNING: No recognized optimization option present in CFLAGS

So you'd have a highly unoptimal build, even if the build worked,
because of you having overridden the CFLAGS.

None of this is directly related to the build failure, but it indicates
that your build is highly unusual in general.

As to the build failure, what's your version of "make" - e.g., what does
"make -v" say?

Post by a***@lx42.de
OpenCL support ............................. no

magnum, when we build without OpenCL support, can we possibly have it
skip copying of the OpenCL kernels? (The copying should have succeeded
anyway, but let's use this as a reminder of this possible improvement.)

Alexander

a***@lx42.de

2017-10-26 13:39:52 UTC

So this looks like a custom or at least unusual Linux distro (or at
least build of gcc), and you use custom build options.

The flags are for openssl only. It is openssl-1.0.2k in this case.

Post by a***@lx42.de
configure: Testing build host's native CPU features
checking for Hyperthreading... yes
checking for MMX... yes
checking for SSE2... yes
checking for SSSE3... no

The OS is CentOS 4.1, the CPU has the ability, but the kernel is
unable to do ssse3:

model name : Intel(R) Xeon(R) CPU E5-4620 v2 @ 2.60GHz
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm
pni cx16 ts

Post by a***@lx42.de
configure: WARNING: No recognized optimization option present in CFLAGS

So you'd have a highly unoptimal build, even if the build worked,
because of you having overridden the CFLAGS.

I like to append the options, how do I achieve this?

Post by Solar Designer
None of this is directly related to the build failure, but it indicates
that your build is highly unusual in general.

The machines are rather old to ensure compatiblity for newer machines.
So upgrading is not an option.

Post by Solar Designer
As to the build failure, what's your version of "make" - e.g., what does
"make -v" say?

GNU Make 3.80

Best Regards,
Alexander

Solar Designer

2017-10-26 19:15:32 UTC

So this looks like a custom or at least unusual Linux distro (or at
least build of gcc), and you use custom build options.

The flags are for openssl only. It is openssl-1.0.2k in this case.

What's in the $CFLAGS $cflags and $LDFLAGS $ldflags environment
variables before you run the configure command?

Post by a***@lx42.de
configure: Testing build host's native CPU features
checking for Hyperthreading... yes
checking for MMX... yes
checking for SSE2... yes
checking for SSSE3... no

The OS is CentOS 4.1, the CPU has the ability, but the kernel is
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx lm
pni cx16 ts

I understand that a kernel this old wouldn't detect SSSE3, but that
doesn't mean JtR's configure also wouldn't, because SSSE3 doesn't
require any special support from the kernel to be usable in userspace.
(AVX does. So you should be able to use up to SSE4.1 with this old
kernel, unfortunately not using this CPU's AVX support.)

I think the reason for non-detection of SSSE3+ here is old gcc: our
configure script test for SSSE3 tries to compile a tiny program that
uses an SSSE3 intrinsic.

So this is sort of "fine". In fact, it might have saved you from worse
problems if the code compiled for AVX but failed to run on this kernel.

Post by a***@lx42.de
configure: WARNING: No recognized optimization option present in CFLAGS

So you'd have a highly unoptimal build, even if the build worked,
because of you having overridden the CFLAGS.

I like to append the options, how do I achieve this?

Per my testing, the only important option that is lost when you override
CFLAGS is -O2, so simply include -O2 in your own CFLAGS. The rest of
our custom CFLAGS are added on top of those you specify.

Post by Solar Designer
None of this is directly related to the build failure, but it indicates
that your build is highly unusual in general.

The machines are rather old to ensure compatiblity for newer machines.
So upgrading is not an option.

CentOS 4.1 is ridiculously old for this hardware, not letting you use it
fully, but OK. I'm surprised it runs there at all. BTW, besides not
supporting AVX in the kernel and not supporting SSSE3+ in gcc, I guess
it also fails to enable turbo boost on these CPUs.

Post by Solar Designer
As to the build failure, what's your version of "make" - e.g., what does
"make -v" say?

GNU Make 3.80

OK, we might try to reproduce the problem with that.

Meanwhile, you may try newer GNU make and optionally newer gcc (but then
you'd have to disable AVX manually) in your home directory.

Alexander

Solar Designer

2017-10-26 19:42:53 UTC

Post by a***@lx42.de
GNU Make 3.80

OK, we might try to reproduce the problem with that.

I just did: the problem manifests itself just like this when using GNU
Make 3.80, but goes away with 3.81.

magnum, we have this weird make rule:

%.cl: | ../run/kernels
$(CP) $< ../run/kernels

Do we really need it like that? I'm not familiar with that trick, and
couldn't easily find it in documentation - the '|' character isn't in
the index here:

https://www.gnu.org/software/make/manual/html_node/Concept-Index.html#Concept-Index

Maybe we can avoid it to relax the GNU Make version requirement?

Alternatively, we can use this test as an opportunity to document the
exact minimum version of GNU Make we require in jumbo - now we know it's
3.81 or newer.

Alexander

magnum

2017-10-26 20:05:01 UTC

Post by a***@lx42.de
GNU Make 3.80

OK, we might try to reproduce the problem with that.

I just did: the problem manifests itself just like this when using GNU
Make 3.80, but goes away with 3.81.
%.cl: | ../run/kernels
$(CP) $< ../run/kernels
Do we really need it like that? I'm not familiar with that trick, and
couldn't easily find it in documentation - the '|' character isn't in
https://www.gnu.org/software/make/manual/html_node/Concept-Index.html#Concept-Index

I can't even remember what it does... we need to figure that out >.<

Post by Solar Designer
Maybe we can avoid it to relax the GNU Make version requirement?
Alternatively, we can use this test as an opportunity to document the
exact minimum version of GNU Make we require in jumbo - now we know it's
3.81 or newer.

That would be an alternative. I wonder how we could make 'make' itself
bail with an informative message if it's too old...

magnum

magnum

2017-11-07 07:18:36 UTC

Post by magnum

%.cl: | ../run/kernels
$(CP) $< ../run/kernels
Do we really need it like that? I'm not familiar with that trick, and
couldn't easily find it in documentation - the '|' character isn't in
https://www.gnu.org/software/make/manual/html_node/Concept-Index.html#Concept-Index

I can't even remember what it does... we need to figure that out >.<

https://www.gnu.org/software/make/manual/html_node/Prerequisite-Types.html

Without the pipe sign, a changed timestamp on ../run/kernels would
trigger an unwanted "rebuild" of the target (since a directory's
timestamp will change whenever a file is put there). We just need
../run/kernels to exist at all.

magnum

a***@lx42.de

2017-11-28 20:40:12 UTC