[PATCHES] Generate shorter name for extra python files

Hi Lukas,

diff --git a/buildtools/wafsamba/samba_python.py b/buildtools/wafsamba/samba_python.py
index f97439c945b4e986bacef39387cf4168d419e158..bfaf5e6250e0ec17e9d300fc7c839bae769cec1e 100644
--- a/buildtools/wafsamba/samba_python.py
+++ b/buildtools/wafsamba/samba_python.py
abi_pattern = os.path.splitext(conf.env['pyext_PATTERN'])[0]
conf.env['PYTHON_SO_ABI_FLAG'] = abi_pattern % ''
+ conf.env['PYTHON_LIBNAME_SO_ABI_FLAG'] = conf.env['PYTHON_VERSION']
conf.env['PYTHON_SO_ABI_FLAG'] = ''
- conf.env['PYTHON_LIBNAME_SO_ABI_FLAG'] = (
- conf.env['PYTHON_SO_ABI_FLAG'].replace('_', '-'))
+ conf.env['PYTHON_LIBNAME_SO_ABI_FLAG'] = ''
@@ -170,7 +170,7 @@ Build.BuildContext.SAMBA_PYTHON = SAMBA_PYTHON
- return name + bld.env['PYTHON_SO_ABI_FLAG']
+ return name + bld.env['PYTHON_VERSION']
return name

what is conf.env['PYTHON_LIBNAME_SO_ABI_FLAG'] used for?
It's write-only in this patchset.

metze

Lukas Slebodnik via samba-technical

2017-07-05 13:43:05 UTC

Post by Stefan Metzmacher via samba-technical
Hi Lukas,

what is conf.env['PYTHON_LIBNAME_SO_ABI_FLAG'] used for?
It's write-only in this patchset.

It is used for generating different name libraries used in python bindings.
And therefore it is also in pkgconfig files

sh$ git grep -n PYTHON_LIBNAME_SO_ABI_FLAG
buildtools/wafsamba/samba_python.py:88: conf.env['PYTHON_LIBNAME_SO_ABI_FLAG'] = conf.env['PYTHON_VERSION']
buildtools/wafsamba/samba_python.py:91: conf.env['PYTHON_LIBNAME_SO_ABI_FLAG'] = ''
lib/ldb/pyldb-util.pc.in:11:Libs: @LIB_RPATH@ -L${libdir} -lpyldb-***@PYTHON_LIBNAME_SO_ABI_FLAG@
lib/talloc/pytalloc-util.pc.in:9:Libs: @LIB_RPATH@ -L${libdir} -lpytalloc-***@PYTHON_LIBNAME_SO_ABI_FLAG@

BTW I added author(s) to CC because I would like to know their opinions.

LS

Lukas Slebodnik via samba-technical

2017-07-06 10:19:16 UTC

Post by Stefan Metzmacher via samba-technical
Hi Lukas,

what is conf.env['PYTHON_LIBNAME_SO_ABI_FLAG'] used for?
It's write-only in this patchset.

BTW PYTHON_LIBNAME_SO_ABI_FLAG is used only for extra python
and therefore pkgconfig file is the same for all architectures with python2
armv7hl https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112827
i686 https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112802
ppc64 https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112787
ppc64le https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112771
s390x https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112816
x86_64 https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112762
aarch64 https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112772

and different pkgconfig files with extra python (python3.6)
armv7hl https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112825
i686 https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112810
ppc64 https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112783
ppc64le https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112764
s390x https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112819
x86_64 https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112760
aarch64 https://koji.fedoraproject.org/koji/rpminfo?rpmID=10112777

It will be much simpler with this patchset and there still be unique name
for different version of python

/usr/lib64/libpyldb-util3.6.so
/usr/lib64/pkgconfig/pyldb-util3.6.pc

I assume that packagers on debian will appreciate it as well
because they build packages on more architectures.

LS

Andreas Schneider via samba-technical

2017-07-06 09:54:24 UTC

On Wednesday, 5 July 2017 15:26:33 CEST Lukas Slebodnik via samba-technical

Post by Lukas Slebodnik via samba-technical
ehlo,
Generating different name with different architecture and with the same
version of python is not ideal. pkgconfig files should be architecture
independent and libraries for different architectures are stored in
different directories
e.g. (/usr/lib64, /usr/lib, /usr/lib/x86_64-linux-gnu/ ...)
Therefore it will be simpler to remove architecture from names
/usr/lib64/pkgconfig/pyldb-util.cpython-36m-x86_64-linux-gnu.pc
vs.
/usr/lib64/pkgconfig/pyldb-util36.pc
LS

RB+

Could we get new releases for talloc and ldb including this? Also
f5cafee0c7a96396798d2b229ff3f9dced1d74f3 is not part of a release!

Andreas

--
Andreas Schneider GPG-ID: CC014E3D
Samba Team ***@samba.org
www.samba.org

Andrew Bartlett via samba-technical

2017-07-06 10:14:09 UTC

Post by Andreas Schneider via samba-technical
On Wednesday, 5 July 2017 15:26:33 CEST Lukas Slebodnik via samba-technical

RB+
Could we get new releases for talloc and ldb including this? Also
f5cafee0c7a96396798d2b229ff3f9dced1d74f3 is not part of a release!

I'm a bit nervous about this. I tried to untangle a mess on Debian in
this area, and ended up abandoning the py3 package for talloc with the
parent commits to this:

https://anonscm.debian.org/cgit/pkg-samba/talloc.git/commit/?id=11a83cac638719df56d8115837f9b8b744d6a69e

I'm not opposed to the idea in the patches, but I'm not sure it is
changing enough, or the right things. The variables there have become
a tangled mess!

I realise this was 'just' trying to sort out pkgconfig, but we do need
to sort out the ABI upstream, not in Debian etc, but I would love it if
someone else could help me ensure that we could now re-enabled py3 in
debian without hacks and with the right thing happening. Ideally we
get this changed once, break the ABI once, and get this right.

I think we will need to break the symbol names for pytalloc-util.
(Does this matter?)

Thanks,

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

Andreas Schneider via samba-technical

2017-07-06 10:16:53 UTC

Post by Andreas Schneider via samba-technical
On Wednesday, 5 July 2017 15:26:33 CEST Lukas Slebodnik via
samba-technical

RB+
Could we get new releases for talloc and ldb including this? Also
f5cafee0c7a96396798d2b229ff3f9dced1d74f3 is not part of a release!

I'm a bit nervous about this. I tried to untangle a mess on Debian in
this area, and ended up abandoning the py3 package for talloc with the
https://anonscm.debian.org/cgit/pkg-samba/talloc.git/commit/?id=11a83cac6387
19df56d8115837f9b8b744d6a69e
I'm not opposed to the idea in the patches, but I'm not sure it is
changing enough, or the right things. The variables there have become
a tangled mess!
I realise this was 'just' trying to sort out pkgconfig, but we do need
to sort out the ABI upstream, not in Debian etc, but I would love it if
someone else could help me ensure that we could now re-enabled py3 in
debian without hacks and with the right thing happening. Ideally we
get this changed once, break the ABI once, and get this right.
I think we will need to break the symbol names for pytalloc-util.
(Does this matter?)

I don't think that python3 is used yet but we need to get it working correctly
for Samba 4.7.

Andreas

--
Andreas Schneider GPG-ID: CC014E3D
Samba Team ***@samba.org
www.samba.org

Lukas Slebodnik via samba-technical

2017-07-06 10:28:50 UTC

Post by Andreas Schneider via samba-technical
On Wednesday, 5 July 2017 15:26:33 CEST Lukas Slebodnik via samba-technical

RB+
Could we get new releases for talloc and ldb including this? Also
f5cafee0c7a96396798d2b229ff3f9dced1d74f3 is not part of a release!

I tent to agree.

Post by Andrew Bartlett via samba-technical
I realise this was 'just' trying to sort out pkgconfig, but we do need
to sort out the ABI upstream, not in Debian etc, but I would love it if
someone else could help me ensure that we could now re-enabled py3 in
debian without hacks and with the right thing happening. Ideally we
get this changed once, break the ABI once, and get this right.

It solved also libraries and not only pkgconfig files

/usr/lib64/libpyldb-util3.6.so
/usr/lib64/pkgconfig/pyldb-util3.6.pc

And we need to have different name for library because
libpyldb-util3.6.so is linked against python3.6 and have to be used
only by python module for 3.6.

Standard libpyldb-util.so is linked with standard python (usually 2.7)

I am open to another suggestion for names. Maybe libpyldb-util-3.6.so
but that's minimal change.

Could you describe what kind of packaging problems did you have on debian?
It is not obvious for me from that commit.

LS

Andrew Bartlett via samba-technical

2017-07-06 10:44:38 UTC

Post by Andreas Schneider via samba-technical
On Wednesday, 5 July 2017 15:26:33 CEST Lukas Slebodnik via samba-technical

RB+
Could we get new releases for talloc and ldb including this? Also
f5cafee0c7a96396798d2b229ff3f9dced1d74f3 is not part of a release!

I tent to agree.

It solved also libraries and not only pkgconfig files
/usr/lib64/libpyldb-util3.6.so
/usr/lib64/pkgconfig/pyldb-util3.6.pc
And we need to have different name for library because
libpyldb-util3.6.so is linked against python3.6 and have to be used
only by python module for 3.6.
Standard libpyldb-util.so is linked with standard python (usually 2.7)
I am open to another suggestion for names. Maybe libpyldb-util-3.6.so
but that's minimal change.
Could you describe what kind of packaging problems did you have on debian?
It is not obvious for me from that commit.

Here is the thread:

https://lists.alioth.debian.org/pipermail/pkg-samba-maint/2016-April/018139.html

It looks like I suggested following Fedora, so I'll generally stick to
that position.

When reading the thread and trying to read symbol versions, remember
the mail archiver has changed @ to _at_ just to confuse you/spammers.

Thanks!

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

Lukas Slebodnik via samba-technical

2017-07-06 11:52:56 UTC

Post by Andreas Schneider via samba-technical
On Wednesday, 5 July 2017 15:26:33 CEST Lukas Slebodnik via samba-technical

RB+
Could we get new releases for talloc and ldb including this? Also
f5cafee0c7a96396798d2b229ff3f9dced1d74f3 is not part of a release!

I tent to agree.

It solved also libraries and not only pkgconfig files
/usr/lib64/libpyldb-util3.6.so
/usr/lib64/pkgconfig/pyldb-util3.6.pc
And we need to have different name for library because
libpyldb-util3.6.so is linked against python3.6 and have to be used
only by python module for 3.6.
Standard libpyldb-util.so is linked with standard python (usually 2.7)
I am open to another suggestion for names. Maybe libpyldb-util-3.6.so
but that's minimal change.
Could you describe what kind of packaging problems did you have on debian?
It is not obvious for me from that commit.

https://lists.alioth.debian.org/pipermail/pkg-samba-maint/2016-April/018139.html
It looks like I suggested following Fedora, so I'll generally stick to
that position.
When reading the thread and trying to read symbol versions, remember

Most of the thread there was discussion about changed version symbols
in library which is not related to python3 work.

The only think which I do not understand is following sentence
"I've tried changing the SAMBA_LIBRARY code
to match, but I still get .py3 and -py3 mixups"

And I do not think that was solved by this patchset.

LS

Andrew Bartlett via samba-technical

2017-07-06 22:18:43 UTC

Post by Lukas Slebodnik via samba-technical
It solved also libraries and not only pkgconfig files
/usr/lib64/libpyldb-util3.6.so
/usr/lib64/pkgconfig/pyldb-util3.6.pc
And we need to have different name for library because
libpyldb-util3.6.so is linked against python3.6 and have to be used
only by python module for 3.6.
Standard libpyldb-util.so is linked with standard python (usually 2.7)
I am open to another suggestion for names. Maybe libpyldb-util-
3.6.so
but that's minimal change.
Could you describe what kind of packaging problems did you have on debian?
It is not obvious for me from that commit.

https://lists.alioth.debian.org/pipermail/pkg-samba-maint/2016-Apri
l/018139.html
It looks like I suggested following Fedora, so I'll generally stick to
that position.
When reading the thread and trying to read symbol versions,
remember
you/spammers.

Most of the thread there was discussion about changed version symbols
in library which is not related to python3 work.

No, it is very related. That is my point. Debian was trying to work
around the same things, but it ended badly. We need to solve it all,
not just bits of it.

Post by Lukas Slebodnik via samba-technical
The only think which I do not understand is following sentence
"I've tried changing the SAMBA_LIBRARY code
to match, but I still get .py3 and -py3 mixups"
And I do not think that was solved by this patchset.

OK. Please continue working until it is.

The things we need:

Remove both cpython-35m-x86_64-linux-gnu and 3.5 from
- ABI/*.sigs files

Then remove at least the cpython-* stuff from
- the pkgconfig files
- the installed library names

I'm not sure what to do with the .vscript files

The problem with the current patch is that it now puts more version
info in than we had before, the Samba ABI/*.sigs files are now being
generated per python version.

Sorry,

Andrew Bartlett

--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team https://samba.org
Samba Development and Support, Catalyst IT
https://catalyst.net.nz/services/samba

Lukas Slebodnik via samba-technical

2017-07-07 12:30:12 UTC

https://lists.alioth.debian.org/pipermail/pkg-samba-maint/2016-Apri
l/018139.html
It looks like I suggested following Fedora, so I'll generally stick to
that position.
When reading the thread and trying to read symbol versions,
remember
you/spammers.

Most of the thread there was discussion about changed version symbols
in library which is not related to python3 work.

No, it is very related. That is my point. Debian was trying to work
around the same things, but it ended badly. We need to solve it all,
not just bits of it.

OK. Please continue working until it is.
Remove both cpython-35m-x86_64-linux-gnu and 3.5 from
- ABI/*.sigs files

I am not sure whether I understand problem correctly.

Here is an output of objdump with **my patches**.
SONAME is obviously different.

[***@e71533851f88 ~]# objdump -p /usr/lib64/libpyldb-util.so | grep SONAME
SONAME libpyldb-util.so.1
[***@e71533851f88 ~]# objdump -p /usr/lib64/libpyldb-util3.5.so | grep SONAME
SONAME libpyldb-util3.5.so.1

Python3 version of library libpyldb-util has only version from 1.2.0
Python2 version of library libpyldb-util has all version since 1.1.2

[***@e71533851f88 ~]# objdump -p /usr/lib64/libpyldb-util3.5.so | sed -ne '/Version definitions/,/Version References/p'
Version definitions:
1 0x01 0x0c4f47b1 libpyldb-util3.5.so.1
2 0x00 0x064f6b10 PYLDB_UTIL3.5_1.2.0

Version References:
[***@e71533851f88 ~]# objdump -p /usr/lib64/libpyldb-util.so | sed -ne '/Version definitions/,/Version References/p'
Version definitions:
1 0x01 0x0a7956b1 libpyldb-util.so.1
2 0x00 0x0aa3e6f2 PYLDB_UTIL_1.1.2
3 0x02 0x0aa3e6f3 PYLDB_UTIL_1.1.3
PYLDB_UTIL_1.1.2
4 0x02 0x0aa3e6f4 PYLDB_UTIL_1.1.4
PYLDB_UTIL_1.1.3
5 0x02 0x0aa3e6f5 PYLDB_UTIL_1.1.5
PYLDB_UTIL_1.1.4
6 0x02 0x0aa3e6f6 PYLDB_UTIL_1.1.6
PYLDB_UTIL_1.1.5
7 0x02 0x0aa3e6f7 PYLDB_UTIL_1.1.7
PYLDB_UTIL_1.1.6
8 0x02 0x0aa3e6f8 PYLDB_UTIL_1.1.8
PYLDB_UTIL_1.1.7
9 0x02 0x0aa3e6f9 PYLDB_UTIL_1.1.9
PYLDB_UTIL_1.1.8
10 0x02 0x0a3e6fe0 PYLDB_UTIL_1.1.10
PYLDB_UTIL_1.1.9
11 0x02 0x0a3e6fe1 PYLDB_UTIL_1.1.11
PYLDB_UTIL_1.1.10
12 0x02 0x0a3e6fe2 PYLDB_UTIL_1.1.12
PYLDB_UTIL_1.1.11
13 0x02 0x0a3e6fe3 PYLDB_UTIL_1.1.13
PYLDB_UTIL_1.1.12
14 0x02 0x0a3e6fe4 PYLDB_UTIL_1.1.14
PYLDB_UTIL_1.1.13
15 0x02 0x0a3e6fe5 PYLDB_UTIL_1.1.15
PYLDB_UTIL_1.1.14
16 0x02 0x0a3e6fe6 PYLDB_UTIL_1.1.16
PYLDB_UTIL_1.1.15
17 0x02 0x0a3e6fe7 PYLDB_UTIL_1.1.17
PYLDB_UTIL_1.1.16
18 0x02 0x0a3e6fe8 PYLDB_UTIL_1.1.18
PYLDB_UTIL_1.1.17
19 0x02 0x0a3e6fe9 PYLDB_UTIL_1.1.19
PYLDB_UTIL_1.1.18
20 0x02 0x0a3e6ff0 PYLDB_UTIL_1.1.20
PYLDB_UTIL_1.1.19
21 0x02 0x0a3e6ff1 PYLDB_UTIL_1.1.21
PYLDB_UTIL_1.1.20
22 0x02 0x0a3e6ff2 PYLDB_UTIL_1.1.22
PYLDB_UTIL_1.1.21
23 0x02 0x0a3e6ff3 PYLDB_UTIL_1.1.23
PYLDB_UTIL_1.1.22
24 0x02 0x0a3e6ff4 PYLDB_UTIL_1.1.24
PYLDB_UTIL_1.1.23
25 0x02 0x0a3e6ff5 PYLDB_UTIL_1.1.25
PYLDB_UTIL_1.1.24
26 0x02 0x0a3e6ff6 PYLDB_UTIL_1.1.26
PYLDB_UTIL_1.1.25
27 0x02 0x0a3e6ff7 PYLDB_UTIL_1.1.27
PYLDB_UTIL_1.1.26
28 0x02 0x0a3e6ff8 PYLDB_UTIL_1.1.28
PYLDB_UTIL_1.1.27
29 0x02 0x0a3e6ff9 PYLDB_UTIL_1.1.29
PYLDB_UTIL_1.1.28
30 0x02 0x0a3e6fc0 PYLDB_UTIL_1.1.30
PYLDB_UTIL_1.1.29
31 0x02 0x0a3e6fc1 PYLDB_UTIL_1.1.31
PYLDB_UTIL_1.1.30
32 0x00 0x0aa3e3f0 PYLDB_UTIL_1.2.0

Version References:

[***@e71533851f88 ~]# objdump -T /usr/lib64/libpyldb-util.so | grep PYLDB_UTIL | sort
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.10 PYLDB_UTIL_1.1.10
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.11 PYLDB_UTIL_1.1.11
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.12 PYLDB_UTIL_1.1.12
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.13 PYLDB_UTIL_1.1.13
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.14 PYLDB_UTIL_1.1.14
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.15 PYLDB_UTIL_1.1.15
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.16 PYLDB_UTIL_1.1.16
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.17 PYLDB_UTIL_1.1.17
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.18 PYLDB_UTIL_1.1.18
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.19 PYLDB_UTIL_1.1.19
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.2 PYLDB_UTIL_1.1.2
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.20 PYLDB_UTIL_1.1.20
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.21 PYLDB_UTIL_1.1.21
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.22 PYLDB_UTIL_1.1.22
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.23 PYLDB_UTIL_1.1.23
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.24 PYLDB_UTIL_1.1.24
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.25 PYLDB_UTIL_1.1.25
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.26 PYLDB_UTIL_1.1.26
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.27 PYLDB_UTIL_1.1.27
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.28 PYLDB_UTIL_1.1.28
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.29 PYLDB_UTIL_1.1.29
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.3 PYLDB_UTIL_1.1.3
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.30 PYLDB_UTIL_1.1.30
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.31 PYLDB_UTIL_1.1.31
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.4 PYLDB_UTIL_1.1.4
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.5 PYLDB_UTIL_1.1.5
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.6 PYLDB_UTIL_1.1.6
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.7 PYLDB_UTIL_1.1.7
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.8 PYLDB_UTIL_1.1.8
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.1.9 PYLDB_UTIL_1.1.9
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL_1.2.0 PYLDB_UTIL_1.2.0
00000000000014c0 g DF .text 00000000000000c4 PYLDB_UTIL_1.1.2 pyldb_Object_AsDn
0000000000001590 g DF .text 0000000000000081 PYLDB_UTIL_1.1.2 pyldb_Dn_FromDn
[***@e71533851f88 ~]# objdump -T /usr/lib64/libpyldb-util3.5.so | grep PYLDB_UTIL | sort
0000000000000000 g DO *ABS* 0000000000000000 PYLDB_UTIL3.5_1.2.0 PYLDB_UTIL3.5_1.2.0
0000000000000b00 g DF .text 00000000000000e4 PYLDB_UTIL3.5_1.2.0 pyldb_Object_AsDn
0000000000000bf0 g DF .text 0000000000000081 PYLDB_UTIL3.5_1.2.0 pyldb_Dn_FromDn

I am not really sure how are symbols generated in libldb with waf
from files in directory lib/ldb/ABI/ because I am used to different definition
of version symbols in libraries http://ftp.gnu.org/old-gnu/Manuals/ld-2.9.1/html_node/ld_25.html
But IIRC it does not work on all UNIX like systems which are supported by samba.

Could you describe what is a problematic? Why it is problematic?
And how it should look like?

Post by Andrew Bartlett via samba-technical
Then remove at least the cpython-* stuff from
- the pkgconfig files
- the installed library names

Already done in patchset

[***@e71533851f88 ~]# cat /usr/lib64/pkgconfig/pyldb-util
pyldb-util.pc pyldb-util3.5.pc
[***@e71533851f88 ~]# cat /usr/lib64/pkgconfig/pyldb-util*
prefix=/usr
exec_prefix=${prefix}
libdir=${prefix}/lib64
includedir=${prefix}/include
modulesdir=${prefix}/lib64/ldb/modules/ldb

Name: pyldb-util2.7
Description: Python bindings for LDB
Version: 1.2.0
Requires: ldb
Libs: -L${libdir} -lpyldb-util
Cflags: -I${includedir}
URL: http://ldb.samba.org/
prefix=/usr
exec_prefix=${prefix}
libdir=${prefix}/lib64
includedir=${prefix}/include
modulesdir=${prefix}/lib64/ldb/modules/ldb

Name: pyldb-util3.5
Description: Python bindings for LDB
Version: 1.2.0
Requires: ldb
Libs: -L${libdir} -lpyldb-util3.5
Cflags: -I${includedir}
URL: http://ldb.samba.org/

Post by Andrew Bartlett via samba-technical
I'm not sure what to do with the .vscript files
The problem with the current patch is that it now puts more version
info in than we had before, the Samba ABI/*.sigs files are now being
generated per python version.

And you would like to remove dumplication of files. Is it correct?
sh$ ls lib/ldb/ABI/pyldb-util*
lib/ldb/ABI/pyldb-util-1.1.10.sigs lib/ldb/ABI/pyldb-util-1.1.30.sigs
lib/ldb/ABI/pyldb-util-1.1.11.sigs lib/ldb/ABI/pyldb-util-1.1.31.sigs
lib/ldb/ABI/pyldb-util-1.1.12.sigs lib/ldb/ABI/pyldb-util-1.1.3.sigs
lib/ldb/ABI/pyldb-util-1.1.13.sigs lib/ldb/ABI/pyldb-util-1.1.4.sigs
lib/ldb/ABI/pyldb-util-1.1.14.sigs lib/ldb/ABI/pyldb-util-1.1.5.sigs
lib/ldb/ABI/pyldb-util-1.1.15.sigs lib/ldb/ABI/pyldb-util-1.1.6.sigs
lib/ldb/ABI/pyldb-util-1.1.16.sigs lib/ldb/ABI/pyldb-util-1.1.7.sigs
lib/ldb/ABI/pyldb-util-1.1.17.sigs lib/ldb/ABI/pyldb-util-1.1.8.sigs
lib/ldb/ABI/pyldb-util-1.1.18.sigs lib/ldb/ABI/pyldb-util-1.1.9.sigs
lib/ldb/ABI/pyldb-util-1.1.19.sigs lib/ldb/ABI/pyldb-util-1.2.0.sigs
lib/ldb/ABI/pyldb-util-1.1.20.sigs lib/ldb/ABI/pyldb-util.py3-1.1.23.sigs
lib/ldb/ABI/pyldb-util-1.1.21.sigs lib/ldb/ABI/pyldb-util.py3-1.1.24.sigs
lib/ldb/ABI/pyldb-util-1.1.22.sigs lib/ldb/ABI/pyldb-util.py3-1.1.25.sigs
lib/ldb/ABI/pyldb-util-1.1.23.sigs lib/ldb/ABI/pyldb-util.py3-1.1.26.sigs
lib/ldb/ABI/pyldb-util-1.1.24.sigs lib/ldb/ABI/pyldb-util.py3-1.1.27.sigs
lib/ldb/ABI/pyldb-util-1.1.25.sigs lib/ldb/ABI/pyldb-util.py3-1.1.28.sigs
lib/ldb/ABI/pyldb-util-1.1.26.sigs lib/ldb/ABI/pyldb-util.py3-1.1.29.sigs
lib/ldb/ABI/pyldb-util-1.1.27.sigs lib/ldb/ABI/pyldb-util.py3-1.1.30.sigs
lib/ldb/ABI/pyldb-util-1.1.28.sigs lib/ldb/ABI/pyldb-util.py3-1.1.31.sigs
lib/ldb/ABI/pyldb-util-1.1.29.sigs lib/ldb/ABI/pyldb-util.py3-1.2.0.sigs
lib/ldb/ABI/pyldb-util-1.1.2.sigs

Could you point me to the code where these files are processed?

LS

Lukas Slebodnik via samba-technical

2017-07-08 19:16:53 UTC

Post by Andrew Bartlett via samba-technical
OK. Please continue working until it is.
Remove both cpython-35m-x86_64-linux-gnu and 3.5 from
- ABI/*.sigs files

I am not really sure how are symbols generated in libldb with waf
from files in directory lib/ldb/ABI/ because I am used to different definition
of version symbols in libraries http://ftp.gnu.org/old-gnu/Manuals/ld-2.9.1/html_node/ld_25.html
But IIRC it does not work on all UNIX like systems which are supported by samba.

I played a little bit with waf and python debuger (pdb)
And I found how and where is version script(vscript) generated.

Post by Lukas Slebodnik via samba-technical
Could you describe what is a problematic? Why it is problematic?
And how it should look like?

I just would like to know what is problematic for libpyldb-util generated from
standard python and for extra python.

LS

Andrew Bartlett via samba-technical

2017-07-09 20:33:51 UTC

Post by Andrew Bartlett via samba-technical
OK. Please continue working until it is.
Remove both cpython-35m-x86_64-linux-gnu and 3.5 from
- ABI/*.sigs files

I played a little bit with waf and python debuger (pdb)
And I found how and where is version script(vscript) generated.

Post by Lukas Slebodnik via samba-technical
Could you describe what is a problematic? Why it is problematic?
And how it should look like?

I just would like to know what is problematic for libpyldb-util generated from
standard python and for extra python.

The ABI files, which we use in Samba to check when we add new symbols
or change function arguments, need not to be dependent on python
versions, as Samba developers have multiple python3 versions installed,
but we only want one ABI file.

However, the vscript and the generated library needs to be python
version specific.

The old code did this by removing the cpython... string from the ABI
file and symbol names before writing the file. You need to (carefully,
as it is now a shorter and less unique string) do the same.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

Lukas Slebodnik via samba-technical

2017-07-11 08:50:54 UTC

Post by Andrew Bartlett via samba-technical
OK. Please continue working until it is.
Remove both cpython-35m-x86_64-linux-gnu and 3.5 from
- ABI/*.sigs files

I played a little bit with waf and python debuger (pdb)
And I found how and where is version script(vscript) generated.

Post by Lukas Slebodnik via samba-technical
Could you describe what is a problematic? Why it is problematic?
And how it should look like?

I just would like to know what is problematic for libpyldb-util generated from
standard python and for extra python.

The ABI files, which we use in Samba to check when we add new symbols
or change function arguments, need not to be dependent on python
versions, as Samba developers have multiple python3 versions installed,
but we only want one ABI file.
However, the vscript and the generated library needs to be python
version specific.
The old code did this by removing the cpython... string from the ABI
file and symbol names before writing the file. You need to (carefully,
as it is now a shorter and less unique string) do the same.

Here you are.

LS

Andrew Bartlett via samba-technical

2017-07-13 20:43:58 UTC

Post by Andrew Bartlett via samba-technical
The ABI files, which we use in Samba to check when we add new symbols
or change function arguments, need not to be dependent on python
versions, as Samba developers have multiple python3 versions installed,
but we only want one ABI file.
However, the vscript and the generated library needs to be python
version specific.
The old code did this by removing the cpython... string from the ABI
file and symbol names before writing the file. You need to (carefully,
as it is now a shorter and less unique string) do the same.

Here you are.

Is it safe to assert that we won't change the ABI with an IS_PYTHON3 or
similar macro?

If so, then these are OK (in that respect, I've not done a full
review), but if it is likely that we will add python3 only helper
functions to one or other of the py*-util libraries, then we need to
keep the distinct ABI files for 2 and 3, but not for 3.5, 3.6 et al.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

Lukas Slebodnik via samba-technical

2017-07-14 19:49:01 UTC

Here you are.

Is it safe to assert that we won't change the ABI with an IS_PYTHON3 or
similar macro?

py*-util libraries are helper functions for python bindings. And new functions
are added there very rarely. (btw tevent and tdb does not need such libraries).

And these helper libraries will be used just by python bindings.
And IIUC python guarantee ABI in minor release (e.g. 3.4.*)
This is a reaon why there will be different version of library for different
python (2.6, 2.7, 3.5, 3.6 ...) But I am not python developer/mantainer.
Adding Petr Viktorin to CC

Post by Andrew Bartlett via samba-technical
If so, then these are OK (in that respect, I've not done a full
review), but if it is likely that we will add python3 only helper
functions to one or other of the py*-util libraries, then we need to
keep the distinct ABI files for 2 and 3, but not for 3.5, 3.6 et al.

In theory, that could be solved with extra glob
"p?y?3?-" + $old_glob.

Thank you very much for review.

Sorry for duplicate mail I accidentally removed samba-technical from CC.

LS

Stefan Metzmacher via samba-technical

2017-07-21 12:39:38 UTC

Hi,

as this will take a bit to discuss, I'll prepare new talloc, tevent
and ldb releases without this.

We need an ldb release for 4.7.0rc3 which is planed for next Tuesday.

metze

Here you are.

Is it safe to assert that we won't change the ABI with an IS_PYTHON3 or
similar macro?
If so, then these are OK (in that respect, I've not done a full
review), but if it is likely that we will add python3 only helper
functions to one or other of the py*-util libraries, then we need to
keep the distinct ABI files for 2 and 3, but not for 3.5, 3.6 et al.
Andrew Bartlett

Lukas Slebodnik via samba-technical

2017-07-21 13:32:46 UTC

Post by Stefan Metzmacher via samba-technical
Hi,
as this will take a bit to discuss, I'll prepare new talloc, tevent
and ldb releases without this.
We need an ldb release for 4.7.0rc3 which is planed for next Tuesday.

I thought I answer all questions and Abrew wrote that:
"(in that respect, I've not done a full review)"

I am not sure how to move this forward. Or where is a problem.
But I would really appreciate solution which works for upstream
and all downstream debian, fedora ...

LS

Stefan Metzmacher via samba-technical

2017-07-21 13:46:49 UTC

Hi Lukas,

Post by Stefan Metzmacher via samba-technical
as this will take a bit to discuss, I'll prepare new talloc, tevent
and ldb releases without this.
We need an ldb release for 4.7.0rc3 which is planed for next Tuesday.

"(in that respect, I've not done a full review)"
I am not sure how to move this forward. Or where is a problem.
But I would really appreciate solution which works for upstream
and all downstream debian, fedora ...

It's just time, some people are/will be on vacation, so it takes a bit
of time to have a solid solution.

We should not rush this into the release on Thursday. There will
be at least 1 or 2 rc's before 4.7.0.

But the most important thing is that we should not add a solution
to ldb-1.2.1, which we than have to revert for ldb-1.2.2.

Thanks in advance for your patience.
metze

Andrew Bartlett via samba-technical

2017-07-22 01:43:34 UTC

On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-

Post by Stefan Metzmacher via samba-technical
Hi Lukas,

It's just time, some people are/will be on vacation, so it takes a bit
of time to have a solid solution.
We should not rush this into the release on Thursday. There will
be at least 1 or 2 rc's before 4.7.0.
But the most important thing is that we should not add a solution
to ldb-1.2.1, which we than have to revert for ldb-1.2.2.

I agree, we need to get this right. Skipping it for rc3 should be
fine.

This area is complex, and I've been (rather unsuccessfully ;-) on leave
for the past couple of weeks, and it need some serious thought, and I
think the input of the original author to understand the original
rationale.

Then we need to publish the ABI files and binaries, then add a new
function, re-publish and check it actually works, the new function is
tagged with the new version, the old functions untagged in the ABI, the
.so and then the thing debian uses to track those.

That will give me the confidence we have got it right.

Thanks,

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

Lukas Slebodnik via samba-technical

2017-07-24 15:54:59 UTC

Post by Andrew Bartlett via samba-technical
On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-

Post by Stefan Metzmacher via samba-technical
Hi Lukas,

It's just time, some people are/will be on vacation, so it takes a bit
of time to have a solid solution.
We should not rush this into the release on Thursday. There will
be at least 1 or 2 rc's before 4.7.0.
But the most important thing is that we should not add a solution
to ldb-1.2.1, which we than have to revert for ldb-1.2.2.

Do you mean purpose what was purpose of ABI/*.sigs instead of writing
version script directly? Or something related to names for extra
python util libraries.

Post by Andrew Bartlett via samba-technical
Then we need to publish the ABI files and binaries, then add a new
function, re-publish and check it actually works, the new function is
tagged with the new version, the old functions untagged in the ABI, the
.so and then the thing debian uses to track those.

Do you mean something like removing pytalloc_BaseObject_check
from pytalloc-util-2.1.6.sigs, pytalloc-util-2.1.7.sigs
pytalloc-util-2.1.8.sigs ?

It would be like moving pytalloc-util-2.1.6.sigs from 2.1.6
to 2.1.9. Which sounds a little bit simpler to me for testing purposes.

LS

Lukas Slebodnik via samba-technical

2017-08-15 12:22:58 UTC

Post by Andrew Bartlett via samba-technical
On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-

Post by Stefan Metzmacher via samba-technical
Hi Lukas,

It's just time, some people are/will be on vacation, so it takes a bit
of time to have a solid solution.
We should not rush this into the release on Thursday. There will
be at least 1 or 2 rc's before 4.7.0.
But the most important thing is that we should not add a solution
to ldb-1.2.1, which we than have to revert for ldb-1.2.2.

I agree, we need to get this right. Skipping it for rc3 should be
fine.

And we skipped rc4 as well. But I hope it is still possible
to get patches to final 4.7.0 :-)

Post by Andrew Bartlett via samba-technical
This area is complex, and I've been (rather unsuccessfully ;-) on leave
for the past couple of weeks, and it need some serious thought, and I
think the input of the original author to understand the original
rationale.
Then we need to publish the ABI files and binaries, then add a new
function, re-publish and check it actually works, the new function is
tagged with the new version, the old functions untagged in the ABI, the
.so and then the thing debian uses to track those.
That will give me the confidence we have got it right.

Andrew Bartlett via samba-technical

2017-08-15 22:02:24 UTC

Post by Andrew Bartlett via samba-technical
On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-

Post by Stefan Metzmacher via samba-technical
Hi Lukas,

It's just time, some people are/will be on vacation, so it takes a bit
of time to have a solid solution.
We should not rush this into the release on Thursday. There will
be at least 1 or 2 rc's before 4.7.0.
But the most important thing is that we should not add a solution
to ldb-1.2.1, which we than have to revert for ldb-1.2.2.

I agree, we need to get this right. Skipping it for rc3 should be
fine.

And we skipped rc4 as well. But I hope it is still possible
to get patches to final 4.7.0 :-)

Perhaps. Sadly we are all busy, but this is still our aim. Is there a
particular need for this in 4.7?

This part is critical.

Petr,

Can you comment on what the design intent was here?

This is still critical. We need to ensure this works for a new
incremental release, as that is where the last similar scheme failed.

Andrew Bartlett

--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team https://samba.org
Samba Development and Support, Catalyst IT
https://catalyst.net.nz/services/samba

Alexander Bokovoy via samba-technical

2017-08-16 09:47:56 UTC

Post by Andrew Bartlett via samba-technical
On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-

Post by Stefan Metzmacher via samba-technical
Hi Lukas,

It's just time, some people are/will be on vacation, so it takes a bit
of time to have a solid solution.
We should not rush this into the release on Thursday. There will
be at least 1 or 2 rc's before 4.7.0.
But the most important thing is that we should not add a solution
to ldb-1.2.1, which we than have to revert for ldb-1.2.2.

I agree, we need to get this right. Skipping it for rc3 should be
fine.

And we skipped rc4 as well. But I hope it is still possible
to get patches to final 4.7.0 :-)

Perhaps. Sadly we are all busy, but this is still our aim. Is there a
particular need for this in 4.7?

It sets Python module names for a release. Changing them later would be a bad
practice. Even though they aren't like sonames, they still better to be
fixed in advance.

I could give my RB+ but I thought there is a benefit from having it all
vetted by a wider community.

--
/ Alexander Bokovoy

Lukas Slebodnik via samba-technical

2017-08-16 10:51:59 UTC

Post by Andrew Bartlett via samba-technical
On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-

Post by Stefan Metzmacher via samba-technical
Hi Lukas,

It's just time, some people are/will be on vacation, so it takes a bit
of time to have a solid solution.
We should not rush this into the release on Thursday. There will
be at least 1 or 2 rc's before 4.7.0.
But the most important thing is that we should not add a solution
to ldb-1.2.1, which we than have to revert for ldb-1.2.2.

I agree, we need to get this right. Skipping it for rc3 should be
fine.

And we skipped rc4 as well. But I hope it is still possible
to get patches to final 4.7.0 :-)

Perhaps. Sadly we are all busy, but this is still our aim. Is there a
particular need for this in 4.7?

ATM current version of libraries (with architecture in names for python3)
are not widely used among distributions (only fedora 27 which is in devel phase)
But official samba 4.7.0 might be adopted quite fast due to samba-ad-dc with
MIT krb5. This is a reason why I would prefer to solve this till official
4.7.0. And I would prefer solution which is acceptable for most of
distributions. Downstream only patches are bad.

LS

Andreas Schneider via samba-technical

2017-08-16 11:54:01 UTC

On Wednesday, 16 August 2017 12:51:59 CEST Lukas Slebodnik via samba-technical

Post by Andrew Bartlett via samba-technical
On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-

Post by Stefan Metzmacher via samba-technical
Hi Lukas,

It's just time, some people are/will be on vacation, so it takes a bit
of time to have a solid solution.
We should not rush this into the release on Thursday. There will
be at least 1 or 2 rc's before 4.7.0.
But the most important thing is that we should not add a solution
to ldb-1.2.1, which we than have to revert for ldb-1.2.2.

I agree, we need to get this right. Skipping it for rc3 should be
fine.

And we skipped rc4 as well. But I hope it is still possible
to get patches to final 4.7.0 :-)

Perhaps. Sadly we are all busy, but this is still our aim. Is there a
particular need for this in 4.7?

ATM current version of libraries (with architecture in names for python3)
are not widely used among distributions (only fedora 27 which is in devel
phase) But official samba 4.7.0 might be adopted quite fast due to
samba-ad-dc with MIT krb5. This is a reason why I would prefer to solve
this till official 4.7.0. And I would prefer solution which is acceptable
for most of distributions. Downstream only patches are bad.

I agree we NEED to fix this for 4.7. It will be a pain changing it later!

I would argue that this is a blocker for the 4.7 release.

Andreas

--
Andreas Schneider GPG-ID: CC014E3D
Samba Team ***@samba.org
www.samba.org

Andrew Bartlett via samba-technical

2017-08-17 19:33:07 UTC

Post by Andreas Schneider via samba-technical
On Wednesday, 16 August 2017 12:51:59 CEST Lukas Slebodnik via samba-technical

Post by Andrew Bartlett via samba-technical
On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-

Post by Stefan Metzmacher via samba-technical
Hi Lukas,

Post by Stefan Metzmacher via samba-technical
as this will take a bit to discuss, I'll prepare new talloc,
tevent
and ldb releases without this.
We need an ldb release for 4.7.0rc3 which is planed for next
Tuesday.

It's just time, some people are/will be on vacation, so it takes a bit
of time to have a solid solution.
We should not rush this into the release on Thursday. There will
be at least 1 or 2 rc's before 4.7.0.
But the most important thing is that we should not add a solution
to ldb-1.2.1, which we than have to revert for ldb-1.2.2.

I agree, we need to get this right. Skipping it for rc3 should be
fine.

And we skipped rc4 as well. But I hope it is still possible
to get patches to final 4.7.0 :-)

Perhaps. Sadly we are all busy, but this is still our aim. Is there a
particular need for this in 4.7?

ATM current version of libraries (with architecture in names for python3)
are not widely used among distributions (only fedora 27 which is in devel
phase) But official samba 4.7.0 might be adopted quite fast due to
samba-ad-dc with MIT krb5. This is a reason why I would prefer to solve
this till official 4.7.0. And I would prefer solution which is acceptable
for most of distributions. Downstream only patches are bad.

I agree we NEED to fix this for 4.7. It will be a pain changing it later!
I would argue that this is a blocker for the 4.7 release.

Can you help out on this? The work has been stalled waiting for:
- A comment from Petr on the original design intent
- confirmation of which symbols are generated after another new
version beyond the introduction of this patch. (Because that is what
broke, and made me revert, a similar change in Debian).

Thanks,

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

Petr Viktorin via samba-technical

2017-08-22 09:25:00 UTC

Post by Andreas Schneider via samba-technical
On Wednesday, 16 August 2017 12:51:59 CEST Lukas Slebodnik via samba-technical

Post by Andrew Bartlett via samba-technical
On Fri, 2017-07-21 at 15:46 +0200, Stefan Metzmacher via samba-

Post by Stefan Metzmacher via samba-technical
Hi Lukas,

It's just time, some people are/will be on vacation, so it takes a bit
of time to have a solid solution.
We should not rush this into the release on Thursday. There will
be at least 1 or 2 rc's before 4.7.0.
But the most important thing is that we should not add a solution
to ldb-1.2.1, which we than have to revert for ldb-1.2.2.

I agree, we need to get this right. Skipping it for rc3 should be
fine.

And we skipped rc4 as well. But I hope it is still possible
to get patches to final 4.7.0 :-)

Perhaps. Sadly we are all busy, but this is still our aim. Is there a
particular need for this in 4.7?

ATM current version of libraries (with architecture in names for python3)
are not widely used among distributions (only fedora 27 which is in devel
phase) But official samba 4.7.0 might be adopted quite fast due to
samba-ad-dc with MIT krb5. This is a reason why I would prefer to solve
this till official 4.7.0. And I would prefer solution which is acceptable
for most of distributions. Downstream only patches are bad.

I agree we NEED to fix this for 4.7. It will be a pain changing it later!
I would argue that this is a blocker for the 4.7 release.

- A comment from Petr on the original design intent

I'm sorry for missing this thread; my vacation-mode filtering is too eager.

As for the original design, "extra python" allows building two versions
of Python by a single `make` invocation. Building for both 2 and 3 at
once was a requirement to get the Python3 patches in.
To make this work, I've reused the mechanism Python uses to allow binary
files for multiple versions to co-exist. The "ABI tag" contains just the
information needed to distinguish between ABI-incompatible versions.

Note that "/usr/lib64/pkgconfig/pyldb-util36.pc" does not include all
relevant information: the "m" in "cpython-36m-x86_64-linux-gnu" encodes
build-time configuration; "dm" would mean a debug build with
incompatible ABI.
See [PEP 3149] if you want details.

Using the ABI tag as-is essentially puts the burden of distinguishing
ABI incompatibilities on Python developers/distributors. It's also quite
easy to ask Python for it [0], so, hopefully, buildsystems of packages
that depend on the utils can be made to support this.

That said, I'm not familiar with pkg-config or multilib packaging
outside of Python, or across distros. This is the Python way to do it;
please adjust if it makes sense.

Another note on the original design intent: generally, the "extra
python" needs to be version 3, and the "normal" Python could be either 2
*or* a different version of 3. It's never been tested with py3 as the
non-extra Python, and I'm sure it doesn't actually work now, but at some
point it'll be necessary to make the "non-extra" Python be Python 3,
preferably with the relevant name mangling. Please don't assume that
"not extra-python" implies Python 2.

[PEP 3149]: https://www.python.org/dev/peps/pep-3149/
[0] import sysconfig; sysconfig.get_config_var('SOABI')

--
Petr Viktorin

Andrew Bartlett via samba-technical

2017-08-27 07:02:36 UTC

On Tue, 2017-08-22 at 11:25 +0200, Petr Viktorin via samba-technical

Post by Petr Viktorin via samba-technical
I'm sorry for missing this thread; my vacation-mode filtering is too eager.
As for the original design, "extra python" allows building two versions
of Python by a single `make` invocation. Building for both 2 and 3 at
once was a requirement to get the Python3 patches in.
To make this work, I've reused the mechanism Python uses to allow binary
files for multiple versions to co-exist. The "ABI tag" contains just the
information needed to distinguish between ABI-incompatible versions.
Note that "/usr/lib64/pkgconfig/pyldb-util36.pc" does not include all
relevant information: the "m" in "cpython-36m-x86_64-linux-gnu" encodes
build-time configuration; "dm" would mean a debug build with
incompatible ABI.
See [PEP 3149] if you want details.
Using the ABI tag as-is essentially puts the burden of distinguishing
ABI incompatibilities on Python developers/distributors. It's also quite
easy to ask Python for it [0], so, hopefully, buildsystems of packages
that depend on the utils can be made to support this.
That said, I'm not familiar with pkg-config or multilib packaging
outside of Python, or across distros. This is the Python way to do it;
please adjust if it makes sense.
Another note on the original design intent: generally, the "extra
python" needs to be version 3, and the "normal" Python could be either 2
*or* a different version of 3. It's never been tested with py3 as the
non-extra Python, and I'm sure it doesn't actually work now, but at some
point it'll be necessary to make the "non-extra" Python be Python 3,
preferably with the relevant name mangling. Please don't assume that
"not extra-python" implies Python 2.
[PEP 3149]: https://www.python.org/dev/peps/pep-3149/
[0] import sysconfig; sysconfig.get_config_var('SOABI')

Thanks for the details. If you could work with Lumir and Andreas to
propose something that follows the above and meets Lumir's requirements
that would be great. We can't be the first package to have produced a
python C binding, surely there is some guidance already?

Finally, awkward as it is, if we don't even have a common design intent
I can't support blocking 4.7 on this. A rush really isn't good for
this kind of thing.

Thanks,

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

Petr Viktorin via samba-technical

2017-08-29 08:35:13 UTC

Post by Andrew Bartlett via samba-technical
On Tue, 2017-08-22 at 11:25 +0200, Petr Viktorin via samba-technical

Samba is not the first package with a Python binding, but those "_util"
libraries are quite unique.

General guidance for providing a C API for Python extension modules is
provided in Python documentation [0]: roughly, put the function pointers
(and a version) in a struct, wrap it in a Capsule object, and provide
that as a Python-level object.
To use this, there's a PyCapsule_Import helper that imports a module,
fetches the attribute, and unwraps it to get the struct.

This solves problems with naming/discovery/linking of C-level utilities:
everything is found via the extension module.
But in Samba's case, it would require changes to all code that currently
uses the _util libraries.

[0]
https://docs.python.org/3/extending/extending.html#providing-a-c-api-for-an-extension-module

--
Petr Viktorin

Lukas Slebodnik via samba-technical

2017-09-13 14:06:15 UTC

Post by Andrew Bartlett via samba-technical
On Tue, 2017-08-22 at 11:25 +0200, Petr Viktorin via samba-technical

Samba is not the first package with a Python binding, but those "_util"
libraries are quite unique.
General guidance for providing a C API for Python extension modules is
provided in Python documentation [0]: roughly, put the function pointers (and
a version) in a struct, wrap it in a Capsule object, and provide that as a
Python-level object.
To use this, there's a PyCapsule_Import helper that imports a module, fetches
the attribute, and unwraps it to get the struct.
everything is found via the extension module.
But in Samba's case, it would require changes to all code that currently uses
the _util libraries.
[0] https://docs.python.org/3/extending/extending.html#providing-a-c-api-for-an-extension-module

I think LDVERSION (+ fallback to VERSION) should work well in this case
and does not contain architecture or platform in the string,
which is fine for dynamic library.

sh$ python3
Python 3.6.2 (default, Sep 1 2017, 12:03:48)
[GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.

Post by Petr Viktorin via samba-technical
import sysconfig; sysconfig.get_config_var('SOABI')

'cpython-36m-x86_64-linux-gnu'

Post by Petr Viktorin via samba-technical
from distutils.sysconfig import get_config_var
get_config_var('LDVERSION') or get_config_var('VERSION')

'3.6m'

sh$ python2
Python 2.7.13 (default, Aug 16 2017, 12:56:26)
[GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Post by Petr Viktorin via samba-technical
import sysconfig; sysconfig.get_config_var('SOABI')
from distutils.sysconfig import get_config_var
get_config_var('LDVERSION') or get_config_var('VERSION')

'2.7'

Sorry that it took me long time to update patches.

LS

Petr Viktorin via samba-technical

2017-09-19 12:30:57 UTC

Post by Andrew Bartlett via samba-technical
On Tue, 2017-08-22 at 11:25 +0200, Petr Viktorin via samba-technical

Samba is not the first package with a Python binding, but those "_util"
libraries are quite unique.
General guidance for providing a C API for Python extension modules is
provided in Python documentation [0]: roughly, put the function pointers (and
a version) in a struct, wrap it in a Capsule object, and provide that as a
Python-level object.
To use this, there's a PyCapsule_Import helper that imports a module, fetches
the attribute, and unwraps it to get the struct.
everything is found via the extension module.
But in Samba's case, it would require changes to all code that currently uses
the _util libraries.
[0] https://docs.python.org/3/extending/extending.html#providing-a-c-api-for-an-extension-module

I think LDVERSION (+ fallback to VERSION) should work well in this case
and does not contain architecture or platform in the string,
which is fine for dynamic library.
sh$ python3
Python 3.6.2 (default, Sep 1 2017, 12:03:48)
[GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.

Post by Petr Viktorin via samba-technical
import sysconfig; sysconfig.get_config_var('SOABI')

'cpython-36m-x86_64-linux-gnu'

Post by Petr Viktorin via samba-technical
from distutils.sysconfig import get_config_var
get_config_var('LDVERSION') or get_config_var('VERSION')

'3.6m'
sh$ python2
Python 2.7.13 (default, Aug 16 2017, 12:56:26)
[GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Post by Petr Viktorin via samba-technical
import sysconfig; sysconfig.get_config_var('SOABI')
from distutils.sysconfig import get_config_var
get_config_var('LDVERSION') or get_config_var('VERSION')

'2.7'
Sorry that it took me long time to update patches.

This looks reasonable to me, two nitpicks:

The resulting filename is "libpyldb-util36m.so.1", with no separator
before the tag. Shouldn't there be one?
If just "-" is used as a separator, the tag can't be a number otherwise
the buildsystem apparently considers it a version. I suggest adding
either "-py" in front, or using "-cp" (short for "CPython") to match
[PEP 425].

And, env['PYTHON_SO_ABI_FLAG'] is no longer necessary.

[PEP 425]: https://www.python.org/dev/peps/pep-0425/#abi-tag

--
Petr Viktorin

Lukas Slebodnik via samba-technical

2017-09-20 20:44:59 UTC

Post by Andrew Bartlett via samba-technical
On Tue, 2017-08-22 at 11:25 +0200, Petr Viktorin via samba-technical

Samba is not the first package with a Python binding, but those "_util"
libraries are quite unique.
General guidance for providing a C API for Python extension modules is
provided in Python documentation [0]: roughly, put the function pointers (and
a version) in a struct, wrap it in a Capsule object, and provide that as a
Python-level object.
To use this, there's a PyCapsule_Import helper that imports a module, fetches
the attribute, and unwraps it to get the struct.
everything is found via the extension module.
But in Samba's case, it would require changes to all code that currently uses
the _util libraries.
[0] https://docs.python.org/3/extending/extending.html#providing-a-c-api-for-an-extension-module

I think LDVERSION (+ fallback to VERSION) should work well in this case
and does not contain architecture or platform in the string,
which is fine for dynamic library.
sh$ python3
Python 3.6.2 (default, Sep 1 2017, 12:03:48)
[GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux
Type "help", "copyright", "credits" or "license" for more information.

Post by Petr Viktorin via samba-technical
import sysconfig; sysconfig.get_config_var('SOABI')

'cpython-36m-x86_64-linux-gnu'

Post by Petr Viktorin via samba-technical
from distutils.sysconfig import get_config_var
get_config_var('LDVERSION') or get_config_var('VERSION')

'3.6m'
sh$ python2
Python 2.7.13 (default, Aug 16 2017, 12:56:26)
[GCC 7.1.1 20170802 (Red Hat 7.1.1-7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

Post by Petr Viktorin via samba-technical
import sysconfig; sysconfig.get_config_var('SOABI')
from distutils.sysconfig import get_config_var
get_config_var('LDVERSION') or get_config_var('VERSION')

'2.7'
Sorry that it took me long time to update patches.

The resulting filename is "libpyldb-util36m.so.1", with no separator before
the tag. Shouldn't there be one?
If just "-" is used as a separator, the tag can't be a number otherwise the
buildsystem apparently considers it a version. I suggest adding either "-py"
in front, or using "-cp" (short for "CPython") to match [PEP 425].

Sounds reasonable.

And, env['PYTHON_SO_ABI_FLAG'] is no longer necessary.
[PEP 425]: https://www.python.org/dev/peps/pep-0425/#abi-tag

Thank you for comments and review.

From 15c791b2ceeabab1fc938d6aab85c8c5e2313776 Mon Sep 17 00:00:00 2001
Date: Tue, 19 Sep 2017 14:00:32 +0200
Subject: [PATCH 1/2] Separate the Python ABI flag from the rest of the so name
---

I hope you don't mind that I squashed this patch to the 1st one in patchset.

LS

Andrew Bartlett via samba-technical

2017-09-20 21:27:31 UTC

Thanks for sending in the new patch set.

My concern is that by having the same python ABI file in samba, it will
break with:

#if PY_MAJOR_VERSION < 3

static void py_cobject_talloc_free(void *ptr)
{
        talloc_free(ptr);
}

_PUBLIC_ PyObject *pytalloc_CObject_FromTallocPtr(void *ptr)
{
        if (ptr == NULL) {
                Py_RETURN_NONE;
        }
        return PyCObject_FromVoidPtr(ptr, py_cobject_talloc_free);
}

#endif

from pytalloc-util.c

If you build with python 2 that will not be included in the ABI, and if
you build with python3 it will.

That is why we need distinct ABI files for python 2 and python 3.

Andrew Bartlett

--
Andrew Bartlett
https://samba.org/~abartlet/
Authentication Developer, Samba Team https://samba.org
Samba Development and Support, Catalyst IT
https://catalyst.net.nz/services/samba

Lukas Slebodnik via samba-technical

2017-09-21 20:38:29 UTC

Post by Andrew Bartlett via samba-technical
Thanks for sending in the new patch set.
My concern is that by having the same python ABI file in samba, it will
#if PY_MAJOR_VERSION < 3
static void py_cobject_talloc_free(void *ptr)
{
        talloc_free(ptr);
}
_PUBLIC_ PyObject *pytalloc_CObject_FromTallocPtr(void *ptr)
{
        if (ptr == NULL) {
                Py_RETURN_NONE;
        }
        return PyCObject_FromVoidPtr(ptr, py_cobject_talloc_free);
}
#endif
from pytalloc-util.c
If you build with python 2 that will not be included in the ABI, and if
you build with python3 it will.
That is why we need distinct ABI files for python 2 and python 3.

I do not have a problem with distinct ABI files for python 2 and python 3.
Initial version of patches didn't contain it. Files were merged
per requests in previous mails. Or did I misunderstand something?

LS

Lukas Slebodnik via samba-technical

2017-09-21 22:07:34 UTC

Actually,
I cannot see any problem with current patches and
pytalloc_CObject_FromTallocPtr

sh# objdump -T /usr/lib64/libpytalloc-util.so.2.1.10 | grep CObject
0000000000000000 DF *UND* 0000000000000000 PyCObject_FromVoidPtr
0000000000001820 g DF .text 0000000000000024 PYTALLOC_UTIL_2.0.6 pytalloc_CObject_FromTallocPtr
sh# objdump -T /usr/lib64/libpytalloc-util-cp36m.so.2.1.10 | grep CObject
sh# echo $?
1

Andrew,
Could you provide me steps to reproduce your case?

LS

Andrew Bartlett via samba-technical

2017-09-22 20:16:37 UTC

Actually,
I cannot see any problem with current patches and
pytalloc_CObject_FromTallocPtr
sh# objdump -T /usr/lib64/libpytalloc-util.so.2.1.10 | grep CObject
0000000000000000 DF *UND* 0000000000000000 PyCObject_FromVoidPtr
0000000000001820 g DF .text 0000000000000024 PYTALLOC_UTIL_2.0.6
pytalloc_CObject_FromTallocPtr
sh# objdump -T /usr/lib64/libpytalloc-util-cp36m.so.2.1.10 | grep CObject
sh# echo $?
1
Andrew,
Could you provide me steps to reproduce your case?

I'm talking about the files in Samba's ABI/ subdirectories.

I misunderstood, I thought there was only going to be one ABI
file. However while there are now python3 ABI files I see two
problems:

- The name (the cp3m bit needs to become .py3 again)
lib/talloc/ABI/pytalloc-util-cp35m-2.1.10.sigs

- The vscript:
cat bin/default/lib/talloc/pytalloc-util-cp35m.vscript | grep CObject
pytalloc_CObject_FromTallocPtr;

This contains references to CObject_FromTallocPtr

This suggests to me that there is a cross-leakage between the python2
and python3 ABI stuff.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

Lukas Slebodnik via samba-technical

2017-09-22 20:49:36 UTC

Actually,
I cannot see any problem with current patches and
pytalloc_CObject_FromTallocPtr
sh# objdump -T /usr/lib64/libpytalloc-util.so.2.1.10 | grep CObject
0000000000000000 DF *UND* 0000000000000000 PyCObject_FromVoidPtr
0000000000001820 g DF .text 0000000000000024 PYTALLOC_UTIL_2.0.6
pytalloc_CObject_FromTallocPtr
sh# objdump -T /usr/lib64/libpytalloc-util-cp36m.so.2.1.10 | grep CObject
sh# echo $?
1
Andrew,
Could you provide me steps to reproduce your case?

I'm talking about the files in Samba's ABI/ subdirectories.
I misunderstood, I thought there was only going to be one ABI
file. However while there are now python3 ABI files I see two
- The name (the cp3m bit needs to become .py3 again)
lib/talloc/ABI/pytalloc-util-cp35m-2.1.10.sigs
cat bin/default/lib/talloc/pytalloc-util-cp35m.vscript | grep CObject
pytalloc_CObject_FromTallocPtr;
This contains references to CObject_FromTallocPtr

But as i showed you in previous example it is not a problem that
symbol is in version script because it is not part of util library
for python3. Version script is used as a whitelist to allow symbols from
objects *.o appear in library. But if symbol is missing in object files
it will not be part of library even though it is in version script.

https://www.akkadia.org/drepper/dsohowto.pdf
Section 2.2 says:
The concept of export maps is to tell the linker explicitly
which symbols to export from the generated object.

And it is much simpler to use version script(export maps)
then __attribute__ ((visibility ("hidden")))

But if you prefer to have separate version script(ABI/*) for python2
and python3 I do not have a problem with that. I can just see a small
benefit in de-duplication.

LS

Andrew Bartlett via samba-technical

2017-09-22 22:15:17 UTC

Post by Lukas Slebodnik via samba-technical
But as i showed you in previous example it is not a problem that
symbol is in version script because it is not part of util library
for python3. Version script is used as a whitelist to allow symbols from
objects *.o appear in library. But if symbol is missing in object files
it will not be part of library even though it is in version script.
https://www.akkadia.org/drepper/dsohowto.pdf
The concept of export maps is to tell the linker explicitly
which symbols to export from the generated object.
And it is much simpler to use version script(export maps)
then __attribute__ ((visibility ("hidden")))
But if you prefer to have separate version script(ABI/*) for python2
and python3 I do not have a problem with that. I can just see a small
benefit in de-duplication.

The ABI files are not just for the version scripts. They are also used
to determine if a symbol is removed or the signature changed, and
triggers logic to reject such a build.

If there is just one ABI file for both, then if python2 requires a
different ABI to python3, then either on the other will fail a build.

However, we don't want ABI files for each possible python3 version (35m
etc), just for python3 overall, hence the .py3 thing.

I hope this helps clarify things.

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

Lukas Slebodnik via samba-technical

2018-04-16 22:07:46 UTC

The ABI files are not just for the version scripts. They are also used
to determine if a symbol is removed or the signature changed, and
triggers logic to reject such a build.
If there is just one ABI file for both, then if python2 requires a
different ABI to python3, then either on the other will fail a build.
However, we don't want ABI files for each possible python3 version (35m
etc), just for python3 overall, hence the .py3 thing.
I hope this helps clarify things.

I would like to apologize that it took ages to send updated version of
patches. Hopefully, it would work for everyone.

Adding Timur to CC for checking it for FreeBSD

LS

Lukas Slebodnik via samba-technical

2018-04-17 07:42:07 UTC

The ABI files are not just for the version scripts. They are also used
to determine if a symbol is removed or the signature changed, and
triggers logic to reject such a build.
If there is just one ABI file for both, then if python2 requires a
different ABI to python3, then either on the other will fail a build.
However, we don't want ABI files for each possible python3 version (35m
etc), just for python3 overall, hence the .py3 thing.
I hope this helps clarify things.

I would like to apologize that it took ages to send updated version of
patches. Hopefully, it would work for everyone.

Adding Timur to CC for checking it for FreeBSD

And now with patch. :-) I had a suspicion that sending patches after midnight
is not the best idea.

LS

Timur I. Bakeyev via samba-technical

2018-04-28 22:05:12 UTC

Hi, Lukas!

Just a short update - I've tested your patch on FreeBSD, it doesn't work
OOTB. One issue is that `pyext_PATTERN` on FreeBSD is set simply to `%s.so`
for all Python versions, so the trick we use to to build EXTRA_PYTHON
version doesn't work - binaries for 3.x just simply override 2.x ones. I've
addressed that by explicitly setting:

conf.env['pyext_PATTERN']

in the EXTRA_PYTHON branch of the buildtools/wafsamba/samba_python.py. That
does work with the old patch and, kind of works with the new one, except
that `make test` is unable to load PY extension for 3.x. I'm still
investigating what is different with regard to the old patch, which was
working fine in that case. Lost development VM doesn't help to speed up
this process though :)

With best regards,
Timur.

The ABI files are not just for the version scripts. They are also used
to determine if a symbol is removed or the signature changed, and
triggers logic to reject such a build.
If there is just one ABI file for both, then if python2 requires a
different ABI to python3, then either on the other will fail a build.
However, we don't want ABI files for each possible python3 version (35m
etc), just for python3 overall, hence the .py3 thing.
I hope this helps clarify things.

I would like to apologize that it took ages to send updated version of
patches. Hopefully, it would work for everyone.
Adding Timur to CC for checking it for FreeBSD
And now with patch. :-) I had a suspicion that sending patches after midnight
is not the best idea.
LS

Lukas Slebodnik via samba-technical

2018-05-05 08:01:37 UTC