[dpdk-dev] [PATCH] mk: fix the combined library problems by replacing it with a linker script

[Panu Matilainen]

The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.

Since creating the linker script is practically zero cost, remove the
config option and just create it always.

Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

Suggested-by: Sergio Gonzalez Monroy <***@intel.com>
Suggested-by: Neil Horman <***@tuxdriver.com>
Signed-off-by: Panu Matilainen <***@redhat.com>
---
config/common_bsdapp | 5 ---
config/common_linuxapp | 5 ---
drivers/net/Makefile | 1 -
drivers/net/mlx4/Makefile | 6 ---
drivers/net/mlx5/Makefile | 6 ---
lib/Makefile | 1 -
mk/rte.app.mk | 10 -----
mk/rte.combinedlib.mk | 57 ++++++++++++++++++++++++++
mk/rte.lib.mk | 16 --------
mk/rte.sdkbuild.mk | 4 +-
mk/rte.sharelib.mk | 101 ----------------------------------------------
11 files changed, 59 insertions(+), 153 deletions(-)
create mode 100644 mk/rte.combinedlib.mk
delete mode 100644 mk/rte.sharelib.mk

diff --git a/config/common_bsdapp b/config/common_bsdapp
index bdf1fcd..f38cdc0 100644
--- a/config/common_bsdapp
+++ b/config/common_bsdapp
@@ -84,11 +84,6 @@ CONFIG_RTE_ARCH_STRICT_ALIGN=n
CONFIG_RTE_BUILD_SHARED_LIB=n

#
-# Combine to one single library
-#
-CONFIG_RTE_BUILD_COMBINE_LIBS=n
-
-#
# Use newest code breaking previous ABI
#
CONFIG_RTE_NEXT_ABI=y
diff --git a/config/common_linuxapp b/config/common_linuxapp
index a565153..60f5d92 100644
--- a/config/common_linuxapp
+++ b/config/common_linuxapp
@@ -84,11 +84,6 @@ CONFIG_RTE_ARCH_STRICT_ALIGN=n
CONFIG_RTE_BUILD_SHARED_LIB=n

#
-# Combine to one single library
-#
-CONFIG_RTE_BUILD_COMBINE_LIBS=n
-
-#
# Use newest code breaking previous ABI
#
CONFIG_RTE_NEXT_ABI=y
diff --git a/drivers/net/Makefile b/drivers/net/Makefile
index cddcd57..5d9c585 100644
--- a/drivers/net/Makefile
+++ b/drivers/net/Makefile
@@ -51,5 +51,4 @@ DIRS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio
DIRS-$(CONFIG_RTE_LIBRTE_VMXNET3_PMD) += vmxnet3
DIRS-$(CONFIG_RTE_LIBRTE_PMD_XENVIRT) += xenvirt

-include $(RTE_SDK)/mk/rte.sharelib.mk
include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/net/mlx4/Makefile b/drivers/net/mlx4/Makefile
index 23b766d..d2f5692 100644
--- a/drivers/net/mlx4/Makefile
+++ b/drivers/net/mlx4/Makefile
@@ -31,12 +31,6 @@

include $(RTE_SDK)/mk/rte.vars.mk

-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS)$(CONFIG_RTE_BUILD_SHARED_LIB),yy)
-all:
- @echo 'MLX4: Not supported in a combined shared library'
- @false
-endif
-
# Library name.
LIB = librte_pmd_mlx4.a

diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index ae568e6..736d205 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -31,12 +31,6 @@

include $(RTE_SDK)/mk/rte.vars.mk

-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS)$(CONFIG_RTE_BUILD_SHARED_LIB),yy)
-all:
- @echo 'MLX5: Not supported in a combined shared library'
- @false
-endif
-
# Library name.
LIB = librte_pmd_mlx5.a

diff --git a/lib/Makefile b/lib/Makefile
index 9727b83..a8fe4b2 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -62,5 +62,4 @@ DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
DIRS-$(CONFIG_RTE_LIBRTE_IVSHMEM) += librte_ivshmem
endif

-include $(RTE_SDK)/mk/rte.sharelib.mk
include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 148653e..2a8fdf7 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -59,10 +59,6 @@ _LDLIBS-y += -L$(RTE_SDK_BIN)/lib

_LDLIBS-y += --whole-archive

-_LDLIBS-$(CONFIG_RTE_BUILD_COMBINE_LIBS) += -l$(RTE_LIBNAME)
-
-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS),n)
-
_LDLIBS-$(CONFIG_RTE_LIBRTE_DISTRIBUTOR) += -lrte_distributor
_LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder

@@ -88,8 +84,6 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrt

_LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lrte_vhost

-endif # ! CONFIG_RTE_BUILD_COMBINE_LIBS
-
_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_PCAP) += -lpcap

_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_SZEDATA2) += -lsze2
@@ -114,8 +108,6 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_BNX2X_PMD) += -lz

_LDLIBS-y += --start-group

-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS),n)
-
_LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS) += -lrte_kvargs
_LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF) += -lrte_mbuf
_LDLIBS-$(CONFIG_RTE_LIBRTE_IP_FRAG) += -lrte_ip_frag
@@ -153,8 +145,6 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_NULL) += -lrte_pmd_null

endif # ! $(CONFIG_RTE_BUILD_SHARED_LIB)

-endif # ! CONFIG_RTE_BUILD_COMBINE_LIBS
-
_LDLIBS-y += $(EXECENV_LDLIBS)
_LDLIBS-y += --end-group
_LDLIBS-y += --no-whole-archive
diff --git a/mk/rte.combinedlib.mk b/mk/rte.combinedlib.mk
new file mode 100644
index 0000000..033287c
--- /dev/null
+++ b/mk/rte.combinedlib.mk
@@ -0,0 +1,57 @@
+# BSD LICENSE
+#
+# Copyright(c) 2010-2015 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+default: all
+
+ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
+EXT:=.so
+else
+EXT:=.a
+endif
+
+COMBINEDLIB := lib$(RTE_LIBNAME)$(EXT)
+
+LIBS := $(notdir $(wildcard $(RTE_OUTPUT)/lib/*$(EXT)))
+
+all: FORCE
+ $(Q)echo "GROUP ( $(LIBS) )" > $(RTE_OUTPUT)/lib/$(COMBINEDLIB)
+
+#
+# Clean all generated files
+#
+.PHONY: clean
+clean:
+ $(Q)rm -f $(RTE_OUTPUT)/lib/$(COMBINEDLIB)
+
+.PHONY: FORCE
+FORCE:
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index 06a1519..92c9d9e 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -134,14 +134,6 @@ endif
$(depfile_newer)),\
$(O_TO_S_DO))

-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS),y)
- $(if $(or \
- $(file_missing),\
- $(call cmdline_changed,$(O_TO_C_STR)),\
- $(depfile_missing),\
- $(depfile_newer)),\
- $(O_TO_C_DO))
-endif
else
$(LIB): $(OBJS-y) $(DEP_$(LIB)) FORCE
@[ -d $(dir $@) ] || mkdir -p $(dir $@)
@@ -157,14 +149,6 @@ $(LIB): $(OBJS-y) $(DEP_$(LIB)) FORCE
$(depfile_missing),\
$(depfile_newer)),\
$(O_TO_A_DO))
-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS),y)
- $(if $(or \
- $(file_missing),\
- $(call cmdline_changed,$(O_TO_C_STR)),\
- $(depfile_missing),\
- $(depfile_newer)),\
- $(O_TO_C_DO))
-endif
endif

#
diff --git a/mk/rte.sdkbuild.mk b/mk/rte.sdkbuild.mk
index 38ec7bd..319fe38 100644
--- a/mk/rte.sdkbuild.mk
+++ b/mk/rte.sdkbuild.mk
@@ -93,8 +93,8 @@ $(ROOTDIRS-y):
@[ -d $(BUILDDIR)/$@ ] || mkdir -p $(BUILDDIR)/$@
@echo "== Build $@"
$(Q)$(MAKE) S=$@ -f $(RTE_SRCDIR)/$@/Makefile -C $(BUILDDIR)/$@ all
- @if [ $@ = drivers -a $(CONFIG_RTE_BUILD_COMBINE_LIBS) = y ]; then \
- $(MAKE) -f $(RTE_SDK)/lib/Makefile sharelib; \
+ @if [ $@ = drivers ]; then \
+ $(MAKE) -f $(RTE_SDK)/mk/rte.combinedlib.mk; \
fi

%_clean:
diff --git a/mk/rte.sharelib.mk b/mk/rte.sharelib.mk
deleted file mode 100644
index 7bb7219..0000000
--- a/mk/rte.sharelib.mk
+++ /dev/null
@@ -1,101 +0,0 @@
-# BSD LICENSE
-#
-# Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
-# All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions
-# are met:
-#
-# * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-# * Redistributions in binary form must reproduce the above copyright
-# notice, this list of conditions and the following disclaimer in
-# the documentation and/or other materials provided with the
-# distribution.
-# * Neither the name of Intel Corporation nor the names of its
-# contributors may be used to endorse or promote products derived
-# from this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-include $(RTE_SDK)/mk/internal/rte.build-pre.mk
-
-# VPATH contains at least SRCDIR
-VPATH += $(SRCDIR)
-
-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS),y)
-ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
-LIB_ONE := lib$(RTE_LIBNAME).so
-else
-LIB_ONE := lib$(RTE_LIBNAME).a
-endif
-endif
-
-.PHONY:sharelib
-sharelib: $(LIB_ONE) FORCE
-
-OBJS = $(wildcard $(RTE_OUTPUT)/build/lib/*.o)
-
-ifeq ($(LINK_USING_CC),1)
-# Override the definition of LD here, since we're linking with CC
-LD := $(CC) $(CPU_CFLAGS)
-O_TO_S = $(LD) $(call linkerprefix,$(CPU_LDFLAGS)) \
- -shared $(OBJS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
-else
-O_TO_S = $(LD) $(CPU_LDFLAGS) \
- -shared $(OBJS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
-endif
-
-O_TO_S_STR = $(subst ','\'',$(O_TO_S)) #'# fix syntax highlight
-O_TO_S_DISP = $(if $(V),"$(O_TO_S_STR)"," LD $(@)")
-O_TO_S_CMD = "cmd_$@ = $(O_TO_S_STR)"
-O_TO_S_DO = @set -e; \
- echo $(O_TO_S_DISP); \
- $(O_TO_S)
-
-O_TO_A = $(AR) crus $(RTE_OUTPUT)/lib/$(LIB_ONE) $(OBJS)
-O_TO_A_STR = $(subst ','\'',$(O_TO_A)) #'# fix syntax highlight
-O_TO_A_DISP = $(if $(V),"$(O_TO_A_STR)"," LD $(@)")
-O_TO_A_CMD = "cmd_$@ = $(O_TO_A_STR)"
-O_TO_A_DO = @set -e; \
- echo $(O_TO_A_DISP); \
- $(O_TO_A)
-#
-# Archive objects to share library
-#
-
-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS),y)
-ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
-$(LIB_ONE): FORCE
- @[ -d $(dir $@) ] || mkdir -p $(dir $@)
- $(O_TO_S_DO)
-else
-$(LIB_ONE): FORCE
- @[ -d $(dir $@) ] || mkdir -p $(dir $@)
- $(O_TO_A_DO)
-endif
-endif
-
-#
-# Clean all generated files
-#
-.PHONY: clean
-clean: _postclean
-
-.PHONY: doclean
-doclean:
- $(Q)rm -rf $(LIB_ONE)
-
-.PHONY: FORCE
-FORCE:

[Neil Horman]

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

Acked-by: Neil Horman <***@tuxdriver.com>
Nice work

[Aaron Conole]

Post by Neil Horman

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

Nice work

+1, and just for good measure:

Acked-by: Aaron Conole <***@redhat.com>

[Stephen Hemminger]

On Tue, 24 Nov 2015 16:31:17 +0200

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

But it now means distros have to ship 20 libraries which seems like
a step back.

[Panu Matilainen]

Post by Stephen Hemminger
On Tue, 24 Nov 2015 16:31:17 +0200

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

But it now means distros have to ship 20 libraries which seems like
a step back.

That's how Fedora and RHEL are shipping it already and nobody has so
much as noticed anything strange, much less complained about it. 20
libraries is but a drop in the ocean on a average distro. But more to
the point, distros will prefer 50 working libraries over one that doesn't.

The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.

- Panu -

[Neil Horman]

Post by Stephen Hemminger
On Tue, 24 Nov 2015 16:31:17 +0200

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

But it now means distros have to ship 20 libraries which seems like
a step back.

That's how Fedora and RHEL are shipping it already and nobody has so much as
noticed anything strange, much less complained about it. 20 libraries is but
a drop in the ocean on a average distro. But more to the point, distros will
prefer 50 working libraries over one that doesn't.
The combined library as it is simply is no longer a viable option. Besides
just being broken (witness the strange hacks people are coming up with to
work around issues in it) its ugly because it basically gives the middle
finger to all the effort going into version compatibility, and its also big.
Few projects will use every library in DPDK, but with the combined library
they're forced to lug the 800 pound gorilla along needlessly.

Agreed, This solves a ton of problems, and from a distro standpoint, no one
really cares how many libraries it is under the covers. Its all just one
rpm/deb package anyway.

Neil

[Stephen Hemminger]

On Wed, 25 Nov 2015 10:38:48 +0200

Post by Panu Matilainen

Post by Stephen Hemminger
On Tue, 24 Nov 2015 16:31:17 +0200

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

But it now means distros have to ship 20 libraries which seems like
a step back.

That's how Fedora and RHEL are shipping it already and nobody has so
much as noticed anything strange, much less complained about it. 20
libraries is but a drop in the ocean on a average distro. But more to
the point, distros will prefer 50 working libraries over one that doesn't.
The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.
- Panu -

Fixing the combined library took less than an hour for us.

[Panu Matilainen]

Post by Stephen Hemminger
On Wed, 25 Nov 2015 10:38:48 +0200

Post by Panu Matilainen

Post by Stephen Hemminger
On Tue, 24 Nov 2015 16:31:17 +0200

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

But it now means distros have to ship 20 libraries which seems like
a step back.

That's how Fedora and RHEL are shipping it already and nobody has so
much as noticed anything strange, much less complained about it. 20
libraries is but a drop in the ocean on a average distro. But more to
the point, distros will prefer 50 working libraries over one that doesn't.
The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.
- Panu -

Fixing the combined library took less than an hour for us.

An hour (and many more in times to come) that I bet you could've used on
something more interesting than fighting that abomination.

- Panu -

[Neil Horman]

Post by Stephen Hemminger
On Wed, 25 Nov 2015 10:38:48 +0200

Post by Panu Matilainen

Post by Stephen Hemminger
On Tue, 24 Nov 2015 16:31:17 +0200

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

But it now means distros have to ship 20 libraries which seems like
a step back.

That's how Fedora and RHEL are shipping it already and nobody has so
much as noticed anything strange, much less complained about it. 20
libraries is but a drop in the ocean on a average distro. But more to
the point, distros will prefer 50 working libraries over one that doesn't.
The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.
- Panu -

Fixing the combined library took less than an hour for us.

How did you fix the versioning issue?

Neil

[Stephen Hemminger]

On Mon, 30 Nov 2015 10:03:43 -0500

Post by Neil Horman

Post by Stephen Hemminger
On Wed, 25 Nov 2015 10:38:48 +0200

Post by Panu Matilainen

Post by Stephen Hemminger
On Tue, 24 Nov 2015 16:31:17 +0200

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

But it now means distros have to ship 20 libraries which seems like
a step back.

That's how Fedora and RHEL are shipping it already and nobody has so
much as noticed anything strange, much less complained about it. 20
libraries is but a drop in the ocean on a average distro. But more to
the point, distros will prefer 50 working libraries over one that doesn't.
The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.
- Panu -

Fixing the combined library took less than an hour for us.

How did you fix the versioning issue?
Neil

This is what I did.
Also decided to keep shared library version == major DPDK version
to avoid confusion.


mk: fix when building combined shared library

The DPDK mk file does not set shared object name or version
information as required by Debian.

Signed-off-by: Stephen Hemminger <***@networkplumber.org>

--- a/mk/rte.sharelib.mk
+++ b/mk/rte.sharelib.mk
@@ -51,10 +51,10 @@ ifeq ($(LINK_USING_CC),1)
# Override the definition of LD here, since we're linking with CC
LD := $(CC) $(CPU_CFLAGS)
O_TO_S = $(LD) $(call linkerprefix,$(CPU_LDFLAGS)) \
- -shared $(OBJS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
+ -shared $(OBJS) -Wl,-soname,$(LIB_ONE).$(RTE_LIBVERS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
else
O_TO_S = $(LD) $(CPU_LDFLAGS) \
- -shared $(OBJS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
+ -shared $(OBJS) -soname $(LIB_ONE).$(RTE_LIBVERS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
endif

O_TO_S_STR = $(subst ','\'',$(O_TO_S)) #'# fix syntax highlight
--- a/mk/rte.vars.mk
+++ b/mk/rte.vars.mk
@@ -74,8 +74,10 @@ ifneq ($(BUILDING_RTE_SDK),)
endif

RTE_LIBNAME := $(CONFIG_RTE_LIBNAME:"%"=%)
+RTE_LIBVERS := $(CONFIG_RTE_LIBVERS:"%"=%)
ifeq ($(RTE_LIBNAME),)
RTE_LIBNAME := intel_dpdk
+RTE_LIBVERS := 2
endif

# RTE_TARGET is deducted from config when we are building the SDK.

[Panu Matilainen]

Post by Stephen Hemminger
On Mon, 30 Nov 2015 10:03:43 -0500

Post by Neil Horman

Post by Stephen Hemminger
On Wed, 25 Nov 2015 10:38:48 +0200

Post by Panu Matilainen

Post by Stephen Hemminger
On Tue, 24 Nov 2015 16:31:17 +0200

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

But it now means distros have to ship 20 libraries which seems like
a step back.

That's how Fedora and RHEL are shipping it already and nobody has so
much as noticed anything strange, much less complained about it. 20
libraries is but a drop in the ocean on a average distro. But more to
the point, distros will prefer 50 working libraries over one that doesn't.
The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.
- Panu -

Fixing the combined library took less than an hour for us.

How did you fix the versioning issue?
Neil

This is what I did.
Also decided to keep shared library version == major DPDK version
to avoid confusion.
mk: fix when building combined shared library
The DPDK mk file does not set shared object name or version
information as required by Debian.
--- a/mk/rte.sharelib.mk
+++ b/mk/rte.sharelib.mk
@@ -51,10 +51,10 @@ ifeq ($(LINK_USING_CC),1)
# Override the definition of LD here, since we're linking with CC
LD := $(CC) $(CPU_CFLAGS)
O_TO_S = $(LD) $(call linkerprefix,$(CPU_LDFLAGS)) \
- -shared $(OBJS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
+ -shared $(OBJS) -Wl,-soname,$(LIB_ONE).$(RTE_LIBVERS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
else
O_TO_S = $(LD) $(CPU_LDFLAGS) \
- -shared $(OBJS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
+ -shared $(OBJS) -soname $(LIB_ONE).$(RTE_LIBVERS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
endif
O_TO_S_STR = $(subst ','\'',$(O_TO_S)) #'# fix syntax highlight
--- a/mk/rte.vars.mk
+++ b/mk/rte.vars.mk
@@ -74,8 +74,10 @@ ifneq ($(BUILDING_RTE_SDK),)
endif
RTE_LIBNAME := $(CONFIG_RTE_LIBNAME:"%"=%)
+RTE_LIBVERS := $(CONFIG_RTE_LIBVERS:"%"=%)
ifeq ($(RTE_LIBNAME),)
RTE_LIBNAME := intel_dpdk
+RTE_LIBVERS := 2
endif
# RTE_TARGET is deducted from config when we are building the SDK.

Adding a soname and a semi-arbitrary version does not fix the
fundamental problems:

Since the library lumps together everything in DPDK, you'd have to bump
its version whenever any of the individual libraries bumps its version
to have the version mean anything. DPDK 2.0 and 2.1 are supposedly
binary compatible but 2.2 certainly is not, and beyond that who knows.

That in turn forces all apps to be rebuild whenever one of the libraries
changes version, whether those apps use that particular library or not.

The combined library doesn't have symbol versioning, so besides the
better version compatibility tracking it loses other benefits like
limited symbol visibility.

Not to mention the extra complexity in makefiles to support it, the
increasing amount of duct-tape required to hold it together. And still
eg the MLX pmds declare the configuration not supported at all.

- Panu -

[Robie Basak]

Adding a soname and a semi-arbitrary version does not fix the fundamental
Since the library lumps together everything in DPDK, you'd have to bump its
version whenever any of the individual libraries bumps its version to have
the version mean anything. DPDK 2.0 and 2.1 are supposedly binary compatible
but 2.2 certainly is not, and beyond that who knows.
That in turn forces all apps to be rebuild whenever one of the libraries
changes version, whether those apps use that particular library or not.

If we bundle all the libraries together into one package, then in
distributions we have to rebuild anyway when any of the libraries
changes version since a dependent package can't just depend on any later
version, because we don't know in advance what ABI breaks might occur.
It's also trivial to do rebuilds in a distribution. I'd prefer to see
ABI versioning done right to avoid the pain that might occur there.
Rebuilding dependent packages is on the other hand straightforward.

The combined library doesn't have symbol versioning, so besides the better
version compatibility tracking it loses other benefits like limited symbol
visibility.

The combined library *should* have symbol versioning, which I've brought
up before. This isn't a reason to not have a combined library; it is a
reason to fix the combined library.

Why is limited symbol visibility a benefit in this case?

Not to mention the extra complexity in makefiles to support it, the
increasing amount of duct-tape required to hold it together. And still eg
the MLX pmds declare the configuration not supported at all.

I'd argue that this is because the build system is unnecessarily complex
currently. A library consumer should just be able to #include
<dpdk/foo.h> and link with -ldpdk. It should not have a build system or
custom flags imposed on it by one of the libraries it uses.

Robie

[Neil Horman]

Post by Robie Basak

Adding a soname and a semi-arbitrary version does not fix the fundamental
Since the library lumps together everything in DPDK, you'd have to bump its
version whenever any of the individual libraries bumps its version to have
the version mean anything. DPDK 2.0 and 2.1 are supposedly binary compatible
but 2.2 certainly is not, and beyond that who knows.
That in turn forces all apps to be rebuild whenever one of the libraries
changes version, whether those apps use that particular library or not.

If we bundle all the libraries together into one package, then in
distributions we have to rebuild anyway when any of the libraries
changes version since a dependent package can't just depend on any later
version, because we don't know in advance what ABI breaks might occur.
It's also trivial to do rebuilds in a distribution. I'd prefer to see
ABI versioning done right to avoid the pain that might occur there.
Rebuilding dependent packages is on the other hand straightforward.

The combined library doesn't have symbol versioning, so besides the better
version compatibility tracking it loses other benefits like limited symbol
visibility.

The combined library *should* have symbol versioning, which I've brought
up before. This isn't a reason to not have a combined library; it is a
reason to fix the combined library.

Agreed, but using a linker script as the combined library gets you the proper
symbol versioning.

Post by Robie Basak
Why is limited symbol visibility a benefit in this case?

Because it prevents an application from inadvertently using symbols that would
otherwise appear in another library (i.e. if not using the combined library, you
know you've used a symbol in another library because you are then forced to add
that library to the build.

Post by Robie Basak

Not to mention the extra complexity in makefiles to support it, the
increasing amount of duct-tape required to hold it together. And still eg
the MLX pmds declare the configuration not supported at all.

I'd argue that this is because the build system is unnecessarily complex
currently. A library consumer should just be able to #include
<dpdk/foo.h> and link with -ldpdk. It should not have a build system or
custom flags imposed on it by one of the libraries it uses.

I don't disagree here, but again, modeling libdpdk.so as a linker script gets
you to that point (or 99% of the way there at least).

Neil

Post by Robie Basak
Robie

[Neil Horman]

Post by Stephen Hemminger
On Mon, 30 Nov 2015 10:03:43 -0500

Post by Neil Horman

Post by Stephen Hemminger
On Wed, 25 Nov 2015 10:38:48 +0200

Post by Panu Matilainen

Post by Stephen Hemminger
On Tue, 24 Nov 2015 16:31:17 +0200

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

But it now means distros have to ship 20 libraries which seems like
a step back.

That's how Fedora and RHEL are shipping it already and nobody has so
much as noticed anything strange, much less complained about it. 20
libraries is but a drop in the ocean on a average distro. But more to
the point, distros will prefer 50 working libraries over one that doesn't.
The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.
- Panu -

Fixing the combined library took less than an hour for us.

How did you fix the versioning issue?
Neil

This is what I did.
Also decided to keep shared library version == major DPDK version
to avoid confusion.

Ok, thats the library versioning, which is great, but I was more concerned about
the symbol versioning - i.e. the collection of version linker scripts that get
applied when you build individual libraries, but completely ignored when you
build the combined library

Neil

Post by Stephen Hemminger
mk: fix when building combined shared library
The DPDK mk file does not set shared object name or version
information as required by Debian.
--- a/mk/rte.sharelib.mk
+++ b/mk/rte.sharelib.mk
@@ -51,10 +51,10 @@ ifeq ($(LINK_USING_CC),1)
# Override the definition of LD here, since we're linking with CC
LD := $(CC) $(CPU_CFLAGS)
O_TO_S = $(LD) $(call linkerprefix,$(CPU_LDFLAGS)) \
- -shared $(OBJS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
+ -shared $(OBJS) -Wl,-soname,$(LIB_ONE).$(RTE_LIBVERS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
else
O_TO_S = $(LD) $(CPU_LDFLAGS) \
- -shared $(OBJS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
+ -shared $(OBJS) -soname $(LIB_ONE).$(RTE_LIBVERS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
endif
O_TO_S_STR = $(subst ','\'',$(O_TO_S)) #'# fix syntax highlight
--- a/mk/rte.vars.mk
+++ b/mk/rte.vars.mk
@@ -74,8 +74,10 @@ ifneq ($(BUILDING_RTE_SDK),)
endif
RTE_LIBNAME := $(CONFIG_RTE_LIBNAME:"%"=%)
+RTE_LIBVERS := $(CONFIG_RTE_LIBVERS:"%"=%)
ifeq ($(RTE_LIBNAME),)
RTE_LIBNAME := intel_dpdk
+RTE_LIBVERS := 2
endif
# RTE_TARGET is deducted from config when we are building the SDK.

[Robie Basak]

Re-sending this unsigned since the ML rejected my signed email.

-1 from Ubuntu without further discussion since it will break us. Please
don't commit this patch yet.

I don't understand why we must have the complexity of so many shared
libraries. From a distribution packaging perspective, all I see is that
this multiplies potential work by twenty times and makes it awkward to
work with without special tooling (which then needs maintaining).

Before I go into details, it would be nice if someone could please
explain why DPDK has to be "special" in needing to do this? I don't
understand why DPDK must be different to every other userspace library
out there. If DPDK has a good need to be different then that's fair
enough. But I feel that if DPDK is deviating from the norm then we need
to frame the discussion from the perspective of "why DPDK must be
different", rather than having me trying to explain why the norm is the
right way to do it.

That's how Fedora and RHEL are shipping it already and nobody has so much
as noticed anything strange, much less complained about it. 20 libraries
is but a drop in the ocean on a average distro.

No, it is 20 times the work from the perspective of DPDK package
maintenance. Let me explain why.

In Debian and Ubuntu, we manage a library transition (an ABI bump in a
library together with all dependencies moving to use the new ABI) by
concurrently packaging both the old and new libraries at once. This
works well with the norm for libraries. We ship one binary package per
soname, with the major version as part of the package name. This allows
a system to have two (or more) ABIs installed simultaneously. For a
library transition, we just package the new version and then that can
land and work concurrently as we then individually update every
dependent (library-consuming) package.

This works because of conventions around sonames, which DPDK breaks
unless we treat it as twenty different libraries which changes our work
from easy to painful.

Usually a library transition is managed by hand by the package
maintainer. It's not taxing because it's straightforward and well
understood. Update and upload the new ABI source package, then find all
reverse dependencies and sort them out, recursively. But if the
maintainer must do it twenty times, then it becomes taxing and prone to
error. And if the reverse dependency tree differs depending on the split
library used by library consumers, then it gets far more complex to
follow.

Admittedly we could tool around this to make it easier, but that's extra
work (both initially and in maintenance) and prone to error (because
we'd only be doing it for DPDK).

Packaging a library is usually virtually a no-op in Debian and Ubuntu
nowadays. Our tooling does it all for us. But packaging DPDK is far from
this currently because of all this added complexity. From my perspective
this is unnecessary and makes no sense. We could do all kinds of things
to work around it (that's what packaging is about) but then we'd have to
maintain that specialness and I don't see why it must be awkward like
this instead of just doing it the same way as every other library.

The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.

It's broken because it's broken upstream, and that's what we should fix.
Why is it not viable? How does it give the middle finger to effort going
into version compatibility? Doing it the right way like every other
userspace library is what *gives us* version compatibility because then
distributions can straightforwardly install multiple ABI versions at
once.

Finally, I fail to see any "lug the 800 pound gorilla along" saving. We
(Ubuntu and Fedora) are both shipping all the libraries in one package,
whether split or combined, so they are all being lugged onto disk
anyway. Whether split or combined, there is no saving there. And memory
is hardly saved either because the kernel will just page in and out what
is needed in both cases. So how does this proposed change give us any
saving at all?

If distributions are expected to ship everything lumped together on one
package, then we don't get any benefit of having the library split up.

I did bring this up on this list[1] and my understanding of the outcome
then was that it would be fine for us to use the combined library, and
in time we could better define its ABI. Thus I'm not happy that you're
proposing to change tack on this, both because I'm far from convinced
it's a good idea for the project and wider ecosystem and also because it
creates more work for us.

I understand that in the embedded world you might want to build a subset
of the split libraries, and I'm not suggesting that we take away this
ability if there users who want it (though in that scenario I think
static builds probably make more sense). But in the distribution world,
for binaries we ship, I'd prefer to see things work the standard way
with standard tooling and nothing special going on when there is no need
for it.

Please can you explain why having a combined library is so much of a
problem?

Thanks,

Robie


[1] Message-ID: <***@mal.justgohome.co.uk>
Archive: http://dpdk.org/ml/archives/dev/2015-September/023180.html

[Neil Horman]

Post by Robie Basak
Re-sending this unsigned since the ML rejected my signed email.
-1 from Ubuntu without further discussion since it will break us. Please
don't commit this patch yet.
I don't understand why we must have the complexity of so many shared
libraries. From a distribution packaging perspective, all I see is that
this multiplies potential work by twenty times and makes it awkward to
work with without special tooling (which then needs maintaining).

Theres nothing "complex" about the simple fact that a project builds lots of
libraries. Its extreemely common. Any graphic window manager has exactly the
same situation, as do any number of tools that have multiple hardware backends
impelmented in user space (v4l, sane, iptables, to name just a few).

Post by Robie Basak
Before I go into details, it would be nice if someone could please
explain why DPDK has to be "special" in needing to do this? I don't

Its not special, see above. Not saying the build environment cant be improved,
but the fact that there are multiple libraries is pretty straightforward.

Post by Robie Basak
understand why DPDK must be different to every other userspace library
out there. If DPDK has a good need to be different then that's fair
enough. But I feel that if DPDK is deviating from the norm then we need
to frame the discussion from the perspective of "why DPDK must be
different", rather than having me trying to explain why the norm is the
right way to do it.

That's how Fedora and RHEL are shipping it already and nobody has so much
as noticed anything strange, much less complained about it. 20 libraries
is but a drop in the ocean on a average distro.

No, it is 20 times the work from the perspective of DPDK package
maintenance. Let me explain why.
In Debian and Ubuntu, we manage a library transition (an ABI bump in a
library together with all dependencies moving to use the new ABI) by
concurrently packaging both the old and new libraries at once. This
works well with the norm for libraries. We ship one binary package per
soname, with the major version as part of the package name. This allows
a system to have two (or more) ABIs installed simultaneously. For a
library transition, we just package the new version and then that can
land and work concurrently as we then individually update every
dependent (library-consuming) package.

So thats, a distribution choice, not an upstream problem. And it seems like a
problem you should have already solved (note the examples above). If you feel
like you need to package multiple ABI versions in the same library, you can,
just update the LIBABIVER of all the libraries, instead of the ones that truly
change, so that each library is guaranteed a newer so version, to make the
library file name unique. Yes you have to make a small change from upstream,
but thats part of the work that distribution maintainers do.

Post by Robie Basak
This works because of conventions around sonames, which DPDK breaks
unless we treat it as twenty different libraries which changes our work
from easy to painful.
Usually a library transition is managed by hand by the package
maintainer. It's not taxing because it's straightforward and well
understood. Update and upload the new ABI source package, then find all
reverse dependencies and sort them out, recursively. But if the
maintainer must do it twenty times, then it becomes taxing and prone to
error. And if the reverse dependency tree differs depending on the split
library used by library consumers, then it gets far more complex to
follow.
Admittedly we could tool around this to make it easier, but that's extra
work (both initially and in maintenance) and prone to error (because
we'd only be doing it for DPDK).

You must already have a solution to this, I can't imagine you package all the
libraries for kde or gnome (or even pam) separately)

Post by Robie Basak
Packaging a library is usually virtually a no-op in Debian and Ubuntu
nowadays. Our tooling does it all for us. But packaging DPDK is far from
this currently because of all this added complexity. From my perspective
this is unnecessary and makes no sense. We could do all kinds of things
to work around it (that's what packaging is about) but then we'd have to
maintain that specialness and I don't see why it must be awkward like
this instead of just doing it the same way as every other library.

The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.

It's broken because it's broken upstream, and that's what we should fix.
Why is it not viable? How does it give the middle finger to effort going
into version compatibility?

Because each individual library has a version script that gets applied during
link to version symbols properly. Those scripts dont get applied when building
the combined library.

Post by Robie Basak
Doing it the right way like every other
userspace library is what *gives us* version compatibility because then
distributions can straightforwardly install multiple ABI versions at
once.

Again, Not at all uncommon. You're packaging methodology is the issue here,
not the fact that there are multiple libraries.

Post by Robie Basak
Finally, I fail to see any "lug the 800 pound gorilla along" saving. We
(Ubuntu and Fedora) are both shipping all the libraries in one package,
whether split or combined, so they are all being lugged onto disk
anyway. Whether split or combined, there is no saving there. And memory
is hardly saved either because the kernel will just page in and out what
is needed in both cases. So how does this proposed change give us any
saving at all?

Not true, initalization constructors for PMD's at the very least mean that every
pmd will get paged in weather you want it or not using the combined library.
Individual libraries let you dynamically load them (via dlopen). I think the
same is true of several other facets of dpdk.

Neil

[Ferruh Yigit]

Post by Neil Horman

Post by Robie Basak
Re-sending this unsigned since the ML rejected my signed email.
-1 from Ubuntu without further discussion since it will break us. Please
don't commit this patch yet.
I don't understand why we must have the complexity of so many shared
libraries. From a distribution packaging perspective, all I see is that
this multiplies potential work by twenty times and makes it awkward to
work with without special tooling (which then needs maintaining).

Theres nothing "complex" about the simple fact that a project builds lots of
libraries. Its extreemely common. Any graphic window manager has exactly the
same situation, as do any number of tools that have multiple hardware backends
impelmented in user space (v4l, sane, iptables, to name just a few).

Post by Robie Basak
Before I go into details, it would be nice if someone could please
explain why DPDK has to be "special" in needing to do this? I don't

Its not special, see above. Not saying the build environment cant be improved,
but the fact that there are multiple libraries is pretty straightforward.

Post by Robie Basak
understand why DPDK must be different to every other userspace library
out there. If DPDK has a good need to be different then that's fair
enough. But I feel that if DPDK is deviating from the norm then we need
to frame the discussion from the perspective of "why DPDK must be
different", rather than having me trying to explain why the norm is the
right way to do it.

That's how Fedora and RHEL are shipping it already and nobody has so much
as noticed anything strange, much less complained about it. 20 libraries
is but a drop in the ocean on a average distro.

No, it is 20 times the work from the perspective of DPDK package
maintenance. Let me explain why.
In Debian and Ubuntu, we manage a library transition (an ABI bump in a
library together with all dependencies moving to use the new ABI) by
concurrently packaging both the old and new libraries at once. This
works well with the norm for libraries. We ship one binary package per
soname, with the major version as part of the package name. This allows
a system to have two (or more) ABIs installed simultaneously. For a
library transition, we just package the new version and then that can
land and work concurrently as we then individually update every
dependent (library-consuming) package.

So thats, a distribution choice, not an upstream problem. And it seems like a
problem you should have already solved (note the examples above). If you feel
like you need to package multiple ABI versions in the same library, you can,
just update the LIBABIVER of all the libraries, instead of the ones that truly
change, so that each library is guaranteed a newer so version, to make the
library file name unique. Yes you have to make a small change from upstream,
but thats part of the work that distribution maintainers do.

Post by Robie Basak
This works because of conventions around sonames, which DPDK breaks
unless we treat it as twenty different libraries which changes our work
from easy to painful.
Usually a library transition is managed by hand by the package
maintainer. It's not taxing because it's straightforward and well
understood. Update and upload the new ABI source package, then find all
reverse dependencies and sort them out, recursively. But if the
maintainer must do it twenty times, then it becomes taxing and prone to
error. And if the reverse dependency tree differs depending on the split
library used by library consumers, then it gets far more complex to
follow.
Admittedly we could tool around this to make it easier, but that's extra
work (both initially and in maintenance) and prone to error (because
we'd only be doing it for DPDK).

You must already have a solution to this, I can't imagine you package all the
libraries for kde or gnome (or even pam) separately)

Post by Robie Basak
Packaging a library is usually virtually a no-op in Debian and Ubuntu
nowadays. Our tooling does it all for us. But packaging DPDK is far from
this currently because of all this added complexity. From my perspective
this is unnecessary and makes no sense. We could do all kinds of things
to work around it (that's what packaging is about) but then we'd have to
maintain that specialness and I don't see why it must be awkward like
this instead of just doing it the same way as every other library.

The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.

It's broken because it's broken upstream, and that's what we should fix.
Why is it not viable? How does it give the middle finger to effort going
into version compatibility?

Because each individual library has a version script that gets applied during
link to version symbols properly. Those scripts dont get applied when building
the combined library.

Post by Robie Basak
Doing it the right way like every other
userspace library is what *gives us* version compatibility because then
distributions can straightforwardly install multiple ABI versions at
once.

Again, Not at all uncommon. You're packaging methodology is the issue here,
not the fact that there are multiple libraries.

Post by Robie Basak
Finally, I fail to see any "lug the 800 pound gorilla along" saving. We
(Ubuntu and Fedora) are both shipping all the libraries in one package,
whether split or combined, so they are all being lugged onto disk
anyway. Whether split or combined, there is no saving there. And memory
is hardly saved either because the kernel will just page in and out what
is needed in both cases. So how does this proposed change give us any
saving at all?

Not true, initalization constructors for PMD's at the very least mean that every
pmd will get paged in weather you want it or not using the combined library.
Individual libraries let you dynamically load them (via dlopen). I think the
same is true of several other facets of dpdk.
Neil

I just sent a patch that fixes versioning in combined library, can you please check it: http://dpdk.org/dev/patchwork/patch/9259/

Thanks,
ferruh

[Christian Ehrhardt]

Hi Ferruh,
while not tackling the "soname for combined lib" which I felt to be
the center of all this discussion.
I like that with your patch the symbols in the combined lib are no
more anonymous, but versioned according to the maps the DPDK sub
libraries are maintaining anyway.
Some more technical feedback on your post.

I wonder if we would find a similar solution for the overall soname version.
Unfortunately all naive approaches that came to my mind so far
(summing up LIBABIVER, biggest of LIBABIVER, ...) all have all too
obvious faults.
And I didn't have time for a complex one yet.
I feel that - in favor of non-perfect, but simple solution - it might
be easier to carry one global LIBABIVER for the combined lib that is
increased on each DPDK release IF (very likely) one of the ABIs was
changed in the Release.
Something like a LIBABISUBVER could be increased on any release that
did change, but not break the old ABI.
Well, likely still too naive - I need more time to really get into all
that - like to better understand all the benefits and drawbacks of the
linker script approach.

Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

Post by Ferruh Yigit

Post by Neil Horman

Post by Robie Basak
Re-sending this unsigned since the ML rejected my signed email.
-1 from Ubuntu without further discussion since it will break us. Please
don't commit this patch yet.
I don't understand why we must have the complexity of so many shared
libraries. From a distribution packaging perspective, all I see is that
this multiplies potential work by twenty times and makes it awkward to
work with without special tooling (which then needs maintaining).

Theres nothing "complex" about the simple fact that a project builds lots of
libraries. Its extreemely common. Any graphic window manager has exactly the
same situation, as do any number of tools that have multiple hardware backends
impelmented in user space (v4l, sane, iptables, to name just a few).

Post by Robie Basak
Before I go into details, it would be nice if someone could please
explain why DPDK has to be "special" in needing to do this? I don't

Its not special, see above. Not saying the build environment cant be improved,
but the fact that there are multiple libraries is pretty straightforward.

Post by Robie Basak
understand why DPDK must be different to every other userspace library
out there. If DPDK has a good need to be different then that's fair
enough. But I feel that if DPDK is deviating from the norm then we need
to frame the discussion from the perspective of "why DPDK must be
different", rather than having me trying to explain why the norm is the
right way to do it.

That's how Fedora and RHEL are shipping it already and nobody has so much
as noticed anything strange, much less complained about it. 20 libraries
is but a drop in the ocean on a average distro.

No, it is 20 times the work from the perspective of DPDK package
maintenance. Let me explain why.
In Debian and Ubuntu, we manage a library transition (an ABI bump in a
library together with all dependencies moving to use the new ABI) by
concurrently packaging both the old and new libraries at once. This
works well with the norm for libraries. We ship one binary package per
soname, with the major version as part of the package name. This allows
a system to have two (or more) ABIs installed simultaneously. For a
library transition, we just package the new version and then that can
land and work concurrently as we then individually update every
dependent (library-consuming) package.

So thats, a distribution choice, not an upstream problem. And it seems like a
problem you should have already solved (note the examples above). If you feel
like you need to package multiple ABI versions in the same library, you can,
just update the LIBABIVER of all the libraries, instead of the ones that truly
change, so that each library is guaranteed a newer so version, to make the
library file name unique. Yes you have to make a small change from upstream,
but thats part of the work that distribution maintainers do.

Post by Robie Basak
This works because of conventions around sonames, which DPDK breaks
unless we treat it as twenty different libraries which changes our work
from easy to painful.
Usually a library transition is managed by hand by the package
maintainer. It's not taxing because it's straightforward and well
understood. Update and upload the new ABI source package, then find all
reverse dependencies and sort them out, recursively. But if the
maintainer must do it twenty times, then it becomes taxing and prone to
error. And if the reverse dependency tree differs depending on the split
library used by library consumers, then it gets far more complex to
follow.
Admittedly we could tool around this to make it easier, but that's extra
work (both initially and in maintenance) and prone to error (because
we'd only be doing it for DPDK).

You must already have a solution to this, I can't imagine you package all the
libraries for kde or gnome (or even pam) separately)

Post by Robie Basak
Packaging a library is usually virtually a no-op in Debian and Ubuntu
nowadays. Our tooling does it all for us. But packaging DPDK is far from
this currently because of all this added complexity. From my perspective
this is unnecessary and makes no sense. We could do all kinds of things
to work around it (that's what packaging is about) but then we'd have to
maintain that specialness and I don't see why it must be awkward like
this instead of just doing it the same way as every other library.

The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.

It's broken because it's broken upstream, and that's what we should fix.
Why is it not viable? How does it give the middle finger to effort going
into version compatibility?

Because each individual library has a version script that gets applied during
link to version symbols properly. Those scripts dont get applied when building
the combined library.

Post by Robie Basak
Doing it the right way like every other
userspace library is what *gives us* version compatibility because then
distributions can straightforwardly install multiple ABI versions at
once.

Again, Not at all uncommon. You're packaging methodology is the issue here,
not the fact that there are multiple libraries.

Post by Robie Basak
Finally, I fail to see any "lug the 800 pound gorilla along" saving. We
(Ubuntu and Fedora) are both shipping all the libraries in one package,
whether split or combined, so they are all being lugged onto disk
anyway. Whether split or combined, there is no saving there. And memory
is hardly saved either because the kernel will just page in and out what
is needed in both cases. So how does this proposed change give us any
saving at all?

Not true, initalization constructors for PMD's at the very least mean that every
pmd will get paged in weather you want it or not using the combined library.
Individual libraries let you dynamically load them (via dlopen). I think the
same is true of several other facets of dpdk.
Neil

I just sent a patch that fixes versioning in combined library, can you please check it: http://dpdk.org/dev/patchwork/patch/9259/
Thanks,
ferruh

[Neil Horman]

Post by Ferruh Yigit

Post by Neil Horman

Post by Robie Basak
Re-sending this unsigned since the ML rejected my signed email.
-1 from Ubuntu without further discussion since it will break us. Please
don't commit this patch yet.
I don't understand why we must have the complexity of so many shared
libraries. From a distribution packaging perspective, all I see is that
this multiplies potential work by twenty times and makes it awkward to
work with without special tooling (which then needs maintaining).

Theres nothing "complex" about the simple fact that a project builds lots of
libraries. Its extreemely common. Any graphic window manager has exactly the
same situation, as do any number of tools that have multiple hardware backends
impelmented in user space (v4l, sane, iptables, to name just a few).

Post by Robie Basak
Before I go into details, it would be nice if someone could please
explain why DPDK has to be "special" in needing to do this? I don't

Its not special, see above. Not saying the build environment cant be improved,
but the fact that there are multiple libraries is pretty straightforward.

Post by Robie Basak
understand why DPDK must be different to every other userspace library
out there. If DPDK has a good need to be different then that's fair
enough. But I feel that if DPDK is deviating from the norm then we need
to frame the discussion from the perspective of "why DPDK must be
different", rather than having me trying to explain why the norm is the
right way to do it.

That's how Fedora and RHEL are shipping it already and nobody has so much
as noticed anything strange, much less complained about it. 20 libraries
is but a drop in the ocean on a average distro.

No, it is 20 times the work from the perspective of DPDK package
maintenance. Let me explain why.
In Debian and Ubuntu, we manage a library transition (an ABI bump in a
library together with all dependencies moving to use the new ABI) by
concurrently packaging both the old and new libraries at once. This
works well with the norm for libraries. We ship one binary package per
soname, with the major version as part of the package name. This allows
a system to have two (or more) ABIs installed simultaneously. For a
library transition, we just package the new version and then that can
land and work concurrently as we then individually update every
dependent (library-consuming) package.

So thats, a distribution choice, not an upstream problem. And it seems like a
problem you should have already solved (note the examples above). If you feel
like you need to package multiple ABI versions in the same library, you can,
just update the LIBABIVER of all the libraries, instead of the ones that truly
change, so that each library is guaranteed a newer so version, to make the
library file name unique. Yes you have to make a small change from upstream,
but thats part of the work that distribution maintainers do.

Post by Robie Basak
This works because of conventions around sonames, which DPDK breaks
unless we treat it as twenty different libraries which changes our work
from easy to painful.
Usually a library transition is managed by hand by the package
maintainer. It's not taxing because it's straightforward and well
understood. Update and upload the new ABI source package, then find all
reverse dependencies and sort them out, recursively. But if the
maintainer must do it twenty times, then it becomes taxing and prone to
error. And if the reverse dependency tree differs depending on the split
library used by library consumers, then it gets far more complex to
follow.
Admittedly we could tool around this to make it easier, but that's extra
work (both initially and in maintenance) and prone to error (because
we'd only be doing it for DPDK).

You must already have a solution to this, I can't imagine you package all the
libraries for kde or gnome (or even pam) separately)

Post by Robie Basak
Packaging a library is usually virtually a no-op in Debian and Ubuntu
nowadays. Our tooling does it all for us. But packaging DPDK is far from
this currently because of all this added complexity. From my perspective
this is unnecessary and makes no sense. We could do all kinds of things
to work around it (that's what packaging is about) but then we'd have to
maintain that specialness and I don't see why it must be awkward like
this instead of just doing it the same way as every other library.

The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.

It's broken because it's broken upstream, and that's what we should fix.
Why is it not viable? How does it give the middle finger to effort going
into version compatibility?

Because each individual library has a version script that gets applied during
link to version symbols properly. Those scripts dont get applied when building
the combined library.

Post by Robie Basak
Doing it the right way like every other
userspace library is what *gives us* version compatibility because then
distributions can straightforwardly install multiple ABI versions at
once.

Again, Not at all uncommon. You're packaging methodology is the issue here,
not the fact that there are multiple libraries.

Post by Robie Basak
Finally, I fail to see any "lug the 800 pound gorilla along" saving. We
(Ubuntu and Fedora) are both shipping all the libraries in one package,
whether split or combined, so they are all being lugged onto disk
anyway. Whether split or combined, there is no saving there. And memory
is hardly saved either because the kernel will just page in and out what
is needed in both cases. So how does this proposed change give us any
saving at all?

Not true, initalization constructors for PMD's at the very least mean that every
pmd will get paged in weather you want it or not using the combined library.
Individual libraries let you dynamically load them (via dlopen). I think the
same is true of several other facets of dpdk.
Neil

I just sent a patch that fixes versioning in combined library, can you please check it: http://dpdk.org/dev/patchwork/patch/9259/
Thanks,
ferruh

I've seen the patch, and I appreciate the effort, but it really seems to me like
more of the same. That is to say, its a good effort but it really creates
additional ifrastructure to allow a single library to be built, but the fact of
the matter is, a single library isn't needed. The build system is setup to
crate multiple libraries, and a linker scripts allows for the combined library
functionality, without adding additional clutter to the Makefiles. The argument
that its more work to support multiple libraries in some distributions simply
doesn't ring true with me, because that must be a problem which is already
solved for other popular projects which are architected in a simmilar fashion.

Regards
Neil

[Thomas Monjalon]

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

As it is a big change and discussion is not totally closed,
it is deferred to release 2.3.
The fix from Ferruh could be sufficient for 2.2.

[Christian Ehrhardt]

Hi,
FYI I kind of "gave up" (not as bad as it sounds) and started looking into
shipping it as individual libraries + linker script as well.
To me it seemed what was more accepted in all the former discussions.

It will surely cause more work for me in the short term, but I hope after
the initial hill I have to climb to make it happen it will be not too much
in future releases.

So if "we" were the only one causing this to be deferred consider it for
2.2.
That way distributions would become more similar which might help consumers
of the DPDK libraries.
In the worst case I can reverse apply it for 2.2 to get some more time to
get it to work properly for us later on.

Looking at the great changes to "make install" by Thomas being in 2.2 -
getting the linker script "official" in 2.2 as well would also help to not
get a major overhaul to packaging every version :-)

have a great week,
Christian



Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

Post by Thomas Monjalon

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

As it is a big change and discussion is not totally closed,
it is deferred to release 2.3.
The fix from Ferruh could be sufficient for 2.2.

[Martinx - ジェームズ]

Hi Christian,

You can count on me to help testing DPDK for Ubuntu, I have plans for it!

I have some experience with Debian packaging too... I'm currently
maintaining few Ubuntu PPAs, for fun... =)

Also, I have hardware available, with 10G, 40G and 100G NIC cards and
traffic generators.

I would love to help! Specially when with DPDK on Xen (plans for in
on PVM, HVM, XenServer and Amazon EC2).

Just a curiosity, I'm the designer/maintainer of "Xen LiveCD v2.0"
and I would like to build a new version of it, that will be based on
Xenial with DPDK.

http://wiki.xenproject.org/wiki/Live_CD_(Xen_3.2_%2B_Debian_5.0)

https://github.com/tmartinx/xenlivecd

Hope to see DPDK compiling with Xen on 32-bit (it is broken now), so
we can enable it on Ubuntu!

Cheers!
Thiago

On 7 December 2015 at 06:27, Christian Ehrhardt

Post by Christian Ehrhardt
Hi,
FYI I kind of "gave up" (not as bad as it sounds) and started looking into
shipping it as individual libraries + linker script as well.
To me it seemed what was more accepted in all the former discussions.
It will surely cause more work for me in the short term, but I hope after
the initial hill I have to climb to make it happen it will be not too much
in future releases.
So if "we" were the only one causing this to be deferred consider it for
2.2.
That way distributions would become more similar which might help consumers
of the DPDK libraries.
In the worst case I can reverse apply it for 2.2 to get some more time to
get it to work properly for us later on.
Looking at the great changes to "make install" by Thomas being in 2.2 -
getting the linker script "official" in 2.2 as well would also help to not
get a major overhaul to packaging every version :-)
have a great week,
Christian
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

Post by Thomas Monjalon

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

As it is a big change and discussion is not totally closed,
it is deferred to release 2.3.
The fix from Ferruh could be sufficient for 2.2.

[Robie Basak]

Post by Neil Horman
Theres nothing "complex" about the simple fact that a project builds lots of
libraries. Its extreemely common. Any graphic window manager has exactly the
same situation, as do any number of tools that have multiple hardware backends
impelmented in user space (v4l, sane, iptables, to name just a few).

Post by Robie Basak
Before I go into details, it would be nice if someone could please
explain why DPDK has to be "special" in needing to do this? I don't

Its not special, see above. Not saying the build environment cant be improved,
but the fact that there are multiple libraries is pretty straightforward.

It's fine in principle for an upstream to ship multiple shared
libraries, but it is extra and unnecessary work unless there's a
*reason* to have multiple shared libraries. What are the reasons for
DPDK?

Post by Neil Horman

Post by Robie Basak
In Debian and Ubuntu, we manage a library transition (an ABI bump in a
library together with all dependencies moving to use the new ABI) by
concurrently packaging both the old and new libraries at once. This
works well with the norm for libraries. We ship one binary package per
soname, with the major version as part of the package name. This allows
a system to have two (or more) ABIs installed simultaneously. For a
library transition, we just package the new version and then that can
land and work concurrently as we then individually update every
dependent (library-consuming) package.

So thats, a distribution choice, not an upstream problem.

No, that's how shared libraries work. By design, multiple ABI versions
can be co-installed. That's why sonames have the ABI major version
inside them and the filenames reflect the sonames.

It is a distribution choice to exploit this capability. But it is an
upstream problem if this capability is broken.

By shipping multiple shared libraries, DPDK isn't breaking this
capability per se. But if the upstream expectation is that it's no
additional work for distributions because the multiple libraries can
just be bundled together into a single distribution package, then _this_
is what breaks the capability.

Instead DPDK needs to acknowledge that splitting libraries _does_ cause
additional packaging work for any distribution that wants to use the
multiple co-installed ABI feature of shared libraries as they are
designed.

Then, it becomes for upstream a question of the trade-off: does the
benefit of split libraries outweigh the extra work this creates on
packagers? To understand this, first I need to understand the rationale
for shipping multiple shared libaries specifically in DPDK, and I feel
that you (well, Red Hat) have yet to present a case.

Post by Neil Horman
And it seems like a
problem you should have already solved (note the examples above). If you feel
like you need to package multiple ABI versions in the same library, you can,
just update the LIBABIVER of all the libraries, instead of the ones that truly
change, so that each library is guaranteed a newer so version, to make the
library file name unique. Yes you have to make a small change from upstream,
but thats part of the work that distribution maintainers do.

If it makes sense for upstream, it would be better for all if the code
was maintained in once place rather than fragmented across distribution
patches. My argument here is that _does_ make sense for upstream, which
is why I took the question to this list before we uploaded our first
patched version to Ubuntu.

Post by Neil Horman
You must already have a solution to this, I can't imagine you package all the
libraries for kde or gnome (or even pam) separately)

PAM modules are unversioned, since they are dynamically loaded plugins
and nothing actually links to them (in the sense that there are
no executables that link to them at exec time). The ABI is defined by
the version of PAM installed, not the version of the plugin. So I don't
think we can really compare to PAM.

I'm less familiar with KDE and GNOME packaging since I specialise on
server. But taking GNOME, for example, I am unable to find any binary
packages where multiple versioned shared objects have been bundled.
Their shared library packaging matches my expectations. For
example (source -> binary -> filename):

gdk-pixbuf -> libgdk-pixbuf2.0-0 -> /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so.0
gconf -> libgconf-2-4 -> /usr/lib/x86_64-linux-gnu/libgconf-2.so.4
gtk+3.0 -> libgtk-3-0 -> /usr/lib/x86_64-linux-gnu/libgtk-3.so.0
pango1.0 -> libpango-1.0-0 -> /usr/lib/x86_64-linux-gnu/libpango-1.0.so.0

Each binary package supplies no more than one soname. (Again I've ignored
unversioned pluggable modules for the same reason as PAM). If this isn't
what you mean, please can you find me a counter-example? Given a soname
you can find the binary package that provides it at
https://www.debian.org/distrib/packages under "Search the contents of
packages". I suggest you set the distribution to "testing" to find more
current sonames.

Christian points out to me that libc6 does ship multiple sonames in a
single package, but I think it's acceptable to consider this to be a
special case that DPDK cannot really look to as an example. We don't
normally co-install multiple ABI versions of libc because a major ABI
bump in libc is extremely rare, and when we do it's a very special case
that is handled as a major distribution-wide project.

In answer to "You must already have a solution to this", we do. Our
solution is to produce one binary package per soname. My point is that
in the case of DPDK, this creates extra unnecessary work. Alternatively,
we could treat DPDK packaging as the same sort of gargantuan task that
packaging GNOME and KDE are, but without a good reason to split
libraries this would be an artifical and unnecessary burden placed on
packagers by DPDK upstream, which is why I am against upstream doing
this.

Post by Neil Horman

Post by Robie Basak
Packaging a library is usually virtually a no-op in Debian and Ubuntu
nowadays. Our tooling does it all for us. But packaging DPDK is far from
this currently because of all this added complexity. From my perspective
this is unnecessary and makes no sense. We could do all kinds of things
to work around it (that's what packaging is about) but then we'd have to
maintain that specialness and I don't see why it must be awkward like
this instead of just doing it the same way as every other library.

Post by Panu Matilainen
The combined library as it is simply is no longer a viable option.
Besides just being broken (witness the strange hacks people are coming
up with to work around issues in it) its ugly because it basically gives
the middle finger to all the effort going into version compatibility,
and its also big. Few projects will use every library in DPDK, but with
the combined library they're forced to lug the 800 pound gorilla along
needlessly.

It's broken because it's broken upstream, and that's what we should fix.
Why is it not viable? How does it give the middle finger to effort going
into version compatibility?

Because each individual library has a version script that gets applied during
link to version symbols properly. Those scripts dont get applied when building
the combined library.

So this is just an upstream bug that needs resolving in the combined
library case? Then I appreciate Ferruh Yigit's efforts in fixing this
bug upstream. Thank you Ferruh Yigit.

Post by Neil Horman

Post by Robie Basak
Doing it the right way like every other
userspace library is what *gives us* version compatibility because then
distributions can straightforwardly install multiple ABI versions at
once.

Again, Not at all uncommon. You're packaging methodology is the issue here,
not the fact that there are multiple libraries.

No, our packaging methdology is sound as I hope I've explained well
enough above. The real issue is the yet-to-be-justified decision to
split libraries creating unnecessary packaging work given that we wish
to shared libraries properly rather than bundling all the sonames
together (which defeats the point of split libraries in the first
place).

Post by Neil Horman

Post by Robie Basak
Finally, I fail to see any "lug the 800 pound gorilla along" saving. We
(Ubuntu and Fedora) are both shipping all the libraries in one package,
whether split or combined, so they are all being lugged onto disk
anyway. Whether split or combined, there is no saving there. And memory
is hardly saved either because the kernel will just page in and out what
is needed in both cases. So how does this proposed change give us any
saving at all?

Not true, initalization constructors for PMD's at the very least mean that every
pmd will get paged in weather you want it or not using the combined library.
Individual libraries let you dynamically load them (via dlopen). I think the
same is true of several other facets of dpdk.

What's the objective impact of this? Can you quantify your claimed
saving? How does it compare to, say, the extra IOPS required in loading
multiple shared libraries and the extra pages that they could consume?
Are these things at all significant in an issue someone will face in the
real world?

Post by Neil Horman

Post by Robie Basak
Why is limited symbol visibility a benefit in this case?

Because it prevents an application from inadvertently using symbols that would
otherwise appear in another library (i.e. if not using the combined library, you
know you've used a symbol in another library because you are then forced to add
that library to the build.

Does Ferruh Yigit's patch address this?

Post by Neil Horman
I've seen the patch, and I appreciate the effort, but it really seems to me like
more of the same. That is to say, its a good effort but it really creates
additional ifrastructure to allow a single library to be built, but the fact of
the matter is, a single library isn't needed. The build system is setup to
crate multiple libraries, and a linker scripts allows for the combined library
functionality, without adding additional clutter to the Makefiles. The argument
that its more work to support multiple libraries in some distributions simply
doesn't ring true with me, because that must be a problem which is already
solved for other popular projects which are architected in a simmilar fashion.

I think I've rebutted all of this above. If you think there's any part
left here that I've failed to address, please let me know and I can go
into it.

Thanks,

Robie

[Neil Horman]

Post by Robie Basak

Post by Neil Horman
Theres nothing "complex" about the simple fact that a project builds lots of
libraries. Its extreemely common. Any graphic window manager has exactly the
same situation, as do any number of tools that have multiple hardware backends
impelmented in user space (v4l, sane, iptables, to name just a few).

Post by Robie Basak
Before I go into details, it would be nice if someone could please
explain why DPDK has to be "special" in needing to do this? I don't

Its not special, see above. Not saying the build environment cant be improved,
but the fact that there are multiple libraries is pretty straightforward.

It's fine in principle for an upstream to ship multiple shared
libraries, but it is extra and unnecessary work unless there's a
*reason* to have multiple shared libraries. What are the reasons for
DPDK?

Separation of functionality. There is no need to build a library that supports
10 different NICS when a given application is only using one. Likewise with
other discrete functionality, e.g. you don't link against the ACL library if you
dont' want to use ACL's.

Post by Robie Basak

Post by Neil Horman

Post by Robie Basak
In Debian and Ubuntu, we manage a library transition (an ABI bump in a
library together with all dependencies moving to use the new ABI) by
concurrently packaging both the old and new libraries at once. This
works well with the norm for libraries. We ship one binary package per
soname, with the major version as part of the package name. This allows
a system to have two (or more) ABIs installed simultaneously. For a
library transition, we just package the new version and then that can
land and work concurrently as we then individually update every
dependent (library-consuming) package.

So thats, a distribution choice, not an upstream problem.

No, that's how shared libraries work. By design, multiple ABI versions
can be co-installed. That's why sonames have the ABI major version
inside them and the filenames reflect the sonames.

We're talking about different things. The DPDK support ABI versioning in their
sonames, if you would look at the makefiles and output, you would clearly see
that. Theres no reason that multiple versions of DPDK can't be co-installed,
thats easy, the question here is weather it should support multiple discrete
libraries or only a single large library. It seems from your comments that a
single monolithic library should be the only option, and thats clearly the less
flexible one.

Post by Robie Basak
It is a distribution choice to exploit this capability. But it is an
upstream problem if this capability is broken.

Its not broken. In what way do you think soname versioning is broken in DPDK?
And in what way is it broken that the only solution is to merge all the
libraries into a single monolithic one?

Post by Robie Basak
By shipping multiple shared libraries, DPDK isn't breaking this
capability per se. But if the upstream expectation is that it's no
additional work for distributions because the multiple libraries can
just be bundled together into a single distribution package, then _this_
is what breaks the capability.
Instead DPDK needs to acknowledge that splitting libraries _does_ cause
additional packaging work for any distribution that wants to use the
multiple co-installed ABI feature of shared libraries as they are
designed.

Very well, allowing multiple separate libraries causes some extra work for
packaging. Specifically it requires that distributions carry a patch that
instantiate a specific library ABI major version number that is incremented ni
lock step for every library in a given build (which is admittedly not what
upstream currently does). I don't see that as overly hard to do, but to each
their own.

However, the solution there is to propose a patch that defines a single ABI
variable in the makefile structure that is applied to every library on a symbol
version bump. The answer is _not_ to somehow entangle that with the idea that
we have to have a single monolithic library. Their separate issues, and you
can solve the problem that you are complaining about without throwing the
proverbial baby out with the bathwater.

Post by Robie Basak
Then, it becomes for upstream a question of the trade-off: does the
benefit of split libraries outweigh the extra work this creates on
packagers? To understand this, first I need to understand the rationale
for shipping multiple shared libaries specifically in DPDK, and I feel
that you (well, Red Hat) have yet to present a case.

Some of the reasons I've mentioned above. Regardless however, the solution
your advocating to the problem you describe is orthogonal to the complaints
you're making there. If your goal is to allow multiple ABI versions in a given
package, then propose that ABI versions be incremented monolithically in the
upstream build. Even if a given library hasn't changed, it doesn't hurt to bump
the version number - that is to say, as a distribution, if an application links
against library A version 2, it will also link against library B version 2
(assuming it uses library B), even if library B has no ABI changes in it (thats
simply an artifact of how packaging works, you dont' typically mix and match
libraries from separate builds). So...just bump the soname version number, and
while you're at it, make it a global build setting, not a per library build
setting. Then you can use it to append a version number to the combined library
(script) as well

What you shouldn't do is assume that each library _has_ to have an independent
ABI version number, and use that to bootstrap an argument that the only solution
is a single combined library.

Post by Robie Basak

Post by Neil Horman
And it seems like a
problem you should have already solved (note the examples above). If you feel
like you need to package multiple ABI versions in the same library, you can,
just update the LIBABIVER of all the libraries, instead of the ones that truly
change, so that each library is guaranteed a newer so version, to make the
library file name unique. Yes you have to make a small change from upstream,
but thats part of the work that distribution maintainers do.

Yes, this makes good sense.

Post by Robie Basak
If it makes sense for upstream, it would be better for all if the code
was maintained in once place rather than fragmented across distribution
patches. My argument here is that _does_ make sense for upstream, which
is why I took the question to this list before we uploaded our first
patched version to Ubuntu.

yes, thats fine. So, it seems like perhaps we're talking about the same
solution here. If we have a single LIBABIVER variable that is applied to each
DSO, why do we then further need to only allow a single combined library?

Neil

[Thomas Monjalon]

Hi,

I'm reviving this old thread.
My understanding is that everybody prefer the linker script
than the current combined library which had neither symbol versioning
nor library dependency informations.

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.

[...]

Post by Panu Matilainen
--- /dev/null
+++ b/mk/rte.combinedlib.mk
@@ -0,0 +1,57 @@
+# BSD LICENSE
+#
+# Copyright(c) 2010-2015 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.

Why this header, Panu?
I think you should write your own copyright, and assume the linker script ;)

It needs to be rebased and some docs comments must be removed or updated.
I'll send a v2.

[Panu Matilainen]

Post by Thomas Monjalon
Hi,
I'm reviving this old thread.

Thanks.

Post by Thomas Monjalon
My understanding is that everybody prefer the linker script
than the current combined library which had neither symbol versioning
nor library dependency informations.

Yeah it seemed to me most (if not everybody) had converged on the side
of the linker script approach.

Post by Thomas Monjalon

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.

[...]

Post by Panu Matilainen
--- /dev/null
+++ b/mk/rte.combinedlib.mk
@@ -0,0 +1,57 @@
+# BSD LICENSE
+#
+# Copyright(c) 2010-2015 Intel Corporation. All rights reserved.
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+#
+# * Redistributions of source code must retain the above copyright
+# notice, this list of conditions and the following disclaimer.
+# * Redistributions in binary form must reproduce the above copyright
+# notice, this list of conditions and the following disclaimer in
+# the documentation and/or other materials provided with the
+# distribution.
+# * Neither the name of Intel Corporation nor the names of its
+# contributors may be used to endorse or promote products derived
+# from this software without specific prior written permission.

Why this header, Panu?
I think you should write your own copyright, and assume the linker script ;)

Its just inherited from the original patch by Sergio. As he's the actual
author here, it didn't seem appropriate for me to remove it.

Post by Thomas Monjalon
It needs to be rebased and some docs comments must be removed or updated.
I'll send a v2.

Thanks,

- Panu -

[Thomas Monjalon]

From: Panu Matilainen <***@redhat.com>

The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.

Since creating the linker script is practically zero cost, remove the
config option and just create it always.

Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.

Suggested-by: Sergio Gonzalez Monroy <***@intel.com>
Suggested-by: Neil Horman <***@tuxdriver.com>
Signed-off-by: Panu Matilainen <***@redhat.com>
Signed-off-by: Thomas Monjalon <***@6wind.com>
---
v2:
- move RTE_LIBNAME assignment rte.vars.mk to rte.combinedlib.mk
- update crypto
- update doc
- update rte.app.mk
- update test-build.sh

config/common_bsdapp | 5 --
config/common_linuxapp | 5 --
doc/guides/contributing/patches.rst | 12 ++-
doc/guides/nics/mlx4.rst | 5 --
doc/guides/nics/mlx5.rst | 5 --
drivers/crypto/Makefile | 3 +-
drivers/net/Makefile | 1 -
drivers/net/mlx4/Makefile | 6 --
drivers/net/mlx5/Makefile | 6 --
lib/Makefile | 1 -
mk/rte.app.mk | 17 +---
drivers/crypto/Makefile => mk/rte.combinedlib.mk | 28 +++++-
mk/rte.lib.mk | 16 ----
mk/rte.sdkbuild.mk | 4 +-
mk/rte.sharelib.mk | 105 -----------------------
mk/rte.vars.mk | 2 -
scripts/test-build.sh | 7 +-
17 files changed, 36 insertions(+), 192 deletions(-)
copy drivers/crypto/Makefile => mk/rte.combinedlib.mk (81%)
delete mode 100644 mk/rte.sharelib.mk

diff --git a/config/common_bsdapp b/config/common_bsdapp
index 696382c..7df5ac6 100644
--- a/config/common_bsdapp
+++ b/config/common_bsdapp
@@ -74,11 +74,6 @@ CONFIG_RTE_ARCH_STRICT_ALIGN=n
CONFIG_RTE_BUILD_SHARED_LIB=n

#
-# Combine to one single library
-#
-CONFIG_RTE_BUILD_COMBINE_LIBS=n
-
-#
# Use newest code breaking previous ABI
#
CONFIG_RTE_NEXT_ABI=y
diff --git a/config/common_linuxapp b/config/common_linuxapp
index f1638db..26df137 100644
--- a/config/common_linuxapp
+++ b/config/common_linuxapp
@@ -74,11 +74,6 @@ CONFIG_RTE_ARCH_STRICT_ALIGN=n
CONFIG_RTE_BUILD_SHARED_LIB=n

#
-# Combine to one single library
-#
-CONFIG_RTE_BUILD_COMBINE_LIBS=n
-
-#
# Use newest code breaking previous ABI
#
CONFIG_RTE_NEXT_ABI=y
diff --git a/doc/guides/contributing/patches.rst b/doc/guides/contributing/patches.rst
index 5dd8f79..3ebe95b 100644
--- a/doc/guides/contributing/patches.rst
+++ b/doc/guides/contributing/patches.rst
@@ -267,7 +267,7 @@ Checking Compilation
Compilation of patches and changes should be tested using the the ``test-build.sh`` script in the ``scripts``
directory of the DPDK repo::

- scripts/test-build.sh x86_64-native-linuxapp-gcc+next+shared+combined
+ scripts/test-build.sh x86_64-native-linuxapp-gcc+next+shared

The script usage is::

@@ -283,10 +283,8 @@ Where:
Examples of configs are::

x86_64-native-linuxapp-gcc
- x86_64-native-linuxapp-gcc+next+shared+combined
- x86_64-native-linuxapp-gcc+shared+next
- x86_64-native-linuxapp-clang+shared+combined
- i686-native-linuxapp-gcc+combined
+ x86_64-native-linuxapp-gcc+next+shared
+ x86_64-native-linuxapp-clang+shared

The builds can be modifies via the following environmental variables:

@@ -302,8 +300,8 @@ These can be set from the command line or in the config files shown above in the
The recommended configurations and options to test compilation prior to submitting patches are::

x86_64-native-linuxapp-gcc+shared+next
- x86_64-native-linuxapp-clang+shared+combined
- i686-native-linuxapp-gcc+combined
+ x86_64-native-linuxapp-clang+shared
+ i686-native-linuxapp-gcc

export DPDK_DEP_ZLIB=y
export DPDK_DEP_PCAP=y
diff --git a/doc/guides/nics/mlx4.rst b/doc/guides/nics/mlx4.rst
index 7757013..49f4626 100644
--- a/doc/guides/nics/mlx4.rst
+++ b/doc/guides/nics/mlx4.rst
@@ -47,11 +47,6 @@ There is also a `section dedicated to this poll mode driver
be enabled manually by setting ``CONFIG_RTE_LIBRTE_MLX4_PMD=y`` and
recompiling DPDK.

-.. warning::
-
- ``CONFIG_RTE_BUILD_COMBINE_LIBS`` with ``CONFIG_RTE_BUILD_SHARED_LIB``
- is not supported and thus the compilation will fail with this configuration.
-
Implementation details
----------------------

diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index b2a12ce..66794e6 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -48,11 +48,6 @@ There is also a `section dedicated to this poll mode driver
be enabled manually by setting ``CONFIG_RTE_LIBRTE_MLX5_PMD=y`` and
recompiling DPDK.

-.. warning::
-
- ``CONFIG_RTE_BUILD_COMBINE_LIBS`` with ``CONFIG_RTE_BUILD_SHARED_LIB``
- is not supported and thus the compilation will fail with this configuration.
-
Implementation details
----------------------

diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile
index d07ee96..d0258da 100644
--- a/drivers/crypto/Makefile
+++ b/drivers/crypto/Makefile
@@ -34,5 +34,4 @@ include $(RTE_SDK)/mk/rte.vars.mk
DIRS-$(CONFIG_RTE_LIBRTE_PMD_AESNI_MB) += aesni_mb
DIRS-$(CONFIG_RTE_LIBRTE_PMD_QAT) += qat

-include $(RTE_SDK)/mk/rte.sharelib.mk
-include $(RTE_SDK)/mk/rte.subdir.mk
\ No newline at end of file
+include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/net/Makefile b/drivers/net/Makefile
index 6e4497e..0c3393f 100644
--- a/drivers/net/Makefile
+++ b/drivers/net/Makefile
@@ -52,5 +52,4 @@ DIRS-$(CONFIG_RTE_LIBRTE_VIRTIO_PMD) += virtio
DIRS-$(CONFIG_RTE_LIBRTE_VMXNET3_PMD) += vmxnet3
DIRS-$(CONFIG_RTE_LIBRTE_PMD_XENVIRT) += xenvirt

-include $(RTE_SDK)/mk/rte.sharelib.mk
include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/drivers/net/mlx4/Makefile b/drivers/net/mlx4/Makefile
index 23b766d..d2f5692 100644
--- a/drivers/net/mlx4/Makefile
+++ b/drivers/net/mlx4/Makefile
@@ -31,12 +31,6 @@

include $(RTE_SDK)/mk/rte.vars.mk

-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS)$(CONFIG_RTE_BUILD_SHARED_LIB),yy)
-all:
- @echo 'MLX4: Not supported in a combined shared library'
- @false
-endif
-
# Library name.
LIB = librte_pmd_mlx4.a

diff --git a/drivers/net/mlx5/Makefile b/drivers/net/mlx5/Makefile
index ae568e6..736d205 100644
--- a/drivers/net/mlx5/Makefile
+++ b/drivers/net/mlx5/Makefile
@@ -31,12 +31,6 @@

include $(RTE_SDK)/mk/rte.vars.mk

-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS)$(CONFIG_RTE_BUILD_SHARED_LIB),yy)
-all:
- @echo 'MLX5: Not supported in a combined shared library'
- @false
-endif
-
# Library name.
LIB = librte_pmd_mlx5.a

diff --git a/lib/Makefile b/lib/Makefile
index ef172ea..6840f87 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -64,5 +64,4 @@ DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
DIRS-$(CONFIG_RTE_LIBRTE_IVSHMEM) += librte_ivshmem
endif

-include $(RTE_SDK)/mk/rte.sharelib.mk
include $(RTE_SDK)/mk/rte.subdir.mk
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index 8ecab41..daac09f 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -59,10 +59,6 @@ _LDLIBS-y += -L$(RTE_SDK_BIN)/lib

_LDLIBS-y += --whole-archive

-_LDLIBS-$(CONFIG_RTE_BUILD_COMBINE_LIBS) += -l$(RTE_LIBNAME)
-
-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS),n)
-
_LDLIBS-$(CONFIG_RTE_LIBRTE_DISTRIBUTOR) += -lrte_distributor
_LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder

@@ -88,8 +84,6 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrt

_LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lrte_vhost

-endif # ! CONFIG_RTE_BUILD_COMBINE_LIBS
-
ifeq ($(CONFIG_RTE_LIBRTE_VHOST_NUMA),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lnuma
endif
@@ -99,9 +93,8 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_VHOST) += -lfuse
endif

# The static libraries do not know their dependencies.
-# The combined library fails also to store this information.
-# So linking with static or combined library requires explicit dependencies.
-ifneq ($(CONFIG_RTE_BUILD_COMBINE_LIBS)$(CONFIG_RTE_BUILD_SHARED_LIB),ny)
+# So linking with static library requires explicit dependencies.
+ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),n)
_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_PCAP) += -lpcap
_LDLIBS-$(CONFIG_RTE_LIBRTE_BNX2X_PMD) += -lz
_LDLIBS-$(CONFIG_RTE_LIBRTE_MLX4_PMD) += -libverbs
@@ -111,12 +104,10 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_XENVIRT) += -lxenstore
_LDLIBS-$(CONFIG_RTE_LIBRTE_MPIPE_PMD) += -lgxio
# QAT PMD has a dependency on libcrypto (from openssl) for calculating HMAC precomputes
_LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_QAT) += -lcrypto
-endif # CONFIG_RTE_BUILD_COMBINE_LIBS or not CONFIG_RTE_BUILD_SHARED_LIBS
+endif # !CONFIG_RTE_BUILD_SHARED_LIBS

_LDLIBS-y += --start-group

-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS),n)
-
_LDLIBS-$(CONFIG_RTE_LIBRTE_KVARGS) += -lrte_kvargs
_LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF) += -lrte_mbuf
_LDLIBS-$(CONFIG_RTE_LIBRTE_MBUF_OFFLOAD) += -lrte_mbuf_offload
@@ -161,8 +152,6 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_PMD_AESNI_MB) += -L$(AESNI_MULTI_BUFFER_LIB_PATH)

endif # ! $(CONFIG_RTE_BUILD_SHARED_LIB)

-endif # ! CONFIG_RTE_BUILD_COMBINE_LIBS
-
_LDLIBS-y += $(EXECENV_LDLIBS)
_LDLIBS-y += --end-group
_LDLIBS-y += --no-whole-archive
diff --git a/drivers/crypto/Makefile b/mk/rte.combinedlib.mk
similarity index 81%
copy from drivers/crypto/Makefile
copy to mk/rte.combinedlib.mk
index d07ee96..fe4817b 100644
--- a/drivers/crypto/Makefile
+++ b/mk/rte.combinedlib.mk
@@ -31,8 +31,28 @@

include $(RTE_SDK)/mk/rte.vars.mk

-DIRS-$(CONFIG_RTE_LIBRTE_PMD_AESNI_MB) += aesni_mb
-DIRS-$(CONFIG_RTE_LIBRTE_PMD_QAT) += qat
+default: all

-include $(RTE_SDK)/mk/rte.sharelib.mk
-include $(RTE_SDK)/mk/rte.subdir.mk
\ No newline at end of file
+ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
+EXT:=.so
+else
+EXT:=.a
+endif
+
+RTE_LIBNAME := dpdk
+COMBINEDLIB := lib$(RTE_LIBNAME)$(EXT)
+
+LIBS := $(notdir $(wildcard $(RTE_OUTPUT)/lib/*$(EXT)))
+
+all: FORCE
+ $(Q)echo "GROUP ( $(LIBS) )" > $(RTE_OUTPUT)/lib/$(COMBINEDLIB)
+
+#
+# Clean all generated files
+#
+.PHONY: clean
+clean:
+ $(Q)rm -f $(RTE_OUTPUT)/lib/$(COMBINEDLIB)
+
+.PHONY: FORCE
+FORCE:
diff --git a/mk/rte.lib.mk b/mk/rte.lib.mk
index 24c81e7..8f7e021 100644
--- a/mk/rte.lib.mk
+++ b/mk/rte.lib.mk
@@ -138,14 +138,6 @@ endif
$(depfile_newer)),\
$(O_TO_S_DO))

-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS)$(EXTLIB_BUILD),yn)
- $(if $(or \
- $(file_missing),\
- $(call cmdline_changed,$(O_TO_C_STR)),\
- $(depfile_missing),\
- $(depfile_newer)),\
- $(O_TO_C_DO))
-endif
else
$(LIB): $(OBJS-y) $(DEP_$(LIB)) FORCE
@[ -d $(dir $@) ] || mkdir -p $(dir $@)
@@ -161,14 +153,6 @@ $(LIB): $(OBJS-y) $(DEP_$(LIB)) FORCE
$(depfile_missing),\
$(depfile_newer)),\
$(O_TO_A_DO))
-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS)$(EXTLIB_BUILD),yn)
- $(if $(or \
- $(file_missing),\
- $(call cmdline_changed,$(O_TO_C_STR)),\
- $(depfile_missing),\
- $(depfile_newer)),\
- $(O_TO_C_DO))
-endif
endif

#
diff --git a/mk/rte.sdkbuild.mk b/mk/rte.sdkbuild.mk
index 85f603c..eec5241 100644
--- a/mk/rte.sdkbuild.mk
+++ b/mk/rte.sdkbuild.mk
@@ -77,8 +77,8 @@ $(ROOTDIRS-y):
@[ -d $(BUILDDIR)/$@ ] || mkdir -p $(BUILDDIR)/$@
@echo "== Build $@"
$(Q)$(MAKE) S=$@ -f $(RTE_SRCDIR)/$@/Makefile -C $(BUILDDIR)/$@ all
- @if [ $@ = drivers -a $(CONFIG_RTE_BUILD_COMBINE_LIBS) = y ]; then \
- $(MAKE) -f $(RTE_SDK)/lib/Makefile sharelib; \
+ @if [ $@ = drivers ]; then \
+ $(MAKE) -f $(RTE_SDK)/mk/rte.combinedlib.mk; \
fi

%_clean:
diff --git a/mk/rte.sharelib.mk b/mk/rte.sharelib.mk
deleted file mode 100644
index 70c49a8..0000000
--- a/mk/rte.sharelib.mk
+++ /dev/null
@@ -1,105 +0,0 @@
-# BSD LICENSE
-#
-# Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
-# All rights reserved.
-#
-# Redistribution and use in source and binary forms, with or without
-# modification, are permitted provided that the following conditions
-# are met:
-#
-# * Redistributions of source code must retain the above copyright
-# notice, this list of conditions and the following disclaimer.
-# * Redistributions in binary form must reproduce the above copyright
-# notice, this list of conditions and the following disclaimer in
-# the documentation and/or other materials provided with the
-# distribution.
-# * Neither the name of Intel Corporation nor the names of its
-# contributors may be used to endorse or promote products derived
-# from this software without specific prior written permission.
-#
-# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
-# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
-# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
-# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
-# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
-# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
-# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
-# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
-# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
-# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-
-include $(RTE_SDK)/mk/internal/rte.build-pre.mk
-
-# VPATH contains at least SRCDIR
-VPATH += $(SRCDIR)
-
-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS),y)
-ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
-LIB_ONE := lib$(RTE_LIBNAME).so
-else
-LIB_ONE := lib$(RTE_LIBNAME).a
-endif
-COMBINED_MAP=$(BUILDDIR)/lib/libdpdk.map
-COMBINED_LDFLAGS += --version-script=$(COMBINED_MAP)
-endif
-
-.PHONY:sharelib
-sharelib: $(LIB_ONE) FORCE
-
-OBJS = $(wildcard $(RTE_OUTPUT)/build/lib/*.o)
-
-ifeq ($(LINK_USING_CC),1)
-# Override the definition of LD here, since we're linking with CC
-LD := $(CC) $(CPU_CFLAGS)
-O_TO_S = $(LD) $(call linkerprefix,$(CPU_LDFLAGS)) \
- $(call linkerprefix,$(COMBINED_LDFLAGS)) \
- -shared $(OBJS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
-else
-O_TO_S = $(LD) $(CPU_LDFLAGS) $(COMBINED_LDFLAGS) \
- -shared $(OBJS) -o $(RTE_OUTPUT)/lib/$(LIB_ONE)
-endif
-
-O_TO_S_STR = $(subst ','\'',$(O_TO_S)) #'# fix syntax highlight
-O_TO_S_DISP = $(if $(V),"$(O_TO_S_STR)"," LD $(@)")
-O_TO_S_CMD = "cmd_$@ = $(O_TO_S_STR)"
-O_TO_S_DO = @set -e; \
- echo $(O_TO_S_DISP); \
- $(O_TO_S)
-
-O_TO_A = $(AR) crus $(RTE_OUTPUT)/lib/$(LIB_ONE) $(OBJS)
-O_TO_A_STR = $(subst ','\'',$(O_TO_A)) #'# fix syntax highlight
-O_TO_A_DISP = $(if $(V),"$(O_TO_A_STR)"," LD $(@)")
-O_TO_A_CMD = "cmd_$@ = $(O_TO_A_STR)"
-O_TO_A_DO = @set -e; \
- echo $(O_TO_A_DISP); \
- $(O_TO_A)
-#
-# Archive objects to share library
-#
-
-ifeq ($(CONFIG_RTE_BUILD_COMBINE_LIBS),y)
-ifeq ($(CONFIG_RTE_BUILD_SHARED_LIB),y)
-$(LIB_ONE): FORCE
- @[ -d $(dir $@) ] || mkdir -p $(dir $@)
- @$(SRCDIR)/scripts/merge-maps.sh > $(COMBINED_MAP)
- $(O_TO_S_DO)
-else
-$(LIB_ONE): FORCE
- @[ -d $(dir $@) ] || mkdir -p $(dir $@)
- $(O_TO_A_DO)
-endif
-endif
-
-#
-# Clean all generated files
-#
-.PHONY: clean
-clean: _postclean
-
-.PHONY: doclean
-doclean:
- $(Q)rm -rf $(LIB_ONE)
-
-.PHONY: FORCE
-FORCE:
diff --git a/mk/rte.vars.mk b/mk/rte.vars.mk
index 7e7ee14..2d734bd 100644
--- a/mk/rte.vars.mk
+++ b/mk/rte.vars.mk
@@ -66,8 +66,6 @@ endif

RTE_TARGET ?= $(RTE_ARCH)-$(RTE_MACHINE)-$(RTE_EXEC_ENV)-$(RTE_TOOLCHAIN)

-RTE_LIBNAME := dpdk
-
ifeq ($(BUILDING_RTE_SDK),)
# if we are building an external app/lib, include internal/rte.extvars.mk that will
# define RTE_OUTPUT, RTE_SRCDIR, RTE_EXTMK, RTE_SDK_BIN, (etc ...)
diff --git a/scripts/test-build.sh b/scripts/test-build.sh
index 6d28c5d..ead7b47 100755
--- a/scripts/test-build.sh
+++ b/scripts/test-build.sh
@@ -55,7 +55,7 @@ print_help () {
-s short test with only first config without examples/doc

config: defconfig name followed by switches delimited with "+" sign
- Example: x86_64-native-linuxapp-gcc+next+shared+combined
+ Example: x86_64-native-linuxapp-gcc+next+shared
Default is to enable most of the options.
The external dependencies are setup with DPDK_DEP_* variables.
END_OF_HELP
@@ -101,8 +101,6 @@ config () # <directory> <target> <options>
sed -ri 's,(NEXT_ABI=)y,\1n,' $1/.config
! echo $3 | grep -q shared || \
sed -ri 's,(SHARED_LIB=)n,\1y,' $1/.config
- ! echo $3 | grep -q combined || \
- sed -ri 's,(COMBINE_LIBS=)n,\1y,' $1/.config
echo $2 | grep -q '^i686' || \
sed -ri 's,(NUMA=)n,\1y,' $1/.config
sed -ri 's,(PCI_CONFIG=)n,\1y,' $1/.config
@@ -110,7 +108,6 @@ config () # <directory> <target> <options>
sed -ri 's,(BYPASS=)n,\1y,' $1/.config
test "$DPDK_DEP_MOFED" != y || \
echo $2 | grep -q '^clang$' || \
- echo $3 | grep -q 'shared.*combined' || \
sed -ri 's,(MLX._PMD=)n,\1y,' $1/.config
test "$DPDK_DEP_SZE" != y || \
echo $2 | grep -q '^i686' || \
@@ -122,11 +119,9 @@ config () # <directory> <target> <options>
sed -ri 's,(PCAP=)n,\1y,' $1/.config
test -z "$AESNI_MULTI_BUFFER_LIB_PATH" || \
echo $2 | grep -q '^i686' || \
- echo $3 | grep -q 'shared.*combined' || \
sed -ri 's,(PMD_AESNI_MB=)n,\1y,' $1/.config
test "$DPDK_DEP_SSL" != y || \
echo $2 | grep -q '^i686' || \
- echo $3 | grep -q 'shared.*combined' || \
sed -ri 's,(PMD_QAT=)n,\1y,' $1/.config
sed -ri 's,(KNI_VHOST.*=)n,\1y,' $1/.config
sed -ri 's,(SCHED_.*=)n,\1y,' $1/.config

[Thomas Monjalon]

ping
I would like to be sure nothing is forgotten in this new revision.

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.
---
- move RTE_LIBNAME assignment rte.vars.mk to rte.combinedlib.mk
- update crypto
- update doc
- update rte.app.mk
- update test-build.sh

[Panu Matilainen]

Post by Thomas Monjalon
ping
I would like to be sure nothing is forgotten in this new revision.

Sorry, didn't realize you were waiting for input from me, it feels a bit
strange to comment on something supposedly coming from myself :)

Post by Thomas Monjalon

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.
---
- move RTE_LIBNAME assignment rte.vars.mk to rte.combinedlib.mk
- update crypto
- update doc
- update rte.app.mk
- update test-build.sh

Briefly tested, gets generated and installed as it should etc - looks
good to me.

- Panu -

[Panu Matilainen]

Post by Panu Matilainen

Post by Thomas Monjalon
ping
I would like to be sure nothing is forgotten in this new revision.

Sorry, didn't realize you were waiting for input from me, it feels a bit
strange to comment on something supposedly coming from myself :)

Post by Thomas Monjalon

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.
---
- move RTE_LIBNAME assignment rte.vars.mk to rte.combinedlib.mk
- update crypto
- update doc
- update rte.app.mk
- update test-build.sh

Briefly tested, gets generated and installed as it should etc - looks
good to me.

Forgot to note that the patch doesn't apply anymore because of
scripts/test-build.sh changes, so it needs a rebase. Want me to send a
v3 or will you handle it when committing?

On a related note, if this is about to go in then I'd rather have it
sooner than later because it also conflicts with the LDLIBS fixing
that's been slowly going on for months and months but been on hold
lately, partly because of this hangup.

- Panu -

[Thomas Monjalon]

Post by Panu Matilainen

Post by Panu Matilainen

Post by Thomas Monjalon
ping
I would like to be sure nothing is forgotten in this new revision.

Sorry, didn't realize you were waiting for input from me, it feels a bit
strange to comment on something supposedly coming from myself :)

Post by Thomas Monjalon

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.
---
- move RTE_LIBNAME assignment rte.vars.mk to rte.combinedlib.mk
- update crypto
- update doc
- update rte.app.mk
- update test-build.sh

Briefly tested, gets generated and installed as it should etc - looks
good to me.

Forgot to note that the patch doesn't apply anymore because of
scripts/test-build.sh changes, so it needs a rebase. Want me to send a
v3 or will you handle it when committing?
On a related note, if this is about to go in then I'd rather have it
sooner than later because it also conflicts with the LDLIBS fixing
that's been slowly going on for months and months but been on hold
lately, partly because of this hangup.

Applied, thanks

[Panu Matilainen]

Post by Thomas Monjalon

Post by Panu Matilainen

Post by Panu Matilainen

Post by Thomas Monjalon
ping
I would like to be sure nothing is forgotten in this new revision.

Sorry, didn't realize you were waiting for input from me, it feels a bit
strange to comment on something supposedly coming from myself :)

Post by Thomas Monjalon

Post by Panu Matilainen
The physically linked-together combined library has been an increasing
source of problems, as was predicted when library and symbol versioning
was introduced. Replace the complex and fragile construction with a
simple linker script which achieves the same without all the problems,
remove the related kludges from eg mlx drivers.
Since creating the linker script is practically zero cost, remove the
config option and just create it always.
Based on a patch by Sergio Gonzales Monroy, linker script approach
initially suggested by Neil Horman.
---
- move RTE_LIBNAME assignment rte.vars.mk to rte.combinedlib.mk
- update crypto
- update doc
- update rte.app.mk
- update test-build.sh

Briefly tested, gets generated and installed as it should etc - looks
good to me.

Forgot to note that the patch doesn't apply anymore because of
scripts/test-build.sh changes, so it needs a rebase. Want me to send a
v3 or will you handle it when committing?
On a related note, if this is about to go in then I'd rather have it
sooner than later because it also conflicts with the LDLIBS fixing
that's been slowly going on for months and months but been on hold
lately, partly because of this hangup.

Applied, thanks

Awesome, thank you! :)

- Panu -