[PATCH] kiss: add multilib support

[PATCH] kiss: add multilib support

From: Cem Keylan
To: dev
Related GitHub issue: https://github.com/kiss-community/kiss/issues/20

Using readelf for fixing dependencies outputs only the soname of dynamic
libraries, which causes confusion between folders such as `/lib` and
`/lib32`.

This change ensures that we retrieve the correct path for the library
dependency by also using ldd. readelf provides the left handside of the
ldd dependency output, and we retrieve the right handside from that
information.

However, we don't want to call the dynamic loader lots of times, as that
would slow the operation. Therefore, we save ldd output in a buffer at
the beginning for each file, and we call that buffer instead when we
need to reference it.
---
 kiss | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/kiss b/kiss
index 67f37f9..881e2a5 100755
--- a/kiss
+++ b/kiss
@@ -435,6 +435,14 @@ pkg_strip() {
     done 2>/dev/null ||:
 }
 
+pkg_fix_deps_fullpath() {
+    # Return the canonical path of libraries extracted by readelf.
+    while read -r dep _ rslv _; do
+        [ "$dep" = "$1" ] || continue
+        printf '%s\n' "$rslv"
+    done
+}
+
 pkg_fix_deps() {
     # Dynamically look for missing runtime dependencies by checking each
     # binary and library with 'ldd'. This catches any extra libraries and or
@@ -451,13 +459,19 @@ pkg_fix_deps() {
     find "$pkg_dir/${PWD##*/}/" -type f 2>/dev/null |
 
     while read -r file; do
+        # We call ldd regardless here, because we also use it to retrieve the
+        # fullpath of a library when using readelf. Best use we have here is
+        # saving it in a buffer, so we don't use the dynamic loader everytime we
+        # need to reference it.
+        lddbuf=$(ldd -- "$file" 2>/dev/null)
+
         case $elf_cmd in
             *readelf)
                 "$elf_cmd" -d "$file"
             ;;
 
             *)
-                ldd -- "$file"
+                printf '%s\n' "$lddbuf"
             ;;
         esac 2>/dev/null |
 
@@ -468,6 +482,12 @@ pkg_fix_deps() {
                     line=${line##*\[}
                     line=${line%%\]*}
 
+                    # Retrieve the fullpath of the library from our ldd buffer.
+                    case $elf_cmd in
+                        *readelf) line=$(printf '%s\n' "$lddbuf" |
+                                         pkg_fix_deps_fullpath "$line")
+                    esac
+
                     # ldd:     libjson-c.so.5 => /lib/libjson-c.so.5 ...
                     line=${line##*=> }
                     line=${line%% *}
-- 
2.31.1

2 replies

Re: [PATCH] kiss: add multilib support

From: Cem Keylan
To: dev
For this patch, I have run the following command as a method of
benchmarking:

time sh -c 'for _ in $(seq 20); do kiss b make; done'

This builds `GNU make` 20 times

Without patch:

    real    3m 41.07s
    user    1m 36.35s
    sys     0m 43.59s

With patch:

    real    3m 44.64s
    user    1m 33.29s
    sys     0m 42.73s


The difference is obviously there, but it is barely noticeable. It
increased from 221 seconds to 224 seconds.

--
Best Regards,
Cem Keylan

Re: [PATCH] kiss: add multilib support

From: Dilyn Corner
Likewise tested, using bmake and hikari to see if the number of dependencies
make a substantial impact:
bmake - static, two binaries:
    Before: 3m59.08s real     4m01.65s user     4m14.00s system
    After:  4m09.42s real     4m12.14s user     4m14.52s system
 
hikari - shared, single binary:
    Before: 2m37.15s real    12m11.60s user    16m53.46s system
    After:  2m25.78s real    12m03.73s user    16m46.83s system

wlroots - static, single lib:
    Before: 1m39.71s real     5m47.44s user     0m47.51s system
    After:  1m26.33s real     5m34.76s user     0m45.45s system

So it seems... faster for things which have more dependencies. Which I'm totally
cool with. Surprisingly faster with hikari, which I wasn't expecting? (I tested
each about six times just to be sure; it was consistent).

Either way, I can swallow an extra second building gmake. And I'm very cool with
a clean ~1s speed increase for the other two...

Isn't broken, isn't bad, passes shellcheck, lgtm.


On Sun Apr 18, 2021 at 2:58 AM EDT, Cem Keylan wrote:
>
> For this patch, I have run the following command as a method of
> benchmarking:
>
> time sh -c 'for _ in $(seq 20); do kiss b make; done'
>
> This builds `GNU make` 20 times
>
> Without patch:
>
> real 3m 41.07s
> user 1m 36.35s
> sys 0m 43.59s
>
> With patch:
>
> real 3m 44.64s
> user 1m 33.29s
> sys 0m 42.73s
>
>
> The difference is obviously there, but it is barely noticeable. It
> increased from 221 seconds to 224 seconds.
>
> --
> Best Regards,
> Cem Keylan

1 reply

Re: [PATCH] kiss: add multilib support

From: Cem Keylan
To: dev
For this patch, I have run the following command as a method of
benchmarking:

time sh -c 'for _ in $(seq 20); do kiss b make; done'

This builds `GNU make` 20 times

Without patch:

    real    3m 41.07s
    user    1m 36.35s
    sys     0m 43.59s

With patch:

    real    3m 44.64s
    user    1m 33.29s
    sys     0m 42.73s


The difference is obviously there, but it is barely noticeable. It
increased from 221 seconds to 224 seconds.

--
Best Regards,
Cem Keylan

1 reply

Re: [PATCH] kiss: add multilib support

From: Cem Keylan
To: dev
For this patch, I have run the following command as a method of
benchmarking:

time sh -c 'for _ in $(seq 20); do kiss b make; done'

This builds `GNU make` 20 times

Without patch:

    real    3m 41.07s
    user    1m 36.35s
    sys     0m 43.59s

With patch:

    real    3m 44.64s
    user    1m 33.29s
    sys     0m 42.73s


The difference is obviously there, but it is barely noticeable. It
increased from 221 seconds to 224 seconds.

--
Best Regards,
Cem Keylan

1 reply

Re: [PATCH] kiss: add multilib support

From: Dilyn Corner
To: dev
On Sat Apr 17, 2021 at 2:32 PM EDT, Cem Keylan wrote:
> Related GitHub issue: https://github.com/kiss-community/kiss/issues/20
>
> Using readelf for fixing dependencies outputs only the soname of dynamic
> libraries, which causes confusion between folders such as `/lib` and
> `/lib32`.
>
> This change ensures that we retrieve the correct path for the library
> dependency by also using ldd. readelf provides the left handside of the
> ldd dependency output, and we retrieve the right handside from that
> information.
>
> However, we don't want to call the dynamic loader lots of times, as that
> would slow the operation. Therefore, we save ldd output in a buffer at
> the beginning for each file, and we call that buffer instead when we
> need to reference it.
> ---
> kiss | 22 +++++++++++++++++++++-
> 1 file changed, 21 insertions(+), 1 deletion(-)
>
> diff --git a/kiss b/kiss
> index 67f37f9..881e2a5 100755
> --- a/kiss
> +++ b/kiss
> @@ -435,6 +435,14 @@ pkg_strip() {
> done 2>/dev/null ||:
> }
>  
> +pkg_fix_deps_fullpath() {
> + # Return the canonical path of libraries extracted by readelf.
> + while read -r dep _ rslv _; do
> + [ "$dep" = "$1" ] || continue
> + printf '%s\n' "$rslv"
> + done
> +}
> +
> pkg_fix_deps() {
> # Dynamically look for missing runtime dependencies by checking each
> # binary and library with 'ldd'. This catches any extra libraries and or
> @@ -451,13 +459,19 @@ pkg_fix_deps() {
> find "$pkg_dir/${PWD##*/}/" -type f 2>/dev/null |
>  
> while read -r file; do
> + # We call ldd regardless here, because we also use it to retrieve the
> + # fullpath of a library when using readelf. Best use we have here is
> + # saving it in a buffer, so we don't use the dynamic loader everytime
> we
> + # need to reference it.
> + lddbuf=$(ldd -- "$file" 2>/dev/null)
> +
> case $elf_cmd in
> *readelf)
> "$elf_cmd" -d "$file"
> ;;
>  
> *)
> - ldd -- "$file"
> + printf '%s\n' "$lddbuf"
> ;;
> esac 2>/dev/null |
>  
> @@ -468,6 +482,12 @@ pkg_fix_deps() {
> line=${line##*\[}
> line=${line%%\]*}
>  
> + # Retrieve the fullpath of the library from our ldd buffer.
> + case $elf_cmd in
> + *readelf) line=$(printf '%s\n' "$lddbuf" |
> + pkg_fix_deps_fullpath "$line")
> + esac
> +
> # ldd: libjson-c.so.5 => /lib/libjson-c.so.5 ...
> line=${line##*=> }
> line=${line%% *}
> --
> 2.31.1

This has been merged. Thanks!

Dilyn