[Scons-users] Setting macOS shared library versioning information with scons

Discussion:

Bill Deegan

2018-07-17 02:41:29 UTC

I added onto the macports bug.

Here's my question:

Given SHLIBVERSION = 1.2.3

What would you expect/want the arguments to -current_version and
-compatibility_version to be?

Both to SHLIBVERSION (1.2.3) or compatibility_version to 1.2?

(SCons developer here)

I'm a developer with the MacPorts package management system.
We noticed that after updating our scons port to 2.4.1, serf's shared

library was built incorrectly, in that its current version and
compatibility version were no longer being set. Here is our bug report

https://trac.macports.org/ticket/50854
It seems that scons's shared library generation code was refactored in

2.4.1, and in the process, the code that sets the -current_version and
-compatibility_version flags was removed. Was this regression intentional,
and if so, how are projects that use scons to build shared libraries on
macOS now supposed to set library version information? Should they manually
specify -current_version and -compatibility_version?

I realize scons 2.4.1 was released 2.5 years ago, but the problem

remains in scons 3.0.1. The developers of serf are preparing to release
serf 1.4.0, and I'm trying to help them ensure that they finally fix the
shared library versioning issue on their end, hopefully in a way that's
compatible with both current and future versions of scons.

Looks like (as you're pointing out), the updates to support shared

library versioning on other platforms broke mac?

Can you point me to the serf logic where they build the shared library

using SCons?

(So I can see how they're calling SharedLibrary(),etc.)
Certainly that's an indicator that however it was being used our tests

weren't covering that usage to ensure it didn't get broken.
https://svn.apache.org/repos/asf/serf/trunk/SConstruct
# This version string is used in the dynamic library name, and for Mac OS
X also
# for the compatibility_version option in the .dylib.
soversion = '%d.%d.%d' % (MAJOR, MINOR, 0)
libversion = '%d.%d.%d' % (MAJOR, MINOR, PATCH)
env['SHLIBVERSION'] = soversion
install_shared = env.InstallVersionedLib(libdir, lib_shared)
- InstallVersionedLib() doesn't use SHLIBVERSION anymore.
So the change sounds deliberate. But it doesn't explain why.
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Andrew C. Morrow

2018-10-10 19:01:00 UTC

Permalink

I don't think you can draw that conclusion. It depends on the semantics of
the versioning scheme for the library and any broader piece of software of
which the library may be a subcomponent. In particular, you could have a
project that has reached version 8.1.3. Within it, you have a library
libfoo that is on its second ABI revision. In the change from 8.1.2 to
8.1.3 a new function was for the first time added (not a breaking change)
to the v2 ABI of libfoo. I'd expect the following on linux and darwin,
respectively:

SHLIBVERSION=8.1.3
SONAME=libfoo.so.2
libfoo.so -> libfoo.so.2 -> libfoo.so.8.1.3

SHLIBVERSION=8.1.3
SONAME=libfoo.2.dylib
-current_version=1.1 -compatibility_version=1.0
libfoo.dylib -> libfoo.2.dylib -> libfoo.8.1.3.dylib

The important thing is that the -current_version and -compatibility_version
are meant to track only minor ABI changes, not major. Major's go in the
SONAME/install_name. I was also surprised to note that the
InstallVersionedLib stuff simply seems not to work at all on darwin
platforms?

For a simpler package, with just one library, where the major version *is*
the ABI and we are following semver only for the ABI, then sure, maybe it
is reasonable to decompose SHLIBVERSION=1.2.3 as using 1 for SONAME, and
then 2.3 as -current-version and 2.0 as -compatibility_version.

But the point is I don't think you can assume either way.

Post by Bill Deegan
I added onto the macports bug.
Given SHLIBVERSION = 1.2.3
What would you expect/want the arguments to -current_version and
-compatibility_version to be?
Both to SHLIBVERSION (1.2.3) or compatibility_version to 1.2?
(SCons developer here)

I'm a developer with the MacPorts package management system.
We noticed that after updating our scons port to 2.4.1, serf's shared

library was built incorrectly, in that its current version and
compatibility version were no longer being set. Here is our bug report

https://trac.macports.org/ticket/50854
It seems that scons's shared library generation code was refactored in

I realize scons 2.4.1 was released 2.5 years ago, but the problem

Looks like (as you're pointing out), the updates to support shared

library versioning on other platforms broke mac?

Can you point me to the serf logic where they build the shared library

using SCons?

(So I can see how they're calling SharedLibrary(),etc.)
Certainly that's an indicator that however it was being used our tests

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Bill Deegan

2018-10-10 21:01:51 UTC

Permalink

SO a
SOLIBVERSION=2.3
SOLIBCOMPATVERSION= defaults to say 2.0 in this case, but user can set?

-Bill

Post by Andrew C. Morrow
I don't think you can draw that conclusion. It depends on the semantics of
the versioning scheme for the library and any broader piece of software of
which the library may be a subcomponent. In particular, you could have a
project that has reached version 8.1.3. Within it, you have a library
libfoo that is on its second ABI revision. In the change from 8.1.2 to
8.1.3 a new function was for the first time added (not a breaking change)
to the v2 ABI of libfoo. I'd expect the following on linux and darwin,
SHLIBVERSION=8.1.3
SONAME=libfoo.so.2
libfoo.so -> libfoo.so.2 -> libfoo.so.8.1.3
SHLIBVERSION=8.1.3
SONAME=libfoo.2.dylib
-current_version=1.1 -compatibility_version=1.0
libfoo.dylib -> libfoo.2.dylib -> libfoo.8.1.3.dylib
The important thing is that the -current_version and
-compatibility_version are meant to track only minor ABI changes, not
major. Major's go in the SONAME/install_name. I was also surprised to note
that the InstallVersionedLib stuff simply seems not to work at all on
darwin platforms?
For a simpler package, with just one library, where the major version *is*
the ABI and we are following semver only for the ABI, then sure, maybe it
is reasonable to decompose SHLIBVERSION=1.2.3 as using 1 for SONAME, and
then 2.3 as -current-version and 2.0 as -compatibility_version.
But the point is I don't think you can assume either way.

I'm a developer with the MacPorts package management system.
We noticed that after updating our scons port to 2.4.1, serf's shared

library was built incorrectly, in that its current version and
compatibility version were no longer being set. Here is our bug report

https://trac.macports.org/ticket/50854
It seems that scons's shared library generation code was refactored

in 2.4.1, and in the process, the code that sets the -current_version and
-compatibility_version flags was removed. Was this regression intentional,
and if so, how are projects that use scons to build shared libraries on
macOS now supposed to set library version information? Should they manually
specify -current_version and -compatibility_version?

I realize scons 2.4.1 was released 2.5 years ago, but the problem

Looks like (as you're pointing out), the updates to support shared

library versioning on other platforms broke mac?

Can you point me to the serf logic where they build the shared library

using SCons?

(So I can see how they're calling SharedLibrary(),etc.)
Certainly that's an indicator that however it was being used our tests

weren't covering that usage to ensure it didn't get broken.
https://svn.apache.org/repos/asf/serf/trunk/SConstruct
# This version string is used in the dynamic library name, and for Mac
OS X also
# for the compatibility_version option in the .dylib.
soversion = '%d.%d.%d' % (MAJOR, MINOR, 0)
libversion = '%d.%d.%d' % (MAJOR, MINOR, PATCH)
env['SHLIBVERSION'] = soversion
install_shared = env.InstallVersionedLib(libdir, lib_shared)
- InstallVersionedLib() doesn't use SHLIBVERSION anymore.
So the change sounds deliberate. But it doesn't explain why.
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Andrew C. Morrow

2018-10-11 21:54:02 UTC

Permalink

I think SOLIBVERSION=x.y.z should imply SOLIBCURRENTVERSION=y.z and
SOLIBCOMPATVERSION=y.0, with options to override. The 'x' part of the
version should be in the dylib filename and install_name. The
current_version and compatibility_version appear to be "minor ABI".

I think SOLIBVERSION=x.y should imply SOLIBCURRENTVERSION=y and
SOLIBCOMPATVERSION=y, with options to override? Not sure there.

One trouble with that: values of zero appear to mean "ignore" in this
scheme. Which is awkward if your SOLIBVERSION is 2.0.0, for instance. Not
sure what to do there, as doing (y+1) (z+1) feels wrong.

Post by Bill Deegan
SO a
SOLIBVERSION=2.3
SOLIBCOMPATVERSION= defaults to say 2.0 in this case, but user can set?
-Bill
On Wed, Oct 10, 2018 at 3:01 PM Andrew C. Morrow <

Post by Andrew C. Morrow
I don't think you can draw that conclusion. It depends on the semantics
of the versioning scheme for the library and any broader piece of software
of which the library may be a subcomponent. In particular, you could have a
project that has reached version 8.1.3. Within it, you have a library
libfoo that is on its second ABI revision. In the change from 8.1.2 to
8.1.3 a new function was for the first time added (not a breaking change)
to the v2 ABI of libfoo. I'd expect the following on linux and darwin,
SHLIBVERSION=8.1.3
SONAME=libfoo.so.2
libfoo.so -> libfoo.so.2 -> libfoo.so.8.1.3
SHLIBVERSION=8.1.3
SONAME=libfoo.2.dylib
-current_version=1.1 -compatibility_version=1.0
libfoo.dylib -> libfoo.2.dylib -> libfoo.8.1.3.dylib
The important thing is that the -current_version and
-compatibility_version are meant to track only minor ABI changes, not
major. Major's go in the SONAME/install_name. I was also surprised to note
that the InstallVersionedLib stuff simply seems not to work at all on
darwin platforms?
For a simpler package, with just one library, where the major version
*is* the ABI and we are following semver only for the ABI, then sure, maybe
it is reasonable to decompose SHLIBVERSION=1.2.3 as using 1 for SONAME, and
then 2.3 as -current-version and 2.0 as -compatibility_version.
But the point is I don't think you can assume either way.

I'm a developer with the MacPorts package management system.
We noticed that after updating our scons port to 2.4.1, serf's

shared library was built incorrectly, in that its current version and
compatibility version were no longer being set. Here is our bug report

https://trac.macports.org/ticket/50854
It seems that scons's shared library generation code was refactored

I realize scons 2.4.1 was released 2.5 years ago, but the problem

Looks like (as you're pointing out), the updates to support shared

library versioning on other platforms broke mac?

Can you point me to the serf logic where they build the shared

library using SCons?

(So I can see how they're calling SharedLibrary(),etc.)
Certainly that's an indicator that however it was being used our

tests weren't covering that usage to ensure it didn't get broken.
https://svn.apache.org/repos/asf/serf/trunk/SConstruct
# This version string is used in the dynamic library name, and for Mac
OS X also
# for the compatibility_version option in the .dylib.
soversion = '%d.%d.%d' % (MAJOR, MINOR, 0)
libversion = '%d.%d.%d' % (MAJOR, MINOR, PATCH)
env['SHLIBVERSION'] = soversion
install_shared = env.InstallVersionedLib(libdir, lib_shared)
- InstallVersionedLib() doesn't use SHLIBVERSION anymore.
So the change sounds deliberate. But it doesn't explain why.
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Ryan Schmidt

2018-10-12 05:14:03 UTC

Permalink

Yes, current_version and compatibility_version are minor versions, install_name and file name contain the major version. All minor versions are converted to a three-number format (so e.g. 1 and 1.0 are converted to 1.0.0).

I think SOLIBVERSION=x.y should imply SOLIBCURRENTVERSION=y and SOLIBCOMPATVERSION=y, with options to override? Not sure there.
One trouble with that: values of zero appear to mean "ignore" in this scheme.

Zero means ignore to whom? scons?

Which is awkward if your SOLIBVERSION is 2.0.0, for instance. Not sure what to do there, as doing (y+1) (z+1) feels wrong.

Something to bear in mind maybe: The linker in old macOS versions (maybe 10.4 and earlier?) doesn't allow the current minor version or compatibility minor version to be 0. We may not care so much about macOS versions that old anymore, but libtool's handling of macOS dylibs was designed when that was still relevant, so when translating a libtool library's -version-info into macOS dylib version information, it ensures that the current and compatibility minor versions start at 1. This means that for libtool-made libraries these version numbers are 1 higher on macOS than they are on other operating systems that don't have that restriction. This causes problems virtually every time a developer switches their project from autotools to another build system like cmake that doesn't implement that same rule, because the developer inevitably makes the change while on an OS other than macOS, everything looks fine on that other OS, and they release a new version of their project which on macOS ha
s library current and compatibility minor versions that are 1 less than they were on the previous version of their software, which makes them incompatible with any software that's linked with that library until that software is rebuilt. It feels like I encounter this problem all the time in MacPorts as we update libraries.

So what I'm saying is that using y+1 might not be such a bad defalt. But also that giving the developer the opportunity to override the defaults is a good idea too, in case they have an existing library with an existing versioning scheme that they need to match.

Bill Deegan

2018-10-12 15:30:01 UTC

Permalink

From MacOS ld manpage:

Options when creating a dynamic library (dylib)

-install_name name
Sets an internal "install path" (LC_ID_DYLIB) in a
dynamic library. Any clients linked against the library
will record that path as the way dyld should locate this
library. If this option is not specified, then
the -o path will be used. This option is also called
-dylib_install_name for compatibility.
-compatibility_version number
Specifies the compatibility version number of the
library. When a library is loaded by dyld, the compati-
bility version is checked and if the program's version is
greater that the library's version, it is an
error. The format of number is X[.Y[.Z]] where X must be
a positive non-zero number less than or equal to
65535, and .Y and .Z are optional and if present must be
non-negative numbers less than or equal to 255.
If the compatibility version number is not specified, it
has a value of 0 and no checking is done when the
library is used. This option is also called
-dylib_compatibility_version for compatibility.
-current_version number
Specifies the current version number of the library. The
current version of the library can be obtained
programmatically by the user of the library so it can
determine exactly which version of the library it is
using. The format of number is X[.Y[.Z]] where X must be
a positive non-zero number less than or equal to
65535, and .Y and .Z are optional and if present must be
non-negative numbers less than or equal to 255.
If the version number is not specified, it has a value of
0. This option is also called -dylib_cur-
rent_version for compatibility.

...

-rpath path

Add path to the runpath search path list for image being
created. At runtime, dyld uses the runpath when

So it looks like.
1 === 1.0 === 1.0.0?

Also sounds like it might be good to check the legality of the version
strings [1-65535].[0-255].[0-255] being the only legal version strings?

Post by Andrew C. Morrow
I think SOLIBVERSION=x.y.z should imply SOLIBCURRENTVERSION=y.z and

SOLIBCOMPATVERSION=y.0, with options to override. The 'x' part of the
version should be in the dylib filename and install_name. The
current_version and compatibility_version appear to be "minor ABI".
Yes, current_version and compatibility_version are minor versions,
install_name and file name contain the major version. All minor versions
are converted to a three-number format (so e.g. 1 and 1.0 are converted to
1.0.0).

Post by Andrew C. Morrow
I think SOLIBVERSION=x.y should imply SOLIBCURRENTVERSION=y and

SOLIBCOMPATVERSION=y, with options to override? Not sure there.

Post by Andrew C. Morrow
One trouble with that: values of zero appear to mean "ignore" in this

scheme.
Zero means ignore to whom? scons?

Post by Andrew C. Morrow
Which is awkward if your SOLIBVERSION is 2.0.0, for instance. Not sure

what to do there, as doing (y+1) (z+1) feels wrong.
Something to bear in mind maybe: The linker in old macOS versions (maybe
10.4 and earlier?) doesn't allow the current minor version or compatibility
minor version to be 0. We may not care so much about macOS versions that
old anymore, but libtool's handling of macOS dylibs was designed when that
was still relevant, so when translating a libtool library's -version-info
into macOS dylib version information, it ensures that the current and
compatibility minor versions start at 1. This means that for libtool-made
libraries these version numbers are 1 higher on macOS than they are on
other operating systems that don't have that restriction. This causes
problems virtually every time a developer switches their project from
autotools to another build system like cmake that doesn't implement that
same rule, because the developer inevitably makes the change while on an OS
other than macOS, everything looks fine on that other OS, and they release
a new version of their project which on macOS ha
s library current and compatibility minor versions that are 1 less than
they were on the previous version of their software, which makes them
incompatible with any software that's linked with that library until that
software is rebuilt. It feels like I encounter this problem all the time in
MacPorts as we update libraries.
So what I'm saying is that using y+1 might not be such a bad defalt. But
also that giving the developer the opportunity to override the defaults is
a good idea too, in case they have an existing library with an existing
versioning scheme that they need to match.
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Paweł Tomulik

2018-10-13 09:29:07 UTC

Permalink

Hi Ryan,

[...]
- InstallVersionedLib() doesn't use SHLIBVERSION anymore.
So the change sounds deliberate. But it doesn't explain why.

AFAIR, the assumption introduced in 2.4.1 was, that InstallVersionedLib
will take the versioned file name/entry as an argument, that is in

InstallVersionedLib(libdir, lib_shared)

the lib_shared points to something, like 'libfoo.so.1.2.3'. The version
info is provided in the file suffix. This was justified by other changes
- the target file produced by linker was defined as the versioned
'libfoo.so.1.2.3', so the target node could be naturally passed to
InstallVersionedLib. The SHLIBVERSION variable found no use in the
installer, as the installer's purpose was to just copy the versioned
library file to the target directory, and eventually create/copy
symlinks libfoo.so -> libfoo.so.1.2.3, libfoo.so.1 -> libfoo.so.1.2.3.

--
Pawel Tomulik

Paweł Tomulik

2018-10-13 09:42:14 UTC

Permalink

Hello,
I'm a developer with the MacPorts package management system.
https://trac.macports.org/ticket/50854
It seems that scons's shared library generation code was refactored in 2.4.1, and in the process, the code that sets the -current_version and -compatibility_version flags was removed. Was this regression intentional, and if so, how are projects that use scons to build shared libraries on macOS now supposed to set library version information? Should they manually specify -current_version and -compatibility_version?
I realize scons 2.4.1 was released 2.5 years ago, but the problem remains in scons 3.0.1. The developers of serf are preparing to release serf 1.4.0, and I'm trying to help them ensure that they finally fix the shared library versioning issue on their end, hopefully in a way that's compatible with both current and future versions of scons.
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Perhaps this comment (and surrounding comments) could be relevant:

https://bitbucket.org/scons/scons/pull-requests/247/new-versioned-libraries-gnulink-cyglink/diff#comment-10067665

--
Pawel Tomulik

Bill Deegan

2018-10-13 18:07:32 UTC

Permalink

That's true, but macOS's 'shared library" is different than linux (as are
DLL's).
Versioning is embedded in the libraries via the flags discussed in the
thread.
As such, the .1 and .1.2.3 may not be directly mappable to macOS's
versioned library usage.

Hello,
I'm a developer with the MacPorts package management system.
We noticed that after updating our scons port to 2.4.1, serf's shared

library was built incorrectly, in that its current version and
compatibility version were no longer being set. Here is our bug report

https://trac.macports.org/ticket/50854
It seems that scons's shared library generation code was refactored in

I realize scons 2.4.1 was released 2.5 years ago, but the problem

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

https://bitbucket.org/scons/scons/pull-requests/247/new-versioned-libraries-gnulink-cyglink/diff#comment-10067665
--
Pawel Tomulik
_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users

Andrew C. Morrow

2018-10-14 13:50:09 UTC

Permalink

My recommendation would be to not attempt to automatically provide values
for -compatibility_version and -current_version, but to provide variables
that allow users to set them. If you did want something that mapped
SOVERSION=x.y.z to -current_version=y.z -compat_version=y.0 or something
(or the +1 variants of that), I think you could do it with generator
functions that would parse SOVERSION.

# No flags passed for -current_version or -compat_version
env.SharedLibrary(..., SOVERSION=1.2.3, ...)

# User sets explicitly, links with -Wl,-current_version=7.1
-Wl,-compatibility_version=7.0
env.SharedLibrary(..., SOVERSION=1.2.3, APPLELINK_CURRENT_VERSION=7.1,
APPLELINK_COMPATIBILITY_VERSION=7.0)

# User programmatically describes how SOVERSION maps to values for
compat/current.

def currentVersionFromSoVersion(source, target, env, for_signature):
return env['SOVERSION'].split(.)[1:]

def compatVersionFromSoVersion(source, target, env, for_signature):
return env['SOVERSION'].split(.)[1] + ".0"

env['APPLELINK_CURRENT_VERSION'] = currentVersionFromSoVersion
env['APPLELINK_COMPATIBILITY_VERSION'] = compatVersionFromSoVersion

# Links as -Wl,current_version=2.3 -Wl,compat_version=2.0
env.SharedLibrary(..., SOVERSION=1.2.3, ...)

Something like that? Would currentVersionFromSoVersion be able to see the
SOVERSION set in the following:

env.SharedLibrary(
SOVERSION=1.2.3,
APPLELINK_CURRENT_VERSION=currentVersionFromSoVersion
)

You could also add some validation for the rules for current_version and
compatibility_version in whatever piece of applelink.py ends up doing the
expansion of APPLELINK_CURRENT_VERSION and APPLELINK_COMPATIBILITY_VERSION.

Post by Bill Deegan
That's true, but macOS's 'shared library" is different than linux (as are
DLL's).
Versioning is embedded in the libraries via the flags discussed in the
thread.
As such, the .1 and .1.2.3 may not be directly mappable to macOS's
versioned library usage.

Hello,
I'm a developer with the MacPorts package management system.
We noticed that after updating our scons port to 2.4.1, serf's shared

library was built incorrectly, in that its current version and
compatibility version were no longer being set. Here is our bug report

https://trac.macports.org/ticket/50854
It seems that scons's shared library generation code was refactored in

I realize scons 2.4.1 was released 2.5 years ago, but the problem

_______________________________________________
Scons-users mailing list
https://pairlist4.pair.net/mailman/listinfo/scons-users