Fun With Symbol Collision

One of the basic rules of C is that you don’t get to use the same function name twice.

If I write a file with some function in it, say, lib1a.c:

void my_awesome_function(void)
{
    printf("This is my awesome function!\n");
}

And then I write another file lib1b.c with a different implementation of the same function:

void my_awesome_function(void)
{
    printf("This is my ENTIRELY DIFFERENT awesome function!\n");
}

Then if I write a simple main program to call that function…

#include "lib1.h"

int main(int argc, char **argv)
{
    my_awesome_function();
    return 0;
}

It won’t know which one I mean, and when I try to build the program I will get an error message.

/usr/bin/ld: /tmp/cc30kjUg.o: in function `my_awesome_function':
lib1b.c:(.text+0x0): multiple definition of `my_awesome_function'; /tmp/cce04mSV.o:lib1a.c:(.text+0x0): first defined here
collect2: error: ld returned 1 exit status

But this reply reveals that I wasn’t quite accurate in my first sentence there. This is not a basic rule of C. It’s a basic rule of C’s linker. That linker is shared with many other languages, so similar rules end up applying to assembly language as well. Languages like C++ work around this by having the function names in the program not quite match up with the function names that the linker gets to see: C++ is said to “mangle” its symbols on the way to the linker.

But even without that, we have a little leeway. The linker still isn’t as strict as that. It only requires that each specific linker product not export the same symbol twice. This means that C can define the same function multiple times as long as it uses the static directive to constrain it to a single file. The C compiler entirely conceals these functions from the linker, so it never gets a chance to notice the duplication.

Linking the Same Symbol Twice, on Purpose

We can be a little crazier, too. After all, a full program is usually made of many individual linker products. We can create two dynamic libraries that define the same symbol, and we will be permitted to link one program against both libraries simultaneously by Mac, Linux, and Windows.

What happens in these cases is, notionally, system-dependent, but when you build with GCC or Clang, the library we list first when we name our libraries is the one that gets priority in the command-line build.

Linking the Same Symbol Twice, by Accident

We’re pretty unlikely to actually run into the situation we described above in a real system, though. Here’s a more insidious example. Let’s keep our lib1a and lib1b libraries as we had them—but now let’s create lib2a and lib2b libraries to go with them. Here is lib2a.c:

#include "lib1.h"

void function1(void)
{
    my_awesome_function();
}

And here is lib2b.c:

#include "lib1.h"

void function2(void)
{
    my_awesome_function();
}

If we then link lib2a against lib1a and lib2b against lib1b, each of these libraries looks nicely independent; neither exports any symbols that conflict with anybody else, and if we have a main program that looks like this:

#include "lib2.h"

int main(int argc, char **argv)
{
    function1();
    function2();
    return 0;
}

We won’t even list anything other than the apparently-non-conflicting 2a and 2b libraries. We would expect, when we ran this, to get output that looks like this:

This is my awesome function!
This is my ENTIRELY DIFFERENT awesome function!

And, indeed, on Windows and Mac, we do. But on Linux, we don’t; we get two copies of the same sentence, and which sentence we get depends on whether we linked 2a in first or 2b. What gives?

One- vs. Two-Level Namespaces

What gives is that Linux uses a one-level namespace for its linking and loading, and Mac and Windows do not. This is a fairly subtle distinction that almost never matters, but it does here so let’s dig into it a bit.

In a two-level linking scheme, the name of the library a symbol is from becomes, effectively, part of the symbol. Both the PE32 format used by Windows and the Mach-O format used by macOS have this mechanism, and are fairly explicit about it—Windows stack traces name their functions things like kernel32.dll!CreateFileW, and the otool -Lv command on macOS will enumerate any library’s dependencies very precisely. (Usually, given the way macOS applications are expected to work, it’s too precise for my tastes, but that is a tale for another time.)

Linux, on the other hand, uses the ELF format, which provides a single-level (or flat) namespace. The ldd command will produce a list of library dependencies much like otool. However, there is a difference: otool does not list anything that is not encoded in some way within the binary itself. The output of ldd, on the other hand, is computed each time it is run, for its results depend on other files in the system, and possibly even the environment variables at the point the process is started.

Another difference between ldd and otool becomes apparent if we run them over our main2 program. On a Mac, otool reports that main2 links against lib2a and lib2b, just like we commanded. On Linux, though, ldd will produce a list that includes not just those two libraries, but also their dependencies. All of these are loaded in at run time just as if the main program had linked directly against them. This means that, as far as Linux is concerned, we are in the exact situation that we were in when we directly linked conflicting libraries into place, and it behaves accordingly.

We can, however, work around this. The POSIX standard provides a function called dlsym that may load any symbol out of any library, even those that you did not link against in the first place. That’s a mechanism that’s sufficiently general that a library could implement whatever linker/loader scheme it wants. It’s got some neat nonstandard-but-widely-available extensions too. GCC and compatible systems (including macOS’s clang compiler) define a special pseudo-library called RTLD_NEXT which will load the next copy of a function starting just past the library calling dlsym.

We can have some fun with that. Let’s!

Library Wrappers

This is the intended use case for RTLD_NEXT: suppose we have a library, and it does almost what we want. We want to define some kind fo interposing library, so that we can do what extra work we need before handing control off to the original code. For example, the PC release of From Software’s Dark Souls was quite buggy, but fans produced a wrapper around Microsoft’s DirectX library that provided the missing functionality that made the game a much more pleasant user experience. Alternately, one may be trying to bridge two not-quite-compatible libraries—SDL1 and SDL2 share function exports and as such cannot be linked directly into the same executable without running into the kinds of issues we saw at the top of this article, but an interposing library could provide the SDL1 API and forward calls as needed to an SDL2 implementation under the hood. (The use cases for this are somewhat limited, because generally either SDL1 will itself run acceptably or the application logic will need some changes to properly run in the kinds of environments SDL2 expects, but it is not zero.)

Our basic strategy is to expose a library’s API, and then call into its own functions while providing identical ones. The problem is that we are ourselves defining functions that we also want to load out of our dependencies. We want the final application to link against our versions of the functions instead of the underlying library’s, but that is no problem even in the flat namespace—the final application only links against us, so if they call a function that both we and the library we wrap define, it will call us. The problem is how we call those library functions when we’ve defined the names ourselves and thus would otherwise be recursively calling our own functions.

Mac and Linux

On Mac and Linux, the strategy is the same: when we need to call an underlying function, use the dlsym function to extract the necessary call from the underlying library and call the result. For performance reasons, we’ll be wanting to cache the underlying function pointers.

If we want to be strictly POSIX compliant, we will also need to open the underlying library ourselves with dlopen and have some way of reliably finding it. However, with the GNU extensions (that macOS support as well, even with clang), we can rely on the normal linking mechanism by exploiting RTLD_NEXT:

/* We rely on the _GNU_SOURCE extension for wrapping here. */
#define _GNU_SOURCE

#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>
#include "lib1.h"

static void(*secret_awesome_function)(void) = NULL;

void my_awesome_function(void)
{
    char *error = NULL;
    if (!secret_awesome_function) {
        dlerror(); /* Clear previous error */
        secret_awesome_function = dlsym(RTLD_NEXT, "my_awesome_function");
        error = dlerror();
    }
    if (!secret_awesome_function) {
        if (error) {
            printf("Interception failed: '%s'\n", error);
        } else {
            printf("Interception successfully imported a NULL. Nothing to do.\n");
        }
        return;
    }
    printf("Beginning interception!\n");
    secret_awesome_function();
    printf("Ending interception!\n");
}

We can save this out as lib1c.c, compile and link it against lib1a like we did our main program, and then link main1.c against lib1c to get this output:

Beginning interception!
This is my awesome function!
Ending interception!

Microsoft Windows

Windows does not have the dlopen, dlsym, and dlclose functions, but it does have the equivalent functions LoadLibrary, GetProcAddress, and FreeLibrary. As long as you know the name of the library you’re linking against (and you should, because you are linking against it), calls to LoadLibrary and FreeLibrary will only alter reference counts and not actually do any loading.

Other Techniques

Over the course of this article I’ve kind of been glossing over the exact details of how you actually turn source code into DLLs on each of these various platforms, or how to arrange things so that your executables actually find and locate the DLLs when it’s time to run your program. This is because, while the basic strategies for resolving symbol conflicts or extracting symbols at runtime are very close across Windows, Mac, and Linux, all three of these operating systems have drastically different requirements on what it takes to create a DLL or to link executables against them. Playing around with the procedures and options there is at least a full article on its own.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s