This is a deep dive into DLL Hijacking, specifically Search Order Hijacking. This is one of the most common and persistent techniques used by attackers to gain control over a system.

In this guide, we will look at how Windows loads code, how to find vulnerable applications using Sysinternals Process Monitor, and how to exploit the flaw.

How Windows Loads DLLs

DLL stands for Dynamic Link Library. Dynamic Linking means an application does not contain all the code it needs to run. Instead, it reaches out to external files (DLLs) to borrow code when it runs. This saves memory and allows multiple apps to share the same code.

The vulnerability happens because an application often does not know exactly where a DLL is located. It just asks Windows for the file by its name (for example, “I need functions.dll”)

The Safe DLL Search Mode

When an application only provides a filename, Windows searches for it in a specific order. Windows uses a strict priority list called Safe DLL Search Mode, which is turned on by default.

Here is the exact order Windows uses:

1. The Application Directory: The folder where the application is located. This has the highest priority.

2. The System Directory: Usually C:\Windows\System32.

3. The 16-bit System Directory: C:\Windows\System.

4. The Windows Directory: C:\Windows.

5. The Current Working Directory (CWD): The folder you are currently in

6. The PATH Environment Variable: A list of other folders defined in system settings

How the attack works: Most legitimate DLLs live in the System32 folder, which is step 2. But Windows checks the Application Directory (step 1) before it checks System32. If an attacker places a malicious file with the exact same name in the application’s folder, Windows will find the fake file at step 1 and stop searching. It effectively ignores the real file in System32

Finding Vulnerabilities with Process Monitor (ProcMon)

To see this search order in action, we can use a free Microsoft tool called Process Monitor (ProcMon) from the Sysinternals Suite. ProcMon logs every single file and registry operation happening on your computer

Since ProcMon captures millions of events, we need to set up a specific “Vulnerability Hunter” filter. We want to look for applications that are failing to find a DLL

Add these three rules in your ProcMon filter:

1. Process Name -> contains -> [Your Target App] -> Include.
2. Result -> is -> NAME NOT FOUND -> Include.
3. Path -> ends with -> .dll -> Include.

Once these filters are applied, your ProcMon window will go blank. Now, we can test applications to see how they behave

The Baseline: A Secure Application (Notepad)

Let’s look at how a secure application behaves. If we set our filter to watch Notepad.exe and then open Notepad and click around, ProcMon will catch its DLL requests.

Add these three rules in your ProcMon filter(Ctrl + L):

1. Process Name -> contains -> Notepad.exe -> Include. -> Add

2. Result -> is -> NAME NOT FOUND -> Include -> Add

3. Path -> ends with -> .dll -> Include -> Add -> OK

4. Open Notepad and click around (like File > Open). ProcMon will catch its DLL requests

When you look at the “Path” column in ProcMon for a secure application, you will see an Absolute Path

Log Example: C:\Windows\System32\DriverStore\FileRepository\iigd_dch.inf_amd64_9741ef1f4093481f\dxgi.dll

What this means: Notepad explicitly told Windows, “I want the file specifically inside the System32 folder”. It did not just ask for the filename. Because it provided a strict location, it is NOT vulnerable to simple search order hijacking. An attacker cannot simply drop a fake file into System32 because that requires Administrator rights. In this case, Notepad is safe.

The Target: A Vulnerable Application

Now, let’s look at a vulnerable application. A vulnerable app does not use an absolute path; it uses a Relative Path. It just asks for “hijack_me.dll”, so Windows starts "hunting" for it down the search order list.

Let’s prove this works by building a simple vulnerable application and a malicious DLL to hijack it.

Step 1: The Vulnerable App

We will write a harmless C++ program that tries to load a library called “hijack_me.dll”. Crucially, it only asks for the filename, not the full path, triggering the vulnerability.

Save this as “victim.cpp”:

#include <windows.h>
#include <iostream>
int main() {
    std::cout << "[*] Victim Application Started." << std::endl;
    std::cout << "[*] Attempting to load 'hijack_me.dll'..." << std::endl;
        HMODULE hDll = LoadLibraryA("hijack_me.dll");
        if (hDll != NULL) {
        std::cout << "[+] DLL Loaded Successfully!" << std::endl;
    } else {
        std::cout << "[-] Failed to load DLL." << std::endl;
    }
        getchar();
    return 0;
}

Step 2: The Malicious DLL

In Windows, every DLL has a starting function called DllMain. This function runs automatically the moment the DLL attaches to the application. We will program our DLL to show a message box immediately upon loading.

Save this as “evil.cpp”

#include <windows.h>
BOOL APIENTRY DllMain(HMODULE hModule, DWORD ul_reason_for_call, LPVOID lpReserved) {
    switch (ul_reason_for_call) {
    case DLL_PROCESS_ATTACH:
        MessageBoxA(NULL, "You have been HIJACKED!", "Security Alert", MB_OK | MB_ICONWARNING);
        break;
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}

Step 3: Executing the Hijack

Using a compiler like “x64 Native Tools Command Prompt”, compile Victim.cpp into an executable (Victim.exe). Then, compile evil.cpp as a shared library named hijack_me.dll

Compile the Victim.cpp:

    g++ victim.cpp -o Victim.exe

Compile the evil.cpp:

    g++ evil.cpp -shared -o hijack_me.dll

Before we execute Victim.exe, let’s look at how the application behaves without the malicious DLL present

First, let’s set up a filter in ProcMon to watch our victim application:

1. Process Name -> contains -> Victim.exe -> Include -> Add

2. Result -> is -> NAME NOT FOUND -> Include ->

3. Path -> ends with -> .dll -> Include -> Add -> Ok

Test 1: Running Without the Payload

Make sure Victim.exe is in a folder by itself, and run it.

The application will print: “Failed to load DLL.”

If you look at ProcMon, you will see exactly why. You will see a sequence of NAME NOT FOUND results for hijack_me.dll.

1. It looks in the application directory first.
2. Then it looks in “C:\Windows\System32\”
3. Then it looks in “C:\Windows\System\”

Because the file does not exist anywhere, the application gives up. This proves the application is vulnerable because it checks its own local folder first.

If an attacker has permission to write a file into that application folder, they can place a fake hijack_me.dll right there. The app will find it at Step 1 and execute it.

Test 2: Running With the Payload

Move your compiled hijack_me.dll into the exact same folder as Victim.exe.

Run Victim.exe. Immediately, a Message Box pops up saying: "You have been HIJACKED!"

Why did this happen? The victim application asked Windows for the hijack_me.dll. Windows checked Priority #1: The Application Directory. It found our malicious file immediately, loaded it, and executed our payload. In a real cyberattack, instead of a message box, the attacker would use this silent access to open a backdoor into your computer.