Today’s topic is uprobes (user space probes).
The focus will not be on understanding the code but rather on where and how to discover and attach uprobes in the application stack, which is tricky — believe me.
Let me show you.
Consider that we want to extract SSH username and password on each user login. This is potentially a malicious act but a good exercise for this demonstration.
A good first step is to identify the modules or software libraries responsible for key functionalities within the system or application you intend to observe.
This step is much easier if you wrote the application yourself, but if not, it's essential, as there could be multiple ways to achieve your goal.
I’ll bring you up to speed — In the SSH daemon, a.k.a. sshd, Linux PAM (Pluggable Authentication Modules) is a suite of libraries that enable user authentication, so this should be our starting point.
The second step would be to start looking through the module source code on GitHub as we're interested in finding a hook point or, I should say, a function that either handles the authentication or has access to the authentication parameters so we can extract them.
Fast forward — Traversing through the code and searching for specific symbols, one can find a pam_get_authtok
function. This should trigger some interest.
This step was kind of tedious, but since we’re trying to showcase how to find an arbitrary function and trace it, I don’t think there’s a better approach.
At this step, we can only technically speculate that this function is triggered on user login.
In the third step we need to verify this function is utilized by our sshd. We can do so by exporting the libraries it utilizes and searching for the function symbol. We achieve that by first figuring out the sshd PID (Process ID):
ps faux | grep sshd
Using this information, we can list the symbol table of the library using the following command:
sudo readelf -s --wide /proc/<PID>/root/usr/lib/x86_64-linux-gnu/libpam.so.0 | grep pam_get_authtok
💡 Note: You need to replace the <PID> placeholder with the sshd PID. In my case, it was 597234 as seen in the image above (second column)..
We can see that the library does indeed have the pam_get_authtok
function symbol. But is it actually triggered on a user login?
So in the fourth step, we need to verify if the function is actually triggered.
Here's a minimal kernel space program:
//go:build ignore
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
char __license[] SEC("license") = "GPL";
SEC("uretprobe/pam_get_authtok")
int trace_pam_get_authtok(struct pt_regs *ctx) {
bpf_printk("It's triggered - jeii!");
return 0;
};
💡 Note: A uretprobe is a probe that is triggered on the exit of a function, rather than on the entry of it like a uprobe.
And a User Space program:
package main
//go:generate go run github.com/cilium/ebpf/cmd/bpf2go -target amd64 guard guard.c
import (
"log"
"fmt"
"time"
"os/exec"
"strings"
"bufio"
"bytes"
"github.com/cilium/ebpf/link"
"github.com/cilium/ebpf/rlimit"
)
const (
defaultBinPath = "libpam.so.0"
defaultSymbol = "pam_get_authtok"
)
func findLibraryPath(libname string) (string, error) {
cmd := exec.Command("sh", "-c", fmt.Sprintf("ldconfig -p | grep %s", libname))
// Run the command and get the output
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
return "", fmt.Errorf("failed to run ldconfig: %w", err)
}
// Read the first line of output which should have the library path
scanner := bufio.NewScanner(&out)
if scanner.Scan() {
line := scanner.Text()
// Extract the path from the ldconfig output
if start := strings.LastIndex(line, ">"); start != -1 {
path := strings.TrimSpace(line[start+1:])
return path, nil
}
}
return "", fmt.Errorf("library not found")
}
func main() {
// Allow the current process to lock memory for eBPF resources.
if err := rlimit.RemoveMemlock(); err != nil {
log.Fatal(err)
}
// Load pre-compiled programs and maps into the kernel.
objs := guardObjects{}
if err := loadGuardObjects(&objs, nil); err != nil {
log.Fatalf("loading objects: %s", err)
}
defer objs.Close()
pamPath, err := findLibraryPath(defaultBinPath);
if err != nil {
log.Fatal(err)
}
log.Printf("LibPAM path: %s\n", defaultBinPath);
// Open an ELF binary and read its symbols.
ex, err := link.OpenExecutable(pamPath)
if err != nil {
log.Fatalf("opening executable: %s", err)
}
// Set up uretprobes
uretprobe_pam, err := ex.Uretprobe(defaultSymbol, objs.TracePamGetAuthtok, nil)
if err != nil {
log.Fatalf("creating uretprobe - %s: %s", defaultSymbol, err)
}
defer uretprobe_pam.Close()
for {
time.Sleep(1 * time.Second)
}
}
So the kernel program will just print something, but the slightly trickier part is attaching the uretprobe to the libpam binary. We first need to find the path to the library itself and utilize eBPF Link to attach our program to it.
By running this program, we can see that our program was indeed triggered by printing eBPF traces:
sudo bpftool prog trace
This is good — we are on the right path.
Now in the last step, it's just a matter of checking which parameters we need to read from the registers to get the username and password parameters, which we already discussed before.
Based on the source code — In the pam_get_authtok
function, we need to read the pam_handle_t
struct, which we can copy and paste from the source into our code and utilize the bpf_probe_read
helper function to extract the parameters, namely authtok
and user
. In other words, adding the following lines into our kernel code:
//go:build ignore
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
char __license[] SEC("license") = "GPL";
SEC("uretprobe/pam_get_authtok")
int trace_pam_get_authtok(struct pt_regs *ctx) {
bpf_printk("It's triggered - jeii!");
if (!PT_REGS_PARM1(ctx)) {
return 0;
}
pam_handle_t* phandle = (pam_handle_t*)PT_REGS_PARM1(ctx);
u32 pid = bpf_get_current_pid_tgid() >> 32;
u64 password_addr = 0;
bpf_probe_read(&password_addr, sizeof(password_addr), &phandle->authtok);
u64 username_addr = 0;
bpf_probe_read(&username_addr, sizeof(username_addr), &phandle->user);
return 0;
};
This way, we finished the extraction of the variables and technically got what we wanted.
💡 Note: This example only covers username and password authentication, but we could technically expand this to also cover authentication using SSH public and private keys.
To wrap this up, in the final code, I’ve also added the eBPF Ring Buffer Map to forward the extracted data to user space. You can find the complete code on my GitHub.
We have come to the end of this week's newsletter, and to be honest, the hardest part in developing uprobe eBPF programs is finding the right hook points and understanding the library source code. Additionally, it’s important to verify using ps
and readelf
that our applications are actually utilizing the function we plan to track. However, if you're searching for a hook point inside your application or library, this would be more straightforward.
I hope you find this resource as enlightening as I did. Stay tuned for more exciting developments and updates in the world of eBPF in next week's newsletter.
Until then, keep 🐝-ing!
Warm regards, Teodor