Why indirect syscalls slip past some EDR hooks

Plenty of endpoint products inspect behaviour by hooking functions in ntdll.dll in user mode. It’s cheap and it works — right up until code stops calling those functions the expected way. This is a high-level explanation of why that gap exists, aimed at defenders deciding how much to trust user-mode telemetry.

The leaky abstraction

A user-mode hook intercepts a call by rewriting the start of a function so execution detours through the security product first. The catch: that only helps if the code actually calls the hooked function. The transition to the kernel is ultimately just a syscall instruction with a service number in a register — and nothing forces anyone to reach it via the hooked stub.

graph LR
A[Malicious code] -->|expected path| B["ntdll!NtAlloc (hooked)"]
B --> C[EDR sees the call]
A -.->|direct path| D[syscall instruction]
D --> E[Kernel]
C --> E

// Illustrative only — the service number is resolved at runtime,
// then control reaches the kernel without traversing the hooked export.
mov r10, rcx
mov eax, <service_number>
syscall
ret

What it means for defenders

If your detection strategy leans entirely on user-mode hooks, technique like this is a blind spot. The durable answers live lower down:

• Kernel-side telemetry (e.g. ETW threat-intel providers) sees the actual transition.
• Call-stack inspection at the syscall boundary reveals when the return address doesn’t land inside ntdll.
• Anomaly on the source — a syscall originating from outside a loaded module’s .text is itself suspicious.

The takeaway isn’t “hooks are useless” — they catch a great deal of commodity tooling. It’s that you should know exactly where your visibility ends, and not mistake a quiet hook for a quiet host.