Skip to content

Case 64: Calling Convention Changed

Field Value
Verdict 🔴 BREAKING
Category Breaking
Platforms Linux
Flags ABI break
Detected ChangeKinds calling_convention_changed
Source files examples/case64_calling_convention_changed/

Category: Function ABI | Verdict: 🔴 BREAKING

Verdict and consumer impact

vector_dot/vector_scale switch from the default System V AMD64 calling convention to __attribute__((ms_abi)) (Microsoft x64 convention), which puts parameters in different registers:

System V ABI (v1) Microsoft x64 ABI (v2)
Integer args rdi, rsi, rdx, rcx, r8, r9 rcx, rdx, r8, r9
Float args xmm0–xmm7 xmm0–xmm3
Callee-saved rbx, rbp, r12–r15 rbx, rbp, rdi, rsi, r12–r15

A caller compiled against v1 passes a in rdi, b in rsi, len in edx. The v2 function reads a from rcx, b from rdx, len from r8d — whatever stale values those registers happen to hold. Recompilation against v2 is mandatory.

Old/new diff

/* v1: default System V AMD64 calling convention */
double vector_dot(const double *a, const double *b, int len);

/* v2: Microsoft x64 calling convention — different register assignment */
__attribute__((ms_abi))
double vector_dot(const double *a, const double *b, int len);

abicheck command

clang -shared -fPIC -g v1.c -o libfoo_v1.so
clang -shared -fPIC -g v2.c -o libfoo_v2.so
abicheck compare libfoo_v1.so libfoo_v2.so

Expected abicheck finding

Verdict: BREAKING (exit 4)

- calling_convention_changed: Calling convention changed: vector_dot (normal -> unknown(0xc1))
  > Function calling convention changed; registers/stack usage differs, call crashes.
- calling_convention_changed: Calling convention changed: vector_scale (normal -> unknown(0xc1))
  > Function calling convention changed; registers/stack usage differs, call crashes.

Minimum evidence

min_evidence: L1 — DWARF's DW_AT_calling_convention attribute on the subprogram DIE records the convention directly, so debug info alone (no public headers) is enough. Toolchain note: GCC does not emit DW_AT_calling_convention for __attribute__((ms_abi)) — compiling both sides with GCC produces NO_CHANGE (a real false negative, not a fixture bug); Clang does emit the attribute and abicheck reports BREAKING as expected. The command above builds with Clang for exactly that reason.

Why abicheck catches it

abicheck reads each function's DW_AT_calling_convention value from DWARF and compares it between versions; ms_abi and the default System V convention encode to different DWARF constants, so the change is visible without any header/AST evidence — when the compiler emits the attribute at all (see the toolchain note above).

Runtime failure demonstration

Severity: CRITICAL

Scenario: compile app against v1, swap in v2 .so without recompile. (GCC is used here since the calling-convention mismatch itself is a runtime register-passing issue, independent of which compiler emits the DWARF attribute abicheck reads.)

# Build old library + app
gcc -shared -fPIC -g v1.c -o libfoo.so
gcc -g app.c -L. -lfoo -Wl,-rpath,. -o app -lm
./app
# → dot product = 32.0 (expected 32.0)
# → scaled = {2.0, 4.0, 6.0} (expected {2.0, 4.0, 6.0})

# Swap in new library (no recompile)
gcc -shared -fPIC -g v2.c -o libfoo.so
./app
# → dot product = 0.0 (expected 32.0)
# → scaled = {0.0, 0.0, 0.0} (expected {2.0, 4.0, 6.0})
# → WRONG RESULT: calling convention mismatch — parameters passed in wrong registers!

Why CRITICAL: the v2 function expects pointer parameters in rcx/rdx (ms_abi) but receives them in rdi/rsi (System V) — it reads from registers holding stale (typically zero) values, producing garbage results. With different register contents this could also segfault if a stale value is dereferenced as a pointer.

Safe redesign

Never change the calling convention of an exported function. If a new convention is needed, add a new symbol (vector_dot_fast()), keep the old entry point as a thin forwarding wrapper, or ship a separate library with a migration guide.

Real-world example: the Windows ecosystem has dealt with this extensively — __stdcall vs __cdecl vs __fastcall vs __vectorcall have caused countless DLL compatibility issues, which is why the Win32 API froze on __stdcall specifically. Wine must carefully match calling conventions between Linux System V and Windows ms_abi for every thunked function — getting even one wrong causes the exact crash demonstrated here.

Cross-tool comparison

abidw --out-file v1.xml libfoo_v1.so
abidw --out-file v2.xml libfoo_v2.so
abidiff v1.xml v2.xml

References

Source files

  • CMakeLists.txt
  • app.c
  • v1.c
  • v1.h
  • v2.c
  • v2.h

See also: Examples overview · All BREAKING cases · Category: Breaking.