Skip to content

Case 212: Calling Convention Macro #undef'd After Use

Field Value
Verdict 🔴 BREAKING
Category Breaking
Classification Rule
Platforms Linux
Flags ABI break
Detected ChangeKinds calling_convention_changed
Source files catalog/cases/case212_calling_convention_macro_undefined_after_use/
Rule family calling-convention-macro-undefined-after-use
Subject Calling convention and integer-model changes

Category: Calling Convention | Verdict: 🔴 BREAKING

Verdict and consumer impact

vector_dot is declared through MATHLIB_CALL, and the header #undefs the macro right after the declaration. v1 defines it as __attribute__((sysv_abi)), v2 as __attribute__((ms_abi)), so the arguments move from rdi/rsi/rdx to rcx/rdx/r8. An unchanged old consumer reads the wrong registers and computes a wrong result.

The macro no longer exists at the end of the translation unit. A reader that resolves conventions from the final macro table (-E -dM) sees no MATHLIB_CALL at all and reports NO_CHANGE; that is the defect the 2026-10-11 re-audit found on the CastXML lane.

Old/new diff

v1.h v2.h
#define MATHLIB_CALL __attribute__((sysv_abi)) #define MATHLIB_CALL __attribute__((ms_abi))
MATHLIB_CALL double vector_dot(...); same
#undef MATHLIB_CALL same

abicheck command

gcc -shared -fPIC -g v1.c -o libv1.so
gcc -shared -fPIC -g v2.c -o libv2.so
abicheck compare libv1.so libv2.so --header old=v1.h --header new=v2.h

Expected abicheck finding

Verdict: BREAKING (exit 4)
  calling_convention_changed  vector_dot: (default) -> ms_abi

Same result with the CastXML and the Clang header backend (ABICHECK_AST_FRONTEND=castxml|clang).

Minimum evidence

L1 (debug info): a Clang-built library carries the convention in DW_AT_calling_convention. GCC emits none, so a GCC build needs the headers (L2), where the macro is read at the declaration.

Why abicheck catches it

CastXML drops GNU x86-64 conventions from its output, so abicheck re-runs the same CastXML command with -E -dD and reads each declaration's own preprocessed line, mapped back through the line markers. That line already has MATHLIB_CALL expanded as it was at the declaration, before the #undef. The Clang backend reads the attribute from the function type.

Runtime failure demonstration

gcc app.c -L. -l:libv1.so -Wl,-rpath,. -lm -o app && ./app   # 32.0, exit 0
cp libv2.so libv1.so && ./app                                 # wrong result, exit 1

Safe redesign

Keep one convention for the life of an ABI. Scoping a convention macro with #undef is fine; changing what it expands to is a break.

Source files

  • CMakeLists.txt
  • app.c
  • v1.c
  • v1.h
  • v2.c
  • v2.h

See also: Compatibility Catalog · All BREAKING cases · Category: Breaking · Rule: Calling convention macro undefined after use · Subject: Calling convention and integer-model changes.