An integer overflow problem has been discovered leading to hep corruption. When calling GC_MALLOC_ATOMIC(0xFFFFFFFFFFFFF2ABull), the expected behavior should be out of memory obviously and return NULL or abort. However, libgc returns a pointer. The caller thought the allocation succeeded and started to write data into heap via the said pointer and thus heap corruption.
The reason is integer overflow in macro OBJ_SZ_TO_BLOCKS in gc_priv.h:1021. (sz) + HBLKSIZE-1 overflows and become a small positive number. After the overflow, libgc allocates a small block of memory and return the pointer.