gccrs#4599: Refactor dynamic object fat pointers and vtable generation

Link: Rust-GCC/gccrs#4599

Note

This is important

Objective

This PR refactors the memory layout of fat pointers (for dynamic trait objects) and vtable generation to match rustc ABI expectations.

Changes Made

Previously, the compiler incorrectly embedded the entire vtable directly inside the fat pointer as an array, and missed essential trait object metadata.

• The fat pointer is now strictly constrained to 2 words (data pointer and vtable pointer).

• The vtable is generated as a separate global static struct.

• The vtable structure now correctly includes drop_in_place, size, and align fields, followed by trait methods.

• Added a caching mechanism in the compilation context to prevent duplicate vtable generation and linker conflicts for identical Type-Trait combinations.

Let’s examine the following example (this example is for 64-bit):

#![allow(unused)]
fn main() {
struct StructX {
    val: i32,
}

trait TraitA {
    fn do_a(&self) -> i32;
    fn dont_a(&self) -> i32;
}
}

Let’s examine the &StructX -> &dyn TraitA coercion.

# Previous fat-pointer memory              │ # Current fat-pointer memory
                                           │
├───────────────────────┤                  │ ├───────────────────────┤
│ StructX           (4) │ 0x1000           │ │ StructX           (4) │ 0x1000
├───────────────────────┤                  │ ├───────────────────────┤
            .                              │             .
            .                              │             .
├───────────────────────┤ ─┐               │ ├───────────────────────┤ ─┐
│ &StructX          (8) │  │ &dyn TraitA   │ │ &StructX          (8) │  │ &dyn TraitA
├───────────────────────┤  │ (fat pointer) │ ├───────────────────────┤  │ (fat pointer)
│ &TraitA::do_a     (8) │  │ 24 bytes      │ │ &__vtable_TraitA  (8) │  │ 16 bytes
├───────────────────────┤  │               │ ├───────────────────────┤ ─┘
│ &TraitA::dont_a   (8) │  │               │ │ (Does not exist)      │
├───────────────────────┤ ─┘               │ ├───────────────────────┤
                                           │
Note: If TraitA had 10 methods, this fat   │ Note: Regardless of the number of methods
pointer would occupy 88 bytes. It also     │ in TraitA, the fat pointer size is always
lacked size, align, and drop_in_place      │ 16 bytes. All necessary metadata is
pointers.                                  │ successfully preserved in the vtable.

Also you can see the generated vtable by the new fat-pointer generator in static memory:

Generated vtable - 40 bytes
├───────────────────────┤
│ &drop_in_place    (8) │ <- address of method drop_in_place
├───────────────────────┤
│ size              (8) │ <- size information
├───────────────────────┤
│ align             (8) │ <- align information
├───────────────────────┤
│ &TraitA::do_a     (8) │ <- address of method do_a
├───────────────────────┤
│ &TraitA::dont_a   (8) │ <- address of method dont_a
├───────────────────────┤

The main reason behind this design by the Rust team is to allow the creation of arrays of fat pointers (which I will explain in more detail in #4722) and to avoid generating a new vtable for every coercion. When a dyn Trait coercion occurs for a specific type, its vtable is generated statically. Subsequent coercions for the same type simply point to this existing vtable, avoiding any additional cost.

Why is this needed for alloc?

For dynamically sized types (DSTs) and trait objects within alloc to function correctly and efficiently, the fat pointer architecture must exactly match standard Rust behavior.

Test Case

fn print1(label: &str, val: i32) {
    let arrow = "─".repeat(36);
    println!("{:<7}: {:>2} <{}┘   │", label, val, arrow);
}
fn print2(label: &str, val: i32) {
    let arrow = "─".repeat(40);
    println!("{:<7}: {:>2} <{}┘", label, val, arrow);
}

//        _~^~^~_      Ferris says hello!
//    \) /  o o  \ (/
//      '_   -   _'
//      / '-----' \

trait TraitA {
    fn do_a(&self) -> i32;
    fn dont_a(&self) -> i32;
}

struct StructX {
    val: i32,
}

impl TraitA for StructX {
    fn do_a(&self) -> i32 { self.val}     // ────────┐
    fn dont_a(&self) -> i32 { -self.val } // ────┐   │
}                                         //     │   │
                                          //     │   │
fn main() {                               //     │   │
    let x = StructX { val: 1 }; // self = 1 ->   │   │
    let dyn_a_x: &dyn TraitA = &x;        //     │   │
                                          //     │   │
    print1("dont_a", dyn_a_x.dont_a());   //     │   │
    print2("do_a", dyn_a_x.do_a());       //     │   │
}                                         //     │   │