[PATCH 1/4] [clang] Improve nested name specifier AST representation #147835

mizvekov · 2025-07-09T21:39:29Z

This is a major change on how we represent nested name qualifications in the AST.

The nested name specifier itself and how it's stored is changed. The prefixes for types are handled within the type hierarchy, which makes canonicalization for them super cheap, no memory allocation required. Also translating a type into nested name specifier form becomes a no-op. An identifier is stored as a DependentNameType. The nested name specifier gains a lightweight handle class, to be used instead of passing around pointers, which is similar to what is implemented for TemplateName. There is still one free bit available, and this handle can be used within a PointerUnion and PointerIntPair, which should keep bit-packing aficionados happy.
The ElaboratedType node is removed, all type nodes in which it could previously apply to can now store the elaborated keyword and name qualifier, tail allocating when present.
TagTypes can now point to the exact declaration found when producing these, as opposed to the previous situation of there only existing one TagType per entity. This increases the amount of type sugar retained, and can have several applications, for example in tracking module ownership, and other tools which care about source file origins, such as IWYU. These TagTypes are lazily allocated, in order to limit the increase in AST size.

This patch offers a great performance benefit.

It greatly improves compilation time for stdexec. For one datapoint, for test_on2.cpp in that project, which is the slowest compiling test, this patch improves -c compilation time by about 7.2%, with the -fsyntax-only improvement being at ~12%.

This has great results on compile-time-tracker as well:

This patch also further enables other optimziations in the future, and will reduce the performance impact of template specialization resugaring when that lands.

It has some other miscelaneous drive-by fixes.

About the review: Yes the patch is huge, sorry about that. Part of the reason is that I started by the nested name specifier part, before the ElaboratedType part, but that had a huge performance downside, as ElaboratedType is a big performance hog. I didn't have the steam to go back and change the patch after the fact.

There is also a lot of internal API changes, and it made sense to remove ElaboratedType in one go, versus removing it from one type at a time, as that would present much more churn to the users. Also, the nested name specifier having a different API avoids missing changes related to how prefixes work now, which could make existing code compile but not work.

How to review: The important changes are all in clang/include/clang/AST and clang/lib/AST, with also important changes in clang/lib/Sema/TreeTransform.h.

The rest and bulk of the changes are mostly consequences of the changes in API.

PS: TagType::getDecl is renamed to getOriginalDecl in this patch, just for easier to rebasing. I plan to rename it back after this lands.

This is split into four interdependent patches:

Fixes #136624
Fixes #147000
Fixes #43179
Fixes #68670
Fixes #92757

erichkeane

Is there ANYTHING we can do to split this up? This is so huge that github is refusing to load half the changes.

mizvekov · 2025-07-10T17:16:54Z

Is there ANYTHING we can do to split this up? This is so huge that github is refusing to load half the changes.

I broke the commit down into others, now the first commit only has the AST and TreeTransform changes.

erichkeane · 2025-07-10T17:23:26Z

Is there ANYTHING we can do to split this up? This is so huge that github is refusing to load half the changes.

I broke the commit down into others, now the first commit only has the AST and TreeTransform changes.

Can we split them up in reviews/do reviews as a stack if necessary? reviewing each individual 'commit' in the same review is also really difficult.

mizvekov · 2025-07-10T17:36:40Z

Can we split them up in reviews/do reviews as a stack if necessary? reviewing each individual 'commit' in the same review is also really difficult.

Possible, although it would be difficult to split this in a way where each commit works individually.
If we accept that we will have to rely on the top of the stack being rebased in order to get testing feedback, and that this will have to be committed all in one go, I can do it.

erichkeane · 2025-07-10T17:38:40Z

Can we split them up in reviews/do reviews as a stack if necessary? reviewing each individual 'commit' in the same review is also really difficult.

Possible, although it would be difficult to split this in a way where each commit works individually. If we accept that we will have to rely on the top of the stack being rebased in order to get testing feedback, and that this will have to be committed all in one go, I can do it.

I'm definitely OK with accepting all of that, and would very much appreciate it. Note that Graphite is a tool others have used to manage patch-stacks that can do a lot of that for you, if it makes it easier.

mizvekov · 2025-07-10T18:01:21Z

I'm definitely OK with accepting all of that, and would very much appreciate it.

Done.

rjmccall · 2025-07-10T18:28:50Z

You should at minimum be able to layer the NestedNameSpecifier change on top of the TagType change. I can't see why that wouldn't work.

Add the structure for sugary TagTypes.
Replace ElaboratedType with sugary TagTypes.
Change the representation of NNS.

erichkeane

For obvious reasons, I wasn't able to make even a reasonably thorough review here, but I came across a few things that seemed like good ideas. All 4 of the patches are STILL huge.

IMO, unless this can be more bite-sized changes, I don't know how to proceed here.

erichkeane · 2025-07-10T18:12:44Z

clang/include/clang/AST/ASTContext.h

-      return QualType(PrevDecl->TypeForDecl, 0);
-    }
+  QualType getTypeDeclType(const TypeDecl *Decl) const;
+  QualType getTypeDeclType(const TagDecl *) const = delete;


Can we do some sort of comments to explain why these are deleted, so that when someone runs into them, they can see what they should be using instead?

erichkeane · 2025-07-10T18:24:13Z

clang/include/clang/AST/NestedNameSpecifierBase.h

+
+    /// Microsoft's '__super' specifier, stored as a CXXRecordDecl* of
+    /// the class it appeared in.
+    Super,


"While we are here" :) It would be REALLY nice if instead of Super we spelled this as something more obvious what it is. Something likeMicrosoftSuper? Super hits too many other language-keyword-type things for me, and it isn't particularly clear when reading other stuff in this patch that it was what this is.

Yeah, this rename would be straight forward.

erichkeane · 2025-07-10T18:27:30Z

clang/lib/AST/ASTContext.cpp

@@ -3413,7 +3417,10 @@ static void encodeTypeForFunctionPointerAuth(const ASTContext &Ctx,
    //   type, or an unsigned integer type.
    //
    // So we have to treat enum types as integers.
-    QualType UnderlyingType = cast<EnumType>(T)->getDecl()->getIntegerType();
+    QualType UnderlyingType = cast<EnumType>(T)
+                                  ->getOriginalDecl()


This has shown up a few times so far... could this use a helper of some sort (getOriginalDecl->getDefinitionOrSelf)?

erichkeane · 2025-07-10T18:28:52Z

clang/lib/AST/ByteCode/Program.cpp

@@ -326,7 +326,7 @@ Record *Program::getOrCreateRecord(const RecordDecl *RD) {
      const auto *RT = Spec.getType()->getAs<RecordType>();
      if (!RT)
        return nullptr;
-      const RecordDecl *BD = RT->getDecl();
+      const RecordDecl *BD = RT->getOriginalDecl()->getDefinitionOrSelf();


I'm pretty convinced by now, getOriginalDefinitionOrSelf would pay HUGE dividends in this patch :)

Yeah, the problem is that it would be really annoying to provide an inline definition of this helper, since it has to be defined with Type.h, but the definition depends on Decl bits which in turn depend on Type.h.

Not making them inline has a noticeable perf hit.

I had the same problem with providing inline implementations of a bunch of NestedNameSpecifier members.
What I did there, was to split this into NestedNameSpecifierBase.h and NestedNameSpecifier.h, the latter containing only inline definitions, and the former being included in places which would otherwise introduce a circular dependency.

I think making this change in Type.h for this patch would have been much larger though, so I'd propose doing this in a follow up.

Another possibility is providing these as inline free-function helpers, defined in Decl.h, but I think this looks uglier (even though we use this pattern elsewhere).

mizvekov · 2025-07-10T18:48:13Z

You should at minimum be able to layer the NestedNameSpecifier change on top of the TagType change. I can't see why that wouldn't work.

Add the structure for sugary TagTypes.

Replace ElaboratedType with sugary TagTypes.

Change the representation of NNS.

Well it's not just TagTypes which were changed in order to not need ElaboratedType, everything else too (TypedefTypes, TemplateSpecializationTypes, etc).

It would be really annoying to remove ElaboratedType from just TagType, leave it on everything else, while making every test pass, due for example how complicated some AST matchers would become in the intermediary state.

If you meant splitting the ElaboratedType removal from the NestedNameSpecifier representation change, that's easier to do, but still its a lot of work to make it in a way which every test passes.

If you would be happy to have dependent patches which must still be tested together, I can certainly split this patch (part 1) into NestedNameSpecifier changes and ElaboratedType changes, if that helps.

mizvekov · 2025-07-10T18:51:05Z

For obvious reasons, I wasn't able to make even a reasonably thorough review here, but I came across a few things that seemed like good ideas. All 4 of the patches are STILL huge.

IMO, unless this can be more bite-sized changes, I don't know how to proceed here.

Yes this can be more bite-sized changes, as long as we keep the patches not individually testable.

I will follow the suggestion above.

rjmccall · 2025-07-10T18:56:03Z

You should at minimum be able to layer the NestedNameSpecifier change on top of the TagType change. I can't see why that wouldn't work.

Add the structure for sugary TagTypes.

Replace ElaboratedType with sugary TagTypes.

Change the representation of NNS.

Well it's not just TagTypes which were changed in order to not need ElaboratedType, everything else too (TypedefTypes, TemplateSpecializationTypes, etc).

It would be really annoying to remove ElaboratedType from just TagType, leave it on everything else, while making every test pass, due for example how complicated some AST matchers would become in the intermediary state.

I actually don't see why. AST matching becomes heinously complicated in the end state of your change if you don't have methods to ask questions like "was this written as an elaborated type" or whatever else the client code in Sema is looking for. So presumably you do have those predicates, and the end state uses them. You can certainly implement those predicates in terms of ElaboratedType while it still exists. That should be patch 1.

mizvekov · 2025-07-10T19:10:25Z

I actually don't see why. AST matching becomes heinously complicated in the end state of your change if you don't have methods to ask questions like "was this written as an elaborated type" or whatever else the client code in Sema is looking for. So presumably you do have those predicates, and the end state uses them. You can certainly implement those predicates in terms of ElaboratedType while it still exists. That should be patch 1.

Yes, that's possible, maybe this is not as hard as I originally made it seem to be, but any kind of splitting this up in which every test has to pass in intermediary states would be a huge time sink to me in my current condition.

I made this patch while I was working 100% on clang, and then continued while I was unemployed while looking for a new thing. Now I have a new project with another employer, and it's not dedicated to clang upstream anymore, so I am not sure when I will have that kind of time again.

zyn0217 · 2025-07-11T02:14:23Z

Given the scale of the patch, it's not surprising if it undergoes some revert-reapplies cycles after the initial commit.

To reduce the churn, would it be possible to ask google (and other major downstream clients) to test it internally before we merge? @AaronBallman

AaronBallman · 2025-07-11T12:24:01Z

Given the scale of the patch, it's not surprising if it undergoes some revert-reapplies cycles after the initial commit.

To reduce the churn, would it be possible to ask google (and other major downstream clients) to test it internally before we merge? @AaronBallman

We can certainly ask; it's in their best interests to avoid that amount of churn if there are problems. But I'm not certain we have a way to really do that aside from poke individuals and see whether they're willing to try it out internally or not.

Sirraide

Some comments, but I will say that I haven’t nearly reviewed all of this because it’s rather massive (and I’m also not too familiar w/ everything around NNSs).

clang/include/clang/AST/ASTContext.h

clang/include/clang/AST/NestedNameSpecifierBase.h

Sirraide · 2025-07-11T13:45:04Z

clang/include/clang/AST/NestedNameSpecifierBase.h

+    Super,
+  };
+
+  inline Kind getKind() const;


Is there a reason a lot of these declarations are inline? Because from what I know that doesn’t really do anything on declarations that aren’t definitions, does it?

It does, an inline definition can't be provided here due to circular dependencies, but it is provided in NestedNameSpecifier.h. This is a way to get around the interdependence between Decl.h and Type.h.

clang/include/clang/AST/NestedNameSpecifier.h

This is a major change on how we represent nested name qualifications in the AST. * The nested name specifier itself and how it's stored is changed. The prefixes for types are handled within the type hierarchy, which makes canonicalization for them super cheap, no memory allocation required. Also translating a type into nested name specifier form becomes a no-op. An identifier is stored as a DependentNameType. The nested name specifier gains a lightweight handle class, to be used instead of passing around pointers, which is similar to what is implemented for TemplateName. There is still one free bit available, and this handle can be used within a PointerUnion and PointerIntPair, which should keep bit-packing aficionados happy. * The ElaboratedType node is removed, all type nodes in which it could previously apply to can now store the elaborated keyword and name qualifier, tail allocating when present. * TagTypes can now point to the exact declaration found when producing these, as opposed to the previous situation of there only existing one TagType per entity. This increases the amount of type sugar retained, and can have several applications, for example in tracking module ownership, and other tools which care about source file origins, such as IWYU. These TagTypes are lazily allocated, in order to limit the increase in AST size. This patch offers a great performance benefit. It greatly improves compilation time for [stdexec](https://github.com/NVIDIA/stdexec). For one datapoint, for `test_on2.cpp` in that project, which is the slowest compiling test, this patch improves `-c` compilation time by about 7.2%, with the `-fsyntax-only` improvement being at ~12%. This has great results on compile-time-tracker as well: ![image](https://github.com/user-attachments/assets/700dce98-2cab-4aa8-97d1-b038c0bee831) This patch also further enables other optimziations in the future, and will reduce the performance impact of template specialization resugaring when that lands. It has some other miscelaneous drive-by fixes. About the review: Yes the patch is huge, sorry about that. Part of the reason is that I started by the nested name specifier part, before the ElaboratedType part, but that had a huge performance downside, as ElaboratedType is a big performance hog. I didn't have the steam to go back and change the patch after the fact. There is also a lot of internal API changes, and it made sense to remove ElaboratedType in one go, versus removing it from one type at a time, as that would present much more churn to the users. Also, the nested name specifier having a different API avoids missing changes related to how prefixes work now, which could make existing code compile but not work. How to review: The important changes are all in `clang/include/clang/AST` and `clang/lib/AST`, with also important changes in `clang/lib/Sema/TreeTransform.h`. The rest and bulk of the changes are mostly consequences of the changes in API. PS: TagType::getDecl is renamed to `getOriginalDecl` in this patch, just for easier to rebasing. I plan to rename it back after this lands. Fixes #136624 Fixes #147000

mizvekov requested review from zygoloid, rjmccall, cor3ntin, AaronBallman, erichkeane and zyn0217 July 9, 2025 21:39

mizvekov self-assigned this Jul 9, 2025

mizvekov requested review from daniel-grumberg, QuietMisdreavus, cyndyishida and Endilll as code owners July 9, 2025 21:39

cor3ntin requested a review from Sirraide July 10, 2025 06:36

erichkeane reviewed Jul 10, 2025

View reviewed changes

mizvekov force-pushed the users/mizvekov/name-qualification-refactor branch from 69fb353 to 4730e9a Compare July 10, 2025 17:12

mizvekov force-pushed the users/mizvekov/name-qualification-refactor branch from 4730e9a to 374aefc Compare July 10, 2025 17:50

mizvekov mentioned this pull request Jul 10, 2025

[PATCH 2/4] [clang] Improve nested name specifier AST representation #148012

Open

mizvekov changed the title ~~[clang] Improve nested name specifier AST representation~~ [PATCH 1/4] [clang] Improve nested name specifier AST representation Jul 10, 2025

mizvekov mentioned this pull request Jul 10, 2025

[PATCH 3/4] [clang] Improve nested name specifier AST representation #148014

Open

erichkeane reviewed Jul 10, 2025

View reviewed changes

mizvekov force-pushed the users/mizvekov/name-qualification-refactor branch from 374aefc to 8da8c53 Compare July 10, 2025 20:14

mizvekov mentioned this pull request Jul 10, 2025

[PATCH 4/4] [clang] Improve nested name specifier AST representation #148015

Open

mizvekov force-pushed the users/mizvekov/name-qualification-refactor branch from 8da8c53 to 7b74782 Compare July 11, 2025 00:05

Sirraide reviewed Jul 11, 2025

View reviewed changes

mizvekov mentioned this pull request Jul 11, 2025

[Clang][AST][NFC] (RecordDecl -> CXXRecordDecl)::isInjectedClassName #148195

Merged

mizvekov force-pushed the users/mizvekov/name-qualification-refactor branch from 7b74782 to cfe89c1 Compare July 12, 2025 17:40

[PATCH 1/4] [clang] Improve nested name specifier AST representation #147835

Are you sure you want to change the base?

[PATCH 1/4] [clang] Improve nested name specifier AST representation #147835

Conversation

mizvekov commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erichkeane left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mizvekov commented Jul 10, 2025

Uh oh!

erichkeane commented Jul 10, 2025

Uh oh!

mizvekov commented Jul 10, 2025

Uh oh!

erichkeane commented Jul 10, 2025

Uh oh!

mizvekov commented Jul 10, 2025

Uh oh!

rjmccall commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

erichkeane left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mizvekov commented Jul 10, 2025

Uh oh!

mizvekov commented Jul 10, 2025

Uh oh!

rjmccall commented Jul 10, 2025

Uh oh!

mizvekov commented Jul 10, 2025

Uh oh!

zyn0217 commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AaronBallman commented Jul 11, 2025

Uh oh!

Sirraide left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mizvekov commented Jul 9, 2025 •

edited

Loading

erichkeane left a comment •

edited

Loading

rjmccall commented Jul 10, 2025 •

edited

Loading

zyn0217 commented Jul 11, 2025 •

edited

Loading