|
| 1 | ++++ |
| 2 | + |
| 3 | +title = "Announcing Context-Generic Programming (Early Preview)" |
| 4 | + |
| 5 | ++++ |
| 6 | + |
| 7 | +# The Beginning of a New Paradigm |
| 8 | + |
| 9 | +Welcome everyone! This blog post marks the launch of the _context-generic programming_ (CGP) project, |
| 10 | +to introduce a new modular programming paradigm for Rust. |
| 11 | + |
| 12 | +# A Quick Overview of Context-Generic Programming |
| 13 | + |
| 14 | +As its name implied, CGP makes it possible to write _context-generic_ programs that can work |
| 15 | +with any context type in Rust, i.e. the type that we typically refer to as `Self`. |
| 16 | + |
| 17 | +Compared to regular generic programming, CGP supercharges how we can write generic programs, |
| 18 | +by lifting many restrictions imposed by Rust and its trait system. CGP alleviates the needs |
| 19 | +to explicitly specify all generic parameters by position, and allows them to be referenced |
| 20 | +similar to named parameters. CGP also makes it possible to specify overlapping and orphaned |
| 21 | +trait instances, by introducing _provider traits_ replace any reference to `Self` with |
| 22 | +an explicit `Context` generic type. |
| 23 | + |
| 24 | +Compared to object-oriented programming (OOP), CGP makes it possible to write Rust programs |
| 25 | +following popular OOP patterns, such as inheritance and mixins, but in better ways that |
| 26 | +fit Rust's type system. With CGP, one can write highly expressive programs that may look similar |
| 27 | +to dynamic-typed programs, while ensuring that the programs remain type-safe at compile time, |
| 28 | +without any sacrifice to runtime performance. |
| 29 | + |
| 30 | +As a new programming paradigm, CGP significantly changes how Rust programs can be written. |
| 31 | +Because of this, CGP programs may look very different from regular Rust programs, and appear |
| 32 | +intimidating to even experienced Rust programmers. CGP introduces new programming concepts |
| 33 | +in the form of Rust macros, which makes it not as elegant as it could have been if we were |
| 34 | +to introduce them as native language constructs in Rust, or with a whole new programming language. |
| 35 | +Hence, we can think of the current form of CGP as an _experimentation_ for introducing new |
| 36 | +language concepts into Rust, or for programming languages in the future. |
| 37 | + |
| 38 | +It would take too much space of this blog post to give a full picture of how CGP works. |
| 39 | +That would require dedication of an entire website, and several books to cover the entirety of CGP. |
| 40 | +If you are new here, you should check out the [CGP homepage](/) for a proper introduction of CGP. |
| 41 | +Instead of rehearsing the full introduction, this blog post will cover some background about the |
| 42 | +project, the current status, and what to expect from here on. |
| 43 | + |
| 44 | +# How It All Started |
| 45 | + |
| 46 | +My name is [Soares Chen](https://maybevoid.com/soarschen), a.k.a. [MaybeVoid](https://maybevoid.com), |
| 47 | +and I am the creator of CGP. Even though this project is still new to the public, it has been ongoing |
| 48 | +for a while. The work for CGP first started at around July 2022, when I was working on the |
| 49 | +[Hermes IBC Relayer](https://github.com/informalsystems/hermes) at [Informal Systems](https://informal.systems/). |
| 50 | + |
| 51 | +I started developing the techniques used in CGP to help writing large-scale generic applications in Rust. |
| 52 | +At that time, the generic code in our code base all share a large monolithic trait called |
| 53 | +[`ChainHandle`](https://github.com/informalsystems/hermes/blob/master/crates/relayer/src/chain/handle.rs#L398), |
| 54 | +which contains dozens of methods that are hard to implement and also difficult to evolve. I then started |
| 55 | +experimenting on using Rust traits with blanket implementations as a form of _dependency injection_ to |
| 56 | +hide the constraints used on the implementation side. This way, a generic code can require the minimal |
| 57 | +subset of dependencies that it needs, and can be reused by implementations that provide only the given subset |
| 58 | +of dependencies. |
| 59 | + |
| 60 | +As time goes on, I developed more and more design patterns to help further modularize the code, |
| 61 | +which collectively form the basis for CGP. The work I done on Hermes also slowly gets decoupled |
| 62 | +from the main code base, eventually becoming its own project called |
| 63 | +[Hermes SDK](https://github.com/informalsystems/hermes-sdk). If you compare both codebases, |
| 64 | +you may notice that the way context-generic programs are written in Hermes SDK is almost completely |
| 65 | +different than the original Hermes, even though both implement the same functionality. |
| 66 | +Compared to the original version, we are able to extend and customize Hermes SDK much more easily |
| 67 | +to support projects with different very requirements, including host environments, APIs, encodings, |
| 68 | +cryptographic primitives, protocols, concurrency strategy, and many more. |
| 69 | + |
| 70 | +But even before my work at Informal Systems, I have spent over 20 years of my programming journey |
| 71 | +experimenting on various design patterns to enable modular programming. My previous projects include |
| 72 | +the implementation of a [dynamic-typed component system in JavaScript](https://github.com/quiverjs/quiverjs), |
| 73 | +and an extensible |
| 74 | +[algebraic effects library in Haskell using implicit parameters](https://github.com/maybevoid/casimir). |
| 75 | +Compared to my previous attempts, I am hopeful that Rust serves as a sweetspot to be a host |
| 76 | +programming language for modular design patterns, thanks to its advanced type systems as well as |
| 77 | +its rapidly expanding ecosystem. |
| 78 | + |
| 79 | +# Current Status |
| 80 | + |
| 81 | +This blog post serves as an early preview announcement, and kickstarts many efforts that are |
| 82 | +still needed before we can be ready for a full release. |
| 83 | +Previously, I have demonstrated the technical feasibility of various CGP programming techniques in |
| 84 | +Hermes SDK. |
| 85 | +In this new phase, I will start adding documentation and learning resources to help spread the |
| 86 | +knowledge of CGP. |
| 87 | + |
| 88 | +For starter, I have created the [project website](https://contextgeneric.dev) |
| 89 | +and finished the first section of my [first book on CGP](https://patterns.contextgeneric.dev). |
| 90 | +However, there are still a lot more work needed before I can make CGP accessible enough |
| 91 | +to the mainstream programming community. Nevertheless, I would like to make use of this |
| 92 | +early announcement to start building an early adopter community to help me continue |
| 93 | +growing CGP. |
| 94 | + |
| 95 | +Depending on my time availability, it may take a year or more before I am ready for an official release |
| 96 | +of CGP. But in the meanwhile, I will start posting regular updates on my development process, |
| 97 | +which may be of interest for some of you reading this blog post. |
| 98 | + |
| 99 | +# Plans for 2025 |
| 100 | + |
| 101 | +In the upcoming new year 2025, I have many plans laid out to prepare for an official release of |
| 102 | +CGP. This section is less about making promises, but more about making you aware of how much work |
| 103 | +is still needed before you should consider using CGP seriously. |
| 104 | + |
| 105 | +## Finish the CGP Book |
| 106 | + |
| 107 | +The most important goal I have for CGP is to finish writing my first book, |
| 108 | +[Context-Generic Programming Patterns](https://patterns.contextgeneric.dev). |
| 109 | +This book will serve as the minimal knowledge transfer for anyone to fully understand CGP. |
| 110 | +My hope is that the book will help reduce the bus factor of CGP, so that even if I became |
| 111 | +unavailable to continue working on CGP, someone could still use the book as a basis |
| 112 | +to continue the work. |
| 113 | + |
| 114 | +## Improve Error Diagnostics |
| 115 | + |
| 116 | +A critical blocker that makes it challenging for me to teach about CGP is the poor error |
| 117 | +reporting returned from the Rust compiler, when there is any error arise from unsatisfied constraints. |
| 118 | +CGP makes heavy use of blanket implementations to facilitate the wiring of components and provide |
| 119 | +dependency injections. But due to its unconventional use of Rust's trait systems, the error case |
| 120 | +is not handled well by the current Rust compiler. This is a major issue, because without proper |
| 121 | +error reporting, it is very tedious to figure out what went wrong inside the code that use CGP. |
| 122 | + |
| 123 | +To improve the error message from Rust, I have taken the initiative to file issue |
| 124 | +[#134346](https://github.com/rust-lang/rust/issues/134346), and attempted a preliminary fix |
| 125 | +[#134348](https://github.com/rust-lang/rust/pull/134348) that is made of ~30 lines of code. |
| 126 | +Currently, the fix somewhat works, by at least showing sufficient information to allow |
| 127 | +debugging to continue. However, it is not yet general enough to not affect general Rust |
| 128 | +programs that do not use CGP. |
| 129 | + |
| 130 | +I plan to eventually dive deeper into Rust's error reporting code, and write a better patch |
| 131 | +that can report CGP-related errors in better ways. But until I have the patch ready and merged, |
| 132 | +any serious use of CGP would require the use of a fork of Rust compiler that applies my temporary patch. |
| 133 | + |
| 134 | +The progress on improving the error messages is tracked on CGP's GitHub issue |
| 135 | +[#44](https://github.com/contextgeneric/cgp/issues/44), and more information on how to use |
| 136 | +the forked compiler is documented in the |
| 137 | +[CGP book](https://patterns.contextgeneric.dev/debugging-techniques.html#improving-the-compiler-error-message). |
| 138 | + |
| 139 | +## Document the `cgp` Crate |
| 140 | + |
| 141 | +I have done quite a bit of writing about CGP on the project website and the book. But if you |
| 142 | +look at the Cargo documentation for the [`cgp` crate](https://docs.rs/cgp/), you would see |
| 143 | +almost no documentation about any CGP core construct provided by the crate. |
| 144 | + |
| 145 | +A main reason I haven't focused on documenting the `cgp` crate is that I wanted to avoid |
| 146 | +explaining the full CGP concepts inside the crate documentation. Instead, I plan to finish |
| 147 | +the CGP book first, and then provide links inside the `cgp` crate for readers to learn |
| 148 | +about relevant concepts. |
| 149 | + |
| 150 | +That said, I do plan to provide at least minimal documentation inside the `cgp` crate, |
| 151 | +to help onboarding programmers to projects that use the `cgp` crate. |
| 152 | + |
| 153 | +## Public Speaking |
| 154 | + |
| 155 | +An effective way to spread the awareness of CGP is for me to speak about it at Rust conferences. |
| 156 | +I plan to apply to speak at major Rust conferences located in Europe, and hopefully I will get |
| 157 | +accepted into at least some of them. If you know of a conference that I should speak at, do let |
| 158 | +me know about it. |
| 159 | + |
| 160 | +As an alternative, I also consider talking about CGP by sharing video recording on YouTube, or |
| 161 | +by organizing online meetups. However, this would subject to my time availability and interest |
| 162 | +from the community, as producing tech videos is not exactly my area of expertise. |
| 163 | +But in case if you are interested in such content, do let me know what you would like to see produced. |
| 164 | + |
| 165 | +## Improve the CGP Macros |
| 166 | + |
| 167 | +The proc macros provided by the `cgp` crate were written in haste as quick proof of concepts |
| 168 | +to simplify the syntax for writing CGP programs. As a result, they are not that high in quality, |
| 169 | +and do not provide good UX when there are errors in using the macros. In most cases, |
| 170 | +the macro would just panic, and do not provide much clue to users on what went wrong. |
| 171 | + |
| 172 | +The CGP macros are also not comprehensive enough to support all possible ways users may define |
| 173 | +CGP components. For instance, the use of const generics or associated constants may result in |
| 174 | +macro panics. Other than that, there are known bugs when merging generic parameters coming from |
| 175 | +multiple sources. |
| 176 | + |
| 177 | +When I have the time, I plan to learn more about how to properly implement the proc macros, |
| 178 | +and implement them correctly with proper test coverage. This is important to provide good |
| 179 | +user experience, as developers will use the macros all the time when programming in CGP. |
| 180 | + |
| 181 | +## Developer Tooling |
| 182 | + |
| 183 | +For CGP to gain mainstream adoption, it is not sufficient to just make CGP powerful enough |
| 184 | +to solve difficult programming problems. In addition to that, we also need to make CGP |
| 185 | +_easy_ enough for even beginner programmers to easily pick up. And to move toward that |
| 186 | +goal, we can slowly make CGP easier by building better _tools_ to assist programming in CGP. |
| 187 | + |
| 188 | +Although CGP makes heavy use of Rust's trait system to power its component system, the |
| 189 | +heavy machinery are _not_ strictly necessary for its _users_ who use CGP to build modular applications. |
| 190 | +If we were to implement CGP as native language constructs, we could in principle not require |
| 191 | +beginner programmers to understand anything about traits when they start to learn about CGP. |
| 192 | +But even if CGP is not native Rust constructs, there are probably ways for us to build tools |
| 193 | +that provide first class support for CGP. |
| 194 | + |
| 195 | +One way we can provide such support is to build analyzers that give special treatment to CGP |
| 196 | +traits such as `DelegateComponent`. Our tools can then perform analysis on the component |
| 197 | +dependencies directly, and provide help in performing any necessary wiring. |
| 198 | + |
| 199 | +Ideally, I would like to implement IDE features similar to Rust Analyzer, so that most of |
| 200 | +the cognitive burden of wiring CGP components can be automated by the IDE. But it may take |
| 201 | +much longer than one year for me to implement such features. In the meanwhile, I will probably |
| 202 | +explore on simpler options, such as building simple CLI tools for CGP. |
| 203 | + |
| 204 | +## Implement Advanced CGP features |
| 205 | + |
| 206 | +Aside from improving CGP macros, there are a few more advanced core constructs that I need to |
| 207 | +implement in the `cgp` crate to enable CGP to solve more use cases. |
| 208 | +In particular, I plan to introduce constructs for context-generic construction of struct fields |
| 209 | +(product types), and context-generic matching of enum variants (sum types). |
| 210 | +These constructs are commonly needed in complex applications, and currently they are commonly |
| 211 | +solved using OOP patterns such as factories and visitors. |
| 212 | +CGP offers better alternatives than the existing OOP design patterns, but I have yet able to |
| 213 | +find the time to implement and document them. |
| 214 | + |
| 215 | +In case if you have interest in topics such as row polymorphism, datatype-generic programming, |
| 216 | +and category theory, you might be interested to follow my progress on how I make use of these |
| 217 | +advanced concepts in CGP. |
| 218 | + |
| 219 | +## More Documentation |
| 220 | + |
| 221 | +It would be a big milestone if I am able to finish the first CGP book and document the `cgp` |
| 222 | +crate by the end of 2025. But I also hope to write more documentation in other forms, to |
| 223 | +explain CGP in different ways to different audiences. |
| 224 | + |
| 225 | +It would be helpful if I can write some tutorial series for teaching CGP to beginners, or |
| 226 | +for programmers coming from imperative programming background. But it may be challenging to |
| 227 | +write such tutorials, without first improving the toolings and error handling for CGP. |
| 228 | +Alternatively, I may focus on writing use-case oriented series to explain how to use CGP |
| 229 | +to solve real world problems, such as building web applications, training AI models, |
| 230 | +or programming microcontrollers. |
| 231 | + |
| 232 | +On one hand, I would like to avoid giving the impression that CGP is specifically |
| 233 | +designed to solve a specific application domain. On the other hand, it is a bit tough |
| 234 | +to demonstrate on my own how CGP can be used for all kinds of problem domains, while |
| 235 | +I myself is clearly not an expert in all of them. |
| 236 | + |
| 237 | +Perhaps the best way for me to approach this is for the community to guide me on |
| 238 | +what kind of content I should produce for CGP. If you have a specific programming |
| 239 | +problem that you think CGP may help solving, I would love to hear more about it. |
| 240 | +This can help inform me what kind of topics is popular, and allows me to better |
| 241 | +prepared to produce content for those topics. |
| 242 | + |
| 243 | +# How You Can Help |
| 244 | + |
| 245 | +If you have read till the end of this blog post, thank you for taking your time! |
| 246 | +If you are interested in CGP, the project homepage [lists many ways](/#contribution) |
| 247 | +you can help me continue my development on CGP. |
| 248 | +I look forward to see you again in the future updates for CGP! |
| 249 | + |
| 250 | +Following are some links to the discussions on this blog post: (links to be updated) |
| 251 | + |
| 252 | +- [Hacker News](#) |
| 253 | +- [Reddit](#) |
0 commit comments