Solve the problem of mapping Drupal fields to pattern Variables

[cybtachyon]

Summary:

• Find a Drupal pattern data-mapping solution that is a good fit for our specific implementation and tooling

Benefits:

• We provide a solid solution to the problem of data mapping in Drupal 8 with themes using pattern libraries for our users to get started with

To Do:

• Discuss solutions with community
• Select a "best possible" solution for our particular implementation
• Create required issues to implement the solution

One of the hard problems with making Drupal play nice with a pattern library is mapping your backend Drupal entities and fields to the pattern variables. Some patterns expect strings, or arrays, or can even accept complicated objects for rendering that make use of recursive rendering from the parent platform.

This would be relatively simple to solve if we just use schemas for our patterns and imported those at run-time into Drupal. Unfortunately for that approach, many Drupal sites have a complicated set of backend data, entity, and field organization that makes it impossible to map fields 1:1 to a frontend. This also makes it hard to dynamically generate data models in Drupal for all scenarios, since often we’ll want to include values contained in other fields/entities.

1:1 mapping is easier though than tackling data mapping, and there are a couple of solutions that implement it pretty well.

A common solution for data mapping in Drupal 8 is to use “presenter” twig templates that include a pattern from the library and do all the complicated data mapping in the Drupal “presenter” template. This essentially looks like:

Drupal controller, view node response render -> node.tpl.php (presenter data-mapping template) -> twig include patterns/myCustomPattern.twig

One possible solution is to use a Pattern Renderer API service, that allows a configuration tool to display and set replacement tokens for Drupal to manage. The issue with this is that it relies on developers to create those tokens for all of the backend data.

Another possible solution is to have a Drupal module that has a configurable set of data mapping transformations, much like Views, that allows assigning fields to the pattern schema variables.

Here are some of the current open source community attempts at solving this problem, in order of scope from smallest to largest:

• https://www.drupal.org/project/twig_field_value Simply makes it easier to get the raw field values from fields. It doesn't hook into Drupal's preprocessing systems or renderable output. This is in use on some production Drupal 8 sites.
• https://www.drupal.org/sandbox/cybtachyon/2744911 Recreates Drupal 8 Core's rendering subsystem and returns all preprocess variables In use on multiple enterprise Drupal 8 sites.
• https://www.drupal.org/node/2848222 (and https://www.drupal.org/node/2707849 ) An attempt at adding a mapping GUI to the Components module, possibly using a token or node-based editor. This is one of the more flexible approaches, but still requires every single pattern be mapped by a user at least once.
• https://www.drupal.org/project/ui_patterns Full-scope module that imports pattern schemas to create paragraphs and other fielded entities in Drupal. Requires a 1:1 data mapping relationship with Drupal entities and fields.
• https://github.com/PatternBuilder/pattern-builder-drupal Dependent on PatternKit, this full-scope module imports and caches PatternKit patterns and their schemas, and allows them to be integrated as drag-and-drop panels in Drupal 7. Drupal Tokens are used for variable replacement. In use on some Red Hat Drupal 7 projects.

The teams I work with internally already have a number of projects in this vein and are planning on moving forwards with a consolidated project starting August 21st.

cc @ThatGuyCND @EvanLovely @evanmwillhite

[jesconstantine]

@cybtachyon very excited to see this issue over at the Commonwealth of Massachusetts! On the openmass project we are using a custom module (named after our Pattern Lab design system) to handle data mapping, transformation and structuring. It's resulted in a developer-centric process for adding content types and patterns. I'd love to see the community standardize on a method that takes advantage of site building and Drupal's UI to handle some of this.

Something that I think ties into some of the projects you've listed above is using pattern data structure "documentation" or "definition" (via json schema for pattern builder and yml definition files for ui patterns) in that data mapping process. I think it would help the community to standardize on a format for this definition / documentation and have those files live in Pattern Lab (much like the json schema files live in pattern kit).

What I really like about the way that pattern builder / pattern kit uses json-schema is that it serves multiple purposes:

• documentation for implementers / consumers of patterns
• something to validate actual pattern data (json) against
• a blueprint from which to create webforms BOTH in Drupal (i.e. pattern builder module creates paragraph edit forms) AND in PatternKit (i.e. a "schema editor" tab which allows for both content prototyping and pattern variant / assembly in real time) -- imagine the potential here!

For those reasons json schema would get my vote, but I suspect there are ways to accomplish multiple purposes with yml or some other format as well.

Should I create an issue on the roadmap and link to it from here?

[EvanLovely]

I think this is crucial work; thanks for getting the ball rolling! I don't have the bandwidth to dive fully into this, but it's something I want to. That being said, I think a great first start would involve something along these lines:

Essentials

• Docs on what variable names, what types they are (string, boolean, etc), what default is, and a description along with any possible values a String type could take.
• Ability to see that in Pattern Lab alongside the Patterns. I've traditionally used pattern-name.md files alongside the Twig files for this, though I like using them at the group level instead of the pattern level.
  • About "group level" markdown: if you have atoms/buttons/button-primary.twig and atoms/buttons/button-primary.md that shows up for just that button in the little info pane. You don't want to do var docs for each variation of buttons so instead I make atoms/buttons.md and that shows up above all button variations found in that folder, full width, and always open.
• Pattern variations showing which combo of variables results in what end result

Nice to haves

• Instead of markdown for documenting variables, using DocBlock and have something automatically generate the docs. I can't find a "TwigDoc" tool out there, but I bet we could use another languages one.
• Validation of PL data used so we can see early if someone is passing in a string instead of a boolean or whatever.

Really, really nice to have

• Validation of Drupal data being passed in. The aforementioned PL data validation should be a lot easier since it's all in files. Not sure how we'd do this.

Dreams

• The Pattern would have either DocBlock comments or a JSON Schema file and when it was used in a template like node.html.twig or paragraph.html.twig we could run tests that would fail if the Back End passed down the wrong data. That way we could present Back End devs with a clear schema, and ask them to format the data just like that: then they don't even have to understand Pattern Lab and Front Enders don't even have to understand all that goes into creating the data model.

[cybtachyon]

@jesconstantine @EvanLovely Thanks for the initial support and details. I'm not sure we need another roadmap item if this issue covers all the bases.

I'd like to recommend that a good end result for this roadmap issue would be either a new Drupal contrib module or a fork of an existing Drupal module.

There is a matching issue for Pattern Lab schemas here: pattern-lab/patternlab-php-core#117

In the past I've exclusively used Twig DocBlock annotations and the PHP annotation engine to load pattern schemas for a Drupal editor. Personally, I think it'd be great to support DocBlock, JSON, and YAML, although the RH team will be focusing on JSON initially since that's how our internal pattern lib is built out.

We have an initial D7 draft in https://github.com/PatternBuilder/pattern-builder-drupal that we're looking at:

• Add as a Drupal contrib module
• Port to use Pattern Lab instead of Pattern Kit
• Allow selection between either a REST theming service or a local pattern library
• Add support for variable preprocess (either with Twig filters or PHP token filters)
• Moving the PatternBuilder PatternKit REST server to a standalone service that can be integrated with Drupal Core or Pattern Lab etc.

I definitely like https://www.drupal.org/project/ui_patterns but much of it is architected around a Drupal 8 1:1 mapping of data to view model, which just doesn't work at the enterprise level. A bit of a pipe-dream for me would be for PatternKit's Drupal module to become the D7 version with an upgrade path, and for the D8 ui_patterns to be refactored to allow arbitrary data mapping. Part of the work needed for that has been done here by @ademarco :
nuvoleweb/ui_patterns#81

[sghoweri]

I wholeheartedly agree with everything mentioned here - holy crap yes!

By far, this whole data definition / translation / validation piece has been the single most common Pattern Lab + Drupal painpoint we've experienced over the past year in building up and scaling out our design system across our digital properties.

Considering how far Pattern Lab and Drupal integrations have come in the past couple years, sharing the same templates, sharing the same namespaces, sharing the same data structures, and more recently, sharing the same Twig extensions, I’d say this is probably THE biggest yet-to-be-solved problem moving forward, even if it’ll take some work to get us there.

Regarding Markdown-based Solutions

Just as a quick aside, I think markdown is an excellent way to document patterns in a Design System / Pattern Library if the content or data being documented is automagically always up to date with the source of truth (ie. not hard coded).

For example, let’s say you had a card component in your pattern library that allows it’s background color to be set to one of a handful of options that’ve been set in your CSS.

Rather than having to manually copy and paste the data for these options into your markdown docs (and hope your documentation doesn’t inevitably fall out of sync), we’re instead automatically exporting the Sass map data for these background color options as a JSON file to the pattern lab and referencing those values in Pattern Lab AND in our markdown docs.

👆🏼That’s why I put in this still open PR 9 months ago to have Pattern Lab process markdown files for any embedded templates so that things like this are possible:
https://patterns-dev.pega.com/?p=viewall-molecules-cards

So Where Do We Go From Here?

In my opinion, I think we should take a seriously hard look at @micahgodbolt’s Silex-based (and seriously awesome) Pattern Kit project and incorporating the best aspects of it into Pattern Lab.

Pattern Lab needs to evolve to keep up with the massively growing needs of Design Systems big and small.

Top things that come to mind:

1. Modernize the way patterns get rendered. We should be rendering HTML via server-side, path based routes instead of outputting every single pattern as an HTML file in order to address scalability issues with PL and to seriously cut back on the amount of code being changed with even the smallest of template or json file changes. i.e. render patterns like how every other system like Symphony, Grav, Drupal, Silex, etc renders templates.
2. Decouple the server / rendering side of Pattern Lab from the viewing / Interactive side of Pattern Lab. Pattern Kit does a great job with this — that’s a fairly large reason why / how there’s a Drupal 7 module that talks to the Pattern Kit rendering / validation routes to make sure shared templates validate. Pick your shares template, pass in data, validate the data, render, return the results, profit. Why couldn’t Pattern Lab do the exact same thing?
3. Data validation via json schemas. I love love love the idea of having “blueprint” specs for patterns being defined that @jesconstantine mentioned. Stub out the data structure, the things allowed / disallowed to be passed in, etc and automatically generate form for Pattern Lab / Drupal / whatever to pass data into said pattern. Easily the biggest feature in Pattern Kit with tons of examples out there doing this in React, Angular, or just plain vanilla JS
4. Client side rendering (thanks to #s 1, 2, and 3), even if we need to fake it. Thanks to the JSON schemas defined for each pattern + server rendering + ability to pass in data to patterns in Pattern Kit while doing it, doing things like the a client-side schema / data editor are possible

Pattern Lab isn’t Perfect But Neither Is Pattern Kit

All this said, there are still some issues I have with the current state of Pattern Kit — namely missing many a lot of the most important Pattern Lab features like lineages and pseudo patterns, but also, like pattern Lab, being far too opinionated on where patterns, data files, schema files, etc need to live on the file system.

For example:

1. Why can’t we just have a couple base folders to look for pattern-related files in and then find files with the same base pattern name + special file extension and call it a day?
2. Why do all our Twig templates have to live in the same, single _patterns folder in PL?
3. Why can’t I have a schema file live right next to my Twig file in Pattern Kit?
4. Why can’t I bring my own UI to interface with the behind-the-scenes processing Pattern Lab provides?
5. When are we going to be able to get installable, versioned patterns in Pattern Lab or is automatically symlinking the crap out of your Design System the best way to go?
6. Why does all this have to be so difficult?

Given all the really awesome, really exciting ideas being floated around here would be a major departure from Pattern Lab (and given that we still hasn’t seen any PRs reviewed or approved in over a year 😒...), I seriously think @EvanLovely, @evanmwillhite and I need to have a heart to heart video chat with @bradfrost.

I’m really excited by all the awesome progress that has been made by the Drupal Pattern Lab Working Group in the past 5 months — I just think we need to figure out if this growing initiative we have here can still ultimately get merged back into the main Pattern Lab umbrella or if we’re fast approaching a point where this fork has no choice but to start becoming it’s own separate thing...

[cybtachyon]

@sghoweri The RH team already has these issues up for PatternLab that address almost all of your Pattern Lab feedback and suggestions here:

pattern-lab/patternlab-php-core#116
pattern-lab/patternlab-php-core#117

They could definitely use your ➕ 1.

We don't really see this as a major departure from Pattern Lab - rather we see this as a natural evolution for the ecosystem and part of maturing Pattern Lab to a major enterprise-level UX tool.

As far as PR approvals - you can rest assured they won't hold us back. The RH team has an internal deadline of August 21st to fully slot development of whatever tasks result from these issues... whether or not there's a community consensus, as we have a strong business case and need to see them developed. PR approvals or not, our team needs to move forwards, and we'll be sharing our progress with the open source community the best way we can.

[micahgodbolt]

Love seeing all this activity happening around pattern builder/ kit! @cybtachyon you're in Bellevue? When are you hopping over to the Microsoft campus? We should do lunch.

[sghoweri]

we see this as a natural evolution for the ecosystem and part of maturing Pattern Lab to a major enterprise-level UX tool.

@cybtachyon 100% agree, whole heartedly and we may be able to help quite a bit with some of the heavy lifting with this as well.

I don't want to speak for @EvanLovely or @evanmwillhite but I feel the issues brought up here and here, and here (among the others referenced) would be more than welcomed additions to maturing and strengthening the Pattern Lab ecosystem, hands down.

Monorepo Side Thought

As I'm sure you're aware of, there's a lot of moving parts involved with this overarching initiative -- to the point that I'm now wondering if a monorepo setup like Symphony or Babel would make a lot more sense here.

For example, with regards to Issue #117, I'm thinking about what synchronized changes would be required in the Pattern Lab PHP Core, the Pattern Lab Twig Pattern Engine, and the Pattern Lab Styleguidekit Assets Default repos in order to add something like Pattern Kit's Twig rendering engine behind the scenes (both for rendering and for serving up the HTML) which could then open up the door for getting validation, routing, .patternlabrc configs, etc in place...

Integrating a Schema Editor w/ Pattern Lab...

Regarding the schema editor in Pattern Lab's interface itself (assuming the backend was in place to support it), about a month ago, I took a rough first pass at refactoring the PL Styleguidekit Assets Default project's JS into something that'd be a lot easier to work with / build off of (now using Webpack with proper dependency management) in addition to starting the process of replacing the PL interface's templating engine from Mustache to Twig.

There's still some mustache templates left that I'd like to get removed so the entire PL interface would entirely be getting handled via Twig (and I didn't crack open any major JS refactoring just yet) however this could potentially be a good starting place to get a lot of the new front-end architecture required off the ground...

https://github.com/drupal-pattern-lab/styleguidekit-assets-default/tree/feature/modernize-patternlab-js

[evanmwillhite]

This all sounds amazing. I'm on board for helping with any of it that I can, including doing what's best for the Pattern Lab ecosystem. This is very exciting - thanks to all involved.

[EvanLovely]

What's the MVP here? How can we ship something that just barely fits our needs in two weeks?

[cybtachyon]

What's the MVP here? How can we ship something that just barely fits our needs in two weeks?

@EvanLovely Sorry - to clarify, that's when we're planning on finalizing requirements and starting scheduling work (by August 21st).

The RH team has an internal deadline of August 21st to fully slot development of whatever tasks result from these issues

The hope is to have enough feedback that what we're doing won't preclude the deliverables being useful to the open source community, and coordinate efforts with anyone else working these issues.

[bradfrost]

Hey All,
Thanks so much for all this. Further integration with Drupal sounds really exciting and valuable.

Given all the really awesome, really exciting ideas being floated around here would be a major departure from Pattern Lab (and given that we still hasn’t seen any PRs reviewed or approved in over a year 😒...), I seriously think @EvanLovely, @evanmwillhite and I need to have a heart to heart video chat with @bradfrost.

I guess I just want to clarify that I'm not really in the weeds with the Pattern Lab codebase. I follow along and help in a way with the overall strategy and vision of the tool, but I'm fairly ignorant of the nitty gritty. Both @bmuenzenmeyer (Node) and @dmolsen (PHP) are the primary maintainers of the project's code base, and would be better equipped to field questions, PRs, and so on. I also know the Gitter channel is a place where a lot of conversation happens: https://gitter.im/pattern-lab/

[bmuenzenmeyer]

I've been summoned!

I am interested in all this effort, and it jives well with thoughts I've had regarding the state of the codebase and the direction PL should grow to remain useful to existing and new teams, across platforms.

Where I'm most interested in helping / contributing / providing input is in the frontend code, styleguidekit-assets-default. I am a firm supporter of the core tenant Dave imparted in Pattern Lab 2.0 specification that the frontend should be platform-agnostic, allowing both (and any others) platforms to benefit from a shared user experience and continual updates. Therefore, I'd prefer to avoid a monorepo structure if at all possible.

Scanning through this and pattern-lab/patternlab-php-core#117 (comment) it's clear to me there is energy, manpower, and already in-flight effort behind this proposal and further deepening the offering of PL PHP / Drupal. That's 🔥 🔥 🔥 I haven't looked at the code closely enough to understand if it could "just" be a plugin - but I have a feeling the answer is no.

I'd add that there is equal energy, community and support around Node efforts, with a major release on slow-burn, engines for Mustache, Handlebars, Underscore, React, Twig and others in the works... plus a number of plugins, a full-fledged, CLI, etc etc etc. What I am trying to say is that we all have a lot on our plates and I welcome this effort whole-heartedly - with the request that we make it platform-agnostic.

I've seen a couple clear themes in supporting PL, and specifically the frontend, over the years:

• Make it easier for users to customize the frontend. https://github.com/bmuenzenmeyer/plugin-node-uiextension was created to help with this effort, short of a full-scale rewrite or fork of the shared frontend. It would be nice if users could swap in there own templates, however. Something simple like letting users pass in their own templates as an override of the default, if present, would do the trick I think
• Do not minify the shared assets by default. More often than not this seems to stifle understanding. With an upgrade to something like webpack, we could support dev and production modes a bit better.
• Support a base url parameter that allows users to host a PL instance deep within a url structure
• Review issues and PRs on the front end repo for other good ideas.
• my own comment Keep the frontend templates hogan. Switching to Twig feels like it deviates too far from platform-agnostic to me.

Is there an appetite for creating a tracking issue / milestone in https://github.com/pattern-lab/styleguidekit-assets-default to better understand a punch list?

Lastly, I strongly suggest this large change be formally described using the structure Dave setup in https://github.com/pattern-lab/the-spec/. It has served us well in the past and lets us think through major changes methodically and even track their implementation progress. I see no reason why this schema feature being discussed doesn't qualify as a major cross-platform feature addition describing what Pattern Lab is (which will then also cascade into documentation on the .io site, too)

boring personal note

At the risk of sounding like a broken record, I love love this and am excited to see it make it into the light of day for users. My personal life is calming down a bit finally, after client work, a job change, and a move to a different state, so you should be hearing more from me from now on :)

[sghoweri]

@bmuenzenmeyer 👍👍👍👍

Where I'm most interested in helping / contributing / providing input is in the frontend code, styleguidekit-assets-default. I am a firm supporter of the core tenant Dave imparted in Pattern Lab 2.0 specification that the frontend should be platform-agnostic, allowing both (and any others) platforms to benefit from a shared user experience and continual updates. Therefore, I'd prefer to avoid a monorepo structure if at all possible.

Is there an appetite for creating a tracking issue / milestone in https://github.com/pattern-lab/styleguidekit-assets-default to better understand a punch list?

Very fair points. And I'm 100% for keeping the community and the project ecosystems together wherever possible so that the work being done can mutually benefit everyone across the board. Soooo, would you be opposed to tag-teaming on a major revamp of styleguidekit-assets-default?

I'd be more than happy to put some time into writing up my own recommendations for styleguidekit-assets-default from a front-end architect point of view (keeping everyone mentioned here thus for heavily into consideration).

Quite frankly, I just don't want the experience people have with the Pattern Lab ecosystem (developers, designers, UXers, content authors... everybody) feel antiquated, horribly outdated and sluggish compared to the mind-blowingly awesome work being done in the community.

my own comment Keep the frontend templates hogan. Switching to Twig feels like it deviates too far from platform-agnostic to me.

What about something like composable web components, nunjucks, or Vue.js... (assuming we could maintain IE9+ compatibility out of the box)?

---

Scanning through this and pattern-lab/patternlab-php-core#117 (comment) it's clear to me there is energy, manpower, and already in-flight effort behind this proposal and further deepening the offering of PL PHP / Drupal. That's 🔥 🔥 🔥 I haven't looked at the code closely enough to understand if it could "just" be a plugin - but I have a feeling the answer is no.

Right. This is just one out of many many instances @EvanLovely and I have found ourselves butting heads with Pattern Lab's current limitations without many options (besides a lot of hacks and workarounds using Gulp / Node.js)

I'd add that there is equal energy, community and support around Node efforts, with a major release on slow-burn, engines for Mustache, Handlebars, Underscore, React, Twig and others in the works... plus a number of plugins, a full-fledged, CLI, etc etc etc. What I am trying to say is that we all have a lot on our plates and I welcome this effort whole-heartedly - with the request that we make it platform-agnostic.

One crazy absurd idea to throw out there... what if Pattern Lab PHP and Pattern Lab Node didn't need to be two separate projects... for example - assuming we could still use .php Twig extensions in Pattern Lab (so Drupal 8 compatibility is still maintained), couldn't we use something like https://github.com/bitmade/node-twig to use PHP template rendering, but with all other PL logic living in Pattern Lab Node?

Lots more to think and talk about, especially on how schema validation and mapping might work, how we can use the best parts of Pattern Kit as a source of inspiration, and how best to get the ball rolling on all this!

[cybtachyon]

Thanks everyone for the amazing feedback and suggestions.

It looks like we have two hot items that need addressing:

1. A Drupal module that allows mapping data models from Drupal to a pattern library
2. Platform-agnostic Pattern Lab updates to make it work better as a dev-tool dependency of other projects

A Drupal module that allows mapping data models from Drupal to a pattern library

This topic is primarily what this issue was created for, and I have a summary of proposals based on all the feedback so far:

Drupal 7

• Fork https://github.com/SiliconValet/pkplugins
• Create a stripped-down Drupal.org contrib release
• Add support for local pattern libraries, not just a REST API
• Add variable preprocess chaining support (to allow full data mapping)

Drupal 8

• Fork https://github.com/nuvoleweb/ui_patterns and add an issue to https://www.drupal.org/project/ui_patterns to merge in https://www.drupal.org/node/2848222
  or
• Fork https://github.com/SiliconValet/pkplugins to create a D8 port

One consideration is that ui_patterns could be too heavy weight (6 modules, plus lots of plugins) to bake into a best practices Drupal / Pattern Lab integration or add to core as an experimental solution for ingesting patterns.

Platform-agnostic Pattern Lab updates to make it work better as a dev-tool dependency of other projects

This is a bit of a topic derailing but it is important to making this issue work as well as developing the ecosystem. As @bmuenzenmeyer and @sghoweri recommended, we need the following issues:

• Make Pattern Lab Configurable via a .patternlabrc vs patternlab-config.json and ./config/config.yml pattern-lab/patternlab-php-core#116 Make Pattern Lab Configurable via a .patternlabrc vs patternlab-config.json and ./config/config.yml
• Add a Sample Engine with Schema Support to Pattern Lab pattern-lab/patternlab-php-core#117 Add a Sample Engine with Schema Support to Pattern Lab
• Create a tracking issue / milestone in https://github.com/pattern-lab/styleguidekit-assets-default

If we're able to reach a consensus, we can go ahead and make the necessary issues, forks, and close this one out.

[micahgodbolt]

just wanted to loop @andrewmriley into this discussion.