Access Static Resources And Compile Into Executable

[sardar01]

MVC web frameworks such as Rails and Django typically have a static directory for storing static resources such as images, css and javascript files. In Drogon, it appears that these resources cannot be specified and statically linked into the executable. Evidently <%view > cannot be used in csp files in order to "include" these resources in a HTML page.

A link such as <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script> for example, works perfectly in Drogon but it does not appear to be accessible when the path is local, such as for example: <script src="static/javascript/jquery.min.js">. Am I missing something or is this a missing feature? The second issue is that these types of static resources are typically cached upstream or served by a specialized server. How would Drogon enable such caching. Thank you.

[an-tao]

There are saveral options controlling accessing static resources in drogon.

"document_root"
"locations"
"static_files_cache_time"

please refer to the comments in the configuration file.
for example of the jquery file. you could set the "document_root" to somewhere, etc. "/usr/local/www" and store the js file as "/usr/local/www/static/javascript/jquery.min.js"
the locations option controls more behaviors of accessing static resources.
the use_sendfile, use_gzip, use_brotli, gzip_static and br_static options are also related to static files. please see the configuration file.

[sardar01]

Thank you, I should have looked more closely at the config file.

…

On Tue, Jul 21, 2020 at 9:51 PM An Tao ***@***.***> wrote: There are saveral options controlling accessing static resources in drogon. "document_root" "locations" "static_files_cache_time" please refer to the comments in the configuration file. for example of the jquery file. you could set the "document_root" to somewhere, etc. "/usr/local/www" and store the js file as "/usr/local/www/static/javascript/jquery.min.js" the locations option controls more behaviors of accessing static resources. the use_sendfile, use_gzip, use_brotli, gzip_static and br_static options are also related to static files. please see the configuration file. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#516 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQEHCE2BQN624EQRTWLTGGTR4ZA3LANCNFSM4PEAJK7Q> .

[rbugajewski]

@sardar01 Regarding your first question about static assets @an-tao already answered the question.

Regarding your second question: Drogon currently doesn’t have any sophisticated asset handling. I’m working on better CDN support for assets & adding preprocessor pipelines for CSS/JS, so that one can better integrate tools like Sass/Less or TypeScript/CoffeeScript, but it is currently just on my private branch and very far away from a public release. The goal is to stick to the standard library paradigm and don’t introduce unnecessary abstraction / computation as far as possible.

[an-tao]

@rbugajewski @sardar01 In my opinion, the cache related headers can control the behavior of caching servers.
you could set static resources headers by the static_file_headers option.

        //static_file_headers: Headers for static files
        "static_file_headers": [
          {
                "name": "field-name",
                "value": "field-value"
          }
        ]

[rbugajewski]

@an-tao Thanks for the info. The cache related headers are actually the problem here. In order to improve loading performance, you need a pretty aggressive cache. Many enterprise CDNs have these values very high by default and some even don’t let you change them under a certain threshold. It also takes time for the content to replicate around the world to become available in a distributed fashion across the whole CDN, which is more of a technical restriction of the CDNs itself, but still relevant to this discussion. In general this isn’t an issue, because assets don’t change very often, but when they change you want all clients to use new assets. Because you can’t control browser caches, but have pretty high values in the cache related headers, you have an issue. The only solution you have now is to rename the asset to a filename you are sure wasn’t previously used, and edit your CSP pages everywhere the asset is used which can be a very tedious and boring job depending on the project size. This way you can make sure that the client will get the new asset, as the filename is completely unknown to the browser’s cache.

Use Case

1. You are working in an enterprise business on a digital store front. Assets get deployed on a CDN with a cache policy of 30 days.
2. Legal comes to you and tells you to change a product image until the end of the week, because the current one violates the copyright.

To handle this use case I have the following ideas and already started to work on a couple of them:

1. Configure asset paths / filenames. When you configure a path, everything (files & recursively directories’ files) inside it will be considered as an asset. It should be possible to configure separate files. This should be part of a separate JSON configuration file. Optional: A deploy directory / filename can be configured for every source directory / filename. Otherwise the source directory / filename will be used for the deploy path.
2. Reference assets in CSP pages / source code. In CSP pages this can be done either via a new custom tag or by introducing a new function (e. g. asset("icon.svg")), and I’m not sure which interface would be better here. It should be possible to reference assets in source code. This should be a constant in the production environment in order to keep performance at maximum. During development it should be possible to just replace the file in the source folder, as it would be annoying to deploy an asset during prototyping. This makes it necessary to make a dynamic lookup, so it should be only used in development mode. Optional: Make the dynamic lookup configurable as optional.
3. Deploy assets. This should be part of drogon_ctl, take the configured paths / filenames, and copy them to the specific deploy destinations. I think it makes sense to use the file’s hash appended to the filename, e. g. icon-HASH.svg. It should be possible to deploy all assets or just a specific one. Optional: Use different strategies, like incremental numbering, e. g. icon-1.svg, icon-2.svg, …, but this would require to have some additional JSON stored as part of drogon_ctl that would keep state of the files & increments. It would be possible to generalize this and allow the user to add their own strategies, but I’m not sure what the benefit is over choosing the best implementation in the first place.
4. Asset component is optional. The functionality should be treated as an additional module, be off by default, and configured in the main JSON configuration file.

Do you have some feedback? What do you think about adding this functionality to Drogon once I finish work on this, and clean it up?

[sardar01]

If my memory serves me correctly, the Rails asset pipeline consists of joining all of the static resource files in a directory tree. This joined file is then minimized for size and given a "fingerprint" name which is a cryptographic hash of the minified fie contents. This is meant for deployment in production environments only.

Regarding referencing assets in a CSP file, we can note that these files will often have to include custom tags which have a special meaning for front-end frameworks such as for example, Vue. So using tags keeps with this "HTML" design philosophy. On the other hand if we use function calls to reference individual assets, we open up the ability to manipulate the asset at compile time with function arguments. It appears that the second alternative is better.

The application that I am working on is a lawyer's trust accounting application. Drogon + Vue + Websockets. Previously it would have been done in Ruby/Rails or Python/Django.

[an-tao]

@an-tao Thanks for the info. The cache related headers are actually the problem here. In order to improve loading performance, you need a pretty aggressive cache. Many enterprise CDNs have these values very high by default and some even don’t let you change them under a certain threshold. It also takes time for the content to replicate around the world to become available in a distributed fashion across the whole CDN, which is more of a technical restriction of the CDNs itself, but still relevant to this discussion. In general this isn’t an issue, because assets don’t change very often, but when they change you want all clients to use new assets. Because you can’t control browser caches, but have pretty high values in the cache related headers, you have an issue. The only solution you have now is to rename the asset to a filename you are sure wasn’t previously used, and edit your CSP pages everywhere the asset is used which can be a very tedious and boring job depending on the project size. This way you can make sure that the client will get the new asset, as the filename is completely unknown to the browser’s cache.

Use Case

1. You are working in an enterprise business on a digital store front. Assets get deployed on a CDN with a cache policy of 30 days.
2. Legal comes to you and tells you to change a product image until the end of the week, because the current one violates the copyright.

To handle this use case I have the following ideas and already started to work on a couple of them:

1. Configure asset paths / filenames. When you configure a path, everything (files & recursively directories’ files) inside it will be considered as an asset. It should be possible to configure separate files. This should be part of a separate JSON configuration file. Optional: A deploy directory / filename can be configured for every source directory / filename. Otherwise the source directory / filename will be used for the deploy path.
2. Reference assets in CSP pages / source code. In CSP pages this can be done either via a new custom tag or by introducing a new function (e. g. asset("icon.svg")), and I’m not sure which interface would be better here. It should be possible to reference assets in source code. This should be a constant in the production environment in order to keep performance at maximum. During development it should be possible to just replace the file in the source folder, as it would be annoying to deploy an asset during prototyping. This makes it necessary to make a dynamic lookup, so it should be only used in development mode. Optional: Make the dynamic lookup configurable as optional.
3. Deploy assets. This should be part of drogon_ctl, take the configured paths / filenames, and copy them to the specific deploy destinations. I think it makes sense to use the file’s hash appended to the filename, e. g. icon-HASH.svg. It should be possible to deploy all assets or just a specific one. Optional: Use different strategies, like incremental numbering, e. g. icon-1.svg, icon-2.svg, …, but this would require to have some additional JSON stored as part of drogon_ctl that would keep state of the files & increments. It would be possible to generalize this and allow the user to add their own strategies, but I’m not sure what the benefit is over choosing the best implementation in the first place.
4. Asset component is optional. The functionality should be treated as an additional module, be off by default, and configured in the main JSON configuration file.

Do you have some feedback? What do you think about adding this functionality to Drogon once I finish work on this, and clean it up?

This is very useful functionality, I'm looking forward to this.
Can it be added as a plugin (such as AssetPlugin)? if needed, we can add some AOP advices for it (for example, PreStaticResources and PostStaticResources).

[rbugajewski]

If my memory serves me correctly, the Rails asset pipeline consists of joining all of the static resource files in a directory tree. This joined file is then minimized for size and given a "fingerprint" name which is a cryptographic hash of the minified fie contents. This is meant for deployment in production environments only.

Yes, that’s what most web frameworks I know do to generate JavaScript and CSS files. This shouldn’t be a big deal from the perspective of Drogon, as most of the functionality is part of the other tools in the pipeline (like TypeScript or Sass). We should just make sure to properly integrate these kind of tools into Drogon’s pipeline, where most of the functionality should reside in drogon_ctl.

Regarding referencing assets in a CSP file, we can note that these files will often have to include custom tags which have a special meaning for front-end frameworks such as for example, Vue. So using tags keeps with this "HTML" design philosophy. On the other hand if we use function calls to reference individual assets, we open up the ability to manipulate the asset at compile time with function arguments. It appears that the second alternative is better.

I agree that a custom tag is in line with the developers’ expectations. I would go for a custom tag in this case. You could then always use <%c++ /* code */ %> in the CSP template to access the Asset API in case you need advanced behavior. I think that we would have the best of both worlds with this solution.

Can it be added as a plugin (such as AssetPlugin)? if needed, we can add some AOP advices for it (for example, PreStaticResources and PostStaticResources).

Yes, that’s in fact how some parts are already implemented, but I don’t see a way for plugins to extend the template syntax with additional tags, so this would probably have to be done outside of the plugin?

[an-tao]

@rbugajewski What do you mean when you said extending the template syntax with additional tags? could you show me a example of it?

[rbugajewski]

@an-tao I was thinking about special markup symbols like they are described in the view section of the official documentation, as it is currently the case with [[ and ]] and so on. This would greatly benefit the usage of assets in CSP pages, e. g.:

• <img src="NEW_MARKUP_SYMBOLS_HERE">
• <script src="NEW_MARKUP_SYMBOLS_HERE"></script>

What do you think?

[an-tao]

@an-tao I was thinking about special markup symbols like they are described in the view section of the official documentation, as it is currently the case with [[ and ]] and so on. This would greatly benefit the usage of assets in CSP pages, e. g.:

• <img src="NEW_MARKUP_SYMBOLS_HERE">
• <script src="NEW_MARKUP_SYMBOLS_HERE"></script>

What do you think?

Great! I see what you mean. How about introducing a configuration object into csp files? e. g.:

<%config=./viewconfig.json%>
...
<img src="<%config.img.image1%>">

viewconfig.json
{
 "img":{"image1":"path/to/an/image"}
}

and then we can change static resources without rebuilding anything.

[an-tao]

Another requirement may be related to this, can it also solve the multi-language support?

[sardar01]

I agree that a custom tag is in line with the developers’ expectations. I would go for a custom tag in this case. You could then always use <%c++ /* code */ %> in the CSP template to access the Asset API in case you need advanced behavior. I think that we would have the best of both worlds with this solution.

Yes, I do believe that you are right.

Regarding configuration files:

<%config=./viewconfig.json%>

<img src="<%config.img.image1%>">
<img src="<%config.img.image1%>">

viewconfig.json
{
 "img":{"image1":"path/to/an/image"}
}

This is very cool. I presume, that these JSON files (their properties) can be written/rewritten at compile time and also at run-time. And classes of objects (for example, images) can be associated with a particular config file.

Are you also thinking (alluding?) to the ability to define a new CSP markup symbol from a subject JSON file? Appears to be a very ambitious idea.