FR: Record slow cloud functions / triggers and cloud function timeout

[dblythy]

Is your feature request related to a problem? Please describe.
Although analytics aren't supported, it would be nice if we could track "inefficient" cloud functions and triggers, which could be displayed in the dashboard with a message on how to 🚀 functions and maximise efficiency.

Describe the solution you'd like
It would be nice if we could specify a "timeout" value (e.g 5000), so if a function runs for longer than this, it's recorded as a slow function to be optimised, and a timeout is returned to the client from the server.

It might also help to have some sort of timing trace to be provided to help work out where the bottleneck is happening, so we can easier isolate whether performance issues are Parse Server decoding / encoding etc related, or cloud code.

Describe alternatives you've considered
Manually coding into cloud functions

I've worked on a conceptual PR, that tracks the timings of function called, function decoding, cloud validator, cloud function, and response encoding times. It's not ready for merging yet, more of a concept to help understand the idea around this FR. Here is the link.. This could easily be extended to triggers as well.

Here's what a slow function looks like:

{
  params: {},
  error: { message: 'Script timed out.', code: 141 },
  events: [
    { name: '(Parse) function called', time: 0 },
    { name: '(Parse) function decoding', time: 0 },
    { name: 'cloud validator', time: 0 },
    { name: 'cloud function', time: 4794 },
    { name: '(Parse) response encoding', time: 0 }
  ],
  context: {},
  functionName: 'cloudFunction',
  timeTaken: 4794.519228000194,
  master: false,
}

In this example, a cloud function takes nearly 5s to complete, with a Parse Server timeout of 3s (slowTracking.timeout: 3000). Judging by "events", the bottleneck is all coming from the developers' cloud function.

[mtrezza]

Thanks for suggesting.

This is an interesting feature. There are many commercial (e.g. NewRelic) and open-source (e.g. OpenTelemetry) sophisticated app monitoring services available that provide a plethora of metrics and diagnostics. Some are also interoperable with services that monitor underlying infrastructure and clients to achieve 360° monitoring. We should keep in mind what such a feature is competing with, when we think about its viability.

Some thoughts:

if a function runs for longer than this, it's recorded as a slow function to be optimised

This could be similar to a MongoDB slow query report. Reporting slow functions out of the box could be attractive to beginning developers who do not integrate a 3rd party monitoring service. Thinking about the future of Parse Dashboard, a "Slow Query" stats page would be a great addition.

a timeout is returned to the client from the server

This seems to be a completely separate feature from "tracing", because timeout could also be function specific. This seems to ask for a separate discussion to come up with a concept that is applicable for many use cases.

It might also help to have some sort of timing trace to be provided to help work out where the bottleneck is happening

A slow Cloud Code function is still a "black box". To provide more value and enable developers to do the next step, it could be interesting to allow markers in Cloud Code. For example:

Parse.Cloud.define("myFunc", async(request) => {
   Parse.Tracing.setMarker("DB call");
   // Execute DB query

   Parse.Tracing.setMarker("send SMS");
   // Call 3rd party API
}

So that the tracing shows:

events: [
    { name: 'cloud function', time: 4794,
      markers: [
         { marker: 'a', time: 10 },
         { marker: 'b', time: 4784 },
    }
  ],

[dblythy]

Thanks for the suggestions. I could maybe add the timeout separately to the validator object and apply that feature from there. I agree that it is a seperate feature, I was just concerned that functions that take a long time to resolve would never show up in the "slow query" stats page as they were still resolving (is that even possible?).

I like the idea of the markers, and I agree that without them it doesn't offer much in terms of diagnosis. I think combining markers and slow queries could provide good value to developers trying to speed up their servers, without needing to look through logs or write that much new code.

[mtrezza]

functions that take a long time to resolve would never show up in the "slow query" stats page as they were still resolving

Not sure what you are referring to, a function can only show in the slow query report after it resolved or after nodejs server timeout.

Another aspect to consider is that monitoring that covers only Cloud Code functions captures only a fraction of Parse Server performance, or not much at all, if Cloud Code functions are not used. To give a better picture of Parse Server it may make sense to cover jobs, triggers (where often a lot of magic happens), and direct endpoint calls, or at least considering the possibility of a future extension when deciding for a concept now. That may require to rethink the concept of monitoring. A solution that is conceptually restricted to Cloud Code functions may not be viable.

[dblythy]

I planned on extending it to all features mentioned, I just didn't want to jump the gun and build / conceptualise it if it wasn't in line with this projects' goals. Thanks for the feedback!

[dblythy]

@mtrezza is it okay to overhaul some of the promise callbacks and move towards await / async? It will be easier for me to add the functionality.

[mtrezza]

Any code changes that are not directly related to this feature should be done in a separate PR. Also, I think we are still at the stage of a conceptual discussion.

[dblythy]

Ok. I was thinking of having an option for the cloud validator where you can specify slowQuery of a trigger. If the trigger exceeds the slowQuery, then all relevant information is stored in the _SlowQuery db. Such as:

 Parse.Cloud.beforeSave('cloudFunction', async (req) => {
      await hold(500);
      Parse.Cloud.setMarker('db',req);
      await hold(500);
      Parse.Cloud.setMarker('api',req);
    },{
      slowQuery : 1000
    });

It might also require a default slowQuery config option for tracking slow queries that don't go through a cloud trigger.

Ideally, I was thinking of adding in the "tracing" all the way through the function handler from the express routes, such as here. Again, I was thinking that this could help work out internal performance issues vs external, as well as picking up inefficient queries that don't necessarily depend on a trigger (e.g count).

Alternatively, all the work and tracing could be done in triggers.js, but we'd only be able to trace function start / end, as well as any custom markings. Wouldn't contain any "internal" decoding / encoding or other operations.

What are your thoughts?

[lybaiyy123]

This is a great idea and function that can be quickly operated / analyzed in the control panel,
it like LeanCloud.

[mtrezza]

There have been discussions around similar non-core features (analytics, endpoint rate limiting, etc). The fundamental question is always whether a proposed feature should be considered a core functionality of Parse Server and should be:

• (a) in the core codebase or rather an external add-on and
• (b) maintained by Parse Org

To determine (a):

• How likely is it that the feature has a practical application in a professional, scaled production environment?
• Are there open-source 3rd party tools that are providing the same - or maybe even more - functionality?
• How likely is it that the feature will be developed beyond a PoC to reach a start-of-the-art scope?
• Does it technically make sense to integrate the feature into Parse Server?

To determine (b):

• Will the feature likely be of value to a critical number of users in the community (vs. 3rd party tools)?
• How complex is the further development of the feature?

Btw, what @lybaiyy123's screenshot is showing is likely the MongoDB cluster stats, which is the basis for DB query optimization via indices. A server side query performance measurement seems a rather vague, indicative point to start investigations, because perf is influenced by various factors (server resources, DB resources, network latency, etc).

[Moumouls]

Here some thinking about this feature: #7867 (comment)

[mtrezza]

#7867 (comment):

Here a good idea is may to add some interface to allow developers to plug their favorite performance tool.

I think so too. The optional adapter concept is really at the heart of Parse Server. Easily plugging a 3rd party tool into Parse Server and collecting specific Parse Server metrics that way is lightweight for parse server (low maintaining effort) and gives users the flexibility to integrate metrics into a sophisticated tool with little effort.

For example, if you add the default NewRelic driver to Node.js, it already collects useful metrics about transactional times, including the MongoDB Node.js driver, so that is a load of deep-insigt metrics that you get by adding just 1 line of code. However, these are only general metrics and a new "Parse Server Metrics Adapter" could provide additional Parse Server specific metrics to NewRelic.

But I don't want to preempt the outcome of a discussion around the questions mentioned in #7038 (comment).