Why final result is considered the last stat received in the test ?

[guiracine]

I am trying to understand how the network stats test works under the hood. I understand that the test stabilizes as the time goes up but is it the correct thing to do to present the final result as the last stat received ? Shouldn't we do an average of all stat values received ?

We implemented the test in both iOS and js.
In the javascript version we observed that that packet lost ratio could be > 0.0 in the test but the last stat received is == 0.0 so we consider that there is a good connection when this is not the case...Thats why we thought that we should compute the average of the stat values.

[dylanjha]

👍 wondering the same thing, did you try saving the stats and computing averages yourself?

[dylanjha]

I'm experimenting with saving all the report objects and calculating the mean bitsPerSecond and packetLossRatioPerSecond values.

The thing I'm wondering is this line in the "How does it work" section:

During the test period, the video quality will stabilize, based on the available network connection quality.

It might be useful to try:

• plot the data points out over a time series and see how long it takes to stabilize
• discard outliers before calculating the mean so that we ignore early sporadic data points

[wobbals]

I think we'll be taking another look at this project in the coming months. In the meantime, I did start a separate project that uses the same API to accumulate periodic reports and produce less granular result: https://github.com/wobbals/opentok-mos-estimator - still a very rough draft, but it could be useful to get your feedback if you're looking at improving how to process getstats data.

[dylanjha]

@wobbals this looks cool. I would love to help move opentok-mos-estimator forward. For the way it is set up now, what is a reasonable timeframe to run this test and get an estimate?

For our use case, we're looking at running a 20s or 30s test and making an assessment based on that. Would opentok-mos-estimator fit that use case?

[wobbals]

@dylanjha Probably some modifications might be wise for periodic sampling -- I'll shoot you an email and we can discuss.