form-analysis.md

Form Analysis

Method:

We take an algorithmic approach to caclulate a score for a rep of a certain exercise. Given a time series of 3D keypoints, we calcuate angle between joints, velocity, and distance between joints. To calculate the score, we compare these values to a predefined range of 'perfect' values for a specific exercise

Skeleton Keypoints:

We use the Human3.6m 17 keypoints

Joint Number	Joint Name
0	Root
1	Right Hip
2	Right Knee
3	Right Ankle
4	Left Hip
5	Left Knee
6	Left Ankle
7	Belly
8	Neck
9	Nose
10	Head
11	Left Shoulder
12	Left Elbow
13	Left Wrist
14	Right Shoulder
15	Right Elbow
16	Right Wrist

Rep Segmentation

We split a video of mutliple repeititons of the same exercise into individual clips containing a single repetition.

These clips are used later for analysis and tracking form across a whole set of reps.

The algorithm we used is derived from AIFit. We first achieve an estimate using a fixed clip length and optimize to find individual clip start and stop frames.

Initialization

To obtain an initial estimate of the segmentation, we follow these steps:

1. Assume a fixed-period pose signal: $T_{init}=T(t_{start},\tau )=T_i|t_i=t_{start}+(i-1)\tau$

   • $\tau$: period
   • $t_{start}$: starting point of repetitions

2. Define affinity between two 3D poses: $A(p_m,p_n)$ = negative mean per joint position error (MPJPE) between $p_m$ and $p_n$

3. Determine initial period estimate ${\tau }^{\ast }$ using auto-correlation: $R_{PP}(\tau ,s)=\frac{1}{N-2s-\tau }\sum _{t=s}^{N-s-\tau }A(p_t,p_{t+\tau })$

   • $s$: signal shrinkage at both ends to account for noise
   • $N$: total number of poses

4. Iterate over $s$ and $\tau$ to find ${\tau }^{\ast }$:

   • Select smallest $\tau$ where $R_{PP}(\tau ,s)$ reaches a local maximum
   • Corresponding $s$ becomes $s^{\ast }$

${\tau }^{\ast }$ represents the period that maximizes the auto-correlation of the signal, accounting for noise outside repetitions.

Finding the Start of the First Repetition

After estimating the period $\tau^$, we search for the beginning of the first repetition $t_{start}$ by maximizing the average affinity $A_{avg}$ of $T(t_{start}, \tau^)$:

$A_{avg}(T)=\frac{1}{k_{min}^2}\sum _{i=1}^{k_{min}}\sum _{j=1}^{k_{min}}{A}_{seq}(T_i,T_j)$

where:

$A_{seq}(T_i, T_j) = \frac{1}{\tau^} \sum_{l=1}^{\tau^} A(p_{t_i+l}, p_{t_j+l})$

• $A_{seq}(T_i,T_j)$ computes the similarity between two repetitions of equal period ${\tau }^{\ast }$ (intervals $T_i$ and $T_j$).
• $A_{avg}(T)$ averages similarities between all possible pairs of intervals, representing a global affinity of the repetition segmentation $T$.
• At this stage, $T$ is parameterized only by $t_{start}$, since ${\tau }^{\ast }$ was found in the previous step.

We select $t^{start}$ as the smallest value for which $A{avg}(T(t_{start}, \tau^))$ has a local maximum. This approach:

1. Provides the highest similarity between repetitions.
2. Uses the smallest such maximum to prevent solutions like the beginning of the 2nd/3rd/etc. interval, which are also local maxima.

Calculations:

Metric	Formula
Angle	$\theta ={\cos }^{-1}\left(\frac{\mathbf{a}\cdot \mathbf{b}}{\Vert \mathbf{a}\Vert \cdot \Vert \mathbf{b}\Vert }\right)$
Velocity	$\mathbf{v}{P{F_2/F_1}} = \frac{d\mathbf{r}_2}{dt} = \frac{d(\mathbf{r}1 + \mathbf{r}{1P})}{dt} = \frac{d\mathbf{r}1}{dt} + \frac{d\mathbf{r}{1P}}{dt}$
Distance	$d=\sqrt{(x_2-x_1{)}^2+(y_2-y_1{)}^2+(z_2-z_1{)}^2}$