A computer screen is locked with the message “Workout time you lazy BUM” and a person is doing pushups next to it. A closeup shot of the raspberry pi display is added to show the model output and pushup count.
The whole solution in action
A diagram showing the the OAK-1 running the blazepose model, connected to a raspberry pi running a custom classifier, which over SSH connects to the PC, which in turn locks the screen.
The pushup counter architecture diagram. Blue bubbles are ML models

Locking the computer

  1. Locking the screen automatically is easy, unlocking it again is hard.
  2. Almost every lock screen uses the password from the main system account
A meme showing a person locked behind a fence and shouting LET ME IIIIIIN
The same diagram as above, but the right side marked in red as solved by i3lock + xdotool

The machine learning

Blazepose

An example of pose estimation, on the right a person doing some poses, on the left the digital skeleton the model produces in 3d space.
The vitruvius man, with markings on their face and torso of what the model keeps track of
Image credit blazepose paper: https://arxiv.org/abs/2006.10204
A diagram showing how blazepose first runs a pose detector and then for each next frame only run the pose tracker
Image credit blazepose paper: https://arxiv.org/abs/2006.10204

OAK-1

The OAK-1 sitting next to a coin for comparison
The OAK-1
The model size output of Netron, a model inspection tool, which shows a bunch or output layers and their sizes. None of which fit the original model description

Labeling the data

The labeling tool

A list of supported Voice Recogntion engines
The output of the labeling tool, frequently misunderstanding label me as: “Lidl Lidl Lidl” or “Riddle me”. As well as some profanity it picked up from myself in reaction.
The output of the labeling tool, frequently misunderstanding label me as: “Lidl Lidl Lidl” or “Riddle me”. As well as some profanity it picked up from myself in reaction.

Dataset version management

The first part of the diagram from above along with a visual representation of the different steps in the pipeline. Labeling data goes to 1 dataset version, which gets picked up by preprocessing code, which leads to a second dataset version, which then is picked up by training code.
The ClearML experiment manager interface. Each line in a list is a tracked task.
The ClearML Experiment Manager
A picture of a man in pushup position with all landmarks overlayed. On the right the same picture with less landmarks
Removed unnecessary landmarks
The ClearML details view showing sample data in the debug images section
ClearML Debug Samples interface

Training the classification model

Experiment Manager

A meme with a clean desk labeled “local filesystem” and a very, very unorganized desk labeled “clearml project archive”
The ClearML experiment manager interface. Each line in a list is a tracked task.

Data augmentation

A side by side comparison of a normal image and its augmented version

Training and Hyperparameter Optimization

A meme of press x to doubt
  1. Only fit that scaler to the training data! Same rules as the model itself.
  2. Save the scaler itself as well as the model file. Use that same fitted scaler on any new incoming data, so there are 2 model files to keep track of.

Deploying everything

The model

The UI

The interface. On then left a preview of the video feed, on the right the current pushup count, whether the counter is currently active and the output class of the classifier.

The raspberry pi display