CLAID EMBC2024 - Part 4: Deploying Machine Learning Models

Part 3 has taught you how you can add custom Modules to CLAID. Building on Part 3, Part 4 will cover how you can deploy machine learning models with CLAID.

Objective for Part 4

The goal for Part 4 is to design a simple intervention using machine learning. You will learn how to deploy machine learning models with CLAID, which can process data livestreamed from the smartwatch. Using the result of a deployed machine learning model, we want to display an intervention to the user (e.g., for him to "move more").

Human Activity Recognition

Human Activity Recognition (HAR) is the process of automatically identifying and classifying actions or movements performed by individuals, utilizing data from sensors like accelerometers or cameras. Many research papers have been published in recent years, showing superior performance of HAR using advanced machine learning techniques. For this workshop, we want to use one of the publicly available models for human activity recognition and integrate it with CLAID. To speed up the process, we have already done the research and downloaded a suitable model based on an LSTM architecture, which is able to recognize 3 activies (Jumping, Standing, Walking) based on 5 seconds of acceleration and gyroscope data recorded at 20 Hz:

Further, we prepare some minimalistic code, which allows to load the model and apply data. Code and model are available from the CLAIDWorkshopEMBC2024 folder:

The Python file "human_activity_recognizer" provides a class HumanActivityRecognizer, which allows you to load the model and feed it with data:

HumanActivityRecognizer

class HumanActivityRecognizer:

    # Instantiates and initializes the machine learning model.
    def initialize_recognizer()
    # Takes in acceleration data and gyroscope data and performs activity recognition,
    # by applying the data to the ML model.
    # The Model expects 5 seconds of data at 20 Hz for the 3 axis Accelerometer and Gyroscope.
    # Thus acceleration_data and gyroscope data both need to be a numpy array of shape [3, 100].
    # (index 0 = X-axis, index 1 = Y-axis, index 2 = Z-Axis).
    # Returns a one-hot encoded array with scores for the corresponding classes "Jump", "Stand", "Work".
    def run_inference(acceleration_data, gyroscope_data)    
    # Returns a list of the available labels in the correct order, i.e., ["Jump", "Stand", "Walk"]
    def get_labels()
    # Decodes the model data and returns the recognized label:
    def get_label(model_out)

Your task: Design an intervention based on HAR

Your task for Part 4 of the workshop is to create a CLAID Module in Python, which takes in the acceleration and gyroscope data (live) from the smartwatch and uses the HumanActivityRecognizer to classify the current activity. Once you have the classification running, you can use the recognized activity (label) to design an intervention using any of the UserFeedback Modules that were introduced in Part 3. For starters, you could let your HARModule output the label and connect it to the TextToSpeechModule. This will make the smartwatch "speak out" the activity you are currently performing. Your HARModule in the end should look as follows:

Choose task: From the list below, please choose the task according to your skill level and confidence.

Option1 for beginners: Copying and registering the HAR Module

Inside of the template folder, you can find a HARModuleWorking.py file which contains a fully working HARModule:

Simply copy this file to the "my_modules" folder, to register it to CLAID.

Option2 for beginners: Completing the HAR Module

To help you get started, we have prepared a template for the Module, which you can find in the file "HARModuleTemplate" inside the folder "CLAIDWorkshopEMBC2024":

You can start the implementation of your own HARModule based on this template. The steps you would still need to do are:

1. Store incoming acceleration and gyroscope data (check the separate functions on_acceleration_data and on_gyroscope_data).
2. Once you have 100 acceleration and 100 gyroscope data, call the function run_inference_if_enough_data.
3. Inside the function run_inference_if_enough_data, uncomment the existing code and feed acceleration data and gyroscope data to the recognizer. Arrange your acceleration data and gyroscope data each in the shape (3, 100).
4. Retrieve the output label of the model using recognizer.get_label() as shown in the code. Post the label to the output Channel using self.output_channel.post(label)
5. Clear the list of stored acceleration and gyroscope data, or at least remove some of the oldest samples. Since the Accelerometer and Gyroscope are required to run at 20Hz for the provided machine learning model,
6. Load up your Module in the CLAID Designer and connect an AccelerometerCollector and GyroscopeCollector running at 20Hz frequency. For best performance, use the BATCHED mode.

If you are unsure, you can have a look at the HARModuleWorking.py file inside the template folder, which contains a fully working HARModule.

Option3 for experienced workshop participants: Creating the HARModule from scratch

1. Create a new Python CLAID Module called HARModule (see instructions in Part 3 on how to create a new Module).
2. Add two input Channels and one output Channel to the Module, just as in the picture above. The input Channels need to have data types AccelerationData() and GyroscopeData(). The output channel should have data type str() (string).
3. Instantiate the HumanActivityRecognizer and initialize it inside the initialize function of your Module.
4. Store incoming acceleration and gyroscope data in the respecting callback functions of the input channels. To run inference on the model, you need 5 seconds of acceleration and gyroscope data recorded at 20 Hz (i.e., 100 samples per sensor in total).
5. Once you have 100 AccelerationData and 100 GyroscopeData samples, arrange each of them in a shape (3, 100) and feed them into the HumanActivityRecognizer model.
6. Retrieve the label of the model and post it to the output channel.

If you are unsure, have a look at the HARModuleTemplate.py inside the template folder of the "CLAIDWorkshopEMBC2024" repository. This file contains a basic HARModule with all Channels being set up correctly. If you are still unsure, you can have a look at the HARModuleWorking.py file, which contains a complete HARModule.

Important: In case you do not know how to start, please check out the "templates" folder in the CLAIDWorkshopEMBC2024 directory. There you can find a file "HARModuleTemplate.py", which provides basic code for an HARModule with the correct input channels as seen in the picture above.

Bonus: Closing the Loop - Deploying everything on the Smartwatch alone

CLAID is designed in a way that you can deploy machine learning models either on a server (or Laptop, in our case), or on edge devices (like smartphones or smartwatches). If you have a ML Model running on your server in the form of a CLAID Module, you can simply migrate that Module to a mobile device without having to write any additional code. Specifically, you can upload your Module to a Python Runtime running inside a CLAID Module on the Smartwatch, a process we call injection.

To inject your HARModule, you have to do two things:

1. Make the Module injectable: Tell the ModuleAnnotator that your Module supports injections.
2. Declare any dependencies: Specify any dependecies that your Module might have. In the case of the HARModule, this would be any additional code (i.e., the human_activity_recognizer) as well as the machine learning model. Those dependencies will be injected together with the Module.

To make these two changes, you have to update the annotate_module function of your Module. Check out the snippet below.

Module annotation for injectable HARModule

@staticmethod
def annotate_module(annotator):
    annotator.set_module_category("Custom")
    annotator.set_module_description("A Module allowing to do human activity recognition using machine learning.")

    annotator.describe_subscribe_channel("AccelerationInputData", AccelerationData(), "Acceleration input data")
    annotator.describe_subscribe_channel("GyroscopeInputData", GyroscopeData(), "Gyroscope input data")
    annotator.describe_publish_channel("OutputDataLabel", str(), "Output data")

    annotator.make_injectable()
    annotator.add_dependency("human_activity_recognition/human_activity_recognizer.py")
    annotator.add_dependency("human_activity_recognition/models/har_model.pb")

If you are unsure, ask the workshop team. We can provide you with a working HARModule with everything set up correctly. After you have updated the annotation, restart the CLAID Designer and load up your previous configuration. Now, simply drag and drop the HARModule from the Laptop into the Smartwatch. Check out the video below.

When asked to upload the Module and all the dependencies, click yes. Afterward, reprogram the devices (you can leave the Laptop empty, if you like).

After you press program, please allow up to 50 seconds for the Smartwatch to reconfigure itself. Injecting the Machine Learning model and instantiating it takes some time.

Workshop and CLAID Evaluation (5 minute questionnaire)

Dear participant,

thank you for taking part in the CLAID workshop!
We hope you enjoyed it. In order to improve the workshop and CLAID in the future, we have prepared a few questions for you to provide feedback. We greatly appreciate if you could fill out this questionnaire. It is completely anonymous and should not take more than 5 minutes to complete.

You can find the questionnaire here.

Thank you very much.