DataCollection package¶

This package offers functionalities for serializing data to different data types, writing data to files, configuring data collection rates (in terms of intervall rates or periods) and synchronizing data with a server. The packages offers Modules, Serializers and data types which are required for data collection and upload.

Included Modules¶

This package includes three Modules:

The DataSaverModule can be used to store data from channels to files on a local storage (internal memory or external sd card).
The DataSyncModule can be used to synchronize files ("upload") with another instance of CLAID, e.g., running on a server. Has to be used together with the DataReceiverModule
The DataReceiverModule is used to receive files from the DataSyncModule, which will be stored in a folder as specified.

Important information regarding the Android/WearOS and iOS file systems

All three Modules require to specify paths to where to store files or from where to synchronize files. Compared to a regular operating system (like Linux, macOS or Windows), there are a few platform-specific things to consider on Android/WearOS and iOS:

Android & WearOSiOS

On Android, the internal storage is traditionally referred to as /sdcard/ (i.e., the internal memory of your phone), while an inserted SD card is referred to as "ext_sdcard". Do not confuse /sdcard/ with a micro SD card that you inserted to your phone, for example. The micro SD card would be referred to as ext_sdcard.

Further, Android 11 introduced a concept called scoped storage. This shall enforce Apps to only be able to store data in an app-specific directory provided by the operating system (OS). This prevents apps from accessing data produced by other apps, as every app has their confined space, similiarly to the sandboxed storage on iOS. This prevents us from storing files on /sdcard/ or /ext_sdcard/. To overcome this, the app has to request to act as an external storage manager. CLAID provides the permissions package, which helps requesting this permissions.

On iOS, each app is traditionally sandboxed and can not store data in any globally accessible folder. Hence, iOS apps cannot easily access data stored by another app on the file system. However, iOS introduced the files app with iOS 11. This app allows to view documents data of any app, as long as the app stores the data in their corresponding documents directory. If a CLAID app stores data to that directory, you can see the recorded data from the file system using the files app. On iOS, the DataSaverModule will automatically try to store data in that directory.

DataSaverModule¶

The DataSaverModule can subscribe to any data channel and serialize incoming data to files. For the serialization, a corresponding Serializer has to be registered, which will be invoked with the incoming data and serializes it to the desired format, wich is then written to a file. A typical configuration looks as follows:

Example configuration

<Module class="claid::DataSaverModule">
    <save>
        <!-- Name of the channel with incoming data. -->
        <what>AccelerometerData</what>
        <!-- Naming of the files within the storage or tmp folder.-->
        <!-- You can use to automatically name files based on the recording time -->
        <!-- See here for more details: https://en.cppreference.com/w/cpp/chrono/c/strftime -->
        <!--e.g.: data_29.08.2023_12 for data received from hour 12 to 13 -->
        <fileNameFormat>data_%d.%m.%y_%H.csv</fileNameFormat> 
        <!-- Storage path on where to store the files. -->
        <storagePath>/sdcard/Data/AccelerometerData</storagePath>
        <!-- Temporary storage path  -->
        <tmpStoragePath>/sdcard/tmp/AccelerometerData</tmpStoragePath>
        <!-- Choose the serializer that is used to serialize the data. -->
        <!-- Common options are XMLSerializer, CSVSerializer or BinarySerializer. -->
        <serializer class="claid::CSVSerializer"></serializer>
    </save> 
</Module>

The individual properties of the Module are explained below:

Explanation of the properties

what: Name of the Channel to subscribe to. Data posted to this Channel by other Modules will be received by the DataSaverModule and serialized to a file.
fileNameFormat: Specifies how files written by the DataSaverModule will be called.
- You can provide a simple name here like "data". However, then the DataSaverModule would store all data in a single file. It can make sense to store data in separate files, for example for each hour or each day. Therefore, here you can also use time format identifiers, that will automatically be resolved based on the times that the data comes in.
storagePath: A path to a folder on the internal storage of the device, where files produced by the DataSaverModule will be stored. If the folder does not exist, it will be created.
tmpStoragePath [optional]: A path to a folder on the internal storage of the device.
- If set, the DataSaverModule initially stores files in this path and transfers them to the Data folder only when starting a new file. This allows to differentiate between finished and unfinished files. For instance, if you set the fileNameFormat such that a new file is created every hour, files will be moved from the tmp folder to the Data folder when beginning a new file each hour. This is beneficial when used with the DataSyncModule, which can be set to sync only from the Data folder, ensuring only completed files are uploaded.
serializer: Specifies which serializer to use (see section serializers)

DataSyncModule¶

The DataSyncModule allows to synchronize files between folders, either on the same device or between devices. In the latter case, it can be used to upload data from a device to a server, for example. The DataSyncModule is the counterpat to the DataReceiverModule. The DataSyncModule can be understood as the sender, while the DataReceiverModule is the receiver. The DataSyncModule periodically sends a list of locally available files to the DataReceiverModule. The DataReceiverModule then requests each file that is not yet present in it's receival folder from the DataSyncModule, which sends the files accordingly. A typical configuration for the DataSyncModule looks as follows:

Example configuration

<Module class="claid::DataSyncModule">
    <!-- Path to a folder containing the data to synchronize -->
    <filePath>/sdcard/Data</filePath>
    <!-- Period to synchronize files, e.g., every 30 seconds -->
    <syncingPeriodInMs>30000</syncingPeriodInMs>
    <!-- Indicates wheter to delete files locally, that have been synchronized successfully. -->
    <deleteFileAfterSync>false</deleteFileAfterSync>
</Module>

The individual properties of the Module are explained below:

Explanation of the properties

filePath: Path to a folder containing the files that shall be synchronized with the DataReceiverModule. This folder will be scanned periodically (as specified by syncingPeriodInMs), and all files that do not yet exist in the output folder of the connected DataReceiverModule will be send.
syncingPeriodInMs: Period in milliseconds that indicates how often a synchronization shall be performed (for example, every 30 seconds).
deleteFileAfterSync [optional, default false]: If specified and set to true, every file that was successfully synchronized with the DataReceiverModule is deleted, after the DataReceiverModule acknowledges a succesfull reception.

DataReceiverModule¶

The DataReceiverModule is complementary to the DataSyncModule. It receives files from the DataSyncModule and stores them in a local output folder. The DataSyncModule periodically sends a list of available files, which the DataReceiverModule then compares with a list of files already available in this output folder. It then requests every file which is not yet available. A typical configuration for the DataReceiverModule looks as follows:

Example configuration

<Module class="claid::DataReceiverModule">
    <filePath>/Data/SmartphoneData/</filePath>
</Module>

The individual properties of the Module are explained below:

Explanation of the properties

filePath: Path to a folder in which received files shall be stored. The DataReceiverModule compares the list of files available in this folder with the list sent by the DataSyncModule and requests the missing files, which will then be stored in this folder.

Serializers¶

This package provides various serializers, which can be used together with the DataSaverModule. Serializers are components that allow to transform data from one format to another. For example, incoming accelerometer data can be converted to a CSV or XML representation automatically. The DataSaverModule invokes the specified serializer on the incoming data and stores the output to files.

The following serializers are available with this package:

CSVSerializer (claid::CSVSerializer): Serializes data to CSV format.
XMLSerializer (claid::XMLSerializer): Serializes data serializing data to XML format.
BinarySerializer (claid::BinarySerializer): Serializes data to a binary format. Each variable will be converted to their binary representation. Note, that this is not equal to a raw byte wise representation (e.g., such as in raw audio data), because it will include a data type specific header.

See the serialization examples below:

Serialization example

Class representationXMLSerializerCSVSerializerBinarySerializer

class Person
{
    string name = "Einstein";
    int age = 42;
    string profession = "Researcher";
}

<Person>
    <Name>Einstein</Name>
    <Age>42</Age>
    <Profession>Researcher</Profession>
</Person>

Name, Age, Profession
Einstein, 42, Researcher

Byte-wise (hexadecimal) representation of the class.

Data types¶

The data types included in this package are described in the following:

PeriodicValue¶

The PeriodicValue can be used to store frequency or periods. It allows to easily convert between different units of frequency (Hz, kHz, ...) and periods (seconds, milliseconds, ...). When using an instance of PeriodicValue as a property of a Module, the deserializer will automatically parse frequency or periodic values. Consider the following example configuration:

PeriodicValue as property

<Module class="MyModule">
    <periodicValue>
        <rate>50Hz</rate>
    <periodicValue>
</Module>

For the rate of the PeriodicValue, you can specify a frequency or period using the corresponding SI units and magnitudes. The following are supported:

Supported prefixes/magnitudes

Frequencies:

- GHz   giga Hertz
- MHz   mega Hertz
- Hz    Hertz
- mHz   milli Hertz

Periods:

- h     hour
- m     minute
- s     second
- mus   millisecond
- ns    nanosecond