In November 2019, the Bundestag passed the climate package, which, among other things, aims to renew the German cycling infrastructure. In order to gain an overview of the current condition of cycling paths, conventional road condition monitoring usually requires expensive measuring vehicles. In the field of data science consulting, we as a company have been dealing with topics such as sensor data analytics and predictive maintenance for several years. Consequently, we have asked ourselves how these topics can be combined cost-efficiently to evaluate road conditions. This would be of particular interest to local authorities who could use rental bike fleets equipped with this setup.
The goal of this project is an automated bicycle-based road analysis kit which entails no monitoring tasks for the cyclist. Therefore the data of an acceleration, gyroscope and GPS sensor has to be transmitted to a classification server. Different machine learning models will be compared to each other, aiming for the highest possible accuracy in predicting road type and condition. A distinction is made between the road types tarmac, cobblestone and gravel in conjunction with the conditions smooth, rough and bumpy, resulting in a total of nine classes. Furthermore, the results will be visualized live on a map using an IoT platform.
The setup in this project requires two things in particular: data and a machine learning model. We collect the data sets ourselves, so that the issue of hardware selection for sensor data recording plays an important role. Regarding the classification task different models are tested, from tree-based procedures to deep learning approaches.
The Bosch XDK 110 was chosen for sensor data recording (see middle picture in fig. 1). It is equipped with an accelerometer and a gyroscope, among other sensors. The recorded data can be transmitted wirelessly via Bluetooth, WiFi or SD card. The freely programmable buttons and LEDs serve as a simple user interface. The XDK can be programmed in C using the Eclipse-based XDK Workbench. Furthermore, a normal trekking bike is used (picture top left). The XDK’s bracket, which also serves as rain protection, is mounted in the bike’s basket (bottom left picture). To annotate the collected data, a labelling app was developed using the React Native framework (see pictures on the right). The recorded data of the XDK is transferred to the app via Bluetooth. Moreover, the app uses the smartphone’s GPS sensor to determine the current position. When a route has to be annotated, the plus button is pressed to enter the corresponding information.
By these means a first data set is created: Over a period of 376 minutes, 91 km are covered and about 735,000 readings per sensor channel are recorded, resulting in a sampling rate of 32.5 Hz. The average speed was about 14.6 km/h. To assess the influence of the sampling rate on the accuracy of our models, a second data set was created, in which the sampling rate was to be maximized. For this purpose, the data can no longer be transferred to the labelling app via Bluetooth due to lower throughput. Instead, it is stored directly on the SD card. The buttons and LEDs of the XDK were programmed accordingly for labelling and for starting and stopping the recording. Finally, the second data set covers a distance of about 77 km, which was recorded in 314 minutes. Due to the higher sampling rate of 294 Hz, significantly more data was recorded – about 5.6 million data points per sensor channel. Finally, a test data set was recorded in the same way, except that the bicycle was replaced with a kvv.nextbike to check the transferability of the models to unknown bikes. A total of approx. 9 km were driven, with just over 500,000 measured values stored per sensor channel.
For the machine learning part, different models were compared. On the one hand, tree-based methods such as Random Forest and XGBoost as well as Support Vector Machines were chosen. All these methods have already proven to be very reliable models for a wide range of classification tasks. In addition, tree-based algorithms are quite intuitive and their decisions can be interpreted easily, as opposed to black box models. On the other hand, deep-learning based methods have been used. In addition to LSTM networks, convolutional neural networks were tested, especially deep residual networks. We want to take a closer look at these.
Figure 2 above shows a ResNet developed by Fawaz et al.  for time series classification. Compared to image recognition, there is a peculiarity here: The filters slide over the input data only in one dimension (time). Everything else however works just the same. We see three residual blocks, each containing three convolution layers. The first block uses 64 filters and the other two blocks use 128 filters. The residual connections can prevent the degradation problem, i.e. deeper nets can be built without a rapid loss of training accuracy. Finally, the residual blocks are followed by average pooling and a fully connected output layer.
With InceptionTime, the current state of the art in the field of time series classification is proposed by Fawaz et al.  too. The architecture remains largely unchanged compared to the ResNet before. However, only two instead of three residual blocks are used. Lastly, the individual convolution layers are replaced by inception modules (see figure 3 above). In these modules, parallel convolution operations are running, whose output is merged to a common output.
The models are implemented using Python, scikit-learn and keras. A grid search with tenfold stratified cross-validation is performed for hyperparameter optimization. Prior to this, the input data used by the models are prepared by normalizing them. In addition, summary statistics such as averages, standard deviations, etc. are calculated for the non deep learning based methods. A total of 14 new features per sensor channel serve as input for these models.
With regard to the first data set, the two deep residual networks ResNet and InceptionTime were able to achieve the highest validation accuracy, with both achieving around 77%. The lead over the next best model (CNN-LSTM) is 7%, all other models are below the 70% mark. However, all models have difficulties in recognizing certain classes, and confusion often occurs. Two exemplary sequences of such problem classes can be seen in the lower left picture (fig. 4). In the upper left picture you can see a sequence of rough tarmac, in the lower left picture of rough cobblestone. On the x-axis the time steps are shown, where 200 time steps also correspond to the sequence length of a model input. The y-axis represents the acceleration in x-direction. In both cases the values fluctuate in a similar range, and the period lengths are also almost identical, so that the classes cannot be distinguished from each other easily.
A closer look at these classes led to the assumption that a higher sampling rate may be able to improve accuracy. With the recording setup of the first dataset, a sampling rate of roughly 32 Hz was achieved. The average velocity was 14.6 km/h or about 4.05 m/s. So on average a measurement was taken every 12.5 cm. This seems quite infrequent regarding e.g. the size of a cobblestone, which can be much smaller. Also road imperfections can occur more frequently. Consequently, the second data set was created with a maximized sampling rate. Again, two sequences of the former problem classes are shown (top right image in fig. 4). This time only the x-axis is adjusted. Since the sampling rate is increased by a factor of 9, the axis comprises 1,800 time steps. Now it seems that the two classes can be distinguished more easily since a measurement is taken on average every 1.36 cm. To sum up, the increased sampling rate has proven to be the key success factor. All models were able to achieve a significantly higher validation accuracy; the residual nets even reach values around 99%.
Furthermore, the transferability of the models to another bicycle should be checked, which is why the third data set was recorded. It can be noted that the data deviates quite strongly from the second data set. The standard deviations of the gyroscope values sometimes differ by a factor of 3 to 4 compared to the training data. Consequently, the models trained with the second data set cannot make reliable predictions. That means before enrolling this project in a larger scale with many bikes recording data some setups such as bike type (especially suspension) and tire pressure have to be assured.
In the end, the setup for real-time visualization of the sensor data was realized. The architecture of the test setup is shown in figure 5 below. The XDK is mounted on a bicycle as before and can transmit the accelerometer and gyroscope data wirelessly using the WiFi hotspot of a smartphone. At the same time, the GPS sensor data of a smartphone is recorded by a corresponding app. Both the smartphone and the XDK publish their data on a public Mosquitto test server. The MQTT protocol is used, a publish-subscribe protocol for typical IoT tasks. With this setup, the XDK can achieve a sampling rate of about 250 Hz, while the GPS coordinates are updated whenever the position of the smartphone changes by more than five meters. On the other side there is a computer that performs the classification and visualization. This has an intermediary program running with a Mosquitto broker who subscribes to the channels (topics) of the Mosquitto test server on which the sensor data are published. Then InceptionTime is used to classify the data. After that the intermediary publishes the classification results including the corresponding coordinates on a channel to which the Thingsboard-MQTT- Broker has subscribed. Thingsboard is an open source IoT platform that enables the recording, processing and visualization of sensor data as well as the management of sensor devices. It is operated on the local computer in form of a docker container. The XDK and the intermediary have been registered in the platform in order to be able to manage them. Finally, the subscribed sensor data are visualized in a dashboard.
The picture below (fig. 6) shows the dashboard, which was created in Thingsboard. There, the current position is drawn in a map with a marker. Furthermore, the tooltip shows the coordinates and the classification result. In addition, the driven route is drawn in black. Next to the map, a table is shown in which the last data received is recorded over a period of five minutes.
To sum up, it is challenging but absolutely possible to perform a street classification task like the one presented in a low cost and fully automated manner. Therefore the usage of state of the art machine learning algorithms led to very impressive classification results. Thus, a model suitable for real world use could be created – especially in connection with the automated transmission, classification and visualization of the data.
Moreover, some lessons for future ventures in the field of sensor data analytics could be learned from this project. In particular the sampling rate plays an important role: At the very beginning of such a project the question should be asked at which minimum frequency interesting information can be expected.
Finally, to use this proof of concept in practice, only a few adjustments need to be made. When collecting training data, different bikes should be considered or at least the type of bike that is to be used in production. Ideally the data should be recorded by different cyclists. Otherwise, what has been considered in this project applies as well: Consider as many different routes, air pressures and speeds as possible.
: Fawaz, H. I.; Forestier, G.; Weber, J.; Idoumghar, L.; Muller, P.-A.: Deep learning for time series classification: a review. Data Mining and Knowledge Discovery 33/4, S. 917–963, 2019.
: Fawaz, H. I.; Lucas, B.; Forestier, G.; Pelletier, C.; Schmidt, D. F.; Weber, J.; Webb, G. I.; Idoumghar, L.; Muller, P.-A.; Petitjean, F.: InceptionTime: Finding AlexNet for Time Series Classification. arXiv preprint arXiv:1909.04939/, 2019.