The SWAMP project recently published a paper in the IEEE Latin-American Conference on Communications (LATINCOM 2019), reporting an early experience with data quality assurance for IoT-based smart applications. Foundations of Data Quality Assurance for IoT-based Smart Applications provides a framework for data quality assurance in the IoT-base smart applications end-to-end data flow and reports the results of preliminary experiments using a long term raw sensor data provided by a partner company working with intelligent crop services.
The IoT and Big Data contemporary context brings several challenges, such as providing quality assurance (defined by availability and veracity) for sensor data. This paper proposes a discussion on the adequate foundations of a new general approach aimed at increasing the robustness and antifragility of IoT-based smart applications. Besides, it shows results of preliminary experiments with real data in the context of precision irrigation using multivariate methods to identify relevant situations, such as sensor failures and the mismatch of contextual sensor information due to different spatial granularities capture.
We used LOF (Local Outlier Factor algorithm), one of the most successful anomaly detection techniques for modern Big Data environments. LOF is a multidimensional anomaly detection technique for computing spatial density and providing real numerical value for each data point.
Filled circles denote a behavior considered typical by LOF, whereas points in other shapes represent anomalous behavior. Red triangles represent the soil is previously dry, with no relevant precipitation, although an extreme jump of water availability is observed in the soil, which is highly unexpected. Blue crosses represent significant soil drying jumps, when the expected behavior is a smoother drying process, even for days with no precipitation. Purple stars represent extreme cases of the blue crosses, where soil water availability is high, but the soil dried entirely in only one day, a highly unexpected phenomenon.
