Richard Self

Richard Self

Research Fellow, Big Data Lab and Senior Lecturer in Analytics and Governance at University of Derby

Software Testing For Big Data Analytics In The Era Of Uncertain Data Veracity


The traditional approach to software construction and delivery provides data validation in the input routines to ensure that only valid data is entered into the system. It can also be used to enhance system security by e.g. preventing input buffer overflow problems. It has always been assumed that such systems will retain a high degree of data cleanliness.


In the world of the IoT and development of big data analytics for all forms of internet based data sources such input data validation is not feasible; often the sensors are of very uncertain calibration and accuracy. This led to J Easton’s observation that some 80% of all the data that we need to use is of uncertain veracity. As a result, it is now the case that traditional approaches to system development and testing are no longer sufficient.


The session will evaluate some of the practical and governance issues raised by current approaches to the development and delivery of IoT device apps and areas such as sentiment analysis. Some of the 12 Vs of data governance will be used to suggest new approaches for testing software that uses such data sources.

Richard has 30 years’ experience at a large aerospace company in designing and implementing a wide range of systems. He is now one of the leaders of the development of Data Science and Analytics programmes at the University of Derby. He is particularly interested in the governance consequences of the use of Big Data and the IoT. He is an invited keynote and leadership speaker at national and international conferences across a wide range of business sectors. His recent research has concentrated on the governance consequences of the accuracy of some IoT devices, such as smartphone location services and activity tracker devices.


You can read Richard’s recent article published in TEST Magazine here.