Skip to main content

Statistical and Machine Learning Techniques applied to Noisy Data

Question: Given a form of Noisy Sensor data, How do you apply Statistical/Machine learning techniques to interpret it?

This is a rather broad question with many interpretations. I would approach this to ask follow up questions e.g. What is the goal? One goal could be to get a high confidence value of the data or to understand how noisy the data is?

Possible Answers:

1. If the goal is to model the noisy data, one option can be to model it as a Gaussian or Normal Distribution. In this case, the sample mean or expected value can be a reliable measure of the noisy sensor data. Once, you have fit a distribution to the data - variance can help as a tool to understand the effect of noise.



2. If the goal is to apply machine learning tools to predict next sample, one can use tools like Linear Regression or a Neural Network to fit the data to better understand and predict next sample based on features.

3. Another goal can be predicting outliers, one can apply Clustering methods like K-Means to cluster data in an Unsupervised fashion and predict outliers or even step #1 can be used for the same by filtering by standard deviation steps.



References:
https://towardsdatascience.com/understanding-the-68-95-99-7-rule-for-a-normal-distribution-b7b7cbf760c2

Comments

Popular posts from this blog

What do you mean by Attention?

Question:  What do you mean by Attention? What are the types of Attention in Neural Networks? Expected Answer: Define attention as a representation of a distribution learnt by a Neural Network, use case and an example. You could also specify types of attention and give applications of each.