Anomaly detection

[Mat]

Anyone got any ideas/advice here?

I'll be working with 1200000+ samples and a lot of noise.

I'm looking at knn and LOF atm, with a view to maybe using both side by side and collecting the result. Unfortunately I have no experience with data analysis other than a stats AS level (read as no experience at all).

Thanks for any info

Matt

[taietel]

Mat, what format is the data?

[Mat]

Mat, what format is the data?

You don't want to know.

It's a directory of 20+ csv files each with 60000 rows. Columns are a timestamp then 4 channels (float between +- 10). It's a mess. The reason it's a mess is because I was originally asked to output it in a format that could be opened in excel.

However, I don't think that's really that important. The timestamps can be ignored (intervals are ~constant), and the channels can be analysed separately, so it's just a question of where the anomalous results are in the sample.

[taietel]

If the data is in csv format, you can easily transform it to xls.

You can determine the standard deviation, then see what values are far away from the mean.

[Mat]

The problem is that it's not a case of there being 1 line and a few anomalies. I should have added that at the beginning.

It will change, so it will start at 5 (+-1), step up to 7 +-1 go down to 2 +-1, and it's those changes that are important as well as the anomalies.

[czardas]

Thinking about this in my rule of thumb way of going about things: if there is a large amount of data and a large number of anomalies to watch out for, then I would think about taking smaller samples to look for a wider range of anomalies. Then if you find something that might be a candidate for deeper scrutiny, take larger samples and test them for similar anomalies. A kind of dead reckoning statistics. I don't know how relevant such an approach might be.

Edited September 26, 2011 by czardas

[Mat]

Again, it's a 1 sample anomaly in 100000 that I'm looking for.

Grrr.... Guess there's no miracle cure then. One day someones going to reply and say: Oh yes, someone wrote exactly what you wanted >here< and I won't need to do it myself I'm still waiting for that day, like a monkey at a typewriter.

[czardas]

Just throwing around whatever comes to mind. You'll probably have to get your hands dirty.

[taietel]

Find a way to graphicaly represent the data, then look for a change in the pattern. Human brain can see the difference lot more faster than any written code.

[MvGulik]

"look" ...

Think your ignoring some of the trailing zero's in the numbers Mat posted.

Edited September 26, 2011 by iEvKI3gv9Wrkd41u