One of the interesting questions we get asked at TheatSTOP concerns how long an IP address remains bad once it has been identified as such. The answer is not completely straightforward and varies depending on which threat list it has been put on. Moreover many lists do not have specific "first seen" or "last seen" data on each IP address, rather they simply list the currently active list (where active typically means that they have been identified as bad within the last week or so). Possibly worse for questioner, some of the threat sources we use are distributed under terms that prohibit us from answering the question.

However the DShield organization makes their top N lists public and they contain the first seen and last seen dates so it is possible to analyze that data to figure out how long an IP address remains bad. So we have done some analysis using the maximum DShield list size (Top 10,000). Here is the breakdown from yesterday

Dshield Top 10k Persistence 2010-08-23

As you can see about half the IP addresses remain bad for over a month. However over a quarter are only seen for a week or less and well over 10% (actually about a sixth) disappear in one day or less. I have run this analysis quite often over the last month or so and the numbers, while they vary a little, do remain quite consistent.

What this means is that when people say that blacklists are dead or that the internet needs to move to some kind of "assume everything is guilty" model, they are partly right. Manual blacklist maintenance is clearly ineffective because the chances are high that by the time you add an ip address to the black list it has been fixed (if you touch your black lists once a month you have a 50% chance of this, once a week almost 30% and once a day about 15%). And those repaired ip addresses can a) lull you into a false sense of security and b) cause major user satisfaction issues when there is interaction with one of these ip addresses - the false positive problem.

However the fact that the blacklist needs to be kept up to date doesn't invalidate the utility of the concept. It just means you need something like ThreatSTOP to automatically update the list on a timely basis so that you get coverage against the short-lived threats while they are active.

Interestingly we also have data from last year (17 September 2009) which our CEO used in a presentation at a security conference shows a slight difference in breakdown - just 40% of threats are over 30 days and almost a quarter are one day or less. My hypothesis (I need to do some data checking) is that what has happened is that some computers become periodically reinfected and thus they gradually move from the short term list to the long term one.

Dshield Threat Persistence 2009-09-17

If I am correct then this is yet further evidence to show that frequent updates to a blacklist are the best way to go because the frequent update will catch these ip addresses when they become compromised and then clean them out once they have been cleaned up.