IP block list - Firewall

[Lethos]

I have made a start (for my own personal/business use) at building a large IP block list, from technically free but quality sources, with a lot of technical data to back up why the IP address' why they should be blocked. I have done further filtering, to include only the worst typically and obviously make sure there is no duplicates.

As I was building this list, which quickly got to 2000 IP address', which was far more data than I had gotten in 2 years, in just 1 week, which came from specific IP address' that out there were well known to be "dangerous", compared to attacks which actually happened to my servers.
I realised others might find this useful, if they operate their own servers and can make use of imported IP address' in a specific format, usually compressed text files to help prevent malicious acts against them. I designed them originally to be imported to the IP blocklist add on with pfsense, but I'm sure others will have their own preferences and formats. With the scripts I've written, changing the format shouldn't be a problem.

So I saw the benefits of sharing my block list, I don't plan to profit from it (I won't say no to a donation, but it will always remain free).


Our Block lists are lists of IP address' that allows website administrators to take advantage of the data generated by well known and established sites; These include both community and corporate efforts to keep track of the worst suspicious, malicious and dangerous IP, then keep those attacks away from you.

We output it in a way that makes it easily usable in popular firewall and web server software with ease.
However it's never perfect, but we do our best to filter it as much as possible, and limit it to the worst offenders, as to reduce false positives.
Also we'd like to hear from you if you'd like it in a particular format, as every software prefers slightly different formatting and we might miss one or two.


To-long-didn't-read

So in short, is their any community interest in a free IP block list, that focus' on the worst offenders out there, from multiple sources?

[BlkGalager]

I am working on a similar project in my area. We currently have 300,000 suspect ip's on our servers. I look forward to posting about it when I have it fully functional.

[Lethos]

Good to know others are working on similar projects.

[Lethos]

Nearer to 5000 now.

Not bad for a few weeks building the list up.

[QuantumKiwi]

The trouble with running so many IPs in a block list is that it has to load the block lists up... Having 5k blocks in a iptable isn't going to run well.

Maybe running a blocklist that can be queried from a fast db that you provide for a premium $?

[K1773R]

I have made a start (for my own personal/business use) at building a large IP block list, from technically free but quality sources, with a lot of technical data to back up why the IP address' why they should be blocked. I have done further filtering, to include only the worst typically and obviously make sure there is no duplicates.

As I was building this list, which quickly got to 2000 IP address', which was far more data than I had gotten in 2 years, in just 1 week, which came from specific IP address' that out there were well known to be "dangerous", compared to attacks which actually happened to my servers.
I realised others might find this useful, if they operate their own servers and can make use of imported IP address' in a specific format, usually compressed text files to help prevent malicious acts against them. I designed them originally to be imported to the IP blocklist add on with pfsense, but I'm sure others will have their own preferences and formats. With the scripts I've written, changing the format shouldn't be a problem.

So I saw the benefits of sharing my block list, I don't plan to profit from it (I won't say no to a donation, but it will always remain free).


Our Block lists are lists of IP address' that allows website administrators to take advantage of the data generated by well known and established sites; These include both community and corporate efforts to keep track of the worst suspicious, malicious and dangerous IP, then keep those attacks away from you.

We output it in a way that makes it easily usable in popular firewall and web server software with ease.
However it's never perfect, but we do our best to filter it as much as possible, and limit it to the worst offenders, as to reduce false positives.
Also we'd like to hear from you if you'd like it in a particular format, as every software prefers slightly different formatting and we might miss one or two.


To-long-didn't-read

So in short, is their any community interest in a free IP block list, that focus' on the worst offenders out there, from multiple sources?

how about using fail2ban and this service http://www.blocklist.de/en/index.html ?

[Lethos]

The trouble with running so many IPs in a block list is that it has to load the block lists up... Having 5k blocks in a iptable isn't going to run well.

Maybe running a blocklist that can be queried from a fast db that you provide for a premium $?

In the firewalls I'm used to, they are used to handling 100k to millions of IP address and/or rules to filter (pfsense - pfctl).
However, I'm talking anything from dedicated firewall machines to firewalls on a dedicated server, that is already multi-purpose web-server, so any load issues is insignificant in comparison to what else it's doing, by using CSF and/or fail2ban to build some pretty large block lists.
So iptables is also used there, while it does have smaller limits, 5000 ip address' is still not that big of a deal any more as long as it's not some tiny vps using half a proccessor core and 256mb of ram.

5k is relatively small scale still in my opinion and since the lists are growing so quickly, it will be good for me to break them apart for those who want smaller ones, so don't feel like I'm ignoring this issue. I can still make this useful for the small user who is concerned by large lists impacting their servers performance.

The aim with my project, is to blocking just the worst offenders who are attacking a lot of others but maybe just haven't targeted your server yet. So blocking them advance is all I hoping to do so it's helpful in preventing problems that comes with them.

There are already those services out there, that allow for you to query individual address' from a list(s), usually done via a dns service.
I'd prefer to do this for free, so it gets adopted and by just downloading out list and importing it automatically once a day (or a frequency of your choosing), it just provides a level of protection that is good. This puts minimal load on both my servers and theirs once it's done.
There are better suited services out there, for those who want to know at the time of an attack if an IP address should be blocked.

how about using fail2ban and this service http://www.blocklist.de/en/index.html ?

I already use fail2ban (it's great tool along with CSF). Thanks to pointing out blocklist.de I can check it out to see if it definitely fills all the criteria I hope for in finding a good source of data and IP address to validate how bad they are and if they deserver to be on the block list I'm creating. It should, but I have to be sure, so I take false positives very seriously and if I don't have enough data to judge that, I don't use it.


I'm under no illusions, this list my be of no use to some, as your service provider (web host) might already be blocking most of these. However for those with minimalistic hosting or those fully in charge of the routing and hosting aspects of your servers, you won't have that sort of protection, unless they put it in themselves. So a free option might benefit them, just like it does me.

[Lethos]

There API (blocklist.de) is a little too simple in the information it provides. It's a good project being a part of fail2ban, I can't fault it. But I do have a tough criteria which it doesn't really fully meet as I do need to know more information. Also since people can easily utilise this service themselves, I'm not aiding them that much. My sources so far have mostly been ones not easy to utilise in a firewall solution (proprietary/paid/complicated) and I make it easier and free, but happen to provide lots of information about the IP address' so I can judge if they should be blocked. It's a small window, but It's been effective so far.

Concerning their API.
Because unless I've misunderstood (which is easy with it be primarily in German), is that while it's possible to look up recent IP address' that have attacked or been reported, I don't know how bad (frequency) or for what - Method or reason.
I can look up an individual IP address and find out how many attacks and reports it's had, which does help a little, but I still don't know what for reason or what they did, just the tally.

So unless I can get more information, I won't use this as a source for my list I give out to others, as the only way to insure I'm not putting false positives in my list and only the worst offenders, which require using the API once to grab recent attacks/reports, then using the API again on every single IP address to check what it was for and how many attacks and reports it had, to judge if it's bad enough to be included.
Obviously that is a lot of unnecessary strain on both my server and theirs, for a service easy to implement already.

It's was a good suggestion to try as I had not considered it.

[K1773R]

i had this problem too when a friend of mine asked me for something similiar. we came down to the point where he couldnt take it that IPs of infected user PCs are banned too, i had to explain him that it dosnt matter. malicious is malicious, if its spamming mails, UDP DoS, SYN Flood (if ppl still do this), Spambots @ forums/similiar, ssh brute force, etc it comes down to the same solution -> block it.
he tryd to argue that the ppl who got infected did nothing wrong so i had to tell him that they do something wrong, they use a piece of technolog they dont understand and are being abused, if you block them you may hurt that user so he cant reach ur server but you also block malicious intentions. morals/ethics while creating such a list is a bad thing, either it is malicious or not. those wo want to circumvent the restrictions can do it easely anyway since IP blocklists are very primitive.
if you want to do this right then i suggest following:
create a service that stores IP, what it has done and how often.
create a tool to import this information into iptables. in this tool (preferable a bash script) he can define what kind of things (spamming, and so on as above) as filter he want to use to get a list of IPs. this way you can create a list/service for everyone and its their opinion to decide what limits they want.

heads up

[Lethos]

i had this problem too when a friend of mine asked me for something similiar. we came down to the point where he couldnt take it that IPs of infected user PCs are banned too, i had to explain him that it dosnt matter. malicious is malicious, if its spamming mails, UDP DoS, SYN Flood (if ppl still do this), Spambots @ forums/similiar, ssh brute force, etc it comes down to the same solution -> block it.
he tryd to argue that the ppl who got infected did nothing wrong so i had to tell him that they do something wrong, they use a piece of technolog they dont understand and are being abused, if you block them you may hurt that user so he cant reach ur server but you also block malicious intentions. morals/ethics while creating such a list is a bad thing, either it is malicious or not. those wo want to circumvent the restrictions can do it easely anyway since IP blocklists are very primitive.
if you want to do this right then i suggest following:
create a service that stores IP, what it has done and how often.
create a tool to import this information into iptables. in this tool (preferable a bash script) he can define what kind of things (spamming, and so on as above) as filter he want to use to get a list of IPs. this way you can create a list/service for everyone and its their opinion to decide what limits they want.

heads up

That is pretty good feedback thanks.
Once I've got a better grips of making the lists, in it's more raw format, I'll work on implementations and how people can make use of it, which should include things you suggest.