Warm tip: This article is reproduced from serverfault.com, please click

How to remove IP(s) from a text file based on given CIDR?

发布于 2020-12-01 07:05:35

To demonstrate this, I have few IPs in text file called blocked.txt with the following content:

1.1.1.1
1.1.1.2
1.1.1.3
2.1.1.1
2.1.1.2

So given an input of CIDR of 1.1.1.0/24

I want to remove the IP that belongs this CIDR range which are 1.1.1.1, 1.1.1.2 and 1.1.1.3

The only thing that makes me stuck is how to list out all the IP if CIDR form is given. Example:

#!/bin/bash
cidr_ip="1.1.1.0/24"
ip_exist=$(echo "${cidr_ip}" | grep_all_the_ip_in_that_CIDR in blocked.txt)
echo ${ip_exist} # This will list out all the iP and then I can use this to remove the IP

The expected output, blocked.txt will only have this content:

2.1.1.1
2.1.1.2

=================================

I'm testing with this data:

161.35.169.25
104.228.72.171
177.5.53.176
103.56.43.225
20.58.48.57
27.115.124.6
1.1.1.1
111.229.188.72
27.115.124.70
51.15.179.65
77.245.149.46
180.163.220.68
71.68.239.90
45.142.120.87
42.236.10.125
42.236.10.114
212.70.149.53
1.1.1.0/24
1.1.1.9
1.1.1.10
1.1.1.2
1.1.1.3
2.1.1.0/24
2.1.1.1
3.1.1.0/24
212.70.149.84
103.249.77.2
5.178.86.76
Questioner
Kalib Zen
Viewed
0
RavinderSingh13 2020-12-01 16:21:28

EDIT: Since OP added few more samples to handle lines with / present one could try following.

awk -F'/' -v val="1.1.1.0/24" '
BEGIN{
  match(val,/.*\./)
  matched=substr(val,RSTART,RLENGTH-1)
  split(substr(val,RSTART+RLENGTH),arr,"/")
  for(i=arr[1];i<=arr[2];i++){
   skip[matched"."i]
  }
}
!($1 in skip)
'  Input_file


Could you please try following, written and tested with shown samples in GNU awk. Where variable val is your range of IPs.

awk -v val="1.1.1.0/24" '
BEGIN{
  match(val,/.*\./)
  matched=substr(val,RSTART,RLENGTH-1)
  split(substr(val,RSTART+RLENGTH),arr,"/")
  for(i=arr[1];i<=arr[2];i++){
   skip[matched"."i]
  }
}
!($0 in skip)
'  Input_file

Explanation: Adding detailed explanation for above.

awk -v val="1.1.1.0/24" '                        ##Starting awk program from here and creating variable val which has that range here.
BEGIN{                                           ##Starting BEGIN section of this program from here.
  match(val,/.*\./)                              ##using match function to match everything till . in variable val here.
  matched=substr(val,RSTART,RLENGTH-1)           ##Creating matched which has sub string of matched regex value in var variable.
  split(substr(val,RSTART+RLENGTH),arr,"/")      ##Splitting rest of value of var which is actual range into array arr here.
  for(i=arr[1];i<=arr[2];i++){                   ##Running for loop from 1st item value to 2nd item value of array here.
   skip[matched"."i]                             ##Creating skip array which has index as matched(variable) dot and i here, it contains which ips to be negleted basically.
  }
}
!($0 in skip)                                    ##In main block of program checking condition if current line is NOT present in skip then print that line.
' Input_file                                     ##Mentioning Input_file name here.