DoS: investigating a SYN flood
DoS: investigating a SYN flood
DoS: investigating a SYN flood
results:
If a decision is made to block, we'd probably look to see if it's already blocked. We'd want to update the annotated time, or we better be ready to store the annotated time on the side in JSON, because at some point we're going to need to develop some kind of tooling to remove a block
Here is a very rough outline of the steps that occur, at least as I think they'd occur from a semi automation perspective
fido_event_log => netflow tools => whois lookup => compare to config => email => maybe do something
The human process isn't much difference but "email" is replaced with "ponder"
I will note that as of 2022/03/07 all policed prefixes [4 now] originate from ASN202425. Should we write automation that keeps lists up to date based on yaml and MRT and is operated like jnx-BGPQ3_report ?
in this case the origin AS has an IRR record that includes the mess we've seen
https://cms.uwsys.net/bgpq3-builder/AS202425-v4.json.juniper
I think if we go this route we owe them a single courtesy email to let them know they are a source of stench on the internet
1) You are made aware of a SYN attack by one of the following methods
- Real time observation of FIDO alarm
- Looking at FIDO event log
- Look at SYN counter rrd files
2) You use netflow tools to figure out what's going on. Here are a couple of queries that I used. You'll note that I didn't specify anything other than TCP SYN and to aggregate into a /16. Based on the below output, which can be output in easier modes for machine parsing [try -o json], the search is refined per step 3. With automation I presume we'd put some kind of comparison into the calculated bps/pps rate.
nfdump -M /var/local/flows/live/core -T -r nfcapd.202203020140 -n 10 -s record/packets -A srcip4/16,dstip4/16 "flags S"
results:
Aggregated flows 20170 Top 10 flows ordered by packets: Date first seen Duration Src IP Addr Dst IP Addr Packets Bytes bps Bpp Flows 2022-03-02 01:39:25.824 300.032 80.82.0.0 137.81.0.0 63.5 M 2.5 G 67.7 M 40 248012 2022-03-02 01:39:26.848 302.080 89.248.0.0 129.89.0.0 440576 17.6 M 466711 40 1721 2022-03-02 01:39:26.080 303.104 89.248.0.0 128.104.0.0 430080 17.2 M 454162 40 1680 2022-03-02 01:39:26.080 303.104 89.248.0.0 128.105.0.0 425728 17.0 M 449513 40 1663 2022-03-02 01:39:26.080 301.056 89.248.0.0 146.151.0.0 425472 17.0 M 452380 40 1662 2022-03-02 01:39:26.592 302.592 89.248.0.0 72.33.0.0 418816 16.8 M 443045 40 1636 2022-03-02 01:39:26.848 301.824 89.248.0.0 144.92.0.0 413184 16.5 M 438147 40 1614 2022-03-02 01:16:07.808 1714.432 205.213.0.0 72.33.0.0 296960 421.9 M 2.0 M 1420 59 2022-03-02 01:39:25.824 303.104 89.248.0.0 131.210.0.0 250368 10.0 M 264351 40 978 2022-03-02 01:39:27.360 300.800 89.248.0.0 144.13.0.0 238848 9.6 M 254120 40 933 Summary: total flows: 399073, total bytes: 7.6 G, total packets: 105.3 M, avg bps: 1.6 M, avg pps: 2708, avg bpp: 72 Time window: 2022-02-03 11:25:22 - 2022-03-02 01:44:59 Total flows processed: 1257005, Blocks skipped: 0, Bytes read: 126157164 Sys: 0.302s flows/second: 4149065.4 Wall: 0.302s flows/second: 4157793.8
3) Refining the query
nfdump -M /var/local/flows/live/core -T -r nfcapd.202203020140 -n 10 -s record/packets -A srcip4/24 "flags S and dst net 137.81.0.0/16 and src net 80.82.0.0/16"
results:
Aggregated flows 2 Top 10 flows ordered by packets: Date first seen Duration Src IP Addr Packets Bytes bps Bpp Flows 2022-03-02 01:39:25.824 90.880 80.82.64.0 63.4 M 2.5 G 223.4 M 40 247851 2022-03-02 01:39:29.152 296.704 80.82.77.0 41216 1.7 M 44562 40 161 Summary: total flows: 248012, total bytes: 2.5 G, total packets: 63.5 M, avg bps: 67.7 M, avg pps: 211614, avg bpp: 40 Time window: 2022-02-03 11:25:22 - 2022-03-02 01:44:59 Total flows processed: 1257005, Blocks skipped: 0, Bytes read: 126157164 Sys: 0.304s flows/second: 4121827.9 Wall: 0.315s flows/second: 3979942.1
4) Learn more about the source. whois '80.82.64.0' or perhap use the router proxy or an MRT dump to see the origin AS, what the covering route is, etc. Here, the human can ponder to make a decision about what to do. With automation, this stuff would be shoved into an email
[@mole ~]$ whois 80.82.64.0
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to '80.82.64.0 - 80.82.64.255'
% Abuse contact for '80.82.64.0 - 80.82.64.255' is 'abuse@ipvolume.net'
inetnum: 80.82.64.0 - 80.82.64.255
netname: NET-1-64
descr: IPV NETBLOCK
country: NL
geoloc: 52.370216 4.895168
org: ORG-IVI1-RIPE
admin-c: IVI24-RIPE
tech-c: IVI24-RIPE
status: ASSIGNED PA
mnt-by: IPV
mnt-lower: IPV
mnt-routes: IPV
created: 2010-09-19T16:51:12Z
last-modified: 2019-02-01T18:24:55Z
source: RIPE
organisation: ORG-IVI1-RIPE
org-name: IP Volume inc
org-type: OTHER
address: Suite 9
address: Victoria, Mahe
address: Seychelles
abuse-c: IVNO1-RIPE
mnt-ref: IPV
mnt-by: IPV
created: 2018-05-14T11:46:50Z
last-modified: 2019-01-31T14:39:36Z
source: RIPE # Filtered
role: IPV
address: Suite 9
address: Victoria, Mahe
address: Seychelles
nic-hdl: IVI24-RIPE
mnt-by: IPV
created: 2018-05-16T13:28:41Z
last-modified: 2019-01-31T21:21:20Z
source: RIPE # Filtered
% Information related to '80.82.64.0/24AS202425'
route: 80.82.64.0/24
origin: AS202425
remarks: +-----------------------------------------------
remarks: | For abuse e-mail abuse@ipvolume.net
remarks: | We do not always reply to abuse.
remarks: | But we do take care your report is dealt with!
remarks: +-----------------------------------------------
mnt-by: IPV
created: 2019-01-24T15:07:49Z
last-modified: 2019-02-01T12:32:15Z
source: RIPE
% This query was served by the RIPE Database Query Service version 1.102.2 (BLAARKOP)
===========
@r-uwmadison-cssc-re1> show route 80.82.64.0 table inet.0 active-path
inet.0: 873644 destinations, 3193972 routes (873421 active, 48 holddown, 228 hidden)
+ = Active Route, - = Last Active, * = Both
80.82.64.0/24 *[BGP/170] 9w4d 20:27:11, MED 0, localpref 775, from 143.235.32.10
AS path: 6939 57717 202425 I, validation-state: unverified
> to 143.235.33.47 via et-0/1/1.3434
to 143.235.33.95 via ae3.3439, label-switched-path et-0/1/1.3434:BypassLSP->143.235.32.10
==================
[@bar cms]$ whois AS202425
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to 'AS196608 - AS213403'
as-block: AS196608 - AS213403
descr: RIPE NCC ASN block
remarks: These AS Numbers are assigned to network operators in the RIPE NCC service region.
mnt-by: RIPE-NCC-HM-MNT
created: 2021-11-26T06:58:53Z
last-modified: 2021-11-26T06:58:53Z
source: RIPE
% Information related to 'AS202425'
% Abuse contact for 'AS202425' is 'abuse@ipvolume.net'
aut-num: AS202425
as-name: INT-NETWORK
org: ORG-IVI1-RIPE
import: from AS3356 accept ANY
import: from AS57717 accept ANY
export: to AS3356 announce AS-IPV
export: to AS57717 announce AS-IPV
admin-c: IVI24-RIPE
tech-c: IVI24-RIPE
status: ASSIGNED
mnt-by: RIPE-NCC-END-MNT
mnt-by: IPV
created: 2018-05-17T11:45:04Z
last-modified: 2019-03-31T22:16:51Z
source: RIPE
sponsoring-org: ORG-IL465-RIPE
organisation: ORG-IVI1-RIPE
org-name: IP Volume inc
org-type: OTHER
address: Suite 9
address: Victoria, Mahe
address: Seychelles
abuse-c: IVNO1-RIPE
mnt-ref: IPV
mnt-by: IPV
created: 2018-05-14T11:46:50Z
last-modified: 2019-01-31T14:39:36Z
source: RIPE # Filtered
role: IPV
address: Suite 9
address: Victoria, Mahe
address: Seychelles
nic-hdl: IVI24-RIPE
mnt-by: IPV
created: 2018-05-16T13:28:41Z
last-modified: 2019-01-31T21:21:20Z
source: RIPE # Filtered
% This query was served by the RIPE Database Query Service version 1.102.2 (BLAARKOP)
[@bar cms]$ whois AS57717
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to 'AS56320 - AS58367'
as-block: AS56320 - AS58367
descr: RIPE NCC ASN block
remarks: These AS Numbers are assigned to network operators in the RIPE NCC service region.
mnt-by: RIPE-NCC-HM-MNT
created: 2018-11-22T15:27:34Z
last-modified: 2018-11-22T15:27:34Z
source: RIPE
% Information related to 'AS57717'
% Abuse contact for 'AS57717' is 'abuse@fiberxpress.net'
aut-num: AS57717
as-name: FBX-AS
org: ORG-FB73-RIPE
import: from AS9002 accept ANY
import: from AS56611 accept ANY
export: to AS-ANY announce AS-FBX
admin-c: FB16391-RIPE
tech-c: FB16391-RIPE
status: ASSIGNED
mnt-by: RIPE-NCC-END-MNT
mnt-by: nl-fiberxpress-1-mnt
created: 2016-10-20T07:43:27Z
last-modified: 2020-06-15T09:24:08Z
source: RIPE
organisation: ORG-FB73-RIPE
org-name: FiberXpress BV
country: NL
org-type: LIR
address: Bruynvisweg 11
address: 1531AX
address: Wormer
address: NETHERLANDS
phone: +31757112156
admin-c: FB16391-RIPE
tech-c: FB16391-RIPE
abuse-c: AR37944-RIPE
mnt-ref: nl-fiberxpress-1-mnt
mnt-by: RIPE-NCC-HM-MNT
mnt-by: nl-fiberxpress-1-mnt
created: 2016-10-13T13:20:58Z
last-modified: 2020-12-16T13:19:34Z
source: RIPE # Filtered
role: Fiberxpress BV
address: Bruynvisweg 11
address: 1531 AX
address: Wormer
address: Netherlands
nic-hdl: FB16391-RIPE
mnt-by: nl-fiberxpress-1-mnt
created: 2019-01-31T17:09:24Z
last-modified: 2019-01-31T17:09:24Z
source: RIPE # Filtered
% This query was served by the RIPE Database Query Service version 1.102.2 (ANGUS)
[@bar cms]$ whois AS6939
#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/resources/registry/whois/inaccuracy_reporting/
#
# Copyright 1997-2022, American Registry for Internet Numbers, Ltd.
#
ASNumber: 6939
ASName: HURRICANE
ASHandle: AS6939
RegDate: 1996-06-28
Updated: 2003-11-04
Ref: https://rdap.arin.net/registry/autnum/6939
OrgName: Hurricane Electric LLC
OrgId: HURC
Address: 760 Mission Court
City: Fremont
StateProv: CA
PostalCode: 94539
Country: US
RegDate:
Updated: 2018-02-09
Ref: https://rdap.arin.net/registry/entity/HURC
ReferralServer: rwhois://rwhois.he.net:4321
OrgTechHandle: ZH17-ARIN
OrgTechName: Hurricane Electric
OrgTechPhone: +1-510-580-4100
OrgTechEmail: hostmaster@he.net
OrgTechRef: https://rdap.arin.net/registry/entity/ZH17-ARIN
OrgAbuseHandle: ABUSE1036-ARIN
OrgAbuseName: Abuse Department
OrgAbusePhone: +1-510-580-4100
OrgAbuseEmail: abuse@he.net
OrgAbuseRef: https://rdap.arin.net/registry/entity/ABUSE1036-ARIN
RTechHandle: ZH17-ARIN
RTechName: Hurricane Electric
RTechPhone: +1-510-580-4100
RTechEmail: hostmaster@he.net
RTechRef: https://rdap.arin.net/registry/entity/ZH17-ARIN
#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/resources/registry/whois/inaccuracy_reporting/
#
# Copyright 1997-2022, American Registry for Internet Numbers, Ltd.
#
Found a referral to rwhois.he.net:4321.
%rwhois V-1.5:0012b7:00 concierge.he.net (HE-RWHOISd v:8e2663b)
asn:ID;I:AS6939
asn:Auth-Area:asns
asn:Class-Name:asn
asn:ASN;I:AS6939
asn:ASName;I:HURRICANE
asn:Tech-Contact;I:POC-HE-NOC
asn:Abuse-Contact;I:POC-HE-ABUSE
asn:NOC-Contact;I:POC-HE-NOC
asn:Created:20101117005228000
asn:Updated:20101117005228000
contact:ID;I:POC-HE-NOC
contact:Auth-Area:contacts
contact:Class-Name:contact
contact:Name:Network Operations Center
contact:Company:Hurricane Electric
contact:Street-Address:760 Mission Ct
contact:City:Fremont
contact:Province:CA
contact:Postal-Code:94539
contact:Country-Code:US
contact:Phone:+1-510-580-4100
contact:E-Mail:noc@he.net
contact:Created:20100901200738000
contact:Updated:20100901200738000
contact:ID;I:POC-HE-ABUSE
contact:Auth-Area:contacts
contact:Class-Name:contact
contact:Name:Abuse Department
contact:Company:Hurricane Electric
contact:Street-Address:760 Mission Ct
contact:City:Fremont
contact:Province:CA
contact:Postal-Code:94539
contact:Country-Code:US
contact:Phone:+1-510-580-4100
contact:E-Mail:abuse@he.net
contact:Created:20100901200738000
contact:Updated:20100901200738000
contact:Comment:For email abuse (spam) only
===============
[net@bar cms]$ cat hosts.junos.ASBR | sed 's/\.conf//' | xargs -n1 -P30 jlogin -x cmd.junos.pfx
============
[net@bar cms]$ cat hosts.junos.ASBR
r-chicago-600w.conf
r-chicago-710.conf
r-minneapolis-511.conf
r-uwmadison-cssc-2.conf
r-uwmadison-cssc.conf
r-uwmilwaukee-ems.conf
[net@bar cms]$ cat cmd.junos.pfx
edit
edit policy-options prefix-list sync_lists-naughty_source_subnets
set 80.82.64.0/24
annotate 80.82.64.0/24 "uwsuperior 2022/03/02"
top
commit confirmed 5
exit
[m7h@bar bin]$ whois 80.82.64.0
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf
% Note: this output has been filtered.
% To receive output for a database update, use the "-B" flag.
% Information related to '80.82.64.0 - 80.82.64.255'
% Abuse contact for '80.82.64.0 - 80.82.64.255' is 'abuse@ipvolume.net'
inetnum: 80.82.64.0 - 80.82.64.255
netname: NET-1-64
descr: IPV NETBLOCK
country: NL
geoloc: 52.370216 4.895168
org: ORG-IVI1-RIPE
admin-c: IVI24-RIPE
tech-c: IVI24-RIPE
status: ASSIGNED PA
mnt-by: IPV
mnt-lower: IPV
mnt-routes: IPV
created: 2010-09-19T16:51:12Z
last-modified: 2019-02-01T18:24:55Z
source: RIPE
organisation: ORG-IVI1-RIPE
org-name: IP Volume inc
org-type: OTHER
address: Suite 9
address: Victoria, Mahe
address: Seychelles
abuse-c: IVNO1-RIPE
mnt-ref: IPV
mnt-by: IPV
created: 2018-05-14T11:46:50Z
last-modified: 2019-01-31T14:39:36Z
source: RIPE # Filtered
role: IPV
address: Suite 9
address: Victoria, Mahe
address: Seychelles
nic-hdl: IVI24-RIPE
mnt-by: IPV
created: 2018-05-16T13:28:41Z
last-modified: 2019-01-31T21:21:20Z
source: RIPE # Filtered
% Information related to '80.82.64.0/24AS202425'
route: 80.82.64.0/24
origin: AS202425
remarks: +-----------------------------------------------
remarks: | For abuse e-mail abuse@ipvolume.net
remarks: | We do not always reply to abuse.
remarks: | But we do take care your report is dealt with!
remarks: +-----------------------------------------------
mnt-by: IPV
created: 2019-01-24T15:07:49Z
last-modified: 2019-02-01T12:32:15Z
source: RIPE
% This query was served by the RIPE Database Query Service version 1.102.2 (ANGUS)
If a decision is made to block, we'd probably look to see if it's already blocked. We'd want to update the annotated time, or we better be ready to store the annotated time on the side in JSON, because at some point we're going to need to develop some kind of tooling to remove a block
A human then mass pushes to update the existing prefix-list
example:
cat hosts.junos.ASBR | sed 's/\.conf//' | xargs -n1 -P30 jlogin -x cmd.junos.pfx
[net@bar cms]$ cat hosts.junos.ASBR
r-chicago-600w.conf
r-chicago-710.conf
r-minneapolis-511.conf
r-uwmadison-cssc-2.conf
r-uwmadison-cssc.conf
r-uwmilwaukee-ems.conf
[net@bar cms]$ cat cmd.junos.pfx
edit
edit policy-options prefix-list sync_lists-naughty_source_subnets
set 80.82.64.0/24
annotate 80.82.64.0/24 "uwsuperior 2022/03/02"
top
commit confirmed 5
exit
We can look at rrds, reports and get alarms just like you'd expect
examples:
https://stats.uwsys.net/cgi-bin/shorten.fcgi?i=6108&c=c2185800f05c5699
from https://fido.uwsys.net/cgi-bin/private/fido_show_datafile.cgi?data_file=fido_jnxFirewall_alarms_Packet_thresholds.bin
'device=r-chicago-710.uwsys.net_filter=antispoof-in-ae2-2292-i_counter=uwsuperior-packetPolicer-1Kpps-ae2-2292-i_type=policer_jnx-xml-FWPacketCount.rrd' => {
'rule' => '120',
'threshold' => '100'
},