DoS: investigating a SYN flood

DoS: investigating a SYN flood

DoS: investigating a SYN flood


Here is a very rough outline of the steps that occur, at least as I think they'd occur from a semi automation perspective

fido_event_log => netflow tools => whois lookup => compare to config => email => maybe do something

The human process isn't much difference but "email" is replaced with "ponder"

I will note that as of 2022/03/07 all policed prefixes [4 now] originate from ASN202425.  Should we write automation that keeps lists up to date based on yaml and MRT and is operated like jnx-BGPQ3_report ?

in this case the origin AS has an IRR record that includes the mess we've seen

https://cms.uwsys.net/bgpq3-builder/AS202425-v4.json.juniper

I think if we go this route we owe them a single courtesy email to let them know they are a source of stench on the internet



1) You are made aware of a SYN attack by one of the following methods
  • Real time observation of FIDO alarm
  • Looking at FIDO event log
  • Look at SYN counter rrd files


2) You use netflow tools to figure out what's going on.  Here are a couple of queries that I used.  You'll note that I didn't specify anything other than TCP SYN and to aggregate into a /16.  Based on the below output, which can be output in easier modes for machine parsing [try -o json], the search is refined per step 3.  With automation I presume we'd put some kind of comparison into the calculated bps/pps rate.

nfdump -M /var/local/flows/live/core  -T  -r nfcapd.202203020140 -n 10 -s record/packets -A srcip4/16,dstip4/16 "flags S"

results:
Aggregated flows 20170
Top 10 flows ordered by packets:
Date first seen          Duration       Src IP Addr      Dst IP Addr   Packets    Bytes      bps    Bpp Flows
2022-03-02 01:39:25.824   300.032         80.82.0.0       137.81.0.0    63.5 M    2.5 G   67.7 M     40 248012
2022-03-02 01:39:26.848   302.080        89.248.0.0       129.89.0.0    440576   17.6 M   466711     40  1721
2022-03-02 01:39:26.080   303.104        89.248.0.0      128.104.0.0    430080   17.2 M   454162     40  1680
2022-03-02 01:39:26.080   303.104        89.248.0.0      128.105.0.0    425728   17.0 M   449513     40  1663
2022-03-02 01:39:26.080   301.056        89.248.0.0      146.151.0.0    425472   17.0 M   452380     40  1662
2022-03-02 01:39:26.592   302.592        89.248.0.0        72.33.0.0    418816   16.8 M   443045     40  1636
2022-03-02 01:39:26.848   301.824        89.248.0.0       144.92.0.0    413184   16.5 M   438147     40  1614
2022-03-02 01:16:07.808  1714.432       205.213.0.0        72.33.0.0    296960  421.9 M    2.0 M   1420    59
2022-03-02 01:39:25.824   303.104        89.248.0.0      131.210.0.0    250368   10.0 M   264351     40   978
2022-03-02 01:39:27.360   300.800        89.248.0.0       144.13.0.0    238848    9.6 M   254120     40   933
Summary: total flows: 399073, total bytes: 7.6 G, total packets: 105.3 M, avg bps: 1.6 M, avg pps: 2708, avg bpp: 72
Time window: 2022-02-03 11:25:22 - 2022-03-02 01:44:59
Total flows processed: 1257005, Blocks skipped: 0, Bytes read: 126157164
Sys: 0.302s flows/second: 4149065.4  Wall: 0.302s flows/second: 4157793.8 



3) Refining the query

nfdump -M /var/local/flows/live/core  -T  -r nfcapd.202203020140 -n 10 -s record/packets -A srcip4/24 "flags S and dst net 137.81.0.0/16 and src net 80.82.0.0/16"

results:
Aggregated flows 2
Top 10 flows ordered by packets:
Date first seen          Duration       Src IP Addr   Packets    Bytes      bps    Bpp Flows
2022-03-02 01:39:25.824    90.880        80.82.64.0    63.4 M    2.5 G  223.4 M     40 247851
2022-03-02 01:39:29.152   296.704        80.82.77.0     41216    1.7 M    44562     40   161
Summary: total flows: 248012, total bytes: 2.5 G, total packets: 63.5 M, avg bps: 67.7 M, avg pps: 211614, avg bpp: 40
Time window: 2022-02-03 11:25:22 - 2022-03-02 01:44:59
Total flows processed: 1257005, Blocks skipped: 0, Bytes read: 126157164
Sys: 0.304s flows/second: 4121827.9  Wall: 0.315s flows/second: 3979942.1 


4) Learn more about the source.  whois '80.82.64.0' or perhap use the router proxy or an MRT dump to see the origin AS, what the covering route is, etc. Here, the human can ponder to make a decision about what to do. With automation, this stuff would be shoved into an email

[@mole ~]$ whois 80.82.64.0
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '80.82.64.0 - 80.82.64.255'

% Abuse contact for '80.82.64.0 - 80.82.64.255' is 'abuse@ipvolume.net'

inetnum:        80.82.64.0 - 80.82.64.255
netname:        NET-1-64
descr:          IPV NETBLOCK
country:        NL
geoloc:         52.370216 4.895168
org:            ORG-IVI1-RIPE
admin-c:        IVI24-RIPE
tech-c:         IVI24-RIPE
status:         ASSIGNED PA
mnt-by:         IPV
mnt-lower:      IPV
mnt-routes:     IPV
created:        2010-09-19T16:51:12Z
last-modified:  2019-02-01T18:24:55Z
source:         RIPE

organisation:   ORG-IVI1-RIPE
org-name:       IP Volume inc
org-type:       OTHER
address:        Suite 9
address:        Victoria, Mahe
address:        Seychelles
abuse-c:        IVNO1-RIPE
mnt-ref:        IPV
mnt-by:         IPV
created:        2018-05-14T11:46:50Z
last-modified:  2019-01-31T14:39:36Z
source:         RIPE # Filtered

role:           IPV
address:        Suite 9
address:        Victoria, Mahe
address:        Seychelles
nic-hdl:        IVI24-RIPE
mnt-by:         IPV
created:        2018-05-16T13:28:41Z
last-modified:  2019-01-31T21:21:20Z
source:         RIPE # Filtered

% Information related to '80.82.64.0/24AS202425'

route:          80.82.64.0/24
origin:         AS202425
remarks:        +-----------------------------------------------
remarks:        | For abuse e-mail abuse@ipvolume.net
remarks:        | We do not always reply to abuse.
remarks:        | But we do take care your report is dealt with!
remarks:        +-----------------------------------------------
mnt-by:         IPV
created:        2019-01-24T15:07:49Z
last-modified:  2019-02-01T12:32:15Z
source:         RIPE

% This query was served by the RIPE Database Query Service version 1.102.2 (BLAARKOP)

===========

@r-uwmadison-cssc-re1> show route 80.82.64.0 table inet.0 active-path 

inet.0: 873644 destinations, 3193972 routes (873421 active, 48 holddown, 228 hidden)
+ = Active Route, - = Last Active, * = Both

80.82.64.0/24      *[BGP/170] 9w4d 20:27:11, MED 0, localpref 775, from 143.235.32.10
                      AS path: 6939 57717 202425 I, validation-state: unverified
                    > to 143.235.33.47 via et-0/1/1.3434
                      to 143.235.33.95 via ae3.3439, label-switched-path et-0/1/1.3434:BypassLSP->143.235.32.10

==================

[@bar cms]$ whois AS202425
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to 'AS196608 - AS213403'

as-block:       AS196608 - AS213403
descr:          RIPE NCC ASN block
remarks:        These AS Numbers are assigned to network operators in the RIPE NCC service region.
mnt-by:         RIPE-NCC-HM-MNT
created:        2021-11-26T06:58:53Z
last-modified:  2021-11-26T06:58:53Z
source:         RIPE

% Information related to 'AS202425'

% Abuse contact for 'AS202425' is 'abuse@ipvolume.net'

aut-num:        AS202425
as-name:        INT-NETWORK
org:            ORG-IVI1-RIPE
import:         from AS3356 accept ANY
import:         from AS57717 accept ANY
export:         to AS3356 announce AS-IPV
export:         to AS57717 announce AS-IPV
admin-c:        IVI24-RIPE
tech-c:         IVI24-RIPE
status:         ASSIGNED
mnt-by:         RIPE-NCC-END-MNT
mnt-by:         IPV
created:        2018-05-17T11:45:04Z
last-modified:  2019-03-31T22:16:51Z
source:         RIPE
sponsoring-org: ORG-IL465-RIPE

organisation:   ORG-IVI1-RIPE
org-name:       IP Volume inc
org-type:       OTHER
address:        Suite 9
address:        Victoria, Mahe
address:        Seychelles
abuse-c:        IVNO1-RIPE
mnt-ref:        IPV
mnt-by:         IPV
created:        2018-05-14T11:46:50Z
last-modified:  2019-01-31T14:39:36Z
source:         RIPE # Filtered

role:           IPV
address:        Suite 9
address:        Victoria, Mahe
address:        Seychelles
nic-hdl:        IVI24-RIPE
mnt-by:         IPV
created:        2018-05-16T13:28:41Z
last-modified:  2019-01-31T21:21:20Z
source:         RIPE # Filtered

% This query was served by the RIPE Database Query Service version 1.102.2 (BLAARKOP)

[@bar cms]$ whois AS57717
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to 'AS56320 - AS58367'

as-block:       AS56320 - AS58367
descr:          RIPE NCC ASN block
remarks:        These AS Numbers are assigned to network operators in the RIPE NCC service region.
mnt-by:         RIPE-NCC-HM-MNT
created:        2018-11-22T15:27:34Z
last-modified:  2018-11-22T15:27:34Z
source:         RIPE

% Information related to 'AS57717'

% Abuse contact for 'AS57717' is 'abuse@fiberxpress.net'

aut-num:        AS57717
as-name:        FBX-AS
org:            ORG-FB73-RIPE
import:         from AS9002 accept ANY
import:         from AS56611 accept ANY
export:         to AS-ANY announce AS-FBX
admin-c:        FB16391-RIPE
tech-c:         FB16391-RIPE
status:         ASSIGNED
mnt-by:         RIPE-NCC-END-MNT
mnt-by:         nl-fiberxpress-1-mnt
created:        2016-10-20T07:43:27Z
last-modified:  2020-06-15T09:24:08Z
source:         RIPE

organisation:   ORG-FB73-RIPE
org-name:       FiberXpress BV
country:        NL
org-type:       LIR
address:        Bruynvisweg 11
address:        1531AX
address:        Wormer
address:        NETHERLANDS
phone:          +31757112156
admin-c:        FB16391-RIPE
tech-c:         FB16391-RIPE
abuse-c:        AR37944-RIPE
mnt-ref:        nl-fiberxpress-1-mnt
mnt-by:         RIPE-NCC-HM-MNT
mnt-by:         nl-fiberxpress-1-mnt
created:        2016-10-13T13:20:58Z
last-modified:  2020-12-16T13:19:34Z
source:         RIPE # Filtered

role:           Fiberxpress BV
address:        Bruynvisweg 11
address:        1531 AX
address:        Wormer
address:        Netherlands
nic-hdl:        FB16391-RIPE
mnt-by:         nl-fiberxpress-1-mnt
created:        2019-01-31T17:09:24Z
last-modified:  2019-01-31T17:09:24Z
source:         RIPE # Filtered

% This query was served by the RIPE Database Query Service version 1.102.2 (ANGUS)


[@bar cms]$ whois AS6939

#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/resources/registry/whois/inaccuracy_reporting/
#
# Copyright 1997-2022, American Registry for Internet Numbers, Ltd.
#


ASNumber:       6939
ASName:         HURRICANE
ASHandle:       AS6939
RegDate:        1996-06-28
Updated:        2003-11-04    
Ref:            https://rdap.arin.net/registry/autnum/6939


OrgName:        Hurricane Electric LLC
OrgId:          HURC
Address:        760 Mission Court
City:           Fremont
StateProv:      CA
PostalCode:     94539
Country:        US
RegDate:        
Updated:        2018-02-09
Ref:            https://rdap.arin.net/registry/entity/HURC

ReferralServer:  rwhois://rwhois.he.net:4321

OrgTechHandle: ZH17-ARIN
OrgTechName:   Hurricane Electric
OrgTechPhone:  +1-510-580-4100 
OrgTechEmail:  hostmaster@he.net
OrgTechRef:    https://rdap.arin.net/registry/entity/ZH17-ARIN

OrgAbuseHandle: ABUSE1036-ARIN
OrgAbuseName:   Abuse Department
OrgAbusePhone:  +1-510-580-4100 
OrgAbuseEmail:  abuse@he.net
OrgAbuseRef:    https://rdap.arin.net/registry/entity/ABUSE1036-ARIN

RTechHandle: ZH17-ARIN
RTechName:   Hurricane Electric
RTechPhone:  +1-510-580-4100 
RTechEmail:  hostmaster@he.net
RTechRef:    https://rdap.arin.net/registry/entity/ZH17-ARIN


#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/resources/registry/whois/inaccuracy_reporting/
#
# Copyright 1997-2022, American Registry for Internet Numbers, Ltd.
#



Found a referral to rwhois.he.net:4321.

%rwhois V-1.5:0012b7:00 concierge.he.net (HE-RWHOISd v:8e2663b)
asn:ID;I:AS6939
asn:Auth-Area:asns
asn:Class-Name:asn
asn:ASN;I:AS6939
asn:ASName;I:HURRICANE
asn:Tech-Contact;I:POC-HE-NOC
asn:Abuse-Contact;I:POC-HE-ABUSE
asn:NOC-Contact;I:POC-HE-NOC
asn:Created:20101117005228000
asn:Updated:20101117005228000

contact:ID;I:POC-HE-NOC
contact:Auth-Area:contacts
contact:Class-Name:contact
contact:Name:Network Operations Center
contact:Company:Hurricane Electric
contact:Street-Address:760 Mission Ct
contact:City:Fremont
contact:Province:CA
contact:Postal-Code:94539
contact:Country-Code:US
contact:Phone:+1-510-580-4100
contact:E-Mail:noc@he.net
contact:Created:20100901200738000
contact:Updated:20100901200738000

contact:ID;I:POC-HE-ABUSE
contact:Auth-Area:contacts
contact:Class-Name:contact
contact:Name:Abuse Department
contact:Company:Hurricane Electric
contact:Street-Address:760 Mission Ct
contact:City:Fremont
contact:Province:CA
contact:Postal-Code:94539
contact:Country-Code:US
contact:Phone:+1-510-580-4100
contact:E-Mail:abuse@he.net
contact:Created:20100901200738000
contact:Updated:20100901200738000
contact:Comment:For email abuse (spam) only








===============


[net@bar cms]$ cat hosts.junos.ASBR | sed 's/\.conf//' | xargs -n1 -P30 jlogin -x cmd.junos.pfx 

============


[net@bar cms]$ cat hosts.junos.ASBR 
r-chicago-600w.conf
r-chicago-710.conf
r-minneapolis-511.conf
r-uwmadison-cssc-2.conf
r-uwmadison-cssc.conf
r-uwmilwaukee-ems.conf
[net@bar cms]$ cat cmd.junos.pfx 
edit

edit policy-options prefix-list sync_lists-naughty_source_subnets
set 80.82.64.0/24
annotate 80.82.64.0/24 "uwsuperior 2022/03/02"
top

commit confirmed 5

exit



[m7h@bar bin]$ whois 80.82.64.0
% This is the RIPE Database query service.
% The objects are in RPSL format.
%
% The RIPE Database is subject to Terms and Conditions.
% See http://www.ripe.net/db/support/db-terms-conditions.pdf

% Note: this output has been filtered.
%       To receive output for a database update, use the "-B" flag.

% Information related to '80.82.64.0 - 80.82.64.255'

% Abuse contact for '80.82.64.0 - 80.82.64.255' is 'abuse@ipvolume.net'

inetnum:        80.82.64.0 - 80.82.64.255
netname:        NET-1-64
descr:          IPV NETBLOCK
country:        NL
geoloc:         52.370216 4.895168
org:            ORG-IVI1-RIPE
admin-c:        IVI24-RIPE
tech-c:         IVI24-RIPE
status:         ASSIGNED PA
mnt-by:         IPV
mnt-lower:      IPV
mnt-routes:     IPV
created:        2010-09-19T16:51:12Z
last-modified:  2019-02-01T18:24:55Z
source:         RIPE

organisation:   ORG-IVI1-RIPE
org-name:       IP Volume inc
org-type:       OTHER
address:        Suite 9
address:        Victoria, Mahe
address:        Seychelles
abuse-c:        IVNO1-RIPE
mnt-ref:        IPV
mnt-by:         IPV
created:        2018-05-14T11:46:50Z
last-modified:  2019-01-31T14:39:36Z
source:         RIPE # Filtered

role:           IPV
address:        Suite 9
address:        Victoria, Mahe
address:        Seychelles
nic-hdl:        IVI24-RIPE
mnt-by:         IPV
created:        2018-05-16T13:28:41Z
last-modified:  2019-01-31T21:21:20Z
source:         RIPE # Filtered

% Information related to '80.82.64.0/24AS202425'

route:          80.82.64.0/24
origin:         AS202425
remarks:        +-----------------------------------------------
remarks:        | For abuse e-mail abuse@ipvolume.net
remarks:        | We do not always reply to abuse.
remarks:        | But we do take care your report is dealt with!
remarks:        +-----------------------------------------------
mnt-by:         IPV
created:        2019-01-24T15:07:49Z
last-modified:  2019-02-01T12:32:15Z
source:         RIPE

% This query was served by the RIPE Database Query Service version 1.102.2 (ANGUS)




If a decision is made to block, we'd probably look to see if it's already blocked.  We'd want to update the annotated time, or we better be ready to store the annotated time on the side in JSON, because at some point we're going to need to develop some kind of tooling to remove a block

A human then mass pushes to update the existing prefix-list

example:

cat hosts.junos.ASBR | sed 's/\.conf//' | xargs -n1 -P30 jlogin -x cmd.junos.pfx 

[net@bar cms]$ cat hosts.junos.ASBR 
r-chicago-600w.conf
r-chicago-710.conf
r-minneapolis-511.conf
r-uwmadison-cssc-2.conf
r-uwmadison-cssc.conf
r-uwmilwaukee-ems.conf

[net@bar cms]$ cat cmd.junos.pfx 
edit

edit policy-options prefix-list sync_lists-naughty_source_subnets
set 80.82.64.0/24
annotate 80.82.64.0/24 "uwsuperior 2022/03/02"
top

commit confirmed 5

exit



We can look at rrds, reports and get alarms just like you'd expect

examples:
https://stats.uwsys.net/cgi-bin/shorten.fcgi?i=6108&c=c2185800f05c5699

from  https://fido.uwsys.net/cgi-bin/private/fido_show_datafile.cgi?data_file=fido_jnxFirewall_alarms_Packet_thresholds.bin

  'device=r-chicago-710.uwsys.net_filter=antispoof-in-ae2-2292-i_counter=uwsuperior-packetPolicer-1Kpps-ae2-2292-i_type=policer_jnx-xml-FWPacketCount.rrd' => {
    'rule' => '120',
    'threshold' => '100'
  },




Keywords:DoS: investigating a SYN flood   Doc ID:117080
Owner:Michael H.Group:University of Wisconsin System Network
Created:2022-03-02 14:20 CSTUpdated:2022-03-07 13:20 CST
Sites:University of Wisconsin System Network
Feedback:  0   0