always-on anti-ddos on uwsys.net

> -----Original Message-----
> From: Michael Hare
> Sent: Monday, December 22, 2025 3:20 PM
> To: UW System Network Discussion List <uwsysnet@uwsys.net>
> Subject: FYI about always-on anti-ddos config changes made on uwsys.net in
> March 2025
> 
> Hi,
> 
> Within the last few months, both Oshkosh and Parkside have reported an
> issue with uwsys.net unexpectedly dropping campus bound traffic.
> Investigation of both incidents ended in a false positive requiring a config
> change in uwsys.net dynamic always-on anti-ddos defense configuration.  The
> main purpose of this message is to convey awareness.  If your campus is
> having unexpected issues with packet loss for specific use cases and is listed
> as a 'possible affected' campus, consider reaching out to me.  What I need to
> confirm or deny our culpability is a timestamp, src/dst ip/port and protocol.
> We can adjust our configuration stance as needed.
> 
> More details on what's happening: automated netflow analysis deploys ACL
> rules on the Junipers that connects uwsys.net to the rest of the internet.
> These ACLs describe traffic that is thought to be safe to mark as discard
> eligible based on said analysis.  Discard eligible packets are dropped first
> upon network congestion.  At the macro level, packet drop rates over time is
> something we monitor and trend.    In the most recent case with UW
> Parkside, this analysis/marking feature in combination with a per-campus
> opt-in TCP syn!ack packet policer at the campus border led to a false positive,
> meaning packets were marked as discard eligible that should NOT have been.
> A list of campuses with the syn!ack policer configured follows below at the
> bottom of this message.  The goal of the syn!ack policer is to reduce severity
> of syn!ack floods on campus managed firewalls.  The syn!ack policer takes
> discard eligible packet markings into consideration.  So if the automated
> analysis is incorrect, it can lead to a situation where traffic is being marked as
> discard eligible and may be dropped instead of forwarded to the campus
> border.   The false positive occurs when the affected campus service is being
> offered on a sparsely populated subnet that talks infrequently.  If the usage
> doesn't across a certain threshold, these destinations may be marked as
> discard eligible.  While there are many options [with varying tradeoffs] to
> change this unwanted behavior, in each case it was decided to bless/bypass
> the holding /24 in the analysis software configuration, resolving the issue,
> albeit removing a bit of protection as well.
> 
> The deployed technique has been useful as an automated
> volumetric/congestion based anti-ddos, and I'm hoping campuses see the
> tradeoff as an acceptable compromise.  Since March 2025 I've been operating
> under the assumption if things were majorly broken, I would have heard
> about them by now, but if you never suspected uwsys.net config as a
> possibility, and if it turns out there is much larger and untenable degree of
> false positives that this message shakes out, we can discuss changes/options
> on a per campus or network wide basis on a future tech call in 2026.
> 
> Based on current configs, this issue might =potentially= affect the following
> UWs as they have a syn!ack border policer configured.
> 
> green bay
> la crosse
> madison
> oshkosh
> parkside
> platteville
> river falls
> stevens point
> superior
> 
> I would not expect these UWs to be impacted, as they do NOT have a syn!ack
> policer configured
> 
> eau claire
> milwaukee
> stout
> whitewater
> 
> -Michael