Cybersecurity Elastic Logging Service

This document describes how to gain access to the UW-Madison Office of Cybersecurity's Elastic logging service and how to perform basic tasks such as searching, visualizing data, and creating dashboards.
The Office of Cybersecurity's logging service is a collection of systems centered around the Elastic Cloud Enterprise (ECE) product to provide an enterprise-scale logging platform for UW-Madison administrators. The systems include syslog services, Beats collection agents, Kafka queuing, Logstash parsing and enrichment, Elastic services for storage and searching, and Kibana for front-end operations.

Currently, only logs from central sources are accessible via the Elastic service:
  • Palo Alto firewall system (including GlobalProtect VPN) logs
  • Palo Alto firewall threat logs
  • Palo Alto firewall traffic logs
  • Suricata threat logs
We are not accepting logs from non-DoIT sources for the time being due to funding constraints; in the future we hope to have the capacity for receiving non-DoIT-sourced logs.

  •  Access
    •  
    • Access is generally authorized by subnet, similarly to how AANTS is authorized. To request access to the service, please send an email to cybersecurity@cio.wisc.edu.

  •  Service URL
    •  
    • The Kibana user interface can be found here:
    •  
    • https://csoc.ece.sec.wisc.edu:9243/
    •  
    • Log in using your UW NetID and Duo. The interface is only accessible from a UW-Madison campus IP address (either on campus or via VPN).
    •  

  •  User Interface Navigation
    •  
    • The navigation buttons are located along the upper left of the screen:
    •  
      • Kibana user interface navigation buttons
      • Discover: Basic displaying, filtering, and sorting
      •  
      • Dashboards: Collections of visualizations showing related data in a single location
      •  
      • Visualize: Data displayed in graphical format
      •  
      • Management: Management options (not available to most users)
    •  

  •  Discover (Searching and Displaying Data)
    •  
    • "Discover" displays your data in tabular format. From here, you can search, filter, sort, and download your data. In addition, you can change how the data is displayed by selecting which fields you're interested in. A typical Discover screen looks like this:
    •  
    • The discover screen
    •  
    • Searching for data: In the search bar, type in a search using Kibana Query Language (KQL), in the format field:value. Some examples:
      • Find all entries having source IP address equal to 10.1.1.1:
      source.ip:10.1.1.1
      • Find all entries having destination IP address equal to 10.1.1.1 and destination port equal greater than 1024:
      destination.ip:10.1.1.1 and destination.port>=1024
      • Find all entries denied inbound from the UK or Russia:
      event.outcome:"deny" and (panw.panos.source.geo.country_iso_code:"United Kingdom" or panw.panos.source.geo.country_iso_code:"Russian Federation")

      Please see the KQL guide for a more complete reference to KQL syntax.
    •  
    • Filtering data: While searching uses a scoring-based system to determine how relevant a document is to your query, filtering requires documents to exactly match all criteria in the filter. Filters are also cacheable, which makes them faster to run when querying large datasets. To set a filter, click on "+ Add Filter" and specify the required parameters:

      Adding a filter

      Once a filter is applied, you can change your filter options by clicking on the filter to show the drop-down menu. For example, you can edit the existing filter, invert the filter by choosing "Exclude results," temporarily disable the filter, or delete it:

      Filter options
    •  
    • Choosing which columns to display: Find the field(s) you are interested in from the "Available fields" list on the left, and hover over the field name. Click the "add" button that appears, and repeat for each field of interest:

      Adding a field as a column to display

      Those columns will now be visible in the data viewing area in the center of the page:

      Field headers

      By hovering over a column header, you can sort, move columns left or right, or remove the column from the display:

      Field header options
    •  
    • Saving your view: Once your data is displayed in a format you're satisfied with, you can save the view by choosing "Save" from the menu in the top left. Saving a view is a prerequisite for downloading data (see "Downloading data" in this section) or creating a visualization from scratch (see the "Visualizing Data" section).
    •  
    • Saving a view in discovery
    •  
    • Type a title for the view and click the "Save" button:
    •  
    • The save view dialog box
    •  
    • Downloading data: To export your data to CSV and download it, choose "Share" from the menu at the top left, then choose "Generate CSV":
    •  
    • Downloading data

  •  Visualizing Data
    •  
    • Visualizations provide a way of seeing your data in various graphical formats. To generate a quick visualization from the Discover screen, click on a field from the listing to the left. The top five values within the first 500 results appear. Click the "Visualize" button:
    •  
    • Creating a visualization using the field list
    •  
    • The graph defaults to displaying the total count of events grouped by the field you selected:
    •  
    • A visualization graph
    •  
    • To change the aggregation calculation to something other than count (e.g., the average, max, min, or sum), find the "Metrics" pane to the right and expand the Y-Axis. On this screen, you can also choose which field is used to aggregate your data (other than simply by unique event):
    •  
    • Changing the metrics of a visualization
    •  
    • By default, the data is split into "buckets" by the field you chose. The top thirty such buckets are displayed in descending order. You can also change which bucketed field to display, the number of results to show, and the sort direction, or you can give the field a custom label. To do this, find the "Buckets" pane on the right and expand the X-Axis:
    •  
    • Visualization data buckets
    •  
    • Rather than generating a visualization from the Discovery page, you can select an existing visualization or create a new one from scratch by clicking on the Visualizations navigation icon (see "User Interface Navigation" above):
    •  
    • Listing existing visualizations
    •  
    • Clicking the "Create visualization" button presents a variety of options for displaying your data:
    •  
    • Creating a new visualization from scratch
    •  
    • After choosing the appropriate display option, choose the data source (this is a view you previously saved--see "Saving your view" above):
    •  
    • Choosing a data source for a visualization
    •  
    • The visualization will default to aggregating by event count, but the data will initially not be "bucketed" by a field. To choose a "bucket," find the "data" pane on the right, then click "+ Add". Choose an appropriate aggregation, field, sort order, and number of values to include in the visualization. Then click "Update":
    •  
    • Visualization data displayed
    •  
    • To change display options, such as whether to show labels or which display type to use for your particular visualization, click on the "Options" pane on the right. Set the appropriate values and click the "Update" button:
    •  
    • Changing visualization display options
    •  
    • Your data will now show in the viewing area:
    •  
    • Visualization graph
    •  
    • To save your visualization (required for using it in a dashboard), choose the "Save" menu option in the upper left:
    •  
    • Saving your visualization
    •  
    • Give your visualization a title:
    •  
    • Giving your visualization a title

  •  Dashboards
    •  
    • To create a dashboard, click the Dashboard navigation icon on the left. If this is your first dashboard, you will see the following screen:
    •  
    • Creating a dashboard
    •  
    • After clicking "Create new dashboard," you will be presented with a blank dashboard. Add a new or existing visualization to your dashboard by clicking either "Create new" or "Add an existing," respectively:
    •  
    • Adding a visualization to your dashboard
    •  
    • If you clicked "Create new," you will be directed to the "New Visualization" screen (see the "Visualizations" section above). Adding an existing visualization will bring you to the add panels screen. Choose the visualization you would like to add to the dashboard:
    •  
    • Selecting an existing visualization to add to your dashboard
    •  
    • Once a visualization has been added to the dashboard, you can change it by clicking the gear icon in the upper right-hand corner of the visualization panel:
    •  
    • Changing a visualization
    •  
    • The following options are available:
      • • Edit search: Modify the search parameters underlying the visualization. For example, you can change the sort order or which field is used to aggregate the data.
      • • Replace panel: Replace the visualization with a different one.
      • • Customize panel: Change the panel's title and choose whether to display or hide the title.
      • • Inspect: View statistics about the search, for example how long it took to complete, the number of hits returned, and the json used to query the data.
      • • Full screen: Maximize the visualization across the height and width of the browser.
      • • Delete from dashboard: Remove the visualization from the dashboard.
    •  
    • If you'd like to add another visualization to the dashboard, click the "Add" menu item at the top of the screen:
    •  
    • Adding additional visualizations to a dashboard
    •  
    • When you're done creating or editing your dashboard, click the "Save" menu item in the upper left. The following dialog is displayed:
    •  
    • Saving a dashboard
    •  
    • If you need to make a change to your dashboard, choose "Edit" from the menu in the upper left:
    •  
    • Editing a dashboard
    •  
    • Alternatively, you can edit an existing dashboard by clicking the Dashboard navigation icon, which will display a list of existing dashboards from which to choose:
    •  
    • Editing an existing dashboard using the navigation buttons

Keywords:
cybersecurity cyber security elastic logging syslog log logs service 
Doc ID:
104220
Owned by:
Michael I. in Cybersecurity
Created:
2020-07-22
Updated:
2021-04-28
Sites:
Office of Cybersecurity