School of Education Guidelines for Generative AI

While generative AI offers many benefits, it also introduces real risks and limitations that we should understand before using it in classrooms, research, or institutional work. These risks do not mean AI should be avoided—but they do mean it should be used carefully, transparently, and with human oversight.

1. Privacy and Data Protection Risks

One of the most important concerns is what happens to the information you provide to an AI tool.

Key points to understand:

• Many AI tools store prompts and responses to improve their systems
• Data entered into public or consumer tools may not be private and AI responses for other users could contain or use that data
• Content you input could potentially be accessed by developers or used in training
• When using regulated data, or data protected by an agreement/contract, use of some AI systems can constitute a breach or unauthorized disclosure of data, potentially resulting in financial, legal, reputational, or other harms.

Potential risks include:

• Entering student data, grades, health information, or identifiable details
• Uploading unpublished research, drafts, or sensitive institutional materials
• Sharing confidential discussions or assessment content

Practical guidance:

• Do not enter personally identifiable student information (see FERPA: https://studentprivacy.ed.gov/content/personally-identifiable-information-education-records)
• Avoid uploading confidential, protected, or proprietary material
• Prefer institution‑approved or enterprise tools with privacy agreements
• Treat AI tools as public spaces, not private notebooks

2. Security and Intellectual Property Concerns

AI introduces new questions about ownership and control of content.

Common risks include:

• Loss of intellectual property when sharing original ideas or drafts
• Unclear ownership of AI‑generated materials
• Accidental disclosure of protected research or assessment materials

For researchers and faculty:

• Drafts shared with AI tools may no longer be fully private
• Some publishers and funding bodies restrict AI use without disclosure
• AI‑generated content may not qualify for copyright protection in some jurisdictions

Best practices:

• Check below for UW-Madison institutional policies on the use of AI
• Disclose AI use in research and teaching where appropriate
• Use AI for brainstorming or editing support—not as the sole author

A critical consideration when using generative AI is how reliable its outputs are—and when they should (and should not) be trusted. Although AI systems often sound confident and polished, this does not guarantee accuracy, fairness, or objectivity.

1. Accuracy Limitations and Fabricated Information

Generative AI systems do not verify facts. Instead, they generate responses based on patterns in data, predicting what a plausible answer should look like.

As a result, AI may:

• Produce incorrect explanations that sound convincing
• Invent citations, quotes, or references that do not exist
• Combine correct facts in misleading or inaccurate ways
• Oversimplify complex topics without signaling uncertainty

This tendency is often called “hallucination”—not because the AI is imagining things, but because it lacks a mechanism to distinguish truth from plausibility.

Potential implications:

• Students may assume AI responses are authoritative
• Errors may be harder for novices to detect
• Misconceptions can be reinforced if AI output is used uncritically

Best practices:

• Treat AI output as a starting point, not a final answer
• Require verification against trusted sources
• Use AI examples as opportunities to teach fact‑checking and evaluation

2. Bias, Representation, and Fairness

Generative AI learns from large collections of human‑created content. Because human knowledge and culture contain biases, AI systems can reproduce or amplify those biases.

This may appear as:

• Stereotypical or exclusionary language
• Uneven representation of cultures, perspectives, or disciplines
• Assumptions based on norms that do not apply universally
• Biased framing of historical, social, or ethical topics

Importantly, AI does not understand bias—or recognize when its outputs may be unfair or harmful.

Why this matters:

• AI‑generated materials may unintentionally marginalize students
• Feedback or examples may reflect dominant cultural viewpoints
• Seemingly “neutral” outputs can still shape perceptions and values

Mitigation strategies:

• Review all AI‑generated content before sharing
• Encourage multiple perspectives and alternative viewpoints
• Explicitly discuss bias as part of AI literacy

3. Why AI Confidence Can Be Misleading

One of the risks to trustworthiness is how confident AI appears.

Generative AI:

• Uses fluent, professional language
• Rarely signals uncertainty on its own
• Can present errors with the same tone as correct information

This creates a mismatch between presentation and reliability.

This means:

• Accuracy cannot be judged by tone alone
• AI should never replace subject‑matter expertise

4. A Healthy Trust Model for AI Use

A useful rule of thumb is:

Trust generative AI for form, structure, and ideas—but not for facts without verification.

In practice:

• Trust it to help brainstorm, summarize, or rephrase
• Do not trust it to be factually correct by default
• Always pair AI use with human review and judgment

Different AI tools or services are available to use at UW-Madison depending on what type of data you will use with them. Data is classified as either low risk (public or internal), moderate risk (sensitive) or high risk (restricted). When data includes elements from multiple categories, the most restrictive label must be used. See Information Security: Data Classification | UW Policies for more information. If you have questions about your data, please contact MERIT at helpdesk@education.wisc.edu for assistance.

Public Data - Data made available for public use and consumption and can be openly shared or discussed with anyone (examples).

• You may use any AI tool or service if the data being used with it is public data
• Ensure the data is public prior to using the AI tools or services because it often cannot be recovered or deleted once it has been used

Internal Data - Data made available to internal users but not intended to be openly shared on public websites or in other public forums (examples).

• Tools approved and licensed by UW-Madison. Approval is only through the campus-managed instance of these tools, not the public/free tier or an individual subscription:
  • Google Gemini
  • Microsoft Copilot Chat
  • NotebookLM

Sensitive Data - Data limited to defined users, roles, or groups within the organization based on specific business needs. By default, all institutional data that is not explicitly classified must be treated as Sensitive data (examples).
-or-
Restricted Data - Data limited to a highly defined small set of users (examples).

• The School of Education's Amazon Transcription Service has been reviewed and approved for use with sensitive and restricted data
  • This service can transcribe audio and video recordings to MS Word documnets in bulk
  • Contact MERIT to get started at helpdesk@education.wisc.edu 
• Local/offline instance of an AI model/tool only when no data is available to the vendor or used to train the model
  • For example, OpenAI's Whisper model, running locally on a desktop computer
• Campus-licensed and managed AI meeting assistants
  • Webex AI Meeting Assistant
  • Secure Zoom AI Companion
    • Secure Zoom may be used with PHI/HIPAA data
  • See the SoE guidelines for AI meeting assistants for more information on these tools
• All other AI tools or services must receive a documented risk assessment before they can be used with sensitive or restricted data

New AI tools or services must receive a documented review by designated IT staff to validate compliance with state and federal laws and Universities of Wisconsin and UW-Madison policies. These types of reviews may also be required by industry partners and collaborators as a condition of data sharing, grant funding, or other agreements. Some reviews can include:

• Cybersecurity risk assessment
• Institutional data governance
• Data privacy
• Accessibility
• Purchasing
• Institutional review board (IRB)

To request a new product or tool, start by submitting this form: https://form.asana.com/?k=3-2KbiuV1wURWDMIqaGNzQ&d=941838868189333.

More AI models are available for custom solutions via the UW-Madison managed cloud environments (Amazon Web Services, Google Cloud Platform, and Microsoft Azure) on a pay-as-you-go basis. Please contact MERIT to get started with one of these options at helpdesk@education.wisc.edu

• Generative AI @ UW–‍Madison: use & policies - UW–⁠Madison Information Technology
• AI at work: A practical guide to using UW’s tools safely - UW–⁠Madison Information Technology
• Generative AI Services at UW–Madison: Tools, Policies & Resources
• Legal Affairs on "Free" Generative AI tools - UW–⁠Madison Information Technology
• Home - Generative AI - Research Guides at University of Wisconsin-Madison
• Ethics and Generative AI - Generative AI - Research Guides at University of Wisconsin-Madison
• AI Data Security: Best Practices for Securing Data Used to Train & Operate AI Systems | CISA
• AI Risks and Trustworthiness - AIRC