Many grant applications go through two rounds of external review, once after LOI and again after Full App, during the proposal stage. This section gives instructions on how to assign and manage external reviewers, as well as considerations for assigning review at both the First and Second External Review States.

Where to Look

1. In Fluxx, open the grant to update and navigate to the Internal section.
2. Expand the Reviewers drop down. 
   • The green plus button enables you to add new reviewer sets.
   • The red negative button lets you delete review sets and all associated review records. (Do not use this option unless you know exactly what data is affected and you're okay losing that data.)
   • The blue page and pencil icon lets you edit a review set. Depending on which edits you make, you can also end up deleting reviewers' comments and scores, so use this with caution. See the Edit Review Sets section below for more info.
     

Assign Reviewers

1. In Fluxx, go to the Reviewers section.
2. Click the green plus button
3. In the Add Reviewer Set window, configure the following:
   1. Reviewer: Enter one or more reviewers. If the Review Status and Request Review Theme should be different for certain reviewers, create different review sets.
      Note: The following fields apply to ALL reviewers, so 
   2. Reviewer group: Ignore (this field allows you to add a group of pre-determined reviewers; however due to the way review is often assigned out WPP doesn't tend to use this tool)
   3. Review Status: Enter the status (select only one) at which review forms should be sent out. Typically, select either First External Review or Second External Review. 
      Note: Review records aren't created until the grant is in the selected status. So, you can assign reviewers at an earlier stage, and they won't be able to access the forms until the grant moves into this state. 
      Also note: If a grant is already in the selected state, review records are created immediately. If the grant is past the selected state, Fluxx does not generate the review records.
   4. Request Review Theme: Select the form template you want reviewers to fill out for this grant.
4. Click Save

Edit Review Sets

Editing review sets can lead to data loss, so review this section carefully to determine whether it's safe to edit. When in doubt, reach out to your Fluxx administrator. 

When it's safe to edit a review set:

• The grant has not yet reached the designated Review State, and no review records have been created.
• None of the reviewers in the set have accessed their review forms yet.
• To remove a reviewer from a review set as long as it is okay to delete their review record in Fluxx.

When not to edit a review set:

• For Second External Review. Overriding review sets for first external review can lead to data loss, and your second external review forms will not generate correctly. In this case, create new review sets.
• When review forms are partially filled out. Changing the review state or theme could cause duplicate forms to be created or the deletion or corruption of existing data. Instead, delete the existing set and create a new one (and inform your reviewers).

Troubleshooting

(This section is primarily for Fluxx admins)

Highlights of how this functionality works:

• It is completely safe to add multiple reviewers to a review set. However, I don't know for sure if you can assign both individual users and reviewer groups. Or even if it does let you, I'm not sure what might happen with duplicate assignments. In any case WPP doesn't tend to use groups because they typically assemble a team of reviewers and assign a few grants to each reviewer, rather than having all reviewers review all grants. If they start asking, though, you'll need to do some testing.
• The review forms aren't created until the grant is in the specified Review State. I really don't know what happens if you assign multiple review states. Since the review teams for first and second rounds tend to be different, and they use different forms for each, I doubt we'll get asked much about this.
• If you change the theme, shared fields preserve their data. Some examples to show the ramifications:
  • If you generate an LOI review form, and a reviewer enters a response into Generic #1 field, and you change the theme to Full review form and it also has the Generic #1 field, their response will show up in the new form. 
  • If you generate an LOI review form, and a reviewer enters a response into Generic #2 field, and you change the theme to Full review form and it doesn't have the Generic #2 field, their response will show up in a data export but not the form.
  • If you generate an LOI review form, and a reviewer enters their responses, and you change the theme to another form (doesn't matter whether the field is on it or not), and then you change it back to LOI, all responses will be preserved.

Overriding Review Sets Issue

There aren't many guardrails around this feature, particularly around editing. If we allow staff to edit at all, they can corrupt or lose review data, but it's not always problematic.

One idea I explored (ultimately didn't work) was to create two Reviewer components, one for first review and another for second, so we can lock down different review sets and prevent this issue. Unfortunately, turns out if you add multiple Reviewer components they merely duplicate each other on the same form (I don't think they create multiple review records, just that they show the exact same review sets). So that's a no go. 

I reached out to Fluxx and they said they'd do some research and let me know if they came up with any ideas for our workflow, but who knows if they'll come back with anything useful.

For now, my solution is to create this document to explain the review assignment process, and possibly conduct a training letting everyone know what not to do. I also want to check with leadership whether we should consider taking away edit capabilities. 

References

• Fluxx documentation here. Note that we use the Ad Hoc method of reviewer assignment.
• Fluxx ticket regarding the overridden review sets encountered in August 2023: FS-59392

Attachments:

image2023-9-5_11-16-49.png (image/png)