Background

Counterbalancing is how we randomly assign an equal number of participants to multiple groups within a study. For example, whether participants receive an F1 shift up vs down on a particular stimulus word.

Tracking counterbalancing

Each experiment with counterbalanacing has a file name cbPermutation_[experiment name] (counterbalancing permutation). That file normally lives on the Waisman server in /smng/experiments/[experiment name]/. Loading in that file will load in a variable just called "cbPermutation". cbPermutation is formatted so that each row represents a set of conditions that a participant might be assigned to. Each column represents some indicator of how to set those conditions (see example below). The final column is a count of how many times that row has been used, ie, how many other participants have been assigned that set of conditions.

Caption for above picture: This is the cbPermutation variable from diphthongAdapt. Since there are two rows, there are two conditions a participant could be assigned to. The first is [1 0], the second is [-1 0]. In the context of diphthongAdapt, these values are used to control the amount of F1 shift on two words. Ie, the first word has an F1 shift up or down, and the second word never has a shift, regardless of what condition the participant is assigned to. Column 3 shows that row 1 has been used 16 times, and column 2 has been used 14 times. This is normally an impossible situation, since the two rows should never be more than 1 value out of sync. But the values can be manually edited if needed.

Caption for above picture: This is the cbPermutation variable for vsaPD. Several settings are controlled by the cbPerm file. In this case, columns 1-3 are settings used for visit 1 of the study, and columns 4-6 used for visit 2. Unlike for diphthongAdapt, it's mostly text. Since cbPerm is a cell array, most data types can be used. Column 7 is still a numeric count of how many times that row has been used.

When running an experiment, we have helper functions which will find a row used the fewest number of times (or tied for the fewest), and will return the set of conditions to be used for the current participant. Then at the end of the study, we update the counterbalancing file by incrementing the counter by 1 for the row which was used. If the Waisman server is accessible, that version of the file will be used and incremented. If the server is inaccessible, a version of the cbPerm file on the local drive will be used. If that doesn't exist, it will be created.

If the experiment is run in test mode (bTestMode=1), or if the participant's ID contains the string 'test' or 'pilot', the counter in cbPerm will not be incremented, as it assumes this was a test participant.

One experiment may have multiple "populations" which are tracked in different cbPerm files. For example, we may want 20 "patient" participants to be split 10-10 between two conditions, and the 20 "control" participants to also be split 10-10 between two conditions. In this case, there would be files called cbPermutation_[experiment]_control and cbPermutation_[experiment]_patient. Each file loads in a variable called cbPermutation with 2 rows; since they're in separate files, the counts for each population are counted separately. Most functions take an input argument "population" which can be set to [] if the study only has one population.

New recommendations in 2025

run_diphthongAdapt_expt uses the best practice for counterbalancing setup.

In 2024 and earlier, most experiments determined which row in cbPerm to use, then immediately updated the counter. This caused problems if the experiment ended early and the data from that participant wasn't used; the counts were now off. Now, the count in cbPerm is incremented at the end of the study.

Previously, experiments had code for getting the cbPerm info directly in the run_[experiment]_expt function. Now, that code is in a separate function get_cbPermutation_[experiment]. This helps reduce the length and complexity of the run_[experiment]_expt function.

Previously, the code for generating an experiment's counterbalancing file was not in the experiment code anywhere; people would copy an old cbPerm file and edit it as needed for the new experiment. This caused problems because, when disconnected from the server, there was no way to know what the cbPerm file should look like. Now, the get_cbPermutation_[experiment] function can generate a new cbPerm file for anyone who has the experiment code.

Relevant functions

get_cbPermutation_[experiment]

An experiment-specific function. Call this function near the beginning of run_[experiment]_expt to get the permutation index (row number) and permutation list (cell array with set of conditions). It must be edited for each experiment to define how the cbPerm file should be formatted.

get_cbPermutation

A behind-the-scenes function which loads the actual cbPerm file, picks the correct row, and returns a single set of conditions. Called by get_cbPermutation_[experiment].

set_cbPermutation

Call this at the end of run_[experiment]_expt to increment the count of the row in cbPerm that was used for this participant.