Background

Counterbalancing is how we randomly assign an equal number of participants to multiple groups within a study. For example, whether participants receive an F1 shift up vs down on a particular stimulus word.

Tracking counterbalancing

Each experiment with counterbalancing has a file name cbPermutation_[experiment name] (counterbalancing permutation). That file normally lives on the Waisman server in /smng/experiments/[experiment name]/. Loading in that file will load in a variable just called "cbPermutation". cbPermutation is formatted so that each row represents a set of conditions that a participant might be assigned to. Each column represents some indicator of how to set those conditions (see example below). The final column is a count of how many times that row has been used, ie, how many other participants have been assigned that set of conditions.

Caption for above picture: This is the cbPermutation variable from diphthongAdapt. Since there are two rows, there are two conditions a participant could be assigned to. The first is [1 0], the second is [-1 0]. In the context of diphthongAdapt, these values are used to control the amount of F1 shift on two words. Ie, the first word has an F1 shift up or down, and the second word never has a shift, regardless of what condition the participant is assigned to. Column 3 shows that row 1 has been used 16 times, and column 2 has been used 14 times. This is normally an impossible situation, since the two rows should never be more than 1 value out of sync. But the values can be manually edited if needed.

Caption for above picture: This is the cbPermutation variable for vsaPD. Several settings are controlled by the cbPerm file. In this case, columns 1-3 are settings used for visit 1 of the study, and columns 4-6 used for visit 2. Unlike for diphthongAdapt, it's mostly text. Since cbPerm is a cell array, most data types can be used. Column 7 is still a numeric count of how many times that row has been used.

When running an experiment, we have helper functions which will find a row used the fewest number of times (or tied for the fewest), and will return the set of conditions to be used for the current participant. Then at the end of the study, we update the counterbalancing file by incrementing the counter by 1 for the row which was used. If the Waisman server is accessible, that version of the file will be used and incremented. If the server is inaccessible, a version of the cbPerm file on the local drive will be used. If that doesn't exist, it will be created.

If the experiment is run in test mode (bTestMode=1), or if the participant's ID contains the string 'test' or 'pilot', the counter in cbPerm will not be incremented, as it assumes this was a test participant.

One experiment may have multiple "populations" which are tracked in different cbPerm files. For example, we may want 20 "patient" participants to be split 10-10 between two conditions, and the 20 "control" participants to also be split 10-10 between two conditions. In this case, there would be files called cbPermutation_[experiment]_control and cbPermutation_[experiment]_patient. Each file loads in a variable called cbPermutation with 2 rows; since they're in separate files, the counts for each population are counted separately. Most functions take an input argument "population" which can be set to [] if the study only has one population.

New recommendations in 2025

In short, here is how you should set up your experiment-running function:

1. Define a variable called expt.cbPerm.allPermConds which contains the full set of counterbalancing permutations. For the diphthongAdapt example above, it would be set to {1 0; -1 0}
2. Set expt.cbPerm.population. For studies with only one group, set it to empty brackets []. For other groups set it to 'control', or 'patient', or whatever you're using.
3. Call the following code to set all relevant expt fields. It will generate a cbPermutation file if one doesn't exist.
   
   [expt.cbPerm.ix, expt.cbPerm.list, expt.cbPerm.savePath, expt.cbPerm.fileName] = get_cbPermutation(expt.name, [], expt.cbPerm.population, [], expt.cbPerm.allPermConds); 
4. At the very end of your run_[experiment]_expt script, after you're finished with all your run_[experiment]_audapter calls, add
   
   %% increment counts in cbPermutation, unless test mode or name has 'pilot'/'test'

   if ~expt.bTestMode && ~any(strfind(expt.snum, 'pilot')) && ~any(strfind(expt.snum,'test'))

   set_cbPermutation(expt.name, expt.cbPerm.ix, expt.cbPerm.savePath, expt.cbPerm.population);

   end

run_diphthongAdapt_expt uses this best practice for counterbalancing setup. You can model your experiment off of it. 

How the current recommendation differs from before

In 2024 and earlier, most experiments determined which row in cbPerm to use, then immediately updated the counter. This caused problems if the experiment ended early and the data from that participant wasn't used; the counts were now off. Now, the count in cbPerm is incremented at the end of the study.

Previously, the code for generating an experiment's counterbalancing file was not in the experiment code anywhere, and get_cbPermutation couldn't make a cbPermutation file if it didn't already exist; people would copy an old cbPerm file and edit it as needed for the new experiment. This caused problems because, when disconnected from the server, there was no way to know what the cbPerm file should look like. Now, the run_[experiment]_expt function itself contains the full list of permutations, so anyone who has the experiment-running code also has the definition of the cbPerm file.

Relevant functions

get_cbPermutation

A behind-the-scenes function which loads the actual cbPerm file, picks the correct row, and returns a single set of conditions. If a cbPerm file doesn't exist, it will automatically call gen_cbPermutation and make a cbPerm file. Called by run_[experiment]_expt. 

set_cbPermutation

Call this at the end of run_[experiment]_expt to increment the count of the row in cbPerm that was used for this participant. If you need to decrement (subtract one) a cbPerm file, call this function on its own and set the bSubtract input arg to 1.

gen_cbPermutation

Generates a new counterbalancing file. Called by get_cbPermutation if necessary.