Implementing Unmatched Count Technique in the field

Alegria Olmedo

Ho Chi Minh City

Implementing UCT in the field

The Unmatched Count Technique (UCT) is increasingly used in conservation to measure the prevalence of sensitive behaviours (e.g. purchasing wild turtle meat). There is growing interest in using this technique but its design and implementation can present some challenges, so must not be taken lightly.

As part of the Oxford Martin Programme on Illegal Wildlife Trade Case Study 5, I recently applied UCT to measure the prevalence of pangolin consumption in Ho Chi Minh City (HCMC), Vietnam. This method works by presenting a sensitive behaviour of interest among a list of other related (but not sensitive) items referred to as control items. Respondents are then asked to state how many items apply to them; in this way, respondents are not directly asked about performing the behaviour in question (see this previous blog for further explanation). Although putting together a list of control items somewhat related to a sensitive item seems easy, designing UCT questions is not as simple. To emphasize this point, I share my own experience and learnings in designing and implementing this method.

 

Step 1: LIST DESIGN

I set out to implement a questionnaire with three UCT questions inquiring about the consumption of pangolin meat, pangolin scales and pangolin wine.

I initially decided to use five control items in each of the UCT questions, with the sensitive item (the pangolin product) being the sixth item in the treatment questions. To avoid ceiling and flooring effects (where respondent say that all, or none, of the items on a list applied to them), I was looking to fill my control lists with one high prevalence item (an item that most people consume), two medium prevalence (items that some people consume) and two low prevalence items (those that few people consume), while making sure none of these items were also sensitive. This way, there would be variation in people’s responses to the UCT questions.

Along with my local collaborator, we put together a list of ten potential control items to use for each of the three pangolin UCT questions, so as to have several options for different control items, with various levels of prevalence. This list of thirty control items was then tested in an informal rating exercise, where my team and I opportunistically approached city residents to discuss the items. People were asked to rate whether, in their opinion, each control item was high, medium or low prevalence for residents of HCMC. The results of this exercise were used to create three lists of UCT questions, each one with five control items. The items that were not selected as part of the final lists were those that people were confused by, those that people did not want to discuss (potentially sensitive) and those for which prevalence was still unclear to the research team. Figure 1 shows the control list piloted for the question on pangolin meat (the treatment question is the same but pangolin meat is added to the list).

Please read the list below. How many of the items below have you consumed in the last 12 months (present month included)?

Figure 1. Control question piloted for pangolin meat.
Figure 1. Control question piloted for pangolin meat.

Step 2: PILOTING

During piloting, the UCT questions were introduced to respondents in two different ways: a) half of the Research Assistants presented a questionnaire that included an example UCT question with control items completely unrelated to pangolin consumption, followed by the three pangolin UCT questions. This was to ensure respondents knew how to answer the UCT questions; b) the rest of the Research Assistants presented three printed examples of unrelated UCT questions to respondents, asking them to answer to each question out loud, to ensure the respondent had understood how to answer. These were then followed by the three pangolin UCT questions. All Research Assistants explained that respondents must indicate how many items applied to them, but not which ones. Respondents were randomly assigned to control or treatment groups for each one of the UCT questions of interest.

Responses from this first round of piloting showed that the control items in each one of the lists had a lower prevalence than the rating exercise suggested. This was apparent as more people than expected answered that zero or only one item applied to them, which would result in a flooring effect (See Table 1 for a summary of problems encountered). To understand if this really was the case, direct questions for each one of the control items initially believed to be high and medium prevalence were added at the end of the questionnaire.

The second round of piloting revealed that more than half of the respondents were underreporting their answers in the pangolin UCT questions. Respondents claimed to have consumed more control items when asked directly about them, than when they answered the UCT questions. Underreporting in UCT is not uncommon and has been described in previous research. The level of underreporting was the same regardless of how the example UCT questions were presented to respondents.

Table 1. Prevalence suggested for control items from the first round of piloting.
Table 1. Prevalence suggested for control items from the first round of piloting.

To address this, three aspects were changed for the third round of piloting:

  • Number of control items was reduced from five to four; it has been suggested it is harder for people to keep immediate count of items if the list is longer.
  • Control items were replaced by new lists, as it appeared the first lists included items with lower prevalence than initially expected, or perhaps the terminology used for the items was not as clear as possible, therefore confusing people. We conducted a second opportunistic rating exercise where people were asked if they had consumed each one of the new control items, in addition to their opinion on whether each one was high, medium or low prevalence.
  • Each UCT question was changed to: Please read the list below. How many of the items have you consumed in the last 12 months (present month included)? The products you do not count are those you have NOT CONSUMED EVEN ONCE in the last 12 months. To ensure people were counting products they had consumed even one time.

The questionnaire was piloted a third time and included direct questions relating to the control items at the end of the questionnaire once more. This round resulted in fewer people claiming none or only one of the items in the pangolin UCT questions applied to them, thus avoiding the flooring effect, but the underreporting level remained the same regardless of the way in which these questions were explained to them.

Fortunately, the underreporting rate was the same for control and treatment questions. As the analysis of UCT is based on the difference in means of the treatment and control questions, it would be expected that the underreporting would not affect the difference in means. As time on the field was limited, we proceed with the data collection with the control items selected for the third round of piloting (See Table 2).

Table 2. Control items used for pangolin meat UCT question.
Table 2. Control items used for pangolin meat UCT question.

In summary…

There are a number of things that must be taken into careful consideration when planning the implementation of a successful UCT:

  • The design of the control lists. Although these may seem easy to put together, particularly for those who are familiar with the context of the research at hand, they must be done very carefully and the control items have to be tried out.
  • Assumptions should not be made about control items and whether they are high, medium or low prevalence among the population of interest.
  • Enough time should be budgeted in to pilot UCT questions (sometimes several times), and determine whether flooring or ceiling effects, underreporting, or other problems are taking place so these can be addressed prior to data collection.

I hope my experience sheds some light on the challenges (and patience required) when implementing UCT, and that it’ll help anyone else planning on using this technique.

Check out Alegria's research profile or follow her on twitter @alegriaoc