Research Method

Method

Participatory design

We will conduct interview sessions with 10 MSM with HIV positive status and 5 healthcare providers via zoom. The audio of the zoom sessions will be recorded. Two separate sessions at different days and times will be held to maximize the attendance of 10 patients. For participants who cannot or are unwilling to attend the group workshop, individual interviews will be offered as an alternative (Marent et al. 2018). Separate one-on-one sessions will be scheduled with healthcare providers if they cannot all meet at the same time. The sessions will be recorded, transcribed, and analyzed using content analysis (Hsieh & Shannon, 2005). Participants can choose to turn off their cameras if they have privacy concerns.

During the participatory design sessions, The Turn to Sunshine software will be presented to the participants in the format of a clickable prototype designed using Figma. Participants will have 10 minutes to explore the software prototype. Then, the researcher will elicit feedback on the content and structure of the software. A notetaker will record the suggestions and start drawing the preferred structure of the software and list out the preferred content. Then the researchers will present the sketch of the new design to the participants and invite participants to suggest further changes to produce an initial prototype for the main interfaces. The research team will implement the prototype to develop a WeChat-mini program, which will go through at least two rounds of rapid iterative testing and evaluation.

Usability testing

We will conduct 2 rounds of remote usability testing sessions with at least 5 individuals newly diagnosed with HIV. The audio of the zoom sessions will be recorded. Five is considered as an acceptable sample size for identifying initial usability issues (Macefield, 2009). Participants in usability testing must have not participated in the PD workshop sessions, considering that previous involvement in the design might hinder their ability to stay critical of the product.

All usability testing sessions will be conducted and recorded via zoom. We will examine five dimensions of usability, including learnability, efficiency, memorability, errors, and satisfaction.

Each usability testing session will include a participant and a moderator. The moderator will assign tasks and use open-ended questions to guide participants and to assess the ease of use for user interfaces. The moderator will encourage participants to “verbally express their thoughts as they complete each task”, which could reveal difficulties in completing tasks and provide information for researchers to improve the app. After completing each task, participants will answer open-ended questions about the user experience and fill out a measure to assess the perceived usability of the application. After each session, the researchers will review the session recording, discuss, and write down notes for improving usability.

The aim of this round is to test how well the mini program fits into participants’ life. The participants will be instructed to install the mini program, explore the modules, and interact with the mini program for a week. The frequency and duration of interaction and features that the participants interacted with will be recorded and analyzed to examine user engagement. Participants will fill out SUS again to indicate perceived usability and a measure of acceptability to indicate how likely they are going to use the mini program and recommend it to others.

Round 3

This round focuses on system usability and the content of the intervention. For this round we will recruit individuals who are newly diagnosed with HIV. We will improve the prototypes based on the feedback from the first round. The same tasks in round 1 will be completed with a few extra tasks assessing the second week materials in the intervention. After the usability session, participants will fill out the SUS and Post-Study System Usability Questionnaire (PSSUQ; Sauro & Lewis, 2016). The PSSUQ is a 19-item scale with a 7-point Likert response scale (1 = “strongly agree” to 7 = “strongly disagree”) that assess memorability. The four subscales in PSSUQ include: system usefulness, information quality, interaction quality, and overall usefulness. The response from all items will be summed to produce a total score, with a high total score suggesting a higher memorability.

Round 2

Round 1

This round focuses on the user flow of the app rather than the content. Therefore, we recruit people who come to the clinic but do not necessarily have an HIV positive status. We will evaluate the usability of week 1 materials of the intervention. The tasks include using the provided link to install the mini program, creating a user profile, logging in, and finishing the first week’s reading and homework, and reporting adherence in the past week. After the tasks are completed, participants will fill out the System Usability Scale (SUS; Sauro, n.d.; Sauro & Lewis, 2016). The SUS is a 5-point Likert scale (1 = “strongly agree” to 5 = “strongly disagree”) with 10 items that assess system usability and learnability (Bangor et al., 2009; Peres et al., 2013). SUS has been validated in many studies and its reliability has been demonstrated (Bangor et al., 2009). The total score of all items will be converted to a 0-100 scale, with a higher score suggesting better usability (Sauro & Lewis, 2016).