The Opinions of Multiple Stakeholders Towards Gerontechnology Evaluation Framework: Four Studies Using Delphi Techniques

The 4 studies aim to collect and examine the opinions from multiple stakeholders of gerontechnology product including the demand side, funder, supply side and academia on Gerontechnology Evaluation Framework. The Evaluation Framework will describe the essential knowledge to conduct an evaluation study for 10 selected gerontechnology product themes in local service settings. Specifically, we will (1) develop the indicators to determine if a companion robot, which is one of the product themes, is 'good'; (2) describe feasible study designs, implementation strategies and data analyses to generate evidenced-based evaluation results of companion robots; (3) outline strategies to build organizational research capacity to facilitate the evaluation of gerontechnology products.

In the Preparation Stage of all 4 studies, eligible participants will be recruited under purposive sampling and snowball sampling. Literature review, qualitative interviews and reflection on recent experience of evaluating gerontechnology products will be conducted. The opinions collected from the stakeholders will facilitate questionnaire development for the Delphi Stage. In the Delphi Stage of each of the 4 studies, there will be two rounds of quantitative validation of the grouped statements to generate consensus. Based on the opinions with consensus, the evaluation framework will be revised.

Study Overview

• Status: Completed
• Conditions: Consensus Development
• Intervention / Treatment: Other: End user; Other: Demand side- management; Other: Supply side; Other: Funder; Other: Academia

Detailed Description

Study design For research question (1), two studies, i.e. Study A1 and A2, will be conducted. Study A1 will conduct secondary data analyses on qualitative interviews and a two-round questionnaire using Delphi techniques with end users. Study A2 will conduct focus groups and a two-round questionnaire using Delphi techniques with decision makers. For research question (2), Study B will employ a focus group and a real-time two-round questionnaire using Delphi techniques with decision makers. For research question (3), Study C will employ a real-time two-round questionnaire using Delphi techniques study with decision makers.

For all 4 studies, in the first Delphi round, panellists will rate anonymously the agreement of the opinions on a 5-point Likert scale, give reasons for their ratings and suggest revisions. They will then be presented with the overall analyses and their ratings in the first round. In the second Delphi round, they will rate anonymously the agreement of the revisions, and give reasons for their ratings.

Procedures Preparation Stage In Study A1, for qualitative interviews with product users, we will do a secondary data analysis on individual qualitative interviews with 10 end users and 6 demand side - frontline. All of them were recruited from a residential care home and previously participated in an intervention study using companion robot. Their answers to questions related to 'good' companion robots will be analyzed by framework analysis. The analyzed opinions, upon the approval of the investigators, will be listed as the questionnaire items for use in Delphi part. Additional questionnaire items for the Delphi part will be added based on the literature that explored the opinions of the product users towards companion robots. For the Delphi part, 20 - 60 eligible visitors of Gerontech and Innovation Expo, a local exhibition that provides a platform for stakeholders to discuss and collaborate for broader adoption of gerontechnology, will be recruited by convenience sampling at the time of Expo registration. Their voluntary written consent will be sought before they complete the Delphi questionnaire. They will provide their email address and contact number, so that they could be contacted to do second Delphi round.

In Study A2, for focus groups with decision makers, 5 - 20 decision makers who have joined the development, evaluation or procurement process about companion robots will be purposively recruited by investigators of the project through phone calls and emails. In particular, demand side - managerial personnel will be recruited from the organizations that have adopted companion robots in their service units. Academia will be recruited by purposive sampling on the investigators who have publication(s) on gerontechnology in online databases of PubMed, Web of Science, Cochrane, CINAHL, EMBASE, and by convenient sampling on the nursing educators in the School of Nursing, the University of Hong Kong (HKU) will be conveniently recruited. Funders to be recruited are the representatives nominated Innovation and Technology Fund (that subsidizes social service units in renting and procuring gerontechnology products including companion robots) and. Supply side who expressed willingness to the Hong Kong Council of Social Service on providing companion robot(s) for trial test will be recruited. Similar to Study A1, their opinions will be analyzed by framework analysis. The analyzed opinions, upon the approval of the investigators, will be listed as the questionnaire items for use in Delphi part. For the Delphi part, 15 - 40 eligible visitors of Gerontech and Innovation Expo will be recruited following the same recruitment mechanism as the Delphi part in Study A1.

In Study B, the same focus groups participants who join the focus groups in Study A2 are simultaneously recruited. Their opinions in the focus groups will be analyzed by framework analysis. The analyzed opinions, upon the approval of the investigators, will be listed as the questionnaire items for use in Delphi part. For real-time Delphi questionnaires, in addition to the above decision makers, snowball sampling will be adopted to recruit the partners of Hong Kong Council of Social Service. These partners agreed to join discussions on gerontechnology, and they are eligible to join the Delphi part no matter they have or do not have experience in the development, evaluation or procurement process about companion robots. In total 8 - 15 panelists will be recruited.

In Study C, the questionnaire used in the Delphi stage will contain a list of strategies in enhancing the organizational research capacity, which are consolidated from literature review and investigators' past experience in doing similar research. In the Delphi Stage, the participants will be recruited in the same way as in Study B. In total 8 - 15 panelists will be recruited.

Delphi Stage For all 4 studies, in the first Delphi round, panellists will rate anonymously the agreement of the opinions on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree), explain their ratings of 3 or less and suggest revisions (including additions of new items). Upon the approval of the investigators, items will be revised according to the suggestions. Items in the first Delphi round that reach moderate or low consensus are revised, and rated in the second Delphi round together with the newly suggested items. On the contrary, for items that reach very high consensus or high consensus, if no revisions are suggested, they will not be rated again. Before the second Delphi round, the panellists will be presented with the consensus level analyses, their own ratings in the first round, and proposed revisions based on the comment on the analysis. Items with more than 75% approval of the revised items will be included in the second round Delphi. In this round, more panellists will be recruited to rate anonymously the agreement of the revised items.

For Study A1 and A2, the second Delphi round is expected to take place two weeks after the first Delphi round. For Study B and C that adopt real-time Delphi in a workshop, the investigators will first present the materials consolidated in the Preparation Stage, and panellists will immediately indicate the agreement of the materials in the first Delphi round. This is followed by an open discussion of first round Delphi analysis results among the panellists for around an hour. For diverse opinions on the same issue, different scenarios favouring the opinions will be explored, and the opinions together with their favouring scenarios will be listed in the second Delphi round. When the opinions are consolidated, second Delphi round will then take place. The panellists will rate their agreement on the suggested revisions.

• Study Type: Observational
• Enrollment (Actual): 87

Contacts and Locations

Study Locations

• Hong Kong
  • Hong Kong, Hong Kong
    • The University of Hong Kong

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: Yes

• Sampling Method: Non-Probability Sample

Study Population

Stakeholders of gerontechnology in Hong Kong

Description

The inclusion criteria for product user (i.e. end user and demand side - frontline).

End user

• Is a service user or family member of demand side, and
• Remember what to do with the product during interview

Demand side - frontline

• Is a staff in a social service unit in Hong Kong (HK), and
• Has used companion robot(s) with a service user or family member for at least a month in past 5 years, or is interested in using companion robot(s) in the future

The inclusion criteria for decision maker (i.e. demand side - manager, academia, funder, and supply side):

Demand side - manager

• Is a managerial personnel of a social service unit

Academia

• Is an investigator or nursing educator in gerontology Funder
• Is a staff directly involved in sponsoring the procurement of companion robot(s) in a Hong Kong funding body Supply side
• Has developed or retailed companion robot(s)

Exclusion criteria for all:

• Unable to communicate in Chinese

Study Plan

How is the study designed?

Design Details

• Observational Models: Cohort
• Time Perspectives: Prospective

Number of groups / cohorts

5

Cohorts and Interventions

Group / Cohort	Intervention / Treatment
End user; A Delphi study will be conducted with two rounds to reach consensus	Other: End user; Participants will provide their opinions in two rounds to reach consensus
Demand side management; A Delphi study will be conducted with two rounds to reach consensus	Other: Demand side- management; Participants will provide their opinions in two rounds to reach consensus
Supply side; A Delphi study will be conducted with two rounds to reach consensus	Other: Supply side; Participants will provide their opinions in two rounds to reach consensus
Funder; A Delphi study will be conducted with two rounds to reach consensus	Other: Funder; Participants will provide their opinions in two rounds to reach consensus
Academia; A Delphi study will be conducted with two rounds to reach consensus	Other: Academia; Participants will provide their opinions in two rounds to reach consensus

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Degree of consensus on indicators of a "good" companion robot	The level of agreement on indicators of a "good" companion robot. , the opinions will be categorized into four levels of consensus according to the central tendency, level of dispersion, and the rated level of importance (Shi et al., 2022). The consensus levels are defined as: very high consensus (median = 5, inter-quartile range (IQR) = 0, rating score of 4/5 ≥ 80%); high consensus (median = 5, IQR = 1, rating score of 4/5 ≥ 80%); moderate consensus (median = 4-5, IQR = 1, rating score 4/5 ≥ 75%), and low consensus (median ≤ 4, IQR ≤ 2, rating score 4/5 < 75%).	3 months
Degree of consensus on feasible scientific methods and implementation strategies to generate evidence-based results of companion robots	The level of agreement on feasible scientific methods and implementation strategies to generate evidence-based results of companion robots. , the opinions will be categorized into four levels of consensus according to the central tendency, level of dispersion, and the rated level of importance (Shi et al., 2022). The consensus levels are defined as: very high consensus (median = 5, inter-quartile range (IQR) = 0, rating score of 4/5 ≥ 80%); high consensus (median = 5, IQR = 1, rating score of 4/5 ≥ 80%); moderate consensus (median = 4-5, IQR = 1, rating score 4/5 ≥ 75%), and low consensus (median ≤ 4, IQR ≤ 2, rating score 4/5 < 75%).	3 months
Degree of consensus on strategies to build organizational research capacity facilitating the evaluation of gerontechnology products	The level of agreement on strategies to build organizational research capacity facilitating the evaluation of gerontechnology products. , the opinions will be categorized into four levels of consensus according to the central tendency, level of dispersion, and the rated level of importance (Shi et al., 2022). The consensus levels are defined as: very high consensus (median = 5, inter-quartile range (IQR) = 0, rating score of 4/5 ≥ 80%); high consensus (median = 5, IQR = 1, rating score of 4/5 ≥ 80%); moderate consensus (median = 4-5, IQR = 1, rating score 4/5 ≥ 75%), and low consensus (median ≤ 4, IQR ≤ 2, rating score 4/5 < 75%).	3 months

Collaborators and Investigators

• Sponsor: The University of Hong Kong
• Collaborators: Gerontechnology Platform, The Social Innovation and Entrepreneurship Development Fund
• Investigators: Principal Investigator: Yee Tak Cheung, PhD, The University of Hong Kong

Study record dates

Study Major Dates

• Study Start (Actual): October 21, 2022
• Primary Completion (Actual): March 31, 2023
• Study Completion (Actual): August 31, 2023

Study Registration Dates

• First Submitted: October 19, 2022
• First Submitted That Met QC Criteria: October 21, 2022
• First Posted (Actual): October 26, 2022

Study Record Updates

• Last Update Posted (Actual): November 18, 2023
• Last Update Submitted That Met QC Criteria: November 17, 2023
• Last Verified: November 1, 2023

More Information

Terms related to this study

• Keywords: Delphi; Gerontechnology; Outcome indicators; Study methods; Research capacity
• Other Study ID Numbers: Gerontechnology Delphi

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO
• IPD Plan Description: Individual participant data (IPD) will not be available

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No