Transparent, Reproducible, and Ethical Evidence (TREE) Review Framework Questionnaire
The objectives of the Transparent, Reproducible, and Ethical Evidence (TREE) Review Framework are to: (i) establish ethical standards for evaluations, (ii) better integrate best practices for transparency, reproducibility, and ethics into evaluation workflow; and (iii) establish timely, independent review of risks facing the evaluation team to produce credible, meaningful, ethical evidence and the efforts to mitigate such risks (See Appendix 1 for examples).
The TREE Review Framework is based on 10 standard ethical standards for evaluations:
- Use transparency as a tool
- Maximize social value and meaningful use
- Balance power and align incentives
- Preserve standard of care
- Value and prioritize community engagement
- Use fair methods
- Ensure fair treatment of participants
- Ensure appropriate informed consent and protection of confidentiality
- Ensure favorable risk-benefit ratio
- Ensure favorable cost-benefit ratio
The TREE Review Framework Questionnaire is designed to pose specific questions regarding evaluation teams’ actions and decisions related to achieving the 10 ethical standards. Teams may use the TREE Review Framework Questionnaire for two purposes:
- Documentation only: Team members may use this Questionnaire as a guide for actions and decisions at the funding, design, data collection, and dissemination stages of the evaluation life cycle.
- Documentation and submission for independent assessment and feedback: Team members may use this Questionnaire to document their actions and decisions at the funding, design, data collection, and dissemination stages of the evaluation life cycle and submit to the TREE Review Committee for assessment and feedback.
To fill in the Questionnaire, evaluation team members are asked to provide responses through two mechanisms:
-
Specific questions: For specific questions, the team members are asked to provide the response based on the drop-down options provided. If Not Applicable, the team members should select the Unknown or Not Applicable options. If the team’s response is not available in the drop-down menu, the team can provide Notes in the Evaluation Team’s Notes section.
a. Certain answers to specific questions trigger a request for the team members to provide more detail, a justification, and/or a risk mitigation strategy. These questions have specific instructions in RED FONT.
- Probing/open ended questions: The evaluation team members can provide more detail in the Evaluation Team’s Notes section.
If specific questions or sections are not relevant given the stage of the evaluation life cycle, the evaluation team members can note Not Applicable or Too Early. Evaluation teams should anticipate submitting supporting documentation: Informed Consent/Assent(s), Questionnaire(s), Protocol(s), Data Management Plan(s).
Appendix 1 Potential Risks, Harms, and Risk Mitigation Efforts
Risk | Example(s) of Harm/Advarse events | Risk Migration Efforts |
---|---|---|
NOTE: The risk can extend to PERCEPTION. When others PERCEIVE a violation of ethical principles, there remains a high risk to researchers, evaluators, and others for harms described. |
Example: A study in Nairobi examining effects of various mechanisms to increase payments for water supply, including cutting off water access, resulted in various discussions around the ethics of the study (see discussion on ethical considerations here). |
|
NOTE: The risk can extend to PERCEPTION. When others PERCEIVE a violation of ethical principles, there remains a high risk to researchers and others for harms described. |
|
|
|
|
|
|
|
|
|
|
|
|
Example: UNHCR shared detailed database of the Rohingya refugee population with Myanmar’s government. |
|
|
|
|
|
Example: On April 20, 2010, Arizona State University (ASU) agreed to pay $700,000 to 41 members of the Havasupai Indian tribe to settle legal claims that university researchers improperly used tribe members' blood samples in genetic research |
|
|
|
|
|
Example: Data obtained from an implementing partner used to inform influential study later identified as fraudulent data. |
|
11. Loss of confidentiality – with or without additional linkage to private, sensitive data – is a risk driven by several potential actions, including:
|
Example: UNHCR shared detailed database of the Rohingya refugee population with Myanmar’s government; Biometric data on Afghan security forces potentially available to the Taliban. |
|
Appendix 2: Taxonomy of Vulnerability
TABLE 4.1: Taxonomy of Vulnerability for Study Participants | |||
Vulnerability | Definition | Potential Causes | Ethical Research Practice |
---|---|---|---|
Cognitive Vulnerability | The research subject does not have the capacity to deliberate and decide whether to participate in the study | Immaturity (through age, other cause), dementia, certain types of mental illness, disability; educational deficits and unfamiliarity with the language; situational mental distress/crisis | Mitigated through proper Informed Consent plain-language, advance directives (where incapacity is anticipated), supplementary educational measures to ensure comprehension, and the proper use of surrogates and advocates |
Juridic Vulnerability | The research subject is liable to the authority of others who may have an independent interest in the research subject’s participation | Prisons and the military, where wardens and officers have legal authority over prisoners and enlistees; Children under the authority of their parents, Students subordinated to Professors, Institutionalized persons subject to the authority of custodians, women legally subject to their husbands; | Mitigated through proper Informed Consent devise a consent procedure that will insulate the research subject from the hierarchical system to which he or she is subject. This is particularly challenging if the researcher/project team is a part of the hierarchical system (so program beneficiaries who are surveyed by their benefactors). |
Deferential Vulnerability | The research subject exhibits patterns of deferential behavior that may mask an underlying unwillingness to participate | May be driven by social and political pressures to follow/defer to others despite own desire to not follow/defer (often present with juridic vulnerability) | Mitigated through Sample Recruitment/ Screening and Informed Consent Inclusion Criteria/Sample Selection may require input of local informants or consultants to devise a process that eliminates as much as possible the social pressures a research subject feels. Informed consent mitigation same as above. |
Allocational Vulnerability | The research subject is lacking in important social goods that will be provided because of participating in the research | When participation in the research can provide research subject a social good - money, housing, medical care, childcare, burial benefits, opportunities to benefit the community, freedom – that they otherwise do not have access to | Mitigated through Sample Recruitment/ Screening and Compensation The Inclusion Criteria/Sample Selection may require input of local informants to determine whether or not the offering of research participation may introduce undue influence; Project Teams must also carefully consider Compensation packages to limit their under or over-value and may need to consider not just their research sample, but also neighboring communities /individuals/households that are excluded and may feel resentment for the exclusion. |
Infrastructural Vulnerability | The political, organizational, economic, and social context of the research setting does not possess the integrity and resources needed to manage the study | Research subjects have access to research requirements (phone, transport); Project teams have access to research requirements (skills for specific biomarker tests, psychological tests, etc.; electricity, transport, safety) | Mitigated through Study Design: The study design/protocol should be carefully reviewed for local context and cultural sensitivities. |
Medical Vulnerability | The research subject has been selected, in part, because he or she has a serious health-related condition for which there are no satisfactory remedies | When (i) illness is severe and (ii) no safe, effective, and otherwise satisfactory treatments are available, patients can be primarily driven to participate based on false hope for benefits | Mitigated through Study Design and Informed Consent Given the interests and aspirations of both parties (and the poor bargaining position of the research subject) work toward fair division of the benefits and burdens of cooperation and design the study to maximize the likelihood of subject benefit based on medical intervention found to be safe and effective; communicate benefits and their probabilities for success through Informed Consent. |
Appendix 3 Adverse Events Reporting – Actual Risks and Potential and Actual Harms from Intervention and Study
Risk Description | Harm Description (Potential and Actual) |
Date(s) occurr | Who was notified and when? | What was the response(s) and when? |
---|---|---|---|---|