3ie Survey | Home

Transparent, Reproducible, and Ethical Evidence (TREE) Review Framework Questionnaire

Instructions

The objectives of the Transparent, Reproducible, and Ethical Evidence (TREE) Review Framework are to: (i) establish ethical standards for evaluations, (ii) better integrate best practices for transparency, reproducibility, and ethics into evaluation workflow; and (iii) establish timely, independent review of risks facing the evaluation team to produce credible, meaningful, ethical evidence and the efforts to mitigate such risks (See Appendix 1 for examples).

The TREE Review Framework is based on 10 standard ethical standards for evaluations:

Use transparency as a tool
Maximize social value and meaningful use
Balance power and align incentives
Preserve standard of care
Value and prioritize community engagement
Use fair methods
Ensure fair treatment of participants
Ensure appropriate informed consent and protection of confidentiality
Ensure favorable risk-benefit ratio
Ensure favorable cost-benefit ratio

The TREE Review Framework Questionnaire is designed to pose specific questions regarding evaluation teams’ actions and decisions related to achieving the 10 ethical standards. Teams may use the TREE Review Framework Questionnaire for two purposes:

Documentation only: Team members may use this Questionnaire as a guide for actions and decisions at the funding, design, data collection, and dissemination stages of the evaluation life cycle.
Documentation and submission for independent assessment and feedback: Team members may use this Questionnaire to document their actions and decisions at the funding, design, data collection, and dissemination stages of the evaluation life cycle and submit to the TREE Review Committee for assessment and feedback.

To fill in the Questionnaire, evaluation team members are asked to provide responses through two mechanisms:

Specific questions: For specific questions, the team members are asked to provide the response based on the drop-down options provided. If Not Applicable, the team members should select the Unknown or Not Applicable options. If the team’s response is not available in the drop-down menu, the team can provide Notes in the Evaluation Team’s Notes section.
a. Certain answers to specific questions trigger a request for the team members to provide more detail, a justification, and/or a risk mitigation strategy. These questions have specific instructions in RED FONT.
Probing/open ended questions: The evaluation team members can provide more detail in the Evaluation Team’s Notes section.

If specific questions or sections are not relevant given the stage of the evaluation life cycle, the evaluation team members can note Not Applicable or Too Early. Evaluation teams should anticipate submitting supporting documentation: Informed Consent/Assent(s), Questionnaire(s), Protocol(s), Data Management Plan(s).

Click here to read the concept note

Create Account

Sample TREE form

Risk	Example(s) of Harm/Advarse events	Risk Migration Efforts
ntervention design and/or implementation violates ethical principles (beneficence, respect for persons and/or justice) NOTE: The risk can extend to PERCEPTION. When others PERCEIVE a violation of ethical principles, there remains a high risk to researchers, evaluators, and others for harms described.	Unethical Intervention design and implementation can lead to physical, economic, emotional, social, or legal harm(s) for intervention participants, research participants, bystanders Studying unethical interventions can pose reputational harm to researchers Studying unethical interventions can lead to reduced funding for evaluation Researchers become too risk averse to study controversial interventions and these interventions are allowed to continue without evidence to inform course corrections which result in harmful policies that affect the physical, economic, emotional, social, or legal status of populations of interest Example: A study in Nairobi examining effects of various mechanisms to increase payments for water supply, including cutting off water access, resulted in various discussions around the ethics of the study (see discussion on ethical considerations here).	Carefully examine the standard of care and how the intervention(s) studied may affect the standard of care in the ‘preserve standard of care’ section Carefully examine the social value of the study and whether there is a favorable risk-benefit ratio Ensure all key research team members are trained in foundational principles of protection of human subjects Ensure review of research protocol and informed consent by appropriate Institutional Review Board(s) Complete timely and continuous ethical review using resources like TREE Review questionnaire Establish mechanisms for participants, bystanders, staff to report harms/adverse events Be accepting of risk – some interventions may be unethical, and how interventions are improved or removed may be research demonstrating the adverse effects. However, this requires additional emphasis on the need to use transparency as a tool for how the researcher navigates ethical concerns and abides by research ethical principles.
2. Research design and/or implementation violates ethical principles (beneficence, respect for persons and/or justice) NOTE: The risk can extend to PERCEPTION. When others PERCEIVE a violation of ethical principles, there remains a high risk to researchers and others for harms described.	Unethical research design and implementation (failure to provide sufficient information for informed consent, poorly designed or executed data collection) can lead to physical, economic, emotional, social, or legal harm(s) for research participants as result of study itself Findings from poor quality or biased research design can inform programs/policies that affect the physical, economic, emotional, social, or legal aspect of population of interest Poor quality or biased research design and implementation can pose reputational harm to researchers Unethical research practices can lead to reduced funding for research Example: A study in Montana on the effect of different communications flyers to increase voter turnout was likely illegal and failed to consider ethics of impacts on community.	Ensure all key research team members are trained in foundational principles of protection of human subjects Ensure review of research protocol and informed consent by appropriate Institutional Review Board(s) Complete timely and continuous ethical review using resources like TREE Review questionnaire Establish mechanisms for participants, bystanders, staff to report harms/adverse events Be prepared to end a study if it is determined the study cannot be conducted in a way that aligns with ethical principles
3. Research studies a solution that does not align with the context/problem	Studying interventions that don’t align with problems can bias the interpretation of results of the study. For example, the study can determine the intervention ‘didn’t work’ when it was not aligned with context/original problems. This can affect future programming of policies or interventions that affect the economic and social status of research participant and bystander populations The potential risks and actual costs of the research may outweigh the benefits, violating ‘beneficence’ principle Research waste results in less funding for research	Ensure there is a strong, data-driven understanding of problem the intervention will address in the ‘maximize social value’ assessment
5. Results of study are not useful or used by decision-makers	The potential risks and actual costs of the research outweigh the benefits, violating ‘beneficence’ principle Research waste results in less funding for research	Ensure motivation of study and evidence consumers are identified early and engaged throughout research life cycle through ‘maximize social value’ assessment Ensure quality of research through ensuring fair methods and fair treatment of participants
5. Research team and/or research funders have incentives to prioritize positive results, downplay null or negative results	Selective reporting and/or specification searching can result in biased findings and biased evidence base Bias can affect future programming of policies or interventions that affect the economic and social status of data provider populations Selective reporting and/or specification searching can pose reputational harm to researchers Research waste results in less funding for research	Define research questions and outcomes through pre-specification in registration and pre-analysis plans Assess power dynamics and incentives across stakeholders Use transparency as a tool to mitigate misaligned incentives and allow for assessment of credibility of findings
6. Research participants experience specific vulnerabilities and/or power dynamics that affect their sense of autonomy to participate or refuse to participate in the research	Research participants feel unable to be truthful during data collection, leading to bias in data Research participants feel coerced or unduly influenced to provide private, sensitive data Provision of private, sensitive data can lead to physical, economic, emotional, social, or legal harm(s) for research participants Example: UNHCR shared detailed database of the Rohingya refugee population with Myanmar’s government.	Assess and acknowledge vulnerabilities and power dynamics in ‘balance power and align incentives’ assessment Limit collection of study data with any other purpose (such as intervention eligibility) Define data purpose and use in informed consent De-identify data before sharing if defined and allowed through informed consent process
7. Data collected but not used or needed	Unnecessary identifiable information collected and stored that is linked to private, sensitive data can be breached or have other unauthorized disclosure that leads to physical, economic, emotional, social, or legal harm(s) Research waste results in less funding for research	Define data needs with the pre-analysis plan or other pre-specification tool to clearly map data needs to research questions and analysis models Clearly define data that is collected for ‘unknown’ purposes and assess risk and need Support these efforts with a data inventory (example - Data Inventory Map - Fillable Template - This Is Amos)
8. Data scope creep and/or misuse that results in data used for something other than purpose for which it was collected	Data that is used for purposes outside the original purpose can lead to physical, economic, emotional, social, or legal harm(s) Under certain regulations, this data misuse can be illegal or result in legal repercussions for the research team Misuse of data can cause reputational harm to researchers Example: On April 20, 2010, Arizona State University (ASU) agreed to pay $700,000 to 41 members of the Havasupai Indian tribe to settle legal claims that university researchers improperly used tribe members' blood samples in genetic research	Define data needs with the pre-analysis plan or other pre-specification tool to clearly map data needs to research questions and analysis models (same as above) Define who has access to what data, when, and for what purpose in the Data Management Plan Ensure alignment between Data Management Plan and information provided to the data provider in the Informed Consent and/or Data Sharing Plan Under US HHS regulations, carefully consider use of ‘broad consent’ during informed consent
9. Incomplete data resulting from the study team not considering all data needs required to fulfill research objectives	Incomplete data can bias the results of the study leading to false positive or negative findings that affect future programming of policies or interventions that affect the economic and social status of data provider populations Inability to fulfill research objectives can pose reputational harm to researchers Research waste results in less funding for research	Define data needs with the pre-analysis plan or other pre-specification tool to clearly map data needs to research questions and analysis models (same as above) Ensure alignment with research sample and intervention sample through fair subject selection assessment
10. Poor Quality Data (primary and/or secondary data sources)	Poor quality data can bias the results of the study leading to false positive or negative findings that affect future programming of policies or interventions that affect the economic and social status of data provider populations and can pose reputational harm to researchers Research waste results in less funding for research Example: Data obtained from an implementing partner used to inform influential study later identified as fraudulent data.	Ensure data quality management and assessment practices are established for primary and secondary data collection and detailed under fair methods assessment
11. Loss of confidentiality – with or without additional linkage to private, sensitive data – is a risk driven by several potential actions, including: Intruders breaching systems and accessing raw data Unintentional leakage or unintentional disclosure of either the raw data or of the information/ knowledge resulting from analysis of the data can occur by (a) of a member of the project team; (b) of known third parties (e.g., government, research partners); who have requested or may have access, or who may be motivated to get access in order to misuse the data and information; or (c) by unknown third parties (e.g. due to hackers or other bad actors Anonymized or de-identified datasets combined with other datasets to re-identify participants	Confidential data obtained by nefarious actors can result in physical, economic, emotional, social, or legal harm(s) to research participants Under certain regulations, loss of confidential data may be illegal or result in legal repercussions for the research team Irresponsible data management can cause reputational harm to researchers Example: UNHCR shared detailed database of the Rohingya refugee population with Myanmar’s government; Biometric data on Afghan security forces potentially available to the Taliban.	Align data collected with data needs – if do not need to collect direct identifiers, do not (see above) Ensure staff members have completed the information security awareness and data protection training (Talk with IT team) Ensure the Data Management Plan describes how to: Limit access of identifiable data to staff based on need Encrypt files and devices storing confidential data Store direct identifiers separate from other data as appropriate Ensure there is a data breach plan that is up to date and known by research team Ensure the data sharing strategy is assessed for re-identification risk and appropriate sharing strategy (public, restricted, or no access)

Vulnerability	Definition	Potential Causes	Ethical Research Practice
TABLE 4.1: Taxonomy of Vulnerability for Study Participants
Cognitive Vulnerability	The research subject does not have the capacity to deliberate and decide whether to participate in the study	Immaturity (through age, other cause), dementia, certain types of mental illness, disability; educational deficits and unfamiliarity with the language; situational mental distress/crisis	Mitigated through proper Informed Consent plain-language, advance directives (where incapacity is anticipated), supplementary educational measures to ensure comprehension, and the proper use of surrogates and advocates

Juridic Vulnerability	The research subject is liable to the authority of others who may have an independent interest in the research subject’s participation	Prisons and the military, where wardens and officers have legal authority over prisoners and enlistees; Children under the authority of their parents, Students subordinated to Professors, Institutionalized persons subject to the authority of custodians, women legally subject to their husbands;	Mitigated through proper Informed Consent devise a consent procedure that will insulate the research subject from the hierarchical system to which he or she is subject. This is particularly challenging if the researcher/project team is a part of the hierarchical system (so program beneficiaries who are surveyed by their benefactors).
Deferential Vulnerability	The research subject exhibits patterns of deferential behavior that may mask an underlying unwillingness to participate	May be driven by social and political pressures to follow/defer to others despite own desire to not follow/defer (often present with juridic vulnerability)	Mitigated through Sample Recruitment/ Screening and Informed Consent Inclusion Criteria/Sample Selection may require input of local informants or consultants to devise a process that eliminates as much as possible the social pressures a research subject feels. Informed consent mitigation same as above.
Allocational Vulnerability	The research subject is lacking in important social goods that will be provided because of participating in the research	When participation in the research can provide research subject a social good - money, housing, medical care, childcare, burial benefits, opportunities to benefit the community, freedom – that they otherwise do not have access to	Mitigated through Sample Recruitment/ Screening and Compensation The Inclusion Criteria/Sample Selection may require input of local informants to determine whether or not the offering of research participation may introduce undue influence; Project Teams must also carefully consider Compensation packages to limit their under or over-value and may need to consider not just their research sample, but also neighboring communities /individuals/households that are excluded and may feel resentment for the exclusion.
Infrastructural Vulnerability	The political, organizational, economic, and social context of the research setting does not possess the integrity and resources needed to manage the study	Research subjects have access to research requirements (phone, transport); Project teams have access to research requirements (skills for specific biomarker tests, psychological tests, etc.; electricity, transport, safety)	Mitigated through Study Design: The study design/protocol should be carefully reviewed for local context and cultural sensitivities.
Medical Vulnerability	The research subject has been selected, in part, because he or she has a serious health-related condition for which there are no satisfactory remedies	When (i) illness is severe and (ii) no safe, effective, and otherwise satisfactory treatments are available, patients can be primarily driven to participate based on false hope for benefits	Mitigated through Study Design and Informed Consent Given the interests and aspirations of both parties (and the poor bargaining position of the research subject) work toward fair division of the benefits and burdens of cooperation and design the study to maximize the likelihood of subject benefit based on medical intervention found to be safe and effective; communicate benefits and their probabilities for success through Informed Consent.

Risk Description	Harm Description (Potential and Actual)	Date(s) occurr	Who was notified and when?	What was the response(s) and when?