The Guide to Economic Evaluation and Research (GEAR) online resource is based on the study entitled “Identifying Methodological Issues in Low- and Middle-Income Countries (LMICs).” This study aims to determine the research questions that arise from their results and can be explored to contribute to better quality economic evaluations evidence for health resource allocation. To achieve this, in January 2015, a team of HITAP’s international unit conducted a comprehensive review of literature identifying issues for conducting economic evaluations. Then, the issues identified from the review were used to inform a survey sent to international scholars working all over the world who have conducted economic evaluation in LMICs to prioritize the issues as they have observed in their work. Results from this survey were analysed and outlined as prioritized gaps which were discussed in a face-to-face consultation meeting with policy-makers and HTA practitioners working in LMIC settings in January 2016. This project has two main outputs which will showcase the results and additional research in different ways: the study and the GEAR. The results after discussions were inputted into an electronic database to inform the global community. The database will be updated biennially to facilitate the accuracy and continued relevance of the methodological issues addressed on the database for LMICs.

Potential methodological issues that might be hindrances to the conduct of economic evaluation in low- and middle-income settings were identified through the review of literature in PubMed and University of York’s Centre for Research and Dissemination (CRD) databases.

Key publications discussing the barriers, flaws or difficulties in the conduct of economic evaluations were retrieved from PubMed and supplemented by reference search. The methodological issues in both high-income and low- and middle-income counties (LMICs) were included since although there were differences in the stage of the research between these settings, the issues in the high-income settings also tended to exist in the LMICs. Context-specific issues were cited as barriers to acceptance of economic evaluations in decision making and were therefore included in the study.

The CRD database was employed owing to its critical appraisal of economic evaluation studies. From the CRD, primary economic evaluations of which a commentary is available were identified through the search of LMICs name based on World Bank classifications as of 2015. The search returned 568 hits of which 180 studies were done in LMICs. Of these, commentaries of 100 studies were randomly selected for review of the criticism on its methodological problems and weaknesses. The issues identified from the review were summarised and categorised based on types of methodological issues: on interventions/comparators (e.g. did not use standard comparator), study population (e.g. no mention of randomisation), effectiveness/benefits (no sensitivity and specificity), costs (e.g. discounting not used even when appropriate), and analysis and results (e.g. no discussion on limitations and/or transferability).

The results of the literature review informed the content of a questionnaire that was disseminated globally to researchers and research assistants that have conducted or currently working on economic evaluations in low- and middle-income countries (LMICs). These were divided into two lists – one on methodological challenges and another on context-specific challenges. For both methodological and context-specific components, the respondents were asked to consider the list of issues provided, add an issue that they thought was relevant but not yet on the list, and rank the top 3 methodological issues and the top context-specific issue that they considered most important in their contexts. They were also asked to propose possible solutions and to choose whether the methodological or the context-specific issues were more important.

The study population comprised researchers who had completed at least one economic evaluation project as the primary investigator or as a part of a team in low- and middle-income settings. The low- and middle-income settings are defined per World Data Bank classifications as of 2015. Both individuals and networks were approached to be respondents. Invitation to participate in the survey was sent to the secretariat of health economic and outcome research networks as well as various health technology assessment networks in different regions for the secretariats to distribute the survey to their members. Individuals were also identified through snowball approach (existing respondents recruit future respondents from their network). In total, there were 927 respondents. However, since the mailing lists of each network are classified and respondents were mostly contacted through the network, there is a possibility of duplication which cannot be eliminated and the exact number of recruited respondents may be less than the figure shown.

The top priority methodological and context-specific issues were analysed separately. Since there is more than one methodological issue, a Likert type of system was applied to rank the methodological issues. For each response, the methodological issue that was ranked first would be assigned the score of 3, 2 for the second rank, and 1 for the third rank. The score for each issue in the components was then ranked in a league table to derive the list of top priorities. Where there was more than one issue that received the same score, their frequencies in ranking amongst the higher levels would be considered. The results of the ranked issues were then analysed in terms of WHO regions, and by respondent affiliation and experience. The solutions that fell within the same concerns or areas or same direction were grouped together.

For more information on the study methodology, results, and discussion, please download the following full file of the original report:

Download PDF file of Identifying Priority Methodological Issues in Economic Evaluation in Low- and Middle-Income Countries: Finding the Holy Grail

The prioritized list of methodological issues and context-specific issues are found in the GEAR section of the website. This list informed the creation of the mind maps and the potential solutions outlined. Guideline recommendations were taken from the survey results. Further, the list was used to inform the research questions that would be relevant to pursue, and these will also be ranked on the website.

The content in the GEAR will be continuously updated through the interactive discussions in the Ask an Expert page and other online content as well as through the biennial survey that will be based on the original survey.

The study “Identifying Methodological Issues in Low- and Middle-income Countries (LMICs)” yielded methodological questions to pursue in the future and improve the quality of economic evaluations in LMICs. Through improved tools for economic evaluation, the quality and usability of this kind of evidence will be boosted and the decision-making informed by these evidences will also be better. This will call for sustainable production of evidence and even higher quality and usability evidences. However, the research team foresaw that filling the gaps through research will require substantial amount of time; therefore, providing academics, researchers and economic evaluation practitioners ‘immediate solutions’ which could be applied to tackle methodological difficulties they are currently facing is as important and needed. It aims to be a part of the cycle in Theory of Change of Methodological Research Development outlined below (Diagram 2).

The ultimate goals of this project are to develop a growing, up-to-date database of issues, solutions and possible research questions to fill the gaps, to provide a unique and trustworthy platform for HTA practitioners to overcome methodological challenges that will respond to queries in a timely manner with up-to-date information, to offer neutral and best available advice to practitioners without advocating any particular method and to function as a global public good, and to which the global community can collectively contribute (i.e. a Wiki approach). With more use of this database, the underlying hope is that capacities in economic evaluation, both to generate and to use the evidences, will be improved.

There is a wide range of health economic evaluations guidelines available whereas the details of methodological specifications and perspective vary according to the purpose of these guidelines. Several guidelines were selected and compared in terms of the issues mainly related to the identified methodological difficulties from the survey. Comparing similarities and differences between these guidelines allows exploring the acceptability and applicability of existing guidelines in LMICs. Guidelines included on the GEAR database were selected to capture different perspectives from different guidelines (e.g. guidelines aiming to use either/both in LMICs or/and in HICS). More details in the rationale for included guidelines are found here.

Experts are those with considerable contribution and influence in the field of health economics in specific areas, geographic regions, and/or experience. They have published, taught, and researched extensively on one or more subjects of interest and can reliable provide information on such subjects, e.g. cost-effectiveness thresholds, costing, meta-analysis, etc. Accordingly, each expert will be assigned “tags” that denote to their particular expertise and questions from GEAR users that apply to these tags may fall under their remit. The GEAR database experts are required to respond within 2 weeks of the question’s receipt. Experts are recruited on a voluntary basis through networks and recruitment campaigns, though they undergo an internal verification and selection process through the iDSI before they are included on the panel. Non-response or late responses (receipt after exceeding 2 weeks) to questions may result to experts’ removal from the panel.

The GEAR addresses the short-term and long-term problems researchers face when conducting research, and this is visually represented through the mind map. The right-side provides researchers with immediate solutions for their current challenges based on available information. The left-side shows researchers which questions arise from the methodological challenges presented and the hypotheses they are based on.