Biomedical Engineering - Clinical Technology Assessment

Collection of questions and answers for the oral exam

Other

EXAM PREP Class CTA Type Revision Materials Date Riguardare What is a health technology? The application of organized knowledge and skills in the form of devices, medicines, vaccines, procedures and systems developed to solve a health problem and improve quality of life. What is a medical device? An article, instrument, apparatus or machine that is used in the prevention, diagnosis or treatment of illness or disease or for detecting, measuring, correcting the structure or function of the body for some health purpose. Its purpose is not achieved by pharmacological, immunological or metabolic means. What is a medical equipment? Medical devices requiring calibration, maintenance, repair, user training and decommissioning. It’s used for the specific purposes of diagnosis and treatment of disease or rehabilitation following disease or injury. It excludes implantable, disposable or single-use medical devices. What is HTA? Systematic evaluation of properties, effects, and impacts of a health-care technology. It is conducted by interdisciplinary groups using explicit analytical frameworks drawing from a variety of methods. It may address the direct and indirect consequences of technologies. Its main purpose is to inform technology- related policy-making in health care, by evaluating the safety, efficacy and effectiveness of the technology weighed against its impact on the society. What are the orientations of HTA? Project-oriented: local placement or use of a technology in a particular institution. For example hospital wants to buy a PET scanner;EXAM PREP 1 @11/01/2022 Technology-oriented: characteristics and impacts of a particular technology; Problem-oriented: solutions or strategies for managing a particular diseases, condition or other problem for which alternative or complementary technologies might be used. What is its general framework? Identify assessment topics Specify the problems or questions Retrieve available relevant evidence Generate or collect new evidence Synthesize evidence Formulate findings Disseminate findings Monitor impact What is assessed in HTA? Technical properties Safety Efficacy and effectiveness Economical implications Other impact on social, legal, political aspects Efficacy vs. Effectiveness Efficacy measures how a treatment performs compared to another treatment (usually the standard care but it’s not always the case) in a tightly-controlled setting of a clinical trial. In this case, the population studied is homogeneous since it doesn’t include patients with complex illnesses. The procedures are standardized to reduce any bias to the minimum and the practitioners performing the treatments are experts, whereas all these factors are inevitably variable in real life. Therefore effectiveness measures how is the treatment performing in real life situations, where the setting is very different from that of a clinical trial. What types of outcomes you can have and the effect measures used accordingly Continuous data: mean difference, standardized mean differenceEXAM PREP 2 Dichotomous data: risk ratio, odds ratio, absolute and relative risk reduction, numbers needed to treat Ordinal data Counts and rates Time-to-event: used in survival analysis (censored data etc.) Give examples for RR, OR and what they stand for Risk ratio is the ratio between the incidence in the treatment group and the incidence in the comparator group. It shows a possible association between two factors in a protective way or as a risk factor whether RR < 1 or RR > 1. The risk ratio doesn’t imply a causality since you can have bias in the results i.e. a confounding factor. For example the RR of having lung cancer if you have smoker’s cough vs. non cough is > 1, but it doesn’t imply direct causality between the two factors. In between you have a confounder which is smoking. Relative risk is commonly used to present the results of randomized controlled trials. This can be problematic, if the relative risk is presented without the absolute measures, such as absolute risk, or risk difference, since it doesn’t take into account the total number of patients considered. Odds ratio is the ratio between the probability of occurrence of an event in the treatment group divided by the probability of occurrence in the non-treatment group. It is standard in the medical literature to calculate the odds ratio and then use the rare-disease assumption (which is usually reasonable) to claim that the relative risk is approximately equal to it. Relative risk is easier to understand than the odds ratio, but one reason to use odds ratio is that usually, data on the entire population is not available and random sampling must be used. In the example above, if it were very costly to interview villagers and find out if they were exposed to the radiation, then the prevalence of radiation exposure would not be known, and neither would the values of total number of patients in control and treatment groups. RR vs. OR in estimating risk OR is often mistaken as a RR but it tends to overestimate the risk for high baseline prevalence. If the rare disease assumption holds, then the OR can be approximated to the RR and some considerations can be made. If the prevalence is higher then OR becomes much higher than the RR and overestimates the risk. It’s difficult to obtain the prevalence values to perform a risk ratio calculation.EXAM PREP 3 What is a biomarker? It’s a clinical point of interest in the follow-up time of a trial or any type of observational study, that is measurable and is used as an indicator of a normal biological process, a disease state or effect of a treatment. Why do we need surrogate endpoints? Because we can’t measure a clinical endpoint of interest during a progression, therefore we rely on measuring a biomarker that we perceive as clinically similar to the endpoint, typically death or morbidity. As all the measurements, it carries some bias due to the fact that no biomarker can identically carry the same meaning as death. You could choose the one value that is scientifically proven to be highly associated to the death of a patient, but it will never be 100% equivalent. For example hypertension is a surrogate endpoint for stroke. What are QALYs? How are they used? What are their disadvantages? QALY is a measurement of life expectancy that is clinically adjusted to reflect a patient’s years in full health, which are lower than the life expectancy in the case of a person with illness or disability. It is useful to represent the life years in best possible shape but they are not perfect. One disadvantage is that you must choose how a disability is going to influence the life o a person and this value (utility) is not going to be the same for every person living with it, since the disability and the loss of health comes from various causes such as psychological, physical, emotional states that vary between the patients. Cutoff level It’s a biomarker. They can vary with different population subgroups. The choice of a cutoff point comes with the risk of more false diagnoses (positive or negative depending on the nature of the cutoff). Screening vs. Diagnosis Screening is the term associated with detection of potential disease indicators so the underlying idea is the absence of symptoms of a disease, while diagnosis is the term for the detection of the disease or not, therefore in the presence of symptoms related to a particular disease, you perform a test directed at evaluating its presence. List the primary data methods, what are they? Clinical trials, observational studies, EXAM PREP 4 How do you select the questions to be answered in a clinical trial? How are they divided? Primary questions that want to test the hypothesis that the intervention has a particular outcome which will be different from the outcome in a control group. Secondary questions regarding benefit, other clinical cutoff levels of interest Questions regarding safety and risk Enunciate the primary data studies, their components, what are their objective Give an example of a confounder Design of a clinical trial What is worse between systematic and random error? Systematic error is worse because you don’t know what is the main cause, whereas the random error is probably due to a small population group. How do you reduce an error? Stratified randomization for random errors, for systematic errors you have different solutions according to the different types of bias. What are the types of bias in the design of a trial? Selection bias: systematic differences in the groups Attrition bias: incomplete outcome data because of withdrawal Performance bias: difference in the care provided by personnel Reporting bias: unreported findings because of poor results Detection bias: difference in how outcomes are determined between groups by assessors How can you perform a random sequence generation? Chance, Stratified randomization, Blocked randomization What main components of a study protocol must be followed when setting and running the trial? Set the objectives: primary and secondary questions. From here the outcomes you want to measure to answer those questions, being surrogate endpoint or intermediate ones. Make hypotheses on subgroups for the study setting, if you want more EXAM PREP 5 generalizability in your results, then discuss what type of trial are you conducting, whether it’s superiority, non-inferiority, inferiority etc. Then you have to design the study: inclusion and exclusion criteria, sample size assumptions, then enrollment of participants with written informed consent. Assessment of eligibility, Baseline examination and intervention allocation (randomization methods). Describe the intervetion with scheduling, follow-up period, and measures of compliance. Collect the data, analyse avoiding any type of bias, assess the adverse events, type and frequency, analyse the data..... Discuss internal and external validity, how are they affected during the conduction of the trial Internal validity measures how effective you are with the treatment you want to test within the clinical trial whereas external validity refers to the generalizability of such results. The best method to assess internal validity is to conduct a RCT (at least for a drugs trial), of course the RCT involves the use of standardized population groups, therefore it doesn’t favour generalizability. What are the qualities that favour internal validity and what are the ones favouring external validity? Internal Validity: prospective studies, large population groups, contemporaneous patients groups, blinding, experimental study, controlled setting, allocation concealment External Validity: heterogeneity of patients that sample the population, multi- center, subgroup analyses, minimize attrition bias, study duration consistent with the disease type, What is bias? A partiality that prevents objective consideration of an issue or situation Discuss the different sources of bias, the name, what are they caused by, how to improve on them Selection bias: random sequence generation Attrition bias: incomplete outcome data Performance bias: blinding of participants and personnel Reporting bias: selective reporting Detection bias: blinding of outcome assessorsEXAM PREP 6 How can you assess the risk of bias in a study, discuss the series of actions to be taken How do missing data impact on the validity of the results? Give an example It changes the effect estimates on the primary endpoints considered, since you don’t know how to deal with lost data. For example if you’re considering the absolute risk of a patient of getting a disease if exposed to smog, the missing data will not be considered in the calculation, resulting in a smaller sample size, in less precision and a much wider confidence interval. Therefore the validity of the results is at risk, also because the missing data might be due to an internal bias in allocation of participants and methods of performing the trial. What are secondary data (integrative) methods? Integrative methods (or secondary or synthesis methods) involve combining data or information from existing sources, including from primary data studies. These can range from quantitative, structured approaches such as meta-analyses or systematic literature reviews to informal, unstructured literature reviews. Having considered the merits of individual studies, an assessment group must then integrate, synthesize, or consolidate the available relevant findings . Even where definitive primary studies exist, findings from them may be combined or considered in a broader social and economic contexts in order to help inform policies. How is systematic literature review conducted? You first need to specify the purpose of the systematic review, and specify the evidence questions using the PICOTS structure. Then you specify a review protocol that is explicit and unbiased, by stating inclusion and exclusion criteria for the selection of studies, bibliographic databases used, search terms for each database, methods of review (number of independent parallel reviewers of each study), intention to conduct meta-analysis, publication of the results. Then you perform the literature search, review the results according to the protocol that you’ve set and by searching for potential publication bias; then you systematically extract data from each included study, you assess the quality of the individual studies retrieved/reviewed, perform the meta-analysis and assess the quality of cumulative body of evidence. Assign a grade to the body of evidence produced, and maybe assess the possible need of collecting new evidence through primary studies to further strengthen the findings.EXAM PREP 7 A sensitivity analysis must be conducted in order to understand the impact of the inclusion/exclusion criteria, the publication bias, and plausible variations in estimates of outcomes and other parameters. Subgroup analyses are good if you want to assess the heterogeneity of the effects and for the external validity of the findings. What is PICOTS? Acronym used to identify the relevant questions in order to formulate keywords for the literature research. Population: disease, demographic, comorbidities... Intervention: technology type, dosage and frequency Comparator: placebo, standard care, active control... Outcomes: morbidity or mortality or QALY or adverse events Timing: duration of follow-up Setting: primary, home care, inpatient etc. Why do I need a meta-analysis? AMSTAR checklist What is a meta-analysis and what is its purpose Meta-analysis is a tool of statistical methods for combining data or results of multiple studies to obtain a quantitative estimate of the overall effect of a particular technology on a defined outcome. Its purpose is to: Organise the knowledge in a systematic approach Increase statistical power of the primary end points Increase external validity of the results Resolve uncertainty between the different papers when they disagree by pooling the data and as such make a stronger assumption Assess the variability between the studies (heterogeneity) Identify new primary data collections that might help strengthen the findings Basic steps in meta-analysis A summary statistic is calculated for each study, statistic varying according to the data you’re dealing with (OR, RR, mean difference etc.). Then a pooled effect EXAM PREP 8 estimate is calculated as the weighted average of the single estimates, where the given weight associated to each study is inversely related to its variance. Therefore low variance (large number of population groups) leads to higher weight, as it should be. For studies that do not calculate the exact outcomes, you need to perform a random-effect meta-analysis What is a cumulative meta-analysis It's a form of meta-analysis that is reconducted each time new trial data is available. In this way the pooled effect estimates and the relative uncertainty can be compared with the previous values obtained. Why is inverse variance method used in calculating the weighted average? Because it gives more weight to the studies that have a large population group i.e. lower standard errors, therefore it reduces the uncertainty over the pooled effect estimate. How do I measure heterogeneity? By making the Chi-square test Why can I have heterogeneity in a meta-analysis? What types of heterogeneity can I have? Any kind of variability among studies in a systematic review may be termed heterogeneity. It can be helpful to distinguish between different types of heterogeneity. Variability in the participants, interventions and outcomes studied may be described as  clinical diversity  (sometimes called clinical heterogeneity), and variability in study design and risk of bias may be described as  methodological diversity  (sometimes called methodological heterogeneity). Variability in the intervention effects being evaluated in the different studies is known as  statistical heterogeneity , and is a consequence of clinical or methodological diversity, or both, among the studies. Statistical heterogeneity manifests itself in the observed intervention effects being more different from each other than one would expect due to random error (chance) alone. In a systematic review, the design features of each of the clinical trials included may affect the overall estimate of treatment effect. These include: random allocation, allocation concealment, blinding of outcome assessment and sample size. Other aspects of study design may result in heterogeneity, such as the method used to assess outcome, variation in the timing of outcome, and length of follow-up. The first step in dissecting the source of heterogeneity in a meta-analysis is therefore EXAM PREP 9 to explore whether variations in these factors might account for the variations between studies with sensitivity analyses How to read funnel plots? What does it graph? Funnel plots represent the deviation of the effect estimates in each paper reviewed from the average treatment effect (0). On the X axis is the log of OR while on the y axis is the inverse of the standard error. It is assumed that studies with larger population group (therefore indicating more precision because standard error is lower therefore they are closer to the sacrario effect) are somewhere in the middle of the table whilewill will disseminate around. if the table is symmetric it means that there is good representation of all the possible effects. If the table is asymmetric it means that some studies with negative results have not been published (reporting bias). What are the limitations of meta-analysis? Meta-analysis can be limited by publication bias of the RCTs or other primary studies used, biased selection of available relevant studies, poor quality of the primary studies, unexplainable heterogeneity (or otherwise insufficient comparability) in the primary studies, and biased interpretation of findings. The shortcomings of meta-analyses, which are shared by—though are generally greater in—unstructured literature reviews and other less rigorous synthesis methods, can be minimized by maintaining a systematic approach. What are precision, directness and consistency of results? Precision: the extent to which a measurement is derived from a set of observations with small variation. Directness: ? Consistency: whether the results off studies in a body of evidence are in agreement. What is survival analysis and what are its objectives? Analysis of patterns of event times and estimates of time-to-event for individuals. The main objective is the comparison of distributions of survival times in different groups in a trial, but also to check by how much done factors affect the risk of an event of interest. It's important to estimate the time origin from which to start the data collection, since it uniforms the participants. What are the different types of events in a study with follow-ups and an outcome and the types of survival times?EXAM PREP 10 Transition time: length of time between the starting point and the event if interest. Relapse-free survival time: time between response to treatment and recurrence of the disease. Complete remission: response to treatment Survival time: time from complete remission to event of interest Overall survival time: time to death Censored data: you have since subjects that you can't observe anymore because they're either still alive while the study has ended, or they've withdrawn from the trial and you don't know what is their overall survival time with or without the disease. Describe the different survival functions Survival curve: probability of not going through a transition before a given time t. Integral of the probability density function from t up to infinite. 1- cumulative distribution function Hazard rate: chance of going through a transition the next time interval, given that the subject has not done so earlier. How do you set up a Kaplan Meier curve? You start by creating a table with the participants in a trial and take note of the transition time and censored data of all the participants. Then you calculate the survival probability at each time period. To deal with the censored data you have to remove them from the calculation of participants that make up the total number of people. But their contribution is removed in the survival probability immediately next to the time period at which they've become censored, because it implicitly means that they've survived up to that time period and then we lost their track at t+. Every time you calculate a probability at a specific time period, you multiply it by the previous probabilities calculated to obtain the overall survival probability up to that point. How can I find resources to innovate in progress? Which are the streams of money Out of pocket expenses from the private citizens (equity is not guaranteed). Increasing taxes, but this doesn't generally mean that you will get an increase cash flow since when the plateau is reached then the amount of cash received goes declining because people will get upset with the increase in taxes and stop EXAM PREP 11 paying. Since I can't just increase the funding for all new technologies I must do some RATIONING What is rationing and how can it be divided? Rationing is making sacrifices in one side of health care to ensure that you don't spend more resources than you can spare. Implicit rationing: it's not a choice of which patients to treat first according to some benefit calculation but it's the line you have to do before getting a surgery done or a specialist visit. One person may be lucky and get treated first in another region where there is less demand or a rich person can pay out-of-pocket and be treated immediately. This is still a firm of rationing even though you can't see it. Explicit rationing: understand which patient would have the highest expected benefit with that technology and therefore try to reduce the rationing. How can I minimise rationing? Efficiency of systems to reduce waste of ������ or ⏳ More co-payments Allocative efficiency When you allocate the resources for a technology, which are the steps involved in HTA? What factors come into play? Hard factors: evidence based evaluations (primary data from experimental studies), risk and safety tests, real-life use. Soft factors: priority of the disease in the general socioeconomic landscape. Horizon scanning i.e. understanding what is the world of science promoting in the next two years and base the evaluations in the future imminent products. Organizational impact on the local health structures for example you have to see if a pet scanner newly introduced could fit in the workflow of the hospital, is there m qualified personnel you deal with it, is there enough space to fit it in, are the doors wide enough to install it in the first place? Economic evaluation: it's the last part which in turn would inform the policy makers of the direct benefits in a new technology compared to the golden standard, the monetary benefits, the break-even of the investment etc. The economic evaluation is not a decision of any sort because it's aimed at informing the decision makers which will make the final reimbursement decision. What is an economic evaluation?EXAM PREP 12 It's a tool for decision making since they identify what can happen in different settings when resources are allocated in different ways. 60% of trials for drugs are singular they want to measure the efficacy of their drug next to a placebo! It's easy in this way, why are they not comparing with another technology and doing an incremental benefit analysis? Then if I have two trials on two drugs, I have to make a network metanalysis, take both the results and find the statistical solution to try and compare them replicating the trial. With medical devices it's quite impossible to make trials, let alone have comparative trials. The evidence that I find therefore mostly never does a comparison of new and old, you have to build the comparative evidence or to make out of existing studies! Why do I need to apply a discount rate? Because of inflation Discuss the main types of economic analyses Cost utility Cost effectiveness Cost benefit Cost minimisation What is a decision model? “Systematic approach to decision making under conditions of uncertainty, in which the probability of each possible event, along with the consequences of those events, is explicitly stated”. Why is decision modelling appropriate to use? Why can’t I base my decisions on a RCT? Because RCT doesn’t reflect the real world use of a technology in most cases. As already said, it improves the internal validity of a choice but reduces the external validity i.e. the generalizability of the choice made. What are the steps of a decision analysis? Develop a model that depicts the set of important choices and potential outcomes of these choices, then assing estimates of the probabilities of each potential outcomes, along with estimates of the value of each outcome in terms EXAM PREP 13 of QALY. Calculate the expected value of the outcomes associated with a particular choice. Identify the choice associated to the highest expected value which is going to be the most desirable choice in terms of efficacy then you need to conduct a sensitivity analysis to determine if variations in the estimates of probabilites and outcomes would change the most desirable choice. What is uncertainty of my decision? It’s the probability that the decision maker will pay for a therapy that, in that real- world scenario, doesn’t represent the optimal choice. You’re paying for a therapy that is not good value, or in the opposite case, you’re refusing to pay for a therapy that is optimal. What are the sources of uncertainty? Sampling variation: the same source of bias that we already discussed in the risk of bias assessment, causes a poor decision since the data you’re analysing are biased. The population drawn for the study is not representative of the population in real life. Extrapolation: Since it is not feasible to run an experimental study long enough to detect also the long term effects of a treatment on the survival curve of a population, you need to make some hypotheses on the future advantage of a treatment as compared to the control Generalisability: Degree of confidence with which data obtained in one setting can be used to inform on a decision in a different setting. Research studies often tend to recruit atypical patients, complex patients are excluded. It’s often difficult to generalise a result from a very much controlled clinical setting. Model Structure: Models used to represent the population in the economic analysis, is of course a simplification of the real life cases. The question is whether these simplifications exclude important characteristics of the potential care process. Methodological uncertainty: uncertainty derived from the process of extrapolating the data from different sources to parametrise the model, the choice of the utility values, the model form used in the probabilistic analysis etc. Discuss the different types of sensitivity analyses Univariate: change only one variable at a time assuming that there is no causal relationship with the other variables, otherwise those values would also change in response to a variation. It’s used to understand which parameter has the highest influence on the model resultsEXAM PREP 14 Multivariate: change two variables at the same time Probabilistic: modelling the parameters from mathematical functions that best describe that parameter, for example beta distribution for effectiveness or utilities (since it can only go from 0 to 1), lognormal or gamma for costs. Dirichlet distribution as a multivariate extension of the beta distr. for utilities and effectiveness. Threshold Analysis: change one parameter to check when you reach the target of choice in the range of values of that variable. Analysis of the extremes: setting the parameters of choice to the lowest and highest possible values, (CI) How is the scatterplot made? How do you read it? How do you make it on Excel? Discuss the brief steps The scatter plot is made by calculating the ICER between two treatment groups. In particular you define or derive from literature, a WTP threshold that the decision makers are willing to pay for a year of life in full health of a person. You plot it as a line crossing the ICE plane and you place the calculated ICERs in the plane (remember X axis has QALYs while y axis has costs). The points can be and should be more than one for the analyses to be accurate, so you can perform an analysis of extremes, probabilistic sensitivity analysis. When you have placed the points inside the plane, you can infer assumptions from these data. In particular, if the points are in the NE quadrant above the WTP line then the intervention treatment is more effective but also more costly. If it stays under the line, then it’s likely that the treatment should be adopted. The opposite consideration has to be done if the points are in the SW quadrant. What is the main assumption of Markov model? The main assumption of a Markov model is that the transition probabilities between Markov states is only correlated to the immediate previous cycle and not the old ones. A patient who was diseased once and then transitions to healthy state, has the same probability of getting disease again of a patient who has always been healthy. When is Markov Model used? Markov Model is used when the patients can transition through the states multiple times, and more importantly the patients are followed in time for a longer period. The decision tree instead does not specify when events occur.EXAM PREP 15 Costs and utilities can be discounted so that the impact of the future on money is also taken into consideration. CEAC Geographical quantification of the uncertainty around the expected cost effectiveness. The residual percentage left at a desired lambda, from the probability of being cost-effective to 100%, is the probability that the decision maker is going to pay for the second best treatment. How do you calculate the expected cost of making the wrong decision? (1-p)*lambda where p is the probability of being cost-effective at a specific WTP threshold How does value of information work? Value of information is a method for quantifying decision uncertainty. It shows how much you value having more information that can reduce the uncertainty, in a way that helps in making a better reimbursement decision. In this case we can calculate what is called value of perfect information that is the price I'm willing to pay to know in advance what treatment is going to be the best in that particular real world case for that particular patient. It is calculated by taking the highest net benefit from the two treatments and making the difference with the highest expected net benefit. This value is the mean of all the health benefits that are always the highest between the two in each reiteration. I can calculate this value IF AND ONLY IF I always know which is the best treatment every time in advance. This is of course impossible in real life since the reimbursement decision is made prospectively before being able to test this decision in all the real world cases. What considerations can you make from EVPI? Why can it be useful? EVPI is the expected value of perfect information, it comes from the fact that we have to deal with uncertainty of decisions we make, because we don’t know for certain which treatment is going to be the best for each real world scenario. IF we had the possibility of knowing every time what is the best alternative, we would obtain what is known as maximum monetary/health benefit, which is the expected NMB/NHB that comes from knowing at all times what is the best treatment. The EVPI is the difference between this value and the highest expected net benefit. The EVPI therefore is the value, in monetary terms, that I am willing to pay to have perfect information and know in advance which is the best treatment every time. EXAM PREP 16 What limitations does CEAC have? How do you overcome them? CEAC shows which option has the highest probability of being cost-effective, by calculating the amount of times, out of the total number of iterations, that one treatment has higher INMB than the second treatment. However, it doesn’t show which is the optimal treatment at that lambda value. Since we have to take uncertainty into account, the optimal treatment needs to be found making the average of NMB of all the real world scenarios (in this case the iterations that I’ve made) and not take the best one out of each single iteration. What does CEAF depict? It depicts the uncertainty associated with the optimal intervention over a range of lamba values, since it computes the highest expected net benefit for each lambda. How do you take into account the investment to break even over time in the reimbursement decision? What curve helps you in making a better decision? NBPM is a curve that helps in analysing the investment made in terms of cumulative net benefit, since it shows on the X axis the time periods and on the y axis the cumulative NB. In this way you can see the progression of the impact that the technology will have on the individual's health. If it's above zero, it means that you're giving more health than is dispensed. The NBPM also shows the percentiles over the cumulative NB i.e. the uncertainty over different values of probabilities distributions. The reimbursement from authorities can be delayed upon considerations of this graph especially if the break even has not yet been reached.EXAM PREP 17