Biomedical Engineering - E-health Methods and Applications

Completed notes of the course - Prof. F.

Complete course

Terminologies, Semantic interoperabilityTerminologies, Semantic interoperability 1 Concepts Basic uses of terminology Purpose of classifications Ontology ICD international classification of disease SNOMED CT Concepts Descriptions Relationships Benefits of SNOMED CT Pre-coordinated expression Properties of SNOMED CT UMLS UMLS in use Metathesaurus Semantic Network Relationships may not hold at the concept level Granularity varies across the semantic network SPECIALIST Lexicon & Tools Morphology Syntax Orthography Lexical Entry Formatting Normalization tool Word Index Lexical Variant Generation (LVG) Main UMLS use Natural Language Processing Uses of NLP Why NLP is possible? Steps of NLP Named Entity Recognition problems The need for terminologies is to have different health care professionals contribute to data collection and interpretation. Communication is a huge driver for good care of the patient, through all the central points in healthcare. We should assure privacy, data loss and maintenance of meaning per se, so not only the data physically but also its interpretation. Automatic interpration of data from PC programs must be exploited (critical decision support systems) therefore if the data is not standardized, the machine can’t elaborate it. Data should also be available for epidemiological studies and research, how to deal with pandemic nowadays, or for the overall health monitoring of nations. It is not anymore a problem of understanding the meaning in this case, but it’s only about collecting the data in a standardized way. Clinical Ideas are the essence of anything we want to record and are the building blocs of EHR. Terminology is the set of special words or phrases that are used in a particular field. The idea is to provide a model, terminologies built according to specific rules or procedures so that in automatic processing of terminologies, there is no mistake in the retrieval of the terminologies. Concepts Pre-coordinated vocabulary : the complex term eg. endoscopic emergency intervention exists and has a specific code. Assign a unique identification code to single or aggregates concepts Advantages: Everything is explicit so they have a definition and there is no ambiguityTerminologies, Semantic interoperability 2 Contextual Processing problems Discourse processing Metamap Pipeline Tokenization and Parsing (part of speech identification) Generation of variants Candidate Evaluation MetaMap use All the combinations that are meaningless are eliminated. Disadvantages: The specificity is too extensive and excessive and makes it difficult to access the vocabularies Concepts are repeated many times but are synonyms Post-coordinated vocabulary : here I have single simple concepts and I can create complex concepts, for example intervention, priority emergency, endoscopy. Here I’m structuring the data in a way that I don’t need to remember all the codes for each complex concept, but I will adapt the existing concepts to create a new one. Atomic concepts that you aggregate to create concept aggregates, you should always check in this type of vocabulary that you’re not creating something that doesn’t exist or you’re creating terminologies that share the same meaning. Advantages: Smaller than the pre-coordinated ones in terms of memory size Disadvantages: It needs a set of rules to create complex concepts. Basic uses of terminology Abstraction and classification entail examination of the recorded data and then selection of items from a terminology with which to label the data. It’s an abstraction of reality that according to this, we have one way of putting the data all together. One-dimensional hierarchy of data and I’m classifying the data as such. Representation is the process by which as much detail as possible is coded so I’m going towards the complete model of meaning and not just following a one- dimension hierarchy. Purpose of classifications Subdivision of phenomen into classes that are the basis for ordering things. It’s an abstraction. Univocity of conceptsTerminologies, Semantic interoperability 3 Omnicomprehensive, i.e. it’s pre-coordinated so you’re listing also the complex concepts Mutually exclusive Exhaustive with respect to also the general meaning of things Ontology It’s a representation of reality and it is considered a complete description of a domain of interest by their concepts and the relationship among those concepts. There not also the hierarchical relationships (parent-child) but also non-hierarchical ones. They allow semantic interoperability among different information systems, as they are able to connect different terminologies by their concepts. An example would be myocardial infarction and heart attack, they share the same meaning so they are in the same concept, I don’t need to express two of them. SNOMED CT and UMLS are ontologies. Can be pre-coordinated and post-coordinated terminology It doesn’t allow technical interoperability but semantical interoperability. ICD international classification of disease Used in the world to do epidemiological studies, classify diseases, causes of death and to record health problems in a state in order to decide the quantity of magnitude given to a specific area. The purpose of ICD is to identify trends and statistics globally, provide definitions and organize information to facilitate the analysis on a disease globally in an automatic way. Allows the counting of death as well as diseases, injuries etc. Any concept is uniquely identified by a certain identifier (ID) or code. You can’t build and recreate a complex term by searching in the sections, you can just check the codes for any specific meaning. ICD-9 is used in Italy for the discharge letters, ICD-10 are used for the epidemiologic studies for the causes of death, both the classification need to be somehow updated in each case, for example for COVID-19 both ICD-9 and 10 had to be used by physicians and doctors. Terminologies, Semantic interoperability 4 SNOMED CT Most comprehensive clinical health terminology (ontology). It can be automatized inside a PC along with UMLS Three basic elements Concepts Organized in hierarchies, body structure concept; clinical finding, environmental location, event, observale entity, organism, procedure, clinical scale etc. These are the hierarchies i.e. the first characterization of concepts, for example bone is a body structure that is a SNOMED CT concept or femor is a bone that is a body structure etc. Most of the concepts are clinical findings, then procedure. They are identified by a unique numeric identifier (Concept ID) up to 18 digits that never changes and a FSN Descriptions Terms that describe a concept for which I have at least two descriptions. The FSN which describes fully and in the most appropriate way the concept and then I have at least a synonym There may be also synonyms marked as acceptable because others are not used anymore and are discarded. Descriptions are terms that relate the terms of a SNOMED CT concept to the concept itself. Every preferred language has a FSN for a unique concept ID and preferred synonyms. Relationships Connections between concepts, the list of relationships for a particular concepts makes up for the logical connections around that concept All defined relationships of a concept must be always true for the concept they define! In this ontology I’m not creating a one-dimensional hierarchy because with causative agents and finding sites relationships I can move on different levels of definitions that match with the concept.Terminologies, Semantic interoperability 5 In this ontology there are rules for which relationship types can be matched to a concept, for example the relationship type causative agent can’t be a relationship of a body structure but only of a disorder while for the range (destination concept) the causative agent can be related to organism and not morphlogical abnormality. This in an example of the set of rules defined for attribute relationships. When creating post-coordinated concepts, these are the rules that you should avoid when creating complex concepts, so that you don’t create terminologies that in fact do not exist. Benefits of SNOMED CT Clinical assessment and treatment: in the concept of the pair of the single patient SNOMED CT allows for storing of information and sharing the information to automated systems and also facilitates the enrollment of participants to a trial based on their info and searching for specific criteria, as well as facilitation of the follow-up of these patients. Population monitoring: identify emerging technologies, or monitoring the population for example during the pandemic. Learning healthcare systems for the extended use of data, data used at population and research level to produce new knowledge. Pre-coordinated expression SNOMED CT expression containing a single concept identifier that represents an idea I can also derive this pre-coordinated expression with post-coordinated expression like for example a fracture of bone as a pre-coordinated expression and arrive at the fracture of femur by making relevant attributes. So if I start from pre-coordinated expression of fracture of bone and the finding site attribute of femur I will reach bone structure of femur that is a combined expression therefore a post-coordinated expression which is going to be the same as bone fracture of femur as a pre- coordinated expression. Properties of SNOMED CT 1. All defined relationships of a concept must be always true for the concepts they define. Then it’s called defined concept.Terminologies, Semantic interoperability 6 2. Specification of general concepts make a new distinct object from the parent usually. But when the new concept doesn’t have any new defining relationships that the parent concept doesn’t have then it’s called primitive concept because it’s exactly the same as its parent . UMLS Tries to solve a big problem in semantics and terminologies since there are too many terminologies (we only introduced ICD and SNOMED CT); there are different terminologies for radiology, nuerses, procedure, medical devices, drugs etc. Each field wants to create its own terminology but then you’re back at the start of the problem of communication and interoperability. There are so many of them because there is need of information encoding, and the reasons for not creating a unique terminology is usually geographical and political. Moreover, some terminologies do not necessarily link to each other, because they might come in a variety of formats and data models or some ar not always freely available. The national library of medicine decided to create UMLS, which is not a terminology itself but it’s a set of files and softwares that bring togheter many health and biomedical vocabularies. It facilitates the development of softwares that use UMLS because they contain all the main terminologies for different health-related terminology. There is source transparency because it is always clear where the concept comes from. A methathesaurus is the combination of all the terminologies together, it has a broader scope and reach an extended coverage. It can have finer granularity, it associates anyway unique identifiers to link all the concepts together, you have to connect all the unique identifiers of all the terminologies into one. Synonyms terms are clustered into concepts. UMLS in use I can link together medical terms, drug names, and billing codes across different computer systems which are relevant and couldn’t be done with the single terminologies. For example coordination of codes between your doctor, the pharmacy and the insurance company. Main benefits Provides access to terminology dataTerminologies, Semantic interoperability 7 NLM has negotiated the right to redistribute terminologies via UMLS for use in research. UMLS consists of Methasaurus, the semantic network that is the model of all these terms (135 categories from 22 of SNOMED CT and 54 different relationships between the categories), then we have lexical tools. Each terminology provides grammar tools and softwares to analyze normal text and retrieve important information. They can be used separately or together. Metathesaurus MetamorphoSys helps you refine the metathesaurus according to your needs without having all the concepts that you don’t need. Concepts contain synonymous terms, preferred term is chosen (default can be changed), unique identifier (CUI) is assigned. CUI has no intrinsic meaning, so it’s not hierarchical ID. Then you have the synonyms in other terminologies that you have chosen in the metamorphoSys, that are all referred to the same CUI since they relate to the same meaning. There is always the source (source transparency), then you have the code of the original terminology and the AUI. CUI: concept is a meaning, it’s always the unique identifier associated to multiple terms and synonyms. LUI: Links strings that are lexical variants, for example headache and headaches. Lexical variant generator is a tool that checks if a term is a lexical variant of another term. SUI: Associated to same string present in different terminologies, for example headache in ICD and SNOMED CT are using the same SUI AUI: it’s the code for each concept in a given source, for example both headache of ICD and SNOMED will have a different AUI. Represents ambiguityTerminologies, Semantic interoperability 8 The same term/string can have different meanings, therefore UMLS shows you that it’s connected to different concepts. Same thing can happen within a SUI or LUI. UMLS provides a common data model We are creating a broad terminology according to a variety of data models, but in the end they are all connected to a single CUI. The UMLS represents all the terminology data according to a standard data model. Semantic Network 135 semantic types (in general are included in two big hierarchies called entity and events ). A category can only be one of these two hierarchies. For instance disease is an entity 54 Semantic relationships divided in IS A or associative relationships . The semantic type is what characterises the entity or the event that you have and it’s the first thing associating to a concept. Relationships may not hold at the concept level Same thing with source and destination from SNOMED CT means that here in parent- child language, some associations are not inherited by the child. For example, the relation “evaluation of” holds between the semantic types “sign” and “organism attribute” but it does not between any particular pair that can be a child of the original pair. overweight and fever are child of sign and body weight and body temperature are child of organism attribute. BUT overweigh is not an evaluation of temperature or fever is not an evaluation of body weight. It is not true that the relationship holds when I’m moving down the level of granularity. Inheritence is not generally there for all the children. Granularity varies across the semantic network The node manufactured object has only 2 child nodes: Medical devices and research devices. But it obvious that there are other manufactured objects other than those two categories, but since it is not present in UMLS, any other object will be inserted in the general node Manufactured Object. If someone wants, the granularity can be increased by creating a new child node.Terminologies, Semantic interoperability 9 SPECIALIST Lexicon & Tools Includes about more than 330K words, 550K variants of the words. It includes general english words used with specialistic words used in a scientific research paper. A lot of lexical entries to provide their properties: syntactic, morphological, and orthographic information needed to process text and terms. Morphology Inflection includes terms with plurals, verb types, superlatives, comparatives. Derivation creates a new term by transforming the word and not only modifying it Compound creates new words by combining existing words UMLS already knows if a AUI is a lexical variant of a concept but it can also give you some morphology tools, and grammar tools to retrieve meaning. Syntax The way words are used to form phrases (how words are put together) Orthography Spelling and quality of writing of words Lexical Entry Base form of the term (singular nouns, infinitive verbs) The part of the speech that can be represented (noun, adjective, verb) A unique identifier (if the word has two different roles in a sentence it will have two different lexical entries with EUI, entry unique identifier) Rules for spelling variants Formatting Each unit is a frame structure made of slots and fillers delimited by braces {}. Spelling variants are all the inflections and derivations of a word. For example the word anesthetic can be also spelled as anaesthetic or exercise and exercize. Terminologies, Semantic interoperability 10 Lexical entries are not divided into meanings. An entry represents a spelling-category pairing regardless of semantics. For example the word Act has two different meanings but both are nouns, in this case from a lexical POV it is the same lexical entry and I won’t have two of them. The semantical relationships are not in the lexicon but only in the semantics network and metathesaurus. Normalization tool A transformation of a sentence sharing it in normalized words and string indexes. it means that this software is removing all the punctuation, uppercases, breaking the string into words (from a sentence) and sorts in alphabetical order. It is used to separate in terms before finding similar terms because you can find spelling errors and whatever, through synonyms you can understand the real meaning Word Index Breaks strings into words and produces the Metathesaurus word indexes Lexical Variant Generation (LVG) Starts from the output of the other two trying to perform all the transformations of input words by looking at all the other variants of the words. Main UMLS use It is mainly used to identify meaning from text. Because medical data in the real world is often unstructured so I want to create knowledge from this data and I need UMLS. To improve search and retrieval by annotating records in a research database To find co-occurrences of concepts in text To annotate clinical text on the fly To identify a patient population UMLS tools are used for Natural language processing Terminologies, Semantic interoperability 11 Natural Language Processing Natural Language Processing (NLP) is a range of computational techniques for analyzing and representing naturally occurring texts at one or more levels of linguistic analysis for the purpose of achieving human-like language processing for a range of tasks or applications i.e. NLP is needed to transform relevant information hidden in unstructured text into structured data that it can be used by computer processes. If a sentence is shared, are used to refer to a previous sentence, this complicates a lot the meaning of the reference of the next sentence. There are tricks that can be used but sometimes it is not enough to understand the sentence. Therefore machine learning is very much used in NLP to undestand a referenced long text. If you are able to identify inside a text, concepts that are connected to SUI and LUIs, than it’s much easier to extrapolate knowledge information on that text. The goal is not only to retriece words in unstructured text, but to find concepts and understand relations among them, simulating the human capability to understand the language. Uses of NLP Pharmacovigilance to identify adverse events Semantic information retrieval (from an ontology extract concepts and develop tools using python to extract semantic information with NLP if you don’t want to use UMLS) Question answering, automatic summarization Why NLP is possible? Because natural language is formulaic: it contains discrete symbols (words) and rules (grammar) specifying how different elements can be combined to define a phrase with a specific meaning. Linguistic knowledge consists of different levels: morphology → combination of different morphemes (root, prefixes, suffixes) to produce lexemes (words) syntax → structure of sentences (grammar) and lexiconTerminologies, Semantic interoperability 12 semantics → meaning of words and phrases pragmatics → intent of the speaker / context how can you teach a machine how to undestand the context. An example is for example if I say a patient drinks too few means water but if he drinks heavily it means alcohol. discourse → paragraphs Steps of NLP 1. Named Entity Recognition → associate to an entity a part of speech and some properties, 2. Contextual Attribute Assignment → try to process the meaning of this entity in the context 3. Discourse processing → Co-reference with parts of the discourse Named Entity Recognition problems Linguistic variation: different words with the same meaning Polysemy : same word with different meaning Contextual Processing problems Negation : All the concepts and sentences are negative in most of the cases, for example the kidney is not enlarged is a problem for NLP Uncertainty : when you are checking for hypothesis, you may confute the hypothesis and creates problems in the meaning Temporality : when you are referring to past history in the family or of the patient Discourse processing Report structure Co-reference → how to understand co-reference in the text? Each referential expression has a unique referent, so you must identify it. Metamap Terminologies, Semantic interoperability 13 Tool available in UMLS that unifies the 3 knowledge structures of metathesaurus, semantic networks and lexicon tools. Pipeline Non-machine learning approach, you start from an input text then making tokenization, identifying part of speech taggin. Metamap does the concept mapping using metathesaurus. There is of course the need of a vocabulary and terminology to retrieve each concept and produce a map. Tokenization and Parsing (part of speech identification) Lexical/Syntactic analysis. Terminologies, Semantic interoperability 14 Identification of the nouns which have the main role since they can be the head of a sentence, the simplified sentence that comes out of the input text is the series of blocked words divided by type. Generation of variants It starts from all the single entries and creating all possible variants, also from the different variants itself, so I create a lot of possible candidates as concepts that are Terminologies, Semantic interoperability 15 associated to that entry and they will be evaluated to find the most appropriate concept. The selected concepts are the mapped concepts at the output. The number inside the brackets is the distance of that term from the adjective “ocular”, for example ocular has distance 0 from ocular since it is a synonym. The table on the right shows the distance based on the variant type. Candidate Evaluation For each word you have to evaluate how much it matches with the original input word, computing: centrality: does the candidate contain the head of the input word? (ocular complications, complications is the head of the sentence so it has centrality value 1 and eye which is the variant that I’m evaluating (ocular) has value 0) variation: how much the variants in the candidate string differ from the corresponding words in the phrase? (the table on the right in the image above, so spelling variation is 0, inflectional is 1, synonym is 2 etc.)Terminologies, Semantic interoperability 16 Every candidate in the table will receive a centrality and variation calculation, by using the formula in red. phrase span: number of words of the phrase involved in the match candidate span: number of words of the candidate involved in the match coverage: phrase span / # words in the phrase + 2 * candidate span / #words in the candidate, how much of the text is matched? connective component: maximum # of contiguous words participating in the match cohesiveness: (phrase connective components)2 / (# words in the phrase )^2 + 2 * (candidate connective components )^2 / (# words in the candidate )^2, similar to coverage but emphasized the importance of the connected components The final weight is up to 1000. MetaMap use Interactive use: searching on a search bar Batch use: provide a .txt file JAVA Web API: provide users with the ability to programmatically submit jobs to the scheduler batch and interactive facilities of using the web-based interface. Terminologies, Semantic interoperability 17 Terminologies, Semantic interoperability 18 Methodologies to design, develop and evaluate e-health technologies A technology needs to be distinguished if it wants to be introduced in health data in everyday life but without clinical validation. When a technology wants to provide valid e- health data that should be seen and evaluated by physicians and doctors, then it’s a completely different setting. New MDR in addition to saying that software is a medical device, the MDR says that in order to get a certification for a medical device you should provide a clinical validity. In the past the companies could put into the market devices without proving the clinical validity; right now it is inside the regulation to get the CE mark. We’re interested in producing technology with low-cost tools that are still reliable, so there are experimental and observational studies aimed at obtaining valid clinical Methodologies to design, develop and evaluate e-health technologies 1 Rationale for e-health design 5M eHealth technologies Challenges of E-Health technology development Waterfall Models (90s) for development of technologies Scrum framework Holistic Approach for eHealth design and evaluation Design Principles Cehres roadmap Contextual Inquiry Steps for inquiry Value specification Design Requirements Development Approach Gathering requirements using persona Design prototypes Operationalization & Summative Evaluation Operationalization Summative Evaluation Formative Evaluation evidence. Clinical evaluation does not mean to evaluate your device with the best possible study available (RCT) because it would take too much time for each device. You need to demonstrate that you’re improving the efficacy of the intended use that you want to exploit. Rationale for e-health design The essence of healthcare is to provide the best care possible that meets the needs of patients and their caregivers. The management of comorbidities is central nowadays + moving the healthcare to homecare. We are moving to a cooperative healthcare, in which all stakeholders are involved in the process. Technology is needed to support this change We can categorize eHealth technologies according to: their place in healthcare → the place in which the technology should be placed (hospital, home etc.) their main aim → connected to the intended use their influence on the healthcare system → are you creating a new infrastructure for hospitals or are you creating just a mobile app? Around this influence you should delevop your idea and requirements 5M eHealth technologies measurement monitoring mentoring motivation management of data Challenges of E-Health technology development Adherence The ability to follow a treatment/protocol. One of the main issues of the lack of impact of a technology is the lack of adherence. Something was wrong in the study Methodologies to design, develop and evaluate e-health technologies 2 design, your device wasn’t accepted and that was the problem, not the lack of impact itself. Acceptance It is more of a preliminary tool and doesn’t include the obligation of use of the device. Adoption It’s after the use, but it is not true that if you are using you are adopting. If I’m letting a user test for a month independently a device at home, you can’t test the adoption because it should be a long-term requirement, where the device meets the requirements of the device (if it aims at treating a disease), the adoption is reached when it is regurarly used for a long time. Most difficult of the three to be reached. Problems for adoption: Lack of communications and coordination between the stakeholders Marginal level of user engagement → after covid, seniors are more familiar with technology but it does not mean engagement, because they must see the benefit in using this technology. If you prove the benefit, then the engagement comes and probably also the daily routine use. Solutions not integrated in the daily routine Waterfall Models (90s) for development of technologies Rigid nature of a sequential model, but if the requirements are finished at the beginning, what you are going to have is probably not the same as you intended it, because maybe you could not know in advance all the requirements. Nowadays it is a spiral model which repeats and can go back to the requirements step and repeat the process of design, validation etc. Scrum framework Used nowadays in companies, there is a planning and they come with a sprint backlog, which is a weekly plan with a sprint review at the end of the week to check if the plans were met and then restart with a new planning. This approach can be very flexible because you always know where you are, very useful in health technologies because you can change the requirements as you go on with the project.Methodologies to design, develop and evaluate e-health technologies 3 Holistic Approach for eHealth design and evaluation Holistic means comprehensive, there are 3 actors: technology people context They are completely interrelated and should always act together. Design Principles should be participatory, stakeholders involvement is crucial should be iterative (involving continuous evaluation cycles to refine development), like Scrum framework should include pervasive design techniques (Caiani lecture on pervasive applications) to facilitate adoption to induce behavioural change Only if you know the intended use you can design a pervasive technique to facilitate adoption. should define evaluation methodologies from the beginning (RCT are not well suited for it) Cehres roadmap A guideline recognised for a good development of a eHealth Technology and its evaluation and implementation. Contextual Inquiry It aims at understanding context and people, stakeholders, users and environment. Current situation Rules, regulation and already existing clinical validations of a similar deviceMethodologies to design, develop and evaluate e-health technologies 4 Conditions for the use of technology (from which you extract the intended use) Steps for inquiry Stakeholder identification → people who are affected by an eHealth technology Stakeholder analysis → interdipendencies, responsibilites, and stakes of involved stakeholders Desk research → non-systematic collection of material to contextualize the technology, research on the products that are used, what is mainly done to tackle a specific disease Scoping review → status of the literature in a certain broad field of study Focused group → meetings for collection of qualitative data involving a small number of stakeholders; for example interviewing teachers you can understand that when children start writing they use pencils and not pens, so if you want to develop a pen to tackle dysgraphia then your technology is not useful, unless you create a deletable ink pen. Interviews: structured, or unstructured set of questions Observations: observation of an event while it happens by a researcher Diary Study: keeping of a diary of certain events on a regular basis to understand their routine. Log data analysis: use of data that logs certain behaviours on a system or in daily life to analyze patterns Value specification They are based on value and are the things that stakeholders would like to see reflected and improves by the eHealth technologyMethodologies to design, develop and evaluate e-health technologies 5 Customer profile tries to understand the stakeholder, Value proposition shows how the product creates value for the customer, so after understanding the customer you identify the values you could provide and validating this proposition by understanding if it’s supported by all stakeholders. If not, then you go back to identifying the value propositions again and validating it. Maximal priority for stakeholders to propose a value proposition needs to go to the target customers that in the end are going to be the consumers. Design Building prototypes that are responding to requirements, they are used to start hands- on and test the features. 1. Build funtional and technical requirements (what should the device do? And to do that function what technicalities do you need?)Methodologies to design, develop and evaluate e-health technologies 6 2. Low-fidelity and high-fidelity prototypes; a. Hands-on → Ideas are made tangible through prototyping b. Iterative → Repeat each phase backward and forward and arrive at each decision after rounds of learning and discovery. 3. Usability tests of prototypes with end-users, experts in real life 4. Test phases with feedback to refine requirements Requirements Development Approach Requirements are the building block of the whole design and development process. what the technology should do what data should be managed interfaces and what to display expected user experience Requirements definition is something that should be left at engineers with the direct involvment of stakeholders, since they will define the requirements in the best way. the requirements development is a process that is orthogonal to the roadmap. At the end of the design phase the requirements should be defined. The list of requirements: Content requirements Usability and user experience Functional requirements Service requirements: the best way to organise the service Organizational requirements: integration of the technology into the organizational structure Technical requirements: synonym of technical specifications The approach to develop requirements follow 5 phases that are orthogonal to the cehres roadmap. 1. Preparation 2. End user and stakeholder identificationMethodologies to design, develop and evaluate e-health technologies 7 3. Requirements elicitation 4. Requirements analysis 5. Communicating requirements In the table of requirements, you need to set the possible conflicts with other requirements, and also the priority you want to set, if high you will deal with it first. Gathering requirements using persona Persona is an ideal group of people and stakeholders with the same characteristics, so you can segment the market and develop specific requirements for different personas. For example I want to develop a fitness tracker for seniors and I can pinpoint 2 personas: 1. Persona 1: Robert (low technology user) a. The setting and use of the tracker must be quite simple and automatic, should track steps and heart rate, large font and high contrast. 2. Persona 2: Bridget (high technology user but has early sign of cognitive impairment and eye problems a. provide clear notification also through emails and reminders, interaction approach should reduce memory load because of the cognitive impairments. Here the requirements will be completely different between the two groups and you may interview them to get an idea of the priority requirements that they would like to have. Once settinge the personas, you can observe their routine to understand if there are additional requirements that you can add to your device. Design prototypes You can develop different stages of prototyping like early concept, early planning, design refinement, mid design, late design or post release.Methodologies to design, develop and evaluate e-health technologies 8 For example prototyping of fitness tracker. Wireframe is the idea of functionalities of the interaction with the dashboard and the user experience. Operationalization & Summative Evaluation Operationalization Create a plan to make sure that the technology is introduced and used in practice in the long term. Determine concrete activities to implement the eHealth technology in practice Launch the final version of the technology in practice mobilizing also additional support (user support) → FIRST REAL FIELD TESTING on a very small population set. Usually for seniors the operationalization is a test in a protected environment for a short time. The clinical validation of the technology at this point must be at least proven in the intended use, before doing the first field testing. Summative EvaluationMethodologies to design, develop and evaluate e-health technologies 9 Determining the impact of the technology. Analyzing the uptake of an eHealth technology in terms of adoption or use of the technology by predetermined users and implementation, and use within the intended context. Summative evaluation is a long-time evaluation of the technology with high number of testers at their home. Only this evaluation allows for the calculation of adherence and adoption and the impact of your technology. Formative Evaluation It is the loop that links the steps of the process, it is used to identify potential implementation problems and tackle them in early phases of development. The evaluation can be done at low levels of prototyping, during the development of the device and not at the end of the validation, with experts of the field to understand whether you’re following the guidelines and you are moving in the right direction. It is done with low-fidelity prototype so at the early stages. The testing with target users instead aims at providing different evaluations, not on the guidelines but on the usability, performance etc.Methodologies to design, develop and evaluate e-health technologies 10 Methods to Evaluate Novel eHealth solutions There is some need for new technological solutions like domestic use (not only in clinic), a slim device not bulky, easy to use and no or reduced supervision of an expert. This doesn’t mean that there should not be a revision at all, but the physical supervision is missing, but you can replace it algorithmically if you are able to. You should understand how far you can go with the technology already present in the market, but also the technological solutions that you may come up with. So the idea is that you need to verify and validate every new technological solution; this process involves two groups: Engineers for the funtional and technical requirements Clinicians for the clinical validation to understand if the device is the right tool for that need. Validation Process of Novel technologiesMethods to Evaluate Novel eHealth solutions 1 Validation Process of Novel technologies Measurement error Validity & Reliability Types of reliability Types of validity Relative and Absolute reliability and validity Assess relative reliability and validity Correlation coefficients ICC Assess absolute reliability and validity Bland-Altman Plot Sample size Comparison with clinical scales Usability Satisfaction Acceptance Definition of requirements Measurement error assessment Systematic error Random error Validity and reliability definition statistical methods sample size calculation comparison with clinical scales, when the gold standard care doesn’t measure with the same scale of your device, so you need a validated clinical scale to measure the efficacy usability, satisfaction, acceptability evaluation Measurement error In each context you should analyze what is the error that impacts the most, what are the aspects mostly related to that error and evaluate only these aspects For example I want to evaluate a clinical scale of questionnaire that can be run by operators. The operators usually are affected by random error, because they might make small differences in the assessment. In this case you will evaluate the intervariability of assessment of the clinical scale. High systematic error → low validity Validity is how are we able to estimate the real value? A synonym of accuracy; if we have not reached a valid value it might be because of systematic error. Random error is totally unpredictable on the predicted values. This error produces a lot of variability i.e. low precision. It affects the reliability of the method Validity & Reliability Reliability: How COHERENTLY a method measures what it is intended to measure. It’s a concept closer to precision, where the method measures a similar value every time without a lot of variability.Methods to Evaluate Novel eHealth solutions 2 Validity: how ACCURATELY a method measures what it is intended to measure. The stability of the situation over time is needed to assess the validity of the results, otherwirse I won’t be able to know if it’s reliable Types of reliability Test-retest → tests the stability of the result over time “within observer”. Do you get the same result by repeating the same measurement? The stability of the measure that you’re targeting can affect a lot this test-retest stability. For example if i want to measure the blood pressure, since it is not a stable measure, I will not get a good test-retest stability, so here it’s important also the context at which I’m working with. Inter-raters → consistency of a measure between different observers (raters): do you get the same result when several people conduct the same experiment? When there is an interpretation or qualitative assessment by experts, you need this type of reliability in the outcome rating. It’s called “between observer”. High inter-rater reliability indicates a good level of standardization, while low inter rater reliability indicates a high level of measurement error due to the variability of subjective evaluation.Methods to Evaluate Novel eHealth solutions 3 Types of validity Construct → adherence of a measure to the previous knowledge of the concept in question, applicable to questionnaires Content → how much the measure covers all aspects of the concept Criterion → how much the result of a measure corresponds to other valid measures of the same concept acquired at the same time ( concurrent validity ) or at a later time (predictive validity). Of course concurrent one is usually impossible to obtain Relative and Absolute reliability and validity Relative reliability: It refers to the ability of the same tool to assign the same position (ranking) to individuals in a sample with replicated measures. Not interested in the absolute measurement but in their rankings. Absolute reliability: Agreement between replicated measures of the same phenomenon using the same method and the same unit of measurement Relative validity: how much two measurement methods, regardless of the unit of measurement, are able to assign the same ranking between individuals. Absolute validity: Agreement between two methods with the same unit of measurement. Example: IOT tool to measure grip force, measured by pressure of inflation of a ball. So in fact you’re measuring pressure and not force, so you’re measuring with another tool in another scale of measurement, so you have two measurements assessing the same concepts. First I use a grasping ball and make the transformation between pressure and force, then with a grip force tool that is actually measuring the force, if I have that the two measurements are valid relatively each other, so I can organise an experiment to assess the relative validity but not the absolute validity because the scales and units of measurements are not the same. Assess relative reliability and validity I should find a statistic for analysis on relative terms and not absolute terms, this statistic is the correlation coefficients, which are a measure of the relation between two Methods to Evaluate Novel eHealth solutions 4 variables and not the agreement. Correlation coefficients ICC varies between 0 and 1, it’s not in absolute terms of course and the equivalent of this coefficient for variables not normally distributed is the Lin’s concordance correlation and when the variables are completely dichotomic the tau of kendall is used. ICC I need to know the stability of my results in test-retest and inter-rater reliability. The first thing you’re asking is what type of reliability you have done because then the model type changes for the ICC. In the end you will have a value that ranges between 0 and 1. For relative validity, the only thing I can do is a Pearson correlation that describes the relationship and strength of the linear relationship between two variables. It’s not a strong tool but it’s the only one you can use. Assess absolute reliability and validity Bland-Altman PlotMethods to Evaluate Novel eHealth solutions 5 If I’m interested in absolute measurements, the BA plot is a way to visualize the goodness of a new tool compared to gold standard in absolute term. Y-axis: the difference bw the two measurements for all your samples (so two values for reliability and validity assessments) X-axis: mean of the two measurements To test validity I’m choosing healthy subjects and ask them to use the two instruments, validity assessment is the difference in value against their mean In green there is the mean error, if there is a systematic error between the two instruments, then there is a systematic error different from 0. If all the points you get are inside the blue lines, then it’s usually a good measurement. This plot is also useful to check other attributes, for example particular behaviors like proportional error where it increases with increasing in mean values. Heteroskedasticity when differences between the methods are not constant but depend on the value.Methods to Evaluate Novel eHealth solutions 6 Sample size It’s important to calculate the sample size (A priori!) to ensure that the number of subjects involved allows us to answer the question of interest with sufficient sensitivity and power. A small study will have reduced sensitivity while an oversized study is costly and you might cause a number of subjects to take an ineffective treatment. You need to hypothesise what is the minimum ICC you accept to get the sample size Comparison with clinical scales How can you assess validity when a gold standard does not exist? Comparison of obtained results vs clinical scale (linked to the measure of interest). The problem of these scales is that they’re not granular because you can’t assess the validity if you don’t have accurate values in the scale. Usability Questionnaire to assess SUS: system usability scale Lots of studies aim at standardizing these qualitative data structures by creating a questionnaire that is fixed. 10 questions, grade 1to5 if he’s in agreement or disagreement with the sentence, very Methods to Evaluate Novel eHealth solutions 7 easy to fill in. It is standardized in a way that the odd questions are positive questions and the even ones are negative, so a good result depends on the question. To calculate the score, for the odds you are doing the score - 1, for the even ones is 5 - scale position, then summing all up and multiplying for 2.5 to obtain a value between 0 and 100. 100 is highest usability If SUS > 68 then the instrument has good usability Satisfaction Here it depends on what types of questions you want to ask depending on the use and device you’re evaluating. Acceptance TAM: technology acceptance model Perceived easiness of use Perceived utility Attitude Intention, if I would like to use it or not in the future Same setting as the SUS, but there is no threshold here. Final example of a usability and clinical validity study on the slides Definition of requirementsMethods to Evaluate Novel eHealth solutions 8 Digital Medicine for ageing in place Ageing population will impact on public costs and place new demands on health system. Rationale 1 Fact: for all the seniors the best would be to leave them in their preferred environment (home). Issue: elders are recovered too early in nursing homes because caregivers are too busy to assist them. Role of DM: Change the model of care offering home-based solutions for the use. Rationale 2 Fact: Frailty, clinically recognizable state of increased vulnerability, progresses slowly but constantly over time. During the preclinical phase, there is no clinical monitoring but Digital Medicine for ageing in place 1 Rationale 1 Rationale 2 Rationale 3 Nutcare Development and testing of each object Smart ink pen Plan of validation tests 1. Validity of the force sensors 2. Validity of the inertial sensors to estimate the tilt angle 3. Validity of the algortihms to segment the signals 4. Reliability of the indicators 5. Evaluation of acceptability Smart Insole Validation of gait monitoring with differences between the single task and dual task Smart Ball for grip force monitoring How to evaluate the ball it happens only when it’s too late i.e. when the cognitive functions have decreased in mild dementia. Issue: Clinical monitoring occurs through sporadic geriatric visits, Role of DM: offer a regular monitoring of daily activities in the preclinical phase when the patient is not that old and is somehow more independent, so to have a prompt intervention in an optimistic way. Rationale 3 Recommendations from the WHO for elderlies, to do 150 minutes per week of moderate physical activity, stay mentally active and be an active member of the community. Prevent physical degeneration and social exclusion. Role of DM: adaptive stimulation through activities done alone of with other people. Nutcare European project to assist the elder in living at home, maximize the elder’s motivation toward an active lifestyle and detect early signs of decline to enable prompt interventions. The proposed solution is to provide continuous and transparent home-based monitoring of age-related decline. Through desk-rooms, interviews and questionnaires, it was found that the patients often forgot to keep the tracker on and didn’t have the willingness to participate a lot. What I want to do is to keep the process of activation of the device as minimum as possible, without intervention of the participant so he doesn’t lose interest. The three ideas were: Smart Ink pen: writing is a complex gesture because it requires combination of cognitive and physical skills. In this way it could have been useful to project a possible early symptom of deficiency. Starting from a pen found online, it was possible to dismantle it and place sensors inside to study its functioning. Smart insoles: alterations of gait in elderlies are important precursors of falls Selecting commercial insoles, because of problems of robustness they exploited an Digital Medicine for ageing in place 2 already existing sole with integration of really automatic application for the detection of the gait. Smart ball: decreased grip force is one of the first indicators of frailty. The ball is inflated with air and the idea was coming from the clinician, so it’s an interview with experts. The use of this ball of course is less than the other two objects so it was more difficult to implement it in an effective way Development and testing of each object Smart ink pen Rationale: Handwriting is an important and high-value taskentailing a blend of cognitive, perceptual, and fine motor skills; its monitoring has been proved useful to assess age- related decline or neurological dysfunction. This object is very much used in Parkinson’s disease and other neurological diseases, so it was already proven to be effective in this case of cognitive impairment. Functional Requirements Quantitative writing and tremor evaluation Extracted features automatically uploaded to the cloud database Not digital but with the use of paper to enhance the easiness of use for an elder Ecologically valid The only thing you’re asking to an elder is to use that pen when writing anything! Plan of validation tests Verify the reliability/validity of the sensors validity of the force sensors (is it linear when measuring the force?) validity of the inertial sensors (IMUs to calculate the angle of the pen when writing) Verify the reliability/validity of the indicatorsDigital Medicine for ageing in place 3 algorithms to segment the signals reliability of the indicators This indicators were found studying the literature on quantitative assessment of handwriting and the signal should be segmented in strokes Evaluation of acceptability in real life scenario 1. Validity of the force sensors Aim : validate the force exerted on the pen tip during handwriting Methods : compare the force measured by the pen to the force measured by an ad hoc sensor, gold-standard (load cell) in both static and dynamic (during handwriting) conditions 1. Static Force vs known weights Check whether the static force was measured linearly. The protocol was to measure the pen tip force with different known weights placed over the pen to simulate a static force The results were highly linear (R^2 = 0.99) 2. Dynamic Force vs force sensor For dynamic condition the protocol required a load cell as the gold standard and both the measurements had to be taken simultaneously. The results also in this case were very similar therefore the correlation was very high again. 2. Validity of the inertial sensors to estimate the tilt angle Comparison of the tilt angle obtained by the intertial sensors of the pen and the same angle obtained using an optoelectronic system gold-standard during handwriting trials. The result was not as highly correlated as the force but still a good result. 3. Validity of the algortihms to segment the signals They wanted to validate the algorithm developed to segment the signal with no force and during the pen touching the sheet (on-sheet). The method was to compare the segmentation that was computed starting from the measured force by the external load cell with a sort of manual segmentation because you can easiliy see the points where the force is absent.Digital Medicine for ageing in place 4 One subject was asked to do a free trial of writing with the pen on a paper placed over the external load cell. The trial was repeated with 10 different pens. Data acquired by 3 pens were used to optimize the segmentation algorithm. The 2 segmentation algorithm were compared using a Bland Altman plot on the other 7 trials. On the graph on the left it was on-sheet time calculated manually from the load cell signal (so very accurate) and the on-sheet time of the pen calculated automatically by the sensors. In the Bland Altman plot you can find the comparison of the strokes computed with the 7 pens; there is a little overestimation of 0.11 in mean error but the data is between the two ranges so the algorithm was behaving correctly 4. Reliability of the indicators Verify the reliability of the indicators estimated on the target population (healthy seniors) during free handwriting i.e. handwrite a couple of times but not always the same thing. The protocol to verify the reliability is a test re-test of a free handwriting repeated twice, 1 hour between the 2 trials and the data analyzed is the ICC on “in-air time”, “on-sheet time”, “writing pressure”. To get a statistical power of 80%, error type I 5%, ICC of 0,5 the sample size was of 22 subjects needed. In the end the test re-test was done was on 43 subjects because it was able to get data from young adults (18-59) and 23 seniors. For each indicator, the ICC is always more than 0,75 so there is excellent reliability in all features. For continuous variables they computed the ICC while for count rates since it is not possible to calculate the ICC, Kendall’s tau was calculated. MDC is the medium detectable change, it is important because if you got from 2 measurements in time that the change in the angle field is less of 2 degrees, it could be an error of the measurement itself and not the device error. You want to discriminate bw a random error and a degeneration error or an improvement. With the two groups of age, the first trial of the two was used to see if the indicators were also useful to discriminate the age between the people taking the test. After creating three classes, a statistical analysis was done to see if the indicators could detect the change in age via a trend that could have been discriminated. In-air time and In-air/on-sheet ratio during handwriting increase with age, so there is a trend of values Digital Medicine for ageing in place 5 for increasing age. Of course this indicator was already found in other neurological impairments in literature, and therefore it was an interesting indicator. Approximate entropy and %determinism were also changing with ageing (the regularity of the handwriting) in a way that more regular patterns emerge with age. The test were able to give reliable indicators that were also sensitive to age! 5. Evaluation of acceptability Evaluate the usability of the long-term not supervisioned remote use of the pen, the elders had just to recharge the pen each night, and the sensors were properly working then without activation. At the end of the 3 months, 24 seniors has to reply to an ad hoc questionnaire. 4 questions in a 1 to 5 Likert scale. Smart Insole Rationale: Slow walking speed is a widely used criterion in geriatric assessment. Alterations of gait in elderlies are important precursors of falls. Functional Requirement: design an ecological solution that monitors the daily life gait Technical Requirement: use of sensorized insoles integrated in a mobile app. Soles already developed in the market have an app associated to a clinical use and in this case it was not easy to find an insole with open API that could have been exploited to create a personal app. To maximise user’s acceptance the ideas were: Adoption of smart insoles easy to be charged and used Mobile app completely transparent to the user (no interaction required) Dual-task paradigm integration. When you’re doing something along with walking, if you have impairments or start to degenerate, it’s probable that the gait analysis will degenerate because it becomes more difficult to walk well while doing something else. In this case the app could discriminate between walking with the smartphone kept in the pocket or when the smartphone was used for example to talk on the phone. Proposed solution: Mobile App for gait monitoring, using the GPS of the smartphone to understand where the user was. When the user was going out more than 200 meters from the house, the geofences were triggering the automatic gait monitoring inside the Digital Medicine for ageing in place 6 app while in the background without any intervention from the user. The app had the possibility of detecting the dual-task condition and monitor them accordingly. When the senior comes home, the monitoring is stopped and the data is sent to the cloud when the phone is connected to the cloud automatically Validation of gait monitoring with differences between the single task and dual task Protocol: Outdoor walking for 19 seniors in three walking conditions: Single tasking: walking at comfortable speed Ecological dual-task: walking while speaking at the phone Cognitive dual task: walking and doing a cognitive quiz since it’s obviously more difficult for the brain Indicators measured: cadence, double support, stance time, swing time. It was possible to see that: Cadence was slightly decreasing with the increase of cognitive load, Double support time (the phase in which you are with the weight with both the feet) and it increases when you’re unsure of the way to walk, also increases with the cognitive load. Same for the stance time (time you’re on the ground) The swing time was decreasing (opposite of the stance time) SLOW and SAFE walking when cognitive load is added, we don’t need to ask to do the quiz which is a difficult task but also with the ecological load it could be seen a degeneration in gait. Smart Ball for grip force monitoring There is a strong relationship between lower grip strength and age-related decline. The most important thing to decide in technical requirements is the range of measurements to be analysed; from literature it was found the best values between 0-60N by reading a paper where the grip force were measured with the jammar (dynanometer). Also here the challenge is the user’s acceptance To maximize user’s acceptance:Digital Medicine for ageing in place 7 Maximizing ease of use Maximize engagement and reliability. Since I can’t use it as a stress ball because I need to measure the maximum grip force. The solution was to implement an android app (serious game) that through some small tests, measures the grip force of the patient on the smart ball. Squeeze for 3 seconds, the more was the force, the more were the points acquired. Another step of the game was not to squeeze but to rotate the ball and after 15 seconds the test would start again. The endurance test was to try to squeeze the longest you can (above 70% of the maximum grip) to have a score in the end. How to evaluate the ball 1. Pressure validation against an external force actuator 2. Clinica validation against a gold standard - Jamar a. Concurrent validity b. Reliability c. usability Pressure Validation A linear actuator exerts standard forces from 10-200 N on the ball, 2 repetitions for each session to understand the reliability of the measurements vs the force actuator. The test was done with 4 initial pressures of the ball to see how the initial pressure was working with the reliability of the measurements from the ball. Clinical Validation Asking 26 adults to do a grasping maximum force with the Jamar and the ball. Jamar was measuring force and the ball was measuring pressure so the only possible thing to do was a correlation analysis since the measurements were different! Pearson analysis to describe the linear relationship between the two variables. Important System reliability was also assessed by repeating the trial 3 times. ICC = 0.9046 System usability with SUS scoreDigital Medicine for ageing in place 8 Serious Games Understanding the context Identify the needs and translate them to functional requirements Development choices (hw-sw) Evaluation process We will talk about dysgraphia (context). We first need to fully understand what dysgraphia is and it is not cognitive related impairment. There is a link between delay in learning and decreased self-esteem, behavioural problems and early school abandon. There are both problems of underestimation and overestimation in this case, since when they access private line of diagnosis, the economical implications of such diagnosis leads to overestimation. Teacher’s observation → Parents notification → Diagnosis The current limitations of diagnosis are: it is done late, in the third class of primary schoo