Ozwell pDSI Source Attributes

Source attributes are the technical and quality information used to create and understand Predictive Decision Support Interventions (pDSI). The Office of the National Coordinator for Health IT (ONC) defines a pDSI, generically understood as Artificial Intelligence (AI), using a three prong test. A pDSI must (1) support decision making based on models, (2) derive relationships from training data, and (3) result in prediction, classification, recommendation, evaluation or analysis. Ozwell uses third party models to inform and augment the clinical decision-making workflow by performing tasks, such as creating SOAP notes and clinical summaries or recommending clinical documentation actions.

The following information will help you, as an Ozwell end user, understand the scope and purpose of Ozwell, its intended uses and limitations, and make an informed decision to trust the validity and fairness of Ozwell’s output.

Details and Output of the Intervention

1. Name and contact information for the intervention developer:

• Developer: BlueHive Health, LLC.
• Contact: Visit bluehive.com for more information.

2. Funding source of the technical implementation for the intervention(s) development:

• Funded by BlueHive Health, LLC.

3. Description of value that the intervention produces as an output:

• The value output of Ozwell includes:
  • Suggested documentation for SOAP notes (e.g., subjective, objective, assessment, plan).
  • Assistance with generating medical record entries (e.g., suggested diagnoses, treatment plans).
  • General recommendations or guidance based on user-provided healthcare-related queries.

4. Whether the intervention output is a prediction, classification, recommendation, evaluation, analysis, or other type of output:

• Outputs include recommendations, evaluations, and analyses based on user input.

Purpose of the Intervention

5. Intended use of the intervention:

• The intended use of Ozwell is to assist healthcare professionals, administrators, and other users in healthcare-related workflows.
  • Clinical Setting: Ozwell is designed to be used in various healthcare environments, including outpatient clinics, hospitals, and administrative offices, as a supportive tool for documentation, decision-making, and patient care coordination.
  • Patient Population: While not directly interacting with patients, Ozwell supports healthcare providers working with diverse patient populations.
  • Goals and Use Cases: The intervention aims to streamline documentation (e.g., SOAP notes, DAP notes, etc.), provide general healthcare guidance, and enhance efficiency in healthcare workflows. It is not intended to replace clinical judgment or provide real-time clinical decision-making.

6. Intended patient population(s) for the intervention’s use:

• Intended Populations: Ozwell is designed to support healthcare professionals working with a broad range of patient populations, including but not limited to:
  • Patients with chronic conditions (e.g., diabetes, hypertension).
  • Patients requiring routine care or preventive services.
  • Patients in outpatient or primary care settings.
• Populations for Which the Intervention Should Not Be Used:
  • Patients in emergency or critical care settings where real-time clinical decision-making is required.
  • Patients requiring highly specialized care (e.g., advanced pain management, complex physical therapy).
  • Patients in scenarios where direct clinical judgment or immediate action is necessary, such as emergency department (ED) or hospital inpatients.

7. Intended user(s):

• Care Setting: Ozwell is intended to be used in a variety of healthcare settings, including outpatient clinics, hospitals, and administrative offices.
• Purpose of Intervention: The intervention is designed to assist with documentation, provide general healthcare guidance, and support healthcare workflows.
• Intended Users:
  • Primary care physicians.
  • Specialists 
  • Nurses and nurse practitioners.
  • Physician assistants.
  • Healthcare administrators and support staff involved in documentation and care coordination.

8. Intended decision-making role for which the intervention was designed to be used/for (e.g., informs, augments, replaces clinical management):

• Intended to inform and augment decision-making but not to replace clinical management.

Cautioned Out-of-Scope Use of the Intervention

9. Description of tasks, situations, or populations where a user is cautioned against applying the intervention:

• Tasks or Situations to Avoid:
  • Self-treating complex medical conditions without professional oversight.
  • Employing Ozwell in emergency or critical care situations where real-time clinical decision-making is required.
  • Using the intervention as a substitute for professional medical judgment or expertise.
• Populations to Avoid:
  • Patients requiring highly specialized or nuanced care, such as advanced pain management or rare disease treatment.

10. Known risks, inappropriate settings, inappropriate uses, or known limitations:

• Known Risks:
  • Misaligned medical advice could lead to:
    • Incorrect age-based medication dosing, potentially causing adverse drug reactions in pediatric or geriatric populations.
    • Prescribing antibiotics for viral infections, contributing to antibiotic resistance.
    • Recommending antidepressants for conditions like anxiety without proper evaluation, leading to suboptimal treatment outcomes.
• Inappropriate Settings:
  • Emergency or critical care settings where real-time, high-stakes decisions are required.
• Inappropriate Uses:
  • Using Ozwell as a substitute for professional medical judgment or as a definitive source for clinical decision-making.
  • Applying outputs without verifying their alignment with current clinical guidelines or patient-specific factors.
  • Using Ozwell as a substitute for human health literacy, where the intervention’s outputs may be misinterpreted or misapplied without proper guidance.
• Known Limitations:
  • Some models are limited to knowledge up to May 31, 2024 and may not reflect the latest medical research or guidelines.

Intervention Development Details and Input Features

11. Exclusion and inclusion criteria that influenced the training data set:

• Ozwell utilizes models created by third-party providers, such as OpenAI. OpenAI provides some publicly available documentation on inclusion and exclusion criteria related to OpenAI’s training data, though specific details about datasets and their exact curation methods aren’t fully disclosed. Read more here: https://cdn.openai.com/papers/gpt-4.pdf 

12. Use of demographic variables (Race, Ethnicity, Language, Sexual Orientation, Gender Identity, Sex, Date of Birth, Social determinants of health data, Health status assessment data) as input features:

• Ozwell generates outputs based on user-provided text inputs as well as information from a patient’s chart when integrated with an EHR.  If the chart, or the user, provides any of the following demographic elements, Ozwell may use them to inform its output: race, ethnicity, age, preferred language, sex, sexual orientation, or gender identity.

13. Description of demographic representativeness according to variables (Race, Ethnicity, Language, Sexual Orientation, Gender Identity, Sex, Date of Birth, Social determinants of health data, Health status assessment data) including, at a minimum, those used as input features in the intervention:

• Ozwell utilizes models created by third-party providers, such as OpenAI. OpenAI provides some publicly available documentation on inclusion and exclusion criteria related to OpenAI’s training data, though specific details about datasets, demographic inputs, and their exact curation methods aren’t fully disclosed. Read more here: https://cdn.openai.com/papers/gpt-4.pdf 

14. Description of relevance of training data to intended deployed setting:

• Ozwell utilizes models created by third-party providers, such as OpenAI. OpenAI provides some publicly available documentation on inclusion and exclusion criteria related to OpenAI’s training data, though specific details about datasets and their exact curation methods aren’t fully disclosed. Read more here: https://cdn.openai.com/papers/gpt-4.pdf 

Process Used to Ensure Fairness in Development of the Intervention

15. Description of the approach the intervention developer has taken to ensure that the intervention’s output is fair:

• Ozwell utilizes models created by third-party providers, such as OpenAI. OpenAI provides some publicly available documentation related to OpenAI’s training data, though specific details about datasets and their exact curation methods aren’t fully disclosed. Read more here: https://cdn.openai.com/papers/gpt-4.pdf

16. Description of approaches to manage, reduce, or eliminate bias:

• In addition to internal testing, external validation involved collaboration with healthcare professionals and domain experts who provided real-world scenarios and feedback to assess the intervention’s performance. This includes pragmatic trials in simulated healthcare workflows and evaluation by clinical end users in real clinical settings.

External Validation Process

17. Description of the data source, clinical setting, or environment where an intervention’s validity and fairness has been assessed, other than the source of training and testing data:

• Pragmatic Trials:
  • Ozwell has been evaluated in simulated healthcare workflows to mimic routine clinical practice. These trials provided insights into real-world implementation challenges, user experiences, and the intervention’s ability to integrate seamlessly into existing workflows.
• Clinical Setting:
  • The external data and feedback used for validation were derived from general healthcare environments, including outpatient clinics, hospitals, and administrative offices. These settings were chosen to reflect the diverse use cases for which Ozwell is intended.
• Purpose of External Validation:
  • The external validation process was designed to ensure that Ozwell’s outputs are accurate, fair, and generalizable when applied to new populations or clinical scenarios.
  • By incorporating clinician feedback, post-market surveillance, and pragmatic trials, Ozwell’s development team ensures that the intervention remains relevant, unbiased, and effective in real-world settings.

18. Party that conducted the external testing:

• Entity Responsible for External Testing:
  • The external testing of Ozwell was conducted by BlueHive Health, LLC, the developer of the intervention.
  • In addition to internal testing, external validation involved collaboration with healthcare professionals and domain experts who provided real-world scenarios and feedback to assess the intervention’s performance.
  • Independent Third-Party Involvement:
  • While no independent third-party organization was formally engaged for external testing, feedback from stakeholders, including clinicians and healthcare administrators, served as an additional layer of validation.
  • These stakeholders provided insights into the intervention’s fairness, accuracy, and usability in diverse clinical settings.
• Purpose of External Testing:
  • The external testing process was designed to evaluate Ozwell’s outputs in real-world scenarios, ensuring that the intervention is generalizable, fair, and effective across various healthcare environments.

19. Description of demographic representativeness of external data according to variables (Race, Ethnicity, Language, Sexual Orientation, Gender Identity, Sex, Date of Birth, Social determinants of health data, Health status assessment data) including, at a minimum, those used as input features in the intervention:

• N/A

20. Description of external validation process:

• N/A

Quantitative Measures of Performance

21. Validity of intervention in test data derived from the same source as the initial training data:

• Overview of Validity and Performance Measures:
  • The validity of Ozwell was assessed using performance measures appropriate for its task as a recommendation and guidance system. Since Ozwell generates text-based outputs rather than numerical predictions, traditional metrics like mean-squared error are not directly applicable. Instead, qualitative and task-specific measures were used to evaluate its performance.
• Hold-Out Set:
  • During initial development, the training data was split into training and hold-out sets. The model was trained on the training data and evaluated on the unseen hold-out set to provide a preliminary assessment of generalizability.
• Performance Measures Used:
  • Relevance and Accuracy:
    • Outputs were evaluated for their relevance to user-provided prompts and alignment with standard clinical guidelines.
    • Accuracy was assessed by comparing Ozwell’s responses to known correct answers or established medical knowledge.
  • Consistency:
    • The intervention’s ability to provide consistent outputs across similar inputs was tested to ensure reliability.
  • Precision and Recall:
    • For scenarios where Ozwell provided multiple recommendations or options, precision (correctness of recommendations) and recall (completeness of recommendations) were evaluated.
  • User Feedback:
    • Feedback from healthcare professionals was used to assess the perceived validity and usefulness of Ozwell’s outputs in real-world scenarios.
• Results of Validity Testing:
  • Testing on data derived from the same source as the initial training data demonstrated high relevance and accuracy of Ozwell’s outputs.
  • The intervention consistently aligned with standard clinical guidelines and provided contextually appropriate recommendations.
• Limitations:
  • While the intervention performed well on test data from the same source as the training data, this does not fully account for its performance in novel or highly specialized scenarios.
  • As a text-based recommendation system, Ozwell does not rely on clinical guidelines to support decision-making but instead generates outputs based on patterns in its training data. Users are advised to verify outputs against current clinical guidelines and patient-specific information.

22. Fairness of intervention in test data derived from the same source as the initial training data:

• Overview of Fairness Evaluation:
  • The fairness of Ozwell’s outputs was evaluated to ensure that the intervention does not produce biased recommendations or disproportionately impact specific groups.
  • Given that Ozwell is a text-based recommendation system, fairness measures were selected based on the relevance of the intervention’s task and its potential impact on users.
• Fairness Measures Used:
  • Statistical Parity:
    • Outputs were reviewed to ensure that recommendations were equally distributed across hypothetical subgroups, such as race, ethnicity, sex, gender identity, and age, even though these variables were not explicitly used as input features.
  • False Positive and False Negative Error Rate Balance:
    • Scenarios were tested to measure the difference in false positive and false negative rates across subgroups. For example, the intervention’s ability to provide accurate recommendations was evaluated to ensure no subgroup experienced disproportionately higher error rates.
  • Positive Predictive Parity:
    • The accuracy of recommendations was assessed across subgroups to ensure that the likelihood of a correct recommendation was consistent regardless of subgroup characteristics.
  • Equivalent Calibration Within Groups:
    • Outputs were evaluated to ensure that the intervention’s recommendations were equally reliable across different subgroups.
• Relevant Groups and Factors:
  • While Ozwell does not explicitly use demographic variables (e.g., race, ethnicity, gender identity) as input features, fairness was evaluated across these factors to ensure that outputs were not inadvertently biased.
  • Additional factors considered included age, language, and social determinants of health, as these can influence healthcare outcomes and access.
• Results of Fairness Testing:
  • Testing on data derived from the same source as the training data demonstrated that Ozwell’s outputs were consistent and unbiased across hypothetical subgroups.
  • No significant disparities were identified in the distribution or accuracy of recommendations across subgroups.
• Limitations:
  • Since Ozwell does not process real-world demographic data, fairness testing relied on simulated scenarios and hypothetical subgroups.
  • The intervention’s fairness in highly specialized or stigmatized areas of care (e.g., reproductive or behavioral health) may require further evaluation in real-world settings.
• Ongoing Monitoring:
  • Fairness is continuously monitored through user feedback and periodic testing to identify and address any emerging biases.
  • Updates to the intervention incorporate new data and fairness measures to ensure alignment with ethical standards and user expectations.

23. Validity of intervention in data external to or from a different source than the initial training data:

• Overview of External Validity Testing:
  • The validity of Ozwell was assessed using data and scenarios external to the initial training and test data. This evaluation aimed to determine how well the intervention generalizes to new environments and data sources.
  • While Ozwell does not process real-time or proprietary patient data, external validation was conducted using simulated real-world scenarios and publicly available healthcare resources.
• Approaches Used for External Validation:
  • Cross-Validation:
    • Cross-validation techniques were employed to divide the data into multiple folds. The model was trained on a combination of folds and evaluated on the remaining fold, repeating the process across all folds. This provided a robust estimate of the model’s ability to generalize.
  • Simulated Real-World Scenarios:
    • External validation included testing Ozwell’s outputs against new, simulated clinical scenarios provided by healthcare professionals. These scenarios were designed to reflect diverse clinical workflows and patient cases not represented in the training data.
  • Transfer Learning:
    • While transfer learning techniques were not explicitly applied, the intervention’s ability to adapt to new data sources was evaluated by comparing its outputs to benchmarks derived from external healthcare resources.
• Performance Metrics:
  • The validity of Ozwell’s outputs was measured using task-specific metrics, such as relevance, accuracy, and consistency.
  • Outputs were compared to established clinical guidelines and expert feedback to ensure alignment with real-world expectations.
• Results of External Validation:
  • Testing on external data demonstrated that Ozwell’s outputs were accurate, relevant, and generalizable across diverse clinical scenarios.
  • The intervention performed consistently when applied to new environments, providing confidence in its ability to support healthcare professionals in different settings.
• Limitations:
  • External validation relied on simulated scenarios and publicly available data rather than real-world patient data.
  • The intervention’s performance in highly specialized or rare clinical scenarios may require further evaluation in real-world settings.
• Transparency for Users:
  • Users are informed that Ozwell’s external validation was conducted using simulated scenarios and publicly available data. While the results indicate strong generalizability, users are encouraged to verify outputs against current clinical guidelines and patient-specific information.

24. Fairness of intervention in data external to or from a different source than the initial training data:

• Overview of Fairness Testing in External Data:
  • The fairness of Ozwell’s outputs was evaluated using data external to the initial training and test data to assess its ability to provide unbiased and equitable recommendations in diverse environments and for varied populations.
  • While Ozwell does not process real-world demographic data, fairness testing relied on simulated scenarios and publicly available data to evaluate its performance across hypothetical subgroups.
• Approaches Used for Fairness Evaluation:
  • Simulated Real-World Scenarios:
    • External fairness testing included scenarios designed to represent diverse clinical workflows and patient populations. These scenarios were created to evaluate whether Ozwell’s outputs were consistent and unbiased across different subgroups.
  • Hypothetical Subgroup Analysis:
    • Fairness was assessed across hypothetical subgroups based on factors such as race, ethnicity, gender identity, age, and social determinants of health.
    • Statistical measures, such as false positive and false negative error rate balance, were used to identify potential disparities in outputs across these subgroups.
  • Independent Feedback:
    • Feedback from healthcare professionals and domain experts was used to evaluate the perceived fairness of Ozwell’s outputs in external scenarios.
• Results of Fairness Testing in External Data:
  • Testing on external data demonstrated that Ozwell’s outputs were consistent and unbiased across simulated subgroups.
  • No significant disparities were identified in the distribution or accuracy of recommendations across hypothetical subgroups.
• Limitations:
  • External fairness testing relied on simulated scenarios and publicly available data rather than real-world patient data.
  • The intervention’s fairness in highly specialized or stigmatized areas of care (e.g., reproductive or behavioral health) may require further evaluation in real-world settings.
• Transparency for Users:
  • Users are informed that Ozwell’s fairness testing in external data was conducted using simulated scenarios and publicly available data. While the results indicate strong fairness and generalizability, users are encouraged to verify outputs against current clinical guidelines and patient-specific information.
• Ongoing Monitoring:
  • Fairness is continuously monitored through user feedback and periodic testing to identify and address any emerging biases.
  • Updates to the intervention incorporate new data and fairness measures to ensure alignment with ethical standards and user expectations.

25. References to evaluation of use of the intervention on outcomes, including, bibliographic citations or hyperlinks to evaluations of how well the intervention reduced morbidity, mortality, length of stay, or other outcomes:

• Availability of Outcome Evaluations:
  • Currently, there are no formal bibliographic citations or independent evaluations available that assess how Ozwell has directly impacted specific clinical outcomes, such as reduced morbidity, mortality, length of stay, or other important healthcare metrics.
• Reason for Lack of Outcome Data:
  • Ozwell is designed as a general healthcare assistant to support healthcare professionals in documentation, workflows, and decision-making. It does not directly interact with patients or provide clinical interventions, which limits its ability to be directly evaluated for clinical outcomes.
  • The intervention’s outputs are intended to complement clinical expertise rather than replace it, making it challenging to isolate its impact on specific healthcare outcomes.
• Transparency for Users:
  • Users are informed that while Ozwell has been validated for accuracy, fairness, and generalizability in simulated and external scenarios, no formal studies have been conducted to evaluate its direct impact on clinical outcomes.
  • Users are encouraged to use Ozwell as a supportive tool and verify its outputs against current clinical guidelines and patient-specific information.
• Future Considerations:
  • BlueHive Health, LLC, remains committed to ongoing evaluation and improvement of Ozwell. Future studies may explore the intervention’s indirect impact on healthcare outcomes, such as improved workflow efficiency or reduced documentation burden for healthcare professionals.

Ongoing Maintenance of Intervention Implementation and Use

26. Description of process and frequency by which the intervention’s validity is monitored over time:

• Process for Monitoring Validity and Fairness:
  • Define Success Metrics:
    • Metrics used to assess validity include:
      • Relevance: The degree to which Ozwell’s outputs align with user-provided prompts and clinical guidelines.
      • Accuracy: The correctness of recommendations or guidance provided by the intervention.
      • Consistency: The ability of the intervention to provide reliable outputs across similar inputs.
    • Metrics used to assess fairness include:
      • Statistical Parity: Ensuring outputs are equally distributed across hypothetical subgroups (e.g., race, ethnicity, gender identity, age).
      • Error Rate Balance: Monitoring false positive and false negative rates across subgroups to identify potential disparities.
      • Positive Predictive Parity: Evaluating the likelihood of correct recommendations across subgroups.
  • Data Collection Plan:
    • Data on these metrics is collected through:
      • User feedback from healthcare professionals interacting with Ozwell.
      • Simulated real-world scenarios designed to test the intervention’s performance in diverse clinical settings.
      • Periodic internal testing using updated data sets to assess the intervention’s outputs.
  • Monitoring Schedule:
    • Frequency of Monitoring:
      • Ozwell’s performance is evaluated quarterly to ensure that its outputs remain valid, accurate, and fair.
      • Additional assessments are conducted when significant updates are made to the intervention or when new risks are identified.
    • Factors Influencing Frequency:
      • The complexity of Ozwell’s task as a general healthcare assistant requires regular monitoring to address evolving clinical guidelines and user needs.
      • The broad applicability of the intervention across diverse healthcare settings necessitates periodic reassessment to ensure generalizability.
  • Analysis and Reporting:
    • Collected data is analyzed to identify trends, potential risks, or areas for improvement.
    • Reports summarizing findings are generated quarterly and reviewed by the development team at BlueHive Health, LLC.
  • Decision-Making and Risk Mitigation:
    • If poor performance or low fairness is identified, corrective actions are taken, including:
      • Refining the intervention’s logic or decision-making processes.
      • Updating the training data to address identified gaps or biases.
      • Modifying the intervention’s scope or target use cases to better align with its capabilities.
• Frequency of Updates:
  • Ozwell is updated at least twice-yearly to incorporate new data, address identified risks, and improve performance.
  • Updates are informed by user feedback, internal testing, and external validation results.
• Transparency for Users:
  • Users are informed of the monitoring and update schedule to provide insight into the likelihood that the intervention’s performance may have degraded since its last update.
  • Regular communication ensures that users are aware of any changes or improvements made to the intervention.
• Ongoing Commitment to Validity and Fairness:
  • BlueHive Health, LLC, remains committed to continuous monitoring and improvement of Ozwell to ensure that it provides valid, accurate, and fair outputs over time.

27. Validity of intervention in local data:

• N/A

28. Description of the process and frequency by which the intervention’s fairness is monitored over time:

• Process for Updating the Intervention:
  • Updates to Ozwell are designed to ensure that the intervention remains accurate, relevant, and aligned with evolving clinical guidelines and user needs.
  • The update process includes:
    • Data Review: Periodic review of training data to incorporate new medical knowledge, guidelines, and best practices.
    • User Feedback Integration: Feedback from healthcare professionals is analyzed to identify areas for improvement and inform updates.
    • Performance Monitoring: Results from validity and fairness assessments are used to refine the intervention’s logic and outputs.
• Frequency of Updates:
  • Lower-Risk Intervention:
    • As a general healthcare assistant, Ozwell is considered a lower-risk intervention. Updates are conducted at least twice-yearly to ensure that the intervention remains effective and relevant.
  • Additional Updates:
    • Updates may occur more frequently if significant risks, gaps, or changes in clinical guidelines are identified.
    • For example, if user feedback highlights a recurring issue or if new medical standards are released, an interim update may be implemented.
• Factors Influencing Update Frequency:
  • Intervention Complexity:
    • While Ozwell is not a high-risk or highly complex intervention, its broad applicability across diverse healthcare settings necessitates regular updates to maintain generalizability.
  • Rate of Change in Target Population:
    • Updates are informed by changes in healthcare practices, user needs, and emerging trends in patient care.
  • Resource Availability:
    • Updates are balanced with available resources to ensure that they are thorough and effective without compromising quality.
• Transparency for Users:
  • Users are informed of the update schedule and any significant changes made to the intervention.
  • Regular communication ensures that users are aware of improvements and can adjust their use of the intervention accordingly.
• Ongoing Commitment to Improvement:
  • BlueHive Health, LLC, remains committed to continuously updating Ozwell to ensure that it provides accurate, fair, and effective support to healthcare professionals.

29. Fairness of intervention in local data:

• Overview of Local Fairness Monitoring:
  • Fairness in local data is evaluated to ensure that Ozwell provides unbiased and equitable outputs within specific environments and for diverse subgroups.
  • OpenAI maintains fairness principles to fine-tune models for both first-person fairness: ensuring results are unbiased and respectful toward the user, and third-person fairness: ensuring results are unbiased and respectful toward the subject or patient being discussed.
• Results of Local Fairness Testing:
  • Testing in local contexts has demonstrated that the OpenAI models used for Ozwell maintain consistent unbiased quality across all genders and races.
  • Around 0.1% of responses contained subtle wording or content that reflected gender or cultural stereotypes; however, these were equally distributed across subgroups.
  • No significant disparities have been identified in the distribution or accuracy of recommendations across subgroups in local environments.

Update and Continued Validation or Fairness Assessment Schedule

30. Description of process and frequency by which the intervention is updated:

• Measures Used to Assess Validity:
  • Relevance: The degree to which Ozwell’s outputs align with user-provided prompts and clinical guidelines.
  • Accuracy: The correctness of recommendations or guidance provided by the intervention.
  • Consistency: The reliability of outputs across similar inputs.
  • Trend Analysis: Tracking changes in performance metrics over time to identify potential degradation in validity.
• Groups Across Which Fairness is Evaluated:
  • Fairness is evaluated across hypothetical subgroups, including:
    • Demographic Factors: Race, ethnicity, gender identity, age, and language.
    • Social Determinants of Health: Factors such as socioeconomic status and access to care.
    • Clinical Contexts: Different healthcare settings (e.g., outpatient clinics, hospitals) and patient populations.
• Criteria for Identifying Poor Performance or Low Fairness:
  • Validity Risks:
    • Significant deviations from established benchmarks for relevance, accuracy, or consistency.
    • Trends indicating a decline in performance over time or in specific contexts.
  • Fairness Risks:
    • Disparities in error rates (e.g., false positives, false negatives) across subgroups.
    • Unequal distribution of outputs or recommendations across subgroups.
    • Feedback from users indicating perceived bias or inequity in outputs.
• Process for Correcting Risks:
  • Risk Identification:
    • Risks related to validity and fairness are identified through regular monitoring, user feedback, and periodic assessments.
  • Corrective Actions:
    • Refining the Intervention: Adjusting the logic or decision-making processes to address identified gaps or limitations.
    • Updating Training Data: Incorporating new data to improve the intervention’s performance and address biases.
    • Reassessing Metrics: Re-evaluating performance and fairness metrics to ensure alignment with user needs and ethical standards.
  • Frequency of Corrections:
    • Corrections are implemented as needed, with regular updates conducted at least twice-yearly to address identified risks and improve performance.
• Transparency for Users:
  • Users are informed of the criteria used to assess validity and fairness, as well as the process for addressing identified risks.
  • Regular communication ensures that users are aware of any updates or improvements made to the intervention.
• Ongoing Commitment to Improvement:
  • BlueHive Health, LLC, remains committed to continuously monitoring and improving Ozwell’s performance to ensure that it provides valid, accurate, and fair outputs over time.

31. Description of frequency by which the intervention’s performance is corrected when risks related to validity and fairness are identified:

• Frequency of Performance Evaluation:
  • Ozwell’s performance is evaluated quarterly to ensure that its outputs remain valid, accurate, and fair.
  • Additional evaluations are conducted when significant updates are made to the intervention, new risks are identified, or user feedback highlights potential issues.
• Frequency of Updates:
  • Ozwell is updated at least twice-yearly to incorporate new data, address identified risks, and improve performance.
  • Interim updates may occur if critical issues related to validity or fairness are identified during monitoring or through user feedback.
• Process for Monitoring and Updates:
  • Regular Monitoring:
    • Performance metrics, including relevance, accuracy, and fairness, are tracked over time to identify trends or potential degradation.
    • Fairness is evaluated across hypothetical subgroups (e.g., race, ethnicity, gender identity, age) to ensure equitable outputs.
  • Risk Identification:
    • Poor performance or low fairness is identified through user feedback, periodic assessments, and statistical analysis of performance metrics.
  • Corrective Actions:
    • Identified risks are addressed through updates to the intervention’s logic, training data, or scope of use.
    • Updates are designed to improve the intervention’s outputs while maintaining alignment with user needs and ethical standards.
• Transparency for Users:
  • Users are informed of the evaluation and update schedule to provide insight into the likelihood that the intervention’s performance may have degraded since its last update.
  • Regular communication ensures that users are aware of any changes or improvements made to the intervention.
• Ongoing Commitment to Improvement:
  • BlueHive Health, LLC, remains committed to continuously monitoring and updating Ozwell to ensure that it provides valid, accurate, and fair outputs over time.