AHCPR Fiscal Year 2000 Performance Plan

You Are Here: AHRQ Home > About AHRQ > Strategic Plan > Performance Plans and Performance Report

Goal 4: Evaluate the effectiveness and impact of AHCPR research and associated activities. (HCQO)

(Note: All Agency evaluation activities, including MEPS-related studies, are included under Goal 4. This is because the MEPS budget line covers only costs associated with data design, data collection and analysis, and data products.)

Strategy

As explained in other portions of this document, interim outcomes of research can be evaluated on a relatively short-term basis. However, the ultimate outcome of how the research affects people receiving health care or people interacting with the system requires large, expensive retrospective studies. AHCPR is implementing a growing portfolio of evaluations that will show, iteratively, the outcomes of the investments of Agency funds. The fiscal year 2000 strategy involves assessing the interim outcomes of four tools created with Agency funds for improving health care quality and evaluating the use and usability of MEPS databases for their intended purposes.

Previous Successes

In 1997 an evaluation was completed on Changes Needed in CONQUEST's Software-Based Programs. It identified the the best computer platform for the system; the problems identified by users that can be corrected through minor enhancements, error corrections, and superficial changes; and a long term strategy for disseminating CONQUEST to the widest possible audience through the World Wide Web.
A Customer Satisfaction Survey of AHCPR Publications Recipients provided feedback on four patient/consumer guides: Prescription Medicines and You; Understanding Incontinence; Early Alzheimer's Disease; and You Can Quit Smoking. The results indicated overall satisfaction with the guides. The report also indicated that AHCPR provide the Clearinghouse staff with more information about the contents of each guide so that they can provide increased customer service when dealing with telephone requests.
The final report of the Small Conference Grant Program Evaluation provided AHCPR with recommendations to increase the effectiveness of this program, including: (1) future grant announcements should set time frames for both the review of concept letters and grant applications based on the Agency's experience in handling the reviews; (2) AHCPR should consider implementing a time limit on application submission once the applicant to proceed so that the Agency can develop an accurate list of expected or pending applications for planning purposes; (3) AHCPR should be more proactive in participating in the development of a dissemination strategy for conferences to increase the impact of the conferences; and (4) in the future, the Agency might consider lending its name to conferences, rather than its money, to extend AHCPR's reach without increasing funds allocated to the program.
A Data Collection Evaluation is currently in progress that will assess the usefulness of major AHCPR-supported data collection efforts, such as the Medical Expenditure Panel Survey, the Health Care Cost and Utilization Project, and the HIV Cost and Services Utilization Survey, to policymakers and researchers. The study is examining methods to improve survey operations and timely data release and includes examining how other federal agencies and private foundations conduct various aspects of their large-scale surveys.
The Healthcare Association of New York State applied HCUP Quality Indicators (QIs) to hospitals statewide, and the results prompted a large health system in New York to create a center of excellence for the care of its diabetic patients.
The Hawaii Health Information Corporation received approval from the Joint Commission for Accreditation of Healthcare Organizations to incorporate HCUP QIs in their performance management initiative.

Goal 4 Fiscal 1999 Year Objectives and Indicators

Fiscal 1999 Year Objective 4.1: Evaluate the impact of AHCPR sponsored products in advancing methods to measure and improve health care.

Indicators
Results from the following evaluation projects will be provided in the fiscal year 1999 performance report.

(A) An evaluation of the outcomes of outcomes research and the impact of AHCPR-supported outcomes and effectiveness research on clinical practice.

(B) An evaluation and synthesis of (1) primary care research supported by AHCPR and (2) an assessment of the current state of the science and future directions for primary care research.

(C) AHCPR's State data strategy will be redesigned based on consultations with state policymakers, researchers, hospital associations, and others about their past use of data from Healthcare Cost and Utilization Project (HCUP) as well as additional data needs.

(D) Results of the evaluation of the Consumer Assessment of Health Plan Study (CAHPS®) will be used to improve the usability and usefulness of the tool. Findings are expected to show whether (1) the survey-based information from CAHPS® helps consumers make better health care decisions, (2) the information increases consumer confidence when choosing health care plan, and (3) CAHPS® is used by public and private organizations.

(E) Evaluation studies on: (1) the quality and usefulness of the evidence reports and technology assessments produced by the Evidence-based Practice Centers and (2) the impact of the use of these products on the health care system will be developed and initiated in fiscal year 1999. Final evaluation report will be received in February 2000.

Baseline data for evaluations: N/A

Type and Significance
Outcome: The evaluations outlined in (A) - (E) are of five core Agency programs or projects. AHCPR conducts evaluations of its major programs or products to determine one or more of the following:

Evaluate the current state of the program or product including impact in health care
Improve customer satisfaction with the program or product
Target or prioritize future activities to increase their usability or usefulness

Data Collection
OMB clearance for customer survey received in December 1998. Surveys under development. The information needed for the CAHPS® evaluation is built into the structure of the project, so we anticipate being able to acquire these data.

Indicators
a. AHCPR Clearinghouse customer satisfaction rated at 98 percent. (Baseline: Overall experience in ordering from Clearinghouse—96.4 percent in first half of fiscal year 1997 )

b. Customer satisfaction data on AHCPR consumer publications (useful/relevant) rated at 90 percent. (Baseline: 81.1 percent from 1997 survey)

Followup customer survey will begin 12/28/98. Updated results will be available in late January 1999.

Type and Significance
Outcome: The success of the AHCPR publications clearinghouse and AHCPR publications directly affect the ability of the Agency to get the results of health services research into the hands of users.

Goal 4 Fiscal Year 2000 Objectives and Indicators

Fiscal Year 2000 Objective 4.1: Evaluate the impact of AHCPR sponsored products in advancing methods to measure and improve health care quality.

Indicators
Use of evidence reports and technology assessments to create quality improvement tools in at least 10 organizations.

Baseline under development.

Indicators Summary
Numerator—Number of organizations using reports to create QI tools.
Denominator—Number of organizations requesting reports from all AHCPR sources (e.g., Web site, publications, etc.).

Type and Significance
Outcome: AHCPR currently funds 12 Evidence-based Practice Centers to produce evidence reports and technology assessments. The report topics are requested by organizations with an interest in developing guidelines, performance measures, or other quality improvement tools or in informing decisions (including coverage decisions) related to the effectiveness or appropriateness of health care technologies.

Data Collection
OMB clearance for customer survey received in December 1998.

Indicators
For at least four evidence reports or technology assessments per year, work with partners to measure how the reports or assessments were used and what impact they had on clinical decisionmaking and patient care.

Baseline under development.

Indicators Summary
Evaluation study documenting impact of EPC reports.

Type and Significance
Outcome: The reports, assessments, and related tools have been used by individual clinicians, health plans, providers, purchasers, and consumers.

Data Collection
The information needed for this evaluation is built into the structure of the project, so we anticipate being able to acquire these data.

Indicators
At least three examples of how research informed changes in policies or practices in other Federal agencies.

Baseline under development.

Indicators Summary
Numerator—Number of Federal agencies using research in policy formation.
Denominator—Number of Federal agencies requesting and/or provided research information.

Type and Significance
Outcome: Evidence reports, technology assessments, and products have significant potential to inform Federal health care policymaking. Actual impact will be assessed through this measure.

Data Collection
Collected through coordination with Federal Partners.

Fiscal Year 2000 Objective 4.2: Evaluate the impact of AHCPR sponsored products in advancing methods to measure and improve health care quality.

Indicators
AHCPR will report on the extent to which CONQUEST assists those who are charged with carrying out quality measurement and improvement activities and the extent to which it helps further state-of-the-art in clinical performance measurement.

Baseline will be established by the evaluation study.

Indicators Summary
Numerator—Number of users in study who report that CONQUEST assists their quality measurement and improvement activities.
Denominator—Number sampled.

Type and Significance
Outcome: CONQUEST is a quality improvement software tool developed, implemented, and enhanced with research funding provided by AHCPR. Beginning in late 1994, AHCPR has invested significant resources in to its development. To validate CONQUEST's usefulness and potential for advancing and strengthening health care measurement and improvement, its current use will be evaluated.

Data Collection
The final evaluation study design will be available in February 1999 and will be implemented at appropriate time.

Indicators
CAHPS® has assisted the Health Care Financing Administration (HCFA) in informing Medicare beneficiaries about their health care choices. The use and impact of this information is determined by surveying a sample of these beneficiaries.

Baseline under development.

Indicators Summary
Report on use of CAHPS® by HCFA.

Type and Significance
Outcome: The CAHPS® surveys and reports were developed in the first phase of the project, beginning in September 1995 and continuing until January 1997. In the second phase of the project, running until September 2000, the CAHPS® products are being evaluated for impact on consumers' decisionmaking.

Data Collection
Collaborations with HCFA, NCQA, JCAHO, and others will involve negotiating for needed data collection.

Indicators
At least three quality measures from Q-span (or instances where AHCPR research contributes to the development of measures) are used in the Health Plan Employer Data Information Set (HEDIS) by the National Committee for Quality Assurance (NCQA), measurement activities of the Joint Commission for the Accreditation of Healthcare Organizations (JCAHO), or others who monitor health care quality.

Baseline in fiscal year 1998: One quality measure adopted and one instance of AHCPR-sponsored research contribute to adoption of measures.

Indicators Summary
Numerator—Number of times measures from Q-Span or AHCPR research used to outside organizations' measurement activities.
Denominator—Total number of measures coming out of Q-Span.

Type and Significance
Outcome: Q-Span, Expansion of Quality of Care Measures, is comprised of eight cooperative agreements funded in September 1996. They are 2-5 year projects addressing a range of subject matter, from dental care to subacute care. Conditions such as hip fractures, adult asthma, cardiovascular disease, diabetes, and breast cancer are included. Completed measures and findings will begin to be published in late 1999, and the impact on tools such as HEDIS should be identifiable by early 2000.

Data Collection
The addition of Q-Span measures to quality measurement data sets is done in coordination with AHCPR staff and/or the researcher responsible for the measure development; therefore, this information will be known and collected.

Indicators
Measure of extent to which hospitals change behavior and the impact of this change on patient care based on HCUP quality indicators.

Data to be collected for the first time with the survey.

Indicators Summary
Numerator—Number of hospitals changing behavior based on HCUP quality indicators.
Denominator—Number of hospitals receiving/requesting HCUP quality indicators.

Type and Significance
Outcome: Improved quality based on the HCUP quality measures is a goal of producing this information.

Data Collection
OMB clearance for customer survey received in December 1998.

Fiscal Year 2000 Objective 4.3: Evaluate the impact of MEPS data and associated products on policymaking and research projects.

Indicators
Use of MEPS data in 5 percent of research applications received by AHCPR.

Baseline to be developed in fiscal year 1999. MEPS products started to be available in fiscal year 1998, with more to be available in FY 1999. AHCPR is publishing program announcements indicating interest in receiving grant applications involving the use of MEPS data. The first research proposals using MEPS data are expected in fiscal year 1999.

Indicators Summary
Numerator—Number of research applications using MEPS data.
Denominator—Total number of research applications received.

Type and Significance
Outcome: One of the core purposes for collecting MEPS data is to enable researchers to conduct health services research that wouldn't be possible otherwise. Fiscal year 1999 will be the first time that extramural researchers are expected to begin using the data in their efforts.

Data Collection
For the time being, we will collect the number of grant applications received by the Agency proposing the use of MEPS data as a part of the grant. Will be identified through the grants management database and working with program staff.

Indicators
Distribution of MEPS data sets to at least 2500 requestors.

Baseline in fiscal year 1998—916 data sets downloaded from Web site. 1000 compact disks distributed at conferences and other venues.

Indicators Summary
Numerator—Number of MEPS data sets distributed during fiscal year 2000.

Type and Significance
Outcome: One of the core purposes for collecting MEPS data is to get the data out to potential users so that the Government isn't the only possessing and using this resource. While this objective looks at dissemination, other measures look at use and usability.

Data Collection
Distribution of the data sets will be tracked through Web site and Clearinghouse statistics.

Indicators
Feedback from recipients of MEPS data indicating that the data were timely, useful, and of high significance.

Baseline under development.

Indicators Summary
Numerator—Number of recipients surveyed who report data used for policy or research is timely and significant.
Denominator—Total number of recipients of data for policy or research surveyed.

Data Collection
OMB clearance for customer satisfaction survey received in December 1998. Survey will be conducted in June after the first full-year data are available.

Indicators
At least 10 examples of how research using MEPS has been used to inform decisions by Federal, State, and private sector policymakers.

Baseline under development.

Indicators Summary
Number of stories of use of results of MEPS research by policymakers.

Type and Significance
Outcome: Significance of MEPS Measures—MEPS is a nationally representative survey of health care use, expenditures, sources of payment, and insurance coverage. MEPS provides quick-turnaround, comprehensive data that will support analyses on the specific health services Americans use, how frequently they use them, the cost of these services and how they are paid, as well as data on the cost, scope, and breadth of private health insurance held by and available to the U.S. population. This study will analyze the applicability and use of these data sets for research and to support policy decisions.

Return to Contents
Proceed to Next Section