MEPS Annual Methodology Report

Deliverable Number: 142
Contract Number: 290-02-0005
April 17, 2008

Submitted to:
Agency for Healthcare Research and Quality
540 Gaither Road
Rockville, Maryland 20850

Submitted by:
WESTAT
1650 Research Boulevard
Rockville, Maryland 20850-3195
301-251-1500

TABLE OF CONTENTS

_. Introduction
1. Sample
1.1 Sample Design and Size
1.2 Sample Delivery and Processing
2. Instrument and Materials Designs
2.1 Questionnaire Changes for Spring and Fall 2007
2.2 Field Pretesting of the Blaise/WVS Instrument
2.3 Changes to Materials and Procedures for Spring and Fall 2007
3. Recruiting and Training
3.1 Recruiting for 2007
3.2 2007 Trainings
4. Data Collection
4.1 Schedule
4.2 Operations
4.3 Data Collection Results
5. Home Office Processing and Support
6. Panel 12 Data Collection: Interview Timing and Utilization Measures
7. Data Processing and Data Delivery
Appendix A. Comprehensive Tables – Household Survey
Table A-1 Data collection periods and starting RU-level sample sizes, all panels
Table A-2 MEPS household survey data collection results, all panels
Table A-3 Signing rates for medical provider authorization forms
Table A-4 Signing rates for pharmacy authorization forms
Table A-5 Results of self-administered questionnaire (SAQ) collection
Table A-6 Results of Diabetes Care Supplement (DCS) collection*
Table A-7 Calls to respondent information line
Table 1-1 Initial MEPS sample size and number of NHIS PSUs, all panels
Table 1-2 Data collection periods and starting RU-level sample sizes, Spring 2001 through Fall 2007
Table 1-3 Percentage of NHIS households with partially completed interviews in Panels 3 to 12
Table 2-1 Supplements to the CAPI core questionnaire (including hard-copy materials) for 2007
Table 4-1 Data collection schedule and number of weeks per round of data collection
Table 4-2 Percent of total interviews conducted on travel
Table 4-3 Results of patient profile collection for medications prescribed in 2006
Table 4-4 MEPS HC data collection results, Panels 8 through 12
Table 4-5 Summary of nonresponse for Round 1, 2003-2007
Table 4-6 Summary of nonresponse for Rounds 2 and 4, 2004-2007
Table 4-7 Signing rates for medical provider authorization forms for Panels 8 through 12
Table 4-8 Signing rates for pharmacy authorization forms (Panels 8 through 11)
Table 4-9 Results of self-administered questionnaire (SAQ) collection (Panels 8 through 12)
Table 4-10 Results of diabetes care supplement (DCS) collection* (Panels 8 through 11)
Table 4-11 Summary of MEPS Round 1 response, 2003-2007 panels
Table 4-12 Summary of MEPS Round 1 response, 2003-2007 panels, by NHIS completion status
Table 4-13 Summary of MEPS Round 1 response rates, 2004-2007 panels, by race/ethnicity and NHIS completion status
Table 4-14 Summary of MEPS refusal rates, 2002-2007 panels, by race/ethnicity and NHIS completion status
Table 4-15 Summary of MEPS Panel 12 Round 1 response rates, by sample domain by NHIS completion status
Table 4-16 Summary of MEPS Panel 12, Round 1 results: ever refused, final refusals, and refusal conversion rate, by region
Table 4-17 Summary of MEPS Panel 12 Round 1 results: ever traced and final not located, by region
Table 5-1 Calls to the respondent information line, 2006 and 2007
Table 6-1 Average interview time, by interviewer groups, Panel 12 Round 1 (single-session and legal break cases only)
Table 6-2 Panel 12 Round 1 single-session vs. multi session interviews, by interviewer training groups and interviewer production groups
Table 6-3 Average interview time, by NHIS completion status, Panel 12 Round 1
Table 6-4 MEPS Round 1 Summary Utilization Comparison: as of 7/31/2007
Table 6-5 Mean total events and office-based events, by interviewer training group
Table 6-6 Mean total and office-based events per person, by interviewer production level
Table 6-7 Number of Round 1 completes by month of the field period and mode of interview, Panel 11 Round 1 and Panel 12 Round 1
Table 6-8 Average total events and MV events per RU and per person, by RU size and mode of interview for Panel 12, Round 1
Table 6-9 Mean total events and MV events per person, Panel 12 and Panel 11, Round 1
Table 6-10 Panel 12 Round 2 interview outcome, by mean Round 1 interview time (Round 1 interviews with no breaks)
Table 6-11 Panel 12 MEPS survey participants’ interview outcome in Round 2 by Round 1 interview break status
Table 6-12 Panel 12 MEPS survey participants’ interview outcome in Round 2 by Round 1 ever refused status
Table 6-13 Panel 12 MEPS survey participants’ interview outcome in Round 2 by Round 1 interview month
Table 7-1 Files deliveries during 2007
Figure 7-1 August-October 2007 activities for data deliveries
Figure 7-2 Processing and methods flow of point-in-time files
Figure 7-3 Methods flow for MEPS full-year use and expenditure data delivery

This report documents the principal design, training and data collection activities of the Household Component of the Medical Expenditure Panel Survey for survey year 2007. These activities were conducted under Contract 290-02-0005, awarded in July 2002. As modified, the contract covers MEPS Panels 8-13.

Previous methodology reports have documented activities performed during the calendar year covered by the report. This report covers a somewhat different time span in that it includes all work associated with the fielding and support of the panels and rounds in the field during the survey year. Since much of the work performed for preparing to field a new panel occurs in the latter half of the year preceding the fielding, excluding a description of that work from the report lessens the cohesiveness of the discussion.

The report touches only briefly on procedures and operations that remained unchanged from prior years. It focuses primarily on features of the project that were new or that were changed or enhanced during 2007 and presents the results of the data collection activities conducted during the year. The report also provides a summary overview of data processing activities that supported data deliveries for the year. The tables within the report document 2007 data collection results. A comprehensive set of tables showing data collection results from prior years is included in Appendix A.

Survey year 2007 was met with new challenges as the CAPI conversion from DOS to Windows was implemented for Panel 12 Round 1. In addition, the Panel 12 sample of households was selected from PSUs in the new NHIS PSU sample fielded for the NHIS in 2006, which resulted in an expansion of the number of PSUs where cases were located. At the same time, the household component continued to maintain the ‘steady state’ schedule of recruiting, training, data collection, and data delivery established over the course of the past several years.

Chapter 1 of the report describes sample preparation activities. Chapters 2 through 5 discuss activities associated with the data collection for 2007 including field staff recruiting, training, materials development, questionnaire updates that took place in the Fall of 2006, data collection procedures and results, and home office processing support. Chapter 6 provides an analysis of the results of the data collection for Panel 12 Round 1 and Chapter 7 describes the data processing and delivery tasks that occurred during 2007.

Return To Table Of Contents

1. Sample

This chapter summarizes the activities associated with the preparation for and conduct of data collection for the MEPS Household Component (HC) for survey year 2007. Interviews conducted during the survey year were for Panel 10 Round 5, Panel 11 Rounds 3 and 4, and Panel 12 Rounds 1 and 2.

Return To Table Of Contents

Each year, MEPS draws its household sample from among responding households in the previous year’s National Health Interview Survey (NHIS). In 2006, the NHIS introduced a new sample design, which meant that the MEPS sample for 2007 – the Panel 12 sample -- would be drawn from a new set of primary sampling units. The new NHIS sample design included a group of entirely new PSUs and a sample of new segments in PSUs from the original sample that were retained in the new design. During the spring of 2006, NCHS provided a sample listing that identified the counties in the original (1995-2005) sample design that also appeared in the new design, counties that were excluded from the new design, and counties that were new to the new design. These lists of counties were re-grouped into MEPS-unique PSUs. With the new sample, the spring 2007 workload for MEPS Panels 10, 11, and 12 was distributed in:

• 102 MEPS-PSUs that overlapped with the original design. For 2007 interviewing, these PSUs included households in all three panels;
  
  
• 102 original MEPS-PSUs that were not part of the new NHIS sample. These PSUs had no new sample for Panel 12 but will continue in MEPS until the remaining interviews with Panels 10 and 11 are complete; and
  
  
• 46 MEPS-PSUs that are new to the design. For 2007 interviewing, these PSUs contained only Panel 12 households.

Combining the 2007 panel with the 2005 and 2006 panels scheduled for fielding in the spring of 2007 resulted in a total of 250 MEPS PSUs that needed interviewer coverage, an increase of 55 PSUs from around 195 PSUs that had been in previous MEPS panels.

In order to reduce the burden associated with introducing a new sample design in the same year the new windows-based Round 1 instrument was to be launched, AHRQ reduced the MEPS sample size for Panel 12 from a budgeted size of 9,048 reporting units (RUs) to 7,467 RUs. Unlike prior MEPS panels, which have been selected from the first three quarters of the NHIS, the 2007 sample was limited to the first two quarters of 2006 NHIS interviews. As with the Panel 10 and 11 samples, Panel 12 contained an oversample of Asian, low income, and Black households. Panel 12 also contained an oversample of Hispanic households.

The decision to limit the selection of households to the first two quarters of 2006 NHIS (Panels 1 and 4) interviews enabled AHRQ to deliver the final sample earlier than in prior years. This, in turn, allowed additional time to process the sample and identify new areas requiring interviewer recruitment. For the new sample it was necessary to map the counties in the new design and identify the counties that overlapped in the original sample and those that needed assignment to a new MEPS PSU. The sample sizes in many of the new PSUs were quite small – too small to provide a reasonable workload for a local interviewer. Planning, therefore, had to address the need for coverage in areas without local staff, as well as staffing in new areas.

The full 2007 MEPS sample was received on August 29, 2006, and work began immediately on reviewing NHIS household composition and designating the sample at a reporting unit level (groups of related household members living at a single NHIS dwelling unit (DU).) In addition, new PSU numbers were assigned, sample from new counties that were contiguous with original sample counties were merged into existing PSUs, and field supervisor regions were restructured to accommodate the new PSU design. Project managers assessed the location of current MEPS staff in relation to the new sample locations and in light of expected workloads in the three panels to be fielded in 2007. Recruiting goals were established and recruiting began in earnest in September of 2006.

Table 1-1 shows the starting sample sizes for Panels 1 to 12 and the number of NHIS PSUs from which each panel was drawn.

Table 1-1. Initial MEPS sample size and number of NHIS PSUs, all panels

Return To Table Of Contents

For the new NHIS sample design (introduced in the 2006 NHIS), PSUs were characterized differently than in the earlier sample design. NHIS has broken down what would have been large self-representing PSUs into smaller PSUs which consist of one or more counties and are defined as individual SPSUs (stratification PSUs). The Panel 12 sample contained 183 NHIS PSUs as they would have been defined in the pre-2006 NHIS sample design. These 183 PSUs are associated with 269 SPSUs.

Table 1-2 on the following page summarizes the combined workload for the January-June and July-December periods from spring 2001 through fall 2007. (Table A-1 in Appendix A shows the data collection periods and sample sizes for all panels and rounds.)

Across the three panels that were active during the first half of 2007, the combined workload was 21,326 RUs. This was the smallest composite HC sample fielded since 2001, and was due to the reduction in sample size for Panel 12. For the two panels that were active during the second half of the year, the total initial workload was 12,906 RUs. This sample was the smallest Fall workload on MEPS, again, due to the reduction in size of Panel 12.

Return To Table Of Contents

As mentioned earlier, the entire 2007 sample was delivered at one time and included households interviewed in just the first two quarters of the 2006 NHIS. In addition to work associated with configuring the field structure to accommodate the new PSU design, earlier receipt of the full sample allowed the project to review the NHIS sample file formats to identify new variables or values and to make necessary changes to the project programs that use the sample file information.

Table 1-2. Data collection periods and starting RU-level sample sizes, Spring 2001 through Fall 2007

Return To Table Of Contents

This was especially important for the 2007 MEPS sample which was loaded into a new database structure for use with the Windows application. Since the plan for transition to the windows system retained a link to the Cheshire database, the file processing for the Pane1 12 sample required no adaptation of the processing plan from earlier years. The Panel 12 sample was processed and loaded into Cheshire as had been done in earlier Panels. The data then was transformed from Cheshire into the database structure for the windows based system.

Each year, the NHIS sample includes a percentage of households classified as ‘partial completes’. Table 1-3 shows the percentage of NHIS interviews classified as "partially complete" in panels 3 through 12. The NHIS partial completes are, as a group, more difficult to complete in MEPS than the full NHIS completes and therefore receive special monitoring. For Panel 12 the percentage of partial completes fell to 19 percent from 23 percent in the Panel 11 sample.

Table 1-3. Percentage of NHIS households with partially completed interviews in Panels 3 to 12

Return To Table Of Contents

This chapter describes changes to the computer assisted personal interviewing (CAPI) instrument and supporting field materials made in support of the data collection activities for Spring and Fall 2007 (Panel 10 Round 5, Panel 11 Rounds 3 and 4, and Panel 12 Rounds 1 and 2).

As mentioned earlier, the Panel 12 Rounds 1 and 2 questionnaires were the first instruments programmed in the new windows-based system, Blaise/WVS. Most of the questionnaire changes, as described below, were made to the Panel 12 instruments. Few changes were made to the Panel 10 Round 5 and Panel 11 Rounds 3 and 4 instruments to minimize the effort to maintain the DOS-based system, Cheshire, in which these instruments were programmed. A more detailed discussion about the design effort for the Blaise/WVS instrument is provided in Chapter 3 of the 2006 MEPS Annual Methodology Report, Deliverable 74, June 1, 2007.

Return To Table Of Contents

2.1 Questionnaire Changes for Spring and Fall 2007

During 2007, the following revisions were made to the MEPS CAPI instrument:

• Relationships. Two categories (foster brother and foster sister) were added to the relationship list for all Panel 12 instruments. For Panel 12 Round 2, two questions were added to identify legal guardians.
  
  
• Conditions. The Condition Enumeration section was redesigned for all rounds of Panel 12 so that pregnant household members are identified on a roster. In addition, the questionnaire was changed so that the flag identifying pregnant household members could be set regardless of whether that woman had been identified as pregnant in a previous round.
  
  
• Health Status. Physical and mental health status questions were moved from the CE section to the new Priority Conditions Enumeration section in the Panel 12 interviews.
  
  
• Preventive Care. For Panel 12, seven questions were moved from the Priority Conditions section asked in all rounds to the Preventative Care section, which is only asked in Rounds 3 and 5.
  
  
• Access to Care. Starting in Panel 12 Round 2, all households are asked whether all members are comfortable conversing in English. Also, data about usual source of care providers are recorded at the usual source of care level instead of at the provider level. For both Panel 11 Round 4 and Panel 12 Round 2 all household members are asked if they were born in the United States, and the number of years a household member has resided in the United States is now collected as an exact number instead of as a range.
  
  
• Employment. In all rounds of the Panel 12 instrument, the method of assigning ID numbers to job records was modified.
  
  
• Health Insurance. In Panel 12 Round 2, "Health Insurance Purchasing Alliance" was removed as a response category for questions about other sources of health insurance.
  
  
• Pregnancy Detail. In all rounds of the Panel 12 instrument, the pregnancy detail section was eliminated and questions were moved into event sections (Hospital Stays, Outpatient, and Medical Visits).
  
  
• Outpatient Department. For Panel 12, the response category "Throat Swab" was added to the question that asks which treatments were received during an outpatient visit and to the associated showcard.
  
  
• Medical Provider Visits. For Panel 12, the response category "Throat Swab" was added to the question that asks which treatments were received during an ambulatory office visit and to the associated showcard. The response category "Indian Health Service (IHS) Facility" was added to the question asking where the treatment was received.
  
  
• Charge Payment. A pick list of common sources of payment was added to the pop-up screen in the Panel 12 instrument. For Panel 12 rounds 1-2, and Panel 11 Rounds 3-4, the term "third party payer" was changed to "source of payment."
  
  
• Prescribed Medicines. For all rounds of Panel 12, a new variable was set when a prescribed medicine was reported but the person had not yet taken the medication at the time of the interview.

Table 2-1 shows the supplements in the CAPI instrument for the rounds administered in calendar year 2007.

Return To Table Of Contents

2.2 Field Pretesting of the Blaise/WVS Instrument

Extensive pretesting was done throughout 2006 in preparation for the launch of the Blaise/WVS instrument. After the Round 1 training was completed in February of 2007, further pretesting was performed for the Rounds 2 and 3 instruments.

Table 2-1. Supplements to the CAPI core questionnaire (including hard-copy materials) for 2007

Return To Table Of Contents

• Round 2 Pretest. To test both the Round 2 Blaise/WVS instrument and the home study materials that would be sent out to interviewers in July 2007, a small pretest was conducted. Training took place at Westat on March 13, 2007. Six interviewers read the home study materials and conducted a self-paced mock interview. Home office staff members were available to answer questions and to note any problem areas in the materials or the instrument.
  
  
• A total of 72 interviews were completed between March 13^th and March 20^th, and a debriefing session was held with interviewers on the evening of March 20^th. Interviewers provided feedback on the training materials and discussed navigation issues that seemed to increase the duration of the interview.
  
  
• Round 3 Pretest. Six interviewers (5 experienced and 1 with no previous experience administering the Round 3 questionnaire) were trained on the Round 3 instrument on September 17^th, 2007. The training session included a mock interview and a lecture which covered the supplemental sections in Round 3. A total of 25 interviews were conducted in the field, 21 of which were observed by home office staff. The cases were selected to cover a range of household sizes and health care coverage types. In addition, 5 interviews were conducted in Spanish in order to test the Spanish version of the questionnaire.
  
  Ten additional interviews were conducted by Westat home office staff without respondents to test the instrument’s handling of questions which cross over two survey years. These cases used a February 2008 interview date, and scripted answers were used in order to confirm that dates from the previous year were handled appropriately by the questionnaire.
  
  No problems occurred with the data rollover from Round 2 or with the instrument’s handling of cross-year dates.

Return To Table Of Contents

2.3 Changes to Materials and Procedures for Spring and Fall 2007

With the conversion to the windows-based applications, a number of instructional manuals and supporting materials required updating to reflect changes in field procedures and administration of the new instrument and field management system. Also, there was a heightened awareness of the protection of respondent data in 2007 that led to taking more active steps to assuring the security of data collected. This also led to some procedural and material changes.

Because of the challenges posed by learning and operating in two different data collection systems, changes to materials and procedures were kept to a minimum to ease burden on the interviewers. Respondent contact materials (brochure, advance letters, etc.) were not changed materially; nor were the administrative forms used for record keeping revised in any significant way.

Changes made to MEPS materials and manuals are described below.

• Field Interviewer Manual. The changes in the field interviewer manual were primarily associated with the Interviewer Management System (IMS) that was part of the Basic Field Operating System (BFOS) in the windows-based system. The manual included an additional section with instructions and screenshots for documenting the use of the IMS. It still retained a section on the DOS-based field management system since it was still in use on Panel 10 and 11.



• Field Supervisor Manual. Like the interviewer manual, a section was added to provide instruction on the use of the new Supervisor Management System (SMS). It also included a discussion of the new field reports generated from the SMS.



• Question by Question Specifications. A new set of specifications were developed that included instruction on making entries in the Blaise/WVS questionnaire and provided new screen shots illustrating the questions and paths through the new instrument. The only change made to content was to cover revisions to the Round 1 instrument.



• CAPI Reference Manual. A new CAPI reference manual was written to provide information on the use and care of the new laptop used for the windows-based systems and detailed instructions on data transmission and the new email package. In addition, it covered navigation techniques unique to the new instrument.



• WVS/CAITRAIN. Though this is not a manual, it is an instructional device. It is a self-paced tutorial on the laptop that interviewers used to learn how to navigate in Blaise/WVS. This tutorial is a corporate system that was developed for the first time for use on MEPS.

• Summary of Events. The summary of events is a hard copy document that displays health care utilization reported in a previous round interview. The interviewer showed it to the respondent as an aid to recall and bounding of the reference period for reporting events. This form was included in the hard copy case materials for each household. Due to concerns that lost case materials would not be recovered and the confidential nature of the information on this form, it was no longer provided in the case materials for use in the interview. However, it was still printed and provided to the supervisors who could relay information from the form to the interviewer in preparation for an interview.



• Instructions for reporting lost case materials and stolen laptops. As part of our compliance with the security C&A, interviewers are required each year to read procedures for reporting lost or stolen materials and laptops and sign a receipt indicating they read the material. This procedure takes place at training for new interviewers and is mailed to the existing field staff each year, with new confidentiality pledges to sign and return.



• Laptop Passwords. At the start of each cycle of data collection (Spring and Fall), passwords are changed on all interviewer and supervisor laptops as a safeguard against access to the laptop by an unauthorized user.



• Authorization Forms. Both the pharmacy and medical provider authorization forms were revised to delete an entry area for a social security number. In addition, language changes were made to clarify that revocation of the authorization only applied to data collection that had not yet taken place. The pharmacy authorization form was revised to include language that indicated that the authorization form allowed for collection of prescribed medications related to the treatment of mental health conditions.

Return To Table Of Contents

3.1 Recruiting for 2007

Recruiting for 2007 began in September of 2006 following delivery of the Panel 12 sample. As mentioned earlier, the new sample design for Panel 12 required recruiting in new PSUs and supplementing staff in existing PSUs that were in both the old and new sample designs. The PSUs that were only in the new design had small workloads and many PSUs did not have an adequate number of cases to support a local interviewer. After a detailed review of the location of the new work and the amount of existing work in overlap PSUs, as well as the interviewers who were currently on staff at the time, decisions were made as to where to recruit and how many to recruit. Consideration was also given to the number of talented travelers currently on staff who could also cover new PSUs with small workloads.

There were 153 interviewers recruited and 139 completed the training programs. With the addition of these new trainees, the project began 2007 data collection with a total of 515 interviewers. Of these, 57 worked in PSUs with only Panel 10 Round 5 and Panel 11 Round 3 work and did not attend the windows-based training programs. There were 78 interviewers (15%) who were lost to attrition during the spring interviewing rounds. An additional 47 (11%) of those remaining were lost during the fall round. Total attrition for the year was 24 percent.

Return To Table Of Contents

3.2 2007 Trainings

The conversion to the new windows-based system coupled with the new sample design for Panel 12 resulted in the need for a new training approach. In PSUs in the old and new samples, new interviewers needed to be hired who could interview in the DOS instrument for Panel 10 Round 5 and Panel 11 Round 3 as well as the Windows instrument for Panel 12 Round 1. There were a small number of interviewers that required only a round 1 training on the windows-based applications since they would work in the new PSUs with only round 1 sample. A third group requiring training was the existing MEPS interviewers who needed training only on the windows-based applications.

The conversion to the windows-based platform required training at several levels: field managers and supervisors as well as home office trainers. These sessions were held in December 2006 and are discussed below.

In preparation for the large-scale training effort held in early 2007, all home office and field supervisory staff scheduled to staff the conversion trainings were required to participate in one of two ‘trainer training’ sessions conducted concurrently at the Westat Training and Conference Center in Rockville, MD, December 1-3, 2006.

Field supervisors and managers were trained together, while home office staff comprised the second session. The three day course provided a ‘dry run’ of the experienced interviewer Panel 12 Round 1 windows-based training program and included a review of training logistics, roles, and responsibilities.

Prior to attending ‘trainer training,’ field supervisors and managers received a short, home study memo providing details on the upcoming training, an overview of the Panel 12 Round 1 sample, a highlight of the changes to the round 1 questionnaire, and an introduction to the windows based applications.

Immediately following the ‘trainer training,’ field supervisors and managers remained at Westat to participate in a second, three-day training on the new management system for Panel 12 Round 1. Training included the Supervisor Management System (SMS), Outlook (email), and practice using Microsoft Word and Excel.

Three different interviewer training sessions were held: (1) a "split" training program to teach new interviewers rounds 3 and 5 instruments in the DOS-based system, send them home to work for several weeks, and then bring them back a few weeks later to introduce the windows-based applications for round 1; (2) a round 1-only training on the new windows-based applications for interviewers experienced with the DOS-based system; and (3) a round 1 only training for interviewers working in PSUs that only had round 1 work in Panel 12.

The training for Rounds 3 and 5 in the DOS instrument took place January 10-15 in Los Angeles. Bilingual training was held for Spanish-speaking interviewers on January 16. Of the 114 interviewers attending this training, 109 successfully completed the program. After a few weeks to gain experience completing Round 3/5 interviews in the field, these interviewers returned for Round 1 training in the windows-based applications. This session was held February 1-4. A total of 103 interviewers completed the Round 1 training session.

Two sessions were held to introduce the Round 1 windows-based applications to experienced interviewers; the first was from January 22-25, and the second was from January 28-30. Bilingual trainings for these sessions were held on January 26 and 30, respectively. A total of 319 interviewers were trained in these sessions.

Finally, Round 1 training was held for 39 newly hired interviewers who were working in new PSUs. These interviewers received a modified version of the Round 1 training to accommodate the fact that this group did not have prior experience with Rounds 3 and 5 of the interview in the DOS-based system. Of the 39 trainees, 36 completed the training.

Experienced interviewers received a home study package to review changes in the DOS Round 3/5 instrument. This home study program, modeled on 2006 home study materials, addressed changes both to the DOS-based instrument and to interviewing procedures. Included with the home study package were updates to the question by question (Q by Q) manuals, a glossary, a new job aid booklet, and a laminated flow card.

Prior to their training on the Windows-based instrument, experienced interviewers received another home study package. This discussed the Panel 12 sample, and included an introduction to the windows-based systems, including screenshots.

The fall 2007 rounds of data collection also required training. For the majority of interviewers, this was performed via a home study. Interviewers with no previous experience on Round 2 interviews (those working in new PSUs) were brought to an in-person training at the Westat Home Office in order to introduce Round 2 concepts. This training took place July 30 – August 1 with the remaining 19 interviewers working in new PSUs.

In addition, a "refresher" training was conducted at the same time with a small group of interviewers who had been identified as needing more training with the windows-based instrument. The focus of this training was navigation, but Round 2 concepts and supplements were also reviewed during the session. A total of 14 interviewers attended this session which took place July 30 – 31 at the Westat home office.

Interviewers new to MEPS were sent a home study package consisting of a "What’s New" memo; a Round 2 job aid booklet; new chapters and supplemental pages for the windows-based Q by Q specifications; replacement show cards for items HX-1 and HX-4 (bilingual interviewers received both English and Spanish replacement cards); and a review exercise.

All experienced interviewers (including those attending the refresher training) received the same home study package as the new interviewers. Experienced interviewers also received a booklet on Panel 12 paperwork and materials to complete a scripted mock interview. Field Supervisors determined whether the mock should be completed individually, over the phone with another interviewer, or with the Supervisor.

Return To Table Of Contents

Table 4-1 shows the calendar dates and number of weeks per round in the standardized, "steady state" data collection schedule for the 5 rounds of MEPS household data collection. The data collection schedule has remained essentially unchanged since 2002. There is a two week interval between the end of rounds 1 and 3 and the start of rounds 2 and 4. Rounds 3 and 5 begin in mid-January of each year followed by a February 1 start-up for round 1. The later start of round 1 allows for a minimum 4 week reference period for the first round of MEPS interviews. The fixed schedule for data collection provides a secure anchor for scheduling the related activities that prepare for or immediately follow the data collection, such as the preparation of field materials for subsequent rounds and identification of the sample for the Medical Provider Component.

Table 4-1. Data collection schedule and number of weeks per round of data collection

However, due to response rate issues, the Panel 12 Round 1 data collection period was extended by one week to allow more time for conversion. This delayed the start of Panel 12 Round 2 by one week and shortened the round 2 field period by one week (which maintained its original ending date). More discussion on reasons for extending the round 1 field period is provided in the next section.

Return To Table Of Contents

4.2 Operations

Pre-field activities, including advance letter mail outs, advance contact calls, and assignment material preparation remained unchanged from prior years. Data collection support activities such as home office tracking, disseminating information from the respondent calls to the Alex Scott line, mailing of refusal letters, and so on, were also relatively unchanged from prior years. Procedural changes were kept to a minimum so interviewer tasks were not more burdensome during the conversion year.

Data collection in the spring of 2007 presented new challenges for field interviewers and their supervisors. The challenges stemmed from the conversion to the windows-based instrument and field management system and the introduction of the new PSU sample for Panel 12 Round 1 with the resulting geographic dispersion of the sample.

Interviewers working in PSUs with both the new and old samples were required to interview using two different laptops – one for the DOS-based questionnaire and management system and a second one for the windows-based applications. Interviewers are encouraged to work as efficiently as possible, being prepared to contact as many households as possible in each trip to the field. Thus, interviewers had to be prepared to carry two laptops into the field on most days they were interviewing and they needed to make sure they took the correct laptop to the door when making contact. Interviewers had to report on their contact attempts in two different management systems, each requiring separate transmissions. Similarly, picking up new case assignments and sending completed cases required two separate transmissions, adding time to their reporting tasks.

In order to maintain one system of communication in the field, interviewers who were not working on the new system (i.e., interviewers working in PSUs not continuing in the new sample design) were provided with a second laptop with the same email system used in the windows-based laptop. These interviewers also had the burden of two laptops with two transmissions – one for use with their DOS-based system and one for communication using the new email package.

As described earlier, during the sample preparation for Panel 12, the counties in the new sample were mapped and those that were contiguous to overlap counties (counties in both the new and old samples) were assigned to the same PSU. This resulted in some very large geographic areas covered within a MEPS PSU. Experienced interviewers had acquired extensive knowledge of the location of the segments in the old sample design, but now had to become familiar with new segment locations in different counties, contributing to an increase in the amount of time they spent in the field contacting households.

The new sample design also posed challenges to the supervisors who had to make case assignments in geographic areas they were not familiar with and planning for covering work in very small PSUs without local staff. The supervisors also had two laptops to use for managing the data collection effort, and the Supervisor Management System (SMS) in the windows-based application required on line connectivity to the management database at the home office.

Interviewers went through an adjustment period as they became more familiar with and gained confidence in using the Blaise/WVS instrument and interviewer management system. Although field activities proceeded in much the same manner as in prior years, the interviewers tasks were more challenging, required more time to complete, and interviewers needed more support from their field supervisor and home office.

About 8 weeks before the scheduled end of Panel 12 Round 1 data collection, it was clear that the response rate and production were well behind expectations. AHRQ was kept informed of the problems and a decision was reached to extend the field period for Panel 12 by one week. Although the Panel 12 sample was small in size, its geographic spread made it difficult to adequately work all cases with local staff and the need for travel increased. Panel 10 and 11 work in PSUs that would be leaving the new MEPS sample were also thinly staffed since a decision was made not to replace staff in those PSUs when recruiting for 2007.

Close supervision of the last 6 weeks of the Panel 12 Round 1 field effort was maintained at the home office. Home office staff worked very closely with the field managers to develop plans for raising completion rates and response rates. Phone conferences between the home office staff and the field managers occurred at least twice a week so that plans for travel and use of the best of the field force to complete the work could be coordinated across regions.

Reports were generated that provided the statistics needed to monitor the remaining work at a PSU level. Specifically, each PSU was examined for the number of pending cases by pending status of initial refusal, second refusal, broken appointment, tracking, and other pending, number of NHIS partial completes in the remaining cases, and number and skill level of local interviewing staff. Using all of this information enabled the field managers and home office staff to effectively manage the final weeks of data collection and raise the response rate.

A number of PSUs needed strong, traveling interviewers to help bring up production and conduct refusal conversion to build response rates. Table 4-2 shows the number of interviews completed on travel status for work conducted in spring 2005 through 2007. The table shows a significant increase in the amount of travel in the spring 2007 data collection effort. While the total percentage of interviews completed on travel did not change significantly with the 2007 data collection effort, the percentage of the round 1 cases completed on travel status did increase in a significant way. For Spring 2007, 30.3% of the interviews completed on travel status were round 1 interviews, as compared to 20.2% in Spring 2006 and 18.7% in Spring 2005.

Table 4-2. Percent of total interviews conducted on travel

Return To Table Of Contents

The Medical Provider Component continued to have difficulty securing cooperation from two large pharmacy chains and the procedure for collecting patient profiles from these two pharmacies was folded into the Household Component data collection. Unlike this effort in 2006, the decision to collect the profiles was made before the field period for the Panel 11 Round 4 data collection effort started so the request for profiles could be made at the end of the round 4 interview, instead of mailing the requests later in the field period.

For Panel 11, Round 4 households, letters with instructions and lists of RU members who used the corporate pharmacies were assembled and included in the case folder for each household who reported using one of these pharmacies and for whom authorization forms had been signed. Respondents were told that upon receipt of the patient profile(s), they would be paid $30 for the time and effort made to collect the profile(s).

Since the Panel 10 Round 5 households had completed their last in person MEPS interview, a mail out was organized to send requests to the households reporting use of these pharmacies. The mail out occurred on August 24, 2007 and these households were also told that they would be sent a check for $30 for returning patient profiles.

This approach for collecting patient profiles was more successful than expected. Results of the effort for 2006 and 2007 are shown in Table 4-3. In 2006, when all patient profile requests were made by mail, patient profiles were collected from nearly 13 percent of the households reporting use of Pharmacy 1 and 20% of the households reporting use of Pharmacy 2. This is only slightly lower than the results obtained from the mail request for Panel 10 Round 5 households in 2007 (13% for Pharmacy 1 and 22% for Pharmacy 2.) The gain in response rate came from the increase in collection of profiles when the request was made in person at the end of the Panel 11 Round 4 interview. In person requests resulted in 46 percent of the households providing profiles for Pharmacy 1 and 34 percent of the households providing profiles for Pharmacy 2. Overall, for both in person and mail collection, 30 percent of the households provided profiles from Pharmacy 1 and 29 percent provided them from Pharmacy 2 in 2007.

Table 4-3. Results of patient profile collection for medications prescribed in 2006

Quality control measures received increased attention with the CAPI conversion and new sampled areas. Concerns about the length of the round 1 interview led to an increase in observations by home office staff in order to observe the administration of the new instrument. Validation procedures continued as they have in the past with 20 percent of each panel’s sample pre-selected for validation. In addition, all interviews conducted in less than 30 minutes were validated. Over 31 percent of the finalized cases in Spring 2007 data collection were validated. About 33 percent of the finalized cases in Fall 2007 were validated.

Return To Table Of Contents

Table 4-4 provides an overview of the data collection results, showing sample sizes, average interviewer hours per completed interview, and response rates for Panels 8 through 12. (Table A-2 in Appendix A shows the data collection results for all panels.) In looking at response rates across all rounds of data collection for the past 5 years, with rare exception, the rates have been declining. This observation is not limited to the MEPS survey; it’s a pattern that has occurred across most cross sectional surveys, including the NHIS.

The most notable difference across panels is the hours per complete for Panel 12 Rounds 1 and 2. While the administration time of the Blaise/WVS instruments is significantly higher than the Cheshire instruments, most of the additional hours can be attributed to the new sample design for Panel 12. Many of the PSUs/counties had such small caseloads and interviewers could not build the efficiency needed to keep the hours low. In addition, the cases were geographically dispersed, requiring more driving time. This is substantiated when looking at the number of hours per complete being spent in the current Panel 13 Round 1 and comparing it to Panel 12 Round 1. At week 10 in the round 1 field period, Panel 12 was experiencing 11.3 hours per complete. Panel 13, at week 10, is at 8.8 hours per complete. With the large sample size in Panel 13 combined with the existing Panel 12 work, interviewers can work more efficiently and keep the hours per complete lower.

Table 4-5 shows response rates and the components of nonresponse for round 1 of the five most recent MEPS panels. The refusal rates stayed the same between Panel 11 and 12. The increase in response rate for Panel 12 was the result of fewer cases in the other types of nonresponse category.

As shown in Table 4-6, the Panel 11 round 4 response rate increased by 0.5 percent from Panel 10 Round 4. This change corresponded to a 0.5 percent decline in the refusal rate. In Panel 12 Round 2, the response rate was 0.7 percent lower and the refusal rate was 0.9 percent higher than in Panel 11 Round 2.

Medical provider authorization form signing rates are shown in Table 4-7 for Panels 8 through 12. (Table A-3 in Appendix A shows the signing rates for all panels and rounds to date.) Panel 11 Round 4 had a signing rate of 69.7 percent, the same rate as Panel 10 Round 4. All other rounds worked in 2007 have lower signing rates than corresponding rounds worked in 2006.

Table 4-8 shows signing rates for pharmacy authorization forms for Panels 8 through 11 (Table A-4 in Appendix A shows the signing rates for all panels and rounds to date.) In 2007, the signing rates for these forms for both Panel 10 Round 5 and Panel 11 round 3 also were lower than the previous year’s rates.

Table 4-4. MEPS HC data collection results, Panels 8 through 12

Return To Table Of Contents

Table 4-5. Summary of nonresponse for Round 1, 2003-2007

NOTE: Figures in tables showing results of field work are drawn from the database used to monitor ongoing production and from the ‘delivery’ database, which reflects minor adjustments made in post-data collection processing. This is the source of several discrepancies in totals shown in the tables.

Return To Table Of Contents

Table 4-6. Summary of nonresponse for Rounds 2 and 4, 2004-2007

Return To Table Of Contents

Table 4-7. Signing rates for medical provider authorization forms for Panels 8 through 12

Return To Table Of Contents

Table 4-8. Signing rates for pharmacy authorization forms

Return To Table Of Contents

Signing rates for all authorization forms continue to decline with each panel and each year. In 2007, field supervisors made a concerted effort to increase signing rates. Home office staff provided weekly signing rates to the field and closely monitored signing rates at an interviewer level. However, refusals to sign authorization forms continue to increase.

Table 4-9 shows the results of the Self-Administered Questionnaire (SAQ) data collection. SAQ collection begins in rounds 2 and 4 of a panel, with followup for nonresponse in Rounds 3 and 5. Table 4-9 shows both the round-specific response rate and the combined rate after the followup round was completed. (Table A-5 in Appendix A shows the results of the SAQ collection for all applicable panels and rounds to date.) The combined rates for the first year of Panel 11 and second year of Panel 10 showed decreases in response rates from their counterparts in the prior panels. In 2007, the signing rate for Panel 12 round 2 was 0.6 percent higher than the rate for Panel 11 Round 2.

Table 4-9. Results of self-administered questionnaire (SAQ) collection