Learning to be an NRS Data Detective

What is a data detective? Data detectives are adult education teachers and administrators who explore their data with an eye toward enhancing data quality and improving policies and education services. The training explores how an effective data detective:

• Understands the process and psychology of data collection;
• Examines data including assessment, goal setting, and follow-up data;
• Asks further questions of the data; and
• Makes changes in processes and policies to improve data quality and program services.

 

What are the keys to being a data detective? States can implement their own models of data collection and analysis, but successful data detectives rely upon the following core elements:

• A series of regimented procedures and policies for data collection and analysis (e.g., set times during which specific data tables are reviewed),
• A thorough and efficient data system, and
• A culture of data exploration and action based on findings.

Data detective work means keeping an open mind, testing hypotheses, and using data in context to make a judgment. For example, if data show programs are posttesting only a very small percentage of students, then a problem is identified but further detective work is need to determine the cause before a solution can be found. Is the problem due to students not staying long enough or with the program’s testing procedures? Is the problem limited to a few sites or is it widespread? Thoughtful qualitative analysis is needed to answer questions like these.

 

Data Detective Training

In the summer of 2006, 47 states and territories attended the NRS training titled, Learning to be an NRS Data Detective: The Five Sides of the NRS.  Several states have used the guide, materials, and tools from the 2006 summer training on Learning to be an NRS Data Detective in their own states. Below are examples of professional development training guides and materials that have been developed in Maryland and South Dakota with descriptions of how those materials were implemented. Examples from Tennessee are coming soon.

 

Maryland – Alta Weiss adapted the NRS Data Detective training materials to develop a training activity for Maryland’s program administrators. This training was presented at their fall 2006 meeting. Alta introduced the activity by using a PowerPoint presentation and reviewing the activity’s directions with everyone. Alta used real FY 02 through FY 06 federal data from five unidentified local programs and the state. She selected five programs that would demonstrate a variety of size, demographics, outcomes, anomalies, etc. For analysis of the trend data for the 5 programs she divided Maryland’s 34 program administrators into 5 groups according to similar program enrollment size, and the state staff formed a separate group to review the state data. After a designated time period each group reported its findings to the entire group.

Alta reported: “This training built upon previous trainings we have done with our administrators about data analysis. The primary objective was to have administrators analyze and offer explanations for data trends over time and to begin thinking about using trend data analysis for program improvement. I also wanted them to see how data can be graphically displayed to aid analysis and to motivate their interest in program data. As intended, the charts were the carrot and definitely got their attention! They were very pleased when I announced that later in the year we would be training each program’s data entry specialist how to create trend and annual charts for data from the federal level down to the teacher level.”

“I have continued to use the great NRS resources and my NRS training to design state training activities and to support training activities for programs at the local level. Thanks!”

The PowerPoint presentation that introduced the activity and the two handouts that were used in the Maryland program administrators’ training are available for download.

 

• MD Data Detective Guide
• MD Data Detectives Exercise
• MD Data Detectives Exercise - Charts

 

South Dakota – Following the NRS Data Detective training, Marcia Hess, the state director in South Dakota, rolled out a state-specific training to state and local program staff on using data. Marcia created a workbook with state data across 3–4 years and added a few questions about cost efficiency and cost per educational gain (Note: fictitious data was included for some questions to generate a graph and those are marked with an asterisk). She removed the names of the programs and mixed up the order and then used different groups to illustrate each item. As she notes, “I wanted to have them look at the data and not make criticisms or competitive comparisons about the data. After they got over the shock that I was not going to tell them which bar or number they were, they buckled down and really got into the analysis of the data.”

Marcia notes that the local program directors were “…highly engaged. The discussions of possible causes for the different results were very good.  Often they thought the data was theirs but many times it was not. We broke into small groups of four to discuss the graphs then back to the large group for discussion. When they looked at the information as if it were multiple class sites instead of the state it became even more objective and spot on with the reasons for the results.”

In addition to the workbook, Marcia also drafted a training manual that provides basic information about using data at the state and local level and details on how to use the workbook. It also provides specific notes to facilitators with prompts and ideas for helping local staff to analyze their data

Marcia commented that after the training session, there was a greater shift toward managed enrollment. She notes, “It took a great deal of time to do this project but it was well worth it. I hope to repeat this training with the new directors I have this year since that training.”

Marcia shared with us a draft Workbook and Facilitator’s Guide:

 

• SD Local Program Workbook
• SD Training Guide for Learning to be a Data Detective

 

Tennessee – In Tennessee, Phyllis Pardue and her staff developed a web-based system for their 95 local grantees to enter student data on a weekly or even daily basis.  This Oracle-based system called CMATS provides users with 33 customizable reports that local programs can run to examine their own data for program improvement. 

Local programs in Tennessee that find their numbers reflect poor program quality request assistance from the state to include on-site or more in-depth assistance in finding errors, as well as help in designing efficient data entry procedures and using reports to improve data quality.  While the state contracts with the Center for Literacy Studies to provide state-wide technical support and training for the OMS, there is insufficient budget for Center personnel to travel around the state for such in-depth, on-site support and investigation.

So, beginning in May 2006, Phyllis’ team identified experienced CMATS data entry folks from across the state.  They created a network of roughly 30 folks to provide on-site field support in data entry procedures and quality for local programs.  In August 2006, they trained this group of 30 in data investigation techniques, provided them with a reporting structure, and gave them in-depth information about data collection and entry procedures in CMATS.  So far, four programs were identified by the state to make use of the on-site field support staff, who began site visits in October 2006. 

In the NRS Webinar held on February 8, 2007, Phyllis reported on what the state staff learned about local program investigative techniques, as well as the success of the field support system approach to data quality improvement.  The goal of their on-site field support network is to improve data quality locally and increase the data investigation skills of all programs across the state.  With better data, and a better understanding of what the data "describes," programs (and  state staff) can make more accurate and effective program improvement decisions.

• Coming soon: a downloadable overview of the CMATS Support Network proposal - a more in depth description and schedule for the implementation of the program.