Pre-Conference Presentations, Workshops and Panel

IOMW Pre-Conference Presentations, Workshops, and Panel
In 2016 we offer unlimited exposure to a large selection of the most advanced measurement software available.  The extremely modest workshop registration fee gives you complete unrestricted access to:
  • Software presentations (all in the same room), 8:30 am - 12:00 pm
    • RUMM
    • Winsteps
    • Facets
    • ConQuest
    • ERMA (Rasch in the R programming language)
    • Openbugs (Rasch via Bayes and Monte Carlo methods)
    • jMetrik (A full-featured open-source package that includes a full complement of IRT models, as well as Rasch)
    • Damon-on-Python (an open-source Python package for highly multidimensional data)
  • Panel on the Status of Measurement Software, 1 - 3 pm
  • Introductory workshops (90 minutes), 3 - 5 pm
    • RUMM, Winsteps, Facets, ConQuest
  • Round-table tutorials, as needed (90 minutes), 3 - 5 pm
    • ERMA, Openbugs, jMetrik, Damon-on-Python
See the IOMW Conference Program for details.  Pre-register here or show up at the last minute.  Workshop instructors do not need to register for workshops.

When
All day Monday, April 4, 8:30 am - 5 pm
Workshop registrations will start at 8 am, for those who haven't pre-registered.

Where
Fellowship Hall at the main IOMW conference venue

Food
Lunch and break refreshments will be provided

What to Bring
  • Your laptop (wifi will be available).  Many of the packages are built for a Windows platform.  If you have a Mac, the usual thing to do is run Windows on a virtual machine within the mac.  The open-source VirtualBox is a good alternative.
  • Download software that you're interested in before the workshop (see links below).
  • If you want to just watch and learn, that is perfectly okay!
Experience Necessary
None

The Panel on the Status of Measurement Software
In 2016, we're going to try something new.  In the past seven years, there have been extraordinary advances in the areas of data science, big data, machine learning, and the algorithms that are behind measurement.  We want to host a discussion on the status of existing measurement software, identify weak points, and discuss the future:
  • What software libraries are available for measurement, IRT, and related fields?
  • What can we do if the datasets are too large to fit in memory?
  • How do we simulate data to compare the various Rasch, IRT, and multidimensional models?
  • How work measurement software into larger workflows, e.g., for online testing and automatic web reporting?
  • How deal with the patchwork of specialty psychometric programs, e.g., Stocking-Lord equating, classification accuracy, DIF, etc.?
  • How do we standardize language and formula conventions?
  • How do we standardize data formats beyond tabular and facets-style?
  • How support third-party reproducibility of results?
  • How do we increase the coding skills of measurement practitioners?
  • What open-source alternatives are available?
  • What would it take to build a robust open-source community in measurement?
To our knowledge, the community has never had this discussion.  The panel discussion will take place on Monday, April 4, 1 pm.

We have grouped software packages into two groups:
  • Foundational.  Software that has been the foundation of Rasch measurement for decades and with which it is likely you have at least passing familiarity.
  • New and Exotic.  Software that is relatively new or resides on platforms that are new to most users.
The "foundational" software packages have dedicated 90-minute workshops.  The "new and exotic" software packages are presented in the morning to everybody, with the possibility of round-table tutorials in the afternoon.  To set up a round-table tutorial, contact the relevant instructor at lunch.

The Workshops for Foundational Software (start at 3 pm)
*******
ConQuest, Fellowship Hall
Instructors:  Rebecca Freund

ACER ConQuest 4 is a computer program for fitting both unidimensional and multidimensional item response and latent regression models.  It provides data analysis based on a comprehensive and flexible range of item response models using the framework of the Multidimensional Random Coefficients Multinomial Logit model (MRCML; Adams, Wilson, & Wang, 1997).  ConQuest is an essential component of the modern psychometrician's toolbox.  

This workshop will consist of:
(1) Introduction to ConQuest: Fitting familiar Rasch-family models (including the Rasch, Partial Credit, Rating Scale, and Rater-effects models); the basics of ConQuest syntax and output
(2) The power of ConQuest: the MRCML and design matrices; importing user-written design matrices to run non-standard or newly created models
(3) Multiple latent variables: Fitting multidimensional, longitudinal, and testlet models
The workshop is designed to engage both novices and seasoned ConQuest veterans in a more sophisticated understanding and use of ConQuest.   

Participants are encouraged (but not required) to bring a laptop with ConQuest installed; a free 30-day trial version with full functionality can be obtained from the ACER ConQuest website (http://www.acer.edu.au/conquest). 

*******
RUMM, Room B102
Instructor:  Professor David Andrich, developer of RUMM

RUMM is a comprehensive item analysis package for the polytomous Rasch model with the partial credit, rating and dichotomous parameterisation as special cases, including multiple choice items and up to a two-way facet designs on items. It allows rapid interaction procedures through its user friendly design and layout.  It is a fully operational Windows-based application which means that the creation of Projects, the running of analyses, and the manipulation of screen displays are all facilitated using simple mouse clicks on the range of easy-to-use Windows objects. All information is displayed both graphically and in table format. The tables can be saved or copied by highlighting and pasted directly into EXCEL for further analyses. The graphs can be saved and then edited in Paint ready for publications. Successive analyses can be built on each other in which items can be deleted, rescored, anchored, combined into subsets, split by group, subgroups of persons (e.g. by gender) can be deleted and a tailored analysis conducted. Facilities include for multiple choice items distractor characteristic curves, and for polytomous items category characteristic curves and expected value curves with observed means in up to 10 class intervals, graphical and analytic DIF analyses and information on groups (e.g. gender), principal component of residuals and other tests of person and item fit, person and item/threshold distributions, and equating functions for subsets of items from a single analysis.

*******
Winsteps, Room B103-A
Instructor:  William Boone, Ph.D., author of Rasch Analysis in the Human Sciences

This workshop will provide an introduction to Winsteps Rasch Software (Linacre, 2014). The workshop will provide attendees with an overview of the capabilities of Winsteps. This will be done through an analysis of i) a rating scale data set, ii) a dichotomous data set from a multiple choice test, iii) data from a partial credit test and iv) ranking data. Attendees will learn 1) about the wide range of user friendly and easily interpreted Winsteps output tables, 2)  how data files can be quickly  imported for a Winsteps analysis, 3) the steps to output Winsteps results (e.g. person measures) to data bases and statistical packages, 4) how to conduct detailed respondent diagnostics with Winsteps, 5) how Winsteps facilitates the exploration of multidimensionality through principal component analysis of residuals 6) how Winsteps allows one to easily fine tune an analysis,  7) how to utilize the detailed Winsteps Manual (600+ pages), 8) how to use Rasch Analysis in the Human Sciences (Boone, Staver & Yale, Springer 2014) to guide your Winsteps Rasch analysis. And- most importantly- 8) how Winsteps tables facilitates communication of your results to non-Rasch stakeholders/decision makers/colleagues!

If possible attendees are asked to download the free program Ministeps from the Winsteps WWW. Ministeps allows the analysis of up to 75 respondents and 25 items. The program is identical to Winsteps, but with a limit as to the number of respondents and items which can be evaluated. The presenter will provide the data sets needed to explore and appreciate the power and user friendly aspects of Winsteps. Attendee's are encouraged to attend with and without Ministeps on a laptop. The presenter has utilized Winsteps (and Winsteps pre Windows predecessor) for over 20 years.

*******
Facets, B103-B
Instructor:  John Michael Linacre, Ph.D. (creator of Facets and Winsteps)

A 90-minute Facets workshop utilizing real-world data will be presented to provide participants with a full introduction into the theory, application and interpretation of judge/rater mediated data.  These data sources have been historically analyzed using classical statistical methodologies without regard to the underlying interaction of judge/rater severity, item fit, or person ability.  Each of these three critical elements may be scrutinized under the FACETS model and provide a unique opportunity to communicate inter-rater reliability in a more meaningful way.  This presentation will begin with theory and move through a direct application, including introduction to the software and programming, and will conclude with the interpretation of data as well as recommended applications.

Information on New and Exotic Software (presentations and round-tables)
*******
ERMA (Everyone's Rasch Measurement Analyzer)
Instructor:  George Engelhard, Jr. and Jue Wang (The University of Georgia)

ERMA was created using R. As an R program, ERMA provides free and open source code. ERMA uses a pairwise algorithm for calibrating item parameters, and maximum likelihood estimation for person measures. The results of ERMA have been validated with other Rasch software. The simple structure of ERMA makes it a useful teaching tool for users to gain an understanding of Rasch measurement theory, and also provides the opportunity to modify the program based on their needs. In addition to the traditional R program, ERMA also is available as a ShinyApp. ShinyApp is supported by RStudio, and it is a web application framework for R programs. ShinyERMA provides a user-friendly interface. ShinyERMA uses click-and-go buttons, and provides major outputs found in typical Rasch analyses. The development of ERMA program is still in progress. We plan to extend its use for wider applications in the future, such as rater-mediated assessments. The ERMA and ShinyERMA programs are available upon request at the conference.

*******
Openbugs
Instructor:  Hong Jiao

BUGS is a software package for performing Bayesian inference Using Gibbs Sampling. The user specifies a statistical model, of (almost) arbitrary complexity, by simply stating the relationships between related variables. The software includes an ‘expert system’, which determines an appropriate MCMC (Markov chain Monte Carlo) scheme (based on the Gibbs sampler) for analysing the specified model. The user then controls the execution of the scheme and is free to choose from a wide range of output types.

*******
jMetrik
Instructor:  Patrick Meyer

jMetrik is a free and open source computer program for psychometric analysis. It features a user-friendly interface, integrated database, and a variety of statistical procedures and charts. The interface is intuitive and easy to learn. It also scales to the experience of the user. New users can quickly learn to implement psychometric procedures though point-and-click menus. Experienced users can take advantage of the jMetrik command structure and write command files for executing an analysis.
 
Psychometric methods include classical item analysis, reliability estimation, test scaling, differential item functioning, nonparametric item response theory, Rasch measurement models, item response models (e.g. 3PL, 4PL, GPCM), and item response theory linking and equating. Statistical methods available in jMetrik include frequencies, correlations, descriptive statistics and a variety of graphs. New methods are added to each new version of the program. The integrated nature of the software facilitates comprehensive psychometric analysis and eliminates human errors that occur when using multiple programs, each with different syntax, for conducting an analysis.

*******
Damon on Python
Instructor:  Mark H. Moulton, Ph.D.

Say you have 30 Rasch analyses to do.  Each dataset needs to be formatted.  The results need to be handled in some special way.  In the old days, I would prepare each dataset manually in Excel, maybe SPSS, configure and run each analysis, and do a lot of formatting (and grumbling) in Word and Excel.  That's a lot of back and forth between software programs.  It would take me days and there were always errors.  I dreaded having to redo all my work.  That's when I decided there had to be a way to automate it all.

With Damon I can.  Written on top of the beautiful and all-powerful Python scripting language, Damon is an open-source cross-platform way to write top-level scripts that handle all steps of a psychometric analysis.  It 
  • loads and formats complex datasets
  • does a really good job linking person and item labels to data (no keeping track of row/column numbers)
  • scores, preps, and edits the data to make it analyzable
  •  computes all essential statistics, and then some
  • automatically handles item anchoring and banking across datasets
  • allows me to generate and export results in any table and chart I want, formatted the way I want it
  • makes it really easy to simulate most types of data for doing simulation studies under controlled conditions
Whatever Damon cannot do, Python can.  Say, for instance, say I want to create a Rasch-based web application, something that teachers can use to analyze their tests.  I would run Damon on a back-end server and have it receive data and return results via a website that I create using a Python (or some other) web-framework.  All in one script.  In fact, my company (Educational Data Systems) has offered, through its partners, a Damon-based service just like this for years.

The most amazing thing about Damon is that, while it is fine for regular Rasch modeling, it is built for highly multidimensional data, the kind of data that increasingly dominates the business world (movie ratings, dating profiles, image recognition).  Built on NOUS (see website), it calculates measures and predictions for each cell, item, and subscale in a dataset.  Most important, where data fit NOUS, they automatically meet Rasch criteria for sample independence and reproducibility.  NOUS requires objectivity as a condition of fit.

No software downloads are necessary to use Damon.  Participants can run Damon as a free web application accessed through their preferred web browser.  It works the same on Windows, Mac, and Linux machines.





Comments