Friday, July 17, 2009

Measuring open education - how we value it?

I really enjoy meeting with Russell and Shelagh at the University. I find their conversation stimulating, and I'm excited at the direction they are leading the project to 'measure' Otago Poly's open education work. This is our second meeting, where in the first we agreed to gather data on usage and value. This meeting was to discuss how we would go about measuring value.

Over the past week, I have been gathering usage data such as numbers of views of a resource, responses, and reuses, as well as file sizes with a view to trying to assertain how much we are saving (or not) by using web based services like Youtube and Wikieducator instead of our own. Reviewing this work in today's meeting, we realised that we need to do more of this sort of comparative analysis between "closed" and "open" educational resources and practices - if we want to be able to make recommendations about our open education.

So far I am working with the following questions for the usage analysis:

This is a measure of the usage of OER which includes videos, images, audio files, slide presentations, course outlines, lesson plans, assessments outlines, and other documents based on statistical records maintained by the content management systems that host and serve the OERs, as well as external measurement tools such as Google Analytics and Alexa. It is also a measure of the worth of those content management systems based on how much it would have cost the Polytechnic to set up and run equivalent systems.
  1. How many OER are online and openly accessible?
  2. How long has the OER been online?
  3. What proportion of a course's resources are open?
  4. How many enrolled students are in a course, and how many nominal learning hours were on OER?
  5. How many views or downloads has each OER had?
  6. How many comments, ratings, and other responses has each OER had?
  7. What is the individual and collective data size of the OERs?
  8. How much would it have cost the Polytechnic to serve the amount of data multiplied by the number of views of the OER?
  9. How much would it cost the Polytechnic to install and manage the content management systems currently used for the OER?
  10. What is the worth of the marketing gains based on the rate of views and responses to the OER?
  11. How many OERs has the Polytechnic sampled and reused in its own OER, and what is the financial worth of that reuse?
Todays meeting was really to start pinning down the harder measure though - that of how our convenience sample value their work in open education. Both Shelagh and Russell are willing and able to video interview staff while in the process of doing what they do in their open education work. "Lived experience" video as Shelagh calls it. From those interviews, we will identify key themes based on what is said and not said - the not-said stuff being what Russell calls "white data", or what Shelagh calls "messy data". From that white and messy stuff .. we might be able to determine levels of understanding or perceived importance etc.

We will likely segment what people say and do in the interviews into categories such as behavioral categories like producers, prosumers and consumers, to see if there is any correlation in what we find and what other research that Shelagh referred to in the marketing sector. This also consciously avoids problematic segmentation such as teachers and students.

It has been extremely interesting for me to explain the purpose and objective of the project (flavoured with my usual opinions on how and why) and to have Shelagh and Russell consider it from their research area perspectives - highlighting areas I hadn't considered before, or seen anywhere else in the educational development writing. I'm confident that because Shelagh works in marketing and Russell in learning (as apposed to teaching and education) this project is going to develop methods and incites quite new to the field.

My job now is to seek ethics approval from our Polytechnic's Research Ethics Committee, and gather up all the forms for us to present a method for ethical consideration.

I know now that we will not be able to complete the study before Vancouver's Open Education conference, but I will have usage data to present, and a method for assessing the value - maybe even a few interview samples to show. I expect we will have a report and recommendations ready in time for the paper to IRRODL however. I'm so excited by the possible descoveries in this project that I don't want to rush it.

Russell, Shelagh and I are meeting each Friday afternoon from now on, and we'll hopefully be recording interviews 2-3 weeks from now.

Creative Commons Licence

This work is licensed under a Creative Commons (Attribution) license.


flash said...

Good to know about this ....

Term papers

Term Papers said...

It’s glad to see good information being convey. Thanks for the info.