Tuesday, September 15, 2009

Measuring Open Education. Finishing stage 1: Usage

I think I can say that:
one teacher engaging in open educational practices (at Otago Polytechnic) using popular social media platforms, generates $5011 per year worth of savings and gains to their supporting oranisation. It costs $3000 to adequately train and equip such a person to a suitable level so as to be able to generate this sort of return.
Anyone care to check my math and reasoning on that?

My sickness, absence last month, and the unanticipated complications with ethics approval has delayed stage 2 of the project - to conduct an ethnographic evaluation of people engaged in the production and publishing of educational resources at the Polytechnic. While we try and sort out an approach to ethics, I have been focusing on finishing this usage evaluation.. trying to get an all too simple dollar value on the open ed work.

  • I have selected a sample of open educational resources produced by Otago Poltechnic staff, including resources on Wikieducator, Youtube, Slideshare and Blogs.
  • I have estimated the cost of producing those resources based on a low royalty fee for using a single image, how long it takes to produce a 1 page handout, 10 slide presentation, a 3 minute video. That goes down as a cost or production (although some argue the production of educational resources is part of teaching work).
  • Any free-for-reuse media used in the resources is subtracted from the cost of production based on the same estimate above. That goes down as a saving in production cost.
  • I have asked other institutions to estimate how much it costs to set up and maintain their own Wiki, media sharing site, and a blogging platform. That goes down as a saving in IT support.
  • I asked the marketing unit how much they spend on a billboard and a news paper ad, divided that by how many views they think that gets, to get a value per view. That value is multiplied by how many views the educational resources have. That result determines the worth of open educational resources in terms of simple brand awareness marketing. That goes down as a gain in brand awareness.
  • I estimated the training and technical resourcing of one staff member to produce and publish open educational resources. That goes down as a cost of production.
  • I have not put a unit of measure to things like learning outcomes, job satisfaction, responses as apposed to straight views of a resource.
  • I am still waiting on costings to serve Internet data out from, and into the Polytechnic. This would go down as either a cost or saving, depending on the location of the person who is viewing the resources.
Beth Kanter recently published a presentation on calculating ROI (Return of Investment). I hadn't considered that was what I was doing before, but of course! Beth's presentation offers some food for though on how I might go about measuring some more aspects to this question.

I hope to be able to argue that at the very least, supporting teachers who produce and publish open educational resources on popular social media sites is good for the organisation at a very basic operational level. I might even go as far as to argue that these dollars should be turned into incentives and rewards for those engaging in such practice, and special support for improving the quality and design of some of the more popular resources that have been generated. Another angle is to suggest to the marketing unit that open educational resources on popular social media platforms are a valuable tool for brand awareness marketing (at least) and that they might consider a future marketing budget that compliments the production of educational media instead of (or as well as) billboards and news papers ads.


computingforsustainability said...

Looking good Leigh. I'm wondering if there needs to be a factor for the novelty of resources - reflecting the effort required to make them. If I'm making the equivalent of an introductory text, the content might flow off the keyboard more smoothly than that to support learning that was closer to the cutting edge.
Also, I think the server costs are inflated. I think yours is an average total cost model whereas it should perhaps be a marginal cost (we are running a wiki off the back end of a server that is doing something else, so far it has cost us about nothing).

computingforsustainability said...

Another thought. The maths looks pretty good until the end. You're loading the entire cost of the server saving onto the output of the 3 staff. eg Blogging saving $15K for three staff - whereas in reality they wouldn't get to spend that money if it was just them using it. This should perhaps be divided by a few 1000.

Similarly for the free images, we would avoid using an image rather than paying what you quote.

On the other side of the equation, many of those views would be from "students" so would be difficult to see the value of the brand (along the lines of a logo inside a classroom).

So I think the $7145 gained per staff member is perhaps $7K too high.

Leigh Blackall said...

Thanks CfS, I should anticipate this sort of response in the write up and presentation. I wonder how I might do it in the numbers. How can I preempt this argument, for every point you make there are points I have left off in the other direction, such as the views of an educational resource are obviously worth much more than the number of cars driving past a blue billboard with a log and a phone number on it. For one, they are actual people clicking and looking. But I had no other way to measure it than by brand awareness.. even when the effort in branding open educational resources has been almost none.

The server costs are "true" costs based on how much other institutions have paid to set up the equivalent. That's the necessary staffing to install and maintain it, to learn and control the system and the extensions, backup costs and servers etc.. the old total cost of ownership stuff.

It would be a little ironic to have to defend these costs the other way around now.. but I guess that's how the game goes.

An alternative and easier way to consider it perhaps: Otago Poly pays a provider ~15 grand to run Moodle for us, but I could set up and run moodle on a server for "next to nothing" also. Not sure how much it would be for Mahara, the eporrtfolio/blogging platform being trialed now, but same same.

These server costs are fixed costs no matter how many people use it. We pay ~15G for Moodle, and we have maybe a dozen staff using it... would be the same if we had all 400 staff using it. The early adopters should get credit for their risk taking I reckon. The sample of 13 staff publishing open educational resources on popular platforms represents pretty close to the total number of staff actually..

What you point out is understandable/predictable interpretation of the numbers.. I need to work out a way of expressing them that preempt some of that... I need to account for the value of educational resources over billboards and news paper ads, factor in the value that marketing put on International brand awareness over local, and anticipate the questioning of server costs.

Thanks. More?

Sarah Stewart said...

There's also the stuff that educators do outside work time...eg I blog a lot about things that I reference back to OP in my own time...so in fact I am gifting 'money' to OP in terms of branding etc...eg I am currently working my way through 2 OP courses & blogging about both of them...how much of a monetary value would go on that?

Kerrie Smith said...

It is hard to get marketing people to see that consistent blogging is probably as effective in getting a name/product/capability before educators as all the money spent in bumpf. All they can see is that the blog is not naming the products, although it is demonstrating the interest and capability.

DaveB said...

oohhh kay.

That sounds really cool - so I can run a server out of my bedroom and provide services like Moodle for almost zero cost to me - then charge institutions $15k which is almost all profit. I like it!!

Why am I working for a living when there's money like that to be?

And then maybe I can bank the money that I save by not paying for graphics that I would never buy.

I should be able to retire by lunchtime.

Wikinomics FTW!!

Leigh Blackall said...

Good on you Dave, I knew you'd show up in true Bremmer form.

Go ask Catalyst about their business model. They charge Otago somewhere near that amount for Moodle. It was only a year or so ago we paid more then that on staff to run Blackboard, and then we paid a license fee on top.

If you read the draft that I'm asking for help on, then you'd see the server costs are based on equivalence in other institutions, such as Uni of Cantebury's UCTV set up, and Wikieducator's costing for running, and Sydney and Illawara Institute's of TAFE for their wiki and blogging platform set up and running costs.

The $20 royalty fee for images, and the $50 for audio is a comparative measure to determine the worth of improvements to a resource when a teacher samples them! Its not a saving as you and Sam seem to think I mean, its a gain. Through staff using free-for-reuse images and audio etc, they have produced resources that are slightly - sometimes significantly enhanced without offending copyright, enabling the polytech to publish and reuse the resource widely. That's a lot better then a text resource without images or graphics, or a video with a restricted audio track that shouldn't have been used. You're confused by what I'm trying to give a value for, I'll go back and try and make that clearer.

Sam, I took your advice and removed the server costs from the savings and gains of the sample group, putting it out on its own as a general fixed saving/gain, along with the training and support costs to get staff effective with social media and open education.

You're right, it reduces the sample down to a figure of no sig difference.. or to a point where the investment in open ed pays for itself. But now I'm wondering if its fare to include the cost of producing slide shows and pages of text when that is arguably standard teacher production work and not unique to OER. I think the videos are a step up, although the chefs say producing a video is very easy for them now - perhaps as easy as preparing a slide show. Considering that I am am already including the costs of training people to a level where it is efficient to produce OER.. is it right that I include the production as a cost if it can be considered normal teaching practice?

DaveB said...

And in true Blakall form the point is completely missed. Yes I know that we pay $15K for Moodle (which is rubbish compared to Blackboard).

My sarcasm was directed at your implication that you could run the service for almost no cost.

The server hosting business is quite competitive and international in scale. There's money to be made but it's regular profit - not anything like you're implying. There are reasons for charges like that.

Typical Web2.0 snake-oil economics. Who's paying for it?

Leigh Blackall said...

The only snake oil here is you and your effort to NOT help improve it, but to make snarky remarks. We work for the same organisation, why not make constructive criticism Dave? You never have, never will. Thanks.

Michael Kaisaris said...

In response to some of the critical comments above: the moodle hosting game is a bit of a racket, but it is a racket that schools feel comfortable with. I imagine that school admins feel good when the problem of providing service for their infrastructure is externalized. They make room for the moodle in a budget. It feels substantial, like an order for new library shelving, or a new urinals for the mens washrooms. And like those things, they don't have to think about the cms once it has been implemented because someone is there (not paid university staff) to maintain it.
My schools is just myself, and three other teachers. We don't need that sort of mindset. If we can maintain something useful with our own labour and occasional advice from more expert friends, then that is what we will do.
I just spent the last two days trying to compare mediawiki and drupal as potential replacements for moodle, which I am not a fan of. I can host either one for less than $200cdn per year, just like moodle, but both are more flexible pedigogically, more collaborative, and easier to learn. I don't know which one we will end up choosing, but I feel heartened to know that, just like moodle, we will be able to learn them, and administer them in our spare time.
Why not spend the 15k on teaching resources that would help instructors and their graduate student minions admininister their own moodles, or wikis, or whatever. They already keep their own blogs, their own facebook pages, their own flicker albums, I could go on.
Beyond that, have an otago poly account with dream host and let them have at with all the one-click-installs on their servers.
Cost would be $120/year/teacher, The result would be a massive asymmetrical presence for the school, and personalized teaching environments for each teacher.
Some wouldn't buy in of course, more would get what they want more of the time.
I don't know if this ramble is at all useful, but I know from my own experience that getting what you want is totally incompatible with out of the box solutions, and that compared with those solutions, getting what you want is dead cheap. Couple asymmetry and personalization with open ed resources and the savings would be substantial. The only question that remains in my mind, is would conservative school admins buy in.

Mark Smithers said...

I've come in late on this so I'm not across all of the arguments. I would make the point that my university (50,000 student bodies) is looking at moving blogs and wikis onto virtal servers provided externally that will allow us to pay for only the resources used. The cost per blog/wiki will fall considerably. We would be looking at $7 or $8 per blog/wiki per year. We have about 2000 blogs at the moment but I'm not sure how many are for staff and how many for students. In terms of an LMS - we host BlackBoard internally but we may go outside in future. Roughly we are looking at $10 per student body per year (less per equivalent body). In staff terms it would be about $200 per academic staff member.

The above are hosting costs. We provide centralised technical support for these services. At a rough guess I would say $200 per academic staff member per year.

The other point I would make is that there are some figures available from MIT on their opencourseware program. Are those relevant at all? I don't have a link handy I'm afraid.

DaveB said...

This comment has been removed by the author.

DaveB said...

Oh I'm sorry - I've got the whole service relationship all wrong. As a lecturer trying to develop courses and material for my students, *I* am supposed to be providing resources to your projects. And here I was expecting central departments to be checking out what I was doing and to be providing a service to meet MY needs.

I am so very sorry Leigh.

It is interesting that you are so completely unaware of the various things I've been up to. Just little bits here and there, seldom with a fanfare - but there have been times when I've been pushing teacher-tube and some other things. Your ignorance of my activities in areas which involve free delivery and my use of open source in teaching doesn't surprise me as it doesn't involve something that you're trying to lead.

But lets get back to calculating the cost of blog/wiki shall we? And whether there's anything substantive in the roi.
[replace to fix reversed words]

Leigh Blackall said...

@Michael, thanks for your comments. In the evaluation, I'm not actually looking at Moodle, that came up in this discussion as a comparison to the costing I have for the blogging, wiki and multi media platforms. Those costs I have are based on estimates in time and spending by colleagues working in other institutions that have set up and maintain those platforms.

@ Mark, thanks for that detail.. I've taken CfS's suggestion and moved the savings of hosting costs and put them out as a general savings. So now they are not calculated as a benefit of staff publishing their materials openly and on popular platforms.

computingforsustainability said...

Is getting better but I'm not convinced by the free images in the equation. For the wiki example you say it is worth $29118.60 of brand awareness (OK). But then you add "By using freely licensed images and graphics, these 5 staff have improved the quality of their resources to the worth of $10400....$39519". You consider the cost of production separately ($25000) giving net gain of $14519 (or $4840 per staff member).

If run a coffee shop and start making mocha then you might sell each one for $5. The cost of production is $2 giving net profit of $3. It makes little sense to then say, "but we're using crayons which are free instead of chocolate which would have cost $1.50 so that's improving the quality of the mocha by $1.50 so therefore the net gain per cup is (5+1.5)-2 = $4.50.

Ignore the taste difference in chocolate versus crayons, (I had started the analogy so finished it), the point is that you are adding a new total cost and then trying reducing this by saying that it could have been more. You could make it even more impressive by saying that we would need to hire a photographer and professional models for every image needed costing $10,000 per image and therefore OER is generating truly ridiculous amounts.

I suggest taking the free images out of the equation. The brand awareness alone shows the worth of OER. I think you'd be better identifying other benefits: researcher reputation, networks, funding success, radical transparency etc.

steven parker said...

This book gives a good insight into the services proper admin of "Free" systems like Moodle involve, not cheap in terms of man hours


Leigh Blackall said...

@CfS, so sorry I didn't respond to your last comment. Somehow I missed the alert, and it's not until I have come to review the project much later that I found your suggestion. Sorry, I hope you didn't think I was ignoring your input. It is very helpful, thanks.

I'm still thinking about your argument here though.. the analogy is a bit distracting for me. If I have it right, basically you are saying that we already have a resource that costs $100, replacing part of it (chocolate) with something else (crayon) doesn't improve the resource or reduce the cost of it.

I could be missing the point you are trying to make, put here would be my response:

* The chefs have a text only resource they hand out.
* They want to add images to assist the understanding of the resource.
* They could produce the images themselves (cost)
* Use restricted images in a restricted way (cost)
* Buy images and pay a royalty (cost)
* Use free images licensed CCBy (saving, and gain)

Whatever they choose, the addition of images improves the quality of the resource (or not, but lets give the teachers the benefit of the doubt).

By using free images, the use is unrestricted - adding a quality of flexibility (such as use in marketing, or on the open internet, or for other purposes that a restricted image may prohibit). All that is a gain.

How do you measure the worth of these quality gains and savings? Well, I chose to take the lowest royalty fee of a commercial image.

Is that fair enough? Or have I missed your point?

Leigh Blackall said...

BTW, the text in this post is out of date with the wiki. I will update this post when the 2nd stage of the research is finished.

Leigh Blackall said...

@marksmithers Thanks for this input. I may include it in the comparisons to other institution's costs. Its a real shame these figures aren't made openly available by institutions. Unfortunately, their choice to go with commercial operators who impose non-disclosure agreements compromises their (moral) obligation of transparency to the tax payers (shareholders) who fund the investment... another cost that can't really be accounted for.

Leigh Blackall said...

@SarahStewart, good point.. I would have liked to have added the gains of staff blogging.. I have been told by many people in Wellington that my own blogging helped put Otago Polytechnic squarely in their radars, but again - a gain that is hard to quantify.

Leigh Blackall said...

@MichaelKaisaris your point I think is the same as I make for the use of freely available popular internet services. BlueHost could be considered a (not free) but certainly popular internet service, that offers considerable savings over most our local options. The same argument was made for Moodle when it was being used to challenge the use of Blackboard and the like. Accounting for the skilled human resources was and still is an issue to many in management. No dounbt the model that you run on will become the norm as more and more people in the sector become skilled and experienced with alternataives to the central, outsourced and 'gauranteed' services. But we're not there yet. My own hope is that the proposal I will make out of this project is that we use marketing and other budgets to train teachers in digital literacy that compensates the teacher for their time rather than simply 'requiring' them to turn up in their own time. Releasing them from normal duties doesn't cut it either. I think such an investment ultimately empowers a teacher to make better choices, including the option to take out a BlueHost account and install and manage their own Moodle, Wiki etc. Personally I think that would be a bad idea, but hey.. their informed choice. At the moment, we really invest nothing in developing critical digital literacy in our teaching staff. Another cost or gain not being measured in this project...

Leigh Blackall said...

I've just been pointed to this presentation that set the LMS cost at $30 - $100 per student continuously.