Open Source Courseware -- Evaluation and Rating

By: Rob Reynolds
Category: Emerging Technologies
Date: 04.25.03 4:16pm

Introduction

The cost of doing business is going up for colleges and universities, particularly when it comes to course management systems. Proprietary enterprise solutions for course management -- BlackBoard, WebCT, eCollege -- are beginning to cost the same as other enterprise solutions. Translation -- they're getting very expensive. This rise in cost, along with the traditionally closed architecture of such systems has lead some universities and organizations to develop in-house programs tailored to their specific pedagogical needs and development resources.

Some of these home-grown solutions have been heavily influenced by the aggressive open source movement and have adopted both development and licensing strategies that allow other to adopt the software for free, or participate in the development process. The purpose of this paper is to outline the major open source courseware projects and to rate their usefulness. This is not intended as a definitive ranking but rather as a point of departure for those interested in investigating these systems and who are considering an open source courseware implementation.

Things to Consider -- Let's Get Real Honest

If you're really serious about open source courseware, there are some factors and facts you need to consider carefully. Depending on your needs and circumstances, an open source implementation can either be the perfect solution or your worst nightmare. Personally, I am an open source advocate, but I am also a pragmatist. Some groups are served better by off-the-shelf, proprietary solutions or by the enterprise hosting solutions they offer.

Relying on my own university faculty and IT backgrounds, I have tried to synthesize here questions that you should answer carefully before determining what makes sense for you.

What kind of development resources do you have available? Make no bones about it -- an institutional open source courseware implementation requires development resources. By this, I mean you will likely have/want to do further development or customization of the product before rolling out to your faculty. Authentication schemas may need to be integrated or further architected. The functionality may need to be enhanced. If you have resources, what are the skill sets available. This may sound a bit facile, but if you don't have any java programmers I'm not sure it makes any sense to adopt an open courseware package that is java-based. Also, when you think of your development resources, make sure you think in terms of full-time, professional resources. Once a system is implemented formally, many faculty will have difficulties with an inordinate amount of student development help working on the system.
How Open Do You Want? Not all open source courseware systems are created equal. Some pay much more attention to interoperability standards than others. Some state up front that they follow general database standards but have not yet embraced evolving courseware data structure standards such as those developed under OKI or SCORM. Do those standards make a difference to you? Do you think those standards are evolved enough to be useful to you yet? Do you need them for future implementations?
How scalable is the system? Some open source courseware products were developed as "small" solutions for individual colleges or departments. Others were, from the beginning, intended to be enterprise deployments across clusters of servers. What are your needs for scale (in both directions)? Do you need a portal architecture? How many courses and students do you need to serve? Can you find meaningful implementations of the system that are analogous to yours? What is your security infrastructure? What are your institutional and IT policies regarding FERPA compliance?
What is the pedagogical philosophy touted by the system? Open source courseware, much more than its commercial counterparts, is often driven by specific and particular pedagogical vision. It is helpful to remember that many open development projects are in direct response to an absence of functionality or flexibility in other systems. Is there a particular pedagogical philosophy in online learning shared by your faculty? Is your desire more for a teacher-centric model or a student-centric one? Are you looking for distributed or collaborative learning or a more centralized approach?
What functionality needs to be supported? MathML? LaTeX? Unicode? Some open courseware was developed specifically to address the needs of a particular discipline while other system,s are much more generic and rely on the community to provide enhanced functionality. If you have a system in place currently, ask yourself what the biggest complaints are currently regarding functionality. Is there an open source courseware product that addresses these needs?
What are your grade book and exercise requirements? Do you need to import/export content? Are you looking for quizzing features first and leaning management second? For most organizations, courses still revolve around grades and the grade book. Because of this, the grade book is often the most important tool in a system. How do you use grade books in your institution? Do you want/need to import and export grades or categories? How are activities and assessments integrated with the grade book?

Philosophical Approaches to Architecture

There are many ways to look at open source courseware products. One that I find particularly helpful is to look at them in terms of original architecture or platform purpose. Simply stated, all of these systems are built on one of three basic platform/architecture system types -- content management system, collaborative portal system, Web publishing system. These labels are fairly self-explanatory but I will provide a bit of elaboration. It is important to note that the basic architecture of a system does not mean that the system cannot or does not handle other attributes well. It simply provides a helpful point of departure in reviewing a system.

Content Management System -- These systems have as their primary goal the efficient and effective sharing of documents, images, and other data resources. They tend to support very granular groups handling and can be almost infinitely flexible in terms of broad content manipulation.
Collaborative Portal System -- These systems are built for community building and afford powerful interactive communication. They handle granular groups and administration, and generally allow lots of user customization.
Web Publishing System -- These systems are designed to make it easy for teachers and students to post and share information via the Web. These systems excel in distributed communication and individual customization.

Ratings Categories and Explanation

Each of the courseware systems reviewed is rated on a scale of 1-5 (5 being the best score) in six different categories. Any score under 3 in a category means that the program is not quite ready for prime time in that category. The rating, is somewhat subjective (based on my personal experience with such systems as teacher and administrator) but I have tried to stick to the category questions and to answer them from the perspective of an IT Director or faculty committee member.

(S) Salability -- Is the program suitable for both small and large installations? Will it handle complex security or authentication schemas? Can it run on both single and clustered servers? Does it address adequately Digital Rights Management (DRM) and privacy issues?
(O) Openness -- Other than being free, how open is the source code really? Is it written in a modular format that is designed for easy modification and new, custom modules? Are there clear code specifications for writing new modules? Is there a strong development community associated with the program? Are there other schools of your size that currently utilize the program?
(A) Administration -- How many valuable resource hours will this take to administer and maintain at the server level? How many valuable resource hours will this take to administer and maintain at the program level? How granular and distributed is the administration (the more granular the better). Are all of the data processes automated and will they integrate easily with your other systems? Does the program run on a server platform on which your staff already has excellent expertise?
(I) Implementation -- How fast can you be up and running? What level of expertise is required? What kind of documentation and and assistance are available?
(F) Functionality -- How robust is the feature set for the program? Does it already include all of the teaching "tools" your faculty need? Does it include both synchronous and asynchronous communications tools? Can data be imported and exported easily into/from the program?
(E) Effectiveness -- How well will this program help an average group of faculty deliver their materials online? Will the program require lots of training r is it fairly intuitive to use? How long will it take faculty to set up their courses at a minimal level?

It is important to note that while some of the systems reviewed are "open source" in terms of their treatment of standards, they are not distributable currently (like MIT's OCW/Stella). I have included these systems in the rankings, however, as they provide important benchmarks for comparison. I have marked such systems with a simple *.

Recommended Systems (only fully distributable systems are considered for these recommendations)

Top 3 Systems in terms of Scalability and Development Flexibility

Top 3 Systems in Terms of Pedagogical Flexibility

System Descriptions

I have listed the systems according to the language in which they are developed. My reasoning for this is that the major factor in adopting an open source courseware system is the availability of development resources. This categorization is intended to make that determination easier, as many institutions tend to specialize in specific platform and language expertise.

The actual narrative descriptions of the various systems are taken from their Web sites but the evaluations and ratings are based on reading documentation and working through available demo courses. When possible, I have tried to work directly with groups who have actual installations of the system being reviewed. As this paper and evaluation project evolves, the Xplana team will run test sites of the various products reviewed.

Java

Colloquia
"Colloquia is a software system that supports group working and group learning. It allows any user to set up a working or learning group around a particular topic (a context), add people to it, add resources (web pages, documents etc) to it, set up group tasks, and then engage in group and personal "conversations" about the topic.To support this way of working, users can continually build a library of resources they can use in different contexts, and have access to detailed information about the people they are working with. These are "dragged and dropped" into new contexts. Contexts can have sub-contexts, which can be set up by any participating member, containing sub-groups of members, additional resources and new tasks. Contexts and sub-contexts are only seen by their members. Colloquia does not allow publishing to the Web but also does not use server software."

Colloquia is a groupware product ideal for collaborative management of projects and course assignments. It is an entirely distributed system and does not feature any server software or central administration. Its strength is in ease of use and control as the software actually resides on each individual user's machine. It does not currently support Web publishing, however, and its lack of server software make it inadequate for large deployments or as an enterprise solution.
S =3 O=3 A=4 I=3 F=3 E=3 Total = 19

CourseWork at Stanford *
"CourseWork is Stanford University's course Web site development and distribution system. Using CourseWork, instructors and TA's can set up a course Web site that displays announcements, on-line readings, a dynamic syllabus and schedule, on-line assignments and quizzes, a discussion forum for students, and a grade book. CourseWork is designed both for faculty with little Web experience, who can use CourseWork to develop their Web site quickly, and for expert Web-users, who can use it to organize complex, Web-based materials and link them to Web communication tools."

CourseWork is a non-distributable system that is designed for enterprise use and scale. Stanford is a partner in the development of OKI standards and has designed a robust and stable, if general, course management platform. CourseWork provides good content management functionality with the ability to export grades into other formats for advanced calculations.
S = 4 O=3 A=5 I=3 F=3 E=3 Total = 21

eConf
"eConf is an open source e-learning software, written in Java. It allows to easily record web sessions and has been used to record multiple computer science courses. eConf is an add-on to an HTTP proxy that is able to capture the web pages shown during the session and the voice of the presenter. The audio and the web pages are then synchronized to allow the students to listen to the recorded course. The current release of eConf is built on top of the W3C's Jigsaw proxy but any other proxy may be used. eConf requires a Java virtual machine 1.3 or higher and the Java Media Framework (JMF) ."

eConf is really a dedicated solution for delivering presentations and lectures over the Web. It is a reliable system with good documentation and is useful within its limit pedagogical vision. More appropriately, eConf is not so much a course management system as an information delivery solution.
S =3 O=4 A=4 I=3 F=2 E=3 Total = 19

eLedge Open Learning Management System
"Eledge consists of a collection of Java servlets that utilize a MySQL database back end to store information and course content. The Java programming language, MySQL database and Eledge itself are all open source distributions that can be downloaded free. You can even download the Linux operating system, Apache Web server and Tomcat servlet engine free of charge if you choose to use them."

eLedge is a course management system developed by Chuck Wright at the University of Utah (Department of Chemistry). It is a full-functional CMS with excellent tracking assignment features. It features adequate documentation.
S =3 O=4 A=4 I=4 F=3 E=3 Total = 21

MIT's Open Courseware *
"MIT OCW is a large-scale, Web-based electronic publishing initiative funded jointly by the William and Flora Hewlett Foundation, the Andrew W. Mellon Foundation, and MIT. Its goals are to: provide free, searchable, coherent access to MIT's course materials for educators in the non-profit sector, students, and individual learners around the world; create an efficient, standards-based model that other universities may emulate to publish their own course materials."

OCW is a long-term project at MIT that will eventually result in a powerful, large-scale solution for course management and content production. MIT is one of the original partners for OKI and is devoted to open standards. The system is rich in terms of content management. It is currently not designed or purposed for distribution.
S = 4 O=3 A=5 I=3 F=3 E=4 Total = 22

University of Michigan's CHEF
"The CompreHensive collaborativE Framework (CHEF) initiative has as its goal, the development of a flexible environment for supporting distance learning and collaborative work, and doing research on distance learning and collaborative work. This project is staffed by University of Michigan School of Information and Media Union staff. We are working closely with and are contributing to the OKI reference architecture, and are collaborating with other groups interested in open source collaboration standards. Communities targeted for CHEF use include those involved in the scholarly activities of teaching, learning and research at the University of Michigan, and their students and colleagues involved in teaching, learning and research that are outside of the Michigan community."

Michigan's CHEF is a robust system completely dedicated to open source and distribution. There is good documentation and the product is built for large installations. As a member of the OKI initiative, UM is a strong proponent of open standards. This is a standard portal architecture using open-java tools for basic features such as discussion, chat, and content management. This is a recommended system for a large institution with adequate java development resources.
S = 4 O=5 A=5 I=3 F=3 E=4 Total = 24

PHP

ATutor
"ATutor is an Open Source Web-based Learning Content Management System (LCMS) designed with accessibility and adaptability in mind. Administrators can install or update ATutor in minutes. Educators can quickly assemble Web-based instructional content. Students learn in an adaptive learning environment."

ATutor is a promising system that provides good documentation, ease of installation, and strong potential for developement. While the user interface may not seem intuitive to many, the overall functionality is good (and wide open/modular) and the development team is committed to standards. The system is also install-friendly and receptive to new language versions.
S =3 O=5 A=4 I=4 F=4 E=3 Total = 22

Caroline Classroom Online
"As the example of the Open University has shown, Distance Learning and Networked Learning Teachers/trainers want to create course Web sites quickly. This often means they do not want to learn HTML but just organize forums, calendars, lists of URL's through their favorite browser and publish already existing Word-like, Excel-like or PDF documents. Students want clear and standardized communication tools and readable documents. Organization (universities, schools, ...) want the whole thing to last more than a few years. External partner institutions insist on the importance of using standards. Caroline uses only open formats and languages: PHP, SQL, HTML. It accepts any file type but encourages open formats. As for educational norms (SCORM, IEEE...), we are open to them but we think it is too early yet."

If you don't need a robust grade book and you want to focus on ease of use, collaboration and comforting design, Caroline is your program. It is a popular program internationally and is easy to administer as well as use. The development team has put together good documentation and the development community is active.
S =3 O=5 A=5 I=3 F=3 E=3 Total = 22

ClassWeb at UCLA (PHP/Perl)
"ClassWeb lets instructors easily create and control a class Web site. without learning HTML, FTP or getting a UNIX account.* Instructors just need a standard web browser and a username and password to administer their class sites. Instructors teaching in your building, across campus, or even overseas can all get to their sites, change them, and update materials the same way. The main advantage of ClassWeb is that it offers an easy, low or no training route for faculty to administer a Web site. And because ClassWeb is open source, administrators and staff programmers have extraordinary control. Once you install and customize ClassWeb for your environment, you can change its component programs and "look" as much or as little as you want. For us, this has meant that our History Dept ClassWeb pages look very different than the rest of our depts, even though they're all using ClassWeb and the administration tools are the same."

ClassWeb is a Web site management system for Social Sciences faculty at UCLA. The system allows faculty to create and manage course Web pages, upload files, create links, and send announcements and e-mail. There is no activity or assessment generation, grade book, or tracking. In addition, ClassWeb was not created as a largely scalable system.
S =3 O=4 A=4 I=3 F=2 E=3 Total = 19

eLecture Online Lecturing System
"eLecture/ is a free e-education package to facilitate the design and presentation of electronic lectures. It has evolved from the presentation of a course in a university environment. The guiding idea was to build a system that is simple to use for lecturers, who have some familiarity with writing web-documents (with html-editors or directly as html-code) but are not experts in web-software. It is easily possible to insert and combine existing web-documents. However, it is also possible to include other types of documents, either by linking to them as usual in web pages, or through a working area available to course users."

While eLecture is not a scalable system for large institutions or organizations, it is a nice system for smaller departments or individual teachers who like to work with HTML (or HTML editors) and want enhanced features for their class Web sites. Using PHP scripts, eLecture allows teachers to modify their HTML so that the documents can become collaborative. The system is low on features but does allow for group work and document sharing.
S =2 O=4 A=3 I=3 F=2 E=3 Total = 17

Moodle
"Moodle is a software package for producing internet-based courses and web sites. It's an ongoing development project designed to support a social constructionist framework of education. The word Moodle is an acronym for Modular Object-Oriented Dynamic Learning Environment, which is mostly useful to programmers and education theorists. It's also a verb that describes the process of lazily meandering through something, doing things as it occurs to you to do them, an enjoyable tinkering that often leads to insight and creativity. As such it applies both to the way Moodle was developed, and to the way a student or teacher might approach studying or teaching an online course."

Moodle is one of the most user-friendly and flexible open source courseware products available. It has excellent documentation, strong support for security and administration, and is evolving towards IMS/SCORM standards. The key to Moodle is that it is developed with both pedagogy and technology in mind. A robust development and user community. Great with languages although some development may be needed for robust handling of MathML and enhanced tracking features. Still, this program receives a high recommendation.
S =4 O=5 A=4 I=4 F=3 E=3 Total = 23

Segue Collaborative Learning System
"Segue is really a collaborative learning system. It has a granular permissions structure that allows site owners to assign individuals or groups to be collaborators and specify precisely what content block, page or section of the site they can view, discuss, add to, edit or delete from. Contrast this to e-learning systems such as WebCT or Blackboard in which faculty post the majority of a course's content with tools available only to them. With Segue, students can contribute as much as the instructor or more to a course Web site. In addition students can make their own Web sites that can be associated with a course Web site. or can build a Web site. for their own personal use such as a weblog or an e-portfolio."

Segue is built on a Web publishing platform and is an excellent product for those with a distributed, student-centric philosophy of online learning.
S =4 O=4 A=4 I=4 F=3 E=3 Total = 22

Python/Zope

fle3
"Fle3 is designed to support learner and group centered work that concentrates on creating and developing expressions of knowledge (i.e. knowledge artifacts) and design. Fle3 contains three learning tools and several administration tools. Fle3 WebTops can be used by teachers and students to store different items (documents, files, links, knowledge building notes) related to their studies, organize them to folders and share them with others. WebTop also includes shared "course folder" for each course. The same shared "course folder" is available in the Knowledge Building and Jamming tools as well. The items in the WebTops can be called learning objects - if you wish."

Fle3 may not be for everybody, but if you are pushing a collaborative, group sharing learning environment, you should ditch everything else and give this a try. Fle3 makes abundant use of Zope's powerful content management features and focuses on education as sharing perspectives of different objects (artefacts -- pictures, texts, etc.). Take the demo and see how much fun teaching could be with this system. Unfortunately, it's lack of traditional features make it less attractive for large, traditional implementations. Still, if you like fle3, keep an eye on the developing EduZope project. We can only hope they will add some features and give everyone what they want and need.
S =3 O=5 A=4 I=4 F=2 E=4 Total = 22

ASP

KEWL
"KEWL is an active server pages (ASP) application that runs under any Microsoft operating system that supports ASP. For testing purposes, it will run on a laptop with any Windows operating system from Windows 98 onwards. Aside from needing a Web server, KEWL also requires a database server that may be on the same machine or on a different server on the network. For moderate use, the Microsoft Data Engine (MSDE), which can be downloaded from the Microsoft Web site, provides adequate database services. Thus KEWL will scale from a single desktop—for example, a laptop used for demonstration purposes—to an online learning enterprise such as the one at UWC, where the features are spread across four high-end servers."

KEWL is still under development (in use but evolving) and shows lots of potential. It is less flexible in terms of platforms it will run on and is still in search of a broad range of activities. It does feature excellent presence management and broad support for outside scripting such as javascripts etc.).
S =3 O=4 A=4 I=3 F=3 E=3 Total = 20

Perl

Bazaar
"Bazaar is a web content hosting platform, designed with users (and administration) in mind. Its content-management system, coupled with a flexible permission system –not to mention a modular design and mature API– allows for easy extension and integration into existing administrative systems."

Bazaar provides a an active journaling structure and good collaboration. It also has basic course management features and a nice RSS feature. The system is being developed actively but is not recommended for large deployments yet.
S =3 O=4 A=4 I=3 F=3 E=3 Total = 20

LON-CAPA at Michigan State
"LON-CAPA is a distributed open-source Learning Content Management and Assessment System that provides instructors with a common, scalable platform to assist in all aspects of teaching a course, from lecture preparation to administration of homework assignments and exams. It also enables instructors to create educational materials and to share such learning resources with colleagues across institutions in a simple and efficient manner. "

LON-CARPA is long on functionality and support for extended character sets and computation, but it's a bit short on user friendliness. It is a robust system built for enterprise scalability and handles clustering well. This system is highly recommended as one of the most promising candidates for providing traditional course management functionality for large institutions. LON-CARPA is an active participant in the standards movement and provides strong documentation nd development support.
S =5 O=5 A=4 I=3 F=4 E=3 Total = 24

MimerDesk
"MimerDesk is a web-based groupware environment designed for a wide variety of uses such as personal management, computer-supported collaborative learning, carrying out projects, and setting up communities. Its main strengths include a very customizable group system which allows many groups to work simultaneously on a shared database with tools like Calendar, Tasks, Forums, Links, Chat, Reviews, Voting, Files, Instant Messages, Profiles, and many more."

MimerDesk is really more of a groupware communication and collaboration tool but if that's all you're looking for, it will certainly suit your needs. Using this as courseware is ideal for colleges or departments looking for non-activity-based teaching assistance (more collaboration and document sharing).
S =3 O=4 A=4 I=3 F=2 E=3 Total = 19

WeBWork
"WeBWorK is an Internet-based system for generating and delivering homework problems to students. Its goal is to make homework more effective and efficient."

WeBWork takes full advantage of Perl functionality and is much more easily expandable than CAPA in terms of problem type authoring. On the other hand, it is currenly only a homework/quiz generator and offers no learning management or collaboration features. You also have to learn PG/Perl syntax to be able to author new problem types. In other words, it is the best open source solution for what it does, but incomplete as a course management alternative and inaccessible to many users.
S =3 O=4 A=3 I=3 F=3 E=3 Total = 19

Pcl

.LRN (OpenACS)
"Our (.LRN) goal is to evolve the platform by harnessing the power of open source"user innovation communities." Second, the underlying architecture and technical design is based on principles taught at MIT. Third, the underlying toolkit (OpenACS) is maintained by a talented group of developers worldwide. Fourth, .LRN functionality is based on extensive testing and production use at MIT's Sloan School of Management. Fifth, .LRN will maintain a strong link to some of the best research and technology initiatives at MIT and the Sloan School of Management, including the Open Knowledge Initiative and D-Space."

.LRN is currently a collaborative communication and document sharing tool but is on its way to adding full, traditional course management features. The strength of .LRN is in its administration and flexibility. It also handles group work well and possesses a flexible and modular architecture that makes it easy to install new features as they are added.
S =4 O=5 A=5 I=3 F=2 E=3 Total = 22