Reflection: A Year of Moodle Admin & a Big Project

Note: I wanted to write up a piece that reflected on a year of Moodle admin, talked about how I tackled my biggest upgrade project so far, and also laid out my current approach to performing upgrades. Obviously, that is a lot and turned into too much for a post. So here is part 1 with some reflections. I’ll lay out my upgrade process in a shorter, pt. 2 post, coming (hopefully) soon.

I’m approaching a year of Moodle sys-admin-life and wanted to take a moment to reflect. I remember the uncertainty (sheer terror?) I felt just starting out: I had a lot to learn. Getting used to Linux CLI, Git, working with webservers and MySQL databases were all pretty new. I remember struggling through the first minor point upgrade I ran on my institution’s production Moodle site. I scheduled downtime for the evening hours and worked into the late night. A year later, this process takes me just a few minutes…there was definitely some newbie confusion and just a need to blindly go in and figure things out on my part that led to things taking way longer than they should have (especially figuring out those Git submodules). Yup, I was sweating the details….but I also remember the satisfaction and relief I felt, and the rush of seeing the update script complete and my upgraded site loading for the first time.

Throughout the last year, I kept practicing. I set up sandbox environments to get more experience on test sites (thanks AWS). I ran two more minor updates on production and became more confident each time. I got to experience installing, removing, and updating plugins, and worked with a contract developer to update a custom plugin. I even built out a brand new production Moodle site in the cloud, using Moodle with the Snap theme for a MOOC flavored side-project.

This is all to say that this Summer when it came time to plan not a minor-point update, but a major one – 3.4 to 3.6 – I felt not only ready to tackle the job but to make real improvements to my process.

Making a plan

Since this was the biggest Moodle project I had yet taken on I knew it was critical to come into the upgrade with a solid plan. So first I pored over the official Moodle Docs guide to updating. I made sure my server environment would continue to meet all recommendations, and painstakingly went through all extra plugins to check for compatability and available upgrades.

For the main piece of the update – the Moodle code – I initially researched updating in much the same way I had done the smaller point updates. Since I use Git for version control with submodules for plugins, that would simply entail changing the tracked branch from the official Moodle Git repo and doing a git merge. Unfortunately, this was not as easy as it first seemed, and I found that others in the community struggled with this approach. Unlike minor point updates, merging a major update requires resolving many conflicts or differences between your existing code base and the updated code. I quickly decided instead to use a clean copy of the 3.6 code.

So, I simply cloned the MOODLE_36_STABLE branch from the Moodle GitHub page into a new project directory. All I migrated over from my old folder was my config.php file and my .gitmodules file.

That introduced two new challenges:

  1. How to decouple any existing code customization that was tracked in my main Moodle Git repo (and therefore was not present in a fresh version of Moodle core)
  2. How to update plugins and efficiently migrate them as Git submodules to my new project

Decoupling customization

This was pretty straight-forward if somewhat time-consuming. Since Moodle treats basically everything as a plugin, I simply went to the Site Admin -> Plugins -> Plugins overview -> Additional plugins screen and compared that with my .gitmodules file (the file in git that keeps track of all of your project’s submodules). This let me identify any custom items that were being tracked in my main repository.

Last year I got UP signed up with a GitHub for Education plan, so I had unlimited free private repositories. So I just created new repos for the few items that were either homegrown (a custom HTML block) or that vendors for whatever reason do not make available through a public GitHub. The most annoying was Kaltura’s video plugins, which they put on GitHub as a whole repo with multiple plugins in various directories in a way that won’t work as a submodule, meaning I had to make my own repo for each individual included plugin. If anyone has an idea of how to include something like that wholesale as a submodule, I’d love to hear from you in the comments!

Anyhow, once I had moved everything into it’s own Git repo, I just added those to my .gitmodules file in the new project.

Migrating submodules

To make sure I was getting updated versions of my plugins that were compatible with 3.6, I just updated my .gitmodules file as needed. Honestly, most Moodle developers do a great job of providing branches for multiple Moodle versions and use the same naming convention so doing a find an replace for “MOODLE_34_STABLE” to “MOODLE_36_STABLE” took care of 75% of the work. But the remainder of plugins that needed updating either used non-standard branch naming conventions or had changed remote URLs or maintainers, so there was a fair bit of drudgy manual work to get everything pointed to the right place.

The problem of migrating submodules was a bit less obvious but ultimately much easier and more straight-forward. You can’t just copy a .gitmodules file into a new repo – you end up with a bunch of empty plugin folders. But, thankfully if you know the commands to run it only takes a minute to get up and running (once again, submodule are extremely efficient if you know what you are doing and extremely cumbersome and time-consuming if you don’t). Here is what I ran to bring my submodules back to life in my new project folder. First, make sure the .gitmodules file is updated and contains the submodules you want to include, along with the path to install them in, updated branches you want to track and remote URLs. Then, in the main project folder (where .gitmodules is located):

First, initialize submodules listed in the gitmodules file

sudo git submodule init

Then clone the most recent commit of the specified branch into the install location

sudo git submodule update --remote

With that work done, I had a fresh copy of the 3.6 code with ALL local modifications exiting as submodules that pointed to a 3.6 compatible version of a plugin. I ran the update script and….boom! Everything went through and installed fine on the first try.

Photo by Joshua Chun on Unsplash

It’s All About the Data

This is going to sound stupid, but…One thing I didn’t really think about enough prior to starting my current job is just how much working with data is involved. Yes, I know. Information systems. It’s kind of all about data. But when I imagined what this job would entail, I thought a lot about building things, about creating systems that support and spark teaching and learning. The reams and reams of data that go along for the – names, email addresses, course CRNs, logs, files, etc – just didn’t seem that sexy to me.

But after a few months of being a sysadmin, I’m thinking about data more and more. There are the obvious things: am I doing enough to keep data safe in the age of constant breaches? Are we thinking about giving users control of their data in any meaningful way in light of evolving views about digital privacy? These are big questions. But in the day to day, I’m just working with like, rows and rows and rows.

It turns out, people have questions they want answered about the digital systems that play such an important role in our schools. These are all questions I’ve answered in the last month:

Which instructors used the accessible Moodle theme we provided in their courses?

I accidentally deleted a quiz, is there any way to get the grades each student in my class got on it?

Can you tell me who at the University has not yet completed this mandatory training?

How many people logged into Moodle on the first day of classes?

Here is the CRN numbers of 50 courses – can you get me the course full name, id number, and the instructor of record?

So what have I learned?

Sometimes this data is readily available within an application itself. For example in the recent versions of Moodle each course has a very powerful completion tracking and activity tracking report built-in. It is also be possible to extend this functionality with plugins.

Reports or logs can also generally be exported into a .csv or spreadsheet format. I was already pretty proficient with Excel but I’ve upped my game in the last few months just due to the amount of time I’ve spent in spreadsheets. (I don’t think I’ve opened PowerPoint in that time, I’ve kind of flip-flopped in that regard. Less presenting polished data and more churning through raw data).

In addition to a sheets tool, a good plain-text editor is your friend when data needs to be manipulated to be useful. I’ve been using VS Code and find it really useful for manipulating text, especially the ability to find and replace with a regex expression. This is incredibly handy when you have a comma separated list but it just has to have each item on a new-line to import wherever it is you need it to go.

If built-in reports or exported info isn’t cutting it, I’ve been going straight to the source itself: typically the Moodle database accessed by PHPMyAdmin. This lets me run SQL queries in a pretty friendly GUI environment and export the results. I’ve found that simple SQL itself is relatively easy to write – it’s understanding the complex structure of a huge relational database, how the tables need to be joined, and thinking through the links between tables that takes a bit of time. But, I am getting faster at this process – I find sketching the query out and making kind of a map of the relevant tables and fields helpful – and this is what has allowed me to do things like recover the grades of a deleted quiz or return a list of instructors who have used a certain theme.

Sketching out a SQL query
Sketching out a SQL query

Photo by Markus Spiske on Unsplash