Automating the day away: An annoying Moodle plugin and bash scripts

I’ve spent a lot of time working with Git submodules as a part of my journey in systems admin. They have been a key tool for me in improving my upgrade and deployment strategies for Moodle by making the process faster, less prone to error, and more secure. Meaning, there is less downtime and less risk when I do upgrades. All it takes to update all my site plugins is:

sudo git submodule update –remote

Anyway, the above is my justification for working with submodules. But there are struggles sometimes. They work great with 98% of Moodle plugins, as most plugins are published in publicly accessible Github repositories. However, there are those cases where a plugin is not published publicly — usually a premium plugin that is only available to download directly from a vendor website. When I need to use these, I just make a git repository in the plugin folder on my local computer and push to a private repository I can add to my project as a submodule. When a new version comes out, I swap in the new code in the folder, commit and push changes, and am good to go.

So the thorn in my side are the Kaltura plugins. UP uses Kaltura for video and they have a very extensive and robust set of Moodle plugins. My problem is, they host all 10(!) plugins in a single Github repository. The plugins are all in subdirectories as if they were installed in the root directory of a Moodle site, like mod/kalvidassign. That makes it impossible to track them as submodules. I’ve opened a Github issue to see if they would change this but I doubt they will, so I need to rely on the manual method of creating private repos for each one. When an update comes out, that means an annoying, repetitive process of copying new files into folders, and then adding, committing, and pushing ten git repos. No thank you!

I thought this would be a good chance to improve my bash scripting skills. I know the basic programming/scripting techniques but am not very experienced writing in bash. So I did a little research and came up with the script below. Now my process to update the Kaltura plugins in my repo is:

  • Download and unzip the new version (a single folder that contains all the updated code)
  • Move that folder into the root of my existing local Kaltura plugins repository
  • Run my gitcomitter script

The script does the following:

  • asks for the name of the folder containing the updated code
  • asks for a commit message
  • iterates through each plugin directory and copies over the new code, adds and commits changes, and pushes to the remote repo.
#!/bin/bash
#
# this helper script will go into each plugin directory and add/commit/push any changes
# (assuming you have initialized git repos and set up remotes in them)
# use from the root of the Kaltura plugins folder
# copy in the updated code folder and run this script

#get the name of the folder with the updated plugin code
read -p "Directory of of updated code: " updatedir
#get a commit message
read -p "Commit description: " desc

#create array of plugin paths
MODPATHS[0]='blocks/kalturamediagallery'
MODPATHS[1]='filter/kaltura'
MODPATHS[2]='lib/editor/atto/plugins/kalturamedia'
MODPATHS[3]='lib/editor/tinymce/plugins/kalturamedia'
MODPATHS[4]='local/kaltura'
MODPATHS[5]='local/kalturamediagallery'
MODPATHS[6]='local/mymedia'
MODPATHS[7]='mod/kalvidassign'
MODPATHS[8]='mod/kalvidpres'
MODPATHS[9]='mod/kalvidres'

#loop through array, copy over new files, and do git stuff
for i in ${MODPATHS[@]}

do

   cp -r $updatedir/$i/* $i
   ( cd $i && git add . && git add -u && git commit -m "$desc" && git push )

done

#delete the update folder, we don't need it anymore
rm -r updatedir

Photo by Markus Spiske on Unsplash

My Moodle Development Environment

I’m taking an Intro to Moodle development course through MoodleBites, and found myself needing to set-up a decent developer environment.

I used to have a custom desktop Windows PC but my former office now = toddler playroom. I’m currently using a tiny, underpowered 12″ Macbook as my personal computer (I have a beefier laptop at work but this is a home project). Because the laptop is a little slow, I decided to set-up a Moodle environment on a Digital Ocean droplet in the cloud. It costs $5 USD per month.

I’m using Visual Studio Code as my editor, which has an extension that lets you connect to a server over SSH and write code remotely. So I don’t need to run AMP locally anymore.


For VC I am just using GitHub, as that seems like the easiest thing to do and the ‘Moodle way.’ 

My Top 3 Moodle Plugins of the Year

Year-end lists are an easy excuse to write a blog post, and I thought this year I would shout out some of the Moodle plugins I found to be really cool/helpful/inspiring. They aren’t necessarily new, just new to me. These are all plugins I either added recently to UP’s Moodle or am rolling out next semester. So, here goes, in no particular order:

Group Self-Selection

Moodle is at it’s best when used as a social collaborative learning environment and Moodle groups are a key method to support that pedagogical approach. But sometimes groups can be a chore to manage. This awesome plugin lets students opt into joining the groups of their choice or even provision their own groups. More learner agency and less work for instructors is a win-win.

Boost Campus Theme

I’ve had our site rocking the Boost theme since it first became the standard and have had few complaints. But, what complaints I’ve had, the Boost Campus theme addresses. It keeps the layout and overall look very familiar offers the option to enable all sorts of little quality of life improvements.

My favorite things about Boost Campus are the subtle re-organizing of the navigation drawer that somehow makes it much more intuitive, the improved settings menu, and the return of an ever-present “Editing On” button!

Kickstart Course Format

Our faculty often struggle with their new, blank-slate course shells. It’s generally a lot of support work to guide them in importing past course content or in figuring out how to provide useful course templates. The Kickstart format looked like it could solve that problem for us, giving faculty a useful dashboard of options when they first hit their new semester courses. We tried the free version which allows three templates and we were impressed enough that we purchased the premium version which allows unlimited templates and course imports. A bonus feature, strangely unadvertised, is that when courses are imported via the premium version of this plugin, it automatically updates the due dates for assignments based on the start date of the new course. A simple thing, but I had multiple faculty sending me overjoyed emails as they found out they didn’t have to manually change a bunch of dates!

Photo by Clint Patterson on Unsplash

A Guide to Collecting Drafts in Moodle Assignments

I recently completed my graduate studies, and after 3+ years of intensive study and a lot of writing, I have an ardent belief that scholarly writing is NOT a solitary activity. Good teachers show students to that writing is a process. Usually, this requires collecting drafts and creating opportunities for feedback and iterative improvement.

But, how to be paperless and also manage lots and lots of student drafts? I know that from an instructor’s standpoint, it’s not always apparent how to best set-up Moodle assignments to support drafts, feedback, and revisions. With that in mind, I wanted to write up a quick post on some of the possible workflow options for instructors to collect multiple drafts with Moodle Assignments. There is no perfect combination of settings, but these are some important settings to consider.

Submission types

There are a few powerful settings under the Submission types heading in the Assignment settings.

Files: One or Many?

The question here is: when collecting work at various stages, do you want students to upload a new file each time that sits alongside the previous draft, or replace the previous draft with a new one? If you expect multiple versions to be uploaded at once, ensure that the Maximum number of uploaded files is set to a number that will allow for this. Conversely, if you’d rather just deal with a single file from each student, make sure to set this to 1. As they move through different drafts, students will be able to replace their previous drafts with a new version of their document.

The other settings are less crucial for drafts but may still be on interest. For instance, if you depend on using Track Changes in Microsoft Word, you can use the Accepted file types option to prevent students from submitting PDFs or Apple Pages files.

Pro tip

If you are adventurous, you can skip students uploading files altogether. Instead, ask them to upload a file to a cloud service (at UP we have Microsoft OneDrive) and share a link to an online document with you. This allows a more involved and collaborative feedback process. To try this out, disable the File submissions option under Submission types and enable Online text that students can paste a sharing link to. Note that students may need an extra bit of support if they have not shared writing in this way before.

Submission settings

You’ll want to attend carefully to these settings as they determine how Moodle treats submissions.

Use the Submit Button?

One approach is to let students decide when to share a work-in-progress and when to ask you to consider a submission as final. Moodle assignments can allow for this if you enable the optional Submit button. This works with multiple file or single file submissions.

When the Submit button is on, students can upload files to an assignment, but they will be considered a draft until the student clicks the Submit button. You can provide feedback or grades for a student to review and they can choose what (if any) changes to make to their submission before hitting the Submit button.

One other possible benefit of this approach is that students can no longer make changes to their submissions after clicking the Submit button. When a file is submitted automatically upon upload, their file is considered submitted, but they can still make changes to it unless you manually lock submissions.

You can find this option under Submission settings by clicking Show more… to reveal the Require student to click the submit button selection field.

Note: because our default setting for Assignments is for the Submit button to be off, you should proactively remind students that they will need to click the submit button for your assignment.

Attempts

If you want to allow students to optionally resubmit to work towards a higher grade, you can set the Maximum number of allowed attempts option to a specific number, or to unlimited attempts. You can then set the Attempts reopened setting to Manually if you want to reopen attempts yourself, or Automatically until pass if you set a Passing grade for the assignment.

Notifications

If you feel it would be helpful to get email notices when a student submits, look under the Notifications options and set Notify graders about submissions to Yes.

One Assignment, or Many?

A final piece to consider is if you might want to use separate assignment activities for each draft. In the end this might be the simplest approach, especially if the drafts have different due dates or if you would like to grade each one separately. Another option that I have enjoyed as a student is to use an ungraded activity, like a forum, for learners to post early drafts. This allows the instructor and peers to give constructive feedback that the whole class might benefit from. Longer drafts and the final submission would then be uploaded to a separate Moodle Assignment for grading.

Conclusion

Moodle assignments offer a lot of choices to customize how you want to collect drafts. It can be a bit complex, but hopefully you see how these options can be combined to create a workflow that supports an effective digital writing, feedback and revision process.

Photo by ron dyar on Unsplash

Reflection: A Year of Moodle Admin & a Big Project

Note: I wanted to write up a piece that reflected on a year of Moodle admin, talked about how I tackled my biggest upgrade project so far, and also laid out my current approach to performing upgrades. Obviously, that is a lot and turned into too much for a post. So here is part 1 with some reflections. I’ll lay out my upgrade process in a shorter, pt. 2 post, coming (hopefully) soon.

I’m approaching a year of Moodle sys-admin-life and wanted to take a moment to reflect. I remember the uncertainty (sheer terror?) I felt just starting out: I had a lot to learn. Getting used to Linux CLI, Git, working with webservers and MySQL databases were all pretty new. I remember struggling through the first minor point upgrade I ran on my institution’s production Moodle site. I scheduled downtime for the evening hours and worked into the late night. A year later, this process takes me just a few minutes…there was definitely some newbie confusion and just a need to blindly go in and figure things out on my part that led to things taking way longer than they should have (especially figuring out those Git submodules). Yup, I was sweating the details….but I also remember the satisfaction and relief I felt, and the rush of seeing the update script complete and my upgraded site loading for the first time.

Throughout the last year, I kept practicing. I set up sandbox environments to get more experience on test sites (thanks AWS). I ran two more minor updates on production and became more confident each time. I got to experience installing, removing, and updating plugins, and worked with a contract developer to update a custom plugin. I even built out a brand new production Moodle site in the cloud, using Moodle with the Snap theme for a MOOC flavored side-project.

This is all to say that this Summer when it came time to plan not a minor-point update, but a major one – 3.4 to 3.6 – I felt not only ready to tackle the job but to make real improvements to my process.

Making a plan

Since this was the biggest Moodle project I had yet taken on I knew it was critical to come into the upgrade with a solid plan. So first I pored over the official Moodle Docs guide to updating. I made sure my server environment would continue to meet all recommendations, and painstakingly went through all extra plugins to check for compatability and available upgrades.

For the main piece of the update – the Moodle code – I initially researched updating in much the same way I had done the smaller point updates. Since I use Git for version control with submodules for plugins, that would simply entail changing the tracked branch from the official Moodle Git repo and doing a git merge. Unfortunately, this was not as easy as it first seemed, and I found that others in the community struggled with this approach. Unlike minor point updates, merging a major update requires resolving many conflicts or differences between your existing code base and the updated code. I quickly decided instead to use a clean copy of the 3.6 code.

So, I simply cloned the MOODLE_36_STABLE branch from the Moodle GitHub page into a new project directory. All I migrated over from my old folder was my config.php file and my .gitmodules file.

That introduced two new challenges:

  1. How to decouple any existing code customization that was tracked in my main Moodle Git repo (and therefore was not present in a fresh version of Moodle core)
  2. How to update plugins and efficiently migrate them as Git submodules to my new project

Decoupling customization

This was pretty straight-forward if somewhat time-consuming. Since Moodle treats basically everything as a plugin, I simply went to the Site Admin -> Plugins -> Plugins overview -> Additional plugins screen and compared that with my .gitmodules file (the file in git that keeps track of all of your project’s submodules). This let me identify any custom items that were being tracked in my main repository.

Last year I got UP signed up with a GitHub for Education plan, so I had unlimited free private repositories. So I just created new repos for the few items that were either homegrown (a custom HTML block) or that vendors for whatever reason do not make available through a public GitHub. The most annoying was Kaltura’s video plugins, which they put on GitHub as a whole repo with multiple plugins in various directories in a way that won’t work as a submodule, meaning I had to make my own repo for each individual included plugin. If anyone has an idea of how to include something like that wholesale as a submodule, I’d love to hear from you in the comments!

Anyhow, once I had moved everything into it’s own Git repo, I just added those to my .gitmodules file in the new project.

Migrating submodules

To make sure I was getting updated versions of my plugins that were compatible with 3.6, I just updated my .gitmodules file as needed. Honestly, most Moodle developers do a great job of providing branches for multiple Moodle versions and use the same naming convention so doing a find an replace for “MOODLE_34_STABLE” to “MOODLE_36_STABLE” took care of 75% of the work. But the remainder of plugins that needed updating either used non-standard branch naming conventions or had changed remote URLs or maintainers, so there was a fair bit of drudgy manual work to get everything pointed to the right place.

The problem of migrating submodules was a bit less straight-forward. You can’t just copy a .gitmodules file into a new repo, and as far as I can tell there are no easy commands to run that will look through your list of modules and actually add them to your project – maybe I’m wrong… That said: scripts to the rescue. I genuinely wish I had kept a bookmark so I could give proper credit but I found a script posted online somewhere and modified it to suit my needs. It took a bit of experimentation to get it right, but I was able to just put the script below in my 3.6 project directory (with a .gitmodules file copied from my 3.4 project) and run it to add all of my submodules to the project. If this looks useful to you, note that it requires all of the entries in .gitmodules to include the optional branch = branchname line, which you may not have if you are tracking master. So you may need to manually add a line like branch = master to your .gitmodules entries.

 #!/bin/sh

 ## script that looks in a .gitmodules file and re-adds all of the submodules to the git repo
 ## for future improvement - currently fails if branch not listed in gitmodules. should roll back to default master

 set -e

 git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
     while read path_key path
     do
         url_key=$(echo $path_key | sed 's/\.path/.url/')
         url=$(git config -f .gitmodules --get "$url_key")
         branch_key=$(echo $path_key | sed 's/\.path/.branch/')
         branch=$(git config -f .gitmodules --get "$branch_key")
         git submodule add -f -b $branch $url $path

     done

With that work done, I had a fresh copy of the 3.6 code with ALL local modifications exiting as submodules that pointed to a 3.6 compatible version of a plugin. I ran the update script and….boom! Everything went through and installed fine on the first try.

Photo by Joshua Chun on Unsplash

Migrating a multisite WordPress install to a new server

One project I’m working on this Summer is moving a WordPress multsite install (with about 300 sites) to new hardware. I didn’t find much in the way of documentation or guides on this, especially anything specific to a multisite install. So, to prep for this project, I created a new WP multisite with a few test sites, plugins and posts on an AWS server and practiced migrating it to a new virtual machine. Here are the steps I took and what I discovered. This was using Ubuntu 18.04 and a MySQL 5.6 database.

First, make sure to get a copy of your WordPress code and database. Just zip the entire web root directory of the site:

tar -cvfz wp-code.backup.tar.gz html/

Backup the database with mysqldump. Here my database is named wordpress, I can log in as root without a password, and I am creating a backup file wpdb-backup.sql.

sudo mysqldump --add-drop-table -u root wordpress > wpdb-backup.sql 

…then zip the dump file as well.

tar -czvf wpdb-backup.sql.tar.gz wpdb-backup.sql

Next, we need to transfer the backup files to the new server.  If the servers can talk to each other over SSH, use rsync. Before you can use rsync, create a keypair with no password on the origin server. 

Digital Ocean has a good guide on creating SSH keys here if you aren’t familiar: https://www.digitalocean.com/docs/droplets/how-to/add-ssh-keys/create-with-openssh/

If you don’t have an existing key on the server, you can just type

ssh-keygen

… and hit enter when asked for a password to skip using a passphrase and accept the defaults. If you already have an SSH key, instead of accepting the default name (which would overwrite your key) type in a new name like rsync_rsa_id.

Once your key is created, you now need to copy the contents of the public key you just created. If you used the default settings can see the public key with:

cat ~/.ssh/id_rsa.pub

Next SSH in to the destination server. Copy the public key info to the .ssh/authorized keys file in your home directory.

Sudo nano ~./ssh/authorized_keys

Now you are ready to transfer files. Go back to the origin server and run rsync:

rsync -avz -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" --progress /file/to/transfer/backup.tar.gz   hostNameOrIPAddress:/destination/path 

A quick explanation of the -avz flags used:  

  • -a -archive mode. Makes command recursive, copies permissions, copies symlinks, preserves last modified times, and so on. 
  • -v verbose. See what rsync is doing 
  • -z compress data before transferring 

On target server 

Make sure server has neccesary pre-reqs for WordPress. Apache, PHP, MYSQL, rewrite rules etc. Here is a good guide for Ubuntu: 

https://www.digitalocean.com/community/tutorials/how-to-install-wordpress-with-lamp-on-ubuntu-16-04#step-1-create-a-mysql-database-and-user-for-wordpress

Make sure to create a blank database for wordpress, using the same username and password as on your origin server.

Once the database is set-up it’s time to put your WordPress files in their new homes.

 Move to the root directory and  extract the tarball we created of the WordPress code directory:

cd var/www/html 

sudo tar -xvzf wp-code.tar.gz 

Then extract the database tarball using the same command: sudo tar -xvzf mydbbackup.sql, but instead of moving it to a directory, restore it to the wordpress database you created. 

sudo mysql wordpress < wpdb-backup.sql

That’s it!

Unless your server has a different IP address or hostname, in which case….

If the new server is not the same IP address/hostname as the old one, or if you are using a different database name, username or password, you’ll need to make some quick edits to the wp-config.php file.

Note: Before editing it, ALWAYS BACKUP wp-config.php. Trust me 🙂

cp wp-config.php wp-config-bak1.php 

Then Modify wp-config.

Update the line to your current IP or hostname by editing this line:

define('DOMAIN_CURRENT_SITE', 'newHostName'); 

(If you need to, change the relevant lines for the database as well)

Almost done! For a multisite install going to a host with a different name, you need to make an update directly to the database on the destination server. You can use PHPMyAdmin if you have access to that. Or just use the MySQL command line again:

Access MySQL: 

sudo mysql 

Access the wordpress table:

USE wordpress; 

Update the wp_blogs table with your new address: 

UPDATE wp_blogs SET domain = 'newHostName'; 

Whooo! After all that, you should have an identical functioning copy of your multisite blog network on your new host!

Photo by Barth Bailey on Unsplash

Upgrading PHP on a CentOS Server

My institution uses CentOS for all of our Linux servers, and I understand why. It’s super stable. But! The default software repository also tends to use really, really old packages. Which means I was running a WordPress server that I discovered was using PHP 5.5. No good in general since PHP 7 is much faster and secure, and untenable since the minimum version of PHP has been raised to PHP 5.6 in newer versions of WordPress.

Anyway, it’s easy to use an additional package manager and switch to a newer version of PHP. Here’s how I updated to PHP 7.2 (and by the way, made the WordPress site about twice as fast instantly).

First make sure you have repositories installed:

• EPEL repo
• Remi repos
• Yum-Utils

Steps:

Make sure things are up to date:

sudo yum update
  • Enable PHP 7.2
sudo yum-config-manager --enable remi-php72
  • Install PHP and modules
sudo yum install php php-mcrypt php-cli php-gd php-curl php-mysql php-ldap php-zip php-fileinfo
  • Restart apache
sudo apachectl restart
  • Check PHP version (you should get 7.2.x)
php –v

Photo by AJ Robbie on Unsplash

Tools of the trade: Mac Apps

Today I want to call out a couple of Mac apps that I personally have found to be a great help to me as a systems admin and builder. These are all simple things that help me get work done.

Termius

Termius

Part of the reason I switched from Windows to Mac when I took my job was to have easier access to a bash terminal. But after a few weeks, I found that using the built in Mac Terminal app wasn’t very efficient for me. I had something like a dozen production and development servers to access on a regular basis, plus a growing amount of resources in AWS, and manually typing in hostnames, users, passwords, and tracking keys seemed unnecessarily burdensome. Before long I found Termius, which is a great SSH manager. Now I add my host information into Termius and can login into whatever server I need in seconds.

It can be used on a laptop for free but my boss was kind enough to purchase a license for me so I have extra features and can sync my credentials to multiple devices. That’s cool because Termius works just about anywhere, including my iPad and iPhone. The Mac app looks great, performs well, and also includes an SFTP client and the ability to save “snippets” so you can save and re-use tricky or frequent commands.

Cost: Freemium (60 USD per year)

Pro tip: Organize hosts into Groups and set a default color scheme. I use the classic black and green for my dev servers and a light color scheme on production so I can tell at a glance what environment I’m in.

LastPass

LastPass

Let’s face it, passwords suck. But until biometrics or physical tokens or something else totally take over we’re stuck with them. I had been using this app for awhile personally and my team invested in LastPass about 6 months ago. It has made sharing credentials to access some of our tools much easier (and safer).

Most people probably use LastPass exclusively as a browser plugin that can auto-fill passwords. But it also comes with a pretty nice Mac app if you want to get a bit more in-depth. Beyond username/passwords, there are templates to store lots of useful things: server login info, private SSH keys, account numbers, credit card numbers, and so on.

Cost: Freemium (2 USD per user per month)

Pro tip: When the Mac app is running, you can invoke a quick search of your vault with CMD-Shift-L . From the search results there are buttons to quickly copy usernames or passwords to your clipboard.

Amphetamine

Amphetamine

The life of systems administration sometimes involves a lot of waiting around waiting for a process to finish. Sometimes, you need your laptop to stay awake even if you’d rather step away. Amphetamine is an app that does just that. Just turn it on and your Mac will stay awake without needing to futz with power saving settings. 

Cost: Free!

Pro tip: You can choose criteria to keep your Mac awake: while a certain app is running, for a certain amount of time, or indefinitely. Keep this app on, then lock your screen (CTRL-SHIFT-Power button) to let a process run while you go take a coffee break//walk/nap.

Photo by JESHOOTS.COM on Unsplash