Adding a JavaScript/CSS Library to Drupal with the Libraries API

Let’s say we found a useful bit of JS or CSS that we’d like to use within our Drupal site.  For example, this jQuery ListNav Plugin.  An ideal way to accomplish this is by adding it to Drupal’s Library API.  Here’s how to do it.

First, make sure you have the Libraries API module installed and enabled.

Next, we’re going to write a custom module that implements hook_libraries_info() to add our custom library to Drupal’s set.

The jquery_listnav.info file:

and the hook_libraries_info() implementation within jquery_listnav.module:

First, we create a new $libraries array and then define a key => value pair within $libraries for our new library.  The key is the name of our new library within Drupal’s eyes — in our case it’s “jquery-listnav”.  The value is another array of information about our new library.

We’ve defined the following bits of info about our library:

  • name
  • vendor_url
  • download_url
  • version_arguments (which is itself yet another array)
  • files (which is itself yet another set of multi-dimensional arrays)

The former three (name, vendor_url, and download_url) are pretty self-explanatory but let’s dig into the latter two.

The ‘version arguments’ key tells Drupal where it can find information about the library’s version.

The ‘file’ key points Drupal to where the file is located.  In this case, since our library is only one file, the version is contained therein.  If your library consists of many different JS and/or CSS files, you may have an .info file which contains the version data or it may be in only one of the files.

The ‘pattern’ key is a regex expression that tells Drupal how to find the version of the library (if any exists).

The ‘lines’ (optional) key tells Drupal the maximum number of lines to search for the aforementioned regex.  It defaults to 20 so if you know your version is always on line 1 (say, in a minified version of the code in question), you can save some cycles by specifying such here.

Next, the ‘files’ key is the crucial data point as it tells Drupal where to find the files that should be included in this library.

The ‘js’ key is yet another array that contains the file names of JS files to include in this library.  We’ve packaged the JS file in the same directory as our module but it could easily be in the theme itself.

The ‘css’ key is the same as the ‘js’ key but for CSS files.

There are quite a few more available keys in the $libraries array definition including declaring dependencies and utilizing external libraries.  You can read more about them on the Drupal 7 API Reference page for hook_libraries_info().

Finally, one last thing I’d like to cover is how to load a library and add it to a specific page.

I’d like this library to load on all displays of a specific view named ‘directory’.  So in my template.php file in my theme, I implement theme_views_pre_render() and create an if-block for that case.

Then I call:

to load the library we created earlier.  Now that that JS is available, I call drupal_add_js() to run the pertinent JS function.

Sometimes we’ll want a library loaded and available on all views, or all pages.  In that case, you could employ the same tactic but remove the if-block from the theme_views_pre_render() call.  You could even do it in something like hook_init().

Quickie: A Bash Script for Open Atrium Upgrades

Our webhost, Acquia, provides security updates as a service through Remote Administration Automation.  This means that, when a security update is released for any contrib modules used in any of our sites, Acquia is immediately aware and immediately begins the process of updating said module.  Once they have the module’s updated version committed to our git repository, they can either automatically deploy it or simply notify us of the update so we can review it and deploy it ourselves.

In a Standard Drupal Core instance, this is great!  It cuts the estimated 25% Full Time Employee (FTE)’s time to keep Drupal up to date down to ~5% (my own rough estimate).  However, with an instance of Open Atrium, this automated-update process can cause problems.

With Open Atrium, even with critical security updates, the proper process is to hold off on individual module updates and wait for an official release of the Open Atrium project.  This is because there may be other changes to OA modules/themes that are now required due to the contrib module’s update.

This means that when the automated process detects an available update for a module in OA, it will apply the upgrade onlto the contrib module.  If notified, Acquia will wait for the proper OA release and perform the update with it but it often takes some back and forth correspondence and may introduce extra delay in getting that update to production.  When the update contains a patch for a critical security issue, that may not always be tenable and manually upgrading may be the only option.

My colleague has produced a series of commands that, when executed from a level above your docroot will update OA, keeping your sites directory intact as well as all your dotfiles, robots*.txts, and, in our case, our simplesaml configuration:

I took the liberty to script this for ease-of-use in the future:

Or here’s the upgrade script on PasteBin.

Below line 28, you can add a line for any custom directories that may be unique to your installation.  For example, we have a simplesaml setup for single-sign-on and favicons_etc here.

The usage is as follows (first copy/create the script in the directory above your docroot):

For example, if you wanted to upgrade to OA 2.63 and your docroot directory’s name is “docroot”, you’d copy the script to a level above your docroot and execute:

$./upgrade_oa.sh docroot https://ftp.drupal.org/files/projects/openatrium-7.x-2.63-core.tar.gz

You can find the URLs to OA releases on the Open Atrium project’s releases page.

Creating A Subtheme From Open Atrium 2.6x’s oa_basetheme

Last month, Open Atrium released one of their largest upgrades since version 2.  Included in this upgrade is a complete revamp of the theme layer.  You can read more about the changes in 2.6 here: A Fresh New Look for Atrium.

There is some documentation on creating a subtheme from oa_radix, OpenAtrium 2.5’s base theme but, unfortunately, the documentation for subtheming 2.6’s base theme, oa_basetheme, hasn’t been created by the maintainers just yet.

We’re trying to get this upgrade out to our users ASAP so, today, I spent some time fiddling with creating a new subtheme for Open Atrium 2.6x.  In this tutorial, we’ll go over what I’ve found:

The oa_basetheme comes with bootswatch and, using Drush, we are supposedly able to install an ‘instant subtheme’ by executing:

There are various options for this command, if you’d like to provide a machine name or use a different bootswatch theme.  You can find these options by running:

So, from my site’s directory (sites/default), I tried to run:

and was presented with

This issue had been reported on d.o, but that thread lead to no results.

Creating a Subtheme

After some finagling and trial-and-error, I was able to get the starter subtheme created by performing the following steps:

  1. Copy oa_basetheme from “docroot/profiles/openatrium/themes” to my site’s theme directory, “docroot/sites/default/themes”:
  2. Enable the theme with Drush:
  3. Browse to Appearance Config (/admin/appearance) and select Set As Default for oa_basetheme.
  4. Run the oa_basetheme bootswatch starter Drush command:

    No errors this time!  I’m honestly not sure if the resolution was the fact that I moved oa_basetheme from the Open Atrium profile to sites/default or the fact that I set oa_basetheme as the Default theme.  I’m inclined to think the fix was the latter since the error was a Bootswatch conflict with oa_radix, but, alas, I have yet to actually confirm this.

    This will put the new subtheme in “docroot/sites/all/themes”.  If you don’t need it in /sites/all, be sure to move it to /sites/default.

  5. Browse to Appearance Config (/admin/appearance) and select Set as Default for the original theme from Step 3.
  6. Remove the copy of oa_basetheme from your site’s theme directory

Now, if you have a fresh install of OpenAtrium 2.6x, you can go ahead and “Enable and Set as Default” directly from Appearance Config (/admin/appearance) and you’ll be good to go!

Upgrading from 2.5x to 2.6x?

However, if you, like me, are upgrading an existing OpenAtrium site from 2.5x to 2.6x, there are a few more steps you need to take to enable all the new features of the theme.

  1. Enable the Open Atrium Site Layout Defaults module.  This is a module that ships with OA 2.6x and contains the default panels config required for the new functionality in the new 2.6x theme.

    This will also enable dependencies: oa_site_layout and panels_everywhere
  2. Go ahead and Enable your new subtheme in Appearance Config (/admin/appearance) but do not Set As Default just yet.
  3. The OA base theme’s panel layout is defined in a Page called site_template for the Panels Everywhere module, however, out of the box, it’s configured to be used only when the current theme is either oa_theme or oa_basetheme.Since we have a new subtheme, we need to update the selection rules for the pertinent Variant within that Page to use the Variant for our new subtheme as well.  Sooo… browse to /admin/structure/pages/nojs/operation/site_template/handlers/site_template__atrium-modern/criteria” (or go to Admin > Structure > Pages > Edit site template Tab > Selection Rules)
  4. Under the Criteria Table, select “Current theme” from the drop down and click the “Add” button.

     Selection Rules config page of the Modern layout Variant
    On the Selection Rules config page of the Modern layout Variant, select “Current theme” and click the “Add” button
  5. Select your new subtheme from the dropdown menu and click the “Save” button.

    Theme selection while adding a new Selection Rule to the Modern layout Variant
    Select your new subtheme and click the “Save” button
  6. Click the “Update and Save” button at the bottom of the Variant Options for the Default site template page.

    Screenshot of adding a new selection rule to a variant
    Once your new selection rule has been added, click “Update and Save” on the Variant modal.
  7. Finally, go back to Appearance Config (/admin/appearance) and select Set as Default for your new subtheme.
  8. Clear caches and you’re all set to start working on the theme’s code itself.

One more thing… Regarding Concurrent Themes

We are currently planning to deploy our new subtheme to a subset of users: a beta group.

This requires keeping our existing theme from 2.5x enabled and using ThemeKey and ThemeKey User Profile (packaged with the ThemeKey module) to allow users to “opt-in” to the new theme.

To accomplish that, forego Step 7 of the previous list, leaving your old 2.5x theme as Default and your new subtheme simply Enabled.  Follow these steps to install and configure ThemeKey and ThemeKey User Profile.

  1. Install and enable ThemeKey and ThemeKey User Profile
  2. Browse to the UI Settings for ThemeKey (/admin/config/user-interface/themekey/settings/ui), expand the Selectable Themes table and check the checkbox for your old 2.5x theme and your new 2.6x subtheme.
  3. In the UI Settings table, select “Add theme option to user profile”.
  4. Click the “Save configuration” button.
  5. Browse to User Permissions configuration page, and ensure the permission: “Select different theme” is enabled for your required roles (in most cases, authenticated user).  Or, add the permission to the role with Drush:

Now when a user browses to their Edit Profile page, they’ll be presented with a field to select the theme to use.

Screenshot of Edit Profile page including ThemeKey User Profile selection table
A user can select which theme to use on their Edit Profile page

How to Add a Custom Shortcode to Drupal 7

We recently received a request to embed some JavaScript for a live chat client into multiple pages within our site.  The live chat service provided the JS code snippet to our team along with the ID key we would need to use within the snippet to link the client on our pages with the live-chatters on the other side.  There were also a few settings that could be changed within the snippet — things like background color, offset width, and location on the page.

Since we do not allow JavaScript tags within our WYSIWYG editor, we couldn’t just ask our Content Managers (CMs) to copy/paste the snippet right into the Body (or other) field.  We also didn’t want to be called upon everytime the script needed to be embedded on a different page or when the ID key changed due to a new marketing campaign (or any other reason).  We decided to use Shortcodes.

Shortcodes are little bits of code that, when parsed in a field that allows them, expand to a pre-defined output, replacing any pre-defined variables in the expansion.

For a quick example, a shortcode for embedding a link to Google may be:

When that string is parsed in a shortcode-enabled field, it might render as:

This is useful because, if the link to Google were to ever change, any number of instances of the shortcode can be updated all at once by one modification to the Shortcode code.

As I mentioned, Shortcodes can also take variables.  For example:

might expand to

This tutorial requires the contrib module Shortcode.  We’ll write a custom module to extend its functionality and add a new shortcode. Here’s how to do it.

First, let’s create a module.  I’ll call mine “tmbridge_custom_shortcodes” and, thus, create a directory, tmbridge_custom_shortcodes, in my custom modules directory and two files, tmbridge_custom_shortcodes.module and tmbridge_custom_shortcodes.info, in that directory.

I won’t speak to writing the info file in this post, but you can refer to this page on d.o for more information.

The first thing we need to do in this new module is name and describe our new shortcode and indicate the callback functions it will use:

We need to define the variables (or attributes) for our new shortcode.  We do this by implementing hook_theme():

Next, let’s define our callback function for the new shortcode.  This callback function takes the variables as defined by the user in the shortcode’s use in the WYSIWYG editor and applies any business logic, formatting, or modification (like setting a default, if empty):

Now, we need to define the theme function that our process callback function calls.  In this case, it’s simply ‘shortcode_friendly_alert’.  This theme function returns the output of the shortcode with the variables replaced.  Here’s where the magic of this module happens.  You can put literally any markup you wish into this function’s output and populate as many variables within it with user-defined data (this means you could get data from any Drupal object that’s in scope, as well).  It could be JavaScript (including AJAX functionality), CSS, output from other theme functions (blocks, views, etc.), Or… just run of the mill HTML markup.

In this tutorial, we’re just using some (really) basic JS and HTML to make the point, but I really want to drive home the wide breadth of functionality one could implement herein.

Finally, let’s define our tip callback.  This function outputs text below the WYSIWYG editor that tells the user what the shortcode is used for and shows the user what parameters are available.

And that’s it for the code!  Simply enable the module and be sure to remember to enable the new shortcode in all the Text Formats you’d like it available within (/admin/config/content/formats).  The available shortcodes are listed in the vertical tab “Shortcodes” under “Filter Settings:

Enabling Shortcode in Text Format Settings
Be sure to enable your new Shortcode in the Text Format settings

You will then see your new shortcode available in all fields that use those Text Formats.

Your new shortcode is now available in all WYSIWYG fields that use those Text Formats.
Your new shortcode is now available in all WYSIWYG fields that use those Text Formats.

 

Integrating Webforms with Open Atrium’s Panes by Creating a Custom CTools Plugin

Open Atrium (OA) core utilizes PanelizerFieldable Panels Panes, and CTools to give content managers (CM) a convenient UI for adding new and existing content to a node.  Open Atrium core also includes the Webform contrib module but, unfortunately, neither is it panelized nor is it ready as a CTools content type out-of the box.

This means that (a) core OA does not allow a CM a way to add other panes to a webform node and, thus, no way to add panes native display.  In this post, we will outline how to add a new content type to CTools and how to define that new content type.

First, we need to create a module.  I’m calling mine:

So in my modules/custom directory, I’ve created a directory:

and files:

Of course, you’ll need to write your info file but I’m sure you know that already!  Now we need to tell CTools where to find our plugin files so my module file looks like:

This is boilerplate code that implements hook_ctools_plugin().  It tells CTools to look in directory “plugins” for plugin files when looking for content types.

Within the tmbridge_webform_panes directory, next to the tmbridge_webform_panes.module, let’s create a “plugins” directory and inside that, let’s create a “content_types” directory.  My directory structure now looks like:

CTools knows to look for include files within the directory that is returned from the call to hook_ctools_plugin_directory() so let’s create an include file within the content_types directory.  We’ll call it tmbridge_webform_panes.inc.  Our directory structure now looks like:

The first thing we need in this file is a $plugin array definition that outlines what this plugin does:

Many of these elements are self-explanatory.  The first element, single, tells CTools that this plugin has 0 subtypes.  The next three elements are string definitions that will appear in the UI for CMs.  The latter four elements define the callback functions that will be called when this plugin’s code is run.  Next in this post, we’ll get to the meat-and-potatoes of writing a CTools plugin: callback functions.

We need to determine which types this plugin should be able to render.  Since we are concerned with webforms, we’ll query the database for all subtypes that have webform components and build an array containing that set:

Next, we’ll build the edit form that presents available options to the user by writing the callback function that we defined earlier in the plugin array.

We are only concerned with the webform to embed in this plugin, but this function can be extended to include any kind of user-defined settings you may need.

You may notice that in this callback function, we make a call to a helper function called _webform_panes_get_all_forms($subtype).  This is a custom auxiliary function defined as follows:

Something to note in this function is the line:

This is how we limit the select options on the edit form to include only nodes to which the user has access.  If our use case called for it, we could instead look here for the node_edit permission and only allow a user to include a webform to which they have edit access.  We’d like users to embed any forms to which they have access so all we are looking for is node_access.

Next, we define the form submit callback which saves the form state information:

Finally, we can talk about rendering.  The render function below creates a block from the selected webform and defines the view mode that will be used to render said node.

We check here if the pane has an override title defined and, if not, we use the webform node’s title as default.  A check for permission is made here as well.  We also take this opportunity to populate the $settings variable with a flag that will be used in the corresponding template to easily check if this block is a webform block.

We also define here the theme hook to use to build the block content.  This is completely dependent on the theme you are using, I’ve opted for a custom hook defined in my theme but use whatever suits your needs (we’ll talk more about creating a custom theme hook to use here in another post.).  You can find more information on the theme() function in drupal.org’s API documentation.

And, with one last function, we define the administrative title that will be used for a webform in a pane:

And voilà! We now have a new content type available to CTools to embed a webform within a pane on any Panelized node’s display.  The plugin pulls all accessible webforms into a select box and allows the user to select from that list:

Webform content type selection process -- step 1 and 2
We now have a Webform content type defined for use within Open Atrium.
Webform selection process -- step 3
And we can select from a list of all the webforms to which we have access

Of course, Open Atrium has native functionality to embed blocks and, at least with Webforms 3 and up, we can configure all webform nodes to automatically create an associated block — however, a custom CTools plugin yields a more comfortable user experience for CMs as Webforms are considered a distinct content type.  It also allows more customization of and during the rendering a webform.

Another alternative to a custom CTools plugin would be to “Panelize” the entire Webform content type.  This would allow CMs to add other panes to a webform node itself.  This comes with its own set of pros and cons.  I will cover this process along with its pros and cons in another post.

A corollary to creating a plugin for a new CTools content type is to create a new pane style plugin for CTools.  Combined together, a content type plugin and a style plugin can yield some great functionality.  I plan to write another post on that subject soon.

CTools plugins are incredibly powerful.  The above tutorial is merely one use case but the methodology and structure is consistent across the board.  Be sure to keep them in your toolchest for when a need may arise.

A Drupal 7 Launch Post-Mortem

Last month, we launched a brand new site at our institution.  This site was built by an outside contractor in conjunction with our team.  This image above was our launch pre-, during, and post- launch plans.

As it goes, no launch is perfect.  The launch was successful in that, by the next morning, we had the new site up for all the world to see.  However, our plan was to begin the launch at 8PM and finish by 10PM.  My team and I didn’t log off from our workstations until 4AM.  We also had to work with our hosting provider to get emergency upsizing for our two dedicated webservers.

Accordingly, there were a few things we could have done better and quite a few things to take away from the experience.

What We Could Have Done Better

Environment Space Differences

Within an hour of starting the launch, we ran into our first issue.  Content managers had been working in our Staging environment for the last few months, getting the site ready for launch.  One step of the launch was to migrate all files from the Staging environment to the Production environment.  We opted to simply tar up all the files on staging, scp the entire tarball to production, and untar the file there.

We have a shared gluster between all of our dev and stage environments on that box, thus, while making the tar, we quickly learned that effectively doubling the space usage (by keeping all files + a tar of all files) was going to quickly put us over our space limit.  Despite all of our brainstorming and prep, no one on the team considered file space!  Our production boxen have a much higher space limit so it would be no problem to hold and extract the tar there.  We just needed to make it first on Stage.

Our solution: We simply paused the tar and manually cleared out the files directories for the old version of our sites on the Stage server.  Once we finished the launch, we could, if so desired, delete the tar and pull back down the files from Prod to Stage.

Lesson Learned: In the future, we should be wary of space limitations and perhaps even test the creation of the “launch files tarball”.  Another option would be to use rsync to migrate the files without having to have a moment in time when both the original file and that tar’d version of the file exists on the same server.

Cache Warming

Our biggest hurdle was the lack of a full cache at launch.  Within ten minutes of turning off maintenance mode, our production box fell over.  When our production boxen truly go down, we are at the mercy of our host to restart said boxen.

Graph of server crashing ~10 minutes after taking it out of maintenance mode.
Our production box fell over ~10 minutes after taking it out of maintenance mode.

This was because our site effectively relies on caching to perform.  At our host, we accept lower origin server performance to save on cost, knowing that Varnish and our load balancers will bear the brunt of requests.

Of course, without said Varnish cache properly warmed, that means our production boxes are at high risk to go down with any significant traffic.  This ties into Launch Time & Publicity, which I will speak more to in the next section.

Our solution: We overcame this hurdle by contacting our host and asking them to temporarily upsize our production boxes so we could withstand the higher-than-normal number of requests to origin and give Varnish a chance to warm up.

Lesson Learned: In the future, it would be wise to keep maintenance mode (or something akin to it, like a CDN) on and allow only a specific IP or IP Range.  Then write and use a script to perform a slow-crawl of the site (or, at least, the high traffic pages).  This would allow Varnish to warm up before opening it up to the world.

Lesson Learned: Temporary, emergency (read: possible to implement in minutes or hours, not days) upsizing and/or horizontal scaling is a must when your site truly relies on Varnish to stay up.

Launch Timelines & Publicity

Many people were aware of the details of our launch plan.  This lead to (1) people spamming F5 at 10PM which was only detrimental to our efforts to get the site up without a warmed Varnish cache and (2) people asking questions about progress and reporting “the site is down! the sky is falling!” while we were obviously 100% aware and working on the issues.

Lesson Learned: Inform only those who need to know of all the minute details of your launch plan.  Most people would do fine to know just that “The site will be live on Tuesday morning”, instead of “The site will be live by 10:01PM Monday night.

Lesson Learned: It may be best to launch during “deep” off hours (i.e. 12am-5am).  Doing so would result in fewer people spamming F5 at the tail end of the projected launch window and, likewise, would result in less exposure to the site’s downtime — resulting in fewer frantic stakeholder and fewer duplicate issue reports from constituents.

The Cost of A Bootstrap

In past days, we served a proxies.pac file to all machines that might connect to our network.  We either configured images that were pre-loaded onto our user’s laptops to request this file or gave them a tutorial to configure it on their own.  We are seeing remnants of those days in that there are many systems out there that still contain this legacy configuration.  In fact, the vast majority of requests to our webserver are legacy requests for this file by two orders of magnitude.

In our previous version of the site, requests for this file would never be seen by Drupal because we would redirect the request away as soon as we knew it wasn’t for a Drupal site.  In our new iteration, _everything_ bootstraps Drupal as long as it gets past .htaccess.

This meant reams of requests for proxies.pac (and corollary proxies-new.cgi and proxies-test.cgi) were bootstrapping Drupal!

Immediately after launch and before our production boxen fell over, I noticed that proxies.pac requests were spamming our logs. We had talked about implementing Fast 404 and deemed it unnecessary pre-launch.  Hindsight is 20/20 and it turns out that this massive number of requests was one of the largest factors that sunk our boxen.

Our Solution:  We added a redirect in the top-level .htaccess to intercept these requests before bootstrapping Drupal.

Lesson Learned: Speak up!  If I had called more attention to the massive number of requests as a potential issue, we might have been able to get this fix it and, thus, log off a lot earlier.

Lesson Learned: It wouldn’t have hurt anything at all to install an enable Fast 404 pre-launch.  Looking back, I can’t think of any good reason other than laziness for why we didn’t have Fast 404 enabled at launch.

What We Did Well

Not all was doom and gloom, though.  In the grand scheme of things, this was a huge endeavor!  With that in mind, it was one of the most smooth launches I’ve had the joy of working on and living through.

Be on Acquia

Being on Acquia instead of hosting within our own data center ensured a level of environment parity that we’ve never had before.  We could be relatively certain that if it worked on Stage, it would work on Prod — save for elements that explicitly relied on being in Prod to test.

Rollback plan

We spent some time developing a step-by-step rollback plan and decided together at what point in the night we would resort to revert (it was 5AM, by the way.  We just made it.)  This gave us a level of comfort in knowing that if everything got truly FUBAR, we could always rollback.  Sure, at great cost, but at least we’d still have a site in the morning no matter how you sliced it.

Version control

I really shouldn’t need to mention this but we’ve only had our entire web codebase in version control since mid-2012.  Our last site launch was done without any version control.

Our team has come a long way since then and I’ll once again preach the importance of version control.  It allowed us to iteratively review the work of the vendor and quickly and easily deploy the new version of the site.  It also provided a simple rollback path, if that was needed.

Code and Content Freeze

We were able to work with all our content managers and stakeholders to agree to a code and content freeze a good five hours before the start of the launch process.  This meant ample time to wrap up the pre-launch tasks without having to keep track of updating databases during go-time.

Monitoring

We use NewRelic to monitor our boxen.  This helped us to be aware of what was actually going on, bringing down our servers —  Instead of grepping through and squinting at log files.

Failure Mode Brainstorming

We spent nearly two hours just brainstorming all the possible things that could have gone wrong.  We also brought up all the things that went wrong during previous launches.  If we didn’t already have a measure in place to deal with a potential issue that came up, we made it a point to implement one before launch day.