Software Updates and Upgrades

Top-level overview of the update process

Introduction

The following things happen in typical a software update (or upgrade):

• Files that are new in the new version are added
• Files that have been changed in the new version are replaced with the new versions of those files
• Files that are no longer used in the new version are deleted
• Changes to the database are made to make the database compatible with the latest version of X2CRM
• X2CRM data caches are cleared

These operations are typically handled by the updater utility (UpdaterBehavior) and controlled via the in-app web-based updater or the command line updater. This article will cover both how to use said interfaces to the updater, and how to perform updates manually if the need arises, in addition to troubleshooting and recovering from an update that did not complete cleanly.

Update Notifications

System flash message displaying software update notification.

To get notified of software updates, unless already using X2CRM Professional Edition, one must first "subscribe" to them. This can be done via checking the box "Notify me of software updates" in the installation form. If X2CRM is already installed, this can be done by going to "Updater Settings" from the Admin page and enabling them. If using Professional Edition, these steps are not not necessary.

Once that has been completed, set the update notification interval as desired in "Updater Settings".

Update Methods

Using the Web Updater

The updater utility page.

The web updater can be accessed using the link in the notification message at the top of the window in versions later than 3.5.5, or by going to the Admin page and clicking the "Update" button at the top, in earlier versions.

Advantages

It is generally quicker and easier to use, and it does not require shell/SSH access to the web server.

Disadvantages

It is not recommended for use on large installations of X2CRM, i.e. containing over a million records.

Explanation

The inherent weakness of the web updater (or any web-based updater to any web application) is that it must always rely upon web requests to the server, and it thus must perform extremely critical operations for the update all within the scope of a web request. Web requests, it goes without saying, are not always reliable. For example, one's internet connection might get cut off due being on a wireless network and encountering interference. Furthermore, if any operations take an exceedingly long amount of time, e.g. because of the size of the data set on which the update must operate, the web request for those operations may take longer than the configured server timeout (maximum request time). This would potentially cut short the execution of a critical update stage, and thus the update would require manual completion and recovery.

How to use

First, navigate to the web updater page, and do not cancel (stop or hit your browser's "back" button) until the page has loaded. If the updater utility has undergone changes and needs to self-update to be compatible with the software updates server, this is when it will do so.

There are three ways of getting to the updater page. One is to simply click the link given in the update notification flash message, if available. The other is to click on the "Update" button at the top of the admin page. If software update notifications are not enabled, one can manually check for and install an update by going to "Update X2CRM" under "System Settings" in the Admin page.

Next, make a backup of the database (recommended), follow the instructions on the page, and click the "Update" button when ready.

Using the Command Line Updater

This update method uses all of the same thoroughly-tested underpinnings and processes as the web updater, but does so all within a system command to the Yii console.

Advantages

Safer to use, especially for large, mission-critical systems. The success of a software update or upgrade does not hinge upon a web request completing successfully.

Disadvantages

More difficult to use; requires shell access to the web server. The console command must be run as the same system user that owns all the files and directories of X2CRM, in order to circumvent permission issues.

How to use

While in via SSH, or in a terminal emulator, or in a script, run the following command in the protected subdirectory of X2CRM:

./yiic update app

Note the following optional arguments to the command line updater:

--force

Use "1" for this option (i.e. --force=1) to proceed with the update even if compatibility issues were detected. Default: 0 (halt if compatibility issues were detected).

--backup

Use "1" to specify that a database backup should be performed first before updating, "0" to proceed without creating a backup first. Default: 1.

--lock

Use "1" to specify that the application should be locked during the final installation of the update package to prevent use, "0" to permit use even when changes to the app are being applied. Default: 0.

If running the command gives you the following message:

The updater is now up-to-date and compliant with the updates server. Re-run the command to proceed.

That indicates that the updater first needed to self-update, and succeeded in doing so. You should then run the same command again to complete the update.

Using the Automatic Updater

Auto-update scheduler on the Updater Settings page

This method uses the command line updater, but runs the command on your behalf using the local cron[[wikipedia:Cron]]: a job scheduler for UNIX-like operating systems daemon. To configure, go to the Updater Settings Page, check the box titled "Update Automatically", and save. If you have Professional Edition, you can manage the update schedule in addition to other automated tasks from the Cron Table settings page, which can also be found under "System Settings" in the Admin page.

Advantages

Automates the entire process of updating.

Disadvantages

Only available on UNIX-like systems that have a cron[[wikipedia:Cron]]: a job scheduler for UNIX-like operating systems daemon, and which permit user access to the cron[[wikipedia:Cron]]: a job scheduler for UNIX-like operating systems table.

Performing "Offline" Updates

In some server environments, the updater may not be able to access the updates server by itself, due to (for example) the PHP or network configuration, or because the "zip" PHP extension (requried) is unavailable or has been disabled. It is in these situations necessary to download the update package manually, unpack on the server (or unpack and upload the files to the server), and then run the updater. When performing updates in this way, the updater will not need to access the updates server, but will use the manually-downloaded and extracted package files present in the local filesystem.

Obtaining the Update Package

First, use the link in the updater settings page entitled "Latest Update Package for Version..." to download the update package If the link in the updater settings page does not work, use the following URL to download the update package (if using Open Source Edition), replacing {version} with your X2CRM installation's version:

https://x2planet.com/installs/updates/{version}/none

If using Professional Edition, and your download link is not working, contact customer support.

Unpacking the Update

System flash message displaying notification of pending update.

The update package will be a zip archive. The contents of this archive must be extracted into a folder called "update" in the web root of the CRM. So, in other words, after extracting and uploading to the server, there should be a folder called "update" alongside "protected", "assets", "framework" (etc) containing the following (from the base level of the zip archive):

manifest.json

A file containing important data to be used by the updater

contents.md5

A message digest file containing MD5 checksums of all files in the archive excluding itself.

source

A directory, whose internal structure mimics that of X2CRM, containing files that were changed in the update. If there were no files changed (i.e. only database changes were in the update), this directory will not be present.

data

A directory containing flat data files. These files are typically not used by the updater, but are useful if performing manual updates or recovery.

Applying the Update

Steps of the update process in the event of manually-extracted package (or resuming a preexisting update)

After unpacking, if you run the command line updater, it should (if no problems arise, i.e. incompatible package version) apply the update straight away. Otherwise, log into X2CRM, and you should see a system flash message notifying you of an update in progress. This message shows up whenever there is an update package folder ("update") present on the server. If you then go to the updater page, once the page has finished loading, you should see that there only remain the steps "Review and confirm changes", "Apply changes" and "Reload" remain to be completed. To complete the update, when ready, click "Update"; otherwise, to delete the "update" folder and cancel the update, click "Cancel" either on the updater page or in the system flash message.

Performing Updates Manually

While it is preferable to avoid manually applying changes to X2CRM, and you may never need to, it may in some circumstance or another actually be necessary to do so. For all such situations, the raw update data files generated by the X2CRM update build process are bundled with every update package to help you perform the operations described in "Introduction". This section will describe how to perform each, by itself.

The manifest and the important data it contains

If you so choose, you may write a script to perform all of the operations for an update in one fell swoop. If so, and if the preferred scripting language of choice has a JSON-parsing library available, you will be able to save yourself a lot of effort by using the main data file to load all of the necessary data for the update. It is not necessary to perform the actions in the language of the examples in this section (bash/shell).

The structure the manifest

In updates, the contents of the plain-text file manifest.json are a JSON-encoded object containing the following properties:

data

An array containing the data corresponding to each intermediate version. It is, in essence, all the data stored in the files in update/data. If the update is for an installation of X2CRM that is only one version behind, it will contain only one element. Each element of this array is itself an array containing respective data for the intermediate version.

versions

An array listing the version numbers of each intermediate version in the update. Similar to data, this array will contain only one element if updating to the next version, and that version will be the next version.

deletionList

An array containing the cumulative list of deletions of obsolete files across versions through which the update traverses.

fileList

An array containing the cumulative list of files added or changed across versions through which the update traverses.

fromVersion

A string, containing the version number that the update is from, or in other words, the version of X2CRM before the package is applied. This must be the same as the version of installation to which the package is being applied. If this is not so, the updater utility would exit immediately, as applying the changes in the update would have potentially harmful results to data and/or compatibility with future updates.

targetVersion

A string, containing the version number that the update is to, or in other words, the version of X2CRM before the package is applied.

updaterVersion

A string denoting the version of the updater utility for which the package was compiled. If the structure of the update package or of the manifest changes, the updater must also change to accommodate and recognize the changes; else it might try to access properties that don't exist, or ignore important ones that do. This is the single property of manifest.json that should never change.

scenario

A string, which should be "update" if the package is for an update. In the case of upgrades, which use most of the same functionality of the updater, it should be "upgrade".

targetEdition

A string denoting the edition of the X2CRM installation after the package is applied. It should be "opensource" for open source edition, "pro" for professional edition. In the case of an upgrade package, it differs from the fromEdition property.

fromEdition

A string denoting the edition of the X2CRM installation before the package is applied.

buildDate

An integer denoting the timestamp of the release date of the version of X2CRM denoted by targetVersion

The structure of the "data" array

Each version's data, contained in each element of the data, contains the following nested array properties:

fileList

The list of files added for the version

deletionList

The list of files deleted in the version

sqlList

A list of SQL commands to bring the database structure and contents into compliance with the version from the previous version

sqlForce

A prerequisite list of SQL commands that should be run before those in sqlList, and if any fail, the failures should be ignored. These are added to updates whenever it is necessary to normalize the database schema, i.e. in the event that some installations' databases may not be consistent with others, but the commands might fail on every other system.

version

The version number

edition

The edition

migrationScripts

A list of PHP scripts that would be run in an update for this version. These are a subset of fileList and are typically used for advanced database operations, i.e. restructuring values of JSON-encoded fields, which are extremely impractical (or impossible) to perform using raw SQL commands. The files contained in the list all have paths that look like "protected/migrations/{version}/{timestamp}-name-of-operations.php". Note, if this array is not empty, you may need to reverse-engineer or otherwise copy and re-purpose/re-write any migration scripts listed in order to perform their necessary operations. This is because they may depend on running within a Yii Framework environment, i.e. will contain references to components of the Yii application singleton such as "db".

What you will need for the update

Among this data, what you will need: deletionList, which will give you the full list of files that need to be deleted, fileList, which will give you the full list of files to copy from the update/source directory, updaterVersion/targetVersion/buildDate, which give you values with which to update the main X2CRM configuration file, and, within each element of data, the sub-array properties sqlList, sqlForce and migrationScripts, with which you will construct the list of database commands to run.

A procedure for manually updating X2CRM, with data from manifest.json, thus might look like this:

1. For each element of data (or in other words, each version's changes):
   1. Run all the SQL commands in its corresponding sqlForce sub-array, ignoring errors
   2. Run all SQL commands in its sqlList array
   3. Perform database operations that would be equivalent to those performed in migration scripts
2. For each file in the cumulative list fileList, copy it from the update/source directory into its analogous location in the web root of X2CRM, creating parent directories as necessary if they don't already exist.
3. Delete all files in the cumulative list deletionList from the web root
4. Clear the data cache files in protected/runtime/cache and the temporary asset folders in assets/
5. Clear all session data (x2_sessions) and auth cache data (x2_auth_cache).
6. Update the main X2CRM configuration (protected/config/X2Config.php) file, setting:
   1. $buildDate to the buildDate property,
   2. $version to the targetVersion property, and
   3. $updaterVersion to the updaterVersion property.

Enacting Database Changes

Inside the update/sqlList and update/sqlForce folder is a list of .sql files, named according to their corresponding version's update, that can essentially be run as SQL scripts to perform the update. Note, however, that if the update traverses more than one version, the scripts must be run in the order of their corresponding version. This is because, in the database changes for any version, it is assumed that the database schema and contents conform to the version that immediately preceded it. The PHP function version_compare, and the method "parse_version", in the pkg_resources module of setuptools, should correctly identify which version should come first. Note, the "versions" property of the manifest will contain the list of versions in the correct order.

Furthermore, note, "sqlList" and "sqlForce" may not encompass all necessary database changes; if there are any migration scripts in the update, the operations they perform will similarly need to be performed in the current update (see the description of migrationScripts in "the structure of the 'data' array").

In summary, for each version traversed in the update, the following operations should be run in order, with {version} being the version number of the version in question:

1. Run all of the SQL commands in the data/sqlForce/{version}.sql file, in the order they appear in the file. Ignore if any commands fail. Note, if passed to the MySQL client as a script, any failures might trigger a halt in execution and thus skip the remainder of the commands.
2. Run all of the SQL commands in the data/sqlList/{version}.sql file.
3. Perform operations equivalent to those in each of the migration scripts. The first part of a script file name is the timestamp it was added to the codebase, and this also dictates the order in which they should be run.

Copying new files

Lists of files added in each intermediate version are stored in the folder update/data/fileList and named like "{version}.txt" ({version} being the version to which it applies). However, if the rsync utility is available on the system, you will not need these files, as the contents of update/source are complete can be safely copied into the web root as follows (executed in the web root):

rsync -avc update/source/ ./

Otherwise, you will need to use xargs and cp, two utilities that should be available on every UNIX-like system. Run the following command, ignoring errors:

for f in update/data/fileList/*; do xargs -a $f -d"\n" cp update/source/"$f" ./"$f" ; done

Deleting old files

If there is only one version through which to update, this is very simple, and can be performed as follows (if the version is 3.6.3, for example):

xargs -a update/data/deletionList/3.6.3.txt -d"\n" rm -fv

Otherwise, if there are multiple versions, this may be a more tricky proposition. The necessity arises to create a cumulative list, because there might be the possibility that a file is deleted in one version and re-established in a new version. Nevertheless, while this is possible, it is not very likely, and in most cases it is safe to simply iterate over all the deletionList data files and delete all the files listed therein.