Software Updates and Upgrades

Top-level overview of the update process

Introduction

The following things happen in typical a software update (or upgrade):

• Files that are new in the new version are added
• Files that have been changed in the new version are replaced with the new versions of those files
• Files that are no longer used in the new version are deleted
• Changes to the database are made to make the database compatible with the new version of X2CRM
• The main configuration file, X2Config.php, is edited to reflect the new state of the application.
• X2CRM data caches are cleared

These operations are typically handled by the updater utility (UpdaterBehavior) and controlled via the in-app web-based updater or the command line updater. This article will cover both how to use said interfaces to the updater, and how to perform updates manually if the need arises, in addition to troubleshooting and recovering from an update that did not complete cleanly.

Update Notifications

System flash message displaying software update notification.

To get notified of software updates, unless already using X2CRM Professional Edition, one must first "subscribe" to them. This can be done via checking the box "Notify me of software updates" in the installation form. If X2CRM is already installed, this can be done by going to "Updater Settings" from the Admin page and enabling them. If using Professional Edition, these steps are not not necessary.

Once that has been completed, set the update notification interval as desired in "Updater Settings".

Update Methods

Using the Web Updater

The updater utility page.

The web updater can be accessed using the link in the notification message at the top of the window in versions later than 3.5.5, or by going to the Admin page and clicking the "Update" button at the top, in earlier versions.

Advantages

It is generally quick and is the easiest method to use, and it does not require shell/SSH access to the web server.

Disadvantages

It is not recommended for use on large installations of X2CRM, i.e. containing over a million records.

Explanation

The inherent weakness of the web updater (or any web-based updater to any web application) is that it must always rely upon web requests to the server, and it thus must perform extremely critical operations for the update all within the scope of a web request. Web requests, it goes without saying, are not always reliable. For example, one's internet connection might get cut off due being on a wireless network and encountering interference. Furthermore, if any operations take an exceedingly long amount of time, e.g. because of the size of the data set on which the update must operate, the web request for those operations may take longer than the configured server timeout (maximum request time). This would potentially cut short the execution of a critical update stage, and thus the update would require manual completion and recovery at best, or completely restoring it to a backup at worst.

How to use

First, navigate to the web updater page, and do not cancel (stop or hit your browser's "back" button) until the page has loaded. If the updater utility has undergone changes and needs to self-update to be compatible with the software updates server, this is when it will do so. Thus, it is critical that you do not interrupt this process. If the page does not load, or displays the error message that dependencies could not be retrieved, you may have to #Refreshing_and_Updating_the_Updater_Utility manually patch the updater and perform an "offline" update.

There are three ways of getting to the updater page. One is to simply click the link given in the update notification flash message, if available. The other is to click on the "Update" button at the top of the admin page. If software update notifications are not enabled, one can manually check for and install an update by going to "Update X2CRM" under "System Settings" in the Admin page.

Next, make a backup of the database (recommended), follow the instructions on the page, and click the "Update" button when ready.

Using the Command Line Updater

This update method uses all of the same thoroughly-tested underpinnings and processes as the web updater, but does so all within a system command to the Yii console.

Advantages

Safer to use on large, mission-critical systems. The success of a software update (or upgrade) does not hinge upon a single web request completing successfully.

Disadvantages

More difficult to use; requires shell access to the web server. The console command must be run as the same system user that owns all the files and directories of X2CRM, in order to circumvent permission issues.

How to use

While in via SSH, or in a terminal emulator, or in a script, run the following command in the protected subdirectory of X2CRM:

./yiic update app

Note the following optional arguments to the command line updater:

--force

Use "1" for this option (i.e. --force=1) to proceed with the update even if compatibility issues were detected. Default: 0 (halt if compatibility issues were detected).

--backup

Use "1" to specify that a database backup should be performed first before updating, "0" to proceed without creating a backup first. Default: 1.

--lock

Use "1" to specify that the application should be locked during the final installation of the update package to prevent data entry during the sensitive moments when the database is being altered and the files updated, "0" to permit use even when changes to the app are being applied. Default: 0.

If running the command gives you the following message:

The updater is now up-to-date and compliant with the updates server. Re-run the command to proceed.

That indicates that the updater first needed to self-update, and succeeded in doing so. You should then run the same command again to complete the update.

Using the Automatic Updater

Auto-update scheduler on the Updater Settings page

This method uses the command line updater, but runs the command on your behalf using the local cron[[wikipedia:Cron]]: a job scheduler for UNIX-like operating systems daemon. To configure, go to the Updater Settings Page, check the box titled "Update Automatically", and save. If you have Professional Edition, you can manage the update schedule in addition to other automated tasks from the Cron Table settings page, which can also be found under "System Settings" in the Admin page.

Advantages

Automates the entire process of updating.

Disadvantages

Only available on UNIX-like systems that have a cron[[wikipedia:Cron]]: a job scheduler for UNIX-like operating systems daemon, and which permit user access to the cron[[wikipedia:Cron]]: a job scheduler for UNIX-like operating systems table.

Performing "Offline" Updates

In some server environments, the updater may not be able to access the updates server by itself, due to (for example) the PHP or network configuration, or because the "zip" PHP extension (requried) is unavailable or has been disabled. It is in these situations necessary to download the update package manually, unpack on the server (or unpack and upload the files to the server), and then run the updater. When performing updates in this way, the updater will not need to access the updates server, but will use the manually-downloaded and extracted package files present in the local filesystem.

Refreshing and Updating the Updater Utility

Before starting, note that the updater itself may have undergone changes; the X2CRM update process is subject to change. To address this technical challenge, the updater utility within X2CRM would normally first attempt to download the latest version of itself, in order to be compatible with the updates server. However, if your X2CRM installation is running in a hosting environment that prevents it from making outbound HTTP requests (the requirements check script should tell you if this is the case), you will need to do this manually. That is because, in such cases, the updater will not be able to do so by itself.

To update the updater manually:

1. Determine the version of the updater utility package by visiting this link. You should see a page that's entirely blank except for a version number.
2. Download the updater package. In versions 3.6 and later, a download link for this should be available through the Updater Settings page. Otherwise, use the following URL (if in Open Source Edition):

   https://x2planet.com/installs/updater.zip

   Otherwise, if using Professional Edition, paste the following url, after replacing {key} with your license key, into your browser's address bar:

   https://x2planet.com/installs/{key}/updater-pro.zip

3. Extract the downloaded zip archive and upload the "protected" folder to your server. Merge folders and overwrite existing files when prompted.
4. Open the file protected/config/X2Config.php in a text editor and find the line that looks like this:

   $updaterVersion = '3.5.2';
   

5. Change the text inside the pair of single quotes on that line so that it is the current updater version obtained in step #1.
6. Save the file.

Obtaining the Update Package

First, use the link in the Updater Settings page entitled "Latest Update Package for Version...", to download the update package. If the link in the Updater Settings page does not work, use the following URL to download the update package (if using Open Source Edition), replacing {version} with your X2CRM installation's version:

https://x2planet.com/installs/updates/{version}/none

If using Professional Edition, and your download link is not working, contact customer support.

Unpacking the Update

System flash message displaying notification of pending update.

The update package will be a zip archive. The contents of this archive must be extracted into a folder called "update" in the web root of the CRM. So, in other words, after extracting and uploading to the server, there should be a folder called "update" alongside "protected", "assets", "framework" (etc) containing the following (from the base level of the zip archive):

manifest.json

A file containing important data to be used by the updater

contents.md5

A message digest file containing MD5 checksums of all files in the archive excluding itself.

source

A directory, whose internal structure mimics that of X2CRM, containing files that were changed in the update. If there were no files changed (i.e. only database changes were in the update), this directory will not be present.

data

A directory containing flat data files. These files are typically not used by the updater, but are useful if performing manual updates or recovery.

Applying the Update

Steps of the update process in the event of manually-extracted package (or resuming a preexisting update)

After unpacking, if you run the command line updater, it should (if no problems arise, i.e. incompatible package version) apply the update straight away. Otherwise, log into X2CRM, and you should see a system flash message notifying you of an update in progress. This message shows up whenever there is an update package folder ("update") present on the server. If you then go to the updater page, once the page has finished loading, you should see that there only remain the steps "Review and confirm changes", "Apply changes" and "Reload" remain to be completed. To complete the update, when ready, click "Update"; otherwise, to delete the "update" folder and cancel the update, click "Cancel" either on the updater page or in the system flash message.

Performing Updates Manually

While it is preferable to avoid manually applying changes to X2CRM, and you may never need to, it may in some circumstance or another actually be necessary to do so. For all such situations, the raw update data files generated by the X2CRM update build process are bundled with every update package to help you perform the operations described in "Introduction". This section will describe how to perform each, by itself.

The manifest and the important data it contains

If you so choose, you may write a script to perform all of the operations for an update in one fell swoop. If so, and if the preferred scripting language of choice has a JSON-parsing library available, you will be able to save yourself a lot of effort by using the main data file to load all of the necessary data for the update. It is not necessary to perform the actions in the language of the examples in this section (bash/shell).

The structure the manifest

In updates, the contents of the plain-text file manifest.json are a JSON-encoded object containing the following properties:

data

An array containing the data corresponding to each intermediate version. It is, in essence, all the data stored in the files in update/data. If the update is for an installation of X2CRM that is only one version behind, it will contain only one element. Each element of this array is itself an array containing respective data for the intermediate version.

versions

An array listing the version numbers of each intermediate version in the update. Similar to data, this array will contain only one element if updating to the next version, and that version will be the next version.

deletionList

An array containing the cumulative list of deletions of obsolete files across versions through which the update traverses.

fileList

An array containing the cumulative list of files added or changed across versions through which the update traverses.

fromVersion

A string, containing the version number that the update is from, or in other words, the version of X2CRM before the package is applied. This must be the same as the version of installation to which the package is being applied. If this is not so, the updater utility would exit immediately, as applying the changes in the update would have potentially harmful results to data and/or compatibility with future updates.

targetVersion

A string, containing the version number that the update is to, or in other words, the version of X2CRM before the package is applied.

updaterVersion

A string denoting the version of the updater utility for which the package was compiled. If the structure of the update package or of the manifest changes, the updater must also change to accommodate and recognize the changes; else it might try to access properties that don't exist, or ignore important ones that do. This is the single property of manifest.json that should never change.

scenario

A string, which should be "update" if the package is for an update. In the case of upgrades, which use most of the same functionality of the updater, it should be "upgrade".

targetEdition

A string denoting the edition of the X2CRM installation after the package is applied. It should be "opensource" for open source edition, "pro" for professional edition. In the case of an upgrade package, it differs from the fromEdition property.

fromEdition

A string denoting the edition of the X2CRM installation before the package is applied.

buildDate

An integer denoting the timestamp of the release date of the version of X2CRM denoted by targetVersion

The structure of the "data" array

Each version's data, contained in each element of the data, contains the following nested array properties:

fileList

The list of files added for the version

deletionList

The list of files deleted in the version

sqlList

A list of SQL commands to bring the database structure and contents into compliance with the version from the previous version

sqlForce

A prerequisite list of SQL commands that should be run before those in sqlList, and if any fail, the failures should be ignored. These are added to updates whenever it is necessary to normalize the database schema, i.e. in the event that some installations' databases may not be consistent with others, but the commands might fail on every other system.

version

The version number

edition

The edition

migrationScripts

A list of PHP scripts that would be run in an update for this version. These are a subset of fileList and are typically used for advanced database operations, i.e. restructuring values of JSON-encoded fields, which are extremely impractical (or impossible) to perform using raw SQL commands. The files contained in the list all have paths that look like "protected/migrations/{version}/{timestamp}-name-of-operations.php". Note, if this array is not empty, you may need to reverse-engineer or otherwise copy and re-purpose/re-write any migration scripts listed in order to perform their necessary operations. This is because they may depend on running within a Yii Framework environment, i.e. will contain references to components of the Yii application singleton such as "db".

What you will need for the update

Among this data, what you will need: deletionList, which will give you the full list of files that need to be deleted, fileList, which will give you the full list of files to copy from the update/source directory, updaterVersion/targetVersion/buildDate, which give you values with which to update the main X2CRM configuration file, and, within each element of data, the sub-array properties sqlList, sqlForce and migrationScripts, with which you will construct the list of database commands to run.

A procedure for manually updating X2CRM, with data from manifest.json, thus might look like this:

1. For each element of data (or in other words, each version's changes):
   1. Run all the SQL commands in its corresponding sqlForce sub-array, ignoring errors
   2. Run all SQL commands in its sqlList array
   3. Perform database operations that would be equivalent to those performed in migration scripts
2. For each file in the cumulative list fileList, copy it from the update/source directory into its analogous location in the web root of X2CRM, creating parent directories as necessary if they don't already exist.
3. Delete all files in the cumulative list deletionList from the web root
4. Clear the data cache files in protected/runtime/cache and the temporary asset folders in assets/
5. Clear all session data (x2_sessions) and auth cache data (x2_auth_cache).
6. Update the main X2CRM configuration (protected/config/X2Config.php) file, setting:
   1. $buildDate to the buildDate property,
   2. $version to the targetVersion property, and
   3. $updaterVersion to the updaterVersion property.

Enacting Database Changes

Inside the update/sqlList and update/sqlForce folder is a list of .sql files, named according to their corresponding version's update, that can essentially be run as SQL scripts to perform the update. Note, however, that if the update traverses more than one version, the scripts must be run in the order of their corresponding version. This is because, in the database changes for any version, it is assumed that the database schema and contents conform to the version that immediately preceded it. The PHP function version_compare, and the method "parse_version", in the pkg_resources module of setuptools, should correctly identify which version should come first. Note, the "versions" property of the manifest will contain the list of versions in the correct order.

Furthermore, note, "sqlList" and "sqlForce" may not encompass all necessary database changes; if there are any migration scripts in the update, the operations they perform will similarly need to be performed in the current update (see the description of migrationScripts in "the structure of the 'data' array").

In summary, for each version traversed in the update, the following operations should be run in order, with {version} being the version number of the version in question:

1. Run all of the SQL commands in the data/sqlForce/{version}.sql file, in the order they appear in the file. Ignore if any commands fail. Note, if passed to the MySQL client as a script, any failures might trigger a halt in execution and thus skip the remainder of the commands.
2. Run all of the SQL commands in the data/sqlList/{version}.sql file.
3. Perform operations equivalent to those in each of the migration scripts. The first part of a script file name is the timestamp it was added to the codebase, and this also dictates the order in which they should be run.

Copying new files

Lists of files added in each intermediate version are stored in the folder update/data/fileList and named like "{version}.txt" ({version} being the version to which it applies), each file on its own line in that file. However, if the rsync utility is available on the system, you will not need these files, as the contents of update/source are complete can be safely copied into the web root as follows (executed in the web root):

rsync -avc update/source/ ./

Otherwise, you will need to use xargs and cp, two utilities that should be available on every UNIX-like system. Run the following command, ignoring errors:

for f in update/data/fileList/*; do xargs -a $f -d"\n" cp update/source/"$f" ./"$f" ; done

Deleting old files

Files to be deleted are listed in a carriage-return-delineated list of paths relative to the web root, in files located at paths that look like this: update/data/deletionList/{version}.txt

If there is only one version through which to update, the operation is very simple, and can be performed in one UNIX/Linux shell command as follows (if the version is 3.6.3, for example):

xargs -a update/data/deletionList/3.6.3.txt -d"\n" rm -fv

Otherwise, if there are multiple versions, this may be a more tricky proposition. The necessity arises to create a cumulative list, because there might be the possibility that a file is deleted in one version and re-established in a new version. Nevertheless, while this is possible, it is not very likely, and in most cases it is safe to simply iterate over all the deletionList data files and delete all the files listed therein.

Editing the Configuration File

This final step in the update process, before refreshing, should not be done manually unless you are completely certain that all the files (not counting customizations) and the structure of the database are equivalent to a new installation at the same version as the one you updated to.

To update the configuration manually:

1. Open the file protected/config/X2Config.php in a text editor
2. Find the line with the $version variable in it. It should look like this (but with the version from which you are updating):

   $version = '3.5.2';
   

3. Change the value in quotes to the current version being updated to, and save the file.

Clearing caches and refreshing the application

It is generally always a good idea to clear the application's server-side cache and other ephemeral files/data after updating. This is taken care of automatically by the built-in updater, but outside of that, it can be performed manually as follows:

clear the data cache

delete all ".bin" files in protected/runtime/cache.

clear web assets

delete all sub-folders of assets (in the web root)

clear the "auth cache"

Flush x2_auth_cache, i.e. run

DELETE FROM `x2_auth_cache` WHERE 1;

log all users out

Flush x2_sessions, i.e. run

DELETE FROM `x2_sessions` WHERE 1;

Troubleshooting

User interface (UI) "bugs" after the update

It is extremely common (and easily remedied) that after an important update, certain features of the user interface of X2CRM may not work properly. This is caused, in nearly all cases, by the web browser's cache, which contains out-of-date copies of X2CRM's essential user interface files. To solve these problems, the first step to try is clearing the cache. A step-by-step guide to doing so is given in wikihow:Clear-Your-Browser's-Cache.

Errors in the updater itself

When all else fails, the updater may have to be patched. See Refreshing and Updating the Updater Utility.

General Procedures to Try

It may sometimes be possible that the update did not finish cleanly. In such cases, it will be necessary to manually re-perform some of the final operations in the update, described in Introduction.

Refresh the Source Code

This step is not really the best step to try first, as it is usually more time-consuming. It entails re-uploading the entire application, in order to reset all non-ephemeral files to what they should be in a fresh installation of X2CRM. To do this:

1. Download the refresh package. There should be a download link for this in the Updater Settings page. Otherwise, if that doesn't work and you're using Open Source edition, use this link, or if using Professional Edition, paste the following url (after replacing {key} with your license key) into your browser's address bar:

   https://x2planet.com/installs/{key}/refresh-pro.zip

2. Upload the contents of the downloaded zip archive to the web root of your X2CRM installation, overwriting existing files.

Clear Caches

See "Clearing Caches..." for details.

Regenerate the Configuration

Again, this should not be done unless you are completely certain that all the files (not counting customizations) and the structure of the database are equivalent to a new installation at the same version as the one you updated to. Refer to Editing the Configuration File.

Upgrading

Using the Web Interface

Software upgrade page, in the first step (registration)

To upgrade to Professional Edition

1. Purchase X2CRM Professional Edition Download. Note the registration info in the activation email.
2. Go to the Admin page in X2CRM and find "Upgrade X2CRM" under "System Settings".
3. In the Upgrade page, enter your registration info into the form and click "Register" to continue. Once complete, you will see a list of changes similar to those displayed when performing an update.
4. Review the changes to be applied and then proceed by clicking "Upgrade", similar to performing an update.

Manually Upgrading

The process of upgrading is very similar to that of updating, and uses all of the same underpinnings as the updater. Thus, almost all of the above information and procedures will apply to upgrades. However, note the following crucial differences:

• X2CRM must be at the latest available version in order to upgrade.
• To obtain the upgrade package, you must be an owner of an X2CRM Professional Edition Download-type license. Contact customer support for further details.
• There will be only one "intermediate" version. The changes to apply will move X2CRM from a version to that same version, applying instead the necessary changes to switch its edition from Open Source to Professional.
• There are no file deletions.
• The properties of the package manifest. In upgrades, the properties of the data object are sqlUpgrade and fileUpgrade instead of sqlList, fileList, etc. Note also, the update data is in aptly-named sub-folders of update/data as well, and there are no sqlForce or migrationScripts lists to handle.
• The configuration is not changed. Instead, the value stored in the field unique_id of the admin record in the database table x2_admin must be changed to the license key, and the edition field must be changed to "pro". To perform these both, run the following command on the X2CRM database:

  UPDATE `x2_admin` SET `unique_id`='{license key}',`edition`='pro' WHERE `id`=1;
  

• It is not necessary to clear caches or invalidate user sessions when done. These operations are performed in updates to eliminate ephemeral (session-ingrained) data that may no longer be applicable and thus cause problems. Upgrades will always tend to introduce far less dramatic changes to the system, and thus should never require these measures.