Software Updates and Upgrades

Introduction

The following things happen in typical a software update (or upgrade):

• Files that are new in the new version are added
• Files that have been changed in the new version are replaced with the new versions of those files
• Files that are no longer used in the new version are deleted
• Changes to the database are made to make the database compatible with the new version of X2Engine
• The main configuration file, X2Config.php, is edited to reflect the new state of the application.
• X2Engine data caches are cleared

These operations are typically handled by the updater utility component UpdaterBehavior and controlled via the in-app web-based updater or the command line updater. This article will cover both how to use said interfaces to the updater, and how to perform updates manually or with the "offline" method if the need arises, in addition to troubleshooting and recovering from an update that did not complete cleanly.

Disclaimer

This article covers the standard and recommended update/upgrade methods, in addition to alternate methods and supplemental procedures. As general rules and precautions for using this article:

1. If you do not fully and completely understand any step of any instructions given in this article, do not follow them; ask for help instead on the forums or through contacting X2Engine Technical Support if you are an existing customer.
2. You should always make a backup of all of your files and a full backup copy of your database, preferably in a format (i.e. SQL script) that can be used to create a verbatim/carbon copy of the original database, with all of the features of the original schema, before attempting any steps of the update process manually.
3. Some of the procedures outlined in this article involve manually creating or modifying files. Note that the same filesystem prerequisite for installation, namely that the file ownership must match the user under which PHP executes on the web server, applies to these steps.

Furthermore, in regard to uploading files: note that any instructions to upload folders into X2Engine, and replace/merge contents, refer to replacing files in X2Engine with files in the archive at analogous paths. For example, if you have a zip file that contains a folder protected, and inside of that is a folder components, and inside of that is a file UpdaterBehavior.php, and the instructions are to upload/replace in X2Engine, you should upload UpdaterBehavior.php to replace the existing file of the same name in protected/components inside of X2Engine.

X2Engine Inc. cannot be held accountable for damage/errors caused by any missteps or mistakes in manually-performed update operations. If in doubt, make a sandbox/clone of your X2Engine installation by copying the files and database to a different server, and test performing the operations on that before proceeding with your production install of X2Engine.

Update Notifications

System flash message displaying software update notification.

To get notified of software updates, unless already using X2Engine Professional Edition, one must first "subscribe" to them. This can be done via checking the box "Notify me of software updates" in the installation form. If X2Engine is already installed, this can be done by going to "Updater Settings" from the Admin page and enabling them. If using Professional Edition, these steps are not not necessary.

Once that has been completed, set the update notification interval as desired in "Updater Settings".

Update Methods

Top-level overview of the update process

There are, in essence, just three basic methods of updating, and variations on each. The three main methods are:

1. web updater,
2. command line updater, and
3. manually.

In each case, an update package from the update data server is used as the authoritative list of changes to apply, but in the non-manual methods, X2Engine's own updater class does all the work applying changes from said package.

In web and command line updates, there is also the option to first download and unpack the package manually (see performing "offline" updates). This circumvents the need for the local server (on which X2Engine is hosted) to be connected to the internet. Similarly, in the case of manual updates, it is not necessary for the web server to actually have an internet connection.

Using the Web Updater

The updater utility page.

The web updater can be accessed by any of the following methods:

• Using the link in the notification message at the top of the window in versions later than 3.5.5
• By going to the Admin page and clicking the "Update" button at the top, in earlier versions, with update notifications enabled
• Through "Update X2Engine" under "System Settings" in the Admin page

How to use

First, navigate to the web updater page, and do not cancel the request (stop or hit your browser's "back" button) until the page has loaded. If the updater utility has undergone changes and needs to self-update to be compatible with the software updates server, this is when it will do so. Thus, it is critical that you do not interrupt this process. If the page does not load, or displays the error message that dependencies could not be retrieved, you may have to perform an "offline" update.

There are three ways of getting to the updater page. One is to simply click the link given in the update notification flash message, if available. The other is to click on the "Update" button at the top of the admin page. If software update notifications are not enabled, one can manually check for and install an update by going to "Update X2Engine" under "System Settings" in the Admin page.

Next, make a backup of the database and files (recommended), follow the instructions on the page, and click the "Update" button when ready.

Advantages

It is generally quick and is the easiest method to use, and it does not require shell/SSH access to the web server.

Disadvantages

It is not recommended for use on large installations of X2Engine, i.e. containing over a million records, or for updating from very old versions (i.e. a version more than five months old). This is because the final update process may be cut short by server timeout or an interruption in the network connection. This can result in an incomplete/broken update that requires recovering manually (a potentially long process) at best, or actual loss of data at worst.

Explanation

The inherent weakness of the web updater (or any web-based updater to any web application) is that it must always rely upon web requests to the server, and it thus must perform extremely critical operations for the update all within the scope of a web request. Web requests, it goes without saying, are not always reliable. For example, one's internet connection might get cut off due being on a wireless network and encountering interference. Furthermore, if any operations take an exceedingly long amount of time, e.g. because of the size of the data set (and thus indexes) on which the update must operate, or the number of database changes that need to be applied, the web request for those operations may take longer than the PHP execution time limit as configured on the webserver. This would potentially cut short the execution of a critical update stage, and thus the update would require manual recovery at best, or completely restoring X2Engine to a backup at worst.

How to Avoid Timeouts

You can usually get around this by setting the value of max_execution_time to a higher value in the web server's PHP configuration. It is highly recommended that you either disable the timeout or set it to a very generous value (such as 300 or more seconds) while attempting to update via the web updater, especially if you are many versions behind or have a high volume of data in your CRM. Also, it is generally always a good idea to create a backup copy before updating, and to download it.

Recovering From Crashes

If you downloaded the backup copy of your database, then after you encounter a failure/timeout, you can use the .sql backup script to restore the database to its original form. However, be aware, that if there are any CREATE TABLE commands in the update data (expand the Database Commands to Run list to see), you will need to manually drop the created tables in order to once again make your installation compatible with the update.

Using the Command Line Updater

This update method uses all of the same thoroughly-tested underpinnings and processes as the web updater, but does so all within a system command to the Yii console.

How to use

While in via SSH, or in a terminal emulator, or in a script, run the following command in the protected subdirectory of X2Engine:

./yiic update app

Note the following optional arguments to the command line updater:

--force

Use "1" for this option (i.e. --force=1) to proceed with the update even if compatibility issues were detected. Default: 0 (halt if compatibility issues were detected).

--backup

Use "1" to specify that a database backup should be performed first before updating, "0" to proceed without creating a backup first. Default: 1.

--lock

Use "1" to specify that the application should be locked during the final installation of the update package to prevent data entry during the sensitive moments when the database is being altered and the files updated, "0" to permit use even when changes to the app are being applied. Default: 0.

If running the command gives you the following message:

The updater is now up-to-date and compliant with the updates server. Re-run the command to proceed.

That indicates that the updater first needed to self-update, and succeeded in doing so. You should then run the same command again to complete the update.

Advantages

Safer to use on large, mission-critical systems. The success of a software update (or upgrade) does not hinge upon a single web request completing successfully.

Disadvantages

More difficult to use; requires shell access to the web server. The console command must be run as the same system user that owns all the files and directories of X2Engine, in order to circumvent permission issues when manipulating file system objects.

Using the Automatic Updater

Auto-update scheduler on the Updater Settings page

This method uses the command line updater, but runs the command on your behalf using the local cron[[wikipedia:Cron]]: a job scheduler for UNIX-like operating systems daemon. To configure, go to the Updater Settings Page, check the box titled "Update Automatically", and save. If you have Professional Edition, you can manage the update schedule in addition to other automated tasks from the Cron Table settings page, which can also be found under "System Settings" in the Admin page.

Advantages

Automates the entire process of updating.

Disadvantages

Only available on UNIX-like systems that have a cron[[wikipedia:Cron]]: a job scheduler for UNIX-like operating systems daemon, and which permit user access to the cron[[wikipedia:Cron]]: a job scheduler for UNIX-like operating systems table.

Performing "Offline" Updates

In some server environments, the updater may not be able to access the updates server by itself, due to (for example) the PHP or network configuration, or because the "zip" PHP extension (requried) is unavailable or has been disabled. It is in these situations necessary to download the update package manually, unpack on the server (or unpack and upload the files to the server), and then run the updater. When performing updates in this way, the updater will not need to access the updates server, but will use the manually-downloaded and extracted package files present in the local filesystem.

Refreshing and Updating the Updater Utility

Before starting, note that the updater itself may have undergone changes; the X2Engine update process is subject to change. To address this technical challenge, the updater utility within X2Engine would normally first attempt to download the latest version of itself, in order to be compatible with the updates server. However, this sometimes will not work properly. If your X2Engine installation is running in a hosting environment that prevents it from making outbound HTTP requests (the requirements check script should tell you if this is the case), you will need to do this manually. Also, in version 3.5.5, there was a bug in the updater utility's self-update methods.

To update the updater utility:

1. Determine the version of the updater utility package by visiting this link. You should see a page that's entirely blank except for a version number.
2. Download the updater package. In versions 3.6 and later, a download link for this should be available through the Updater Settings page. Otherwise, use the following URL (if in Open Source Edition):

   https://x2planet.com/installs/updater.zip

   Otherwise, if using Professional Edition, paste the following url, after replacing {key} with your license key, into your browser's address bar:

   https://x2planet.com/installs/{key}/updater-pro.zip

3. Extract the downloaded zip archive and upload the "protected" folder to your server. Merge folders and overwrite existing files when prompted.
4. Open the file protected/config/X2Config.php in a text editor and find the line that looks like this:

   $updaterVersion = '3.5.2';
   

5. Change the text inside the pair of single quotes on that line so that it is the current updater version obtained in step #1.
6. Save the file.

Obtaining the Update Package

First, use the link in the Updater Settings page entitled "Latest Update Package for Version...", to download the update package. If the link in the Updater Settings page does not work, use the following URL to download the update package (if using Open Source Edition), replacing {version} with your X2Engine installation's version:

https://x2planet.com/installs/updates/{version}/none

You should receive, from that URL, a zip archive file called "update.zip".

If using Professional Edition, and your download link is not working, contact customer support.

Unpacking the Update

System flash message displaying notification of pending update.

The update package will be a zip archive. The contents of this archive must be extracted into a folder called "update" in the installation directoryThe directory (folder) on the web server where X2Engine is installed. You will know that you are operating within this directory when you see a file ''constants.php'' and the following subdirectories: ''assets'', ''css'', ''custom'', ''framework'', ''images'', ''js'', ''protected'', ''themes'' and ''uploads''.. So, in other words, after extracting and uploading to the server, there should be a folder called update alongside protected, assets, framework (etc). Within that folder (update) should be the following contents:

manifest.json

A file containing important data to be used by the updater

contents.md5

A message digest file containing MD5 checksums of all files in the archive excluding itself.

source

A sub-folder/directory, whose internal structure mimics that of X2Engine, containing files that were changed in the update. If there were no files changed (i.e. only database changes were in the update), this directory will not be present.

data

A directory containing flat data files. These files are typically not used by the updater, but are useful if performing manual updates or recovery.

Applying the Update

Steps of the update process in the event of manually-extracted package (or resuming a preexisting update)

After unpacking, if you run the command line updater, it should (if no problems arise, i.e. incompatible package version) apply the update straight away. Otherwise, log into X2Engine, and you should see a system flash message notifying you of an update in progress. This message shows up whenever there is an update package folder ("update") present on the server. If you then go to the updater page, once the page has finished loading, you should see that there only remain the steps "Review and confirm changes", "Apply changes" and "Reload" remain to be completed. To complete the update, when ready, click "Update"; otherwise, to delete the "update" folder and cancel the update, click "Cancel" either on the updater page or in the system flash message.

Performing Updates Manually

While it is preferable to avoid manually applying changes to X2Engine, and you may never need to, it may in some circumstance or another actually be necessary to do so. For all such situations, the raw update data files generated by the X2Engine update build process are bundled with every update package to help you perform the operations described in "Introduction". This section will describe how to perform each of those operations.

Prerequisites

First and foremost, all shell commands listed here, and in general, any operations that create or modify files, must be run as the same system user under which the server runs PHP scripts. Otherwise, the ownership and/or permissions of files and directories will have to be changed after completing all changes.

To obtain the update package (which will be necessary), refer to "Obtaining the Update Package".

Furthermore, it is strongly advised that you first make a back-up copy of all files and of X2Engine's database. Your database backup would best be a full, exact copy, such as one that would be made by mysqldump, or the built-in export tools of PHPMyAdmin.

The X2Engine global import/export utility ("Import All Data"/"Export All Data") in Admin is not recommended by itself as the sole backup tool. This is especially important to note because of how this tool cannot re-create ancillary data such as tags or relationships, or user roles. However, using it to supplement your standard database backup method will prove useful in the event that you:

1. don't have any custom roles, relationships, tags, etc.
2. cannot manage to update X2Engine with success, and
3. wish to start over with a fresh installation.

In such a case, you will then be able to import your old data into the new installation.

Introduction: Data Files

In Unpacking the update, a rough overview of an update package's contents were given. In this section, the anatomy and purpose of each of the data files will be introduced.

Included in every update package is two copies of the following data:

• Files to copy
• Files to delete
• Database changes to enact
• Migration scripts to run, if any

The first, immediately visible place where all of this data is stored is in the file manifest.json, as a JSON-encoded array. See the next section for how to use this file.

The second place, which can be useful to users who do not wish to automate or script any step of a manual update, is in the files contained in the data directory. Each file is named after the version to which it corresponds, and contains the data required to update to that version from the version immediately preceding it.' Thus, the versions you should see in each file name are the versions above your X2Engine installation's current version, up to and including the version being updated to.

In an update, the directory will contain the following subdirectories:

fileList

Contains files for each version, each containing the list of files to copy. The format is each line containing a file path in the list.

deletionList

Contains files for each version, each containing the list of obsolete files for that version which will need to be deleted. The format is the same as in fileList.

sqlList

Contains .sql scripts corresponding to each version. Each file is formatted such that it can be directly run on the database as an SQL script, although this is not generally recommended. The list of SQL commands to run is a of the following format:

 [query 1];
 /*&*/
 [query 2];
 /*&*/
 [etc]

sqlForce

Contains .sql scripts with similar naming and formatting conventions to the sqlList scripts. See The structure of the "data" array for explanation of the purpose of sqlForce.

How to use the manifest for updating manually

If you so choose, you may write a script to perform all of the operations for an update in one fell swoop. If so, and if the preferred scripting language of choice has a JSON-parsing library available, you will be able to save yourself a lot of effort by using the main data file (manifest.json) to load all of the necessary data for the update.

The structure the manifest

In updates, the contents of the plain-text file manifest.json are a JSON-encoded object containing the following properties:

data

An array containing the data corresponding to each intermediate version. It is, in essence, all the data stored in the files in update/data. If the update is for an installation of X2Engine that is only one version behind, it will contain only one element. Each element of this array is itself an array containing respective data for the intermediate version.

versions

An array listing the version numbers of each intermediate version in the update. Similar to data, this array will contain only one element if updating to the next version, and that version will be the next version.

deletionList

An array containing the cumulative list of deletions of obsolete files across versions through which the update traverses.

fileList

An array containing the cumulative list of files added or changed across versions through which the update traverses.

fromVersion

A string, containing the version number that the update is from, or in other words, the version of X2Engine before the package is applied. This must be the same as the version of installation to which the package is being applied. If this is not so, the updater utility would exit immediately, as applying the changes in the update would have potentially harmful results to data and/or compatibility with future updates.

targetVersion

A string, containing the version number that the update is to, or in other words, the version of X2Engine before the package is applied.

updaterVersion

A string denoting the version of the updater utility for which the package was compiled. If the structure of the update package or of the manifest changes, the updater must also change to accommodate and recognize the changes; else it might try to access properties that don't exist, or ignore important ones that do. This is the single property of manifest.json that should never change.

scenario

A string, which should be "update" if the package is for an update. In the case of upgrades, which use most of the same functionality of the updater, it should be "upgrade".

targetEdition

A string denoting the edition of the X2Engine installation after the package is applied. It should be "opensource" for open source edition, "pro" for professional edition. In the case of an upgrade package, it differs from the fromEdition property.

fromEdition

A string denoting the edition of the X2Engine installation before the package is applied.

buildDate

An integer denoting the timestamp of the release date of the version of X2Engine denoted by targetVersion

The structure of the "data" array

In updates, each version's data, contained in each element of the data, contains the following nested array properties:

fileList

The list of files added for the version

deletionList

The list of files deleted in the version

sqlList

A list of SQL commands to bring the database structure and contents into compliance with the version from the previous version

sqlForce

A prerequisite list of SQL commands that should be run before those in sqlList, and if any fail, the failures should be ignored. These are added to updates whenever it is necessary to normalize the database schema, i.e. in the event that some installations' databases may not be consistent with others, but the commands might fail on every other system. These SQL commands were intended to be "attempted"; they are not guaranteed to succeed on all installations. An example of when this is necessary would be adding a foreign key constraint when in previous versions the storage engine was left to the database's default settings in old versions. MyISAM does not natively support foreign key constraints. Thus, to standardize this across all installations, including installations on servers where the storage engine defaulted to MyISAM, the constraint must first be dropped (in a sqlForce command) before the relevant tables are changed to the InnoDB storage engine, and the constraint is added back again, in the usual sqlList.

version

The version number

edition

The edition

migrationScripts

A list of PHP scripts that would be run in an update for this version. These are a subset of fileList and are typically used for advanced database operations, i.e. restructuring values of JSON-encoded fields, which are extremely impractical (or impossible) to perform using raw SQL commands. The files contained in the list all have paths that look like "protected/migrations/{version}/{timestamp}-name-of-operations.php". Note, if this array is not empty, and the runmigrationscript Yii command does not work (or isn't available yet), you may need to reverse-engineer or otherwise copy and re-purpose/re-write any migration scripts listed in order to perform their necessary operations. This is because they often depend on running within a Yii Framework environment, i.e. will contain references to components of the Yii application singleton such as "db".

Also note that in upgrades, fileList and sqlList are replaced by fileUpgrade and sqlUpgrade (respectively), and deletionList, sqlForce and migrationScripts are missing (because they aren't necessary).

What you will need for the update, if using the manifest

Among this data, what you will need: deletionList, which will give you the full list of files that need to be deleted, fileList, which will give you the full list of files to copy from the update/source directory, updaterVersion/targetVersion/buildDate, which give you values with which to update the main X2Engine configuration file, and, within each element of data, the sub-array properties sqlList, sqlForce and migrationScripts, with which you will construct the list of database commands to run.

A procedure for manually updating X2Engine, with data from manifest.json, thus might look like this:

1. For each element of data (or in other words, each version's changes):
   1. Run all the SQL commands in its corresponding sqlForce sub-array, ignoring errors
   2. Run all SQL commands in its sqlList array
   3. Perform database operations that would be equivalent to those performed in migration scripts
2. For each file in the cumulative list fileList, copy it from the update/source directory into its analogous location in the installation directoryThe directory (folder) on the web server where X2Engine is installed. You will know that you are operating within this directory when you see a file ''constants.php'' and the following subdirectories: ''assets'', ''css'', ''custom'', ''framework'', ''images'', ''js'', ''protected'', ''themes'' and ''uploads''. of X2Engine, creating parent directories as necessary if they don't already exist.
3. Delete all files in the cumulative list deletionList from the installation directoryThe directory (folder) on the web server where X2Engine is installed. You will know that you are operating within this directory when you see a file ''constants.php'' and the following subdirectories: ''assets'', ''css'', ''custom'', ''framework'', ''images'', ''js'', ''protected'', ''themes'' and ''uploads''.
4. Clear the data cache files in protected/runtime/cache and the temporary asset folders in assets/
5. Clear all session data (x2_sessions) and auth cache data (x2_auth_cache).
6. Update the main X2Engine configuration (protected/config/X2Config.php) file, setting:
   1. $buildDate to the buildDate property,
   2. $version to the targetVersion property, and
   3. $updaterVersion to the updaterVersion property.

Enacting Database Changes

Inside the update/sqlList and update/sqlForce folder is a list of .sql files, named according to their corresponding version's update, that can essentially be run as SQL scripts to perform the update. Note, however, that if the update traverses more than one version, the scripts must be run in the order of their corresponding version. This is because, in the database changes for any version, it is assumed that the database schema and contents conform to the version that immediately preceded it. The PHP function version_compare, and the method "parse_version", in the pkg_resources module of setuptools, should correctly identify which version should come first. Note, the "versions" property of the manifest will contain the list of versions in the correct order.

Furthermore, note, "sqlList" and "sqlForce" may not encompass all necessary database changes; if there are any migration scripts in the update, the operations they perform will similarly need to be performed in the current update (see the description of migrationScripts in "the structure of the 'data' array").

In summary, for each version traversed in the update, the following operations should be run in order, with {version} being the version number of the version in question:

1. Run all of the SQL commands in the data/sqlForce/{version}.sql file, in the order they appear in the file. Ignore if any commands fail. Note, if passed to the MySQL client as a script, any failures might trigger a halt in execution and thus skip the remainder of the commands.
2. Run all of the SQL commands in the data/sqlList/{version}.sql file.
3. Run migration scripts using ./yiic runmigrationscript {path}, if any. The first part of a script file name is the timestamp it was added to the codebase, and this also dictates the order in which they should be run.

Copying new files

Files added/changed in the new version are stored in the update/source directory. In an update, they should be copied into the installation directoryThe directory (folder) on the web server where X2Engine is installed. You will know that you are operating within this directory when you see a file ''constants.php'' and the following subdirectories: ''assets'', ''css'', ''custom'', ''framework'', ''images'', ''js'', ''protected'', ''themes'' and ''uploads''., replacing any existing files of the same name/path. If the rsync utility is available on the system, you can use one command to safely copy these files into the installation directoryThe directory (folder) on the web server where X2Engine is installed. You will know that you are operating within this directory when you see a file ''constants.php'' and the following subdirectories: ''assets'', ''css'', ''custom'', ''framework'', ''images'', ''js'', ''protected'', ''themes'' and ''uploads''. as follows (executed in that same directory):

rsync -avc update/source/ ./

Otherwise, you will need to use cat and cp, two utilities that should be available on every UNIX-like system, and the lists of files added in each intermediate version. These lists are stored in the folder update/data/fileList and named like "{version}.txt" ({version} being the version to which it applies), each file on its own line in that file. Run the following command, ignoring errors:

for f in update/data/fileList/*; do for l in $(cat $f); do cp update/source/"$l" ./"$l" ; done ; done

Deleting old files

Files to be deleted are listed in a similar manner as the files to be added: one file per line. The list of files to delete are located at files with paths that look like this: update/data/deletionList/{version}.txt So, for an update from 5.5 to 5.7 with intermediate version 5.6 (for instance), the files would be update/data/deletionList/5.6 and update/data/deletionList/5.7.

If there is only one version through which to update, the operation is very simple, and can be performed in one UNIX/Linux (bash) shell command as follows (if the version is 3.6.3, for example):

xargs -a update/data/deletionList/3.6.3.txt -d"\n" rm -fv

Otherwise, if there are multiple versions, this may be a more tricky proposition. The necessity arises to create a cumulative list, because there might be the possibility that a file is deleted in one version and re-established in a new version. Nevertheless, while this is possible, it is highly unlikely, and in most cases it is safe to simply iterate over all the deletionList data files and delete all the files listed therein. As a bash command:

for f in update/data/deletionList/*; do xargs -a $f -d"\n" rm -fv ; done

Editing the Configuration File

This final step in the update process, before refreshing, should not be done manually unless you are completely certain that all the files (not counting customizations) and the structure of the database are equivalent to a fresh installation at the same version as the one to which you have updated.

To update the configuration manually:

1. Open the file protected/config/X2Config.php in a text editor
2. Find the line with the $version variable in it. It should look like this (but with the version from which you are updating):

   $version = '3.5.2';
   

3. Change the value in quotes to the current version being updated to, and save the file.

Clearing caches and refreshing the application

It is generally always a good idea to clear the application's server-side cache and other ephemeral files/data after updating. This is taken care of automatically by the built-in updater, but barring proper use and functioning of that, the operations can be performed manually as follows:

clear the data cache

delete all ".bin" files in protected/runtime/cache:

rm -fv protected/runtime/cache/*.bin

clear web assets

delete all sub-folders of assets by running this command (in the installation directoryThe directory (folder) on the web server where X2Engine is installed. You will know that you are operating within this directory when you see a file ''constants.php'' and the following subdirectories: ''assets'', ''css'', ''custom'', ''framework'', ''images'', ''js'', ''protected'', ''themes'' and ''uploads''.):

rm -r assets/*/

clear the "auth cache"

Flush the auth cache table, i.e. run the following command on the database:

DELETE FROM `x2_auth_cache` WHERE 1;

log all users out

Flush the sessions table, i.e. run the following command on the database:

DELETE FROM `x2_sessions` WHERE 1;

Upgrading

Using the Web Interface

Software upgrade page, in the first step (registration)

To upgrade to Professional Edition

1. Purchase X2Engine Professional Edition Download. Note the registration info in the activation email.
2. Go to the Admin page in X2Engine and find "Upgrade X2Engine" under "System Settings".
3. In the Upgrade page, enter your registration info into the form and click "Register" to continue. Once complete, you will see a list of changes similar to those displayed when performing an update.
4. Review the changes to be applied and then proceed by clicking "Upgrade", similar to performing an update.

"Offline" Upgrading

This works in exactly the same way as offline updating. However, to obtain your upgrade package, which you will extract into a folder called "update" on the X2Engine web server, contact customer support. Furthermore, you will need to run the following database command on your X2Engine database when done with the upgrade, replacing {product key} with your product key, to allow your installation to receive Professional Edition software updates:

UPDATE `x2_admin` SET `unique_id`='{product key}' WHERE `id`=1;

Manually Upgrading

The process of upgrading is very similar to that of updating, and uses all of the same underpinnings as the updater. Thus, almost all of the above information and procedures will apply to upgrades. However, note the following crucial differences:

• X2Engine must be at the latest available version in order to upgrade.
• To obtain the upgrade package, you must be an owner of an X2Engine Professional Edition Download-type license. Contact customer support for further details.
• There will be only one "intermediate" version. The changes to apply will move X2Engine from a version to that same version, applying instead the necessary changes to switch its edition from Open Source to Professional.
• There are no file deletions.
• The properties of the package manifest differ slightly. In upgrades, the properties of the data object are sqlUpgrade and fileUpgrade instead of sqlList, fileList, etc. Note also, the update data is in aptly-named sub-folders of update/data as well, and there are no sqlForce or migrationScripts lists to handle.
• The configuration is not changed. Instead, the value stored in the field unique_id of the admin record in the database table x2_admin must be changed to the license key, and the edition field must be changed to "pro". To perform these both, run the following command on the X2Engine database:

  UPDATE `x2_admin` SET `unique_id`='{license key}',`edition`='pro' WHERE `id`=1;
  

• It is not necessary to clear caches or user sessions when done. These operations are performed in updates to eliminate ephemeral (session-ingrained) data that may no longer be applicable and thus cause problems. Upgrades will rarely, if ever, invalidate this data.

Updating to Beta Versions

Beginning at version 4.1.2, after much advocacy to adopt this practice, X2Engine began a two-phase release process in which each version would be released first as a beta, then to the broader public after initial bug finds. The purpose behind this is to avoid updating production installations of X2Engine to new versions that have had zero public exposure, and thus avoid introducing bugs that were not found in initial QA testing (due to a incompatibility with a particular server environment or configuration of X2Engine) into sensitive production environments.

By default, the updater will not update to beta versions. The updater must be configured to use the proper URL on the update server for beta release data, by the following procedure:

1. Create a file "constants-custom.php" in the root level of X2Engine. You can easily do this by renaming "constants-custom.example.php" to that name (remove ".custom").
2. Look for the following line, and in it, change "false" to "true":

   defined('X2_UPDATE_BETA') or define('X2_UPDATE_BETA',false);
   

The advantage of enabling beta releases is that new features will be available much sooner. Additionally, fixes to bugs will also be available sooner. The main disadvantage of enabling beta releases is that they are never guaranteed to be as stable as non-beta releases. We encourage users to clone their installations of X2Engine, if able, and use beta versions on them, and report any issues they encounter for their particular server environment/special configuration/custom implementation of X2Engine. This way, any issues that arise can be fixed in the stable release. The more users participate in beta testing, the more stable the non-beta releases will become.

Recovering From Failed Updates

Uh-oh. You attempted to update, and found out too late that the database was incompatible with the changes. You might have used the web updater and experienced a server timeout, or made changes to the schema outside of X2Engine that conflicts with the update. Your backup is either out of date or nonexistent.

Fortunately, recovery from such a scenario is fairly straightforward provided one has basic knowledge of MySQL, but it requires hands-on work with the files and MySQL database. If you have no experience managing databases either through a MySQL command console or a web UI (i.e. PHPMyAdmin), it is recommended you seek help with this process. Furthermore, like manually updating, this process requires access to the file system. If all that is available to you is accessing your application via the web, i.e. pure SAAS, you will need to contact your hosting provider or system administrator for assistance.

Recovery Checklist

In summary, before beginning, you're going to need the following:

• Access to the file system and database of X2Engine. Without these, your hands are tied, and you cannot effectively do anything. You will need someone else who does have access to fix X2Engine on your behalf.
• The list of SQL changes that ran before the failure. The updater should have displayed this, in addition to the SQL error message (which you should also save and read/consider carefully; it explains what went wrong).
• The updater package for the version you attempted updating/upgrading from. This contains not just the important data but also flat data files that should prove helpful in diagnosing and fixing the problems. See "Obtaining the Update Package."

Using Backups

If a backup copy of the database is available, and the update failed in the middle of database changes, restoring the database to the backup can prove the easiest way to reset the database to a known state and saving effort in the recovery process. Doing so is recommended unless your database is exceedingly large (i.e. several hundred megabytes or more), in which case the restore operation will take a long time and possibly cause more damage if done improperly.

Backup Validity and Applicability

Note that, when making a backup copy of the database, whether using the button in the updater's web interface or some third-party tool, that the backup represents the exact state of the database when it is created. Furthermore, if the backup-database button is used after attempting to update unsuccessfully, that will overwrite the old backup. Thus, it should go without saying: DO NOT back up again if your first attempt failed. You should back up your files and database before you attempt to update, and only then; nothing after that is useful. After the first attempt, any backup made will reflect the broken/intermediate state, in which case it does no good and only returns you to the broken state when you use it.

Restoring a SQL Backup Manually

Database backup copies, especially those made with MySQLDump and the built-in database backup utility of X2Engine, retain both the schema and content of the database. They can be run as SQL scripts to completely restore the database to its previous state. How exactly to do this depends entirely on the end user's preferred method and thus is not covered here (there are many possible ways).

If using a backup copy that is in the form of a SQL script, it is vitally important to first drop all tables in the target database that were not created in the backup script. This is important because the restore script will not drop any tables in the current database that aren't in the backup (which need to be dropped before re-creating them in the backed-up state). This will include any tables created since the time that the backup was made. The reasoning behind this is that if the error is "Base table or view already exists," restoring in itself will not help, because the offending table will still remain in the database after restoring. Note, the auto-restore operation in the updater takes care of this automatically.

Step 1: Examine the Files

The first appropriate step is to look at the modification times of the files that were supposed to be updated or added in the update. Lists of such files, one per line, are in the files stored in data/fileList within the update package. Pick one of them and see if it was changed today. Compare, i.e. using diff, the files in the update package versus the files in the installation.

Bulk Comparison of Files

To compare all files in one command: assuming you are in a Unix/Linux shell session (i.e. bash) on the web server, and you are in the base directory of X2Engine, and the updater package has been unpacked into the "update" directory, run

 diff -rq update/source/ ./

Get a List of Migration Scripts

Migration scripts perform ancillary data transformation after a version's worth of updates that cannot be performed using pure SQL. They are PHP source files, and can be found in the directory source/protected/migrations within the unzipped update package. If any migration scripts are present, this will impact how one must proceed, if finishing the update manually rather than restoring the database to the backup copy and modifying the database for update compatibility.

Step 2: Examine the Database

If any of the files differ and have not been updated, this indicates that the update halted in the middle of database changes, and thus, you will need to determine the state that it is in. On the other hand, if all of the files are identical to the copies in the update package, the chances are good that all of the database and file changes have already been applied, and all that remains to be done is update the configuration with the new version. However, checking that the database is in the expected state is still recommended in this case.

What will essentially be accomplished in this step is choosing the best possible option for actions to take in manually altering the database.

Read the Update Error Output

In many cases, the error output from the updater will tell you exactly what went wrong in the update. If it is a simple problem, i.e. table/column already exists, and you are mostly certain that this is the only compatibility issue, you can skip a full database comparison and opt rather to try removing any incompatibilities in the database first before attempting to update a second time.

Compile The Full Update SQL

To make subsequent tasks easier, one can compile the full list of SQL statements into one file. First, however, it is necessary to compile the chronologically-ordered list of versions. This is important because, while one can iterate over files in data/sqlList, the ordering of files may not result in a listing that accurately reflects the order in which the versions were released.

So, to get the list of files, look for the "versions" property of the json object stored in update/manifest.json, or run the following command to extract it into a file "versionList.txt":

 php -r '$m = json_decode(file_get_contents("update/manifest.json"),1); echo implode("\n",$m["versions"]);' > versionList.txt

Now, with the list of files in versionList.txt:

 for v in `cat versionList.txt`; do echo "/* VERSION $v */"; cat update/data/sqlList/$v.sql; done > update.sql

The file update.sql will now contain a list of all SQL commands, in chronological order, that are part of the update.

Perform a Binary Search for the Current State

This may only be necessary in cases where the database changes were cut short, i.e. a server timeout in the web updater. In this process, one determines which changes have been applied in the update and which have not. It will reveal what if any action is necessary to "repair" the database.

For those unfamiliar with the term, a binary search works as follows:

1. Start with an "upper" and "lower" limit item in the ordered list being searched.
2. Pick an item in the middle of the list, between the upper and lower limits.
3. If the item is before the item we want, change the lower limit to that item. Otherwise, change the upper limit to that item.
4. Repeat steps 1-3 until the desired point in the list is found.

In our case, the ordered list is the full list of SQL to be run in the update, and the desired item is the first database change that didn't get applied. That leaves the binary condition, which is choose the lower bound if the change appears to have already been applied, and choose the upper bound if it hasn't been applied.

To begin, open the .sql file created in the previous part of this step, and:

1. Find the first and last SQL statements.
2. Look in the database schema (and, if necessary, data) for those changes, i.e. the presence of those columns/records. Note:
   • If neither of them have been applied: the problem to be fixed is not an incomplete update; it is an incompatible schema, in which case the best option is removing the incompatibilities.
   • If both of them have been applied: the issue is that the database is already up-to-date and at the latter version, and so no other action need be taken.

Finding out if a given change has been applied requires basic knowledge of MySQL; for instance, given an INSERT command in the update, one will need to know how to compose an appropriate SELECT statement (or know which buttons to press) to search for and find the records that would be inserted by it.

Step 3: Determining the Appropriate Course of Action

If The Database Is Already Up To Date

If this is the case, and you have determined that the database changes have all been applied, then all that remains to be done are the final steps in manually updating. Those steps include everything from copying the new files through the final clean-up process, and they are all the easier steps in the process.

If The Schema Includes a Known Incompatibility

In this scenario, you have a feature which you know that you added, but it somehow conflicts with a database change to be applied. The way to recover from or get around this this could range from very simple to very complex. For the simpler case, i.e. a column name conflict, you can get around this by renaming the column to something else temporarily, and then copying the data into the newly-created column as desired after the update.

If The Schema/Data Is Incompatible vis-a-vis The Updater Error Messages and You Have No Idea Why

If all else fails, or you have no recollection of manually adding the offending database feature (i.e. primary key conflict, table, column, etc), try removing the feature from the database and re-attempting the update.

This could include anything from dropping tables to deleting records; again, this requires knowledge of MySQL.

If The Update Is In a State Between Versions

In this scenario, there are two options worth considering. First, you could try manually applying each subsequent database change and running accompanying migration scripts for each version. In other words, pick up where the database changes left off, as though in the middle of a manual update.

If there are a very large number of database changes between the state of the database and the final change, one might consider reverting the changes instead, i.e. remove columns that get added. Note, however, that migration scripts cannot be applied in reverse, and so this task may be far less simple if any migration scripts have been run already at the stage that the database is currently in.

In either case, if a working/compatible and sufficiently up-to-date database backup has been made before the update attempt and is available, one should try to use that and re-attempt the update first, provided that it does not take too long to restore the database.

Troubleshooting

User interface (UI) "bugs" after the update

It is extremely common (and easily remedied) that after an important update, certain features of the user interface of X2Engine may not work properly. This is caused, in nearly all cases, by the web browser's cache, which contains out-of-date copies of X2Engine's essential user interface files. To solve these problems, the first step to try is clearing the cache. A step-by-step guide to doing so is given in wikihow:Clear-Your-Browser's-Cache.

Errors in the updater itself

When all else fails, the updater may have to be patched due to an irreconcilable backwards compatibility issue. Fortunately, if one has read/write access to the source files on the webserver, this is straightforward to fix. See Refreshing and Updating the Updater Utility.

Professional Edition not in effect after upgrading

This may be caused by Professional Edition not getting "switched on" properly, despite all the necessary filesystem and database changes already being in effect. First, to see if this scenario applies to you, look for a folder "reports" inside protected/modules, and a table x2_reports in the X2Engine database. If you find these things, you can try the following to enable Professional Edition:

1. Find the file constants.php in the installation directoryThe directory (folder) on the web server where X2Engine is installed. You will know that you are operating within this directory when you see a file ''constants.php'' and the following subdirectories: ''assets'', ''css'', ''custom'', ''framework'', ''images'', ''js'', ''protected'', ''themes'' and ''uploads''.. Open it in a text editor, and look for the following line:

   defined('PRO_VERSION') or define('PRO_VERSION',false);
   

2. Change false to true on that line, and save the file.
3. Run the following SQL command (replacing {license key} with your X2Engine license key) on the X2Engine database:

   UPDATE `x2_admin` SET `unique_id`='{license key}',`edition`='pro' WHERE `id`=1;
   

General Procedures to Try

It may sometimes be possible that the update did not finish cleanly. In such cases, it will be necessary to manually re-perform some of the final operations in the update, described in Introduction.

Refresh the Source Code

This step is not really the best step to try first, as it is usually more time-consuming. It entails re-uploading the entire application, in order to reset all non-ephemeral (source code) files to what they should be in a fresh installation of X2Engine. To do this:

1. Download the refresh package. There should be a download link for this in the Updater Settings page. Otherwise, if that doesn't work and you're using Open Source edition, use this link, or if using Professional Edition, paste the following url (after replacing {key} with your license key) into your browser's address bar:

   https://x2planet.com/installs/{key}/refresh-pro.zip

2. Upload the contents of the downloaded zip archive to the installation directoryThe directory (folder) on the web server where X2Engine is installed. You will know that you are operating within this directory when you see a file ''constants.php'' and the following subdirectories: ''assets'', ''css'', ''custom'', ''framework'', ''images'', ''js'', ''protected'', ''themes'' and ''uploads''., overwriting existing files.

Clear Caches

See "Clearing Caches..." for details.

Regenerate the Configuration

Again, this should not be done unless you are completely certain that all the files (not counting customizations) and the structure of the database are equivalent to a new installation at the same version as the one you updated to. Refer to Editing the Configuration File.