Validating a Magento Connect Extension

Like this article? Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

If you’ve followed my work here, on Stack Overflow, or on Twitter, you know I have a perpetual bee in my bonnet over Magento Connect’s file permissions issues. There’s a few common cases where Magento Connect will tell you an extension is installed, but in actuality Connect couldn’t install the extension because of insufficient file permissions. Permissions are always a thorn in a developer’s side — but the Connect application’s willingness to lie to its users about an extension being installed crosses some invisible line of unacceptable behavior.

The Magento 2 team at eBay is aware of the problem, and working on it, but that leaves working Magento developers with the problem of never knowing what’s actually installed when clients or customers use Magento Connect. That’s why I’ve created a new command for n98-magerun called extension:validate. Today we’ll cover how to use the command, as well as discuss some implementation details so you understand what the command is doing.

Using the Command

The extension:validate command requires no arguments. If you run it you’ll see something like this

$ n98-magerun extension:validate

Customchecoutstep
--------------------------------------------------


Pulsestorm_Commercebug
--------------------------------------------------


Auctionmaid_Matrxrate
--------------------------------------------------

Problem: /path/to/magento/app/code/community/Webshopapps/Matrixrate/etc/config.xml
    Hash: MISMATCH
Problem: /path/to/magento/app/etc/modules/Webshopapps_Matrixrate.xml
    Path: FILE NOT FOUND

When invoked with no options, extension:validate will look at all the third party Magento Connect extensions a user has installed on their system, and validate their contents against the package.xml manifest.

In the above example, our system has the Magento Connect extensions Customchecoutstep, Pulsestorm_Commercebug, and Auctionmaid_Matrxrate installed. Customchecoutstep and Pulsestorm_Commercebug were fine, but the Auctionmaid_Matrxrate was missing a file (Webshopapps_Matrixrate.xml), and one of the files didn’t match the stored hash (config.xml). Not matching the stored hash means the file’s been modified/edited.

Troubleshooting Connect Details

While this tool is useful, it’s not a cure all. Once you’ve detected problems with an extension you’ll need to figure out why they happened. In other words, are they real problems or intended changes from more reckless developers. There’s a good chance the original package for an extension will be in the

downloader/.cache/community/

folder. This where Magento Connect downloads the packages prior to installing them. Copying a package from here and extracting the files will let you replace missing files or run a diff to see what’s changed.

The extension:validate command reads the package information from the

downloader/cache.cfg

file. Despite being named cache.cfg, this really isn’t a cache file — it’s the file Magento Connect reads from when it lists out the installed packages on the system, and effectively acts as the source of truth for what is and isn’t installed.

If I speculate, the reason this file is named cache is it’s a cached list of what’s installed on the system. In other words the entire Magento system itself is the actual thing, and this file is just a cache. However, this falls apart when you consider there’s no way to query the Magento system itself for a list of installed packages, and the Magento Connect downloader always reads from this file. Another example of “The best laid plans …” problem.

Command Options

The extension:validate command will also let you check a single extension. Just pass in the extension’s name as the first argument

$ n98-magerun extension:validate Auctionmaid_Matrxrate

Auctionmaid_Matrxrate
--------------------------------------------------

Problem: /path/to/magento/app/code/community/Webshopapps/Matrixrate/etc/config.xml
    Hash: MISMATCH
Problem: /path/to/magento/app/etc/modules/Webshopapps_Matrixrate.xml
    Path: FILE NOT FOUND

This is useful to run right after you’ve installed an extension to make sure it’s installed correctly. Keep in mind you need to use the Magento Connect Name for an extension, and not the Namespace_Modulename module name. (Auctionmaid_Matrxrate vs Webshopapps_Matrixrate above)

If you only want to do a specific type of check — that is only check if a file is missing, or if the hash matches — there’s options to skip each test type.

$ n98-magerun help extension:validate
...
 --skip-file           If set, command will skip reporting the existence of package files
 --skip-hash           If set, command will skip validating the package file hashes

You use them like this.

n98-magerun extension:validate --skip-file 
n98-magerun extension:validate --skip-hash

n98-magerun extension:validate Auctionmaid_Matrxrate --skip-file 
n98-magerun extension:validate Auctionmaid_Matrxrate --skip-hash

There’s also the --full-report command which will list all the files for an extension — not just the problematic ones.

$ n98-magerun extension:validate Auctionmaid_Matrxrate --full-report

Auctionmaid_Matrxrate
--------------------------------------------------

Checking: /path/to/magento/app/code/community/Webshopapps/Matrixrate/Block/Adminhtml/Shipping/Carrier/Matrixrate/Grid.php
    Path: OK
    Hash: OK
Checking: /path/to/magento/app/code/community/Webshopapps/Matrixrate/Block/Adminhtml/System/Config/Form/Field/Exportmatrix.php
    Path: OK
    Hash: OK
//...
Checking: /path/to/magento/app/etc/modules/Webshopapps_Matrixrate.xml
Problem: /path/to/magento/app/etc/modules/Webshopapps_Matrixrate.xml
    Path: FILE NOT FOUND

Useful if you’re curious what, exactly, a Magento Connect package has dropped on your server.

Magento Connect for Upgrades

In addition to installing third party extensions, Magento Connect’s other big feature is enabling package based updates of the Magento system itself. It does this by having the following 36 packages installed by default when you initialize a Magento Connect system.

'Cm_RedisSession','Interface_Adminhtml_Default','
Interface_Frontend_Base_Default','Interface_Frontend_Default','
Interface_Frontend_Rwd_Default','Interface_Install_Default','Lib_Cm','
Lib_Credis','Lib_Google_Checkout','Lib_Js_Calendar','Lib_Js_Ext','
Lib_Js_Mage','Lib_Js_Prototype','Lib_Js_TinyMCE','Lib_LinLibertineFont',
'Lib_Mage','Lib_Magento','Lib_Phpseclib','Lib_Varien','Lib_ZF','
Lib_ZF_Locale','Mage_All_Latest','Mage_Centinel','Mage_Compiler','
Mage_Core_Adminhtml','Mage_Core_Modules','Mage_Downloader','
Mage_Locale_de_DE','Mage_Locale_en_US','Mage_Locale_es_ES','
Mage_Locale_fr_FR','Mage_Locale_nl_NL','Mage_Locale_pt_BR','
Mage_Locale_zh_CN','Magento_Mobile','Phoenix_Moneybookers'

This has always struck me as a little bit crazy. Because Magento Connect installations aren’t transactional, this creates a situation where an install that fails due to network connectivity issues or un-handled corner cases can leave a Magento installation in a “half-upgraded” state. If you’ve spent any time on the Magento forums or the various Stack Exchange sites, you know this is more than a theoretically concern.

Regardless, if you or your client are managing upgrades via Magento Connect, you may include these packages with the --include-default flag.

$ n98-magerun extension:validate --include-default

Be warned though, since it’s effectively running through every Connect managed file in your Magento application it’ll take a bit longer to finish. This is also a potentially useful way to keep your application “core hack” free. Finally, these extension names are hard-coded into the extension:validate command. If Magento adds any packages in the future, extension:validate will see them as 3rd party extensions until the n98-magerun command is updated with a new extension list. If anyone know a way to query something for a list of official packages, pull requests are welcome.

Wrap Up

The extension:validate command is available in the develop branch of the n98-magerun source. It looks like the application is getting monthly-ish releases, so this should be available in the official release soon. If you run into problems or have feature suggestions, please let me know.

Originally published June 9, 2014

Commerce Bug 2.3.1 with Theme Inheritance

Like this article? Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

In case you missed it in the waterfall of post Magento Imagine press releases, I’ve got a new version of Commerce Bug 2 available. Free for registered users — this update brings support for the theme inheritance features of Magento CE 1.9 and Magento EE 1.14. You can read more on the Pulse Storm press release blog. Actually, there’s not much more there — but if you want notices when I update an extension or other product, you should defiantly subscribe.

If you’re new to Magento or new to my work, Commerce Bug is my commercial Magento debugging extension. It provides programmers, developers, system administrators, and designers with the sort of diagnostic information they needed to work with unfamiliar Magento systems. It’s useful for newbies and old-hands alike. The developing Magento with Commerce Bug article series demonstrate how a developer can use Commerce Bug in their job day to day. If you’re being paid to do Magento work Commerce Bug will pay for itself it the first day of use.

I released Commerce Bug four years ago at the first volcano interrupted Magento Imagine. Since then I’ve been slowly but steadily updating it — adding new features and keeping up to date with Magento’s various version changes. Charging for the software lets me provide my users with top notch support. It also allows me the sort of career where I can do the deep research need to write the articles you see here, answer questions on Stack Exchange sites, and contribute other projects to the open source community. Without that income I couldn’t do what I do.

I try not to say it too often, but if you’ve gotten any value out of my Magento work over the past half-decade, I’d really appreciate it if you took a look at the products I sell and give one a try. Your support, as always, is appreciated.

Originally published June 4, 2014

Getting Started with PHP Composer and Magento

Like this article? Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

If you haven’t heard, PHP Composer is a package management system for PHP. It’s closest thing the PHP community has to something like Ruby Gems. The goal of package management systems is to make it super simple to add a new library and/or third party framework to the program you’re working on.

While composer’s starting to catch on within a certain segment of the PHP community, it faces a few challenges that Ruby Gems and other package management systems do not.

First, PHP has no built-in module system. There’s namespaces, which are module-ish, but they’re optional. The core PHP language is still centered around a global namespace of variables, classes, and functions all mushed together. The PSR standards created a blueprint for solving this but there’s no enforcement on a language level. This means any package management system is facing an uphill battle in getting its code to play nice with others.

Second, composer is still a new enough project that it’s very developer centered. Yes, of course, all developer tools are developer centered. It’s more accurate to say composer has a lot of rough edges that haven’t been sanded down yet. An inexperienced ruby programmer can get up and running with Ruby Gems almost instantly — packages are just a gem install away. An inexperienced PHP developer will find a steep learning curve when they’re jumping into composer.

Despite this, there’s still value in using composer to manage your packages, even for a Magento developer. Today we’re going to run through getting started with composer, and using some of the Firegento/Hackathon tools to make composer work with Magento. Our goal will be to make this as simple as possible while giving you a road map to composer’s deeper features.

Also, we assume you’ve been able to download composer from their website and install it. The code examples below assume a working composer.phar somewhere in your command line path. If you’re running composer in a different way, please adjust the composer.phar commands accordingly.

Initializing

To start, we need to initialize our composer project. This is something we need to do regardless of whether we’re starting a new development project with composer, or just using it to install packages. This is as simple as creating a new folder to hold our project and running composer’s init command

$ mkdir composer-example
$ cd composer-example
$ composer.phar -n init

The -n means “no interactive”. Without this flag init asks you a bunch of questions about starting a new composer package. We don’t want to create a new composer package, we’re just using composer to manage other packages. That means the interactive questions don’t apply to us.

If you take a look in the current directory, you’ll see there’s now a skeleton composer.json file.

$ cat composer.json 
{
    "require": {

    }
}

This is the main composer configuration file for our project. The same way a git repository has a .git folder, composer has a composer.json file.

Installing

After we’ve initialized a composer project, we need to install it. This is a little confusing if you’re new to composer. The basic idea is the init command creates the composer project, but the install command will setup the default PSR boilerplate needed for a PHP project based on composer packages. It will also install any packages configured in the require section of composer.json, but we’ll get to that in a minute.

First, let’s install the project. Run the following command.

$ composer.phar install --no-dev    

The --no-dev flag makes sure a bunch of extra stuff we don’t need is skipped. A composer project has two sets of boilerplate and packages — those you need to use the package, and those you need to work on the package (things like tests, build tools, etc). When we use the --no-dev flag, this tells composer that we only want to use these packages — we don’t want to download the “we’re going to work on someone else’s project” developer dependencies.

If you take a look at your composer project folder after running install, you’ll see composer added a number of new files to the vendor folder.

$ find .
./composer.json
./vendor
./vendor/autoload.php
./vendor/composer
./vendor/composer/autoload_classmap.php
./vendor/composer/autoload_namespaces.php
./vendor/composer/autoload_psr4.php
./vendor/composer/autoload_real.php
./vendor/composer/ClassLoader.php

This is the boilerplate code we were talking about earlier — the standard composer autoloader files. These files give composer projects a module like system for running code and using composer packages. If everything’s working correctly you won’t need to worry about these files — composer needs them, but they’re invisible glue. Additionally, since we’re using composer to manage Magento packages we won’t be using these autoloader files. However, composer needs them to run, so don’t remove them. Just put them out of your mind.

Changing the Repository

With our project installed, the next thing we need to talk about is repositories. In composer, a repository is a list of packages. When you ask composer to install a specific package, the composer.phar program asks the repository where the package lives. Composer was built to be decentralized — anyone can create a repository. However, composer has support for a repository named packagist.org baked in. Packagist is meant to be the central location for PHP packages.

For reasons we don’t have time to cover, having Magento packages as a part of packagist.org isn’t feasible. Instead, the crew over at Firegento, via their hackathon, have created their own custom composer repository at packages.firegento.com. This is a composer repository for Magento extensions.

The next two steps to using composer and Magento together are

  1. Tell composer about packages.firegento.com

  2. Disable support for packagist.org

The second step (disabling packagist.org) isn’t technically necessary, but unless you’re an experienced composer user it’s easy to accidentally install packages from one repository when you think you’re installing them from another. Disabling packagist.org removes this as a possible outcome.

There’s one last bit of bad news — composer doesn’t have a command to disable packagist.org or to add a new repository like packages.firegento.com. We’ll need to manually edit composer.json to do what we want. Specifically, we’ll want our file to match the following.

//File: ./composer.json
{
    "require": {

    },    
    "repositories":[     
        {
            "packagist": false
        },         
        {
            "type":"composer",
            "url":"http://packages.firegento.com"
        }
   ]    
}

We’ve added a top level key to the configuration file named repositories. This key contains an array, and there’s two objects in the array.

The first object — {"packagist": false}, disables packagist.org.

The second object

//File: ./composer.json
{
    "type":"composer",
    "url":"http://packages.firegento.com"
}

is a bit of configuration that tells composer to look for a repository at packages.firegento.com.

Whenever you make an update to composer.json, you’ll need to tell composer to update your project with the update command. The update command is similar to the install command in that it installs boilerplate and packages based on your composer.json file. The main difference is you run it after you’ve done the initial installation.

To update your project, run the following

$ composer.phar update --no-dev
Loading composer repositories with package information
Updating dependencies
Nothing to install or update
Generating autoload files

Again, unless you want the development related packages, it’s important to use the --no-dev flag. Although we ran update, composer told us it had nothing to do (Nothing to install or update). If you want to confirm composer knows about the packages.firegento.com repository, run update with the -vvv flags (that’s three vs for three levels of verboseness)

$ composer.phar update --no-dev -vvv
...
Downloading http://packages.firegento.com/packages.json
Writing 
...

We’ve cut out a lot of the verbose output to show the line we’re interested in. The message above lets us know composer has downloaded the packages.json file for packages.firegento.com. This means composer now knows about packages.firegento.com. With this confirmed, we’re ready to move on to the next step.

Adding our First Package

At this point, we’ve got a composer project going, and this composer project knows about the Firegento repository. Next, we’ll install our first package. Specifically, we’re going to install the pulsestorm/magento-better404 package.

To install a package with composer, you use the require command.

$ composer.phar require --update-no-dev pulsestorm/magento-better404
Please provide a version constraint for the pulsestorm/magento-better404 requirement: 

Again, notice we’ve used a flag to tell composer to skip updating the development related packages. For the require command, this flag is named --update-no-dev instead of the usual --no-dev.

When you run require, composer will ask you for a version constraint. This is the version of the package you want to install. We’re going to enter the string 1.* for this value, which will tell composer to always update the package to the latest version of the 1.x release branch. Covering composer’s versioning is beyond the scope of this article, but the composer book has a good section that covers versioning, and also tells you why you might not want to use a raw wildcard here

Caveats out of the way, let’s enter 1.* and hit return

Please provide a version constraint for the pulsestorm/magento-better404 requirement: 1.*
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies
  - Installing pulsestorm/magento-better404 (dev-master 088677b)
    Cloning 088677bd394baf97a876914c6ccb0802caae4b7e

Writing lock file
Generating autoload files

Composer has done two things here. The first is, it’s updated the require section of our composer.json with the following.

//File: ./composer.json
"require": {
    "pulsestorm/magento-better404": "*"
},    

This configures our composer project to use the pulsestorm/magento-better404 package.

The second thing the require command does is automatically run composer.phar update for us. When composer said

Loading composer repositories with package information
Updating dependencies
  - Installing pulsestorm/magento-better404 (dev-master 088677b)
    Cloning 088677bd394baf97a876914c6ccb0802caae4b7e

It was telling us that it was downloading the pulsestorm/magento-better404 files. You can see these files have been downloaded to the vendor folder

$ ls -l vendor/pulsestorm/magento-better404/
total 48
-rw-r--r--  1 alanstorm  staff  1084 May 22 13:23 LICENSE.txt
-rw-r--r--  1 alanstorm  staff    93 May 22 13:23 README.md
drwxr-xr-x  5 alanstorm  staff   170 May 22 13:23 app
-rwxr-xr-x  1 alanstorm  staff   274 May 22 13:23 build-better404.bash
-rw-r--r--  1 alanstorm  staff   551 May 22 13:23 composer.json
-rw-r--r--  1 alanstorm  staff   890 May 22 13:23 magento-tar-to-connect.better404.php
-rw-r--r--  1 alanstorm  staff   684 May 22 13:23 modman

Congratulations! You’ve used composer to download your first Magento package!

Note: We don’t need to use the require command for this. If you want, you can edit the composer.json file yourself and run update manually. Using require is just a convenience.

Installing into Magento

While we’ve successfully downloaded a composer package into the vendor folder, one problem remains. Namely, how are we supposed to get this package into our Magento system. Modern PHP frameworks (like Symfony) that conform to the PSR standard know how to read class files out of vendor. Unfortunately, Magento 1 is not a modern PHP framework.

This is where the magento-hackathon/magento-composer-installer project comes into the picture. Composer allows users to create and install custom plugins that create alternate installation routines. The magento-hackathon/magento-composer-installer plugin creates a composer installer for Magento. Put another way — the magento-hackathon/magento-composer-installer plugin will let you install packages via composer into vender, and will automatically install the same packages into Magento for you.

So, your next questions is probably: How do we install a composer plugin? This is another area that may throw a few people for a loop. To install a composer plugin, you add a specific package to your composer project. That is, despite being a composer plugin, the plugin exists as part of your composer project, and not as part of the composer.phar script/executable.

In some people’s minds this blurs the line between your project and composer. However, another way of thinking about a composer plugin is that it’s just more boilerplate added to your project, similar to the PSR autoloading code.

Regardless, the magento-hackathon/magento-composer-installer project is also part of the packages.firegento.com repository, so all we need to do to install it is use the require command again.

Before we do that though, let’s remove the pulsestorm/magento-better404 package from our composer.json and then update our project. While not technically necessary, it will make a few things down the road a bit easier.

This is another case where composer doesn’t have a command to do what we want to do. We’ll need to manually edit composer.json to remove the the package

//File: ./composer.json
//...
"require": {

},    
//...

Then, we’ll run composer.phar update, (using the --no-dev flag, again)

$ composer.phar --no-dev update
Loading composer repositories with package information
Updating dependencies
  - Removing pulsestorm/magento-better404 (dev-master)
Generating autoload files

If you take a look at your vendor folder, you’ll see the files for this package are now gone.

$ ls -l vendor/pulsestorm/
ls: vendor/pulsestorm/: No such file or directory

With that done, next we’ll use require to install the Magento installer plugin, (using 1.* for the version number again)

$ composer.phar require --update-no-dev  magento-hackathon/magento-composer-installer
Please provide a version constraint for the magento-hackathon/magento-composer-installer requirement: 1.*
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies
  - Installing magento-hackathon/magento-composer-installer (1.3.2)
    Downloading: 100%         

  [ErrorException]                  
  magento root dir "" is not valid  

After running the command we can see that composer has successfully downloaded the plugin

$ ls vendor/magento-hackathon/magento-composer-installer/
README.md   composer.json   res     src

However, composer also yelled at us about an error

[ErrorException]                  
magento root dir "" is not valid  

This is one of those rough edges we were talking about. What happened here is we installed the custom installer plugin. After the plugin was installed, it tried to run itself, but didn’t know where we wanted our packages installed (magento root dir “” is not valid).

To fix this, we’ll need to add one last thing to our composer.json file.

//File: ./composer.json
{
    "require": {
        "pulsestorm/magento-better404": "dev-master",
        "magento-hackathon/magento-composer-installer": "*"
    },    
    //...
    "extra":{
        "magento-root-dir":"/path/to/magento",
        "magento-deploystrategy":"copy"            
    }    
}

What we’ve done here is add a new top level extra key to our composer.json configuration. The extra section contains extra configuration values. Then, within this new top level object we’ve added a two configuration keys.

The first, magento-root-dir, has a value of /path/to/magento. You should replace this with the location of Magento on your local system.

The second, magento-deploystrategy, has a value of copy. This tells the Magento installer plugin that we’d like to copy the files out of the vender folder and into our Magento system. This is an optional parameter — without it the installer will create symlinks to our vendor folder instead of copying files.

With the above in place, all we need to do is run update one last time.

$ composer.phar update --no-dev
Loading composer repositories with package information
Updating dependencies
  - Installing magento-hackathon/magento-composer-installer (1.3.2)
    Loading from cache

Writing lock file
Generating autoload files    

We’ve gotten an error free update — all that’s left is to re-add our packages.

Pulling it all together.

With our custom installer added, lets re-require the better 404 package.

$ composer.phar require --update-no-dev pulsestorm/magento-better404
Please provide a version constraint for the pulsestorm/magento-better404 requirement: 1.*
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies
  - Installing pulsestorm/magento-better404 (dev-master 088677b)
    Cloning 088677bd394baf97a876914c6ccb0802caae4b7e

Writing lock file
Generating autoload files

Although this looks exactly the same as before, behind the scenes the Magento installer plugin has done its magic. If we take a look at our system, we’ll see the files from our vendor package have been automatically copied into our Magento installation.

$ ls -l /path/to/magento/app/code/community/Pulsestorm/Better404
Block   Model   etc

If you’re curious, the installer knows which files to install and where to put them thanks to the modman configuration file included with every Firegento package.

$ cat vendor/pulsestorm/magento-better404/modman 
app/code/community/Pulsestorm/Better404/Block/404.php   app/code/community/Pulsestorm/Better404/Block/404.php
app/code/community/Pulsestorm/Better404/etc/config.xml  app/code/community/Pulsestorm/Better404/etc/config.xml
app/code/community/Pulsestorm/Better404/Model/Lint.php  app/code/community/Pulsestorm/Better404/Model/Lint.php
app/code/community/Pulsestorm/Better404/Model/Observer.php  app/code/community/Pulsestorm/Better404/Model/Observer.php
app/design/frontend/base/default/template/pulsestorm_better404/404.phtml    app/design/frontend/base/default/template/pulsestorm_better404/404.phtml
app/etc/modules/Pulsestorm_Better404.xml    app/etc/modules/Pulsestorm_Better404.xmlYou have new mail in /var/mail/alanstorm

While it’s taken a little work to setup, we now have the ability to install any Magento composer package from the Firegento repository into our local Magento system.

Wrap Up

While it’s fantastic that composer is flexible enough, and the Magento community dedicated enough, to make a a pre-PSR system like Magento 1 manage packages in a modern way, a lot of the jiggery pokery involved shows the transitional position the PHP tooling community is in.

Hopefully, over the next few years as Drupal moves to Symfony, Magento 2 in thawed from its iceberg, and other platforms begin to catch on to what the more general PHP community has been up to, we’ll start to see refinements made to the user experience of these tools, and it won’t be necessary to be an expert (or read an expert’s blog) to get started with them.

Originally published May 24, 2014