Magento 2: DI Compile Pre-Scan with Commerce Bug

Like this article? Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

If you’re following along over on the Pulse Storm blog, you’ll know we’ve just released Commerce bug 3.1. This is mostly a bug fix/Magento 2.1 compatibility release, but there are a few new features hidden beneath the surface.

A software development tool like Commerce Bug encapsulates a certain way of thinking about, analyzing, and manipulating your code. Magento 2 presents a particular challenge, since no one’s quite sure how to think about Magento 2 yet. Baking a feature into the UI that’s going to prove useless one release cycle from now is a recipe for madness. However, not including a new feature robs Commerce Bug users of something useful.

To split the difference, we’ve started to include our new prototype features as Magento CLI commands. You can see a list of these commands by running the following

$ php bin/magento list ps:cb
Available commands for the "ps:cb" namespace:
 ps:cb:scan:context        Scans class for DI params repeated in context object.
 ps:cb:scan:double-param   Scans constructor for double params.

These commands live somewhere between unstable beta and fully polished, ready for a GUI features. They’ll be useful to your development workflows, but may change as new and better best practices develop in the Magento world.

Today we’re going to discuss the first two ps:cb commands in Commerce bug, both of which offer pre-scanning for Magento 2’s setup:di:compile command.

Dependency Injection Compilation

To start, we should explain the problem these commands are trying to solve. When you ship a Magento 2 system to production, you need to run the following command

php bin/magento setup:di:compile

This commands scans through the code in your system and pre-generates a number of things (mostly related to the object manager system and dependency injection) that Magento dynamically loads when you’re running in developer mode. This is both a performance and security thing, and discussing it in full is beyond the scope of this article.

What is in the scope of this article is how slow this process can be. While it’s not a “true” compilation in the computer science sense (transforming high level code into assembly code) it can take as long as traditional compile cycles did back in the day. This becomes extra frustrating if your module code fails some of the validation checks setup:di:compile performs. When this happens, you need to fix your code, and start the compilation process over again for the entire system. There’s no way to compile a single module.

The ps:cb:scan:context and ps:cb:scan:double-param commands each check a single PHP file for some of these compilation checks. These commands will let you validate your module’s code before running a full compilation, or let you quickly correct an error that crops up during compilation.

Double Parameter Check

First, let’s talk about the ps:cb:scan:double-param command. Up until PHP 7 (meaning, for Magento 2 developers, PHP 5.6), PHP was pretty permissive about function parameters. For example, the following code

function example($foo, $foo)
    echo $foo,"\n";
example("Hello World","Goodbye World");

is valid PHP, despite the fact the example function has two parameters with the same name. PHP just uses the value of the second argument.

When developing Magento 2 extensions, I’ve found myself copy/pasting constructor DI parameters from core classes once I’ve found the object that does what I need. I’ve also found this leads to the occasional parameter with the same name, or injecting the same object twice when parameter lists get long. This usually doesn’t show up as a problem until I run my setup:di:compile check on PHP 7.

The ps:cb:scan:double-param command scans a single PHP file for this situation. Scanning a class with a valid constructor

public function __construct(
    \Magento\Framework\App\Action\Context $context,
    \Magento\Framework\View\Result\PageFactory $resultPageFactory)
    $this->resultPageFactory = $resultPageFactory;        
    return parent::__construct($context);

looks like this.

$ php bin/magento  ps:cb:scan:double-param app/code/Pulsestorm/Nofrillslayout/Controller/Index/Index.php 
Param Names is free of dupes.
Param Types is free of dupes.
Scan complete

Whereas scanning a class with an invalid constructor

public function __construct(
    \Magento\Framework\App\Action\Context $context,
    \Magento\Framework\View\Result\PageFactory $resultPageFactory    
    $this->resultPageFactory = $resultPageFactory;        
    return parent::__construct($context);

looks like this.

$ php bin/magento ps:cb:scan:double-param app/code/Pulsestorm/Nofrillslayout/Controller/Index/Index.php 
Param Names has dupes: resultPageFactory
Param Types has dupes: Magento\Framework\View\Result\PageFactory
Scan complete

The command reports on both the repeat of a type hint as well as the parameter name.

Context Object Repeat

Another common problem when running setup:di:compile is accidentally duplicating an injected parameter that’s already available in the context object.

If you know all those words, but are still confused, don’t worry, we’ll explain.

If you’ve worked your way through the object manager series, you know that Magento has a dependency injection system that automatically creates objects for you in object constructors.

With the following constructor

public function __construct(
    \Magento\Framework\View\Result\PageFactory $resultPageFactory    
    $this->resultPageFactory = $resultPageFactory;        

Magento 2 will automatically instantiate a Magento\Framework\View\Result\PageFactory object, and pass it in as the $resultPageFactory parameter.

This dependency injection system is designed to make external object dependencies explicit in a class, make the classes easier to test, and help discourage too many dependencies in a class.

Regarding that last item — while Magento Inc. spent 4 years working on Magento 2, they spent far more time building new systems than they did cleaning up old ones. Many Magento object types have a large number of dependencies, and rather than refactor these out, Magento Inc. simply carried them forward.

For example, a Magento 2 constructor class has eleven dependencies

public function __construct(
    \Magento\Framework\View\Result\PageFactory $resultPageFactory    
    \Magento\Framework\App\RequestInterface $request,
    \Magento\Framework\App\ResponseInterface $response,
    \Magento\Framework\ObjectManagerInterface $objectManager,
    \Magento\Framework\Event\ManagerInterface $eventManager,
    \Magento\Framework\UrlInterface $url,
    \Magento\Framework\App\Response\RedirectInterface $redirect,
    \Magento\Framework\App\ActionFlag $actionFlag,
    \Magento\Framework\App\ViewInterface $view,
    \Magento\Framework\Message\ManagerInterface $messageManager,
    \Magento\Framework\Controller\Result\RedirectFactory $resultRedirectFactory,
    \Magento\Framework\Controller\ResultFactory $resultFactory

These are all the objects a Magento 2 controller needs in order to do its job.

This many classes/objects creates a problem. If end-user-programmers wants to create their own controllers with an injected dependency, this means they’d need to

  1. Copy all these dependencies from the base class
  2. Make a call to parent::__construct with the parameters in the correct order
  3. Keep all those parent:: calls up-to-date when the base class changes in future versions of Magento

This is the problem Context objects solve. If you look at the actual __construct method for a base action controller in Magento 2

#File: vendor/magento/framework/App/Action/Action.php
public function __construct(
    \Magento\Framework\App\Action\Context $context
    $this->_objectManager = $context->getObjectManager();
    $this->_eventManager = $context->getEventManager();
    $this->_url = $context->getUrl();
    $this->_actionFlag = $context->getActionFlag();
    $this->_redirect = $context->getRedirect();
    $this->_view = $context->getView();
    $this->messageManager = $context->getMessageManager();

You’ll see only a single injected parameter — Magento\Framework\App\Action\Context. However, if you look at the source of this context object

#File: vendor/magento/framework/App/Action/Context.php
public function __construct(
    \Magento\Framework\App\RequestInterface $request,
    \Magento\Framework\App\ResponseInterface $response,
    \Magento\Framework\ObjectManagerInterface $objectManager,
    \Magento\Framework\Event\ManagerInterface $eventManager,
    \Magento\Framework\UrlInterface $url,
    \Magento\Framework\App\Response\RedirectInterface $redirect,
    \Magento\Framework\App\ActionFlag $actionFlag,
    \Magento\Framework\App\ViewInterface $view,
    \Magento\Framework\Message\ManagerInterface $messageManager,
    \Magento\Framework\Controller\Result\RedirectFactory $resultRedirectFactory,
    ResultFactory $resultFactory
) {

you’ll see all the needed dependencies. The context objects allow Magento core developers to hide the dependency burden of Magento’s core classes behind a single class. As of Magento 2.1, there are twenty one different context object types


Context and Redundancy

One side effect of all this can be redundancy. If I’m an end-user-programmer creating my own controller class and I need to generate a URL, I may try injecting a Magento\Framework\UrlInterface

public function __construct(
    \Magento\Framework\App\Action\Context $context,
    \Magento\Framework\UrlInterface $urlMaker,        
    $this->urlMaker = $urlMaker;
    return parent::__construct($context);

even though the context object already has a URL object inside it. Magento, by itself, will let you do this.

However, the setup:di:compile command checks for these sorts of redundancies. If it detects one, compilation is halted. This is the situation ps:cb:scan:context scans for.

A scan of a class with the above, invalid constructor, would look like this.

$ php bin/magento ps:cb:scan:context app/code/Pulsestorm/Nofrillslayout/Controller/Index/Index.php 
Found param in original object that's in context object.
 - Magento\Framework\UrlInterface
Original Object: /Users/alanstorm/Sites/
Context  Object: /Users/alanstorm/Sites/

Wrap Up

While far from complete, these two commands should dramatically shorten the release cycle of anyone working on Magento 2 modules meant for production deployment. If there’s additional setup:di:compile code validation that’s tripping you up, please let me know and we’ll get them added to the new feature tracker.

Originally published August 22, 2016

Why SemVer is Failing in Magento 2

Like this article? Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

One of the big changes for Magento 2 developers is the introduction of a formal, “semantic versioning” system to the platform. If you’re a Magento developer and this comes as news to you, don’t feel bad. Although there’s been plenty of talk about the semantic versioning system and some formal documentation, no one’s really sat down and explained or explored the implications of this system for a working Magento developer.

This article won’t answer all your questions, instead it’s the start of a conversation about

  1. What semantic versioning is in Magento 2
  2. Why this system is failing third party developers

If you’re the impatient type, semantic versioning is a system that lets third party developers configure which Magento component versions their third party components need. It’s failing because Magento Inc. has not clearly communicated what things are and aren’t covered under semantic versioning, Magento Inc. has not evangelized the feature properly to developers, Magento Inc. is strongly biased towards you using their component hosting to take advantage of semantic versioning, Magento Inc. has failed to provide a comprehensive API that’s semantically versioned, and Magento Inc.’s semantic versioning only covers PHP code.

All of that probably sounded wah wah wah, so lets get to that rarest of commodities: Context.

What is Semantic Versioning

Semantic versioning, or SemVer, is an informal specification (i.e. no standards bodies involved, as far as I know) that attempts to formalize and give meaning to the common versioning practice that gives software numbers like


In SemVer’s version of the universe, the first number is the “MAJOR” version, the second number is the MINOR version, and the third number is the “PATCH” version.

Where SemVer provides its main value is in what a change to any of those numbers means. When the major version number changes, it means

We’ve broken backwards compatibility with the API

The the minor number changes, it means

We’ve added new functionality, but the API is still backwards compatible

When the patch version changes, it means

We added no new API functionality, but we fixed some bugs in a way that’s backwards compatible with with the old API

All in all, SemVer is a pretty straight forward, if idealistic, idea of what version numbers in a software product mean.

Magento, Versions, and Composer

This brings us to the first failure of Magento’s semantic versioning. Magento Inc., has not, in a definitive way, let the world wide developer community know if semantic versioning applies to individual components, the Magento product/marketing version (2.0.x, 2.1.x), the product meta-package, or some combination of the three.

The product/marketing version is pretty self explanatory, but component and meta-package versions may require additional explanation for developers coming from Magento 1 or new to the platform.

In Magento 1, the atomic particle of software distribution was called a package. This package format was originally based on the PHP pear format, but later forked into its own thing. In very simple terms, a Magento 1 extension was a collection of files and folders installed into your Magento system’s root folder.

In this regard, Magento 2 improves on Magento 1. In Magento 2, the atomic particle of software distribution is something called a component. A component is a collection of PHP files, in a folder, with a file named registration.php in the root. This registration.php file contains code that identifies the collection of files as a Magento Code Module, a Magento Theme, a Magento Code Library, or a Magento Language/Translation Pack.

Magento 2’s code distribution is heavily biased towards using Composer. There’s a 1 to 1 relationship between a Magento component, and a composer package. Most Magento 2 components live in vendor/[vendor-name] — including Magento 2’s core code. Every composer-distributed component has a version number. For example, you can see the version of the magento/module-catalog component/composer package here

//File: vendor/magento/module-catalog/composer.json
"version": "100.0.5",

The 100 is the major version number, the 0 is the minor version number, and the 5 is the patch version number.

Component versions explained, that leaves “meta-package” versions. A “meta-package” is a composer package that lists every package requirement for a major version of Magento 2. For example, the composer meta-package for Magento CE is magento/product-community-edition. The composer.json for a Magento 2 CE system has a require section that looks something like this

//File: ./composer.json
"require": {
    "magento/product-community-edition": "2.1.0",

The magento/product-community-edition package contains no code — but it does contain a list of component packages and versions that make up Magento 2.1.

Magento Inc. has never explicitly said which of these is under semantic versioning. By convention, it would appears to be only the component versions. For example, in the jump from Magento 2.0.x to Magento 2.1 (both the marketing version, and meta-package version) the magento/module-catalog version jumped from 100.0.5 to 101.0.0. The change in the major version of the component version (100 to 101) indicates a breaking change. However, the marketing and meta-package version only jumped a minor point release (2.0.x to 2.1.x). This indicates new functionality, but no breaking changes.

Magento not having a clear explanation for how all this works is the first way semantic versioning fails Magento developers.

Composer, SemVer, and Third Party Code Distribution

The second way semantic versioning fails Magento developers is similar. There’s no clear explanation of what semantic versioning can do for them. i.e. SemVer — What is it Good For?

Concentrating only on components (putting aside the marketing/meta-package version), as a third party developer, when you call Magento code, you should list that component’s version as a requirement in your component’s composer.json file.

For example, if I created a Magento module that used Magento\Catalog\Model\Product objects, code that’s part of the magento/module-catalog component, I would list that module as a requirement in my module’s composer.json file.

//File: vendor/pulsestorm/some-module/composer.json
"require": {
    "magento/module-catalog": "100.0.5",

With the above configuration, I’m saying “this module only works with version 100.0.5 of the magento/module-catalog component. I could indicate more than one version using Composer’s version constraints.

The value of this is, if a user (merchant, integrator) attempts to upgrade their system using Magento’s built in upgrade tools (all based on composer), and that upgrade included version 101.0.0 of the magento/module-catalog component/composer package, the upgrade would fail. However — the upgrade would fail safely, because Magento’s upgrade tools would refuse to upgrade the extension.

Put another way, as an extension developer, instead of getting a support email that says


you’ll end up with a support email that says

Hey, when are you going to release a version of your extension that’s compatible with Magento 2.1

As a merchant, if you’re using extensions that follow semantic versioning, and using Magento’s built-in upgrade tools (and the built in upgrade tools are working correctly), you can proceed with upgrades much more confidently, knowing that your extension vendors have explicitly stated “Our extension works with the new version of Magento”.

Conceptually, as a feature, this is a huge win for everyone. Unfortunately, Magento Inc. has not communicated this message very well, and most distributed components I’ve seen don’t list their Magento dependencies.

The Composer Assumption

This leads into the third failure of semantic versioning, and that’s the composer assumption. It’s fantastic that Magento Inc. has embraced modern PHP package management via composer. What’s less fantastic is semantic versioning benefits are only available if you’re distributing a component via composer.

While composer is the preferred method of distributing Magento components, Magento 2 scans traditional Magento 1 folders like app/code, app/design, lib/internal, and app/i18n for registration.php files. If Magento finds a registration.php file, it will treat the folder as a component. That’s why you can still drop files in app/code/Package/Module and they still work. It’s great, and very important that Magento still has this functionality, but semantic versioning can’t help you here, because semantic versioning is dependent on composer.

In my mind, this would be an acceptable version of progress if there was a straight forward way for independent developers and integrators to distribute their extensions via composer.

While it’s technically possible for a third party developer or a tech savvy agency to do this, Magento Inc. has spent near zero effort documenting or creating tooling around this process. Instead, developers are encouraged to distribute extensions via Magento’s composer based Magento Marketplace.

This is one of those places where technology and business interests intersect, and even Otto von Bismarck would have trouble navigating the waters. However, if we take a realpolitik look at the world, there are (at the time of this writing) four themes on marketplace that have access to the protections of semantic versioning, and (again, at the time of this writing) nineteen themes on Theme Forrest that do not have the protections of semantic versioning. Also, massively popular Magento 1 vendors like Unirgy are skipping Marketplace and distributing via app/code — again, losing the protections of semantic versioning.

A Magento ecosystem where third party developers have straight forward tools for composer hosting would be better for everyone.

SemVer and APIs

The penultimate failure of semantic versioning is “The Magento API”. Here we’re not referring to the HTTP based method (REST, SOAP, etc.) of doing things with a Magento system. Instead we’re referring to the SemVer concept of an API.

Software using Semantic Versioning MUST declare a public API. This API could be declared in the code itself or exist strictly in documentation. However it is done, it should be precise and comprehensive.

Magento 2 does have a SemVer API. Magento 2 uses the @api annotation of PHP Doc Blocks to tag methods that are part of its API. For example, the following method

#File: vendor/magento/framework/DataObject/Copy.php
namespace Magento\Framework\DataObject;

class Copy
     * Copy data from object|array to object|array containing fields from fieldset matching an aspect.
     * Contents of $aspect are a field name in target object or array.
     * If targetField attribute is not provided - will be used the same name as in the source object or array.
     * @param string $fieldset
     * @param string $aspect
     * @param array|\Magento\Framework\DataObject $source
     * @param array|\Magento\Framework\DataObject $target
     * @param string $root
     * @return array|\Magento\Framework\DataObject|null the value of $target
     * @throws \InvalidArgumentException
     * @api
    public function copyFieldsetToTarget($fieldset, $aspect, $source, $target, $root = 'global')

is marked with @api. This means the copyFieldsetToTarget of the Magento\Framework\DataObject\Copy class is part of the Magento SemVer API. This class is part of the magento/framework component. The magento/framework component’s version is (at the time of this writing) 100.1.0

//File: vendor/magento/framework/composer.json 
"version": "100.1.0",

If Magento changed the copyFieldsetToTarget method to fix a bug, but the method behaved the same way, the next released version of the module would be 100.1.1. That is, they’d increment the patch version number.

If Magento changed copyFieldsetToTarget to add some new functionality (by adding an optional parameter), or marked a new method in the Magento\Framework\DataObject\Copy class with @api, the next released version of the module would be 100.2.0. That is, they’d increment the minor version number.

If Magento changed copyFieldsetToTarget so its behavior changed, even when called with identical parameters, this would be a non-backwards compatible change, and the next version of the module would be 101.0.0. That is, they’d increment the major version number.

As we said, Magento has a SemVer API and this is a good thing. However, where Magento’s SemVer API fails is in the comprehensive requirement. Only a minority of Magento’s methods are covered by @api. Changes to most Magento code will not create a component version change because that code is not covered by an @api annotation.

If a module developer stuck to the methods available in the SemVer API, they’d either have a useless module, or would have to reimplement non-@api functionality in their own code. The later defeats the purpose of adopting a platform like Magento in the first place. Worse, despite having real world feedback (in the form of released Magento 2 extensions), Magento 2.1 did not significantly increase the surface area of the SemVer API.

The practical net negative for third party developers is every Magento version change brings with it the possibility of breaking things. A responsible third party developer can never be ready on day one with an extension, even for a small patch change.

SemVer, Javascript, and DSLs

The final failure of SemVer in Magento 2 is its assumption of a PHP only universe. While better evangelism and a real, comprehensive API would be great improvements to Magento 2, SemVer only covers PHP methods, and there’s a huge component of Magento 2 that’s still programming, but falls outside of PHP.

While Magento 2 has adopted a number of modern javascript approaches, there’s no equivalent to SemVer for Magento’s many RequireJS modules. This leaves third party developers out in the cold when it comes to using core Magento RequireJS modules.

Similarly, the two XML based domain specific languages (DSL) Magento uses to generate layouts (Layout Handle XML files and UI Component XML files), have no concept of SemVer. Magento 2 developers have felt this acutely with Magento 2.1, which introduced backward incompatible changes to the backend UI Components that manage grids and forms.. Developers are faced with either leaving customers on the 2.0.x branch behind (a branch which Magento will support for 2 years), or maintaining two branches for their extensions. The multiple branch approach seems the only way forward, but its a labour intensive effort that puts the ROI of Magento 2 extension development in jeopardy.

Wrap Up

While the relationship between Magento 1 and its third party developer ecosystem was strained at times, the Magento 1 core team operated under an informal “don’t break things” policy that worked well (more or less) for the lifetime of the Magento 1 product. As a Magento 1 extension developer, I always worried about little breaking changes to an extension, but never worried about systems vanishing.

Magento 2, on the theoretical level, formalizes this with a semantic versioning policy. While the theory is sound, in practice the only people currently served well by the semantic versioning policy are the Magento core engineers, who now have something to point at when they break something. As a Magento 2 extension developer, version changes fill me with dread that entire sub-systems may be removed, leaving me with a short RC cycle to replace those sub-systems, or remove functionality for my customers.

This is usually the part where I try to find some small light of hope or improvement to point at, but until Magento Inc. can move beyond optics and addresses the needs of its third party developers, Magento 2 extension development will remain a tenuous, uncertain thing.

Originally published August 15, 2016

Pulse Storm Launcher for WordPress

Like this article? Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

I’m happy to announce the immediate availability of the Pulse Storm Launcher UI enhancement for WordPress, including integrations with WooCommerce and WP eCommerce. The source is currently available on GitHub, and I’m hoping to carve out some time to get it submitted to the official WordPress plugin directory.

If you’re not familiar with the Magento version of this plugin, it’s an application launcher that allows quick, keyboard only searching of your WordPress admin and store. Since a picture’s worth 1,000 words, please consider this 4 minute screencast the explainer novel.

Pulse Storm Launcher for WordPress from Alan Storm on Vimeo.

Back to WordPress

It’s been an interesting experience flexing some long atrophied WordPress muscles — despite outside appearances there’s content on that was originally written in WordPress. In 2009 I wrote a PHP shim application that fetches content from the various CMS systems I’ve used over the years. Beyond the programming exercise, this shim application lets me switch back and forth between systems without having to do a giant import/export. I also chose WordPress when I moved my personal Portland site off of Square Space (albeit, published statically via a WP-Cli command).

Although I couldn’t have put it into the same words back then, in 2009 I was looking for a platform, and WordPress was in the running until Magento came along. At the time I was a little skeptical of the old cruft and code their backward compatible religion seemed to encourage.

Seven years later, with a little more systems programming under my belt, I’m significantly impressed with the trade-offs the Wordprss teams have made to keep moving the platform forward while ensuring it remains a collaborative enviornment for the community. Their event based programming (action and filter hooks) has proven a resilient system for plugin developers, and a fertile ground for functional programming in PHP.

The recent (and ongoing) work on the RESTful API is a boon for anyone doing javascript work with WordPress. Its also a nice way for returning-to-the-platform plugin developers (cough) to get at structured data without needing to wade through a decade of post querying code.

All in all, coming back to WordPress felt like visiting the opposite trailhead on that path not taken all those years ago. I’m still available for all your Magento consulting needs, but I do hope the work and business gods conspire to land some more WordPress projects in my inbox.

Originally published August 10, 2016