Bypassing a Slow Composer Repository

Like this article? Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

This week I had a long standing to do item I was looking forward to tackling

Write a dead simple “Getting Started” tutorial for Magento/Composer/FireGento

FireGento are a group of European-centric Magento developers who run a hackathon, and cultivate several projects to help them all do their jobs. One of those projects is the ability to manage Magento modules via PHP Composer, and a corresponding Composer repository dedicated to those same Magento modules.

Unfortunately, when I set out to write my dead simple “Getting Started” tutorial, I ran into a major roadblock. Slowness abounded. PHP Composer was hanging after I ran every command. It was borderline unusable, and a far cry from how things should work.

Complicating things further, for all its revolutionizing PHP package/dependency management, there’s not many developers out there who understand Composer’s internals, and Composer’s documentation has that certain “we assume you’ve been coming to the meetings” feel about it.

Stack Overflow was no help, and I opened bugs in both the Composer and FireGento issue trackers with little activity or actionable advice.

So, instead of a dead simple “Getting Started” tutorial, I decided to write up how I’m working around the performance problems I’ve been having with Composer. While this tutorial specifically covers the FireGento repository, the concepts can apply to any Composer repository that’s suffering from network related latency.

The Problem

Composer repositories are lists of package names that link to git repositories (or subversion, mercurial, etc.). It works something like this

Composer: Hey, repository? Do you know about package foo?

Repository: Yes, I do know about package foo.

Composer: Do you know where I can find package foo?

Repository: Yes, it’s at this git/subversion/mercurial/archive URI

Composer: Thanks repository!

Each repository has a packages.json file that contains a list of every package name with a URI for the package’s real location. You can see the FireGento packages here.

If you tried clicking on that link you may see the problem. There’s over 3,000 unique packages in the FireGento repository — over 11,000 if you consider the different versions. That’s a packages.json file that’s over 12 MB in size. The performance problem I ran into was this. Every time Composer needed information in this file, it would download a new version.

Surprisingly, Composer doesn’t appear to use gzip compression. A quick check of both FireGento’s packages.json and the packagist packages.json (the main Composer repository) revealed neither returned gzip encoded content. There was, however, one seemingly odd thing about the packagist packages.json. Despite being responsible for every general PHP package distributed by Composer, it’s tiny.

#https://packagist.org/packages.json
{
"packages": [],
"notify": "\/downloads\/%package%",
"notify-batch": "\/downloads\/",
"providers-url": "\/p\/%package%$%hash%.json",
"search": "\/search.json?q=%query%",
"provider-includes": {
    "p\/provider-active$%hash%.json": {
        "sha256": "05eeca5abedcc69d4bde0e4ec9da117ee6db21b82966a2880b92d715250bbff3"
    },
    "p\/provider-archived$%hash%.json": {
        "sha256": "e3c4ecbf14703c7c27715c722166bfc415de2bd3b6e362d202cde52f2c0e6486"
    },
    "p\/provider-latest$%hash%.json": {
        "sha256": "12394fec0600f586c0da9aa32ae50920fd9eee6bb3ec2335d0245d677c432c3f"
    },
    "p\/provider-stale$%hash%.json": {
        "sha256": "faab0b5a39c5becd7d6beda2ba118484e5f1dad4b46449cbf0bb3d41dee41296"
    }
}}

Doing some reading, it turns out Composer has an undocumented system for splitting packages.json into multiple files to avoid large file downloads. Because of this, it may be the Composer developers never saw the need to implement a caching system for the main packages.json, or to use gzip compression. Put another way, the lead developer may have specifically decided against a caching system to ensure the team was forced to implement a file splitting system that worked.

Regardless, as a third (or fourth?) party, I’m left with a FireGento packages.json that’s over 12MB in size, and taking minutes to download each time. While not a big deal for something that happens once in a great while, it means adding/removing Composer packages to my project is a tedious, time consuming affair.

Decentralized Repositories

While it’d be ideal if Composer had a better caching strategy, or FireGento had a better implementation of packages.json, the creators of Composer’s architecture have given us a way out. Composer’s repository system is decentralized. While packagist is pushed as the main PHP repository, Composer was very deliberately built to support anyone operating a repository. That’s why FireGento can create and manage their own Composer repository of Magento packages.

It also means we can host a repository on our local machine, and make that repository a mirror of the FireGento repository. Hosting the packages.json file on a local machine (or local network) means the download time for that single, enormous packages.json file will be dramatically reduced.

The simplest solution? We setup a website/domain on our local machine with a copy of packages.json, such that we can access it via the development URL http://packages.pulsestorm.dev/packages.json. The rough steps I took to this were

  1. Setup an apache virtual host for a domain you control, or a fake development domain (pacakges.pulsestorm.dev for me)

  2. Setup DNS for that host to point to your computer (i.e. add 127.0.0.1 packages.pulsestorm.dev to your hosts file)

  3. Download the FireGento packages.json to the root of your new virtual host. I grabbed it with the following curl command: curl -LO http://packages.firegento.com/packages.json

With the above in place, let’s consider the performance differences. First, here’s a dead simple composer.json pointing to the official repository.

#File: composer.json
{
    "require": {
        "magento-hackathon/magento-composer-installer": "*"
    },
    "repositories": [
        {
            "type": "composer",
            "url": "http://packages.firegento.com"
        }
    ],    
    "extra":{
        "magento-root-dir": "magento/"
    }
}

If we run this with Composer’s --profile and -vvv verbose options, we see the performance problem. It takes Composer around 94 seconds to download the entire package.

$ composer.phar install --profile -vvv
[3.7MB/0.01s] Reading ./composer.json
...
[73.2MB/94.84s] Writing /path/to/.composer/cache/repo/http---packages.firegento.com/packages.json into cache

However, if we change the composer.json to point to our local mirror (the url field under the repositories field)

#File: composer.json
{
    "require": {
        "magento-hackathon/magento-composer-installer": "*"
    },
    "repositories": [
        {
            "type": "composer",
            "url": "http://packages.pulsestorm.dev"
        }
    ],    
    "extra":{
        "magento-root-dir": "magento/"
    }
}

we end up with dramatically different numbers.

$ composer.phar install --profile -vvv
[3.7MB/0.01s] Reading ./composer.json
...
[73.2MB/1.29s] Writing /path/to/.composer/cache/repo/http---packages.pulsestorm.dev/packages.json into cache

Here it only took Composer 1.29 second to download the repository file. A dramatic improvement, and one that makes the project usable again.

Official Mirror

While curling the packages.json file down to a local machine scratches my personal nerdy-http itch, it does present one problem — staying up to date with changes to the official FireGento repository. If you want to create a more official/stable mirror, you’ll need to familiarize yourself with the FireGento composer-repository project.

This repository is what the FireGento team uses to manage http://packages.firegento.com. It has two branches. The gh-pages branch is the public repository website branch. If you clone this branch.

git clone -b gh-pages git@github.com:magento-hackathon/composer-repository.git

you’ll end up with the full contents of the existing packages.firegento.com website.

In other words, a full mirror of the repository.

$ ls -1
CNAME
connect20
index.html
packages.json

If you’re curious how this is generated, try checking out the master branch of this project

git clone -b master git@github.com:magento-hackathon/composer-repository.git

When you take a look at the project’s master branch, you’ll see

$ ls -l

README.md
repairedConnectPackages
satis.json
script

This is a Satis project. Satis is a “Simple static Composer repository generator”. In other words — it lets users generate a packages.json for their own repository (using the meta-data in satis.json), but lacks the logic that splits packages.json into multiple files like the official packagist repository. If you want to generate this file yourself, you’ll also need to clone the satis repstory and install it on your computer.

Wrap Up

The FireGento Composer repository, like so much of open source developer infrastructure these days, suffers from lack of resources. Make no mistake, it’s awesome that it exists, but a two day hackathon project is never going to take scaling, usability, and user-adoption concerns into account. On paper (or in bits), a one to two minute download for packages.json doesn’t seem like a big deal, but in practice it makes working with the project a horrible tedium that only the most dedicated/vested developers will stick around for.

For us, this is another great example of how knowing and understanding the underlying concepts behind our tools can make us much more efficient developers. Realizing that Composer isn’t magic, and being able to diagnose a problem without Google means you can solve problems that third parties like FireGento haven’t gotten around to solving yet. As “open source” and software development becomes more and more fragmented, these skills will become more valuable than ever.

Originally published April 23, 2014

Review of Grokking Magento

Like this article? Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

In 2014 a newcomer to Magento is, all things considered, in much better shape than a newcomer was in 2009. While far from complete, the developer’s series (originally written by me, since updated by Magento staff) gives a developer the grounding they need in Magento’s sometimes-convention-sometimes-configuration MVC system.

I’m often emailed by developers who, once grounded in the system conventions, aren’t sure what’s next. Many want to pursue certification, others just want to make sure they’ve jumped down the right rabbit hole when working on their own system customizations.

Until recently, I haven’t had much advice for these folks other than “Do more Magento and ask questions when you’re stuck”. Commerce Bug’s a great tool for any Magento developer, and No Frills Magento Layout gives a front-end developer the grounding they need in the layout system, but until now there hasn’t been a resource that really helps a developer apply their new knowledge.

I say “until now” because of the new Grokking Magento study guide series, which is the perfect next step for a developer looking to sharpen their skills.

Grokking Magento

Grokking Magento grew out of the work of Magento power training duo Vinai Kopp and Ben Marks. Vinai and Ben, both accomplished Magento developers in their own right, have been the public face and lead-trainers of Magento’s developer training sessions for a number of years. This means they not only have the required knowledge to write proficiently about Magento, but it also means they’ve spent thousands of hours training developers and know where the cruxes are in Magento’s learning curve.

This first book in the series, authored by Vinai, uses Magento’s Request Flow as a jumping off point for deeper topics. After a quick chapter on setting up your Magento editing environment and a review of testing frameworks, we get to the meat of the book. Vinai and Ben have taken 7 examples from the Magento study guide

  • Rewrite the sales order model to add customer group ID to email template

  • Redirect to / when users access CMS homepage directly

  • Controller action rewrite to manipulate before/after_auth_url

  • Custom front-end controller route using Response Body

  • Dynamic/Conditional class rewrites

  • Diagram the dispatch process

  • Store view selection priorities

For each topic they restate the problem, cover real world scenarios where the task might be useful, provide a step by step guide into how you’d research the particular issue in Magento’s code base, and then provide a solution to the problem.

Beyond the obvious benefits of providing cookbook style solutions to specific problems, the real value of this book is explaining the thought process of how to solve these problems on your own. While the book is subtitled “Basics and Request Flow”, you’ll end up learning about other overlapping systems. Vinai and Ben teach you how to think like a veteran Magento developer, and that’s a skill you’ll apply over and over again in your day to day work.

The book is also littered with bits of Magento technical/historical trivia that tries to explain some of the seemingly odd technical choices that the core team made over the years. While not strictly necessary, these sections provide a little context around what most of us call “Magento being Magento”, and are a nice treat if your curiosity bends that way.

The book wraps up with a quick appendix section on Varien_Object. Instead of a typical “getter and setter” retread, Vinai and Ben cover a few of the lesser known features of Varien_Object based objects.

All in all, Grokking Magento is an excellent addition to the world of resources available to aspiring and established Magento developers. If you’re wondering where to go next, it’s an excellent choice, and even if you’re an experienced Magento developer you’ll pick up a trick or two.

Originally published April 14, 2014

Magento Ultimate Module Creator Review

Like this article? Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

The PHP community draws developers from all sorts of different backgrounds — there’s the technical and academic on one end of the spectrum, and the bang on the keyboard until it works on the other. Because PHP is such an accessible language, all these people are writing code that solves their problems. In some cases “writing code” may be an overstatement — editing files that contain code would be more accurate.

This presents a dilemma for people on the technical/academic side. Namely — what sort libraries and tools should we be building? Make something too abstract and the folks on one end of spectrum lose out. Make something too simple and we curse ourselves to banging out repetitive error prone code.

Time and time again, the comprise PHP projects come up with is code generation. That is — we accept that some systems will be too abstract for non programers to understand, so we develop systems that automatically generate the boring, repetitive code we don’t want to be stuck writing.

Magento Code Generation

Magento’s no different — although I’ve always been surprised how stunted the evolution of the Magento code generation tools are. The structure is there for something like the Rails command line, but our code generation tools never reached that level of sophistication. The original Netz98 module creator (also on GitHub) took a bold step in that direction, but never went much beyond the generation of a single CRUD model in a Magento module.

Despite this, up until a few days ago I still recommend The Netz98 Module Creator as a way for newbie Magento developers to get a feel for the structure of a Magento admin module. I was Googling for it the other day, and was surprised when a different project popped up higher in the listings.

I’d heard of Silk Software’s creator before — but since it’s not open source I wasn’t interested. However, I’d somehow missed the Ultimate Module Creator. When I clicked through to the Magento Connect page (also on GitHub), I was surprised to see The Magento Stack Exchange champion, Marius. How had I missed this?

After reviewing the features, the Ultimate Module Creator is a big step forward from the venerable Netz98 project. If you’re doing custom Magento development with many entities, you owe it to yourself to take a look.

Ultimate Module Creator Features

The first step forward for the Ultimate Module Creator is — it’s distributed as a proper Magento module. Files are installed in the following locations

app/code/community/Ultimate/ModuleCreator/
app/design/adminhtml/default/default/template/ultimate_modulecreator

After installing the module (manually, or via Magento Connect), you’ll have a new Magento admin menu item: System -> Module Creator

Clicking on this link brings up a grid of all your saved modules. This is the next way the Ultimate Module Creator is superior to projects that have come before — you can save your module’s configuration, come back, make adjustments, and re-save the module. Most other module creators I’ve seen are stateless — once you generate a module, that’s it.

Clicking on the Create New Module button, or editing an existing module, will bring up the Magento standard multi-tab Ultimate Module Creator user interface.

Here the Ultimate Module Creator suffers from its environment. The creation of a Magento module has a lot of different options. Magento default UI widgets are optimized for a simple “this field correspond to this value” user experience. The first time you look at the module creator UI you may feel al little overwhelmed.

The key to these sorts of user-interface patterns is they require the user to invest a bit of time learning about the problem domain the application is solving. This requires more investment that the typical “intuitive on the first run” patterns that are popular in mobile development today — but once you learn the interface, you’ll have a much more powerful feature set on your hands.

Marius seems to understand this trade-off, and accommodate for it. Each field in the UI has an additional help icon

Clicking on this icon will bring up a small help dialog explaining what the field is for.

This allows expert users to quickly center themselves in the application, and gives novice users the nudge in the right direction they need. All of these help fields are also repeated in the Help tab

Clicking on the Entities tab brings up the other major innovation in the Ultimate Module Creator

Namely, you can setup multiple entities in one module. For reasons that are a little opaque to me, the idea that a single Magento module can contain multiple CRUD models was met with resistance in certain development communities. Ultimate Module Creator embraces the natural design of Magento, and lets you create an unlimited number of models entities for your module.

In fact, not only does the Ultimate Module Creator embrace the design of Magento, it brings missing features to the ORM. Once you’ve setup multiple entities, you’ll have access to an Entity Relations tab. Here you can create parent/child or many-to-many relationships between your entity models.

Similarly, there’s also options that will allow you to setup relationships with products.

This feature will also give your entity UI forms an AJAX product picker.

There’s tons of other features as well, including

  • Creation of front-end files, and a front-end view controller for your module
  • Widget creation
  • API Module Creation
  • RSS Feed Creation

All in all the Ultimate Module Creator lives up to its name and is a worthy successor to the pioneering Netz98’s venerable Module Creator project. I’ll be using it on my future Magento projects, and I’m looking forward to the new version Marius is working on. We all owe an open source debt to Marius for creating such a valuable and feature rich tool.

Originally published April 7, 2014