Two Fridays ago I attended a mid-day tech talk at Copious, a fast growing agency here in Portland, OR. The talk was called Scaling Magento, but Copious isn’t a Magento agency. Copious is a full service digital strategy agency, and one of the rare few with an engineering department.
Copious recently took on a high profile client with a large Magento system. The presentation never mentioned this client by name (“SEC concerns”), but if you scan through the slides and look at the sort of order volume Copious scaled for, it’s clear this was a merchant with the sort of sales volume that no system could handle out of the box, and a client whose sales volumes (likely) exceed any cloud provider’s annual revenue.
In other words, the perfect case study for a custom scaling effort.
The presentation was split into three sections. First, Aaron Edmonds talked about scaling Magento’s application code. This was followed by an overview of the data center side of the operation from Kyle Terry. Finally, Reid Parham talked about the black art of total project management and delivery: In other words, scaling your team.
While it’s always interesting to hear how folks scale Magento, it was extra interesting to hear how an agency outside the Magento community of partner agencies tackled these challenges, especially given that two separate (unnamed) Magento partners had tried, and failed, to roll out a system for this particular client.
Scale your Code
The first section of the talk was led by Aaron, and covered scaling your code. Much of this section will be familiar to regular readers of this site. Aaron succinctly summed up much of the tribal knowledge that day-to-day Magento developers have internalized.
Two interesting bits of trivia that came up during Aaron’s section were the size of the Magento project Copious was scaling (820,000+ lines of code), and uncovering Magento’s longest class name
Trying tweeting that in an @ reply.
Aaron also shared a smart bit of Magento reindexing optimization. It turns out Magento will reindex the entire catalog — including products that aren’t currently visible. I can see how CE and EE ship with this behavior on, but Copious tweaked the
_getProducts method on the
Mage_Catalog_Model_Resource_Url class such that only visible products are reindexed during a catalog URL reindex.
If that doesn’t make sense, consider a concrete example. A shoe, with 9 possible sizes. One configurable product, linking 9 simple products. In a stock Magento system, that’s 10 reindexed products. Now consider a store with 100 similar shoes. That’s a 1,000 products to reindex. By pulling only visible products, (i.e. the simple products don’t need URLs), that’s only 100 products to reindex. Now consider multiple configurable attributes, and you can see how this one small optimization reduced the indexing time dramatically.
Scale your System
The next section of the talk was “scaling your system”. The was led by Kyle, and focused on hardware, deployment, and databases. Regarding hardware, there’s no silver bullet, but if you melt that silver bullet down, and use it to buy a lot of hardware, you can scale Magento pretty successfully.
The slides contain a lot of specific information on hardware setup (yes, physical hardware), so I won’t repeat that here. It’s a tried and true LAMP stack scaling solution, with front-end web servers, master/slave replication on the database layer (no slave lag checkout bugs seen!), and dedicated redis caching servers. Be sure to pay extra attention to the processors – while Copious’s client had a significant hardware budget, smart spending on crazy processors for the database, less crazy processors for the web servers, and budget processors for the caching servers let Copious spread that big budget further than a novice consulting shop might.
It’s also interesting to note they made some minor kernel tweaks to their server OS. Again, a less experienced consulting shop might run out and buy new hardware when their servers hit a certain ceiling, but knowing which kernel setting are adjustable with a recompile can help spread your hardware budget even further.
Although it was only mentioned in brief, for me the most interesting part of this section was Copious’s database integrity tests. Any developer who’s spent some time with Magento knows that sometimes foreign key contraints go awry, or key contraints that should be there aren’t, or extensions and lack of transactions in important parts of the system lead to incomplete, invalid data in the database.
It sounds like Copious knows this, accepts this, and has written tests to detect problem situations. Rather than try to fix a system they don’t completely control (Magento), they’ve used engineering skills to scan for, detect, and alert system developers of these integrity problems so they can be fixed before they cause real problems.
Your own personal Agent Smith, running down problems in the Matrix.
Scale your Team (Reid in Repose)
The final section of the talk, led by Reid, was less technical, and more focused on the problems of managing teams on a large project that involves both client deliverables and a custom engineering effort.
That I work alone and independently should tell you everything you need to know about my project management strengths and weaknesses, so there was a lot to digest in this portion of the talk.
One part of this section that stood out for me was the number of vendors involved in the project. There were 10 systems level vendors (ERP, tax calculation, Akami, etc.), and over 20 different product/accessory vendors to deal with. This is one of those hidden challenges to online retail work — because the industry relies so heavily on specific vendors for solutions, a certain project complexity threshold is quickly reached.
For Copious, this meant gaining the trust of their client such that Copious could act as the client with these vendors. As a corollary to Copious’s experience, I’d remind solo developers (or even small teams) that it’s important to get the full vendor picture before agreeing to a project, and making sure you’re not taking on the responsibility of full vendor management, but that it is being taken care of.
It’s also encouraging to hear that, despite working for a top secret client with SEC concerns and the sort of secrecy that’s becoming de-rigor in global business, Copious was able to contribute back to the open source ecosystem. Patches were submitted to Colin’s Redit modules (now officially sanctioned by Magento), as well as Redis itself.
The Web™ has come a long way since 1995. Hosted services like Blogger, Tumblr, Square Space, Shopify, Etsy, etc. means the average web user doesn’t need to think about the technical details of their web presence. While amazing, these steps forward have come at the cost of hiding the incredible technology complexity of handling large scale web systems. When a businesses needs to grow beyond what a cloud provider can offer, the easy solutions are few and far between.
This, more than anything, seems like the future for software service professionals: Learning how to scale systems like cloud providers, and setting client expectations that this is a necessary step. Like everything in business, it’s all about finding partners you can trust, where each party can contribute to the success of the other.
I’m happy to announce a new chapter in my Leanpub published book, No Frills Command Line Magento. The new chapter Built in Shell Commands covers the three
shell commands that ship with Magento. You’ll learn how to clean log files, reindex the site, and use Magento’s compiler to speed up your system — all without ever looking at a browser window.
Already bought No Frills Command Line Magento? Just head on over to Leanpub and download a new copy, 100% free. Haven’t purchased a copy? Eleven dollars gets you a great first Magento book covering the
n98-magerun tool, and access to future updates covering all aspects of Magento command line development.
We’d be remiss if we didn’t mention this is the second book in Pulse Storm’s No Frills Magento series. The first, No Frills Magento Layout, is still as relevant today as the day it was first published. Whether you’re using an up-to-date Magento 1.8 system, or you’re stuck on a duct tape and bubble gum Magento 1.4 system, No Frills Magento Layout will give you the background you need to make sense of Magento’s powerful (but complicated) layout XML system.
Content Returns Soon!
On a personal note, I’ve been knee deep in some end-of-year consulting. New, fresh Magento content will return here shortly, starting next week with a writeup of the Copious Scaling Magento talk.
Apologies for the quiet around here, but I’ve been hard at work finishing up the latest release of Commerce Bug. The big feature this time around? Magento 2 support! Don’t worry, Commerce Bug is still, and will remain, fully functional with Magento 1.
Update: As expected, A few weeks after this post eBay released a significant update to the Magento beta. This broke the new Commerce Bug Magneto 2 functionality. We’re working to catch up with the core team and re-restore Magento 2 functionality for Commerce Bug.
Not using Commerce Bug? Why not? Commerce Bug will save you hours a day in Magento development time. Tasks that used to take the entire morning can be completed in 10 minutes with Commerce Bug at your side. It’s the perfect tool when you’re learning Magento, and the perfect tool for experts who need to cut through the tedium of Magento’s deep class hierarchies. Checkout the feature demonstration screencast or the Learning Magento with Commerce Bug series for detailed information, and then get your copy today.
Why Magento 2
After Commerce Bug’s first year of sales, the plan was for Commerce Bug 2 to be Magento 2 compatible. However, Magento 2 always seemed to be “a few” financial quarters away from release. After a few rounds of that it was easy to read the writing on the broad side of a mixed metaphor. Commerce Bug 2 morphed into a feature release and Pulse Storm’s customers have been happy with the results.
Magento 2 is still “a few” financial quarters away from release. However, there have been signs of life recently from the Magento development team. After a five and half month delay Magento 1.8 went from alpha to full release, the Magento 2 GitHub received a major code update a month ago, and eBay product executives has been quietly making enquires about key Magento 2 features.
Does this mean a Magento 2 release is imminent? I have no idea, but I do think it’s time for serious Magento developers to start looking at the Magento 2 code base, learning about the changes that have taken place, and using their downtime to start refactoring key pieces of code to work in both Magento 1 and Magento 2. Commerce Bug is the best tool money can buy to help you do this, and that’s why I’ve added Magento 2 support to Commerce Bug.
Magento 2’s an odd beast, at one time familiar, and in other ways different and new. Some of the more jarring changes for an old timer were
“Modern” splitting of the application code in
app, and the public web assets in
No more module declaration in
No more code pools, modules live directly in
The full PHP class name has replaced
groupname/classnamealiases in factory methods.
Rewrites are still possible, but the syntax has changed
The entire event/observer system is re-implemented (which is where the majority of my refactoring effort went)
There’s a new dependency injection system in place — I didn’t use it, but had to work around its implementation as I looked for the new rewrite syntax.
Magento still has a
_constructmethod, but it’s implemented in
Abstractclasses and has been removed from
The layout system replaced
text/listblocks with a new container type. Containers aren’t a block type, they’re a second sort of thing you insert into a layout.
I’ve got more to say on these changes over on the Magento Quickies site (tag: Magento2). There’s definitely a shift in tone in the code base — it’s still super class heavy, but there’s not a new class for everything that might possibly need to be data modeled in the future. That said, Magento 2 is still more Symfony than Laravel, and more its own thing than anything else.
While I’ve been able to keep the same module code base for Magento 1 and Magento 2, the packaging had to change. There’s no more
tar archive in the
beta-magento2 folder. This isn’t a connect extension, just an archive to untar into the system.
I imagine things might break the next time we get a code dump in the Magento 2 GitHub — targeting a beta platform is always tricky business. Future updates will contain fixes for these future bugs.
Your support of Pulse Storm and its products is what makes this website possible. As always, thank you for your support.