Categories


Archives


Recent Posts


Categories


Challenges of a Full Packagist Mirror for Magento 2

astorm

Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

Updated for Magento 2! No Frills Magento Layout is the only Magento front end book you'll ever need. Get your copy today!

I recently started a new series over on my main website that sets out to explain everything a Magento 2 developer would need to know about composer. The first article covers setting up a satis mirror of repo.magento.com, Magento 2’s official composer repository.

An ideal deployment system wouldn’t just mirror Magento 2’s official repository, but it would also mirror the standard PHP packages from packagist that Magento uses. This way no external services (Magento, GitHub, Packagist, etc) would be able to block day-to-day deployments and builds of your system.

I ended up punting on that ideal scenario after running into small little edge case after small little edge case. It’s not impossible, but it makes me start to understand the appeal of software like Toran Proxy.

Regardless – here’s a few heads up if you decide to go down that road.

Case Sensitive File System

Your going to want a *nix server and/or VM to build your packagist mirror. I ran into trouble on OS X thanks to the case insensitive file system (running OS X on a case sensitive file system presents its own set of problems).

Satis will un-archive and re-archive every archive file it downloads, which means projects with archives that contain case-sensitive incompatible files (ex. Setup.txt and setup.txt in the same folder). This is more than a theoretical problem – several archives met this criteria and halted the satis mirror build on my OS X machine.

Lots of Disk Space, Bandwidth

You’ll need need a decent chunk of disk space for this. I’ve been experimenting will mirroring only the projects (and those project’s dependencies) that Magento needs, and my dist folder came in at 6.8 GB using tar archives. If if you change things up to zip archives (i.e. you get some file compression) you’re still looking at an almost 2GB chunk of someone’s hard drive.

You’ll also want to be careful you don’t trip any over sensitive network usage precautions your WiFi node and/or work network might have. Building a satis mirror can involve significantly more internet traffic than a simple project build. When I started this project I was working at a local library, and managed to get myself banned from their network for an hour or two.

Make Sure you Have unzip Installed

This one was a big maddening. I was using a PHP 7 virtual machine based on Homestead, and satis was consistently crashing with a Segmentation Fault error whenever it tried to grab the a specific project zip file (codeception’s). It was maddening because this would only happen one satis reached the codeception dependency.

The high level reason for this? I didn’t have the unzip program installed on my virtual machine, and (I assume?) satis/composer falls back to using PHP code to unzip a package when it can’t find a command line unzip. There was likely something about the ZIP archive that didn’t agree with PHP’s implementation of zip – or perhaps PHP 7 just seg faults if you try to use a command line program that isn’t there.

Likely more to come on this.

Copyright © Alan Storm 1975 – 2019 All Rights Reserved

Originally Posted: 18th April 2016