Magento Config Revisited: Interlude

Like this article? Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

We hope you’re enjoying the latest Magento programming series covering the the global config. Unfortunately, the former core team member we have chained in the basement escaped over the weekend without finishing his latest assignment. We have dogs canvasing the neighborhood and a foot patrol tracking the GPS we embedded at the base of his skull, so our next installment will be posted soon enough.

Since we’re here, we’ll take the opportunity for some rank commercialism. These articles are all made possible by Pulse Storm. More specifically, they’re financially supported by the Pulse Storm Store where you can purchase Commerce Bug, the world’s best Magento debugging extension and No Frills Magento Layout, the only book on Magento’s layout system you’ll ever need. Beyond supporting the site, purchasing these products will make you a more efficient Magneto developer, and they’ll pay for themselves in the first day.

That’s it for now. Don’t forget to keep an eye on Magento Quickies, a collection of quick tips for Magento programmers, as well as Stack Overflow’s Magento tag. Happy programming!

Originally published January 31, 2012

Magento Configuration: Loading Declared Modules

This article is part of a longer series exploring the Magento global configuration object. While this article contains useful stand-alone information, you’ll want to read part 1 of the series to fully understand what’s going on.

When our last article finished up, we had successfully instantiated the global config object (Mage_Core_Model_Config), and loaded the base configuration files from app/etc/*.xml. In this article we’ll be covering the loading of Magento’s module configuration.

A key piece of the Varien/Magento business strategy involved relying on third-party developers to fill holes in their startup-developed system. It’s no surprise the Magento engineers chose to implement a formal system of code modules so developers across the world could avoid stepping on each other’s toes. While it’s still possible to create conflicting modules in Magento’s system, it’s done a good enough job and enabled a world wide community of commercial and open-source extensions

Magento’s configuration loading is a vital step to bootstrapping the module system. The Mage_Core_Model_Config object identifies which modules are active in a system, and loads the necessary information into the global config that will allow these modules to function properly.

Time to Load the Modules

We’re going to start in the Magento app object’s run method

#File: app/Mage.php
public function run($params)
{
    $options = isset($params['options']) ? $params['options'] : array();
    $this->baseInit($options);
    Mage::register('application_params', $params);

    if ($this->_cache->processRequest()) {
        $this->getResponse()->sendResponse();
    } else {
        $this->_initModules();
        $this->loadAreaPart(Mage_Core_Model_App_Area::AREA_GLOBAL, Mage_Core_Model_App_Area::PART_EVENTS);

        if ($this->_config->isLocalConfigLoaded()) {
            $scopeCode = isset($params['scope_code']) ? $params['scope_code'] : '';
            $scopeType = isset($params['scope_type']) ? $params['scope_type'] : 'store';
            $this->_initCurrentStore($scopeCode, $scopeType);
            $this->_initRequest();
            Mage_Core_Model_Resource_Setup::applyAllDataUpdates();
        }

        $this->getFrontController()->dispatch();
    }
    return $this;
}

Our last article was all about the call to baseInit. This time we’re interested in the following method invocation

#File: app/Mage.php
$this->_initModules();

This is the method that kicks off loading module information into the Magento global configuration tree. If we look at the _initModules definition

#File: app/Mage.php
protected function _initModules()
{
    if (!$this->_config->loadModulesCache()) {
        $this->_config->loadModules();
        if ($this->_config->isLocalConfigLoaded() && !$this->_shouldSkipProcessModulesUpdates()) {
            Varien_Profiler::start('mage::app::init::apply_db_schema_updates');
            Mage_Core_Model_Resource_Setup::applyAllUpdates();
            Varien_Profiler::stop('mage::app::init::apply_db_schema_updates');
        }
        $this->_config->loadDb();
        $this->_config->saveCache();
    }
    return $this;
}

the method call we’re interested in is 

#File: app/Mage.php
$this->_config->loadModules();

This is the code that will tell the config object it’s time to load the XML information for each module. The surrounding if block ensures the full configuration loading only happens if Magento can’t load the configuration information from cache. This bit of code is why the phrase “clear your cache” is on every Magento developer’s lips.

What Needs to be Loaded?

Taking a look at our config class, we can see the loadModules method.

#File: app/code/core/Mage/Core/Model/Config.php
public function loadModules()
{
    Varien_Profiler::start('config/load-modules');
    $this->_loadDeclaredModules();

    $resourceConfig = sprintf('config.%s.xml', $this->_getResourceConnectionModel('core'));
    $this->loadModulesConfiguration(array('config.xml',$resourceConfig), $this);

    /**
     * Prevent local.xml directives overwriting
     */
    $mergeConfig = clone $this->_prototype;
    $this->_isLocalConfigLoaded = $mergeConfig->loadFile($this->getOptions()->getEtcDir().DS.'local.xml');
    if ($this->_isLocalConfigLoaded) {
        $this->extend($mergeConfig);
    }

    $this->applyExtends();
    Varien_Profiler::stop('config/load-modules');
    return $this;
}

Merging (or extending, see the first article in our series) each module’s config.xml tree into the main config object is actually a two step process.

First, we need to identify which modules are declared in the system, and which code pool they reside in. Once those modules have been identified we can start step two; loading their config.xml files from disk.

It’s worth repeating the two types of configuration files in play here. Files in 

app/etc/modules/*.xml

are for declaring Magento modules. A file here tells Magento to look for a module.

Files in

app/code/[code pool]/Packagename/Modulename/etc/config.xml

are the files that contain the actual configuration information for a module.

Declared Modules

Jumping to the _loadDeclaredModules method, our first step is to grab a list of all the “declared” modules.

To identify which modules are declared, Magento will glob up the files in 

app/etc/modules/*.xml

and merge/extend them into the global config. This will let Magento know which modules are installed into the system, which are active, and which code pool the module files are loaded in. By loading this information into the global configuration tree, Magento ensures the information will be available to any developer who wants it without having to reload everything.

This loading and extending is handled by the call to _loadDeclaredModules. 

#File: app/code/core/Mage/Core/Model/Config.php
protected function _loadDeclaredModules($mergeConfig = null)
{
    $moduleFiles = $this->_getDeclaredModuleFiles();
    if (!$moduleFiles) {
        return ;
    }

A declared modules is a module that has a configuration node in 

app/etc/modules/*.xml

The _getDeclaredModuleFiles method is responsible for fetching a list of these files. Unlike loading the app/etc/*.xml files (per the last article), this is far from a simple file glob. Let’s take a look

#File: app/code/core/Mage/Core/Model/Config.php
protected function _getDeclaredModuleFiles()
{
    $etcDir = $this->getOptions()->getEtcDir();
    $moduleFiles = glob($etcDir . DS . 'modules' . DS . '*.xml');

    if (!$moduleFiles) {
        return false;
    }

    $collectModuleFiles = array(
        'base'   => array(),
        'mage'   => array(),
        'custom' => array()
    );

    foreach ($moduleFiles as $v) {
        $name = explode(DIRECTORY_SEPARATOR, $v);
        $name = substr($name[count($name) - 1], 0, -4);

        if ($name == 'Mage_All') {
            $collectModuleFiles['base'][] = $v;
        } else if (substr($name, 0, 5) == 'Mage_') {
            $collectModuleFiles['mage'][] = $v;
        } else {
            $collectModuleFiles['custom'][] = $v;
        }
    }

    return array_merge(
        $collectModuleFiles['base'],
        $collectModuleFiles['mage'],
        $collectModuleFiles['custom']
    );
}

First, Magento globs up a list of files, 

#File: app/code/core/Mage/Core/Model/Config.php
$etcDir = $this->getOptions()->getEtcDir();
$moduleFiles = glob($etcDir . DS . 'modules' . DS . '*.xml');

after which it needs to reorder the files in the $moduleFiles array before returning a final list. Here’s the plain english explanation of the sorting

  1. Mage_All.xml is always first
  2. Any files that start with Mage_ come next
  3. Any other files (with a non-Mage namespace) are last

This is necessary to ensure the configuration for any core Magento modules load before any third party modules. While Magento offers the ability to configure the loading order via a <depends/> tag, by jury-rigging the order ahead of time Magento ensures third-party modules without such a tag are loaded last.

Back up in _loadDeclaredModules, our $moduleFiles variable now contains an array of values, something like this

array
    0 => string '/path/to/magento/app/etc/modules/Mage_All.xml' (length=80)
    1 => string '/path/to/magento/app/etc/modules/Mage_Api.xml' (length=80)
    2 => string '/path/to/magento/app/etc/modules/Mage_Authorizenet.xml' (length=89)
    3 => string '/path/to/magento/app/etc/modules/Mage_Bundle.xml' (length=83)
    4 => string '/path/to/magento/app/etc/modules/Mage_Centinel.xml' (length=85)
    5 => string '/path/to/magento/app/etc/modules/Mage_Compiler.xml' (length=85)
    6 => string '/path/to/magento/app/etc/modules/Mage_Connect.xml' (length=84)
    7 => string '/path/to/magento/app/etc/modules/Mage_Downloadable.xml' (length=89)
    8 => string '/path/to/magento/app/etc/modules/Mage_ImportExport.xml' (length=89)
    9 => string '/path/to/magento/app/etc/modules/Mage_PageCache.xml' (length=86)
    10 => string '/path/to/magento/app/etc/modules/Mage_Persistent.xml' (length=87)
    11 => string '/path/to/magento/app/etc/modules/Mage_Weee.xml' (length=81)
    12 => string '/path/to/magento/app/etc/modules/Mage_Widget.xml' (length=83)
    13 => string '/path/to/magento/app/etc/modules/Mage_XmlConnect.xml' (length=87)
    ...

Next up is this chunk of code

#File: app/code/core/Mage/Core/Model/Config.php
$unsortedConfig = new Mage_Core_Model_Config_Base();
$unsortedConfig->loadString('<config/>');
$fileConfig = new Mage_Core_Model_Config_Base();

// load modules declarations
foreach ($moduleFiles as $file) {
    $fileConfig->loadFile($file);
    $unsortedConfig->extend($fileConfig);
}

This code creates a new Mage_Core_Model_Config_Base object to hold a merged XML configuration tree. Then, each of the above XML files are loaded into another Mage_Core_Model_Config_Base object, and this object is merged into the first with extend. This is a similar pattern to the one we saw when loading the base config, except we’re merging files into a new first object instead of merging them into the Magneto global config tree (in $this->_xml). Before we can add this node to $this->_xml, we need to ensure the modules are in the correct order, (per the previously mentioned <depends/> tag)

Before we continue, it’s interesting to note another way this pattern is different from loading our base configuration. Rather than cloning the _prototype object to hold a loaded configuration, the hard-coded Mage_Core_Model_Config_Base object is used. It’s unclear if this is a deliberate deviation, two developers working at a similar system level who had a difference of opinion on things, or a half done refactoring job. We mention it mainly to put to rest any confusion you might have about the weird discrepancy.

It’s also worth noting the call to the extend method

#File: app/code/core/Mage/Core/Model/Config.php
$unsortedConfig->extend($fileConfig);

doesn’t pass in extend’s second parameter. This means identical nodes in the second file will replace nodes in the first. The implication here is you could, theoretically, use your own module declaration file to replace the values of a core module’s declaration. While the system was design to allow this, it’s probably not what you want to do, so be careful with your own declaration files. For a full explanation of extends, see part one of our series.

Depends Sorting

At this point, we have a merged tree in $unsortedConfig that contains all the declared modules for our system, along with their code pools, active state, etc. 

<config>
    <modules>
        <Mage_Core>
            <active>true</active>
            <codePool>core</codePool>
        </Mage_Core>
        <Mage_Eav>
            <active>true</active>
            <codePool>core</codePool>
            <depends>
                <Mage_Core/>
            </depends>
        </Mage_Eav>
        <!-- etc ... -->
    </modules>
</config>

However, we’ve done nothing to merge this into our main configuration tree in $this->_xml. Before we do that, we need to reorder this tree such that modules which depend on other modules (configured with a <depends/> tag) appear later in the list of modules. Later on, this will control the order in which modules are loaded into the system, allowing one module to “depend” on another.

For example, if you look at the declaration file for the Mage_Bundle module, you can see it’s been made to depend on the Mage_Catalog module.

<!-- File: app/etc/modules/Mage_Bundle.xml -->
<config>
    <modules>
        <Mage_Bundle>
            <active>true</active>
            <codePool>core</codePool>
            <depends>
                <Mage_Catalog />
            </depends>
        </Mage_Bundle>
    </modules>
</config>

The first step is looping over our just merged configuration tree, and serializing an array of module information. 

#File: app/code/core/Mage/Core/Model/Config.php
$moduleDepends = array();
foreach ($unsortedConfig->getNode('modules')->children() as $moduleName => $moduleNode) {
    if (!$this->_isAllowedModule($moduleName)) {
        continue;
    }

    $depends = array();
    if ($moduleNode->depends) {
        foreach ($moduleNode->depends->children() as $depend) {
            $depends[$depend->getName()] = true;
        }
    }
    $moduleDepends[$moduleName] = array(
        'module'    => $moduleName,
        'depends'   => $depends,
        'active'    => ('true' === (string)$moduleNode->active ? true : false),
    );
}

Next, this information is passed off to a separate method to sort it based on <depends/> rules.

#File: app/code/core/Mage/Core/Model/Config.php
$moduleDepends = $this->_sortModuleDepends($moduleDepends);

We’re going to skip looking into the specifics of _sortModuleDepends, but it’s worth investigating if you’re interested in that sort of thing. Once sorting is complete, the $moduleDepends array will look something like this

array
  0 => 
    array
      'module' => string 'Mage_Core' (length=9)
      'depends' => 
        array
          empty
      'active' => boolean true
  1 => 
    array
      'module' => string 'Mage_Eav' (length=8)
      'depends' => 
        array
          'Mage_Core' => boolean true
      'active' => boolean true
  2 => 
    array
      'module' => string 'Mage_Page' (length=9)
      'depends' => 
        array
          'Mage_Core' => boolean true
      'active' => boolean true
  3 => 
    array
    ...

with each array of module information being in the correct order.

Next, we create a new config object to hold our sorted module information

#File: app/code/core/Mage/Core/Model/Config.php
$sortedConfig = new Mage_Core_Model_Config_Base();
$sortedConfig->loadString('<config><modules/></config>');

and copy (not merge) any top level nodes from the unsorted configuration information that aren’t the <modules/> node into the new $sortedConfig

#File: app/code/core/Mage/Core/Model/Config.php
foreach ($unsortedConfig->getNode()->children() as $nodeName => $node) {
    if ($nodeName != 'modules') {
        $sortedConfig->getNode()->appendChild($node);
    }
}

This is necessary to preserve any additional information added to the declaration files, such as the <any_additional_information> node below

<config>
    <any_additional_information>
        <added_to>the declaration file</added_to>
    </any_additional_information>
    <modules>
        <Foo_Bar>
            <active>true</active>
            <codePool>core</codePool>
        </Foo_Bar>
    </modules>
</config>

With a new XML “shell” tree built up, we loop over the sorted $moduleDepends data array, and use the information therein to append (again, not merge) information from the unsorted config into a new <modules> node in the new sorted config

#File: app/code/core/Mage/Core/Model/Config.php        
foreach ($moduleDepends as $moduleProp) {
    $node = $unsortedConfig->getNode('modules/'.$moduleProp['module']);
    $sortedConfig->getNode('modules')->appendChild($node);
}

Finally, the <modules> configuration tree, now properly loaded and sorted in the $sortedConfig variable, is merged in with our main global config

#File: app/code/core/Mage/Core/Model/Config.php
$this->extend($sortedConfig);

The end results of which will be an updated global XML tree in $this->_xml property.

Loading the Modules

It wasn’t always pretty, but we now have a new <modules/> node in our global configuration tree. This is a list of modules, but we haven’t yet loaded each of our module’s config.xml files. That’s handled by the following two lines in the loadModules method

#File: app/code/core/Mage/Core/Model/Config.php
$resourceConfig = sprintf('config.%s.xml', $this->_getResourceConnectionModel('core'));
$this->loadModulesConfiguration(array('config.xml',$resourceConfig), $this);

If you’re familiar with Magento, you may know the loadModulesConfiguration method can be used to merge together any XML configuration from all the installed modules. When called with a single parameter, Magento will return a merged configuration object. For example, my Simple Page module uses this method to fetch a list of routes from all simplepage.xml in the system.

However, if a configuration object is passed in as the second parameter, then Magento will use that object as the “caller” object for extends. As you can see above, the Mage_Core_Model_Config object passes itself in (as $this), which means the merged files will be added directly to the global config.

The first line of the above pair is another interesting bit of code to note. In versions prior to 1.6, this call looked a little different. 

#File: app/code/core/Mage/Core/Model/Config.php
$this->loadModulesConfiguration('config.xml', $this);

Rather than passing in an array of configuration file names, a single file named config.xml was passed in. Magento 1.6, however, passes in two file names. We’re getting a little ahead of ourselves, but these are the files Magento will attempt to load from each and every installed module’s etc folder. If we expand $resourceConfig’s parameters, the call looks more like this

$this->loadModulesConfiguration(array('config.xml','config.mysql4.xml'), $this);

Magento 1.6 CE, in addition to merging in config.xml files, will also look for a config.mysql4.xml file. It’s unclear if this was an early attempt at breaking out the new database resource configuration into their own files, or if it’s a portent of things to come.

Regardless of all that, let’s take a look the loadModulesConfiguration method itself. To start with, Magento checks if “local” modules have been disabled. 

#File: app/code/core/Mage/Core/Model/Config.php
public function loadModulesConfiguration($fileName, $mergeToObject = null, $mergeModel=null)
{
    $disableLocalModules = !$this->_canUseLocalModules();
    ...

If true, this will be used later to skip loading modules in the local code pool. (In my 3+ years of using Magneto, I’ve never seen this feature used, but it’s worth being aware of).

Next, Magento ensures we have an object that will be used to hold our merged config.

#File: app/code/core/Mage/Core/Model/Config.php
if ($mergeToObject === null) {
    $mergeToObject = clone $this->_prototype;
    $mergeToObject->loadString('<config/>');
}

As previously mentioned, this is the object that Magento will be merging each config.xml into. In our case this will be the Mage_Core_Model_Config object itself. Similarly, Magento sets up a “merge model” object

#File: app/code/core/Mage/Core/Model/Config.php
if ($mergeModel === null) {
    $mergeModel = clone $this->_prototype;
}

This is the object that will be used to initially load the individual config.xml files prior to merging. If that didn’t make sense don’t worry, all will become clear in a moment.

Before moving on, it’s (once again!) worth noting that we’re back to using the prototype cloning to create our merge objects. When loading the declared modules, Magento simply instantiated a Mage_Core_Model_Config_Base object. The core team moves in mysterious ways.

Merging into the Global Config

With our objects instantiated, we’re ready to go. Using the config object’s getNode method, we grab the (previously inserted) top level <modules/> node and all its children

#File: app/code/core/Mage/Core/Model/Config.php
$modules = $this->getNode('modules')->children();

and then loop over each child node, in order

#File: app/code/core/Mage/Core/Model/Config.php
foreach ($modules as $modName=>$module) {
    //...
}

This is why it was so vital our module information nodes be in a particular order. PHP’s children method will return a list of nodes in document order. In other words the order of the nodes in the document dictates the order each module’s config.xml will be loaded

Inside the foreach loop we have the following code

#File: app/code/core/Mage/Core/Model/Config.php
if ($module->is('active')) {
    if ($disableLocalModules && ('local' === (string)$module->codePool)) {
        continue;
    }
    if (!is_array($fileName)) {
        $fileName = array($fileName);
    }

    foreach ($fileName as $configFile) {
        $configFile = $this->getModuleDir('etc', $modName).DS.$configFile;
        if ($mergeModel->loadFile($configFile)) {
            $mergeToObject->extend($mergeModel, true);
        }
    }
}

The if conditional that surrounds the body of the foreach loop has a pretty obvious meaning. If the module isn’t active, then we skip this particular go through the loop. It’s worth looking at the implementation of the is method 

#File: app/code/core/Mage/Core/Model/Config/Element.php    
public function is($var, $value = true)
{
    $flag = $this->$var;

    if ($value === true) {
        $flag = strtolower((string)$flag);
        if (!empty($flag) && 'false' !== $flag && 'off' !== $flag) {
            return true;
        } else {
            return false;
        }
    }

    return !empty($flag) && (0 === strcasecmp($value, (string)$flag));
}

Here we can see the call to $module->is('active') will return false if

  1. There’s no <active/>; node
  2. The <active/> node has a string value of ‘false’
  3. The <active/> node has a string value of ‘off’

While there’s nothing particularly earth shattering in this code, it does serve as an example of why a developer might use the custom class feature of simple XML objects. Magento has given all their SimpleXML config objects this additional is method.

Back in the loop, if Magento determined local modules are disabled, it will continue on to the next iteration of the loop

#File: app/code/core/Mage/Core/Model/Config.php
if ($disableLocalModules && ('local' === (string)$module->codePool)) {
    continue;
}

Pausing again as an aside, this is a curious mixing of coding styles. On one hand, the loop is skipped for inactive modules by placing its contents in an if block. On the other hand, the way the loop is skipped for disabled local modules uses a continue. It’s likely that the local module disabling came later, and the second developer was unwilling to mess with the outer if clause.

Regardless, if we’re still here Magento will ensure our $fileName parameter is an array, even if we only requested the merging of a single file.

#File: app/code/core/Mage/Core/Model/Config.php
if (!is_array()) {
    $fileName = array($fileName);
}

Finally, we’re at the point where the configuration files are merged in. Magento will now loop over each request file 

#File: app/code/core/Mage/Core/Model/Config.php
foreach ($fileName as $configFile) {
    $configFile = $this->getModuleDir('etc', $modName).DS.$configFile;
    if ($mergeModel->loadFile($configFile)) {
        $mergeToObject->extend($mergeModel, true);
    }
}

and for each (in our case, config.xml and config.mysql4.xml) Magento will use the module name to construct a full path to the configuration file. 

$configFile = '/path/to/magento/app/code/community/Foo/Bar/etc/config.xml'

Then, Magento will attempt to load in and merge the XML file at the constructed path 

#File: app/code/core/Mage/Core/Model/Config.php
if ($mergeModel->loadFile($configFile)) {
    $mergeToObject->extend($mergeModel, true);
}

If $configFile doesn’t exist or is unreadable, loadFile will return false and the extend will be skipped.

Magento does this for each and every declared module. Notice that the extend method’s $override parameter is made explicitly true here. That means if there’s conflicts with specific nodes between modules, the value in the config.xml loaded later will win out. This is one of the many ways a module with a poorly configured config.xml can wreck a system, so be vigilant in making sure your nodes don’t conflict.

Cleaning Up

At this point, we have all our module’s config.xml nodes loaded into the system. We can start using getModel to instantiate objects, query the config for template files, or any of the other countless things that Magento needs its module configuration for. If you’ve ever wondered why the magento app model and configuration model are instantiated by regular PHP code 

#File: app/code/core/Mage/Core/Model/Config.php
self::$_app    = new Mage_Core_Model_App();
self::$_config = new Mage_Core_Model_Config($options);

instead of using the getModel factory, this is why. Magento needs its module configuration in the global config before it can use its getModel factory pattern, and these are the classes that get that information into the configuration. This sort of chicken/egg problem is common in systems programming, and while it’s always easy to see a better way after the system’s been in use for a while, it’s not always easy to see that better way at the get go.

We’re not quite done with the loadModules method yet. After we get our configuration loaded, there’s two more things that happen. First, there’s this bit of code

#File: app/code/core/Mage/Core/Model/Config.php
/**
 * Prevent local.xml directives overwriting
 */
$mergeConfig = clone $this->_prototype;
$this->_isLocalConfigLoaded = $mergeConfig->loadFile($this->getOptions()->getEtcDir().DS.'local.xml');
if ($this->_isLocalConfigLoaded) {
    $this->extend($mergeConfig);
}

Here Magento is remerging the app/etc/local.xml file we loaded back in loadBase (see part one of the series). Magento reloads this file in case any of the loaded modules inadvertently or maliciously replace some local.xml nodes via extend. Without this in place, a module author could replace/ruin important information in local.xml (database connection information, etc.).

The last bit of loadModules we’ll consider is this call

#File: app/code/core/Mage/Core/Model/Config.php
$this->applyExtends();

If we take a look at the applyExtends method, we see a curious bit of code

#File: lib/Varien/Simplexml/Config.php    
public function applyExtends()
{
    $targets = $this->getXpath($this->_xpathExtends);
    if (!$targets) {
        return $this;
    }

    foreach ($targets as $target) {
        $sources = $this->getXpath((string)$target['extends']);
        if ($sources) {
            foreach ($sources as $source) {
                $target->extend($source);
            }
        } else {
            #echo "Not found extend: ".(string)$target['extends'];
        }
        #unset($target['extends']);
    }
    return $this;
}

From what I can gather, this method will search the loaded configuration for nodes with an attribue named extends

<foo extends="/some/other/xpath" />

For each “target node” found, it will use the value in the extends attribute to make another xpath query. Nodes found from this query are known as source nodes.

Then, Magento will go through each source node, and use it to extend/merge the target node

#File: app/code/core/Mage/Core/Model/Config.php    
$target->extend($source);

From what I can see, this feature hasn’t been used by core code since at least Magento 1.3.4 CE, and it’s unclear if it ever was. My instincts say this was an early system that would provide an explicit path for third party developers to overwrite existing configuration nodes with their own values. For whatever reasons, this was abandoned/never-adopted, but it remains a curious artifact. If anyone knows the history and/or current usage of this feature, use the comments below or get in touch.

Wrap Up

So, that’s our module configuration loaded into the Magento global config. As you can see, the code is less than elegant in places, but it gets the job done. You can also start to see why Magento relies so heavily on caching. Without configuration caching in place a Magento store would start bottlenecking at disk access in a heartbeat. as each configuration file was loaded for each individual http request.

If you think we’re done with the loading of the global config — you’re wrong!

Magento’s system configuration variables were an early topic here, but I never covered how these values are persisted in Magento, or how they’re read back out of the database. It turns out Magento’s global configuration tree plays a huge role in this vital Magento sub-system, which will be the topic of our next article.

Originally published January 23, 2012

The Magento Global Config, Revisited

Like this article? Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

If you’ve spent any time with Magento, you know the importance the “global config” plays in a the system. However, you may not fully understand how that configuration tree is built. Usually, most tutorial authors (myself included), will gloss over this and say something like

All the module’s etc/config.xml files are merged into one

While this statement is true, it’s only part of the story. Understanding what’s in this tree, and how it’s loaded, is a necessary step on the road to Magento mastery. There’s more than just module configuration in the Magento global config. Over the next few articles we’ll cover exactly what’s in the configuration tree, and where it comes from.

The specifics of this article refer to Magento 1.6.1 CE, but the principles discussed apply to all versions.

Object Hierarchy

If you open up the main Mage class file, you’ll see the run method

#File: app/Mage.php
public static function run($code = '', $type = 'store', $options = array())
{        
    try {
        Varien_Profiler::start('mage');
        self::setRoot();
        self::$_app    = new Mage_Core_Model_App();
        self::$_events = new Varien_Event_Collection();
        self::$_config = new Mage_Core_Model_Config($options);
        self::$_app->run(array(
            'scope_code' => $code,
            'scope_type' => $type,
            'options'    => $options,
        ));
        Varien_Profiler::stop('mage');
    }
    ...

Magento calls this static method at the bottom of the index.php bootstrap file. The code in the run method is responsible for executing a full Magento HTTP request. If you remove the profiling lines, that means Magento is only 5 lines of code! Who says it’s complicated?

The line we’re interested in is

#File: app/Mage.php
self::$_config = new Mage_Core_Model_Config($options);

This is where the Magento global config is instantiated, and this is the object that will be used to load and interact with Magento’s XML configuration files. For those new to Magento, the Layout XML files are a separate system, covered in my No Frills Magento Layout book. Layout XML files are not strictly configuration files. Instead, Layout XML files form the basis for a domain specific programming language which allows users to interact with the Layout and Block objects. Because of this, they don’t fall under the domain of Mage_Core_Model_Config, and will not be covered in this series.

The Mage_Core_Model_Config class extends the Mage_Core_Model_Config_Base class

#File: app/code/core/Mage/Core/Model/Config.php
class Mage_Core_Model_Config extends Mage_Core_Model_Config_Base
{
}

which in turn extends a Varien_Simplexml_Config class

#File: app/code/core/Mage/Core/Model/Config/Base.php
class Mage_Core_Model_Config_Base extends Varien_Simplexml_Config
{
}

which is a general class for dealing with XML configuration files. These classes exist to create a structure and “configuration context” around PHP’s generic XML objects. By ensuring Magento client developers always ask for configuration information via the Config object, Magento system developers can provide those client developers with shared functionality, and also allow them to take advantage of functionality like caching without having to change their code.

Note: Client here doesn’t refer to “client side browser development”. Instead, client developer means someone using Magento to create a store or application. These people are clients of the of the Magento system. The line between client and service developer is often blurred in Magento, but it’s still a useful metaphor for understanding the intent of core code.

Loading the Config

That’s the config object instantiated, but so far we haven’t loaded any XML files. What gives? If you take a look at the class’s constructor

#File: app/code/core/Mage/Core/Model/Config.php
public function __construct($sourceData=null)
{
    $this->setCacheId('config_global');
    $this->_options         = new Mage_Core_Model_Config_Options($sourceData);
    $this->_prototype       = new Mage_Core_Model_Config_Base();
    $this->_cacheChecksum   = null;
    parent::__construct($sourceData);
}

we see some code, but at first glance the code doesn’t look like it loads any XML. That glance would be correct, but we’ll come back to this constructor code it a bit, so keep it in the back of your mind.

To find the point in the application startup process where an XML tree is loaded, we need to follow the application startup execution chain a few methods down the stack. We’ll be breezing past some things that are worth exploring, but don’t relate directly to configuration loading.

We’ll start with the app model object’s run method, which is called by the Mage class’s run method

#File: app/code/core/Mage/Core/Model/App.php
public function run($params)
{
    $options = isset($params['options']) ? $params['options'] : array();
    $this->baseInit($options);
    Mage::register('application_params', $params);        
...

Early on in this method, a call to $this->baseInit($options); is made. Looking at this method

#File: app/code/core/Mage/Core/Model/App.php
public function baseInit($options)
{
    $this->_initEnvironment();

    $this->_config = Mage::getConfig();
    $this->_config->setOptions($options);

    $this->_initBaseConfig();
    $cacheInitOptions = is_array($options) && array_key_exists('cache', $options) ? $options['cache'] : array();
    $this->_initCache($cacheInitOptions);

    return $this;
}

we can se two items of interest. First, we’re assigning a reference to the previously defined config object the the app model object’s _config property

#File: app/code/core/Mage/Core/Model/App.php
$this->_config = Mage::getConfig();
$this->_config->setOptions($options);

This gives the app model object easy access to the config. Remember, in PHP 5+ objects are references. This means both the Mage class and the core/app object have a reference to the same config object. When one changes, they both change.

Out second item of interest is the call to the _initBaseConfig

#File: app/code/core/Mage/Core/Model/App.php
protected function _initBaseConfig()
{
    Varien_Profiler::start('mage::app::init::system_config');
    $this->_config->loadBase();
    Varien_Profiler::stop('mage::app::init::system_config');
    return $this;
}

The _initBaseConfig method makes a call to the configuration object’s loadBase method. This method is where the first XML files will be loaded from disk. Let’s take a look!

#File: app/code/core/Mage/Core/Model/Config.php
public function loadBase()
{
    $etcDir = $this->getOptions()->getEtcDir();
    $files = glob($etcDir.DS.'*.xml');

    $this->loadFile(current($files));
    while ($file = next($files)) {
        $merge = clone $this->_prototype;
        $merge->loadFile($file);
        $this->extend($merge);
    }
    if (in_array($etcDir.DS.'local.xml', $files)) {
        $this->_isLocalConfigLoaded = true;
    }
    return $this;
}

The first two lines of this method

#File: app/code/core/Mage/Core/Model/Config.php
$etcDir = $this->getOptions()->getEtcDir();
$files = glob($etcDir.DS.'*.xml');

will “glob up” any XML files in the app/etc folder. In a standard base system, the value returned by a call to $this->getOptions()->getEtcDir(); is /path/to/magento/app/etc. In a normally operating system the $files variable will contain the following two strings.

/path/to/magento/app/etc/local.xml
/path/to/magento/app/etc/config.xml

Next, we see the following

#File: app/code/core/Mage/Core/Model/Config.php
$this->loadFile(current($files));

This code takes the first filename in that array, and passes it to the loadFile method. Unless you’ve been using PHP for a very long time, you may not be familiar with the current method. Time was, PHP didn’t have a handy foreach loop. In versions prior to PHP 4, the only way to loop over an array was to use a for($i=0;$i<count($array);$i++) style construct or use PHP’s internal array functions (current, next, etc. to manipulate an arrays internal “pointer”.

Why this construct is being used here is unclear. Perhaps there’s a performance gain, or maybe the core developer was new enough to PHP that they didn’t know about the newer constructs and current was encountered before array_shift. Regardless, all you need to know is the first item of the array, (/path/to/magento/app/etc/config.xml), is being passed to the loadFile method. This method is located on the base Varien_Simplexml_Config class.

#File: lib/Varien/Simplexml/Config.php
public function loadFile($filePath)
{
    if (!is_readable($filePath)) {
        //throw new Exception('Can not read xml file '.$filePath);
        return false;
    }

    $fileData = file_get_contents($filePath);
    $fileData = $this->processFileData($fileData);
    return $this->loadString($fileData, $this->_elementClass);
}

Here we have another small, compact method. First, if the file isn’t readable by PHP (root owned, improper permissions, etc.) Magento will bail on attempting to load it. 

#File: lib/Varien/Simplexml/Config.php
if (!is_readable($filePath)) {
    //throw new Exception('Can not read xml file '.$filePath);
    return false;
}

Next, the contents of the file are loaded into memory and passed through the processFileData method.

#File: lib/Varien/Simplexml/Config.php
$fileData = file_get_contents($filePath);
$fileData = $this->processFileData($fileData);

The processFileData method implements a listener method for classes which sub-class Varien_Simplexml_Config. As you can see, the base implementation does nothing

#File: lib/Varien/Simplexml/Config.php
public function processFileData($text)
{
    return $text;
}

The classes which inherit from Varien_Simplexml_Config may implement their own processFileData method to manipulate the XML string before loading it. The Mage_Core_Model_Config class doesn’t take advantage of this for anything, but it’s worth noting, as you never know what another developer may have done with a class rewrite.

Finally, we pass the data to the loadString method

#File: lib/Varien/Simplexml/Config.php
return $this->loadString($fileData, $this->_elementClass);

along with the value of $this->_elementClass.

The XML Object

If we take a look at the loadString method

#File: lib/Varien/Simplexml/Config.php
public function loadString($string)
{
    if (is_string($string)) {
        $xml = simplexml_load_string($string, $this->_elementClass);
        if ($xml instanceof Varien_Simplexml_Element) {
            $this->_xml = $xml;
            return true;
        }
    } else {
        Mage::logException(new Exception('"$string" parameter for simplexml_load_string is not a string'));
    }
    return false;
}

we can see that loadString is responsible for creating a SimpleXML object from the passed in string, and assigning that object to the internal $this->_xml property after ensuring its an instance of Varien_Simplexml_Element.

Wait, what? Where does Varien_Simplexml_Element come from? Shouldn’t a simple XML object be an instance of SimpleXMLElement? Turns out there’s a little known/used feature of the simple xml loading functions that allows you to specify a custom class for your SimpleXMLElement objects. This allows you to create custom methods that are available directly on any simple XML objets you create. That’s because the objects are no longer SimpleXMLElements, they’re now objects that are instances of your class.

If you take a look at the definition of Varien_Simplexml_Element, you can see it inherits from SimpleXMLElement, and adds multiple convenience methods for dealing with XML objects.

#File: lib/Varien/Simplexml/Element.php 
class Varien_Simplexml_Element extends SimpleXMLElement
{    
    public function setParent($element)
    {//...}

    public function getParent()
    {//...}


    public function hasChildren()
    {//...}

    public function getAttribute($name){//...}

    public function descend($path)
    {//...}

    public function asArray()
    {//...}

    public function asCanonicalArray()
    {//...}

    protected function _asArray($isCanonical = false)
    {//...}

    public function asNiceXml($filename='', $level=0)
    {//...}

    public function innerXml($level=0)
    {//...}


    public function xmlentities($value = null)
    {//...}

    public function appendChild($source)
    {//...}

    public function extend($source, $overwrite=false)
    {//...}

    public function extendChild($source, $overwrite=false)
    {//...}

    public function setNode($path, $value, $overwrite=true)
    {//...}
}

If you look at the call to 

#File: lib/Varien/Simplexml/Config.php
$xml = simplexml_load_string($string, $this->_elementClass);

you can see we’re specifying the class to use (the second function parameter) with the $this->_elementClass property. You’ll may also remember this property was passed in as a second parameter of the loadString method. 

#File: lib/Varien/Simplexml/Config.php
return $this->loadString($fileData, $this->_elementClass);

It looks like the core team originally intended to allow a programmer to override the base class that’s used via the method call, but failed or declined to implement this feature.

By default, the element class is a Varien_Simplexml_Element. This default is set directly on the object property in the Varien_Simplexml_Config class definition file. 

#File: lib/Varien/Simplexml/Config.php
protected $_elementClass = 'Varien_Simplexml_Element';

However, if we pop-up to the Mage_Core_Model_Config_Base class’s constructor, we can see we’re setting a custom element class for our config object.

#File: app/code/core/Mage/Core/Model/Config/Base.php
public function __construct($sourceData=null)
{
    $this->_elementClass = 'Mage_Core_Model_Config_Element';
    parent::__construct($sourceData);
}

This means any simple xml object created will be a Mage_Core_Model_Config_Element, which has Varien_Simplexml_Element as an ancestor.

Class Hierarchy Review

Before we continue, we’ve just introduced a number of new classes, and it may be useful to review the hierarchy and what they’re used for.

The Mage_Core_Model_Config class, which extends Mage_Core_Model_Config_Base, which in turn extends Varien_Simplexml_Config will be the class for our Magento global configuration object. Although there’s nothing enforcing this in the system, this object should be thought of as a global and/or singleton object. Many different Magento objects may have a reference to this configuration object, but each of these references should be pointing at the same object.

The Mage_Core_Model_Config object has an _xml property. This property will store a PHP SimpleXML object which represents the tree for the Magento global config. However, this SimpleXML object is instantiated with a custom PHP class. This class is Mage_Core_Model_Config_Element, which extends a Varien_Simplexml_Element, which extends the built in SimpleXMLElement.

This hierarchy of classes may seem overly complicated, and while I’d personally agree, a key piece of the core team’s original design goals for Magento was to make the system as flexible and customizable as possible, and to push the limits of PHP 5’s often under-utilized object oriented features. It’s easy see a situation where one member of the core team though “Hey, let’s use the custom class feature of the simple xml objects”, and another member of the core team though “Configuration XML should have wrapper classes which share the same base”.

If you’re this far along you probably don’t need reminding, but it’s often necessary to put aside how you would have implemented an online retail system and accept how Magento’s been built.

Loading the Rest

Since we’re done with loadFile, let’s get back to the loadBase method

#File: app/code/core/Mage/Core/Model/Config.php
public function loadBase()
{
    $etcDir = $this->getOptions()->getEtcDir();
    $files = glob($etcDir.DS.'*.xml');

    $this->loadFile(current($files));
    while ($file = next($files)) {
        $merge = clone $this->_prototype;
        $merge->loadFile($file);
        $this->extend($merge);
    }
    if (in_array($etcDir.DS.'local.xml', $files)) {
        $this->_isLocalConfigLoaded = true;
    }
    return $this;
}

At this point, we’re about the enter the while loop. Our _xml property contain the XML tree for the file at app/etc/config.xml. This is the absolute minimum configuration Magento needs to bootstrap a basic system. The while loop will go through the remaining XML files and load each one. If you’ve ever accidentally copied another XML file into the app/etc folder and had weird things happen, this is the reason for that weirdness. Magento will load ANY file that ends in .xml.

The loading of these files gets a little tricky. Let’s take a look at the code responsible for this. First, Magento clones the object in $this->_prototype.

#File: app/code/core/Mage/Core/Model/Config.php
$merge = clone $this->_prototype;
$merge->loadFile($file);
$this->extend($merge);

If you remember back to our initial constructor, the object in _prototype was set to be an empty instance of Mage_Core_Model_Config_Base

#File: app/code/core/Mage/Core/Model/Config.php
$this->_prototype       = new Mage_Core_Model_Config_Base();

Mage_Core_Model_Config_Base is the same class that Mage_Core_Model_Config has as an ancestor. When Magento clones this object into the variable $merge, it’s making a copy, and then loading the current XML file into the new object

#File: app/code/core/Mage/Core/Model/Config.php
$merge->loadFile($file);

The core team could have just written the code like this

$merge = new Mage_Core_Model_Config_Base;
$merge->loadFile($file);

but as the configuration system was likely one of the first parts of Magento built, it seems like they were still working out what sorts of abstractions they’d be using, which partially explains this sort-of prototypical inheritance pattern for the configuration objects.

That aside, we’re finally at the really tricky part. We now have two XML config objects. There’s our Magento global config object, and this new config object. Each config object contains an individual XML tree. The question we’re now faced with is

How do we combine these two trees into a single object

The new object’s contents are merged with the first tree via the extend method. 

#File: app/code/core/Mage/Core/Model/Config.php
$this->extend($merge);

If you look at the definition of the extend method

#File: lib/Varien/Simplexml/Config.php
public function extend(Varien_Simplexml_Config $config, $overwrite=true)
{
    $this->getNode()->extend($config->getNode(), $overwrite);
    return $this;
}

you can see the ultimate extend method is implemented on the custom simple XML object class, Varien_Simplexml_Element. This is also the first time we’ve encountered the getNode method. When used without a parameter, getNode returns the contents of the $this->_xml property. 

#File: lib/Varien/Simplexml/Element.php    
public function getNode($path=null)
{
    if (!$this->_xml instanceof Varien_Simplexml_Element) {
        return false;
    } elseif ($path === null) {
        return $this->_xml;
    } else {
        return $this->_xml->descend($path);
    }
}

So extend could also be written as

$this->_xml->extends($config->_xml,...);

if _xml wasn’t a protected property

Extending a Varien XML Node

So what does is mean to “extend” or “merge” two XML trees in Magento? Rather than walk through the code, we’re going to describe how extend works. It’s a method that will recursively copy one xml tree into the first, (recursively!). That is, we’re extending the original XML tree with the second.

The basic metaphor is our current XML tree lacks certain nodes, so its asking the new XML tree to provide them.

Consider the following code

$first = new Mage_Core_Model_Config_Base;
$first->loadString('<config>
    <one></one>
    <two></two>
    <three></three>
</config>');

$second = new Mage_Core_Model_Config_Base;
$second->loadString('<config>
    <four></four>
    <five></five>
</config>');

$first->extend($second);

Here we’ve created one XML configuration object, and we’re extending the first with the second. If, after running the above, we called

echo $first->getNode()->asNiceXml()

we’d get output something like this

<config>
   <one/>
   <two/>
   <three/>
   <four/>
   <five/>
</config>

The extend method has merged the two trees. Now consider something like this

$first = new Mage_Core_Model_Config_Base;
$first->loadString('<config>
    <one></one>
    <two></two>
    <three></three>
</config>');

$second = new Mage_Core_Model_Config_Base;
$second->loadString('<config>
    <one>Hello</one>
    <two>Goodby</two>
</config>');

$first->extend($second);

Here our second XML tree has nodes that conflict with the first. When merging trees like this, the second tree wins. Checking our output again

echo $first->getNode()->asNiceXml();

will show the merged tree

<config>
   <three/>
   <one>Hello</one>
   <two>Goodby</two>
</config>

However, remember that this is a recursive merge. Trees like this

$first = new Mage_Core_Model_Config_Base;
$first->loadString('<config>
    <one>
        <two>
            <three></three>
        </two>
    </one>
</config>');

$second = new Mage_Core_Model_Config_Base;
$second->loadString('<config>
    <one>
        <two>
            <four></four>
        </two>
    </one>
</config>');

$first->extend($second);

will not replace the contents of the <two> node. Instead they’ll be merged together, like this

<config>
   <one>
      <two>
         <three/>
         <four/>
      </two>
   </one>
</config>

It’s only when there’s a depth match that one node will completely replace another. Code like this

$first = new Mage_Core_Model_Config_Base;
$first->loadString('<config>
    <one>
        <two>
            <three>Original Value</three>
        </two>
    </one>
</config>');

$second = new Mage_Core_Model_Config_Base;
$second->loadString('<config>
    <one>
        <two>
            <four></four>
            <three>New Value</three>
        </two>
    </one>
</config>');

$first->extend($second);

would give us a final tree of like this, replacing the original tree’s nodes

<config>
   <one>
      <two>
         <four/>
         <three>New Value</three>
      </two>
   </one>
</config>

That said, this “second tree wins” behavior may be avoided by using the extend method’s second $overwrite parameter

$first->extend($second, false);

When you pass this parameter in as false, you’re instructing Magento NOT to overwrite nodes in the first tree. Only new nodes from the second tree will be added, conflicts will be ignored. This means the values in the second tree are preserved.

The code that does this is worth exploring, especially if you’re interested in recursion. However, it’s beyond the scope of this series to explore that deeply.

One last note before we move on. You’ll notice that, in the extend method mentioned above on the base config class, the $overwrite property defaults to true. However, this call wraps to another extend method, defined on our custom simple xml class

#File: lib/Varien/Simplexml/Element.php
public function extend($source, $overwrite=false)
{
    if (!$source instanceof Varien_Simplexml_Element) {
        return $this;
    }

    foreach ($source->children() as $child) {
        $this->extendChild($child, $overwrite);
    }

    return $this;
}

which has a prototype that defines the default as false. So remember, if you’re working with the config objects, extend will default to overwriting nodes but if you’re working with the raw Varien_Simplexml_Element nodes, the default behavior is to skip merging nodes that conflict. If you’re going to be doing a lot of work directly with both these objects, it’s probably best to be explicit with your override parameter and not rely on the defaults.

Back to the Base

Jumping back up to our loadBase method one last time

#File: app/code/core/Mage/Core/Model/Config.php
public function loadBase()
{
    $etcDir = $this->getOptions()->getEtcDir();
    $files = glob($etcDir.DS.'*.xml');

    $this->loadFile(current($files));
    while ($file = next($files)) {
        $merge = clone $this->_prototype;
        $merge->loadFile($file);
        $this->extend($merge);
    }
    if (in_array($etcDir.DS.'local.xml', $files)) {
        $this->_isLocalConfigLoaded = true;
    }
    return $this;
}

After we’re done merging all the discovered XML files into a single base object, Magento checks if one of the files we just loaded was app/etc/local.xml

if (in_array($etcDir.DS.'local.xml', $files)) {
    $this->_isLocalConfigLoaded = true;
}

If so, it marks a property flag to note this. 

#File: app/code/core/Mage/Core/Model/Config.php
$this->_isLocalConfigLoaded = true;

The local.xml file is created when you go through the installation steps after running Magento for the first time. By marking this flag as true, the core code has given Magento client developers the ability to programmatically determine if this is an “installed” system, or if it’s a system that still needs its local.xml file created.

At this point, we’ve now loaded our base Magento configuration.

Wrap Up

So that’s the first part in our series on the Magento global config. So far we’ve covered loading the base configuration tree, as well as the general mechanism Magento uses to merge multiple XML configuration trees. Although we’re several thousand words in, we’ve only scratched the surface. Next time we’ll be covering the where and how of Magento’s module configuration files, and what effect this loading has on our global configuration object.

Originally published January 16, 2012