Memcached integration for performance improvement

 

 

One of the most required tasks by our customers is related to the performance of their site. For those clients that require a lower TTFB (Time To First Byte) as a priority instead of progressive enhancements or other development strategy to load their sites, then read on. Aside from helping to make improvements to the Search Rank of a Website, it also helps to increase sales and generate a better user experience for all their customers.

E-Commerce platforms like Magento include a decent caching system that improves the loading time of static content, however you can’t avoid that sometimes customers access uncached pages using urls with dynamic query parameters or when cache has been cleared to show an update in the content. In those cases it is necessary to provide an improved site load speed and reduce server usage because uncached pages generate a lot of database requests to display content.  This is specially more evident on high traffic sites or after executing large marketing campaigns (email, television, radio, etc.) that can overwhelm your site’s current capacity.

Memcached is one of several excellent options for those non-cached pages, helping to reduce database requests and allowing us to store large amounts of data in memory that can be displayed or used quickly. This means that we can store complex data in a string representation; i.e. Memcached lets us save html content that has been generated previously with database requests and data processing.

In this article I will explain how to use Memcached with a Magento 2 site as an alternative to store and load information for product pages and improve a site’s loading time.

First we need to install and configure Memcached on our server.  Magento 2 can also be integrated with Memcached for sessions, however we will use it for a different purpose in this case.  Magento provides documentation for Memcached installation, so you can use this as a guide to install and verify that Memcache has been configured correctly on your server:

https://devdocs.magento.com/guides/v2.3/config-guide/memcache/memcache_ubuntu.html .

After verifying that memcached is working correctly we need to create a custom module.  This will let us extend the core functionality of Magento instead of modifying core files (currently modifying core files is frowned upon given it tends to endanger the breaking of current functionality).

We need to start by creating the base of the extension to be detected by Magento

app/code/Qxd/Memcached/registration.php

 

<?php

\Magento\Framework\Component\ComponentRegistrar::register(

   \Magento\Framework\Component\ComponentRegistrar::MODULE,

   ‘Qxd_Memcached’,

   __DIR__

);

 

app/code/Qxd/Memcached/etc/module.xml

 

<?xml version=”1.0″?>

<config xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:noNamespaceSchemaLocation=”urn:magento:framework:Module/etc/module.xsd”>

   <module name=”Qxd_Memcached” setup_version=”1.0.0″ active=”true”>

   </module>

</config>

 

We can also create the acl.xml configuration to allow only specific roles to edit the configuration of Memcached.

 

app/code/Qxd/Memcached/etc/acl.xml

 

<?xml version=”1.0″?>

<config xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:noNamespaceSchemaLocation=”../../../../../lib/internal/Magento/Framework/Acl/etc/acl.xsd”>

   <acl>

       <resources>

           <resource id=”Magento_Backend::admin”>

               <resource id=”Magento_Backend::stores”>

                   <resource id=”Magento_Backend::stores_settings”>

                       <resource id=”Magento_Config::config”>

                           <resource id=”Qxd_Section::qxd_section” title=”Qxd Section”>

                               <resource id=”Qxd_Memcached::config” title=”Memcached Integration”/>

                           </resource>

                       </resource>

                   </resource>

               </resource>

           </resource>

       </resources>

   </acl>

</config>

 

We need to create a section in the admin panel to configure the connection with Memcached. If your site is already using Memcached for sessions is a better idea to use a different server to store the content cached in case you need to clear all the information saved in the Memcached instance.

app/code/Qxd/Memcached/etc/adminhtml/system.xml

<?xml version=”1.0″?>

 

   <config xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:noNamespaceSchemaLocation=”urn:magento:module:Magento_Config:etc/system_file.xsd”>

       <system>

           <tab id=”qxd” translate=”label” sortOrder=”50″>

               <label>QXD</label>

           </tab>

           <section id=”memcached_options” translate=”label” sortOrder=”1″ showInDefault=”1″ showInWebsite=”1″ showInStore=”1″>

               <label>Memcached Integration</label>

               <tab>qxd</tab>

               <resource>Qxd_Memcached::config</resource>

               <group id=”configure_memcached” translate=”label” type=”text” sortOrder=”1″ showInDefault=”1″ showInWebsite=”1″ showInStore=”1″>

                   <label>General Configuration</label>

                   

                   <field id=”memcached_host” translate=”label” type=”text” sortOrder=”1″ showInDefault=”1″ showInWebsite=”1″ showInStore=”0″>

                       <label>Host</label>

                       <validate>required-entry</validate>

                   </field>

 

                   <field id=”memcached_port” translate=”label” type=”text” sortOrder=”2″ showInDefault=”1″ showInWebsite=”1″ showInStore=”0″>

                       <label>Port</label>

                       <validate>required-entry</validate>

                   </field>

               </group>

           </section>

       </system>

   </config>

 

After upgrading and compiling the code the custom module will generate the next view in the admin panel:

Memcached_pages-to-jpg-0004

 

The Host and the Port will let us connect to a remote server or even to our local server and store and load the information without problems.

 

Now we need to provide a connector to access memcached and be able to store and load the information. We will create a Singleton to avoid creating extra instances for each connection and reuse the same instance to access data.

 

app/code/Qxd/Memcached/Model/Connector.php 

<?php

namespace Qxd\Memcached\Model;

class Connector extends \Magento\Framework\Model\AbstractModel

{

   private static $instance;

   private static $scopeConfig;

   public $memcachedConn;

 

   public function __construct(

       \Magento\Framework\App\Config\ScopeConfigInterface $scopeConfig

   ) {

       self::$scopeConfig = $scopeConfig;

   }

 

   /**

    *

    * @return Connector – Singleton

    */

   public static function getInstance(){

 

       if (self::$instance == null){

           $className = __CLASS__;

           self::$instance = new $className(self::$scopeConfig);

       }

 

       return self::$instance;

   }

 

   /**

    *

    * @return \Memcached

    */

   private static function initConnection(){

       $mem = self::getInstance();

       $mem->memcachedConn = new \Memcached;

       $host = self::$scopeConfig->getValue(

           ‘memcached_options/configure_memcached/memcached_host’

       );

       $port = self::$scopeConfig->getValue(

           ‘memcached_options/configure_memcached/memcached_port’

       );

 

       $mem->memcachedConn->addServer($host, $port);

       return $mem;

   }

 

   /**

    * @return Memcached connection

    */

   public static function getMemcachedConn() {

       try {

           $memcached = self::initConnection();

           return $memcached->memcachedConn;

       } catch (Exception $e) {

           $writer = new \Zend\Log\Writer\Stream(BP . ‘/var/log/QXD_Memcached_Error.log’);

           $logger = new \Zend\Log\Logger();

           $logger->addWriter($writer);

           $logger->info(‘observer before save product’);

           return null;

       }

   }

}

 

$host = self::$scopeConfig->getValue(

           ‘memcached_options/configure_memcached/memcached_host’

       );


$port = self::$scopeConfig->getValue(

           ‘memcached_options/configure_memcached/memcached_port’

       );

 

We will used the information stored on admin panel to establish the connection with Memcached

Additionally we will use a Helper to allow the access of the Singleton from every block,controller or template as we want.

app/code/Qxd/Memcached/Helper/Data.php
<?php

namespace Qxd\Memcached\Helper;
class Data extends \Magento\Framework\App\Helper\AbstractHelper

{   

   public function __construct(

       \Qxd\Memcached\Model\Connector $connector

   ) {

       $this->_connector = $connector;

   }
   /*

   * Start memcached connection

   */
   public function initMemcached()

   {

       $_memcached = $this->_connector->getInstance();

       $memcachedConnector=null;

       if($_memcached){ $memcachedConnector=$_memcached->getMemcachedConn(); }
       return $memcachedConnector;

   }

}

 

Now as an example we can extend the functionality used to load the Json configuration used for price calculation, for this purpose we need to override the block

vendor/magento/module-catalog/Block/Product/View.php

For this purpose we need to create the /app/code/Qxd/Memcached/etc/di.xml file and update it with the preference information

 

<?xml version=”1.0″ ?><config xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:noNamespaceSchemaLocation=”urn:magento:framework:ObjectManager/etc/config.xsd”>

   <preference for=”Magento\Catalog\Block\Product\View” type=”Qxd\Memcached\Block\Product\View”/>

</config>

After that we need to make a copy of the block file and place it in the following path app/code/Qxd/Memcached/Block/Product/View.php

The most important changes for this file are:

  1. The namespace: We update it to use the path of our custom module.

     namespace Qxd\Memcached\Block\Product;
  2. The class definition: We adjust it to extend from the class that we are overwriting.

     class View extends \Magento\Catalog\Block\Product\View
  3. The constructor: We include the Helper used to create the connection with Memcached

    public function __construct(

           \Magento\Catalog\Block\Product\Context $context,

           \Magento\Framework\Url\EncoderInterface $urlEncoder,

           \Magento\Framework\Json\EncoderInterface $jsonEncoder,

           \Magento\Framework\Stdlib\StringUtils $string,

           \Magento\Catalog\Helper\Product $productHelper,

           \Magento\Catalog\Model\ProductTypes\ConfigInterface $productTypeConfig,

           \Magento\Framework\Locale\FormatInterface $localeFormat,

           \Magento\Customer\Model\Session $customerSession,

           ProductRepositoryInterface $productRepository,

           \Magento\Framework\Pricing\PriceCurrencyInterface $priceCurrency,

           \Qxd\Memcached\Helper\Data $memcachedHelper,

           array $data = []

       ) {

           $this->_productHelper = $productHelper;

           $this->urlEncoder = $urlEncoder;

           $this->_jsonEncoder = $jsonEncoder;

           $this->productTypeConfig = $productTypeConfig;

           $this->string = $string;

           $this->_localeFormat = $localeFormat;

           $this->customerSession = $customerSession;

           $this->productRepository = $productRepository;

           $this->priceCurrency = $priceCurrency;

           $this->_memcachedHelper = $memcachedHelper;
           parent::__construct($context, $urlEncoder, $jsonEncoder, $string, $productHelper, $productTypeConfig, $localeFormat, $customerSession, $productRepository, $priceCurrency, $data);

       }

 

    4. The method getJsonConfig: We extend it adding the required changes to store the information in memcached and loading it from there instead from database if there is information saved for the current product

public function getJsonConfig()

   {

       $writer = new \Zend\Log\Writer\Stream(BP . ‘/var/log/test.log’);

       $logger = new \Zend\Log\Logger();

       $logger->addWriter($writer);

       /* @var $product \Magento\Catalog\Model\Product */

       $product = $this->getProduct();

       $memcached=$this->_memcachedHelper->initMemcached();
       if(!$memcached || !$memcached->get(‘jsonConfig_’.$product->getId()))

       {

           if (!$this->hasOptions()) {
               $config = [

                   ‘productId’ => $product->getId(),

                   ‘priceFormat’ => $this->_localeFormat->getPriceFormat()

               ];
               if($memcached)

               {

                   $logger->info(‘NOT CACHED’);

                   $memcached->set(‘jsonConfig_’.$product->getId(),$this->_jsonEncoder->encode($config));

               }

               return $this->_jsonEncoder->encode($config);

           }
           $tierPrices = [];

           $tierPricesList = $product->getPriceInfo()->getPrice(‘tier_price’)->getTierPriceList();

           foreach ($tierPricesList as $tierPrice) {

               $tierPrices[] = $tierPrice[‘price’]->getValue();

           }

           $config = [

               ‘productId’ => $product->getId(),

               ‘priceFormat’ => $this->_localeFormat->getPriceFormat(),

               ‘prices’ => [

                   ‘oldPrice’ => [

                       ‘amount’ => $product->getPriceInfo()->getPrice(‘regular_price’)->getAmount()->getValue(),

                       ‘adjustments’ => []

                   ],

                   ‘basePrice’ => [

                       ‘amount’ => $product->getPriceInfo()->getPrice(‘final_price’)->getAmount()->getBaseAmount(),

                       ‘adjustments’ => []

                   ],

                   ‘finalPrice’ => [

                       ‘amount’ => $product->getPriceInfo()->getPrice(‘final_price’)->getAmount()->getValue(),

                       ‘adjustments’ => []

                   ]

               ],

               ‘idSuffix’ => ‘_clone’,

               ‘tierPrices’ => $tierPrices

           ];
           $responseObject = new \Magento\Framework\DataObject();

           $this->_eventManager->dispatch(‘catalog_product_view_config’, [‘response_object’ => $responseObject]);

           if (is_array($responseObject->getAdditionalOptions())) {

               foreach ($responseObject->getAdditionalOptions() as $option => $value) {

                   $config[$option] = $value;

               }

           }
           if($memcached)

           {

               $memcached->set(‘jsonConfig_’.$product->getId(),$this->_jsonEncoder->encode($config));

           }
           return $this->_jsonEncoder->encode($config);

       }else{

           $logger->info(‘CACHED’);

           return $memcached->get(‘jsonConfig_’.$product->getId());

       }

   }
 if($memcached)
Is important to validate that the connection with Memcached is working to avoid server errors breaking the functionality.
$writer = new \Zend\Log\Writer\Stream(BP . ‘/var/log/test.log’);

$logger = new \Zend\Log\Logger();

$logger->addWriter($writer);

$logger->info(‘CACHED’);

$logger->info(‘NOT CACHED’);

 

Is important to validate that the connection with Memcached is working to avoid server errors breaking the functionality.

 $writer = new \Zend\Log\Writer\Stream(BP . ‘/var/log/test.log’); $logger = new \Zend\Log\Logger();

$logger->addWriter($writer);

$logger->info(‘CACHED’);

$logger->info(‘NOT CACHED’);

 

I added some logs to verify that content is loaded from memcached instead from database. The information recorded in the log shows the message: 2019-02-22T03:09:30+00:00 INFO (6): NOT CACHED the first time that the product page is accessed, the next time the message recorded in the log is: 2019-02-22T03:09:38+00:00 INFO (6): CACHED meaning that information has been loading from Memcached directly.

Now we need a way to clear the content stored in Memcached, for this reason we will create an observer to allow the admin user to clear the content and show an updated version by saving the product from admin panel.

We will create the file app/code/Qxd/Memcached/etc/events.xml to define the event that we want to track:

 <?xml version=”1.0″?><config xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:noNamespaceSchemaLocation=”urn:magento:framework:Event/etc/events.xsd”>
  <event name=”controller_action_catalog_product_save_entity_after”>

      <observer name=”qxd_memcached_product” instance=”Qxd\Memcached\Observer\Productsave” />

  </event>
</config>

After that we will create the observer with the required code to be executed after saving a product from admin panel:

app/code/Qxd/Memcached/Observer/Productsave.php
<?php

namespace Qxd\Memcached\Observer;
use Magento\Framework\Event\Observer;

use Magento\Framework\Event\ObserverInterface;
class Productsave implements ObserverInterface

{

public function __construct(

       \Qxd\Memcached\Helper\Data $helper
   ) {

       $this->_helper = $helper;

   }

   /*

   * Clear memcached key related with the product

   */
   public function execute(Observer $observer)

   {

       

       $product = $observer->getProduct();

       $productId = $product->getId();
       $_memcached = $this->_helper->initMemcached();

 

       if($_memcached){

           $_memcachedKeys=$_memcached->getMulti(array(‘jsonConfig_’.$productId,’reviewSummary_’.$productId));
           if(isset($_memcachedKeys) && !empty($_memcachedKeys))

           {

               foreach($_memcachedKeys as $k=>$data) { if($data){ $_memcached->delete($k); } }

           }

       }

   }

}
$_memcachedKeys=$_memcached->getMulti(array(‘jsonConfig_’.$productId,’reviewSummary_’.$productId));

The getMulti method will let us get multiple keys from memcached if they exist, after that we will use the method delete to remove the key and the content from Memcached.

After saving a product we can access again the product page and will find in the log that content was loaded from database.

As we could see we can improve the site speed in catalog pages using this open source tool to get most of the product page content and not only save customer’s sessions. However its functionality also can be extended to store content loaded for different pages and even information used in the checkout process like shipping rules, for example.