Fast, Low Memory Drupal 6 System Module

A Drupal 5 version of this module is also available.  If you would like this patch to be committed to Drupal core, please do not leave a comment on this page—please instead add your comment to Drupal issue #455092.

This is a drop-in replacement for the system.module of Drupal 6.37 which makes your Drupal 6 site use less memory and may even make it faster. A test I ran in a development environment with a stock Drupal 6 installation suggested that I got:

  • 2—4.4% reduction in average page load time
  • 3% less memory

In a single test with much higher load on a production server, this patch gave a much higher boost to performance, somewhere between 18—40% reduction in average page load time.

If this sounds interesting, please feel free to install it on your Drupal 6 site. I'd be interested to hear feedback as to whether this makes any difference if you have an opcode cache installed. If it works for you, you might be interested in downloading my other low memory modules as well.

Performance and Memory Benchmark (6.10 version)

This version of the module performs between 2—4.4% faster than the stock module in my single test:

Benchmark run on stock Fedora 10 with Apache 2.2.10 (httpd-2.2.10-2.i386) and MySQL 5.0.37 (mysql-server-5.0.67-2.fc10.i386), using ab -n 100 http://hostname/user on a default install of Drupal 6.10.

The stock system.module (with -c 1):

Requests per second:    4.89 [#/sec] (mean)
Time per request: 204.443 [ms] (mean)
Time per request: 204.443 [ms] (mean, across all concurrent requests)
Transfer rate: 20.97 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 198 204 8.0 202 238
Waiting: 185 196 8.3 194 230
Total: 198 204 8.0 202 238

My low memory version (with -c 1):

Requests per second:    4.99 [#/sec] (mean)
Time per request: 200.267 [ms] (mean)
Time per request: 200.267 [ms] (mean, across all concurrent requests)
Transfer rate: 21.41 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 0 0.0 0 0
Processing: 194 200 7.6 197 229
Waiting: 182 191 8.0 189 222
Total: 194 200 7.6 197 229

The stock system.module (with -c 10):

Requests per second:    4.86 [#/sec] (mean)
Time per request: 2056.485 [ms] (mean)
Time per request: 205.649 [ms] (mean, across all concurrent requests)
Transfer rate: 20.85 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 9 30.5 0 156
Processing: 1455 2022 149.7 2049 2331
Waiting: 1452 1988 147.3 2007 2300
Total: 1455 2031 150.9 2057 2331

My low memory version (with -c 10):

Requests per second:    5.09 [#/sec] (mean)
Time per request: 1965.231 [ms] (mean)
Time per request: 196.523 [ms] (mean, across all concurrent requests)
Transfer rate: 21.81 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 6 21.3 0 103
Processing: 1334 1933 146.0 1967 2191
Waiting: 1333 1901 142.9 1923 2190
Total: 1334 1939 149.8 1975 2191

Replacing the stock Drupal system.module with this module saved 214K on a stock Drupal install or a little bit over 3% of overall site memory.

Caveat Emptor

  1. As this code is based on Drupal, which is licensed under the GPL, this code is also licensed under the GPL, which means (in the word of said license):
    This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Installation (whole module version)

To install, simply download the archive, unpack it, and put the files in your modules/system directory. The system.module from the archive will replace the stock drupal system.module.

If you're on a command line, you can do so by switching into the drupal installation directory (running ls should show index.php and cron.php), and running:

wget -O - http://wtanaka.com/system/files/system-6.37.tar.bz2 | bzcat | tar xvf -

Installation (patch version)

From the drupal installation directory (running ls should show index.php and cron.php), run:

wget -O - http://wtanaka.com/system/files/system-6.37.patch.bz2 | bzcat | patch -p0

Uninstall

From the drupal installation directory (running ls should show index.php and cron.php), run:

ln -s . drupal-6.37
wget -O - http://ftp.drupal.org/files/projects/drupal-6.37.tar.gz|gzip -dc|tar xvf - drupal-6.37/modules/system/system.module
rm -f drupal-6.37
rm -f modules/system/*.inc.php
AttachmentSize
Binary Data system-6.10.patch.bz220.58 KB
Binary Data system-6.10.tar.bz216.19 KB
Binary Data system-6.11.patch.bz220.49 KB
Binary Data system-6.11.tar.bz216.22 KB
Binary Data system-6.12.patch.bz220.49 KB
Binary Data system-6.12.tar.bz216.22 KB
Binary Data system-6.13.patch.bz220.49 KB
Binary Data system-6.13.tar.bz216.23 KB
Binary Data system-6.14.patch.bz222.7 KB
Binary Data system-6.14.tar.bz216.55 KB
Binary Data system-6.15.patch.bz221.38 KB
Binary Data system-6.15.tar.bz216.57 KB
Binary Data system-6.16.patch.bz220.91 KB
Binary Data system-6.16.tar.bz216.56 KB
Binary Data system-6.17.patch.bz221.19 KB
Binary Data system-6.17.tar.bz216.76 KB
Binary Data system-6.18.patch.bz221.23 KB
Binary Data system-6.18.tar.bz216.78 KB
Binary Data system-6.19.patch.bz221.2 KB
Binary Data system-6.19.tar.bz216.78 KB
Binary Data system-6.20.patch.bz221.23 KB
Binary Data system-6.20.tar.bz216.78 KB
Binary Data system-6.22.patch.bz221.3 KB
Binary Data system-6.22.tar.bz216.79 KB
Binary Data system-6.23.patch.bz221.28 KB
Binary Data system-6.23.tar.bz216.79 KB
Binary Data system-6.24.patch.bz221.75 KB
Binary Data system-6.24.tar.bz217.11 KB
Binary Data system-6.25.patch.bz221.83 KB
Binary Data system-6.25.tar.bz217.21 KB
Binary Data system-6.26.patch.bz221.82 KB
Binary Data system-6.26.tar.bz217.19 KB
Binary Data system-6.27.patch.bz221.84 KB
Binary Data system-6.27.tar.bz217.2 KB
Binary Data system-6.28.patch.bz221.78 KB
Binary Data system-6.28.tar.bz217.14 KB
Binary Data system-6.29.patch.bz221.76 KB
Binary Data system-6.29.tar.bz217.13 KB
Binary Data system-6.30.patch.bz221.77 KB
Binary Data system-6.30.tar.bz217.15 KB
Binary Data system-6.31.patch.bz221.77 KB
Binary Data system-6.31.tar.bz217.13 KB
Binary Data system-6.32.patch.bz221.78 KB
Binary Data system-6.32.tar.bz217.14 KB
Binary Data system-6.33.patch.bz221.78 KB
Binary Data system-6.33.tar.bz217.13 KB
Binary Data system-6.34.patch.bz221.79 KB
Binary Data system-6.34.tar.bz217.13 KB
Binary Data system-6.35.patch.bz221.79 KB
Binary Data system-6.35.tar.bz217.13 KB
Binary Data system-6.36.patch.bz221.78 KB
Binary Data system-6.36.tar.bz217.12 KB
Binary Data system-6.37.patch.bz221.79 KB
Binary Data system-6.37.tar.bz217.14 KB

Comments

It would be interested to see benchmarks with a heavily module loaded D6 site. I notice with the naked eye D6 out of the box is very fast (logged in user), but after getting the site close to launch, it is much much slower for logged in users. I normally use some 10 - 20 modules on average.

Thanks for your work and keep the Drupal Community posted on future results - we all want faster sites. ;)

Testing this on a 6.10 site which currently has 87 modules installed, including the core modules. This is largely down to the fact that ubercart is installed.

The site is running on about 512Mb of memory but occasionally reports a fatal error due to running out, so we'll see if this low memery version makes any difference.

Is there a chance you could roll your performance improvements as a patch against Drupal core, so Drupal 7 will get that boost by default?

If I find the time to, yes definitely!  I'm not even sure that this change will be relevant in Drupal 7 though!

Hello there!!!
I wanna lower the usage of memory.
Can this module be useful? how?

Regards/

It's as secure as the stock Drupal core module as far as I know.  If your site uses a lot of modules already, then the difference that this one makes will be a smaller percentage of your overall memory usage.

Thank you so much / now i can use my drupal system within a small shared hosting account.

4.4% reduction in load time is something excellent. Some how I have installed. Thank u.

Instead of advising people to the awful practice of patching core, why don't you contribute your findings and possible fix? I'm also wondering what your test environment is Drupal-wise and why you base all this on just a single test. I'm all for a faster Drupal but I think your effort might better be put towards Drupal 7. For instance by looking into the registry: http://drupal.org/node/345118

Again: please don't advising people to break their core Drupal installation.

I have contributed these findings and patches on Drupal issue #455092.  I would greatly appreciate it if you would help advocate for inclusion on that bug trail!  I have not had time to look into Drupal 7 yet.  The development test environment is described above (stock RPMs on Fedora).  I believe the production server was stock RPMs on CentOS 5 on a VPS, so I was not able to run a test there that I'd have enough confidence in to publish.  I haven't tested more extensively for two reasons.  First, I don't have access to a wide variety of high end hardware.  Second, even if I did, my first priority is, I hope you understand, to test on the configuration that I am personally using.

Incidentally, for maintaining patches on top of "vendor branches", I can recommend using Git for source control.  It makes merging a custom patch set with updated vendor branches pretty easy.  Forking is no longer necessarily a thing to be avoided when merging is easy to do.  The big headache would occur if you're trying to maintain this lower memory system module in addition to another set of patches to the system.module file.  I'm not sure what to recommend in this case, perhaps the only thing that we can do is to hope that these changes get included into Drupal core.  Again, if you can help advocate for that on Drupal issue #455092, I'd appreciate it greatly!

As far as trying to do this without touching core, unfortunately the strategy being used here is fundamentally tied in to modifying core files.  If you have any ideas for how to do the same thing without touching core, I would be grateful for your input, as it would reduce my effort to maintain this each time a new Drupal core update comes out.

I don't advise people to break their core Drupal installation. I know from experience how time consuming and fragile it is to maintain patches on top of vendor branches.  But I figure that publishing these changes is better than not publishing them, given that I'm maintaining them anyway.  My hope is to get more eyeballs and more testing so that I can have more confidence in this code, and of course in the end people can choose to use this or not as they please, right?

I'm so glad that you already created an issue for this in May. What you're doing here is actually quite similar to what the menu module introduced in Drupal 6 to decrease overhead and increase performance: Page handler include files. The best way to make these changes public is to put them in a module, especially since you're actively maintaining them, but that is indeed hard since your changes must happen in the core system.

>The best way to make these changes public is to put them in a module, especially since you're actively maintaining them, but that is indeed hard since your changes must happen in the core system.

That statement is not accurate. If I am wrong, please point to an issue.

You can override core Drupal classes in your own module. Panels and cTools do this I believe, as do many many other modules.

It does require a little bit more structure to override things, vs. making a self-contained module that does not affect core any, but progress is progress and you want to give people an easy way to back this module.

I also have to agree - don't ever hack core for an installation. None of these users who do so will be able to apply future security patches, putting their systems at risk (and other servers in the ecosystem at risk, since a compromised server nearly always ends up attacking someone else).

If you get this into a module, it is MORE likely to be noticed, as there's a ton of version specific Drupal patches in the issue queue.

Can't you have people put this in sites/all/modules/system? You'd have to copy the modules/system directory to sites/all/modules and then add your drop-in replacement, but it would be better than "breaking core".

Keep up the interesting work. We need to find as many ways as possible to speed up Drupal.

This would be the same as hacking core I'm afraid... If an update to core was released in which for example a new function is added to one of the files in /modules/system then and the function was called from elsewhere in the codebase then the site would break.  Not hacking core means alloweing changes made to all parts of core be reflected in the "loaded" code base.

I applied this patch to 6.13... and all was well and I believe faster.

I updated core to 6.14 and expected to apply the new version of this patch - when I did i was told that it was already applied.... but I've just replaced all the patched code with the latest unpatched core.

What is the correct procedure on updates?

You can either follow the uninstall instructions, and then apply the 6.14 patch, or you can install the "whole module" version which will replace all of the necessary files with the patched 6.14 versions.

Hi,

thank you very much for your effort. Drupal is such a memory hog and its good people like you care about it.

However there is one thing I do not understand:
I am using APC and a bunch of modules. So I can safely assume that most of the system function will get called sooner or later and their include files will end up being cached too.

In such a constillation, couln't it be that it is actually more efficient to have just one file? As far as I understand, my php process would be still using more memory, but due to the reduced number of functions and include_once routines on something already cached I might end up with higher loading times?

I don't know the internals of how APC works, and I'm not really all that familiar with file system caching on modern operating systems either, actually, so I don't have a guess.  My suggestion is to try it out if you think it might make a difference.  If you can also report your results here and/or on the drupal issue trail, that would be useful too, since other people seem to have that same question.

Thanks for adding this patch to your list. I will also try installing Drupal onto my current Turnkey LAMP node. What is the difference is between doing my own install on my Turnkey LAMP node vs. using the prepackaged Drupal appliance. Is there a list of what changes were made to LAMP to turn it into the Drupal appliance? When updating my image, is it recommended to get Drupal updates from Debian packages instead of the Drupal website? I don't want to break your security update process. Cheers, -Dan

We've been happily using this modification for sometime now - it has become part of my upgrade routine....

Is it going to be available for 6.20?

Many thanks for the great work!

Thanks for the feedback.  I just uploaded a version for 6.20.

which directory im must repalace 6.x.patch file??

Hi, I've done some tests using the 6.22 patch version on a medium drupal site (about 10.000 nodes).

I want to share a small summary of my results:

  • ab -n 100 -c 10  http://localhost/
    • cache enabled: with patch I've had a loss of 3%of response time
    • cache disabled: with patch I've had a gain of 2% of response time
  • ab -n 1000 -c 50  http://localhost/
    • cache enabled: with patch I've had a gain of 1% of response time
    • cache disabled: with patch I've had a gain of 8% of response time
  • ab -n 1000 -c 100  http://localhost/
    • cache enabled: with patch I've had a gain of 0,2% of response time
    • cache disabled: the server was not able to complete the test

I've repeated many times the tests rebooting the webserver, cleaning Drupal cache and removing tail and head results in order to have the better statistical results.

The patch works fine when Drupal cache is disabled, instead with Drupal cache enabled the patch shows only small improvements with -c 50 and -c 100. I suggest everyone interested on using the patch to try it and apply only if really needed.

Anyway I like it :-)

 

 

Subscribe to All Posts - Wesley Tanaka