OpenX Tips

So, business is booming, and you are approaching your OpenX installation’s maximum capacity — you know this, because you measured your system capacity. You know that it’s time to scale your OpenX installation.

Or, perhaps you have no idea what you system’s maximum capacity is, but you have noticed that your system is a little bit slower than it used to be, and you think that scaling your OpenX installation might be a good idea¹.

How do you go about scaling an existing OpenX installation?

There are essentially two different options for doing this simply.

Vertical scaling

Vertical scaling is the easiest way of scaling your OpenX installation — simply upgrade your existing server (perhaps by adding more RAM or changing the hard drives so that they have faster I/O) or move your OpenX installation onto a larger, faster server.

Certainly, this will not be a hassle-free process — performing hardware upgrades or migrating services from one server to another are never going to The Guru’s idea of a fun day’s work — but if you currently run OpenX on a single server, then it can be comforting to know that, after you have completed a vertical upgrade, you will still have a familiar, single server setup.

Unfortunately, the main drawback with vertical scaling is that sooner or later, depending on your budget, it will simply become too expensive to perform a vertical upgrade².

Simple horizontal scaling

Simple horizontal scaling as an approach where OpenX is scaled horizontally (that is, over multiple servers) at the delivery/UI server level, but where there is still only one database.

Using this approach, there are only four things that are different from a single-server installation of OpenX that you need to worry about:

1. Database access: In a single server installation, ensuring that OpenX can access the database is generally trivial, as all that is needed is a username/password. However, now that your delivery/UI servers are separate from the server that is running the database, you may need to update your database server’s accounts and/or permissions, to ensure that the OpenX code on the remote delivery/UI servers can connect to the database.
2. Load balancing: To distribute the task of banner delivery and OpenX UI access across all of your delivery/UI servers, you will need some form of load balancing solution.
3. Code synchronization: As OpenX is now installed on more than one server, you need to ensure that all servers remain synchronized with each other, so that when you make configuration changes, or perform an upgrade, you don’t end up with one (or more) of the servers in a different state from the others. This is relatively easy to achieve by using rsync, or a similar code synchronization system³.
4. Running maintenance: When OpenX is installed on more than one server, but you have a single, central database, it is important to ensure that only one server runs maintenance⁴. Pick one of your delivery/UI servers, and set up maintenance to run on just that server. Disable maintenance from running on all the other servers. It’s even more important to run maintenance manually in this kind of setup, otherwise the issues of not doing so will be increased, as a result of having more delivery/UI servers!

Of course, with this form of simple horizontal scaling, the central database server can become a bottleneck. If this happens, you can either scale your database server vertically, or you will need to consider using OpenX’s “distributed statistics” mode for horizontal scaling, which allows the database load to be (partially) distributed across multiple servers — however, scaling OpenX in this was is most certainly not a simple means of scaling, and is a topic for another day…

1. Assuming that you have at least tried basic performance tuning first! [↩]
2. Let’s face it, there are not many OpenX users that can afford to go and drop $50k or even more on a top-of-the-line server. [↩]
3. It is often easier to manage this process if you distribute banner delivery across all of the servers, while limiting UI access to one server, as this means there is only one UI server where changes can be made, and upgrades can be run. However, this of course also limits the level scaling you can apply to your OpenX UI service. [↩]
4. If all of your servers ran maintenance, and they all ran maintenance at the same time, this should not be an issue, as OpenX does have code in place to try and ensure that if you run more than one instance of maintenance at a time, it won’t matter; but it’s better not to do this, if you can avoid it [↩]

As you may be aware, the OpenX ad server summarizes data and calculated priorities for Contract campaigns on the basis of the configured Operation Interval.

You may also be aware that it is the maintenance script that summarizes the data and calculates the priorities. But how does this script run? There are two different ways — either automatically, or manually.

By default, an out of the box OpenX ad server installation will be configured to run the maintenance script automatically. The OpenX admin guide lists some advantages to running the maintenance script automatically:

• Outstanding maintenance tasks are triggered by banner delivery;
• Enabled by default on new installations; and
• No need to create scheduled tasks.

However, there are also some disadvantages to relying on automatic maintenance, in The Guru’s opinion:

• Automatic maintenance is called from the “lg.php” and “avw.php” delivery scripts. Respectively, these are the scripts responsible for performing impression logging after banner delivery, and for delivery of banners for the Image and No Cookie Image zone tag types. However, you may have noticed that the XML-RPC zone tag delivery script (“axmlrpc.php”) is not mentioned here. In fact, delivery via XML-RPC, which does not use a logging beacon, will not trigger the automatic maintenance process — therefore relying on automatic maintenance when you are using XML-RPC delivery may deliver inconsistent results, as maintenance will not be triggered.
• As a result of needing to test for when maintenance needs to be run, and then actually running it, the “lg.php” and “avw.php” delivery scripts will not complete as quickly as when automatic maintenance is turned off. Although the OpenX ad server tries to minimize the effects as much as possible, there will nevertheless be some users that occasionally experience a delay in the scripts completing when visiting your website.
• Finally, if you have a low volume site, it is possible that some Operation Intervals will not have the automatic maintenance script run at all. This can lead to some odd campaign priority values, as the script has been designed to run each and every interval.

As a result, running the maintenance script manually is The Guru’s preferred method. This is very easy to set up in crontab, as the OpenX admin guide shows.

However, once you have set up a crontab job to run maintenance manually, you should also update your OpenX configuration, to obtain the maximum benefits. While working as the Administrator account, go to Configuration > Maintenance Settings and then disable the automatic maintenance system — this will ensure the delivery scripts do not need to test to see if maintenance has been run recently, and will ensure you get the maximum performance from your delivery scripts.

OpenX with automatic maintenance fully disabled.

Once you have done this, again while working as the Administrator account, go to Configuration > Maintenance, and OpenX will report the current maintenance script configuration. (Please note that you may need to wait for the end of the current Operation Interval to see the current settings.)

Finally, don’t forget to update your crontab entry to run the maintenance script once per Operation Interval, if you have opted to change the value from the default!

One of the least understood configuration options in OpenX is the “Operation Interval” setting. What is it, and what does it do?

Operation Interval in the Maintenance Settings

The first and foremost thing to know about the Operation Interval is that it is not a means of changing how often statistics will be updated in the user interface. OpenX, no matter what version you have installed, will only ever update the statistics in the user interface once per hour. That is it — end of story, end of discussion. Changing the Operation Interval to a value less than 60 minutes will not change this, and if you modify the Operation Interval setting expecting this to be the case, then you will be sorely disappointed.

So, if this is the case, what is the Operation Interval?

The Operation Interval is the mechanism by which OpenX performs internal statistics data aggregation and is the basis on which priority calculations are performed. That is:

• Statistics data is logged based on the Operation Interval (for OpenX 2.8, anyway);
• Statistics data is summarized and stored based on the Operation Interval; and
• Priority values for Contract Campaigns are calculated on the basis of the Operation Interval.

So, given that the Operation Interval is something that only relates to OpenX’s internal operations, why would you want to change it?

To answer this question, you need to understand the two effects that reducing the Operation Interval from the default value of 60 minutes has. (The Operation Interval can only be set to 5, 10, 15, 20, 30 or 60 minutes.)

Firstly, because data aggregation is performed using the Operation Interval value, if you reduce the value from the default of 60 minutes, then OpenX will need to store more data rows. This is because if the Operation Interval is reduced, then there are more Operation Intervals per hour, which means there will be less data aggregation based on the Operation Interval, and so more data rows that need to be stored. As the Operation Interval is the basis by which data is summarized and stored in the “data_intermediate_%” and “data_summary_%” tables in OpenX, which account for most of the database size, then you can expect that an OpenX installation with an Operation Interval of 30 minutes would require twice the database storage as an equivalent OpenX installation with an Operation Interval of 60 minutes¹.

Secondly, because the maintenance process which calculates Contract Campaign priority values does so on the basis of the Operation Interval, if you reduce the value from the default of 60 minutes, then the OpenX maintenance process will be required to perform more work to calculate the priorities. This is because the maintenance script calculates a priority for every active banner/zone link in the system for every Operation Interval, so if the Operation Interval is reduced, then there are more Operation Intervals per hour, which means there are more priority values to be calculated. The part of the maintenance script that calculates priorities for Contract Campaigns would be expected to take at least twice as long to run on an OpenX installation with an Operation Interval of 30 minutes than on an equivalent system with an Operation Interval of 60 minutes.

So, given the increase in database size and the decrease in performance of the maintenance script that comes with reducing the Operation Interval value, why would you ever change the Operation Interval from the default value of 60 minutes?

The answer is that if you run a very high end OpenX installation, having the Contract Campaign priority values updated only once per hour may sometimes not be enough, and you really do need these priority values updated every half an hour, or every 15 minutes, or maybe every 5 minutes. Be aware, though, that you will need some seriously expensive hardware if you expect to be able to run a large OpenX installation with an Operation Interval of 5 minutes².

Of course, if you do change the Operation Interval value, don’t forget:

• You need to update your system crontab so that the maintenance script is run once (and only once) per Operation Interval. There’s no point in setting your Operation Interval to 30 minutes, so that Contract Campaign priorities can be updated more often, and then not running the maintenance script every Operation Interval to do so!
• You should review your banner delivery cache setting, as there is no point in calculating the Contract Campaign priorities more often, only to have these values ignored due to a long cache value.

Finally, it is worth noting that if you don’t actually run any Contract Campaigns, then there is no point in reducing the Operation Interval from the default of 60 minutes, as there will be no benefit for you at all in this situation!

1. Note, however, that OpenX does not update previously logged and summarized data when you change the Operation Interval — the new value only applies to data logged and summarized after you make the change. So, if you’ve had the Operation Interval set to 30 minutes, changing it back to the 60 minute default will only mean that you will save space on all new data. [↩]
2. If you do have an Operation Interval value that is less than 60 minutes, you should probably review this value. Is it really necessary to have it lower than 60 minutes? Is the additional database space required really worth the benefits? Is the maintenance script actually able to complete running within the defined Operation Interval time, or is it taking longer than the Operation Interval? If so, you are not really obtaining any benefits, and you are probably causing issues by not running the maintenance script completely and fully every Operation Interval. [↩]

The Guru was browsing through a list of old links the other day, and came across an article on optimizing MySQL’s InnoDB storage engine. It’s definitely worth a read if you’ve never seen it before, but if you’re just after the high level summary:

• Add more memory to your server. Memory is good.
• If you have lots of memory, don’t forget that you should be running a 64-bit O/S, with the 64-bit version of MySQL, if you are going to be able to utilize that memory!
• The OpenX ad server is a write intensive application, so RAID-5 is bad. Very, very bad. Use RAID 0, or RAID 10.
• Tune your InnoDB settings. Remember, tuning is a process of measurement, followed by making a change, and then re-measuring.

If you’re a small website publisher, and you run your own OpenX ad server on the same server as your website, then you know that when your server goes down, everything stops. That’s just a fact of life for small website publishers, and there’s not much you can to but get your server back up and running as soon as possible.

However, if you are lucky enough to have seen your site grow up over time, or you work for a company that has a large site, you may be in a position where you run OpenX on a server that is different to the one that runs your website. (If you use the OpenX Hosted service, then you’re in the same situation — OpenX is running on a server that is different to the one that runs your website.)

If this is the case, then there is a possibility that the OpenX ad server system might go down, but your website server will still be up and running.

If this has ever happened to you, then you will know that this matters — and not just because your banners will not display. It can actually be a major issue because if the OpenX ad server is down, then your website may “hang” when it gets to the part of the site where a banner should be displayed¹. This is bad news for your users, because they won’t be able to view your website properly.

Below is the complete list of OpenX’s different zone tag types², and their susceptibility to this problem:

• Singe Page Call: Susceptible³.
• Javascript: Susceptible.
• iframe: Not susceptible.
• Interstitial or Floating DHTML: Susceptible.
• Image: Not susceptible.
• No Cookie Image: Not susceptible.
• Popup: Susceptible.
• XML-RPC: Susceptible⁴.

In all of the above cases, remember, the zone tags that are susceptible will cause your website to hang at the point where the tag is located in your site’s HTML during loading, so that your users won’t be able to see the entire page without some kind of delay. This means that it’s got nothing to do with whether banners load or not, as in all cases, if your OpenX ad server is not responding, you won’t get any banners displayed⁵.

So, if you have an OpenX ad server in use that doesn’t live on the same server as your website, what can you do to protect yourself from this situation?

Use a tag type that is not susceptible to the hanging page problem

As you may be aware, the Single Page Call and the Javascript zone tags provide advanced features that are not available in other tag types, just as supporting companion positioning, and the ability to avoid duplicating banners, etc. Assuming that you don’t need these features, then simply using the iframe or Image zone tag types makes sense⁶.

End place tags & position with CSS

Assuming that you would like the advanced features of the Single Page Call or Javascript zone tags, then it is possible to avoid hanging issues by placing the tags at the very end of your page’s HTML, and then positioning the banners on the page via CSS.

For example, here’s how you might place a Single Page Call tag at the end of a page, and position the banner via CSS⁷:

...

<p>End of HTML page!</p>

<!-- Generated by OpenX 2.8.1 -->
<script type='text/javascript'
  src='http://www.example.com/openx/www/delivery/spcjs.php?id=1'>
</script>

<div style="position: absolute; top: 10px; right: 10px;">
<script type='text/javascript'><!--// <![CDATA[
 /* [id1] Website 1 - Zone 1 */
 OA_show(1);
// ]]> --></script><noscript><a target='_blank'
  href='http://www.example.com/openx/www/delivery/ck.php?n=f1bb21f'>
  <img border='0' alt=''
  src='http://www.example.com/openx/www/delivery/avw.php?zoneid=1&amp;n=f1bb21f'
/></a>
</noscript>
</div>

</body>
</html>

There are many ways of CSS positioning, of course. Use whatever method suits you best!

Disadvantages

It is worth mentioning that is you do decide to use end placement of tags and position with CSS, this means that your banners will be the very last items to load on your web page. This means that you may find that you have a reduced number of impressions, as a result of users “navigating away” from the page before the banners have loaded; this in turn may result in reduced income (either through fewer impressions for CPM campaigns, or through fewer clicks/conversions for CPC/CPA campaigns).

However, this is something you will need to make a judgement call on — is it better to sacrifice a little income to ensure that your users can view your website immediately, albeit without banners, in the event that your external OpenX ad server has a problem; or is it better that if your OpenX ad server goes down, your website is not accessible?

Modify the OpenX tags

Finally, if you really want to get creative, you can modify the OpenX tags. This is for advanced users only, though. Best of luck!

1. How long might your site hang for before the rest of the content loads? This depends on exactly how the OpenX ad server has failed. The worst case, however, is when the server is not responding to requests, in which case most browsers will hang for minutes. [↩]
2. The Local Mode zone tag is not included as this tag can only be used when OpenX is installed on the same server as the website. [↩]
3. Additionally, as the Single Page Call tag, by default, recommends that part of the tag code be placed in the HTML document’s <head> section, this will cause the page to hang before any part of your web site has been displayed. [↩]
4. Results may vary, depending on the implementation language used. With the default PHP-based XML-RPC zone tag, the implementation actually causes the web page to hang at the start of the document, so page loading hangs before any content has been displayed. However, the default tag has a timeout of 15 seconds, which is much shorted than the hanging time for the other tags. [↩]
5. It also means that in all cases, the user’s browser will not appear to complete loading the page contents, even for the non-susceptible tags — it’s just that even though the browser will keep spinning, trying to load the banners, at least all of the other page content will have loaded for the user to see. [↩]
6. Remember that the zone tag type can only display image based banners; it can’t display Flash, text and HTML banners, for example. The No Cookie Image zone tag type is another special case, for displaying image based banners in Email/Newsletter zones. [↩]
7. It’s a contrived example, as you’d probably never use the Single Page Call zone tag for just one zone, but it illustrates the concept [↩]