Experimentation with ZeroMQ
At the weekend I was experimenting with ZeroMQ (0MQ for short here) for handling requests for items in an online shop. The idea behind it was simple – schedule requests for items to be added to a shopping cart, and don’t allow overselling of said stock items. The legacy code handles requests by wrapping in a transaction and checking for the amount of items currently available (sales + in existing carts), then adding/rejecting as appropriate.
If however there is significant traffic, this can result in oversells as the number of items available can be reported the same to more than 1 user. If both users request the final item, your oversell happens. I wanted to remove the likelihood of this happening, and so I turned to a queueing system to see how this could make a difference. I’d seen the video of Ian Barber’s presentation at PHP UK last year, and liked the apparent simplicity of 0MQ coupled with the lightweight in-memory approach so decided to experiment a bit.
After downloading and installing the 0MQ library, and the pecl extension for PHP, I was all set to go.
The idea seemed simple:
- Take all requests for items, and pop them into a queue
- If a response from the queue isn’t available immediately, the customer should be shown a page which polls periodically for the result.
- When the result is available, redirect the customer to their cart (success) or back to the item page (failure)
The 0MQ documentation at first seemed thorough, but reading through I realised it was a bit too verbose for my liking. I can appreciate the use of metaphors and friendly language, but it seemed to hinder understanding of what 0MQ could offer, coupled with everything being on the same page. Slightly overwhelming! In contrast, RabbitMQ’s tutorial pages seemed to be a lot clearer. For a developer new to queueing system, this was definitely a win for RabbitMQ – a case of which-image-represents-best-what-I-want *click* etc. However, I stuck with 0MQ for now for the reasons outlined above.
I settled on the Push-Pull methodology, “Divide and Conquer” in the 0MQ docs. This would satisfy the requirements above of being able to dump a request into a queue and periodically (via another mechanism) check for a result. I’ve shamelessly ripped off the accompanying image from their docs here in order to illustrate my point :-)
In the above image, the part labelled “Ventilator” can be equated to the user requesting a quantity of items to add to a shopping cart, and my queue handler is a single instance of a worker. In my implementation, I went for something like the following process:
- User requests 1 of item A via form
- Memcache entry is created with a unique key for the user’s request
- 0MQ request is created, “allocate me 1 of item A, for my session ID X and unique key ABC”
- … polling happens here …
- 0MQ worker receives message and performs relevant checks
- Memcache value for supplied unique key is updated with response
- Polling for this updated value causes a redirect for the user accordingly
Polling checked the memcache key for a successful response. I decided to use memcache for this as it was quick to get going and I only technically needed a key-value setup. I also wanted to avoid unnecessary hits on a database if possible, preferring to keep this for transactional data. Plus I also wanted to play with memcache a bit ;-)
Testing
In tests, this setup seemed to work well, and the updated memcache value ended up being a serialized array with a success/error code, and some additional information eg “Sold out”, “Item is unavailable”, “You requested 2 but only 1 is available”. This enabled better feedback for the user when their request was unsuccessful.
Running this in small-scale tests was great. It was good to be able to start and stop the queue handler, and see the user being automatically bounced to the “please wait” polling page whenever the queue handler wasn’t running or had a backlog of messages to process. Fire up the handler, checks take place and the user is then redirected accordingly.
When running this under heavier loads however, we hit instances where occasionally (due to intentionally-bad Apache settings), the server processes ran out of memory and required a restart of Apache. This was done deliberately to see how the queue could recover from different scenarios. As a result, restarting Apache would cause a flood of messages to be sent to the queue handler in one go, presumably because they’d been blocked somehow previously with the large load. This was confusing and unexpected behaviour to me, however I presumed this was because Apache (with mod_php) had been struggling to send the messages out under the load, and a restart caused a flush of some sorts. I also presume this isn’t down to any fault of 0MQ. It does however pose the question of what to do in a scenario where messages can potentially be lost, but this could be solved by a message queue that persists messages somewhere until completion.
Despite this teething issue which wasn’t really down to any fault of 0MQ, the system works great. Being new to queueing like this, I’m not entirely sure how this would scale and still keep the ability to not oversell an item – database transactions from multiple workers would need to be synchronised somehow to ensure that figures are kept up to date. But overall, I’m very happy with it as a starting point for a replacement solution.
Mini post. I can never find this information on the PEAR site, so this is just a reference for me really :-)
After all code changes have been made and pushed into the repository, do the following
[rich@localhost]# pfm * Follow instructions here, eg update version number, release notes * Regenerate Contents * Save & Exit [rich@localhost]# git add package.xml [rich@localhost]# git commit [rich@localhost]# git tag "RELEASE-x.y.z" [rich@localhost]# pear package * Package file is created, release this through PEAR website as normal
Rendering emails with Twig in Symfony2
I’ve been experimenting with sending emails but rendering the content with Twig. This allows emails to extend a standard layout where required, but also gives us the flexibility to insert other content into the template which may be required. In the first instance, we’re interested in supplying the email subject here.
The concept is simple – within Symfony2, firstly get Twig from the container or inject it accordingly. You can then render the template as a whole, and also extract any other items required. My example is taken from a listener class I’ve been using, but the concepts can be applied appropriately depending on where you want to use it:
class MyListener { protected $mailer; protected $twig; public function __construct(\Swift_Mailer $mailer, \Twig_Environment $twig) { $this->mailer = $mailer; $this->twig = $twig; } public function onMyevent($myEvent) { $emailTo = $myEvent->getEmailTo(); // Load the template in $templateFile = "MyBundle:emails:myEmailTemplate.twig"; $templateContent = $this->twig->loadTemplate($templateFile); // Render the whole template including any layouts etc $body = $templateContent->render(array("someParam" => "foo")); // Get the subject block out // If the block doesn't exist, use a default $subject = ($templateContent->hasBlock("subject") ? $templateContent->renderBlock("subject", array("param1" => $someInformation->foo)) : "Default subject here"); $subject = trim($subject); // Send email $message = \Swift_Message::newInstance() ->setSubject($subject) ->setFrom("info@example.com") ->setTo($emailTo) ->setBody($body) ; $this->mailer->send($message); } }
This means we can put something like the following in our template file:
{% extends "MyBundle:emails:emailLayout.twig" %} {# This block is rendered in emailLayout.twig #} {% block body %} Hello, world. This is your email content here {% endblock %} {% block subject %} Your subject line goes here {% endblock %}
I was hoping to use the Twig {% spaceless %} tag, but it appears that it’s only available for HTML (sadly this PR has been closed; shame as it would have been useful). Hence the trim() call when retrieving the subject.
Of course, you can extend the above to inject in default email-FROM-addresses and names, template names and so on where appropriate.
Update: Thanks to Fabien for pointing out (comments below) that you can remove whitespace by using the following (hyphens at the end of the opening block tag and at the start of endblock:
{% block subject -%} Your subject line goes here {%- endblock %}
