At the weekend I was experimenting with ZeroMQ (0MQ for short here) for handling requests for items in an online shop. The idea behind it was simple – schedule requests for items to be added to a shopping cart, and don’t allow overselling of said stock items. The legacy code handles requests by wrapping in a transaction and checking for the amount of items currently available (sales + in existing carts), then adding/rejecting as appropriate.

If however there is significant traffic, this can result in oversells as the number of items available can be reported the same to more than 1 user. If both users request the final item, your oversell happens. I wanted to remove the likelihood of this happening, and so I turned to a queueing system to see how this could make a difference. I’d seen the video of Ian Barber’s presentation at PHP UK last year, and liked the apparent simplicity of 0MQ coupled with the lightweight in-memory approach so decided to experiment a bit.

After downloading and installing the 0MQ library, and the pecl extension for PHP, I was all set to go.

The idea seemed simple:

• Take all requests for items, and pop them into a queue
• If a response from the queue isn’t available immediately, the customer should be shown a page which polls periodically for the result.
• When the result is available, redirect the customer to their cart (success) or back to the item page (failure)

The 0MQ documentation at first seemed thorough, but reading through I realised it was a bit too verbose for my liking. I can appreciate the use of metaphors and friendly language, but it seemed to hinder understanding of what 0MQ could offer, coupled with everything being on the same page. Slightly overwhelming! In contrast, RabbitMQ’s tutorial pages seemed to be a lot clearer. For a developer new to queueing system, this was definitely a win for RabbitMQ – a case of which-image-represents-best-what-I-want *click* etc. However, I stuck with 0MQ for now for the reasons outlined above.

I settled on the Push-Pull methodology, “Divide and Conquer” in the 0MQ docs. This would satisfy the requirements above of being able to dump a request into a queue and periodically (via another mechanism) check for a result. I’ve shamelessly ripped off the accompanying image from their docs here in order to illustrate my point :-)

In the above image, the part labelled “Ventilator” can be equated to the user requesting a quantity of items to add to a shopping cart, and my queue handler is a single instance of a worker. In my implementation, I went for something like the following process:

• User requests 1 of item A via form
• Memcache entry is created with a unique key for the user’s request
• 0MQ request is created, “allocate me 1 of item A, for my session ID X and unique key ABC”
• … polling happens here …
• 0MQ worker receives message and performs relevant checks
• Memcache value for supplied unique key is updated with response
• Polling for this updated value causes a redirect for the user accordingly

Polling checked the memcache key for a successful response. I decided to use memcache for this as it was quick to get going and I only technically needed a key-value setup. I also wanted to avoid unnecessary hits on a database if possible, preferring to keep this for transactional data. Plus I also wanted to play with memcache a bit ;-)

Testing

In tests, this setup seemed to work well, and the updated memcache value ended up being a serialized array with a success/error code, and some additional information eg “Sold out”, “Item is unavailable”, “You requested 2 but only 1 is available”. This enabled better feedback for the user when their request was unsuccessful.

Running this in small-scale tests was great. It was good to be able to start and stop the queue handler, and see the user being automatically bounced to the “please wait” polling page whenever the queue handler wasn’t running or had a backlog of messages to process. Fire up the handler, checks take place and the user is then redirected accordingly.

When running this under heavier loads however, we hit instances where occasionally (due to intentionally-bad Apache settings), the server processes ran out of memory and required a restart of Apache. This was done deliberately to see how the queue could recover from different scenarios. As a result, restarting Apache would cause a flood of messages to be sent to the queue handler in one go, presumably because they’d been blocked somehow previously with the large load. This was confusing and unexpected behaviour to me, however I presumed this was because Apache (with mod_php) had been struggling to send the messages out under the load, and a restart caused a flush of some sorts. I also presume this isn’t down to any fault of 0MQ. It does however pose the question of what to do in a scenario where messages can potentially be lost, but this could be solved by a message queue that persists messages somewhere until completion.

Despite this teething issue which wasn’t really down to any fault of 0MQ, the system works great. Being new to queueing like this, I’m not entirely sure how this would scale and still keep the ability to not oversell an item – database transactions from multiple workers would need to be synchronised somehow to ensure that figures are kept up to date. But overall, I’m very happy with it as a starting point for a replacement solution.

Mini post. I can never find this information on the PEAR site, so this is just a reference for me really :-)

After all code changes have been made and pushed into the repository, do the following

[rich@localhost]# pfm
* Follow instructions here, eg update version number, release notes
* Regenerate Contents
* Save & Exit
[rich@localhost]# git add package.xml
[rich@localhost]# git commit
[rich@localhost]# git tag "RELEASE-x.y.z"
[rich@localhost]# pear package
* Package file is created, release this through PEAR website as normal

I’ve been experimenting with sending emails but rendering the content with Twig. This allows emails to extend a standard layout where required, but also gives us the flexibility to insert other content into the template which may be required. In the first instance, we’re interested in supplying the email subject here.

The concept is simple – within Symfony2, firstly get Twig from the container or inject it accordingly. You can then render the template as a whole, and also extract any other items required. My example is taken from a listener class I’ve been using, but the concepts can be applied appropriately depending on where you want to use it:

class MyListener
{
    protected $mailer;
    protected $twig;
 
    public function __construct(\Swift_Mailer $mailer, \Twig_Environment $twig)
    {
        $this->mailer = $mailer;
        $this->twig = $twig;
    }
 
    public function onMyevent($myEvent)
    {
        $emailTo = $myEvent->getEmailTo();
 
        // Load the template in
        $templateFile = "MyBundle:emails:myEmailTemplate.twig";
        $templateContent = $this->twig->loadTemplate($templateFile);
 
        // Render the whole template including any layouts etc
        $body = $templateContent->render(array("someParam" => "foo"));
 
        // Get the subject block out
        // If the block doesn't exist, use a default
        $subject = ($templateContent->hasBlock("subject")
            ? $templateContent->renderBlock("subject", array("param1" => $someInformation->foo))
            : "Default subject here");
 
        $subject = trim($subject);
 
        // Send email
        $message = \Swift_Message::newInstance()
            ->setSubject($subject)
            ->setFrom("info@example.com")
            ->setTo($emailTo)
            ->setBody($body)
        ;
        $this->mailer->send($message);
    }
}

This means we can put something like the following in our template file:

{% extends "MyBundle:emails:emailLayout.twig" %}
 
{# This block is rendered in emailLayout.twig #}
{% block body %}
Hello, world. This is your email content here
{% endblock %}
 
{% block subject %}
Your subject line goes here
{% endblock %}

I was hoping to use the Twig {% spaceless %} tag, but it appears that it’s only available for HTML (sadly this PR has been closed; shame as it would have been useful). Hence the trim() call when retrieving the subject.

Of course, you can extend the above to inject in default email-FROM-addresses and names, template names and so on where appropriate.

Update: Thanks to Fabien for pointing out (comments below) that you can remove whitespace by using the following (hyphens at the end of the opening block tag and at the start of endblock:

{% block subject -%}
Your subject line goes here
{%- endblock %}

I’ve been thinking about next year’s New Year’s Resolutions again. One which I’m going to repeat from this year was to blog more (inspired by Lorna’s post about blogging more back in January), but also a new one I want to achieve is to speak at a conference.

I’ve presented papers at a couple of non-web related conferences a number of years ago, but nothing in the web field, and I’m unsure as to the correct approach to take with this. My main reason for wanting to speak is that I want to be able to share knowledge I’ve learnt in the hope that this may be of interest to at least one person. I’m also passionate about the technology I use in day-to-day development and would most probably like to include this. My main concern however is “would it be interesting enough?”, which I’m inclined to feel is a catch-22 situation – I won’t know until presenting it, but then I won’t get a chance to present it unless I submit.

Calls for papers are continuously springing up, and I’m wondering about the best approach here to get my foot in the door so to speak. Is it a case of writing up something that I feel is interesting and would benefit others, and just taking the leap? Finding local Uncons/user groups and trying the material out there?

I’d be interested to hear any tips and pointers :-)

NB: Cross-posted on our work blog at White October :-)

I spent some time last week working on automated MailChimp integration with one of our Symfony2 sites. The project uses Behat and Mink extensively for BDD testing, so I wanted to be able to test that when users registered on our site, they were automatically added to the client’s MailChimp distribution list. Unsubscription could happen via a checkbox in the user’s account settings and this needed to work the same way behind the scenes.

I opted to use one of the available MailChimp bundles to aid integration and with a bit of tweaking this worked great. The bundle comes with a stub connection which is able to collate requests sent and make them available for inspection. I wanted to be able to utilise this neatly inside a set of tests, and potentially also make it available in the dev environment. A bit of skimming the Symfony2 cookbook led me to the entry on Symfony2 data collectors.

The concept is simple – a data collector is registered and is able to, well, collect data! For example, the number of queries during a request, or the number of emails sent. This can then be accessed via the profiler in the relevant environment (dev, test etc). In my case I wanted to collect and subsequently inspect all requests sent to MailChimp.

First step – create me a data collector:

< ?php
 
namespace Jirafe\Bundle\MailChimpBundle\DataCollector;
 
use Symfony\Component\HttpKernel\DataCollector\DataCollector;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use Jirafe\Bundle\MailChimpBundle\Connection\ConnectionInterface;
 
class MailChimpDataCollector extends DataCollector
{
    private $connection;
 
    public function __construct(ConnectionInterface $connection)
    {
        $this->connection = $connection;
    }
 
    public function collect(Request $request, Response $response, \Exception $exception = null)
    {
        $this->data = array(
            'requests'  => $this->connection->getRequests(),
        );
    }
 
    public function getRequestCount()
    {
        return count($this->data['requests']);
    }
 
    public function getRequests()
    {
        return $this->data['requests'];
    }
 
    public function getName()
    {
        return 'mail_chimp';
    }
}

The collect() method is the interesting part here – this is called whenever data collection is required – usually once per request. We collect the requests from the connection supplied to the constructor.

Next step, register this during the bundle’s load() method:

// Add connection to data collector
$definition = $container->getDefinition('mail_chimp.data_collector');
$definition->addArgument(new Reference(sprintf('mail_chimp.connection.%s', $config['connection'])));

The above uses the connection type specified in the bundle’s configuration option and previously registered as a service in the same method. If you’re creating your own custom collector, this is just the normal parameters you’d pass in to your service when it’s instantiated.

Finally, I added a method and subsequent implementation to the interface and related connection classes within the bundle, so that MailChimp request objects were collected in an array, and made available to the data collector upon request. This is the getRequests() method listed above in the data collector; called in the collect() method. Quick ‘n’ dirty here, since there would only ever be a single MailChimp request (possibly 2) made in any one request.

Once all this was in place, I could then test correctly. Note that I’d rather have done the other way round (test and then develop) but since this was experimentation, I figured it was OK ;-) In my Behat scenarios, I added the getSymfonyProfiler() method detailed in a Behat/Mink cookbook entry, and proceeded to write my step (Behat 1.x-style):

$steps->Then('/^my email address is unsubscribed from the relevant MailChimp list$/', function($world) {
 
    $profiler = $world->getSymfonyProfiler();
    if (!$profiler)
    {
      throw new \RuntimeException("Symfony2 profiler not present for tests, wrong driver?");
    }
    $mcDataCollector = $profiler->getCollector("mail_chimp");
    $listID = $world->container->getParameter("mailchimp_list_id");
 
    // Check the requests made to make sure we have an unsubscribe
    assertEquals(1, $mcDataCollector->getRequestCount());
 
    $requests = $mcDataCollector->getRequests();
    $req1 = $requests[0];
    assertEquals("listUnsubscribe", $req1->getMethod());
 
    $params = $req1->getParams();
    assertEquals($listID, $params["id"]);
 
    // Fire the redirect on the client
    $world->getSession()->getDriver()->getClient()->followRedirect();
});

The final line here is as a result of disabling redirects in a previous step since the data collector operates on a per-request basis. If you’re subscribing eg via a form POST, then you’ll need to disable redirects prior to submitting the form, otherwise your data collector will be empty when you check it. Disable them and then redirect after you’ve carried out your checks.

The implementation for the above is up on our Github MailChimpBundle fork here; I’m not particularly fond of having to hack about the connection classes, and ideally it would be nice to layer the data collector on top without having to adjust existing code. That said, it does the job fine here, and it’s good to use even more features of Symfony2 ;-)

Came across this one today whilst integrating some new code into my current Symfony2 project. The project has 2 custom Twig extensions – one with some filters in, one with some custom methods. When creating the latter, I’d copied the class from the former and tweaked to suit. In doing so, I’d left these lines in:

    /**
     * Name of this extension
     *
     * @return string
     */
    public function getName()
    {
        return "myfirstextension";
    }

The Twig documentation states that all extensions should be named uniquely, unless you need to replace any existing extensions, in which case you should name them the same. See here for a better explanation :-) Great for extensibility and adjusting extensions that maybe you don’t have control over. Not so good when both extensions do something different and you don’t quite understand why it’s not working….

Anyway, quick question in the #symfony IRC channel and Stof came to the rescue. Give that man a medal :-) and name your extensions uniquely if you don’t want them to overwrite others!

Quick post this, as I know I’ll forget/lose this code otherwise… Sometimes you need the ability to force a logout of a user. The example I’ll use here is a user requesting their account be deactivated – you assume that upon confirmation of the deactivation the user’s model is marked as such, and the user is subsequently logged out and shown a confirmation page:

// Update your model here. Example:
$user = clone $this->get("security.context")->getToken()->getUser();
$user->setEnabled(false);
$user->save();
 
// Log the user out
$this->get("request")->getSession()->invalidate();
$this->get("security.context")->setToken(null);
 
// Flash var to enable the user to see the deactivation success page
$this->get("session")->setFlash('deactivate_account.success', true);
 
// and redirect to success page
return new RedirectResponse($this->get("router")->generate("deactivate_account_success"));

Your deactivation page should allow viewing without requiring authentication (anonymous users can view it). The above code should be self-explanatory – invalidate the current session, and set the security token to null (rather than the previously-authenticated token). The flash variable I set so that viewing of the success page is one-shot rather than being permanently viewable. Just my preference :-)

Just came across this article on ubuntuforums.org which describes how to add a greyscale printing option when using a Canon MP270 printer under Ubuntu. Reprinting here in case the original disappears. Dead simple!:

After reading the manual from Canon, I have made the grayscale printing working too. For those who need it, here is the fast tip:

Under System–>Administration–>Printing

right click MP250 that you have already setup, then select Properties from the popup menu, go to Job Options, scroll down to the bottom, add this option in:

CNGrayscale

click the Add button to add the option to the list.

Beside the CNGrayscale option, type in TRUE in the textfield, then click Apply, and close the Properties dialog.

By doing this, this printer is setup to be Grayscale printing by default. You can definitely have another copy of printer setup, pointing to the same printer, but for color printing. This is what i have done.

I’ve had a case recently whereby I needed to implement a registration form in my current Symfony2 application, based on a Mandango model. This form consisted of selected fields from my model, along with a separate “I accept the terms and conditions” checkbox; pretty standard. In Symfony 1, it was straightforward to add in non-model fields, however in Symfony2 the convention for forms is to create a domain model which represents your form, which then gets populated with the data.

Initially I went with extending from the Mandango model (in my case, Model\MyOwnBundle\User being the original model) and adding in a separate get/setTermsAndConditions() method, but this caused problems when saving the model down, as Mandango couldn’t find the document class (since I’d extended it and called it UserRegistration).

After a bit of digging and experimentation however, the following code does what I need it to, without having to extend my existing model or create a new one (which I’d then have to use to populate my actual Mandango model):

class RegisterFormType extends AbstractType
{
    /**
     * Constructs our registration form
     * 
     * @param \Symfony\Component\Form\FormBuilder $builder
     * @param array $options
     * @return void
     */
    public function buildForm(FormBuilder $builder, array $options)
    {
        // Build the form
        $builder->
            add("firstName", "text")->
            add("lastName", "text")->
            add("emailAddress", "email")->
            add("t_and_c",
                "checkbox",
                array(
                    "property_path" => false,
                )
            );
 
        // "True" validator on the form for t&c
        $builder->
            addValidator(new CallbackValidator(function(FormInterface $form)
            {
                if (!$form["t_and_c"]->getData())
                {
                    $form->addError(new FormError('Please accept the terms and conditions in order to register'));
                }
            })
        );
    }
 
    // ...
}

The above adds the t_and_c field but using "property_path" => false means it’s not validated against the model and therefore no need to implement separate get/set() methods. Instead we use a callback validator on the form to check the value of the t_and_c field and ensure it’s been checked. Et voila :-)

NB: This was originally posted over at the White October blog but I figured it would fit in nicely here too :-)

I’ve spent the past week starting a new sprint on our latest large project, and as we’re keen to get our clients as involved as possible during sprints, we decided to get down with some Behaviour-Driven Development (BDD) which allows them to contribute to writing the actual acceptance criteria for the user stories. We’re pretty big on Test-Driven Development here at WO Towers but sometimes it’s necessary to strike a balance between testing every minute detail and no testing. BDD seems to meet that balance whilst providing good code coverage with tests. Note: here I speak of functional tests rather than unit tests.

The beauty of BDD is that it’s written in plain everyday language, and this can be directly translated to code snippets. Therefore the client is happy working “in plain english”, and the developer can crack on with writing the actual code to test these parts.

We’ve chosen to use Symfony2 for the project (more on that in a separate post) and for the BDD aspect we’ve gone with Behat (coupled with Mink); both of which have bundles available for Symfony2. Essentially, BDD boils down to Features, consisting of a number of Scenarios, which in turn consist of a number of Steps. Here’s an example:

Feature: Login to the admin area
	As an administrator
	I want to be able to log into an admin area
	So I can administer MyApp

	Scenario: Joe Bloggs logs into the admin area
	  Given I visit the admin area
	  When I am shown the login form
	  And I click login with my correct credentials
	  Then I should be on /admin/home

This could translate to code such as the following:

$steps->Given('/^I visit the admin area$/', function($world) {
 
    $world->getSession()->visit($world->getPathTo("/admin"));
});
 
$steps->When('/^I am shown the login form$/', function($world) {
 
    $doc = $world->getSession()->getPage();
 
    assertTrue($doc->hasField("_username"));
    assertTrue($doc->hasField("_password"));
});
 
$steps->And('/^I click login with my correct credentials$/', function($world) {
 
    $doc = $world->getSession()->getPage();
 
    $doc->fillField("_username", "joe");
    $doc->fillField("_password", "bloggs");
    $doc->clickButton("Log in");
});
 
$steps->Then('/^(?:|I )should be on (?P<page>.+)$/', function($world, $page) {
 
    assertEquals(
        parse_url($world->getPathTo($page), PHP_URL_PATH),
        parse_url($world->getSession()->getCurrentUrl(), PHP_URL_PATH)
    );
});
</page>

The beauty of Behat is that the step definitions are reusable, meaning that if you stick to a standard way of describing your steps, the amount of code you need to write is vastly reduced, compared with normal functional TDD. Steps that aren’t implemented yet are given default code as per:

$steps->Given('/^I visit the admin area$/', function($world) {
    throw new \Behat\Behat\Exception\PendingException();
});

which will translate into the output shown as a step needing to be implemented. Behat also takes note of the Pending exception being thrown and automatically skips any steps following this in the scenario, notifying you of this at the end of the test run.

Behat also comes with a number of standard step definitions built-in, although these are somewhat buried in the code and not in the docs :-( So for example the following steps are part of the included ones:

I go to /my/url:
 
$steps->When('/^(?:|I )go to (?P<page>.+)$/', function($world, $page) {
    $world->getSession()->visit($world->getPathTo($page));
});
</page>

and

I fill in "username" with "foo":
 
$steps->When('/^(?:|I )fill in "(?P<field>[^"]*)" with "(?P<value>[^"]*)"$/', function($world, $field, $value) {
    $world->getSession()->getPage()->fillField($field, $value);
});
</value></field>

which translates as even less code to write. Bonus!

Tips ‘n’ tricks

The Mink (browser emulator abstraction layer) setup comes with a Symfony2 driver which is great for just testing normal usage. In our project however we’re integrating Facebook authentication using the FOS FacebookBundle, and we needed a way to test that the user was being bounced to Facebook in order to log in. The solution for this was simple – switch to using Goutte as the driver, as per the BehatBundle docs, and voila, you can test the current URL contains eg facebook.com or whatever. The actual testing of the authentication itself I’ll write up in a separate blog post as it’s a bit epic for here.

The documentation mentions using a separate front controller for testing, with a separate environment configuration. This is definitely recommended, particularly if you’re using Goutte. If you need services to be available, for example your entity manager, pop something like this in your env.php file:

$kernel = new AppKernel('test', true);
$kernel->loadClassCache();
$service = $kernel->getContainer()->get("your.service.here");

You can also use the env.php file for setting up fixtures that need to be reset for each Scenario that you create, eg pre-defined user states and so on.

Caveats

Unless you’re using Sahi, you obviously can’t test things such as Javascript popups effectively. I’m in two minds as to whether this should be handled by a new type of Exception which can be thrown e.g. a TestInRealBrowserException(). That way it could be caught and notified to the user running the tests, or flagged as part of an automated build process for manual testing.

Overall, getting into BDD has been a fun and rewarding experience. At times the lack of docs make it somewhat frustrating, but this is easily solved by a) digging into the code and b) probably me getting off my ass and submitting some docs back. The ease of which the steps can be written though is a massive time saver when writing your tests, coupled with the reusability of the code. Big thumbs-up from me!