Replacing meteor spiderable with alternative server side snippet rendering
After getting fed up with meteor’s spiderable spawning phantomjs instances all over the place when receiving a couple of concurrent requests for different resources we decided to switch to a different strategy to support spiders/crawlers.
Instead of using the spiderable package to fully render the application including all the logic and data in the blaze template we decided to serve alternative snippets for our SEO heavy content whenever google, facebook or another crawler hits the application server with specific user agent.
The following piece of meteor backend code shows how to create a router that is activated triggered when the botagent or google’s “unescaped fragment” query param is detected.
The SeoRouter can then be configured to match specific request urls. This example shows matching a profile url, retrieve a user profile and use the SSR.compileTemplate method to render the specified blaze template to an html string. Please note that the helpers used in this template should also be defined here.
The last example shows the actual html template, which is a stripped down html page containing just the SEO meta tags, a heading and a paragraph.
Please note that this method of search engine optimalization is called “Cloaking” and could be considered as “Black hat SEO” if the content you are serving in the snippet is completely different from the content you would be serving in the single page application. In our case it has proven to be a scalable and effective method of solving the seo problem in our meteor application.