Is Anyone Testing On Mobile?

OK, this is nuts. It all started with DOM vs Abstraction, it ended up with me testing almost every mobile device I have ... guess what I've discovered ...
First off, it came out that the Ember benchmark version tested in Parashuram's blog, was actually introduced during the "Is Ember fast yet?" site launch, which states pretty clearly the Ember team is aware, and working hard, to be there in pole position for performance too, but it's not there yet, so the benchmark is totally foolish, based on a pre-alpha release.
Debunked the "Ember is too damn slow" myth, it's actually unbelievable how badly is both compatibility and performance of all libraries mentioned in this benchmark:

• Facebook React is actually apparently based on the runtime transformer. This justifies the longer initial overhead/time to first draw, but not poor performance all over. Excluding Ember, which is again in a pre-alpha state in this benchmark, React, when it works, is the worst performing abstraction out there: Every. Single. Time.
  I wish there was a better benchmark to show real React muscles, because it looks like somebody forgot to test on real-world devices here ... I don't know what's going on: either the test is unfair, based on a pre-pre-release too, or this React + Virtual DOM story is very not suitable for Mobile.
  I hope the former, 'cause today Google officially started privileging Mobile capable website in its search engine ... so, you know ...
• Reactive performs decently only when it's capable of showing up. It's good on Desktop browsers, but it fails in many devices. Probably worst compatibility.
• paperclip, the absolute winner in terms of raw performance when it comes to choose an abstraction, unfortunately penalized by some incompatibility on some device. Kudos to the effort, please test more on Mobile since so far you are the best bet out there!

A big kudos goes to underscore/backbone too, this benchmark demonstrated this is the most scalable and reliable abstraction out there: it just fucking work, no matter where I've tested it, and performance are truly good! The only exception has been IE9 Mobile, but I'm pretty sure it's a matter of little tweak to run in there too.
Last, but the opposite of least, I cannot stress enough what I've said already in my previous post: the absolute winner in terms of performance AND compatibility has been the Pure DOM version of this benchmark!
Check it yourself, I've used modern devices, as well as more outdated. You can read the User Agent at the very beginning of each video, and realize this is true.

iOS 8.3 on iPad Mini Retina

This is a very modern device, and React performs 2X slower than DOM. paperclip seems to do some wizardry here, I'm not sure it's even faster than just DOM! underscore/backbone goes pretty smooth too.

iOS 8.2 on latest iPod

Basically the equivalent of an iPhone 5, the latest generation of iPod are a pretty good mobile target.
And yet, React takes its runtime delay to start and shows poor performance compared with everything else. Reactive takes a little bit too to start, but once it's there it's very fast. paperclip and DOM are again head to head, while underscore/backbone keeps scaling with its linear performance.

Blackberry 10.3 and Android 4.4

These are basically the most common, most updated versions, out there for both BB10 and Android.
It's quite embarrassing to see how every solution performs quite bad in here, but it's clear that paperclip and pure DOM won.

Early FirefoxOS and Ubuntu Phone

There's Chrome pulsing behind Ubuntu Phone, so regardless the hardware is not the best it performs decently in every occasion. However, this is the first time reactive shows its incompatibility (now supported, please read the update) ... FirefoxOS is not that common target but it's based on Firefox, and Firefox usually means Standards. I wasn't expecting any test to fail in here, but that happened. React still the slowest, excluding Ember.

Mobile IE9 and IE11

React here never appears in IE9, and it performs like 1 frame per each 3 seconds performance. Worst than pre-alpha Ember, and less compatible.
To be fair, every other bench performed crap except for paperclip and DOM. It's hard to tell who won here, 'cause why I have the feeling paperclip is slightly faster on IE11, it doesn't do so well in IE9 Mobile.

Kindle Fire HD

I'd say everything would perform quite well, for a real-world table with data app scenario, but I'd put perf down in ascending order: Ember, React, Reactive, underscore/backbone, and not sure between paperclip and DOM.

The Humble BB7

I like this phone, there's a reason die-hard mouse users like this too!
OK, OK, I've put this in just for compatibility sake. BB7 is based on WebKit and it performs actually reasonably well, thanks to its small screen.
However, the only survivors here are underscore/backbone and DOM. Even paperclip failed, while DOM won again.
This is just another case where choosing DOM would give you extra, wider, compatibility, at zero cost ;-)

As Summary

When anyone claims performance, I'll be there testing on real devices because these are where performance matters for real. We have quad core 2GHz CPUs and 8 GB of RAM on Desktop browsers, going fast there is like playing catch the dot with a laser pointer and your cat: deadly easy, everyone is fast already.
I wonder if anyone does these kind of tests, 'cause I've been paying money out of my pockets to be sure I could test all devices I was trying to support.
This is not necessarily something that requires a huge amount of money, you can buy cheap 2nd hand devices if you'd like to have little backward compatibility.
I have the impression most developers forget testing for real ... it's unfortunately not as easy as using a VM and testing Windows on Mac or Linux, but I can assure you it's a great way to be sure that things work, to start with, and if you claim any performance improvement you gonna actually see it for real where it matters.
Android 2.X is still at 6.8% of market share the 6th of April 2015.
Please remember to do some bench on your old phone, not everyone buys or gets a new phone every 6 months, many still buy 2nd hands devices 'cause they don't need to spend more, but they still need to have access to the Web.

Reactive Update

Rich uploaded a version that should support more legacy, like IE8. I've given previously unsupported devices a spin and indeed there's some extra compatibility.
FirefoxOS and webOS now run. I've compared performance only against the DOM.
However, it took 4 attempts to have webOS non stuck, and same happened to the BB7 device, which I've left out of the video to avoid troubles.
IE 9 Mobile, which is different from IE9 Desktop, is still a white page. This is not a big deal, considering how poor is IE9 usage and online de-facto support anyway, but Android 2.3 is still unsupported.



I'd say good job on webOS, I can't tell the difference, but yet DOM is clearly faster on FirefoxOS ;-)
Also the bootstrap could be probably improved on reactive side, pre-compiling the benchmark, if that's an option (I don't actually know).