My JavaScript book is out! Don't miss the opportunity to upgrade your beginner or average dev skills.

Wednesday, April 22, 2015

Is Anyone Testing On Mobile?

OK, this is nuts. It all started with DOM vs Abstraction, it ended up with me testing almost every mobile device I have ... guess what I've discovered ...
First off, it came out that the Ember benchmark version tested in Parashuram's blog, was actually introduced during the "Is Ember fast yet?" site launch, which states pretty clearly the Ember team is aware, and working hard, to be there in pole position for performance too, but it's not there yet, so the benchmark is totally foolish, based on a pre-alpha release.
Debunked the "Ember is too damn slow" myth, it's actually unbelievable how badly is both compatibility and performance of all libraries mentioned in this benchmark:
  • Facebook React is actually apparently based on the runtime transformer. This justifies the longer initial overhead/time to first draw, but not poor performance all over. Excluding Ember, which is again in a pre-alpha state in this benchmark, React, when it works, is the worst performing abstraction out there: Every. Single. Time.
    I wish there was a better benchmark to show real React muscles, because it looks like somebody forgot to test on real-world devices here ... I don't know what's going on: either the test is unfair, based on a pre-pre-release too, or this React + Virtual DOM story is very not suitable for Mobile.
    I hope the former, 'cause today Google officially started privileging Mobile capable website in its search engine ... so, you know ...
  • Reactive performs decently only when it's capable of showing up. It's good on Desktop browsers, but it fails in many devices. Probably worst compatibility.
  • paperclip, the absolute winner in terms of raw performance when it comes to choose an abstraction, unfortunately penalized by some incompatibility on some device. Kudos to the effort, please test more on Mobile since so far you are the best bet out there!
A big kudos goes to underscore/backbone too, this benchmark demonstrated this is the most scalable and reliable abstraction out there: it just fucking work, no matter where I've tested it, and performance are truly good! The only exception has been IE9 Mobile, but I'm pretty sure it's a matter of little tweak to run in there too.
Last, but the opposite of least, I cannot stress enough what I've said already in my previous post: the absolute winner in terms of performance AND compatibility has been the Pure DOM version of this benchmark!
Check it yourself, I've used modern devices, as well as more outdated. You can read the User Agent at the very beginning of each video, and realize this is true.

iOS 8.3 on iPad Mini Retina

This is a very modern device, and React performs 2X slower than DOM. paperclip seems to do some wizardry here, I'm not sure it's even faster than just DOM! underscore/backbone goes pretty smooth too.

iOS 8.2 on latest iPod

Basically the equivalent of an iPhone 5, the latest generation of iPod are a pretty good mobile target.
And yet, React takes its runtime delay to start and shows poor performance compared with everything else. Reactive takes a little bit too to start, but once it's there it's very fast. paperclip and DOM are again head to head, while underscore/backbone keeps scaling with its linear performance.

Blackberry 10.3 and Android 4.4

These are basically the most common, most updated versions, out there for both BB10 and Android.
It's quite embarrassing to see how every solution performs quite bad in here, but it's clear that paperclip and pure DOM won.

Early FirefoxOS and Ubuntu Phone

There's Chrome pulsing behind Ubuntu Phone, so regardless the hardware is not the best it performs decently in every occasion. However, this is the first time reactive shows its incompatibility (now supported, please read the update) ... FirefoxOS is not that common target but it's based on Firefox, and Firefox usually means Standards. I wasn't expecting any test to fail in here, but that happened. React still the slowest, excluding Ember.

Mobile IE9 and IE11

React here never appears in IE9, and it performs like 1 frame per each 3 seconds performance. Worst than pre-alpha Ember, and less compatible.
To be fair, every other bench performed crap except for paperclip and DOM. It's hard to tell who won here, 'cause why I have the feeling paperclip is slightly faster on IE11, it doesn't do so well in IE9 Mobile.

Kindle Fire HD

I'd say everything would perform quite well, for a real-world table with data app scenario, but I'd put perf down in ascending order: Ember, React, Reactive, underscore/backbone, and not sure between paperclip and DOM.

The Humble BB7

I like this phone, there's a reason die-hard mouse users like this too!
OK, OK, I've put this in just for compatibility sake. BB7 is based on WebKit and it performs actually reasonably well, thanks to its small screen.
However, the only survivors here are underscore/backbone and DOM. Even paperclip failed, while DOM won again.
This is just another case where choosing DOM would give you extra, wider, compatibility, at zero cost ;-)

As Summary

When anyone claims performance, I'll be there testing on real devices because these are where performance matters for real. We have quad core 2GHz CPUs and 8 GB of RAM on Desktop browsers, going fast there is like playing catch the dot with a laser pointer and your cat: deadly easy, everyone is fast already.
I wonder if anyone does these kind of tests, 'cause I've been paying money out of my pockets to be sure I could test all devices I was trying to support.
This is not necessarily something that requires a huge amount of money, you can buy cheap 2nd hand devices if you'd like to have little backward compatibility.
I have the impression most developers forget testing for real ... it's unfortunately not as easy as using a VM and testing Windows on Mac or Linux, but I can assure you it's a great way to be sure that things work, to start with, and if you claim any performance improvement you gonna actually see it for real where it matters.
Android 2.X is still at 6.8% of market share the 6th of April 2015.
Please remember to do some bench on your old phone, not everyone buys or gets a new phone every 6 months, many still buy 2nd hands devices 'cause they don't need to spend more, but they still need to have access to the Web.

Reactive Update

Rich uploaded a version that should support more legacy, like IE8. I've given previously unsupported devices a spin and indeed there's some extra compatibility.
FirefoxOS and webOS now run. I've compared performance only against the DOM.
However, it took 4 attempts to have webOS non stuck, and same happened to the BB7 device, which I've left out of the video to avoid troubles.
IE 9 Mobile, which is different from IE9 Desktop, is still a white page. This is not a big deal, considering how poor is IE9 usage and online de-facto support anyway, but Android 2.3 is still unsupported.



I'd say good job on webOS, I can't tell the difference, but yet DOM is clearly faster on FirefoxOS ;-)
Also the bootstrap could be probably improved on reactive side, pre-compiling the benchmark, if that's an option (I don't actually know).

6 comments:

Unknown said...

@Rich_Harris here again - looks like Blogger doesn't want to recognise my Google account so I'll still show up as Unknown.

The default Ractive build supports all modern browsers. We also do a 'legacy' build which has been tested down to IE8. I've updated the Ractive DBMonster app to use the legacy build rather than the default build - I don't know for certain whether it'll now run on Android 2/Blackberry/FirefoxOS because I don't have any devices to hand, but it should be more widely compatible now.

Andrea Giammarchi said...

> I don't know for certain whether it'll now run on Android 2/Blackberry/FirefoxOS

That is exactly what I am asking in this post ;-)

I will try again and eventually update the post with this extra details.

I won't have time to re-test everything though ... but thanks for coming back on this matter.

Andrea Giammarchi said...

updated

Unknown said...

Rich here.

> pre-compiling the benchmark, if that's an option (I don't actually know)

Yes, if you have a build step you can precompile your templates. Also, with our upcoming rearchitecture, we've seen something like a 10-20x startup performance increase in similar cases - like I said on the other post, if you use a library, you get free performance upgrades every so often :)

Nolan Lawson said...

The mobile-optimized frameworks like Ionic and Famous are definitely testing on mobile. :)

I've played with Ionic a lot, on a variety of devices. I find it falls short of native performance (60fps), but it's still fairly good on a wide variety of devices.

Performance of the scrolling list on Nexus 5: http://nolanlawson.s3.amazonaws.com/www/ionic_list_perf/index.html

Performance of the side menu on Firefox Flame: https://www.youtube.com/watch?v=OxLtyG-CMEs

Still need to test out Famous, though.

BTW this is @nolanlawson; Google login seems not to work.

Andrea Giammarchi said...

AFAIK Famous was targeting iOS only, I hope they made their target wider than that.

If I remember, not long time ago, IE wasn't even mentioned.

Mobile Web is not as "small" in variety as Desktop, but everyone doing a website today should understand that testing on mobile too is mandatory.

Not just on latest-greatest phones and OSs, also on not so outdated devices.

Of course if we want to program 60FPS games, and WebGL is a target, testing on latest greatest would be just fine. But I wonder if in that case we are using the right tool for the job.

Thanks for the Ionic info, and yes, I can definitively read your name ;-)