My JavaScript book is out! Don't miss the opportunity to upgrade your beginner or average dev skills.
Showing posts with label Mouse. Show all posts
Showing posts with label Mouse. Show all posts

Tuesday, June 19, 2012

Dealing With Future Pointers

I have talked about this already in both JSConfEU and QCon - London and I have also blogged a couple of times about this problem that many developers keep ignoring ...

The Unexpected Input


In JSConfEU I have showed with my slides this picture:
iPad and magic mouse
which is simply the reason most of the so called mobile websites won't work as expected.
Why that? Because "ontouchend" in window, the most pointless feature detection ever, will produce a positive result and if you decide which event should be attached to the document or any DOM node relying this check, the moment the device that exposes touch events BUT has a trackpad, mouse, or any other alternative pointer device connected, it will fail, it won't work, it will look broken!

A Hybrid Solution

We could make the assumption that if the user is using fingers, the user will keep using fingers ... while if the user chooses the alternative pointer, there shouldn't be a case where fingers are again on the screen.
For this kind of situation/assumption, a lazy pointer detection is the only one we can rely.

// useless/unoptimized code - used to explain and nothing else
function initPointerEvents(e) {
// decide what should be used later on
switch (e.type) {
case "touchstart": return useTouches();
// other possible cases
default: return useMouse();
}
document.removeEventListener("touchstart", initPointerEvents, true);
document.removeEventListener("mousedown", initPointerEvents, true);
document.removeEventListener("mousemove", initPointerEvents, true);
document.removeEventListener("mousewheel", initPointerEvents, true);
}
// note the usage of true, so that we can be sure these detections are performed
// before any other listener attached with "false" ( bubbling phase )
document.addEventListener("touchstart", initPointerEvents, true);
document.addEventListener("mousedown", initPointerEvents, true);
document.addEventListener("mousemove", initPointerEvents, true);
document.addEventListener("mousewheel", initPointerEvents, true);

With above example it becomes really easy to understand what's the user choice: either touch, or keyboard.
No matter what kind of device we are dealing with, the very first event will tell us what kind/group of events we should use.

Well ... Still Not Friendly

What if the user combines touches and trackpad?
Microsoft Surface Tablet
The Microsoft Surface proposal looks great to me, but at the same time it might fail in most common mobile oriented websites.
First of all, MS has introduced kinda proprietary events to deal with pointers but I wonder how these events will behave once the user decide to switch between trackpad and screen.
I imagine that the mouse cursor will appear and disappear accordingly, at least that is what could make sense to me, so that the page could actually implement both :hover styles and still be based on touch events but the simple fact is that all this is a mess for web pages.
Here an example of how things could go wrong, assuming the classic trackpad in older laptop won't mean that msPointerEnabled is true ( being that screen not touchable )

// the MS suggested feature detection
if (window.navigator.msPointerEnabled) {
// fails when the user uses the trackpad instead of the screen
} else {
// fails when the user uses the screen instead of the trackpad
}

So, unless Windows 8 does not prefer MS events regardless the hardware, the inline feature detection will fail in this case as well.

We Need A Better Interface

This call is mainly for W3C, everything we have right now does not scale with more complex, modern, hybrid, devices.
This is also not only about mouses, trackpads, and screens, this is related to virtual keyboards too, the most annoying thing could appear and without notification/control in a webpage screen.
What we could do is to create yet another library able to deal in a totally abstract way with all these cases and in real time.
But how big this overhead would be? Think how many times we attach events to single nodes, rather than into the document only, and think how many checks we need to do in order to normalize properly and cross platform user actions.
The ideal library should be able to switch runtime and handle combination of any sort of pointer ... maybe you scroll a page with fingers, then you draw on canvas with a USB pen ... then you type on the screen, then suddenly you press enter in the keyboard ... and so on ...
Asus Transformer Prime


One Problem At The Time

There is no library out there able to truly behave as expected and switch runtime all these possibilities ... so I might decide to write one but the architecture should be both simple and able to scale.
I am not a huge fun of too abstract architectures for the simple reason that these rarely have good performances but this is a problem that later we deal with, more problems we'll have.
If you have any library I am not aware of that is that smart, please let me know and I'll update the post linking to this library.

Sunday, October 25, 2009

Android DOM "Finger" Events


This is the Android based device I am talking about and the installed software is version 1.6, apparently with multi-touch support for more recent devices so mine is not the case.

Disclaimer

The reason I am writing this post is my attempt to reproduce the Android OS interface via browser, hilariously possible with every browser except the original one: the WebKit in Android.
Why would I do that? The mouse has been around for ages, thanks Xerox, but later these years we've seen so many touch screen based devices without common mouse pattern. Since I think both Android, iPhone, and others, have done an excellent work with the way we use the device UI I thought: Why not! It could be a great time to start analysis and piece of libraries to replicate in a cross browser way some device independent event. Well, as I've already said, and as you'll see next, right now it is not possible!

Mouse Events .. ahem, Finger Events

Every mouse related event we know does not make much sense with a touchscreen, at least the tested one. Think about it: a finger points something (aka: mousedown) move somewhere (aka: mousemove) remove the finger itself (aka: mouseup). This is not what truly happens.

Mouse Events Order

  • mouseout, optionally fired when point if before we pointed another element
  • mouseover, optionally fired if the pointed event is different (or no elements have been pointed before)
  • mousemove, always before mousedown, sometimes fired twice if the internal cursor is in position 0,0 - this event is not performed while we move the finger!
  • mousedown
  • mouseup
  • click, which is so far the only event that makes sense
  • DOMActivated, a common WebKit "bug"


Mousemove Events ... Whatever ...

Fired events are device dependent, my device fires these events just in order but whatever we change in the DOM wont be called/rendered until we release. Mousemove is fired once, maximum twice, but never during movement.
In few words the only "mouse event" that makes sense is click, not performed if we scroll.

Forget The Drag And Drop!

Even if the most interesting part of the Android interface is the drag and drop, via WebKit mobile we cannot even try to emulate it just because the only event fired while we move the pointer is scroll.

Scroll ... Whatever ...

This event could be considered just a notify and nothing else. If we do whatever action while we scroll a page nothing will happen until we release the pointed element. Accordingly we cannot use the element to synchronize somehow the rendered page since again, nothing will happen during the scroll. The phone browser is basically frozen during the entire lifecycle of a gesture, or a pointer (via finger) operation.

(window || document) // it does not matter
.addEventListener(
"scroll",
(function(i){
return function scroll(e){
// it's a waste of time
// to perform whatever
// action here
// let's speed up things
// adding delay
if(i)
clearTimeout(i);
i = setTimeout(function(){
i = 0;
onScroll();

}, /* about */ 250);
};
})(0)
)
;

Above snippet represents a delayed onScroll event which aim is to be performed faster without slowing down the page while we are scrolling.
I know this could sound a nonsense but operation will be performed in any cases but never rendered so why should we let our phone compute stuff will never be showed?
Also a scroll could be not perfect, since the mouse wheel won't be there and a pointer has 10 pixels of tolerance so just wait the end of the action, and only at that point perform something.

Resize

Both window or document, no difference in this case, will always perform a resize event. This is the last reliable event we have so far in a G1 device.

Drop Down Menu Not Worth It

Common JavaScript drop down menu usually activated via mouseover and removed on mouseout won't work as expected in common touchscreens while if we implement a "clever" click manager Desktop users will be a bit disoriented 'cause we failed implementing these menus everywhere just because these are strictly device pointer dependent, so not re-adaptable/usable at all.
Moreover, a finger is not a mouse and it's more complicated to click the right menu/link at first shot. We need to rethink a little bit the UI if our aim is to make a website usable, without having 2 versions to maintain, with both desktop and mobile or touchscreen.

As Summary

We have basically 3 events, and few others, such onload, DOMContentLoaded, focus and key related for inputs, but these a part other events won't necessarily act as expected.
We need to bear in mind that mousemove or mouseover/out are quite meaningless since there is no mouse and no sensor able to monitor the finger activity when close to the screen. iPhone implemented a touch event but still, touch does not allow drag and drop so if we would like to emulate the Mobile OS interface via the Mobile Browser, we simply cannot: what a joke, but that's how it is.