Implement pan/zoom feature with multi-touches screen (Part 2)

In last post, we already know how to make a proper zoom internally with OpenGL. This post will help you build a natural zooming on multi touches screen.
We will first, define how zooming gesture works with some UI tests, based on that we generalize a Mathematical problem, solve it, then go to some implementation note

If you are not interested in pan/zoom, you may find some useful info about Multitouches UI Tests, and tips for handling UITouch objects.

UI Tests
Put your fingers in Maps application, then feel the map's movement when you move your fingers. It's really intuitive and natural. You can enlarge then make it small, then large again as many times as you can, you can move the map up, move it down easily with your finger.

So, how to brief this cool feeling in detail?
A natural pan/zoom function like Maps app should pass these UI tests:

Let's say we put 2 fingers on screen at A and B, we move our finger to new points on screen A' and B'. Maps application zoom/pan have these properties:
1. Zoomed with appropriate scale with respect to how opened your finger is

The image is enlarged with a scale A'B'/AB

2. In Maps, if you put your first finger on location A, another finger on location B, after preforming pan/zoom, both locations are still right under your finger, no matter how many time you zoom out, then in.

2 point O and X are still under 2 fingers after zooming

3. Hold a finger on screen, move another, the map's zoomed, 2 marks is still under your finger.

4. If 2 fingers touch the screen at different timestamp, you still can zoom.
5. Pan and zoom at the same time

6. Twist 2 finger around to form a circle, the map is not panned or zoomed

2 point O and X are still in the same place

A simple conclusion: a natural zooming will keep 2 marks stay under your fingers in most cases


Now, pan/zoom seems more clear, but how to put these tests into code. We generalize the previous conclusion into Maths:
Equivalent Mathematical problem for pan/zoom feature.

On a plan, given 2 pair of points A, B and A', B'. How to use transformations (translation, resizing, rotation) to move A to A' and B to B'?

Try to solve this yourself, or you can continue with a simple, step by step solution below:

- First. move A to A' by using a Translation with vector AA', B is also moved to B1
- Second, move B1 to B2 by using a resize with origin in A', scale A'B2/A'B1
- Finally, use a Rotation with origin A', angle (A'B2, A'B') to make vector A'B2 to same direction with A'B', B2 is moved to B'

Next step, we go to implement all of these. To do this, we need to understand how touch data is transferred from your fingers and iPhone screen into your code.


Tips for handling UITouch
1. What is UITouch object?
This except from Apple iOS Event Guideline is extremely useful:

In iPhone OS, a UITouch object represents a touch, and a UIEvent object represents an event. An event object contains all touch objects for the current multi-touch sequence and can provide touch objects specific to a view or window (see Figure 3-2). A touch object is persistent for a given finger during a sequence, and UIKit mutates it as it tracks the finger throughout it. The touch attributes that change are the phase of the touch, its location in a view, its previous location, and its timestamp. Event-handling code evaluates these attributes to determine how to respond to the event. (more discussion)

2. How UITouch objects sent through touchesBegan, touchesMoved, touchesEnded, touchesCanceled?

Sequence of passing objects during a series of events

UITouch	touchBegan	touchMoved	touchEnded	touchCanceled
Put 1 finger on screen	A
Move 1		A
Put another finger	B
Move 2 fingers		A,B
Move 2nd finger		B
Put 1st finger out of screen			A

1. Number of touches depends on event, so it is not represent how many touches are on screen
2. An UITouch object represent for a finger on screen, its data is mutable and will changed went the finger moved. You can not retain UITouch objects, thus cannot use an NSArray or NSDictionary to store touches data.
3. Number of touches go throughout touchesBegan is equal to number of touches go through touchesEnded + touchesCanceled

Pan/zoom implementation tips
- To handle how many touches are on screen, we need to remember the UITouch objects' pointers, we can use a C array of UITouch to store, and use a pointer comparison if we want to check for a pointer

Now you have all information needed for an implementation. In each UITouch delegates of your UIView, do these task:
Touches Began

• Remember UITouch objects by saving it into array

Touches Moved

• Check touches array
• If there has 1 finger on screen, preform a pan
• If there have more than or equal to 2 fingers on screen, use our transformations to make a zoom

A = [touch1 previousLocationInView:self];
A' = [touch1 locationInView:self];
B = [touch2 previousLocationInView:self];
B' = [touch2 locationInView:self];

+ Translation with vector AA': AA' = (x_A' - x_A, y_A' - y_A)
Calculate B1

+ Resize at origin A', scale k = A'B'/AB, use the function we build in previous post
- (void)zoomAtPoint:(CGPoint)point scale:(CGFloat)scale;
Calculate B2

+ Rotation:
At this point we want to make rotation to bring B2 to B'. In real, we can not rotate the maps, so we will do a simple trick to bring B2 nearer to B': we perform a pan with vector B2B'/2

Touches Ended/Touches Canceled
Remove touches out of touches array

That's it! Feel free for clarification and discussion :)

Implement pan/zoom feature with multi-touches screen (Part 1)

(Image source: top9tip.com)

When Apple rolled out it multi-touches device -  the iPhone, they also defined some useful, intuitive gestures for a certain purposes like pinch to zoom, pan to moving around in many of apps like Photos, Maps. These gestures become industry standard for their tasks and even used more and more in other apps like pinch to open/close an article in Flipboard..., pan/zoom in drawing app and game like Happy Farm...
This post will give you some ideas on how to implement your own pan/zoom.


Platform: iOS - OpenGL
OpenGL pre-configuration: Draw an off-screen texture to an onscreen frame buffer EAGLView

Texture ------> onscreen FBO ---> render buffer -> context

We can understand the task easier and get it done by simplifying it into smaller steps:

Step 1: Implement simple and acceptable Zoom/Pan function


Goal
A simple pan/zoom function, you can zoom & pan to any point of the drawing.

Technical

We define a pan by using a vector, and a zoom by a scale number. Let’s store these values in a struct:

typedef struct {
GLfloat x;
GLfloat y;
  GLfloat zoom;
} Transforms;
// Where (x, y) is panning vector, and zoom is zooming scale level.

OpenGL
You can change make our draw bigger with glScalef, and move it by a vector with glTranslatef:.
We need to modify the params of glTranslatef and glScalef when drawing view:

glTranslatef(transforms.x/screenWidth, - transforms.y/screenHeight, 0);  

glScalef(transforms.zoom, transforms.zoom, 1.0);

Multitouches
To glue these internal settings with multitouches gesture:
- In your touchesMoved:withEvent: methods, check if event touches included 1 or 2 touches
For pan: Apply for event with 1 touch (which moved from point A to point A_ in screen), get pan's vector by:

touch1 = [[touches allObjects] objectAtIndex:0];
A = [touch1 previousLocationInView:self];
A_ = [touch1 locationInView:self];
(transform.x, transform.y) = panning vector = vector AA_

For zoom: Apply for event with 2 touches
Similarly, calculating locations for touch2: B & B_

transforms.zoom = zooming scale = A_B_ / AB

Result
Now you have a good enough pan/zoom, you can zoom and go to any point of the drawing.

You also may notice that depend your OpenGL configuration (in iOS or MacOSX) then your view will be always zoomed at center or lower left corner.  A point at a corner after being zoomed will be moved out of the screen.

To make the drawing zoom at a corner:

• We first make a zoom at center of screen.
• Then drag/perform pan drawing to the corner.

Step 2: Make it better - Implement zooming at any point on drawing


Goal
Open Maps app, when you double click at any point, the map will be zoomed right at that point, even if that point is in a corner How to make a zooming like that?

We are going to build zooming function at a point:

- (void)zoomAtPoint:(CGPoint)point scale:(CGFloat)scale;

This method will modify transform setting appropriately when program receive zoom gestures
A simple thought to solve this is to utilize code in step 1, and add an appropriate pan right after a zoom.

Technical
Remember we have an off-screen texture, then we draw it on an on-screen frame buffer, again this buffer will be rendered to the view. This means we have 3 layers which affect how a drawing is displayed.

In which layer will zooming (the drawing is double in size) actually happen?
In Step 1, we know that transforms.zoom is used for glScalef, and this function will take effect when texture is drawn into onscreen frame buffer. In this process, zooming effect is actually happened


So, why do last time we always zoom on center of the screen?
That is happened when the double drawing is rendered into the view, OpenGLES 3D coordinate system has origin at the center of the screen (while in Mac OSX, this origin is in lower left corner of the screen)

Thus, after we double click on a point in screen, that point is not displayed under our finger in screen anymore, but is moved farther the center of the screen

We need to pan that point from Ao to A (vector AoA), but how to know the coordinate of Ao?
To do this, we need to go deeper into 3 layers of displaying. Let’s name some points:
• A_w, Ao_w: coordination of A and Ao in view layer (which is also A and Ao)
• A_d, Ao_d: actually coordination of A and Ao in the draw, without any affect display of zoomed/panned drawing
• A_t, Ao_t: coordination of A and Ao in the texture, zoom double affect will be performed on A_t point

Follow this sequence of transformation to know coordination of Ao
A_w
--transform/remove pan-zoom effect--> A_d
--transform/convertToGL--> A_t  
--zoom at center--> Ao_t
--transform/convert GL to view--> Ao_d
--apply pan-zoom affect--> Ao_w

Create function for each steps, take into account the difference between coordinate of each layers:
• View: origin at lower - left corner
• On-screen buffer: origin at center
• Texture: origin at top - left corner

This is one of a tough part of implementation, it requires a little knowledge about mathematic and coordinate transform. Try to have unit test and make sure each task done correctly before jump into next task. Below is instruction for first transforms: remove pan-zoom effect

//I. Remove panning
    x -= transforms.x;
    y -= transforms.y;
/*II. Remove zooming
  1. Move from bottom-left to center
    2. scale with current zoom (not new zoom caused by double click)
    3. move from center back to bottom-left */
   x = (x - screenWidth/2)/transforms.zoom + screenWidth/2;
   y = (y - screenHeight/2)/transforms.zoom + screenHeight/2;

You can find the idea for the second here: convertToGL, the forth and fifth is the opposite transform of the first and second one.
When you get all point converting done correct, perform a pan with vector AoA will bring Ao back to under your finger. Eureka!

Step 3: Make it feel cool like Maps app - Integrate multi-touches gestures: pinch to zoom
(to be continued)