Warpic: A four-finger system for designing two-dimensional image warping animations
Brian Edmonds & Jarek Rossignac, PhD
Image warping can be defined as the manipulation of portions of an image through user input. We propose a software system called Warpic that allows for the creation of two-dimensional animations from static images via user controlled image warping. The initial positions of the user's four fingers are recorded when the user places them on a touch screen device. As the user drags their fingers over the screen, Warpic computes a space warp for each frame and applies it to the image. An Android application was built in order for users to create animations using four fingers as simultaneous inputs on a mobile device. A back-end web server was implemented for users to upload and share their animations. The system provides a unique user interface that humans can easily navigate. Additionally, the Warpic software allows users to create most animations in real-time, without any time spent waiting for rendering or configuration.
Animated image warping, also known as image deformation, has widespread applications in computer graphics. Perhaps the most notable example in pop culture is Michael Jackson's famous "Black or White" music video, which employed image warping techniques to align the face of one dancer to another during their morph. In the context of this paper, image warping is defined as the manipulation of portions of an image through user input. We have designed a software system called Warpic that creates spatially-smooth warps, in real-time, for such applications as making a character appear to wink or breathe and for making moving backgrounds for movies.
Several "measures of goodness" are defined as a barometer for success. Warpic should work within the real-time constraint, meaning that all rendering, input and feedback from the software should be performed without any perceivable delay. Additionally, the software should evoke "natural" responses, meaning that the input is handled in such a way that the user feels as if he or she is manipulating real objects. Warpic's user interface must be intuitive and must have dynamic and versatile abilities, meaning that the user can specify many different animations and create the desired acceleration effects. Acceleration effects refer to the speed at which certain portions of the animation move at once.
Several problems make accomplishing all these goals challenging. Specifying user input to generate image warps in an efficient and easy manner is hard. The traditional point and click interface requires time and much trial and error from the user when specifying many different control points. Additionally, keeping a warping animation spatially smooth is a challenge. However, much progress has been made on image deformation challenges. Schaefer and McPhail developed a strategy, Moving Least Squares (MLS), for deforming two-dimensional images in 2006 . This work succeeded in generating spatially smooth warps as well as calculating resulting warps in real-time. Also, MLS is not restricted to four control points. The warped image is defined as a weighted average of the current position of the control points. The weights are computed using the distances of the optimized points to ensure that the warp interpolates the control points. Ma and Zhao expanded on this technique with their paper Moving Regularized Least Squares . The deformation scheme involves the user choosing a set of handles and destinations for these handles. The handles can be edges or points.
Several different techniques exist to generate animations. Key frame animation is used often in the animation pipeline, however the user often must specify many different frames to build a substantial animation. When using key frames, creating expressive animations with sudden accelerations or subtle movements may be tedious. Furthermore, this approach may require designing and editing a large number of key frame curves. In contrast, puppeteering a form of motion capture and popular alternative to key framing, captures detailed movements easily. Puppeteering does have two drawbacks; it does not support editing and it limits the number of parameters that can be controlled by the user.
Relevant work exists that utilizes puppeteering to control 2 and 3 dimensional virtual characters within a computer. For example, Shitori and others at Microsoft Research allow the user to move a control piece in 3 dimensions, hence creating puppeteering animations . They created a system that offers real-time feedback and provides a much quicker and flexible interface than key framing. Although 3D sensing devices make it possible to track user's hands and fingers, we focus on touch screen tablet interfaces and limit the tracking to four fingers. The user can maintain a real-time connection with their animation and have fine-grained control of animation dynamics. The captured animation replays at the speed by which the user moves his or her hands. This input scheme generates a predictable, natural and simple output.
When choosing the number of fingers to accept as input, a few previous studies helped guide our decision to use four fingers. In 1986, Buxton and Myers published a paper stating that bi-manual interfaces in computing are natural for humans . With two hands, the user can easily create symmetrical effects within his or her animations. Moscovich and Hughes studied the effects of different input mappings for multi touch inputs, noting that often a widget or component present on touchscreens is smaller than a person's actual fingers. With this in mind, we designed Warpic with a teleport interface, meaning Warpic allows the user to place fingers on the screen anywhere and avoids the necessity to "grab" specific handles on the screen.
User workflow for animation generation
Warpic does not require any complicated settings nor does it require the specification of any key frames or animations curves. The user simply starts with a static image that can be loaded from a file or taken with the tablet's camera (Figure 1). Upon choosing an image the user can begin pre-warping the image. The user places four fingers on the screen and warps the static image into a starting point for motion capture. Next the user specifies an animation thru puppeteering, a simplified version of motion capture in which the user moves four fingers on the touch screen (Figure 1). Finally, Warpic calculates the ensuing animations (Figure 1). In Warpic, the initial positions of the four fingers are recorded when the user places them on the touch screen. Then, as the user drags the fingers over the screen at each frame, Warpic computes a space warp and applies it to the image. Once the user lifts all four fingers the motion capture phase begins. The motion capture phase allows the user to record a motion path of four fingers, while showing the warping animation they are creating in real-time. Finally, once the user is done designing his or her animation, user can view the animation that he or she created on the tablet during the animation playback phase. Animation playback iterates through all the history lists to generate the warped image that the user created at each frame of the motion capture phase. The animation will play for the elapsed time of the motion capture phase.
Finger tracking algorithm
We designed a four-finger interface that has the sole responsibility of intercepting the Android motion events created by the user's fingers on the screen. Motion events are the system data that result from different user input. A challenge with this includes correctly indexing the data that is received as the user lifts, drags and touches the screen with multiple fingers simultaneously. Due to the fact that the frame rate of animation must be constant but the user creates motion events at an unpredictable rate, our interface queues these events so that they can be handled at a set point in the draw loop.
Vertex and grid formation
At every frame, Warpic defines a warp. The software creates a grid beneath the image and the vertices of this grid are displaced according to the user's input as interpreted by our four-finger interface. Performance of Warpic bottlenecks at the resolution of this grid. As the resolution of the grid increases, the ability of the tablet to maintain its real-time connection with the user falters. One can begin to see noticeable lag in the users' actions and the displayed warp when the grid is 100 vertices by 100 vertices. For testing purposes the grid resolution stays at 50 X 50.
Four fingers of user input is mapped to two edges L and R, where a user's thumb and forefinger of the left hand are points on the x,y plane LA and LB (Figure 2). Predictably, a user's thumb and forefinger from the right hand are (x,y) points referred to as RA and RA. Warpic keeps track of the positions of these edges when the user first puts his or hands on the screen as L0 and R0 (Figure 2). Warpic also keeps track of the current position of the edges at the current moment in time as Lt and Rt. Finally the image is painted over the grid using texture mapping based on the warp calculated from inputs L and R. During the motion capture phase, the user moves his or her four fingers around on the screen and at every frame, in addition to calculating a new warp, the values of LAt, LBt, RAt, RBt are added to "history" lists. The history lists represent a motion path defined by the user and are iterated over during animation playback.
Calculation of warp
Given an edge between the user's forefinger and thumb, consider the following points:
A0: the first location that the thumb touched the screen
At: the current location of the thumb
B0: the first location that the forefinger touched the screen
Bt: the current location that the forefinger touched the screen.
The software follows the below steps to calculate our warp at every frame, which essentially is a displacement of vertices (Figure 3).
1. Calculate the Spiral Angle, a, between A0B0 and AtBt
Compute the coordinates x,y of vector AtBt with respect to vector A0B0 as follows: A0B0 • AtBt
x = At Bt • A0 B0 and y = At Bt° • A0 B0° where AB denotes the vector from A to B, V denotes the normalized (unit) version of vector v, and v° denotes the version of v rotated by 90°.
a = atan2((y,x) • At Bt, A0 B0 • At Bt)
where atan2 calculates the angle, in radians, from a specified point to the coordinate origin as measured from the positive x-axis. a is a float in the range from PI to -PI.
2. Calculate the Spiral Scale, s,
s = |At Bt | / | A0 B0|
3. Calculate point C as the convergence of a family of spirals that also intersect at A0, At, B0 and Bt. The software uses the stroke motion method outlined in the Rossignac and Whited's paper, InBetweenIt .
4. Next rotate a given point P, around the spiral center C, by the calculated angle a giving us point P'.
P'= C + satCP°
P' is then linearly interpolated towards C with our calculated spiral scale, s (Figure 3). Warpic follows these steps for both hands. Displacing the vertices in the grid once for each hand, essentially blending the effects of both hands. Applying this procedure to all of the vertices in the grid will result in what we call a "global warp." In order to give the users local control of his or her deformation we introduce a Region of Influence, or ROI, and apply the effects of s and a based on decay function to the displacement of the vertices of the grid the farther they are away from the averaged center of L0 and R0.
We implemented Warpic on a Samsung Galaxy Tab III that runs Android 4.2. Our graphics platform is Processing for Android . Testing and user studies were conducted within the Graphics labs in the Georgia Tech GVU center.
When testing Warpic through user studies, we asked a user to create some basic warping animations. For example, a user was asked to create an animation of a cartoon character jumping and then landing. Additionally a user was asked to make a frog breathe. These users found the interaction to be intuitive, however, often times, the user wanted to be able to change the ROI depending on the type of animation that he or she wished to achieve. Some animations, such as wrinkling a nose, or blinking an eye, were easier with a small ROI. Other animations, such as making a character move whole parts of his body, are easier to perform with large ROI's. For this reason, we developed an interface that allows the user to have direct control of the size of the region of influence. The user is given control over a ratio (r/R) through the use of a slider widget. From this ratio, Warpic generates finger proxies that simulate fingers on the screen that are artificially larger or smaller than the user's actual fingers (Figure 4). The ratio (r/R) is set to 1 by default. A proxy point, P, is calculated for each point on both edges L and R with the formula: P = C + rF/R where C is the average point of L0 and R0. Next the software will substitute the values LAt, LBt, RAt, RBt for F and calculate a finger proxy point P for each of these.
With two hands engaged in creating animations, the tablet has the propensity to slide around on the table with nothing to steady it. The nature of the warping scheme, where bring two fingers close together causes certain parts of the image to get smaller and moving two fingers apart makes part of an image bigger, is reminiscent of the widely accepted pinch to zoom interface mechanism . This is an advantage because the model is not symmetrical. The above findings represent a prototype of an animation tool with a narrow scope. Warpic's warping effects can captivate users with its unique capabilities. During user studies, we noticed that the four fingers did have a downfall. The software maps finger placement based on the order that the user places them on the screen. This can cause confusion as to how the edges will be established.
Future plans for this project include allowing the user to "layer" warps on top of each other. This would allow the user to create effects such as moving a figures head side to side while the character is winking at the same time. Also, we would like the user to have the ability to concatenate separate warps on top of each other, perhaps by loading two different warps from a "motion path" library. Additionally, the ability to export a video file either directly from the Android application or from the replay webpage is a necessity should professionals want to use this system. Overall, Warpic allows for the creation of spatially-smooth, two-dimensional image warping animations for professional graphic design and consumer content creation applications.
- Schaefer, S., T. McPhail, and J. Warren, Image Deformation Using Moving Least Squares. ACM SIGGRAPH, 2006.
- Ma, J., J. Zhao, and J. Tian, Nonrigid Image Deformation Using Moving Regularized Least Squares. 2013.
- Shiratori, T., et al., Expressing Animated Performances through Puppeteering. 2013.
- Buxton, W. and B.A. Myers, A Study in Two-Handed Input. CHI '86 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1986: p. Pages 321-326
- Moscovich, T. and J.F. Hughes, Indirect Mappings of Multi-touch Input Using One and Two Hands. Proceedings of CHI, 2008.
- Processing for Android. Available from: http://wiki.processing.org/w/Android.
- Rossignac, J., et al., BetweenIT: An Interactive Tool for Tight Inbetweening. EUROGRAPHICS, 2010. 29.
- Incorporated, A., Basic Zooming Using the Pinch Gestures, in Scroll View Programming Guide for iOS. 2011: ios Developer Library.