Dynamic Space Management for User Interfaces

 

Blaine A Bell

 

Steven K Feiner

 

Department of Computer Science

Columbia University

 

A Program for Managing Display Screen Real Estate

 

Background

Computer display screens are a ubiquitous feature of contemporary life. Their presence is increasing rapidly as the displays on appliances, wireless phones, personal digital assistants (PDAs), web pads and the like join the already enormous population of personal computer (PC) displays. Many of these new devices will make use of smaller display screens than are typically found on PCs, although some will support higher resolution output.

 

One of the major problems already faced by PC, PDA, and wireless phone application designers is how to most advantageously manage the limited real estate on their display screens. In a society that values multitasking, display screens are often filled with a confusion of overlapping and layered windows, all opened on the screen at the same time.

 

Current window managers and similar software do not make it easy for application programs to determine how to lay out multiple screen objects effectively.  While it is easy for an application to specify a precise size or position for an object, it is not easy for an application to determine how to place or move objects such that they avoid overlapping other objects that are already being displayed. It is often difficult just to determine whether it is possible to position an object of a desired size anywhere on the screen without overlapping other objects.

 

Our Invention

This invention enables graphical applications to do a better job of managing display screen real estate. The inventors have developed an efficient approach for representing and querying the empty space on a screen so that objects can be added or moved to better meet the user's needs.

 

The invention makes possible a number of novel approaches to these screen real estate management problems. In one demonstration, the inventors' empty space representation has been applied to a conventional desktop window manager to avoid window overlap when possible. Suppose that the user drags a window and drops it such that it overlaps other windows. In this case, the invention allows the window manager to find the closest empty space to the original destination at which the dragged window can be dropped at its current size, and to avoid overlap by automatically moving the dragged window there. In a variation on this approach, the dragged window remains at its selected destination, and any windows that it overlaps are automatically moved to the nearest available empty spaces that are large enough to avoid overlap. In general, when new objects are created or old ones are moved or resized, the invention makes it easy to determine a location or size that avoids overlaps with all other objects or specific sets of objects.

 

The invention will add needed flexibility and adaptability to any application or window manager. This will be especially important in an era of new display screen-based devices.

 

FAQ

This FAQ covers notable advantages with DSM in 7 areas:

1.        MDI applications

2.        Moving beyond tiled windows

3.        Non-window environments

4.        3D environments

5.        Applications for games

6.        Value for use in Cell phones, PDAs, 2-way pagers

7.        Ads and ad revenue models

8.        Online documents and e-Books

9.        Closed-Caption encoding/viewers

 

1. What is the outstanding value of DSM for MDI (Multiple Document Interface) Applications?

 

Many commercial applications use multiple windows for displaying tools and other capabilities in addition to windows used for a work area (documents).  Typically, current available software places new windows in either a default position or in the position it was in the last time it was visible.  Our tool allows new windows to be dynamically placed in a position without interfering with the windows that are already open.  This gives the user the ability to continue working without having to rearrange the windows manually. Furthermore, our spatial representation can also be used to keep un-highlighted windows visible when the user is interactively moving highlighted windows to desired places on the screen.

 

2. Does the DSM technology differ from the X window system tiled window?

 

The functionality of DSM gives a normal window manager some functionality that a tiled window has (namely non-overlapping windows) without the drawbacks of a tiled window manager.

 

The DSM space manager does not replace or introduce a new window manager.  The spatial representation that DSM introduces can be used (but not limited to) certain functions of a window manager.  For example, one function that it can accomplish is given a certain number of possibly overlapping windows, a new window can be placed in a position where it does not overlap any other windows.

 

 

3. Does the DSM technology only apply to Windows Environments?

 

The spatial representation that is introduced does not only apply to windows environments.  This representation is generic to computing empty space given used space, and can be used within applications for introducing information using tool tips, billboards, application level windows, and 2D or 3D interfaces in general.

 

 

4. Can the DSM technology apply to Virtual 3D Environments?

 

DSM also uses the spatial representation to introduce information into a virtual 3D scene by projecting the 3D objects to the 2D screen.  This allows us to introduce information to a user with a certain perspective without occluding any of the objects in the scene.  We have also extended the spatial representation to represent the 3D space which gives the ability to introduce new objects in the 3D world without overlapping other objects.

 

 5. What key opportunities does DSM offer for games?

 

The introduction of information during a game such as clues and other facts may be presented without taking away the user's ability to visualize the game's current scene.  These clues and other facts can be placed on the screen so they can directly reference objects in the scene without occluding other important objects that might take away from the clues, facts, or the user's experience.

 

6. Does the DSM technology apply to Cell Phones, PDAs and 2-Way Pagers?

 

The growing opportunity to display and manipulate graphic information on smaller screens is an opportunity to take advantage of simultaneous displaying, without interference, a variety of graphic and textual information.

 

7. What is the value of DSM advertising placement and revenue models?

 

Advertising used in graphical interfaces typically use-reserved spaces on the screen.  For example, banner adds on the Internet are placed in a position on the web page, and many free ISPs use a reserved space either on the top or bottom of the screen.  Our spatial representation allows for a more adaptive placement of advertisements by placing them in positions on the screen that aren't being used.  Since applications can use space on the screen dynamically, so should the advertisements.  By knowing all of the space that is not used, more advertisements can be displayed in these places.  For ISPs, all of the space on the screen that is not used by a window can be used for advertisement.  Thus, since size and placement should compose the advertisement cost, more revenue can be generated.

 

8. How does DSM apply to online documents and e-Books?

 

Online document viewing applications give users functionality that they wouldn’t experience reading a book or magazine.  For example, if the user comes across a word that he or she doesn’t understand, the definition could pop up on the user’s request.  In current applications, the placement of the popup information is relative to the cursor or word.  Our spatial representation gives the application the ability to dynamically place the pop up information where it doesn’t obstruct the user from seeing the rest of the document.  This provides the user with a better visual experience and continuity.  It also allows the application to pop up multiple things because it will not obstruct the user from continuing to read.  This idea can also be applied to advertising (as described above in #7) and electronic sticky notes.

 

9. How does DSM apply to closed-caption encoding/decoding?

 

Currently, the placement of closed-captions on video is usually done in the same place on the screen.  Sometimes, short phrases are manually placed closer to an actor to convey who is saying the caption, although this can only be done on pre-recorded video because of the work that is involved.  New video encodings, such as MPEG-4, have information about objects that are segmented in the video stream.  Our space representation would allow the closed-caption systems to dynamically place closed-caption text in places on the screen so that it wouldn’t cover important objects to the viewer.  For example, during football games or news programs, statistics are projected.  In the current systems, there is a good chance that the closed-caption viewers will not see the statistics.  By tracking the screen placement of the statistical information, our spatial representation could place the closed-caption text so the closed-caption viewers would be able to see the statistics and the text at the same time.

 

Video

An online demo of the screen manager can be accessed:

 

Internet Explorer version 5.5 or above and Windows Media Player version 6.4 or above is required

Note: If you are using a connection speed less than the desired speed, please download the file to disk and play it locally.  Otherwise, buffering will happen too frequently.

High Quality (approximately 11Megs, made for connections 384k and greater)

Medium Quality (approximately 3Megs made for connections 128k and greater)

 

Demonstration

We have put together some web-based examples to show how our space manager can be used on the Internet.  The main benefit to using this technique is that no matter what size the scrollable, resizable browser window is on the user’s screen; the images (or of course more text) can be displayed without occluding anything else on the web page.  In each of these examples, we use the same logic to present the new images.  Although this is not the way typical advertising should be done, (i.e. popping up and fading away might be considered really annoying to the user!) it shows how our representation can use all the space that isn’t used by the initial web page setup.  We apply it to this page, a sample yahoo page.

 

Note: Our representation does not limit you from choosing which objects the new image cannot overlap.  In these examples, we choose the objects by opinion (notice in the yahoo demo we allow the images to overlap the colored places on the screen).  This is strictly a design choice, and gives developers/designers flexibility to use the representation however they wish.

 

Tech Note: Internet Explorer version 5.5 is needed to run these demos. IE 5.0 will not support resizing the window. Also, our algorithm traverses the HTML and calls a IE specific function “getClientRects()” for each object on the page.  This function is not well defined in documentation, and behaves differently for many different HTML objects.  For example, it does not return the bounds for radio buttons, and it is very strange when working with tables and the elements inside the tables.  This is currently under investigation, as well as supporting Netscape. But we hope you get the point! J

 

Download Java Libraries

Space Management is now available for non-commercial use. In order to download these libraries, you need to read the license agreement, and fill out some information.  In turn, we will email you the jar file that includes the source, compiled code, and can be used to generate the javadoc.

 

Contact Information

If you are interested or have more questions you can contact me at blaine@cs.columbia.edu or Steve Feiner at feiner@cs.columbia.edu, or if you wish to get more information on licensing, please contact:

Tony Mauriello

Columbia Innovation Enterprise, New York, New York 10027

Phone: 212-854-8444

Email: Rockbuilt@aol.com

 

Last Updated: November 26, 2001 by Blaine Bell