Data Science
Interactive Analysis of Hierarchical Image Segmentation NOTE: This paper was published in the Proceedings of the 2000 International Geoscience and Remote Sensing Symposium (IGARSS '00), Honolulu, HI, July 24-28, 2000.
 

Interactive Analysis of Hierarchical Image Segmentation

James C. Tilton
NASA's Goddard Space Flight Center
Mail Code 935, Greenbelt, Maryland 20771 USA
Telephone: (301) 286-9510
Facsimile: (301) 286-1776
E-mail: James.C.Tilton.1@gsfc.nasa.gov

William T. Lawrence
Natural Sciences, Bowie State University
Bowie, Maryland 20715 USA
Telephone: (301) 464-6121
Facsimile: (301) 809-4066
E-mail: blawrenc@cs.bowiestate.edu


At IGARSS'98 we described a hybrid image segmentation approach based on region growing and spectral clustering [1]. This approach produces a hierarchically related set of image segmentations in which the regions may consist of non-spatially connected objects. Here we describe a "Region Labeling Tool" that can be used produce customized image labelings from the set of hierarchical segmentations. We demonstrate the Region Labeling Tool in a land cover analysis of a Landsat TM data set from over the Baltimore, MD / Washington, DC area. We have seen that a key benefit of using the Region Labeling Tool with hierarchical segmentations is that a much fuller utilization of the spectral and spatial information content of imagery is obtained than possible with conventional pixel-oriented analysis techniques.

BACKGROUND

Image segmentation is a partitioning of an image into related sections or regions. These regions may be later associated with ground cover or land use class, but the segmentation process simply gives generic labels (region 1, region 2, etc.) to each region. For multispectral or hyperspectral remotely sensing imagery data, the regions consist of groupings of image pixels that have similar data feature values. These data feature values may be the multispectral or hyperspectral data values themselves and/or they may be derived features such as band ratios or textural features.

Image segmentation is a key first step in a number of approaches to image analysis. The group of pixels contained in each region provides a good statistical sampling of data values for more reliable labeling based on multispectral or hyperspectral feature values. In addition, the region shape can be analyzed as an additional clue for the appropriate labeling of the region. Labeling based on image region features rather than individual image pixels often gives a more satisfactory analysis result because spatial information is taken into account. Also, region-based analysis can be quicker than pixel-based analysis, since usually many fewer regions than pixels need to be considered in the labeling step.

For a general discussion of image segmentation approaches see [2]. We prefer the region growing approach for application to remotely sensed imagery data because it exploits spatial information and guarantees the formation of closed connected regions. However, region growing is not without its problems.

With straightforward region growing, spectrally similar but spatially disjoint regions are never associated together, thus complicating their identification. Also, it is often not clear at what point the region growing process should be terminated, resulting in under- or over-segmentation. In addition, region growing tends to be a computationally intensive process.

A hybrid region growing and spectral clustering approach, HSEG, has been developed to overcome these problems. The hybridization with spectral clustering allows association of spectrally similar but spatially disjoint regions. The approach also includes the detection of natural convergence points. The region label maps produced at each natural convergence point are combined to form the hierarchical image segmentation product. Storing the region label map only for the most detailed segmentation, and storing relabeling lists for the subsequent levels of the segmentation hierarchy substantially reduces disk space requirements. A recursive implementation of this approach reduces the computational requirement [3]. This recursive implementation can be very effectively programmed on parallel computers, greatly reducing the amount of time required to segment large images with this approach [4].

Described in the following section is an interactive tool for selecting an application specific segmentation from the segmentation hierarchy produced by the HSEG algorithm. We anticipate future development of approaches for the automatic selection of such segmentations from the segmentation hierarchy.

THE REGION LABELING TOOL

A "Region Labeling Tool" has been developed to facilitate the selection and identification of a particular application specific segmentation from the hierarchical set of segmentations produced by HSEG algorithm briefly described in the proceeding section. With this tool an analyst selects a particular data point (based on an appropriate graphical representation of the original data), and all other data points in the same region in the segmentation (either at the finest level or coarsest level of detail) are highlighted. The analyst then can choose to adjust the level of segmentation detail by selecting other levels from the segmentation hierarchy. "Region of Interest" and other tools are available to adjust the highlighted areas. Once the analyst is satisfied with the areas highlighted, he/she can label the segment with a particular alphanumeric phrase and color. This process can be continued until the whole data set is labeled. Other features of the Region Labeling Tool allow for a refinement of this process.

Due to the graphical nature of the Region Labeling Tool, it is difficult to adequately describe it in a three page proceedings article with no color pictures. A much fuller description of the Region Labeling Tool can be found in [5]. (Note that a version of this reference is available via the World Wide Web.)

Using the Region Labeling Tool - We describe the use of the Region Labeling tool through a narrative description of the labeling of a hierarchical set of image segmentations produced by processing a portion of a Landsat TM data set with the HSEG algorithm. The Landsat TM data set used in this example is a six-band 2504-by-2504 pixel section taken over the Washington, DC / Baltimore, MD area on September 16, 1991 (WRS II path/row 15/33). The thermal band was not used.

Upon initiation, the Region Labeling Tool displays a GUI panel. Upon left mouse clicking on a button labeled "RGB Image", an RGB image can be displayed. This image is used for reference so that the analyst can move about in the image, selecting, identifying and labeling known land cover types within the data set. The "RGB Image" can be any band combination or enhancement of the original image deemed useful for labeling purposes. Up to three of these "reference" images can be included to facilitate labeling as long as they share the same spatial extent and resolution as the input image data. Other data could be topographic, land cover or other thematic maps, as well as previously labeled results. The additional reference data can be selected for use by clicking on the "1st Reference Class" or "2nd Reference Class" buttons on the GUI panel. All images are displayed at full resolution. For images that are greater than the extent of the display, a reduced resolution "Pan" window is used for roaming within the larger-than-screen full data set.

Once an initial labeling area is displayed from suitable reference data, the labeling process can begin. For the purpose of description, assume we pan the image so that it is centered to the southeast of Baltimore, MD, over the west central part of the Chesapeake Bay.

Navigating the Hierarchical Segmentation - In the case of the example image, the predominant class is water; specifically the Chesapeake Bay. This being the case, we start our example by labeling the water class. First we select a single pixel located in a spectrally uniform area of the Chesapeake Bay on the "RGB Image" by clicking the left mouse button at this point in the Chesapeake Bay, and then clicking on the "Select Region" button in the upper left corner of the GUI panel. Next, with a click on the "Current Class Labels" button on the GUI panel we see a display in which all other pixels in the same region in the current level in the segmentation hierarchy are highlighted in purple. By default, this is the finest segmentation from the segmentation hierarchy containing the pixel selected. In this example, most of the Chesapeake Bay is highlighted in purple.

At this point, we have two choices. If we are satisfied with the current selection, we can label the currently highlighted region. If not satisfied, we can refine the region further by looking at other segmentation levels within the segmentation hierarchy. A good way to make this decision is to inspect the spatial extent of the highlighted region in comparison to the "RGB Image" rendition of the image or any other available reference images. Whether or not more "water" should be included with the currently highlighted area depends on our application requirements. In our case, we are not interested in differentiating between water classes by such criterion as sediment load, so we choose to include more "water" regions together with the currently highlighted area. This can be easily done by moving up the segmentation hierarchy to find coarser levels of segmentation that included the initially selected pixel.

The simplest way to highlight the next coarsest segmentation from the segmentation hierarchy is to click the left mouse button on the button labeled "Select Next Coarser Segmentation". In doing so, we see (through inspection) that the highlighting of the Chesapeake Bay is nearly complete, and that some smaller water bodies are also included. We must now decide to either label the currently highlighted region or to look at even coarser segmentations from the segmentation hierarchy so as to capture any remaining unhighlighted water segments. We are aided in making this decision by the information displayed in "Region Label Tool Informational Output" text output area in the bottom portion of the Region Labeling Tool GUI. We are currently viewing level 1 of the segmentation hierarchy (level 0 is the finest and level 15 is the coarsest). The informational output shows that there are 972,833 pixels in the level 1 region. This increases only slightly to 974,377 pixels in the level 14 region. This being the case, we decide to look at the level 14 segmentation from the segmentation hierarchy. The easiest way to do this is to enter the number 14 (followed by the "Enter" key) in the space after the label "Segmentation Level".

We pause here to comment on the usefulness of the information provided in the "Region Label Tool Informa­tional Output" text output area. Significant changes in the number of pixels, region criterion value (a measure of the difference of the region mean from the original image values), and/or the region mean vector are indications of significant changes in the entities that the region is representing. For example, the combination of two distinct ground cover types, such as concrete and forest, would cause significant changes in the criterion value and region mean vector (assuming that roughly equal numbers of each class are combined).

Now back to the labeling problem at hand. An inspection of the highlighted region shows that the level 14 segmentation from the segmentation hierarchy not only highlights the Chesapeake Bay, but also most bodies of water throughout the whole image - including small lakes and thin stretches of river. However, going to level 15 highlights nearly the whole image, mixing land and water.

Labeling a Class - We now decide to label the highlighted area from the level 14 segmentation as it encompasses all expected areas except for the margins of water bodies, which are probably mixed water and vegetation regions. This final labeling is initiated by clicking the left mouse button on the button labeled "Label Region". When this is done, the Label Region panel appears.

The Label Region panel contains a set of 39 preprogrammed colors for use in color labeling regions. Associated with each of these colors is a text window, which is initialized as "(undefined)". We can choose any of the preprogrammed colors to label a particular region and associate any alphanumeric phrase with the region color. In addition, we can optionally adjust any region color through the explicit specification of red, green and blue intensity values. In this example, however, we simply choose the preprogrammed blue color for labeling the highlighted area. In addition, we associate the phrase "Water" with this region. The area highlighted in purple is recolored to blue when we click the left mouse button on the "Label" button to the left of the blue square.

Other Region Labeling Tool Functions - Comparing the RGB image and the Current Class Labeling, we note that the Chesapeake Bay Bridge is visible in the RGB image, but has been largely lost in the Current Class Labeling. We are assisted in the observation by the Zoom panels obtained by left mouse button clicking on the Zoom buttons under the RGB image and Current Class Labeling buttons.

If the hierarchical segmentation separates the Chesapeake Bay Bridge from the bay, we can distinguished the bridge from the bay in the labeling with the following approach. We select a pixel that is labeled as "Water" (blue) in the Current Class Labeling, but that appears to on the bridge in the RGB Image display, and click on the "Select Region" button on the Region Label Tool GUI panel. In the resulting Current Class Labeling Zoom panel we can see that the Chesapeake Bay Bridge is nicely highlighted, along with a very small number of other "water edge" pixels (this is the case for a small number of additional pixels throughout the image). We decide to label the highlighted pixels with a brown color and the alphanumeric phrase "Bridges/Water Edge".

Labeling More Classes - Now that the dominant water related classes are labeled, other land cover classes can be labeled in a similar manner. In this particular scene we may want to label vegetation classes such as "Evergreen Forest", "Deciduous Forest", "Mixed Evergreen/Deciduous", "Shrubland", "Cropland", etc. Other non-vegetation classes might be "Urban/Industrial", and "Residential."

Saving the Work So Far - To save our work we simply exit the program by clicking on the "Exit" button in the upper right corner of the Region Label Tool GUI panel. The latest labeling is stored in the "label_out" file specified when the program was initiated. The color and alphanumeric labels are stored in the "ascii_out" file.

Later, when we would like to restart the labeling process, we can copy the "ascii_out" file to the "ascii_in" file and copy the "label_out" file to the "label_in" file, and restart the program with all the previous labeling work intact.

COMMENTS

A key feature of the HSEG algorithm and the Region Labeling Tool is the capability to choose any level of segmentation within the data set and tailor the fineness of the segmentation to the user's needs. As an example, an ecologist might choose to label five or six specific vegetation classes and a single urban class, while an urban planner could use the same data and label the vegetation as a single class while greatly expanding the land cover classes labeled within the urban category. This can all be done with the Region Labeling Tool by particular selections of segmentation from the same segmentation hierarchy. A user is not forced to accept the segmentation produced by any particular level of the segmentation hierarchy, but he/she can choose the level appropriate for the to the class being label. In this way the Region Labeling Tool puts tremendous flexibility in the hands of the analyst.

The Region Labeling Tool is intuitive to use and greatly speeds the selection and labeling of classes from the segmentation hierarchy produced by running the HSEG algorithm on imagery data. For example, a typical Landsat MSS scene can be labeled very satisfactorily in less than an hour. Without the Region Labeling Tool it would be nearly impossible to work with the results of the hierarchical segmentation algorithm, and this valuable analysis tool would be inaccessible to most users.

REFERENCES

  1. James C. Tilton, "Image segmentation by region growing and spectral clustering with a natural convergence criterion," Proc. of the 1998 Intern'l Geoscience and Remote Sensing Symp., Seattle, WA, July 6-10, 1998.
  2. Robert M. Haralick and Linda G. Shapiro, "Image segmentation techniques," Computer Vision, Graphics and Image Processing," Vol. 29, pp. 100-132, 1985.
  3. James C. Tilton, "Method for recursive hierarchical segmentation by region growing and spectral clustering with a natural convergence criterion," Disclosure of Invention and New Technology: NASA Case Number GSC 14,328-1, February 28, 2000. See also http://code935.gsfc.nasa.gov/code935/tilton.
  4. James C. Tilton, "Method for recursive implementation of hierarchical segmentation on parallel computers," Disclosure of Invention and New Technology: NASA Case Number GSC 14,305-1, February 22, 2000.
  5. James C. Tilton, "A region labeling tool for use with hierarchical segmentation," Disc. of Invention and New Technology: NASA Case No. GSC 14,331-1, February 29, 2000. See also http://code935.gsfc.nasa.gov/code935/tilton.