Connected Filtering on Tree-Based Shape-Spaces

Connected filters are well-known for their good contour preservation property. A popular implementation strategy relies on tree-based image representations: for example, one can compute an attribute characterizing the connected component represented by each node of the tree and keep only the nodes for which the attribute is sufficiently high. This operation can be seen as a thresholding of the tree, seen as a graph whose nodes are weighted by the attribute. Rather than being satisfied with a mere thresholding, we propose to expand on this idea, and to apply connected filters on this latest graph. Consequently, the filtering is performed not in the space of the image, but in the space of shapes built from the image. Such a processing of shape-space filtering is a generalization of the existing tree-based connected operators. Indeed, the framework includes the classical existing connected operators by attributes. It also allows us to propose a class of novel connected operators from the leveling family, based on non-increasing attributes. Finally, we also propose a new class of connected operators that we call morphological shapings. Some illustrations and quantitative evaluations demonstrate the usefulness and robustness of the proposed shape-space filters.


INTRODUCTION
M ATHEMATICAL morphology, as originally devel- oped by Matheron and Serra [1], proposes a set of morphological operators based on structuring elements.Later, Salembier and Serra [2], followed by Breen and Jones [3], proposed morphological operators based on attributes, rather than on elements.Such operators, also known as attribute filters or connected filters, have been popularized notably by Salembier, Wilkinson, and Ouzounis [4,5,6,7,8].One popular implementation of such operators relies on transforming an image into an equivalent representation, namely a tree of components of the level sets of the image.Such trees are equivalent to the original image in the sense that the image can be reconstructed from its associated tree.Filtering then involves the design of a shape attribute that weighs how much a node of the tree fits a given shape.This treebased implementation is depicted by the black path in Fig. 1.
Several approaches for filtering a tree of components (and hence the image) have been proposed.The more evolved approach consists in pruning the tree by removing some entire branches of the tree, an operation particurlary relevant if the attribute is increasing on the tree (i.e., if the attribute is always higher for the ancestors of a node).However, most shape attributes are not increasing.In this case, three pruning strategies have been Fig. 1: Classical connected operators (black path) and the proposed shape-space filtering scheme (black+red path).
proposed (Min, Max, Viterbi; see Section 4.1 and [9,4] for more details).They all choose a particular node on which to make a decision, and remove the subtree rooted at that node.Though it may give interesting results in some cases, it does not take into account the possibility that several relevant objects can have some inclusion relationship, meaning that they are on the same branch of the tree (e.g., a ring object in a tree of shapes [10], see Fig. 6 (a)).
For non-increasing attributes, another commonly used approach is to simply remove the nodes of the tree for which the attribute is lower than a given threshold [9,11,4].It is however often impossible to retrieve all expected objects with one unique (global) threshold.Fig. 2 shows the evolution of a shape attribute, the circularity [12], along two branches of the tree of shapes.The light round shape and the dark one are both meaningful round objects when compared to their context.However, their attribute values are very different.In order to obtain the light object, a high threshold is required, but then some non-desired shapes appear in the background in Fig. 2 (f).
This classical thresholding process can be seen as a simple filtering of the tree.In particular, it does not take into account the intrinsic parenthood relationship of the tree.The founding idea of this paper is to apply connected filters in the space of all the components of the image, such space being structured into a graph by the parenthood relationship (i.e., the neighbors of a node are its children and its parent).This process is illustrated by the black+red path in Fig. 1.This surprising and simple idea has several deep consequences that were first exposed in a preliminary version of this study [13], where it is shown that this framework encompasses some usual attribute filtering operators.Novel connected filters based on non-increasing criteria can also be proposed.When the first tree T is respectively a Min-tree or a Max-tree [9], such filters are novel morphological lower or upper levelings [14,15] based on shapes.When the first tree T is a tree of shapes [10], it gives birth to a novel family of connected filters that we call morphological shapings.
The main contributions of the present paper are: 1) an extensive discussion of the theory behind the framework of shape-space filtering; 2) the study of some theoretical properties of shape-space filters (i.e., novel connected operators provided by this framework), morphological shapings, in particular, are also studied; and 3) some filtering illustrations allowig a qualitative comparison to classical thresholding-based methods, as well as an illustration of shape-based upper levelings to retinal blood vessel segmentation.
The rest of the paper is organized as follows.In Section 2 we briefly review a number of previous developments on which our methodology is based: connected operators and attribute filters.The theory of the frame-work of shape-space filtering that we propose is detailed in Section 3. Section 4 is dedicated to demonstrate that this framework encompasses some classical attribute filters.Theoretical properties of various novel shape-space filters are explained in Section 5. Section 6 gives a set of illustrative experimental results using those shape-space filters.Finally we discuss and conclude in Section 7.

BACKGROUND
A discrete image is defined on a domain which can be seen as an undirected graph represented by a pair G = (V, E), where V is the finite set of vertices and E ✓ (V ⇥ V ) is the set of edges.Each vertex v 2 V represents a pixel or a voxel of the image domain, and each edge e 2 E models the neighborhood relationship (classically, 4-or 8-connectivity for 2D images, and 6-or 26-connectivity in 3D images) between the two vertices composing e.
A graph (V, E) is said to be connected if, for any x, y 2 V , there exists a path from x to y, which is a sequence of n 1 vertices (x 0 = x, x 1 , . . ., x n = y) such that every x i 2 V , and every (x i , x i+1 ) 2 E. In this paper, the image domain V is connected.
A binary image X is a subset of the image domain that induces a subgraph (V X , E X ), such that V X is the set of vertices representing the set of points of X, and a binary image X is a connected subset of X with the maximal extent.This means for any C 0 such that C ✓ C 0 ✓ X, if C 0 is connected, then we have C 0 = C.More details about the notion of connectivity can be found in [16,17,7,8].
Let x 2 V .In the following, x denotes the operator that associates with a binary image X the connected component of X containing x if x 2 V X , otherwise ;.Let be the operator that associates the set of connected components with a binary image X. Formally, (X) denoted also by ( In the sequel, we denote by F the set of mappings V !R or Z.A grayscale image f is then an element of F. Given a graph G = (V, E), a pair (G, f ) also denoted in the following by (V, E, f ) is called a node-weighted graph, and it models an image f .In this paper, as a convention, operators on binary images are denoted with a capital Greek letter; the corresponding lowercase letter is used to denote the grayscale version.

Connected operators
In this section we briefly review the connected operators.More details can be found in [18,2,9,4].Generally speaking, such operators deal with a set of connected components C ✓ P(V ) of an image, where P denotes the set of all subsets of V .The connected operators act by filtering out some elements in C and/or by changing their associated gray levels.
For a binary image X, the set C is composed of two types of connected components: those in the foreground (X) and those in the background (X c ), where X c is the complement of X.We have V X c = V \V X .A connected operator working on X simply preserves or removes te connected components of X.This implies that the difference between X and (X) is exclusively composed of the union of connected components of X and/or of its complement X c .
For a grayscale image f , there are many possibilities for the set C. The most trivial one is the set of elemental connected components called flat zones [2].Let L h (V, f ), also denoted simply by L h (f ), be the set of points having a gray level h: L ) is the flat zone containing that point.The set of flat zones denoted by C FZ of a grayscale image f is given by where FZ(V, E, f ), also denoted simply by FZ(f ), is the operator that gives the set of flat zones of a grayscale image f .Such a set of flat zones C FZ forms a partition of the image domain V called partition of flat zones [2].The general definition of connected operators for grayscale images is based on this notion of partition of flat zones.An operator on a grayscale image f is a connected operator if the partition of flat zones of (f ) is coarser than the one of f , which means that ).The flat zones of (f ) are created by merging flat zones of f , implying that connected operators will not split the flat zones of f .Hence, connected operators do not introduce any new contour, and perfectly keep the location and shape of contours in the input image.
In practice, the flat zones in C FZ are usually individual points or small in size.Hence the decision whether or not to remove a connected components might be difficult to take.For that reason, other methods to create the set C of a grayscale image have been proposed in the literature.A popular strategy is based on two types of threshold decomposition: upper level sets and lower level sets.For a given threshold value h, the upper level set X h (V, f ) or simply X h (f ) and the lower level set X h (V, f ) or simply X h (f ) are defined respectively by: ( This leads to two different sets of connected components.They are given by the connected components of all possible upper (resp.lower) level sets denoted by C > (resp.C < ): A third set of connected components is given by the fusion of C > and C < through the notion of shapes [10] (see [19,20] for more details), where a shape is a connected component of an upper or lower level set with all cavities (i.e., connected components of its complement) filled in.We denote this third type of set of connected components by C < > , it is given by: where Sat is the saturation operator that acts by filling the cavities (connected components of the background).

Attribute filters
Attribute filters [3,9,21] deal with connected components instead of individual points (as in the case of classical morphological operators originally developed by Matheron and Serra [1]).They act by preserving or by removing the connected components based on some attribute criterion.We denote by A the set of mappings (C, F) !R or Z.An attribute function is then an element of A that measures some interesting feature A(C, f ) on each connected component C. When no confusion can occur, we use the notation A(C).In this paper, we distinguish the following attribute functions: • An attribute function A can be as simple as the gray level at which a connected component is obtained.
We denote this specific function by F, and we term is the smallest (resp.largest) gray level inside each connected component C.
• An attribute function A can also be more complicated, for example it can be some shape information measurement, such as circularity, compactness, or elongation [5].Note that these attributes do not depend on the gray levels of the points inside the connected components, which means for any connected component C 2 C, and any two different mapping functions f 1 2 F and f 2 2 F, f 1 6 = f 2 , we always have A(C, f 1 ) = A(C, f 2 ).We call such attribute functions shape attributes.• Some attribute functions based on the gray levels of points inside the connected components might also be interesting, such as the height or some other attributes [12].For a given connected component Let T be an attribute criterion: C ! {true, false}.Typically, T is given by the comparison of the attribute function A to a given threshold : ).Then the trivial attribute filter T on a connected component C returns the connected component C itself if T (C) is true, and ; otherwise, with the convention that T (;) = ;.The binary anti-extensive attribute filter T for a binary image X is defined by T (X) = S x2X T ( x (X)).Recall that an operator on a binary image X is said to be anti T is then an attribute opening, otherwise T is an attribute thinning [3].
The extension of the attribute filters to grayscale images is based on the set of connected components C obtained from an image.The attribute filters on a grayscale image f consists in preserving or removing T T > given by Eq. ( 3), the grayscale anti-extensive attribute filter T for a given image f at any point x is defined by Recall that an operator on a grayscale image f is said to be antiextensive if (f )  f .
The extensive attribute filters for binary images and grayscale images can be easily defined by the duality relationship with the anti-extensive versions defined above.

SHAPE-SPACE FILTERING
In the literature connected operators and attribute filters have generally been applied to the node-weighted graph (V, E, f ) or shortly (G, f ) of the image with G = (V, E).They deal with the connected components of (G, f ) and act by preserving or removing some of them in the set of connected components C. We begin with studying some basic features of this set of connected components C in Section 3.1.This leads to the introduction of the shapespace, a graph representation G C in Section 3.2.Then we present in Section 3.3 the principle of the framework of shape-space filtering, which is, in short, applying the connected operators and attribute filters to the nodeweighted graph (G C , A).Finally, in Section 3.4, we detail how to use this framework efficiently, relying on treebased image representations (as depicted in Fig. 1).A synthetic example of the scheme of shape-space filtering based on tree representation is given in Fig. 3.

Any two different elements
h2 and X h1 ✓ X h2 .Note also that the data of the set of connected components C > or C < and the associated level function F are sufficient to reconstruct the image thanks to the threshold superpositin principle [22]: Let us denote this reconstruction operator by C 1 .More generally, for any set of connected components C obtained from some image f (whether it is C > , C < , or C < > ) and its associated level function F, the image f can be reconstructed by: where Thanks to this reconstruction principle, the set of connected components C is an equivalent representation of the input image.
Classical connected operators and attribute filters remove connected components from the set C (e.g., circles on the top-middle of Fig. 3), yielding a subset of connected components C 0 ✓ C.Then, a filtered image f 0 is obtained from C 0 based on the reconstruction principle in Eq. ( 6).Note that a slightly modified reconstruction rule has also been proposed in [11], see Section 4.1 for more details.
Such a classical filtering process is easy to apply if the attribute function is an increasing attribute j is also preserved.For a nonincreasing attribute A, several filtering rules have been proposed in the literature [4].The simplest one among them is the thresholding-based filtering, where each connected component C 2 C is analyzed individually.We note that the nesting property (represented by the lines linking any pair of circles on the top-middle of Fig. 3) of the family of connected components in C is not taken into account in such a thresholding.

Shape-space
For any C 2 C, we denote by Pr (C) the smallest element of C containing C and different from C: We observe that for any C 2 C, with C different from the image domain, Pr (C) 6 = ;.We also denote by Pr 1 (C) the set of largest connected components contained in C and different from C, given by: Note that Pr 1 (C) might be empty.These two operators Pr and Pr 1 encode an adjacency relationship between the set of connected components in C. We have Consequently, we can structure the family of connected components of C into a graph representation denoted by G C = (C, E C ), where E C is given by Pr and Pr 1 : We call this graph representation G C the shape-space.As compared to the image space, which is also modeled by a graph G = (V, E), each node in G C is a connected component of the image space.And each edge in G C encodes the inclusion relationship of the connected components in C. For example, the structure on the top-middle of Fig. 3 is a shape-space, where nodes are represented by circles, and edges by lines.This shape-space has several interesting features.Firstly, it is invariant to contrast changes, that are increasing functions g applied to the image f , and covariant to continuous (topological) transformations.Besides, such a shape-space inherently embeds a morphological scalespace (the size of Pr (C) 2 C is always larger than the size of C 2 C).Furthermore, this shape-space is an equivalent representation of an image, in the sense that the image can be reconstructed from the shapespace (thanks to Eq. ( 6) if F is given).This shapespace also has another interesting property: contrary to scale-spaces, the contours of a given shape (topological boundaries of the connected components) correspond to actual contours in the image, without any blurring due to convolution with a kernel in the case of classical scale-space.Finally, this shape-space satisfies the principle of causality which is certainly the most fundamental principle of multi-scale analysis [23].From this principle, for any pair of scales 2 > 1 , the "structures" found at scale 2 should find a "cause" at scale 1 .Indeed, each connected component C 2 C can be seen as a kind of "cause" of the connected component Pr (C) 2 C.
As an illustration of the usefulness of shape-spaces, we would like to mention two proposals of local feature detectors.The first proposal is the Maximally Stable Extremal Regions (MSER) [24,25].Note that MSER, broadly speaking, are equivalent to the Maximal Meaningful Lines previously proposed by Desloneux et al [26].Indeed, a small variation in the components area occurs because of strong gradients in the image, which is the measure that is used in Desolneux et al.'s work to characterize meaningful level lines.The second proposal is the recent Tree-Based Morse Regions (TBMR) [27]; TBMR are truly invariant to contrast changes, a property which, combined with the other properties of shapespaces, allows us to obtain state-of-the-art results in image registration and in multi-view 3D reconstruction.

Principle of shape-space filtering
For each element in the shape-space, the attribute function A weighs how much that element fits a given criterion.The pair (G C , A) is a node-weighted graph.In the classical use of attribute filters as described in Section 2.2, the filtering decision for each connected component C 2 C is usually made upon the comparison of its attribute function A(C) to a given threshold value .
This thresholding process can be seen as a simple filtering of the graph (G C , A).However, many sophisticated signal/image processing filters exist.Besides, it is wellknown that a pertinent result with a unique thresholding value is usually difficult to obtain.Instead of just thresholding the graph (G C , A), we propose to apply some signal/image filtering tools to the graph (G C , A).Indeed, graphs are generic data structures that have been widely used in many scientific and engineering fields, notably for image analysis and computer vision [28,29].Since they are very flexible representations, a current trend is to use the classical tools for signal/image processing also for graphs [30].Specifically, in this paper, we apply some connected operators to the graph (G C , A).This process is the basis of the proposed framework.
Exactly as described in Section 2.1, in order to apply the connected operators to the graph (G C , A), we create a set of connected components of the graph (G C , A).We denote this second set of connected components by CC, and we note that CC ✓ P(C).Each connected component CC 2 CC is composed of a set of nested connected components C 2 C. We can also associate a second attribute AA to each connected component CC 2 CC, where AA is an element of the set of mappings (CC, A) !R or Z denoted by AA.This second attribute function is usually designed to be increasing so that the filtering process is easy to apply.We filter out all the connected components CC 2 CC such that AA(CC ) is smaller than a given threshold .This filtering yields a subset of the connected components CC 0 ✓ CC given by: For each connected component Then the final filtered subset of connected components C 0 of the graph (G, f ) is given by: As described in Eq. ( 6), we can reconstruct an image f 0 from this subset of connected components C 0 and its associated level function F: . This reconstructed image f 0 is the final filtered image.

Shape-space filtering using trees
In practice, an efficient implementation of the connected operators and the framework of shape-space filtering rely on some tree-based representations (see [9,10] for some more details).Indeed, if the whole domain of the graph G (resp.G C ) belongs to the set of connected components V 2 C (resp.C 2 CC), then the set of connected components C (resp.CC) can be organized into a tree structure denoted by T (resp.T T ) thanks to the nesting property.Each node of the tree T (resp.T T ) corresponds to a connected component C 2 C (resp.CC 2 CC).The parenthood relationship of the tree T (resp.T T ) is given by Pr defined in Eq (7) (resp.PPr which is similar to Pr but defined on CC).The node Pr (C) 2 T (resp.PPr (CC ) 2 T T ) is said to be the parent of the node C 2 T (resp.CC 2 T T ), and C (resp.CC ) is said to be a child of Pr (C) (resp.PPr (CC )).Note that the node representing V 2 C or C 2 CC has no parent.The set of connected components C > in Eq. ( 3) and C < in Eq. ( 4) are organized into respectively Max-tree and Min-tree representation [9].And the set of connected components C < > in Eq. ( 5) gives the tree of shapes [10].Using these tree-based representations, we call the shape-space G C = (C, E C ) described in Section 3.2 the tree-based shape-space.We also denote it by G T .Each node of the graph G T is adjacent to its parent, so each node has its parent and its children as its neighbors.
A schematic overview of the framework of shapespace filtering relying on the tree-based representations is depicted by the black+red path in Fig. 1.The filtering is performed on the second tree T T , that contrasts with classical connected operators which filter the first tree T .The whole process is composed of: 1) The first tree T construction to create C of graph (G, f ); 2) The second tree T T construction to create CC of graph (G C , A); 3) The second tree pruning that yields a subset CC 0 ✓ CC contained in the simplified tree T T 0 ; 4) A simplified tree T 0 restitution to produce the filtered subset C 0 ✓ C, and an image reconstruction from C 0 .Fig. 3 depicts a synthetic example of the proposed framework using tree representation.We hereafter rely on this example to detail each step of the proposed framework.

Creation of C by tree representation T
First of all, we build a tree representation of graph (G, f ) to create the set of connected components C. The choice of the type of tree T depends on the targeted application: the shapes (objects) of interest have to be present at some nodes of the chosen tree.The tree of shapes is used in Fig. 3.During the tree T construction, we are able to compute incrementally a lot of information, based on which the attribute A (e.g., area, gray level F, or some more evolved shape attributes) is derived.The pair (G T , A) is a node-weighted graph modeling the treebased shape space.The graph (G T , A) in Fig. 3 has four regional minima (represented by red circles).

Construction of CC by the second tree T T
As we have done in the previous step for the graph (G, f ), we construct the set of connected components CC of graph (G T , A) by constructing a tree representation T T .In this paper, this second tree T T is either a Mintree or a Max-tree representation.The choice between them depends on the application and on the nature of the attribute A. Indeed, the attribute A encodes the probability for a shape to be of a given type.If we want to filter out the shapes different from that type, we use a Min-tree for T T , so that the minima of the space of shapes are the less probable shapes.In other words, for a node of T to be a leaf of T T , its parent and its children have to be more probable than the node itself.
Conversely, if we want to preserve the shapes of a given type, then we use a Max-Tree for T T .
Two different "objects" located in the same branch of T are now possibly present in two different branches of T T .For example, in Fig. 3, nodes C and H of T are now in two distinct branches of T T (Min-tree of T ).Hence, these two "objects" can be extracted separately, which is difficult to achieve using only T .In this example, the four regional minima of the graph (G T , A) are represented by the four leaves (red circles) of T T .

Filtering of T T
A tree filtering of T T is performed based on a second attribute AA.This attribute AA is always an increasing attribute in order to ensure that the second tree filtering can be achieved with a simple pruning strategy.The design of this second attribute AA is quite flexible.Usually, it can also be computed incrementally during T T construction, based on the first attribute A (for instance, the height of A), or based on the contextual information on the image domain around the shapes that CC represents.
The pruning is then based on comparing AA to a given threshold.Let us remark that depending on the application, two different pruning strategies can be used.For the purpose of filtering out some non-desired shapes, the nodes to be pruned are the subtrees rooted just above the leaves.If we want to select the shapes corresponding to the desired shapes (represented by the leaves of T T ), the pruning strategy removes the nodes that are close to the root node of T T .This is equivalent to preservation of subtrees containing the leaves.
In Fig. 3, the second tree is filtered by pruning the nodes CC whose attribute value AA, e.g., the height of A applied on shape-space, is less than 2. This is an example of filtering out the non-desired shapes around the leaves.

Tree restitution and image restitution
The tree restitution step is trivial.The simplified tree T 0 is reconstructed by removing the set of nodes {C i } i contained in the series of filtered nodes {CC k } k , and by updating the parenthood relationship.For example, the simplified tree T 0 in Fig. 3 is obtained by removing the nodes (dashed circles) contained in the set of filtered nodes of T T 0 , and each preserved node in T takes its lowest preserved ancestor as its parent.
For any point x 2 V , let C a x 2 T 0 be the lowest preserved ancestor node that contains the point x.Then based on the principle of image reconstruction described in Section 3.1, the filtered image f 0 at any point x is given by: A similar reconstruction strategy by maintaining the local contrast of the preserved nodes is detailed in Section 4.1.
Note that in Fig. 3, the nodes C and H are removed, while the node E lying between them in the same branch is preserved.None of the existing pruning strategies described in Section 4.1 can achieve such a result.Indeed, the nodes E and J with A = 3 are preserved, while the node F with A = 3 and even the node D with A = 4 are filtered out.Such a behavior cannot be obtained with a threshold-based strategy.

CONNECTED OPERATORS
The framework of shape-space filtering deals with connected components which are composed of flat zones.This guarantees that the operators of the proposed framework belong to the class of connected operators.In this section, we show in Section 4.2 that the framework of shape-space filtering encompasses some classical strategies of attribute filters which are detailed in Section 4.1.

Classical rules for tree filtering
Filtering the tree T with some increasing attributes A " is rather straightforward: it is simply performed by removing all the nodes that do not satisfy the attribute criterion T .As the attribute A " is increasing, it amounts to removing a branch of the tree: if a node does not satisfy T , then none of its descendants satisfies it.The pruned points take the respective gray levels of their respective lowest preserved ancestors.For nonincreasing attributes, which is the usual case for many shape attributes, there is no straightforward strategy to filter the tree.Salembier et al. [9,4] propose three pruning strategies (Min, Max, Viterbi) and a threshold-based rule (Direct).Urbach et al. [11] also propose a threshold-based strategy (Subtractive).The decisions of these rules for a given attribute threshold are described as follows: • Min: a node C is removed if A(C) < or if there exists one of its ancestors C a such that A(C a ) < .• Max: a node C is removed if A(C) < and for all its descendants C d , A(C d ) < holds.• Viterbi: removal and preservation of nodes are determined by a cost optimization process.Each transition is assigned with a cost.For each leaf node, the branch with the lowest cost to the root node is taken.See [9] for more details.• Direct: a node C is removed if A(C) < .The pixels stored in C but not in any of its descendants take the gray level of the first preserved ancestor starting from C to the root.• Subtractive: as the direct rule, but it differs in image reconstruction.The change in the gray level of the modified node is applied to all its descendants, so that the contrast with the local background remains unchanged.Let us notice that all those rules have some drawbacks.Pruning strategies can not deal with the case where two interesting objects are present in the same branch.With the two threshold-based rules, it is often impossible to retrieve at the same time all the expected objects with one unique global threshold.

Encompassing some classical attribute filters
We now explore the effect on the image of filtering T T , the Min-tree of the node-weighted graph (G T , A), with some specific attributes.We show that those attributes allow us to retrieve the classical filtering rules described in the previous section.

Case A: Pruning of increasing attribute A
In the most trivial case, the attribute A is increasing, and the classical connected filters are equivalent to a pruning of the tree.
Proposition 1: For any tree representation T , if A is strictly increasing, let T T be the Min-tree of the nodeweighted graph (G T , A), then T T is isomorphic to T .
Proof: Since A is strictly increasing, so for any given node C, A(C) < A(C a ) holds for any ancestor node C a of C, which means that the leaves of T are regional minima of graph (G T , A).These regional minima lie also on the leaves of T T being the Min-tree.Furthermore, for any pair of neighboring nodes Suppose that the former one holds, then i be any neighboring node of C j , since the parent of each node on a tree structure is unique, so In consequence, we have CC Ci = PPr (CC Cj ).So T T has the same structure of T .
Let AA be the current level of the second tree T T , which means 8 C 2 C, AA(CC C ) = A(C).Pruning T T is equivalent to pruning T .In other words, shape-space filtering encompasses the classical filtering strategy in this case, but we do not have to test whether the attribute A is increasing or not.

Case B: Thresholding of non-increasing attribute A
A shape attribute A is more often non-increasing.In such a case, there exists some pair of vertices (C i , C j ) 2 E C , such that C j = P (C i ), and A(C j ) < A(C i ).For example, let T T be the Min-tree of the graph (G T , A), then the node CC Ci 2 CC is hence an ancestor of CC Cj 2 CC on T T .So T T is different from T .Furthermore, just like the increasing attribute case, let AA be the current level of T T , we have AA(CC C ) = A(C) for any C 2 C. Pruning T T by AA is equivalent to thresholding T .Accordingly, shape-space filtering encompasses the thresholdbased strategies (Direct and Subtractive rules), which are context-free or adjacency stable operators, as explained in [31].Let us remark that, although shape-space filtering theoretically encompasses the cases presented above, a direct implementation (not relying on the proposed framework) is more efficient.Let us also remark that it is impossible (without a global rule) to retrieve in the proposed framework the contextual filtering [32] pruning strategies described in Section 4.1, based on Min, Max, and Viterbi.
The second attribute AA can be different from A. For example, it can be any measure based on A or even some new attribute/measure computed from the image domain (e.g., the total variation inside the context region represented by the node CC).This is when shape-space filtering becomes particularly useful.

SOME NOVEL CONNECTED OPERATORS
Shape-space filtering is more flexible than conventional connected operators and offers some new possibilities.In this section, we introduce two novel classes of connected operators: shape-based lower/upper levelings in Section 5.1 and morphological shapings in Section 5.2.
In the rest of this paper, we study the following process: • we first build a family C of the connected components of level sets of the image.In other words, C is made of the connected components corresponding to the nodes of either the Min-tree, the Max-tree or the tree of shapes; • using the parenthood relationship in Eq. ( 7), we build the graph (C, E C ), where E C is given by Eq. ( 9); • we choose an attribute A that evaluates any element of C.This attribute does not need any particular property, other than the ones needed for the application at hand; • we choose an attribute AA, increasing for the connected components of node-weighted graph (C, E C , A).This attribute AA evaluates the strength of the attribute A in the graph (C, E C ); • we then make an attribute filtering on the nodeweighted graph (C, E C , A) with the attribute AA.We call such an attribute-based connected operator a shape-space filter.Note that this definition is not to be confused with the definition of shape filter given in [11]: specifically, a shape filter is scale, rotation and translation invariant.

Shape-based lower/upper levelings
Let us now detail the first type of novel connected operators offered by the framework of shape-space filtering, in the case of T being a Max-tree or Min-tree representation of the image f .For that, we first recall the notion of levelings [14].
The levelings form a subclass of connected operators.They act by enlarging the flat zones by suppressing many details, while keeping the sharpness of the transition zones.The levelings are the intersection of two subclasses: the lower levelings and upper levelings.Definition 1: An operator is a lower leveling of a grayscale image f if and only if for any pair of neighboring vertices (x, y) Definition 3: An operator is a leveling of a grayscale image f if and only if for any pair of neighboring points (x, y) : The definition 3 states that if there is a transition in the output image after leveling, the transition exists in the initial image.This is because Furthermore, the interval of the transition in the output image [ (f )(y), (f )(x)] is contained in the interval of the transition in the input image [f (y), f(x)].
The readers can refer to [14,15] for more details about the properties of levelings.
Proposition 2: If C is made from a Max-tree T , and if the attribute function is a (non-increasing) attribute, any shape-space filter is an upper leveling.We term such an operator s" a shape-based upper leveling.Proof: Let T be a Max-tree representation, then no matter what type of tree T T is and no matter how T T is pruned, the simplified tree T 0 has always a Max-tree structure in the sense that gray level for the ancestors is always lowered.In the image reconstruction step, the pixels stored in some removed node C r take the gray level of the first preserved ancestor C a (Direct rule) or even lowered with the change induced by the removed ancestors (Subtractive rule).Hence, 8x 2 V, s" (f )(x)  f (x) always holds.By Definition 2, such an operator s" is an upper leveling.Proposition 3: If C is made from a Min-tree T , and if the attribute function is a (non-increasing) attribute, any shape-space filter is a lower leveling.We term such an operator s# a shape-based lower leveling.Proof: Let T be a Min-tree, the simplified tree T 0 is still a Min-tree in the sense that the gray levels of the ancestors are always higher.So, similarly to the proof of Prop.2, 8x 2 V, s# (f )(x) f (x) holds.Based on Definition 1, such an operator s# is a lower leveling.When the first tree T is a Max-tree (resp Min-tree), the reason why shape-space filtering gives shape-based upper (resp.lower) leveling is that any anti-extensive (resp.extensive) operator is an upper (resp.lower) leveling.Let us also remark that in both cases, the operator s is not a leveling.Indeed, for any pair of points (x, y) 2 E, we have either f (x) s (f )(x) and f (y) s (f )(y), or s (f )(x) f (x) and s (f )(y) f (y).Therefore, the operator s is not a leveling (except when s (f ) is a constant), according to Definition 3.
A classical upper (resp.lower) leveling removes some details around the regional minima (resp.maxima).In practice, this is equivalent to pruning the Min-tree (resp.Max-tree) with an increasing attribute A " .Nevertheless, a shape-based upper (resp.lower) leveling is based on some connected filtering on the shape-space built from the Max-tree (resp.Min-tree) representation.It filters out some details of unwanted bright (resp.dark) shapes on the basis of a user defined non-increasing attributes.
Recall that an operator is said to be idempotent if and only if ( (f )) = (f ).

Proposition 4:
If the second tree T T filtering is idempotent then the shape-based upper leveling s" and lower-leveling s# are idempotent.Proof: Let T 0 , T T 0 , T T 0 0 , and T 0 0 be the tree structures corresponding to the shape-based morphological operator s applied to f .T 1 , T T 1 , T T 0 1 , and T 0 1 are the tree structures corresponding to s applied to f 0 = s (f ).As T 0 0 and T 1 are either a Max-tree or a Min-tree, it is trivial to see that T 1 = T 0 0 , thanks to the strict fixed order of the gray level between neighboring nodes of those trees.As the second tree filtering is based on the pruning strategy, it is equivalent to removing some blobs around minima or maxima of the graph (G T0 , A).As a consequence, the second tree of graph (G T 0 0 , A) is the same as T T 0 0 , which means that T T 1 = T T 0 0 .The idempotent second tree filtering yields:

Morphological shapings
Unlike the shape-based upper or lower levelings, which deals only with bright or dark shapes, we introduce in this section a second type of connected operators which process both bright and dark shapes at the same time.
Proposition 5: If C is made from the tree of shapes T , and if A is a non-increasing attribute, the operator s given by the shape-space filtering is in general not a leveling.
Proof: Since A is non-increasing, and any shape-space operator gives a filtered tree T 0 which does not result from a pruning of T , a pair of neighboring vertices (x, y) can exist, such that C x ⇢ C y , and C y is removed while C x is preserved.Let C z be the lowest preserved ancestor of C y .Then we have However, as the tree T is the tree of shapes, the order of the gray level of a node and any of its ancestors is not monotonous a So it is possible that f (z) < f(x) and f (z) < f(y).In this case, we have . This is in contradiction with the definition of a leveling (see Definition (3)).

Definition 4:
If C is made from the connected components corresponding to the tree of shapes T , and if A is a non-increasing attribute, then the associated shape-space filter is called a morphological shaping.
In the sequel, the symbol S is used for denoting a morphological shaping.Proposition 6: A morphological shaping S with shape attribute A is a self-dual operator.
Proof: Since the tree of shapes T is self-dual, the tree of shapes of graph (G, f ) denoted by T + has the same structure as the one of graph (G, f ) denoted by T .Consequently, the set of connected components made from T + denoted as C + is equal to the one made from T denoted as C .By definition, the shape attribute A does not depend on the gray levels of the points inside the connected components, which means for any Thus the shaping S is a self-dual operator.
To make a shaping S idempotent, the first condition to satisfy is that the tree of shapes of S(f ) denoted by T 1 is equivalent to the simplified tree T 0 0 from which the filtered image S(f ) is reconstructed.However, this is not a trivial requirement due to the unfixed order of gray levels of the neighboring nodes on the tree of shapes T .A node C k 2 C of T is obtained by filling the cavities of either a component of a upper level set (see Eq. ( 1)), or a component of a lower level set (see Eq. ( 2)).Let us denote the family of these two types of nodes respectively by < .Proposition 7: Let T be the tree of shapes, C 0 be the set of preserved connected components, and P 0 be the parenthood relationship between connected components in C 0 .For each preserved node < of T , and > of T , then the tree T 1 constructed from f 0 is equal to T 0 from which f 0 is reconstructed.
Proof: Thresholding the reconstructed image f 0 by Eq. ( 1) and Eq. ( 2) yields some shapes which can also be found in T 0 .This correspondence is guaranteed by the condition 8 C 2 Based on the definition of the tree of shapes [10], the inclusion relationship between those shapes yields a unique tree, which is equal to T 0 .Proposition 8: If the second tree filtering T T is idempotent and if the type (either a < node) of the preserved nodes does not change, then the shaping S is idempotent.
Proof: See Prop.7 and the proof of Prop. 4.

ILLUSTRATIONS AND EXPERIMENTS
In this section, we present some illustrations and experimental results based on the novel connected operators introduced in the previous section.First of all, in Section 6.1, we provide several filtering results of some shape-space filters, and we compare them, from a qualitative point of view, to those based on thresholding strategies.This qualitative comparison demonstrates the robustness and flexibility of the proposed framework.Then, in Section 6.2, we evaluate quantitatively one of the shape-space filters against its corresponding threshold-based filter, in the context of retinal blood vessel segmentation.

Comparison with classical approaches
We compare several shape-space filters to their corresponding classical attribute filters based on thresholding strategies.First of all, we illustrate two examples of shape-based lower levelings.In Fig. 4, we aim to preserve the round disks while filtering out the other structures in Fig. 4 (a).The first tree representation is a Min-tree, and we use for the attribute A the inverse of the compactness [5,11], the compactness being defined by L 2 / (4 ⇡ A), where A and L are respectively the area and the perimeter of the component.Fig. 4 (c) is the result of a subtractive reconstruction and Fig. 4 (d) is obtained by taking the average value for the pixels inside each preserved node excluding its preserved descendants.We observe that, with a low thresholding value, some unwanted objects remain.When we augment the thresholding value (shown in Fig. 4 (e) and (f)), some of the expected objects are filtered out.At the same time,  the second tree T T is a Min-tree, and the second attribute function AA is the combination of the attribute A itself with the total variation inside the context region represented by the node CC.We observe that all the five round disks are preserved while all the other structures have been filtered out.In Fig. 5, we use the average of the gradient magnitude on the shape contour as attribute A. As illustrated in Fig. 5 (c) and (d), thresholding this attribute with a low value preserves some structures (inside the red boxes) in the background.To get rid of these unexpected structures, a higher thresholding value is applied.But, as shown in Fig. 5 (e) and (f), some of the expected structures (inside the blue boxes) are also filtered out.The result of the shape-based lower leveling (Fig. 5 (b)) is cleaner.We preserve any node in the second tree T T (a Max-tree) with a small height, and if furthermore the node contains a maximum of ) having a high extinction value [33] and a high attribute A.
When we want to perform a self-dual filtering, i.e., to process at the same time both upper and lower level sets, we choose as tree T the tree of shapes.Such an operator is a morphological shaping.In both Fig. 2 and Fig. 6, we use for attribute A the circularity, 1 l min /l max , where l min and l max denote the respective lengths of the small-  In Fig. 6, we compare a self-dual morphological shaping to a variant of the state-of-the-art thresholding approach [11].In the latter, when the threshold of A is low, some objects do not appear (Fig. 6 (c)).To be able to get all expected objects, we have to set a high threshold.However, in this case, too many unwanted objects are present (Fig. 6 (d)).The filtering strategy used in the shaping consists in preserving the minima of the graph G C based on their extinction values [33].More details about this filtering strategy can be found in [13].
With the shaping, all the expected objects can be found, as depicted in Fig. 6 (b).The results can be improved by combining some shape attributes.In Fig. 6 (e) and Fig. 6 (f), we use a combination of the circularity and of the non-compactness attribute, Tr(I) / A 2 , where I is the moment-of-inertia tensor and A is the area [5,11].The combination of shape attributes significantly improves the results.Yet, the shaping in Fig. 6 (f) performs much better than the threshold-based strategy in Fig. 6 (e).In Fig. 7, we give another example of comparison between our shaping and classical attribute thresholding.The used attribute function A is the inverse of the context-based energy estimator proposed in [34].Just like in active contours, this energy estimator measures the significance of each region, based on the image contents around the region boundary.Small values of the estimator correspond to meaningful objects.More details about it can be found in [34].We construct a Min-tree  T T on G C , and we choose for the second attribute AA the total variation (as in Fig. 4).As depicted in Fig. 7, classical attribute thresholding gives reasonable results (in Fig. 7 (c) and (d)), but some unwanted structures in the background remain.In contrast, as observed in Fig. 7 (b), the shaping preserves all the expected structures on a clean background.Shape-space filters provide more possibilities and flexibilities for image filtering as compared to classical attribute filters.In all those above illustrations, our shapespace filters give results that are more robust than the classical attribute thresholding methods.And some of them are impossible to obtain with classical methods.

Blood vessels segmentation in retinal
In this section, we evaluate quantitatively one of the shape-space filters on the specific application of retinal blood vessel segmentation, where it is used as a segmentation tool.The motivation here is not to develop a complete process, but rather to demonstrate that the qualitative results of the previous section 6.1 are of some practical importance.In particular, we are going to see that a shape-based filter can advantageously replace its threshold-based counterpart.
As in many other existing approaches [35], we work here on the green channel of the color retinal image.To improve the visibility of the blood vessels, for each color retinal image f c , a black top-hat transform, given by b5 (f g ) f g , where is morphological closing using a disk with radius 5 as structuring element b 5 , is applied to the green channel f g .When a mask of eye fundus is available, we combine it with the black top-hat f t .We thus obtain an image f i in which the blood vessels are visible, and the main structures of the blood vessels are present in the Max-tree T representation of f i .Blood vessels are usually long and thin structures.Hence, the attribute used here is the elongation A e .For each node C of the Max-tree T , the elongation attribute is given by where |•| denotes the cardinality (area) and l max denotes the length of the largest axis of the best fitting ellipse for the connected component C. We expect that nodes with low attribute A e correspond to blood vessels.The core of this application is the filtering of the Maxtree T .A mere thresholding of the elongation A e already provides interesting results, but also gives unwanted objects (noise).However, a very low thresholding value t min on A e ensures that thresholded nodes are blood vessels.These initially extracted nodes are used as seeds in the sequel.We then apply a morphological filtering with a depth criterion: using the Min-tree T T of the node weighted graph (G T , A e ), we only preserve the nodes that have an height smaller than a given thresholding value d 0 in T T and that furthermore contain the seeds.The connected components contained in the preserved nodes of T T are considered as segmented blood vessels.The whole process is one of the many variants of shape-based upper levelings.An example of such a blood vessel segmentation process is given in Fig. 8.The elongation-based upper leveling correctly segments most of the blood vessels (see Fig. 8 (e)).
We have tested this specific shape-based upper leveling on the database of Digital Retinal Images for Vessel Extraction (DRIVE) [36,37] and on the database of STructured Analysis of the Retina (STARE) [38,39].Fig. 9 shows two segmentation results on DRIVE database, where the parameter t min is set to 0.05, and the parameter d 0 is set to 0.09.Two segmentation results on STARE database are depicted in Fig. 10 in Appendix A for this paper.Qualitatively, most of the blood vessels are correctly extracted, although some noise at the end of the vessels is also extracted.In addition some very thin blood vessels are missed.
Quantitative assessment is based on three performance measurements: sensitivity, specificity, and accuracy [37] (see Appendix A and [37] for details about these measurements).High values of these measurements are required for good segmentation results.A benchmark of our approach compared to the threshold-based attribute filtering [40], and one of the best dedicated method [35] on DRIVE database is provided in Table 1.The proposed elongation-based upper leveling is more robust than the thresholding-based subtractive filter given in [40], where the thresholding value is set manually for each      image.In addition, in [40], a branch filtering [41] is also required to remove some unwanted neighboring objects of the expected objects or noise in the background.In contrast, the elongation-based upper-leveling achieves clean background while preserving expected objects.In the Appendix A of this paper, a complete benchmark of different methods on DRIVE and STARE databases is provided in Table 2 and in Table 3.

CONCLUSION
Connected operators are filtering tools that have very good contour preservation properties and are appropriate for both low level filtering and high level object recognition [4].In this paper, we have expanded on the idea of tree-based connected operators.We propose to apply connected operators to an equivalent tree-based representation of the image.We term such a processing shape-space filtering.We have shown that the framework encompasses the classical attribute filters.We have introduced two novel classes of connected filters: the shape-based lower/upper levelings (obtained with T being the Min-tree or Max-tree) and the morphological shapings (obtained with T being the tree of shapes).
Those connected operators filter out image components based on some non-increasing criteria.Furthermore, we have studied some of the properties of those novel connected operators.
We have compared the shape-space filters to their corresponding threshold-based methods through some filtering illustrations.Shape-space filters usually provide more robust results (clean background while, at the same time, preserving the desired structures) and achieve some results that are impossible to obtain with classical approaches.We have evaluated one of the many shapebased upper levelings on retinal blood vessel segmentation, showing that it outperforms a dedicated approach relying on a classical threshold-based filter.
We believe that shape-space filtering has a large potential.In this paper, we have only presented the aspect of filtering, and hardly skimmed the other surface of what they have to offer to the scientific community.In [34], we have used this framework to perform object segmenta-tion on the tree of shapes.The idea is to propose an efficient context-based energy estimator whose minima correspond to meaningful objects.Other criteria that can be used in this context are for example the number of false alarms [42] and the snake energy [43].A practical problem is that many minima of such a criterion do not correspond to meaningful components of the image.In [34], a morphological closing in the space of shapes helps to filter out those spurious minima.When augmenting the filtering strength, more and more minima disappear.Indeed, the extinction value [33] measures the significance of minima, based on which we can produce extinction-based saliency maps [44,45].In [46], we have shown an example of such obtained saliency maps, which is an extension of the notion of constrained connectivity [47] to non-increasing constraints, using as an example a criterion from [48].
In future work, two different research directions are of interest.On the one hand, many other attributes than the ones tested in this paper need to be investigated.In this respect, learning the attributes rather than using hand-crafted ones is an interesting perspective.On the other hand, building G C from other trees than the Min-tree, Max-tree, or the tree of shapes deserve further studies: obvious examples are the binary partition tree [49] (see [46] for some preliminary results), or treebased shape spaces using mask-based second generation connectivity [7] or hyperconnectivity [8].More generally, the proposed approach can further be generalized to other spaces, i.e., an appealing idea is to build G C from graphs that do not necessarily have a tree structure: family of segmentations, component-graphs of multivalued images [50,51], directed graphs (considering nonsymmetric adjacency relations [52,53]) are promising examples of such graphs.
As a final note, we would like to stress out that the filters proposed in this paper are part of the C++ image processing library Milena [54], available on the Internet as free software.The present paper also demonstrates the interest of applying image processing algorithms to different types of graphs, in other words, the interest of genericity for image processing.

APPENDIX A ADDITIONAL RESULTS FOR BLOOD VESSEL SEGMENTATION IN RETINAL IMAGES
Fig. 10 shows two segmentation results on the STARE database, where the parameter t min is set to 0.055, and the parameter d 0 is set to 0.01.Fig. 10: Some blood vessel segmentation results on the STARE database.On the right side, black pixels are true positive, white pixels are true negative, cyan pixels are false positive and red pixels are false negative.
Quantitative assessment is based on three performance measurements: sensitivity, specificity, and accuracy [37].Sensitivity measures the true positive rate (TPR), specificity measures the true negative rate (TNR), and accuracy measures the rate of pixels correctly classified.Another performance measurement, named kappa values, is also used in [37] for the DRIVE database.It is a statistical measure of inter-rater agreement, which measures the agreement between two raters who each classify N items into M mutually exclusive categories.A complete benchmark of different methods is depicted in Table 2 and Table 3.In the case of DRIVE database (in Table 2), our result is slightly under the best results given by the method of Mendonc ¸a [35].Note also that the approaches of Staal [37] and Niemeijer [55] are supervised approaches.On the STARE database (in Table 3), the proposed method performs also very well.It is better than the method of Mendon¸ca [35].Both methods are very close to the second human observer.
We would like to stress out that, as said in Section 6.2, our goal in this experiment was to demonstrate that   shape-based filters can in many cases advantageously replace their threshold-based homologue.Hence, we did not try everything that is possible to do for blood vessel segmentation.Many other processing are possible, for example, it has been shown in [40] that using the three channels instead of only the green one can improve the results, as does image enhancement.Developing a complete application dedicated to the problem of retinal blood-vessel segmentation is left for further research.

Fig. 2 :
Fig. 2: (a) Evolution of the circularity attribute on two branches (containing respectively the red and blue shapes) of the tree of shapes [10]; (b to e) Some shapes; (f) Result of attribute thresholding; (g) Result of a shaping.

Fig. 3 :
Fig. 3: An example of the workflow of shape-space filtering framework with T T being the Min-tree of T .Circles with a capital letter inside: nodes of C; Blue values: first attribute A; Circles without a letter inside: nodes of CC; Red values: second attribute AA; Dashed circles: filtered nodes with a given threshold value 2.

Definition 2 :
An operator is an upper leveling of a grayscale image f if and only if for any pair of neighboring vertices (x, y) : (a) Input image.(b) Result of s# .(c) Low threshold of A. (d) Low threshold A. (e) High threshold of A. (f) High threshold of A.

Fig. 4 :
Fig. 4: Comparison between shape-based lower leveling and attribute thresholding, with A being the compactness.Images (c,e) are obtained with the subtractive rule and images (d, f) with the average [see text].
Low threshold of A. (d) Higher threshold of A. (e) Threshold strategy.(f)Shaping 2.
(a) Input image.(b) Result of s# .(c) Low threshold of A (subtractive).(d) Low threshold of A (average).(e)Higher threshold of A (subtractive).(f) Higher threshold of A (average).

Fig. 5 :
Fig. 5: Comparison of shape-based lower leveling to attribute thresholding; the attribute function is the average of the gradient magnitude on the shape contour.
(a) Input image.(b) Result of shaping.(c) Low threshold of A. (d) Higher threshold of A.

Fig. 8 :
Fig. 8: Steps for the blood vessel segmentation in retinal image using elongation-based upper leveling.Images in (c) and (d) are reversed.In (e), black pixels are true positives, white pixels are true negatives, cyan pixels are false positives and red pixels are false negatives.t min is set to 0.05, and d 0 is set to 0.09.

Fig. 9 :
Fig. 9: Illustration of blood vessel segmentation on the DRIVE database.In the segmented results, black pixels are true positives, white pixels are true negatives, cyan pixels are false positives and red pixels are false negatives.
These measurements are defined as: sensitivity = TPR = TP/P = TP/(TP + FN), specificity = TNR = TN/N = TN/(TN + FP), accuracy = (TP + TN)/(TP + TN + FP + FN), kappa = (Pb(a) Pb(e))/(1 Pb(e)), where TP stands for true positive, FP for false positive, TN for true negative and FN for false negative.Pb(a) is the relative observed agreement among raters, and Pb(e) is the hypothetical probability of chance agreement.If the raters are in complete agreement then kappa = 1.If there is no agreement among the raters other than what would be expected by (as defined by Pb(e)), kappa = 0.

TABLE 1 :
Benchmark of several blood vessel segmentation approaches on DRIVE database.

TABLE 3 :
Benchmark of different blood vessel segmentation approaches on the STARE database.