Constrained palette-space exploration

Color palettes are widely used by artists to define colors of artworks and explore color designs. In general, artists select the colors of a palette by following a set of rules, e.g. contrast or relative luminance. Existing interactive palette exploration tools explore palette spaces following limited constraints defined as geometric configurations in color space e.g. harmony rules on the color wheel. Palette search algorithms sample palettes from color relations learned from an input dataset, however they cannot provide interactive user edits and palette refinement. We introduce in this work a new versatile formulation enabling the creation of constraint-based interactive palette exploration systems. Our technical contribution is a graph-based palette representation, from which we define palette exploration as a minimization problem that can be solved efficiently and provide real-time feedback. Based on our formulation, we introduce two interactive palette exploration strategies: constrained palette exploration, and for the first time, constrained palette interpolation. We demonstrate the performances of our approach on various application cases and evaluate how it helps users finding trade-offs between concurrent constraints.


INTRODUCTION
Colors are central components of any visual information. In a computer-generated image, they can directly be pixel values or parameters of a visualized model (e.g. a material component). Their individual manipulation is tedious while being of primary importance for controlling the image appearance. This raises the need for high level color representations encoding the image colors in a compact set of representative parameters. A Color palette is a well established representation [Chang et al. 2015;Lin et al. 2013], which stores the colors that are the most representative of the content, This work was partially funded by the French National Research Agency (Mapstyle project ). The authors wish to thank the anonymous reviewers for their insightful remarks and suggestions, Mathias Paulin and Pascal Barla for their comments, and Kartini Thomas for her support and gure materials. Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or a liate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only. © 2017 Copyright held by the owner/author(s). Publication rights licensed to ACM. 0730-0301/2017/7-ART60 $15.00 DOI: http://dx.doi.org/10. 1145/3072959.3073650 regarding its usage and application context. Providing colors satisfying both aesthetic aspects and application constraints requires however extensive e orts from experienced users [Zhou and Hansen 2015].
We identify four main strategies followed by artists to design palettes. In the rst, the user starts with an existing palette, either selected from a palette database [Brewer 2015;Murray et al. 2012], or extracted from the principal colors of an image [Chang et al. 2015]. This palette is usually considered as a bootstrap and can be further adjusted by the user [Jalal et al. 2015]. In the second, an existing palette is modi ed using interactive tools [Adobe 2015]. Here, the user edits the palette colors and lets the system ensure the palette coherence by considering color harmony rules (e.g. analogous, complementary) [Itten 1974;O'Donovan et al. 2011]. In the third, one or several palettes are generated by the optimization of a set of speci cally de ned constraints between colors. Even though palettes are not generated interactively, such systems enable the modeling of complex constraints, e.g. simulating color vision deciencies [Yanagida et al. 2015] or decreasing energy consumption of display devices [Chuang et al. 2009]. In the last strategy, probabilistic constraints are learned in pre-process from sets of segmented images and their palettes [Lin et al. 2013;Ritchie et al. 2015]. Then, starting from an input segmented image, a set of palettes generated by stochastic sampling is proposed to the user.
From the end-user point of view, computer driven palette design can be done either by (1) using interactive tools limited to simple constraints, or by (2) browsing palettes suggestions designed by other users or generated computationally. To our knowledge, no solution has been proposed for the interactive exploration of palettes complying with advanced constraints In this work, we introduce a new versatile formulation enabling the creation of constraint-based interactive palette exploration systems. We de ne the exploration of palettes as a minimization problem that we solve e ciently to provide real-time feedback. Our technical contribution is a graph-based palette representation  in which nodes are colors and edges are constraints modeled as cost functions. Based on our formulation, we introduce interactive strategies for constrained palette exploration and constrained palette interpolation (Figure 1-c,d, Section 5). We also provide guidelines to setup constrained palettes by de ning their sets of colors and constraints (Section 6). These exploration schemes are versatile enough to serve as core components for the creation of interactive tools, which we illustrate with several applications (Section 7). Thanks to their interactive feedback, the proposed palette exploration systems assist the user in nding a palette considered as a skin pastel shades b) c) d) a) Fig. 1. a) In this work we represent the color relations (e.g. shades, anchors and contrasts) of an input pale e as a constraint hypergraph (b), from which we derive two optimization-based exploration strategies. c) While the user manipulates a color, here the sweater color, the pale e is updated to satisfy the constraints stored in the hypergraph. d) Given two input pale es, we compute continuous color interpolation paths to smoothly blend between the two inputs while preserving their interpolated constraints.
trade-o between multiple constraints. With our approach, users have a direct control over the exploration by adjusting the colors and the constraints if needed, while the system ensures the constraints preservation. We evaluate the bene t of our approach on exploration and interpolation scenarios (see user study in Section 8).

PREVIOUS WORK
In this section, we review the main approaches for creating and modifying color palettes. We rst present user-centered approaches for the interactive design of color palettes and we follow by the exploration techniques (e.g. optimization and stochastic search) used in computational approaches. As most of the proposed approaches come with their own set of constraints, we present in a third section an overview of the principal types of color constraints.

User-centered pale e exploration
Today, several tools are available for the interactive design of palettes. Meier et al. [2004] and more recently Jalal et al. [2015] study typical creation work ows, e.g. palette design, color replacement, and the generation of palettes similar to a given input palette. An important conclusion of their work is that users tend to experiment with the colors: they explore the space of feasible palettes by tweaking the individual colors and observe their arrangement.
Semi-automatic tools have been designed to ease this process. Adobe Kuler [Adobe 2015] and Paletton [Staníček 2015] are very popular tools that allow the creation and exploration of palettes with four or ve colors, by moving color points within a color wheel. Those systems recompute the palette by applying hue and harmony templates (e.g. triad, complementary) interactively. The Color gradients explorer [Jégou 2014] focuses on cartographic palettes controlled by color gradients.
With these tools, the user has a strong feeling of control on the palette, and relies on the system to ensure constraint conformity according to simple rules (e.g. color harmonies). The strength of these systems is their predictability, however they rely on hard constraints and cannot deal with over or under-constrained con gurations.

Computational pale e exploration
Computational approaches have been proposed to generate or modify palettes. In opposition to the aforementioned approaches, the goal is not to focus on user control during the generation process, but rather to support more advanced constraints. We divide these approaches in two categories: those targeting speci c applications and those for general purpose application.
Application-speci c exploration: In the last decades, computational palette exploration has been largely studied for map design, energy e ciency (i.e. to reduce the energy required to display the palette) and adaptation to Color Vision De ciencies (CVD).
Palette optimization for map design has mainly been addressed as a problem of color contrasts related to perceptual and semantic constraints. Constraints are designed to convey the notions of difference, order and association between colors [Bertin 1983;Brewer 1994;Buard and Ruas 2009]. Christophe [2011] models a Constraint Satisfaction Problem (CSP) to handle cartographic and artistic constraints. This formalism implies to consider discrete color spaces, and uses heuristics to handle over-constrained problems.
Palette optimization for Color Vision De ciencies (CVD) rather focuses on nding a balance between aesthetic and accessibility constraints. Such problems are usually over-constrained, and enable the use of optimization techniques such as genetic algorithms [Ichikawa et al. 2003;Troiano et al. 2008] and probabilistic exploration [Zhou et al. 2014]. More recently, Yanagida et al. [2015] present an exploration method based on Fuzzy Constraint Satisfaction Problem (F-CSP). In contrast to the CSP used in map design, F-CSP can partially satisfy the input constraints while providing an associated satisfaction degree.
Recent advances on display devices have led to the di usion of OLED screens, which require a variable amount of energy according to the displayed colors. Energy Aware Color Sets [Chuang et al. 2009] is an approach to optimize the color palette controlling the appearance of the displayed content in order to reduce the display energy cost. Colors are modi ed using a non-linear optimization scheme in order to satisfy both a global energy cost and local contrasts between each pair of colors. Fig. 2. a) Our approach takes as input a constrained pale e defined as a set of colors connected in a graph by n-ary constraints. b) When the user changes a color or a constraint parameter (e.g. target contrast between colors) , c) the system updates in real-time the other colors so they respect the constraints. d) Starting from two constrained pale es, our system continuously interpolates both colors and constraints to generate new designs.
General-purpose exploration: More recently, computational tools have been proposed to generate palettes for arbitrary applications, i.e. palettes that do not rely on explicit a priori knowledge to constrain the problem de nition.
I want Hue [Jacomy 2015] is a system generating a palette from a set of input colors controlled by the user using a k-means algorithm.
The key idea of this approach is to prevent similar colors in the output palette by extracting well distributed clusters in color space. Other constraints, such as saturation bounds, can be applied during the generation. This approach can generate multiple palettes from a single input (using random seed initialization), but it remains limited by the de nition of the metric used for the classi cation.
An orthogonal work by Lin et al. [2013] tackles this problem by learning the constraints as discrete probability distribution functions from a representative set of pairs of palettes and images. Palette generation is then performed by sampling the probability space using either Markov Chain Monte Carlo or Hamiltonian Monte Carlo [Ritchie et al. 2015] methods. While very attractive, these approaches are limited in two ways regarding our context. Firstly, the diversity of the generated palettes is restricted by the learning set, which must be carefully designed a priori to produce representative distribution functions. Secondly, once learned, the distribution functions are xed and constraints cannot be adjusted easily, which limits the user controls during exploration.

Color constraints
Most of the aforementioned approaches provide their own constraint de nitions. We classify these constraints in three categories that we use later in our formulation.
Color properties: A rst type of constraints is related to the control of the properties of a color, regardless of the other palette colors. Low level constraints operate directly on the color coordinates in a given color space, e.g. hue or lightness. High level constraints have also been considered, for instance to control a color according to its temperature or activity [Flatla et al. 2013;Ou et al. 2004].
Color relations: A second type of constraints applies on small groups of colors in relation, such as contrasts and harmonies. These relations are usually modeled by preserving inter-color distances in a dedicated color space or subspace [Ichikawa et al. 2003;Sharma 2002], and have been deeply theorized and experimented by artists [Itten 1974;Munsell and Birren 1973;von Goethe 1971]. Hue and harmonic templates coming from these fundamental works are now provided by all color manipulation tools as xed sets of rotations on the hue color wheel in the HSV space. While usual harmony templates ignore the HSV value components, Meier et al. [2004] also propose non-planar harmony templates between colors. In addition to distances, Chang et al. [2015] propose to preserve the luminance order between colors, as this property has a strong in uence on human perception.
Context-based measures: The aforementioned constraints consider color palettes regardless of their application context. A third type of constraints takes into account the colors, the associated media (e.g. vector graphic drawing, photography or map), and/or its usage (e.g.display type and context). A typical example consists in modulating the in uence of color contrasts according to the spatial relations of the elements in a represented image [Ichikawa et al. 2003], or the spatial contrasts in maps ]. Regarding the palette usage, several approaches constrain the palette colors to reduce the energy consumption required to display the media on speci c devices while preserving its legibility [Chen et al. 2014;Chuang et al. 2009;Hoarau 2011]. In the context of Color Vision De ciencies (CVD) adaptation, several works preserve contrasts and harmonies on a palette generated by simulating the CVD on the input palette [Ichikawa et al. 2003;Troiano et al. 2008]. In their recent work on Factor Graphs, Lin et al. [2013] learn distribution functions of color and relations properties (e.g. respectively lightness and perceptual di erence), and also estimate labels on the input images segments (noise, foreground, background).
We de ne our palettes as a set of colors constrained by a set of properties called constraints, for instance enforcing the contrasts between the palette colors (Section 4.2). More formally, a palette is represented as an hypergraph P = (C, E) whose nodes C = {c i ∈ G, i = 1..k } are the k palette colors c i expressed in a color gamut G.
The hyperedges E = {(e j , w j ), j = 1..k E } are the k E constraints e j on these colors with their respective weights w j . Constraint weights are used to modulate the relative in uence of each constraint within the palette. The right inset illustrates a palette P with three colors c i and ve weighted constraints (e j , w j ). Constraints e j can be of di erent arity, varying from one to the number k of colors in the palette. In this example k = 3, constraints e 1 , e 2 and e 3 are binary, e 4 is unary and e 5 is ternary.
We express a constraint of arity n j as a function e j : G n j → R taking as input a subset C j ⊂ C of n j colors. The real value e j (C j ) indicates how much the constraint is violated by the palette colors, i.e. the smaller the constraint absolute value, the better it is satis ed by the colors.
Using the weighted constraint values, we introduce the palette conformity error Π P of a palette P (conformity for short), which measures how much palette colors C are respecting the weighted constraints E (Section 4.3), i.e. Π P = 0 when all palette constraints are exactly satis ed.
We design the exploration of palettes as an optimization process operating on the palette colors and minimizing the palette conformity. We present in Section 5 two interactive palette exploration strategies based upon this representation. The rst strategy consists in editing parameters of an existing palette (e.g. colors or/and constraints, see Figure 2-b), which modi es its constraint values and thus its conformity Π P . New palettes are then produced in real-time by optimizing the colors so that the conformity is minimized (see Section 5.1). The second strategy is a palette interpolation scheme based on the optimization of interpolation paths (in color space) between paired colors of two interpolated palettes (Figure 2-d), so that the interpolation conformity error Π I is minimized. This conformity error, called interpolation conformity, is the integral of the conformity Π P of all intermediate palettes P obtained when the interpolation parameter varies in [0, 1] (Section 5.2). While the definition of a palette is agnostic to the color space, we use the Lab space to represent colors and interpolation paths and thus bene t from a perceptually uniform spacing of colors [Jain 1989].
We present in Section 6 how users can design colors and constraints to setup a palette P. We explain how high level relations between palette colors (e.g. harmony and semantic meaning) can be expressed with constraints. We illustrate the pertinence and the practical capabilities of our exploration strategies on numerous potential application cases (Section 7). We conduct a user study to evaluate (1) how our approach helps users to perform a trade o between constraints, and (2) how well users appreciate the intermediate palettes obtained by our approach (Section 8).

CONSTRAINTS AND CONFORMITY
As explained in the previous section, a constrained palette P is composed of colors c i and constraints e j de ned as cost functions. The goal of these constraints is to evaluate the deviation of the palette colors properties p(c i ) -e.g. color coordinates, hue or energy consumption -to a target value f r ef . We rst present the color properties used in this work (Section 4.1) and, based on these properties, the derivation of unary, binary and n-ary constraints (Section 4.2). Once our constrained palettes well de ned, we present the computation of our palette conformity error in Section 4.3.

Color properties
Color space coordinates: In our approach, a color is represented by its coordinates c in the Lab color space. These coordinates dene a property of the represented colors from which they can be compared. In practice, any other color space or sub-space can be used to constrain di erent color properties. For instance, we can use any of the Lab coordinates, the luminance, the hue, saturation and value (HSV) components to measure contrasts, harmonies or shades. In that case, properties are de ned by the transformations to the new color space or by the computation of a color component. For instance p hsv (c) denotes the transformation of c to the HSV color space and de nes as property the coordinates of c in that space. Following equivalent notations, p hue (c) is the property for the hue color component and p lum (c) the one for the luminance.
Color Vision De ciencies: Several Color Vision De ciencies (CVD) are represented as a transformation of the color coordinates expressed in the XYZ space. We simulate two types of CVD using the approach proposed by Rasche et al. [2005]. We denote as p deut (c) and p prot (c) the color properties returning the Lab coordinates of the color as seen by respectively a deuteranope and a proteranope.
Energy consumption: Recent display devices, such as OLED screens, consume a variable amount of energy for the display of di erent pixel colors. Based on the results presented by Chuang et al. [2009], we derive the property computing the energy required for the display of a color as: where p red (c), p green (c) and p blue (c) are respectively the red, blue and green color coordinates in the RGB space.

Constraints
General formulation: A constraint function e is a real-valued signed cost function computing how much a set of colors violates a constraint, zero representing a perfect match. In general, any realvalued function f (C) can be used to de ne a constraint function, by evaluating its deviation to a known target value f r ef , using: (1) In this research, we apply our constraints on the color properties. These constraints are presented below and in their de nitions, we denote as p(c) any of the aforementioned properties. Also notice that even when f is strictly positive, e can be negative when f is lower than the reference f r ef .

Contrasts:
We de ne contrast constraints e c as binary constraints penalizing the variation of di erence between two colors properties. This di erence is signed and takes into account the components ordering if the property is a single color component (e.g. the hue), while it is a distance if the property is a vector (e.g. Lab coordinates). Following Eq. 1, contrast constraints are thus de ned with: and the reference value f r e f is the desired contrast between the two color properties. In this equation, Card(p(c)) is the cardinality of the color property p(c) and we use | • | as the L 2 norm for computational performance, even though alternative de nition such as ∆E * ab in Lab might also be used.
We also de ne the clamped contrast constraintē c , which penalizes only contrast decrease. It is de ned by clamping the color distance to f r e f as follow: Anchors: We de ne anchor constraints e a R as unary constraints penalizing the distance between a color c and a reference set of colors R with: In this constraint, the anchors target is de ned by R, and thus the reference value f r e f in Eq. 1 is set to 0.
We present two types of anchors, de ning R either as a single target color c r e f or as a target color set or manifold. Anchors provide a way to model semantic information, for instance, forcing a color representing the sea within a map to remain in blue shades (e.g. by setting p as p hue , and R = {blue}).
Mean property target value: N-ary targets constraints provide a control over a set of colors by constraining the weighted mean value of the colors' properties to a desired value f r e f . They are de ned with: The weights w c i set the colors importance within the constraint.

Conformity
As mentioned in the overview, we introduce the notion of palette conformity. The conformity Π P of a palette P is a real-valued function evaluating the deviation of the palette colors from their constraints. It also takes into account eventual out-of-gamut colors. We thus de ne the conformity as the sum of two error functions: where, Π P constr , is the sum of the palette weighted squared constraint values: and Π P gamut , which penalizes out-of-gamut colors: with (c i ) the distance to the gamut for out-of-gamut colors: The term λ in Equation 2 balances the two errors when dealing with out-of-gamut colors (when all colors are valid, Π P amut is equal to 0 and Π P = Π P const r ). We empirically set λ = 10 3 for all our applications, as we expect palette colors to stay within their gamut. Note that what matters is the magnitude of λ and we observed that computations remain unsensitive to slight modi cations of its value.

PALETTE-SPACE EXPLORATION
On top of our palette representation and its conformity computation, we introduce two palette exploration strategies. The rst is based on a direct editing of the input palette parameters while the second relies on the interpolation between two input palettes.

Pale e exploration
As presented in Section 2, interactive exploration techniques use simple, yet e cient constraints to adapt a palette to user inputs. Following the same concept, our interactive palette exploration strategy produces new palettes P * with colors optimized from an edited palette P so that P * always minimizes the conformity Π P : In general, the palette exploration is guided by the user who directly edits the palette parameters (colors or/and constraints). Another interesting solution is to generate new sets of palette colors with a stochastic search as input editings for conformity minimizations, and let the user select the palette P * he prefers (as nal result or starting point for an exploration).
In practice, we use the Levenberg Marquardt algorithm (LM) to solve this non-linear minimization problem interactively (see the timings in Table 1 and the accompanying video). By its nature, LM will not necessary nd global optimal results but rather palettes corresponding to local minima. This can be seen as trade-o s between the user inputs and the conformity measure, and since in most cases the conformity is a non-convex function, these local minima provide a variety of con gurations to be explored. The conformity value of these con gurations is given by the optimization residuals.
When optimizing the colors, care must be taken to always generate nal colors inside the gamut. During the optimization, Equation 4 penalizes colors that are out-of-gamut, but it does not actually force the colors to remain in gamut. In practice, we observed that invalid colors are unlikely to be far from the gamut at the end of the optimization and therefore we project them on the gamut at the end of the process. This approach, allowing the optimizer to follow paths going temporary out-of-gamut, provides adequate results that are better than those obtained by always projecting the colors in the gamut during the optimization.

Pale e interpolation
In the past decade, several approaches have been proposed to explore high-dimensional parametric domains (e.g. shape deformation) using linear subspaces. The key idea behind such techniques is to compute a linear parametric domain in which the user can explore a set of valid designs interactively. Recently, Nguyen et al. [2015] have presented an approach to extract color manifolds from labeled image datasets, in order to de ne one or two dimensional linear color subspaces associated to a label (e.g. sky, skin, apples).
In this work, we present the computation of a one-dimensional linear palette subspace by interpolating between a pair of constrained palettes P 0 (C 0 , E 0 ) and P 1 (C 1 , E 1 ) (see Figures 3 and 4). We assume that the mapping between the colors is provided by the user, such that c 0,i is mapped to c 1,i .

Pale e matching.
In order to perform a meaningful interpolation, both palettes need to have the same topology, i.e. the same number of colors and the same constraint hypergraph topology. When the number of colors is di erent, we assume that parts of the colorized media are a ected by respectively one color in a palette, and multiple colors in the other, as in Figure 16 (road colors). In that case, colors are duplicated, as illustrated in right inset, in order to linear (Lab) optimized (our)

Parametric space.
Our exploration outputs an interpolation space parametrized by α ∈ [0, 1] and de ned by B the set of paths b i (α) linking the associated palette colors with b i (0) = c 0,i and b i (1) = c 1,i . We note C α = {b 1 (α), .., b k (α)} the set of interpolated palette colors.

Pale e interpolation.
As we de ne the palette exploration as the result of an optimization minimizing the conformity Π P (with Equation 6), we obtain the interpolated colors C α by computing the set B * of curves whose parameters optimization minimizes an interpolation conformity error Π I : 5.2.4 Interpolation conformity. In this equation, the interpolation conformity Π I is de ned as the sum of three error functions: Π I const r evaluating the constraints deviations of all palettes C α as the integral of the error function Π P const r (see Equation 3 Template-driven mode Simple UI Automatic con guration Fig. 6. Overview of the pale e design pipeline: a) Select the number of colors within the pale e, possibly using pale e extraction tools. b) Select groups of colors and assign them a relation type provided by the system (anchor, level of shade, harmony). c) Optionally select additional relations between groups and/or colors. d) Configure pale e parameters: colors (when not already provided), constraints reference values and weights.
when α varies in [0, 1], Π I amut penalizing out-of-gamut colors, and Π I smoot h favoring smoother interpolation curves. Π I is thus de ned as: where λ is set as in Section 4.3 and γ limits the in uence of the smoothness criterion to ambiguous cases only, i.e. when Π I const r and Π I amut are small (see supplementary materials for numerical values). The evaluation of constraints deviations is computed as: with constraints e α, j de ned for any value α ∈ [0, 1] as the linear interpolation of the constraints of the interpolated palettes P 0 and P 1 : e α, j (C α, j ) = (1 − α)e 0, j (C α, j ) + αe 1, j (C α, j ).
The penalization of out-of-gamut colors is computed as the integral of the out-of-gamut penalization Π P amut (see Equations 4 and 5) for all palettes obtained when α varies in [0, 1]: Finally, when the problem de ned by the constraints is underconstrained, it might exist an in nity of valid paths connecting the two palettes (see Figure 7). In consequence, we de ne Π I smooth P 0 P 1 c) a) b) Fig. 7. Example of an underconstrained problem taking benefit from the Laplacian constraint. a) Two input pale es, with a unique contrast constraint f c between two colors expressed in Lab. b) Interpolation path produced by our approach, which boils down to a linear interpolation naturally preserving the contrast. c) Another solution preserving the contrast, but producing unexpected color variations.
to prompt the optimization to converge on smooth interpolations closer to linear paths in Lab: where ∆ denotes the Laplace-Beltrami operator. We implement the interpolation paths with Bézier curves. They offer a convenient parametric domain, are easy to di erentiate and manipulate with a few control points. We use the Levenberg-Marquardt algorithm for the optimization of the curves' control points. The end-points of our paths are constrained by the interpolated color coordinates (b i (0) = c 0,i and b i (1) = c 1,i ), i.e. the rst and last Bézier curves control points are directly set and thus, they are not degrees of freedom for the optimization. All the interpolation results shown in this paper are obtained with quartic Bézier curves providing three free control points per path for the optimization. We illustrate how the palette conformity value can oscillate during an interpolation in Figure 3-c, and experimentation with other Bézier curve degrees (up to 12) are shown in supplementary materials.

CONSTRAINED PALETTE DESIGN
The design of a palette is done with the following pipeline, composed of 4 main steps illustrated in Figure 6: a) setting the number of colors, b) grouping the colors w.r.t. their relations (which generates constraints between colors), c) optionally setting the groups relations (which generates constraints between groups of colors), d) con gure the constraints (e.g. reference value and weights) and when required, generate a rst set of colors.
Step a) can be performed in several ways. When working with raster images, colors might be extracted using palettes extraction techniques [Chang et al. 2015]. In some other cases (e.g. raster images with indexed colors or vector images), input colors are directly extracted from the media. Alternatively, palettes can be designed from scratch using random colors or manual settings.
We detail in Sections 6.1 and 6.2 the systems required to design the constraint graph (steps b,c) and to set the constraint parameters (step d) respectively. Two examples of systems implementing this palette design pipeline are presented in Section 6.3.

Design system for constraints definition
In this work, we model high-level relations between colors (e.g. levels of shade, color harmonies, or even semantics) as arrangements of constraint functions, represented by the constraint graph. Palette design is usually a hierarchical process: colors are rst grouped w.r.t. to their relations (e.g. harmony template). Then, groups of colors can be connected, for instance to preserve contrast between two harmony groups. We identi ed 4 main types of high-level relations between colors that can be implemented with the following guidelines: • Color readability is preserved by constraining the contrasts perceived between the colors. Speci cally, the colors of a group are connected by a complete graph of contrast constraints e c Lab . • Color ordering is often related to shading e ects (luminance order [Chang et al. 2015]) and gradients. We preserve ordering with component-wise contrast constraints e c X with X the color component associated to the e ect (i.e. luminance for shading).
• Colors associated to entities with a speci c color meaning (e.g. the sky) require unary anchors e a R to maintain colors similar to a reference color value or a manifold.
• Color harmonies are broadly used for palette design and color compositions. They are represented in our approach by a complete graph associating hue, saturation and value contrast constraints between the considered colors. The number of colors and the desired contrast values de ne the harmony template (e.g. respectively dual, triadic and analogous, complementary).
In a practical system, high-level relations and groups can be designed either via the UI or by learning the relations from data, for instance using an approach like Probabilistic Color-by-Numbers [Lin et al. 2013]. With this approach, the color-based relations learned from data directly translate to our approach as follows: (1) color properties de ne anchors (lightness as e a R,L , saturation as e a R,sat ), and (2) pairwise color functions de ne binary constraints (e.g. perceptual di erence as e c Lab , relative saturation as e c sat ). As a result, a constrained palette can be generated automatically from a factor graph trained on a given learning set. The constraint weights can be set directly from the learned weights, and the initial reference values can be initialized from the constraints probability distribution functions (e.g. maximum probability value).

Design system for constraints configuration
At this step, the graph topology is set; colors, constraint reference values f r e f and weights w i remain to be de ned. Colors and reference values are tightly related, each in uencing the others. Consequently, we propose two scenarios: the computation of the reference values from input colors, and the estimation of colors from reference values.
• Reference values from colors: Using Equation 1, we compute the reference values as f r e f = f (C j ) to obtain a conformity equal to zero.
Input image + palette Stochastic search output Fig. 8. Color estimation from a constraint graph with known topology and reference values using stochastic search. Input gray-scales illustrate the areas with di erent labels in the image.
• Colors from reference values: Unknown colors are obtained by stochastic search, i.e. initial colors are randomly generated and then optimized to minimize the palette conformity (see Figure 8). This scenario usually occurs for palettes with small cardinality, for instance following harmony templates. When neither colors nor reference values are known, references can be initialized with a default template (e.g. analogous or complementary colors). Then, the values can be re ned during the exploration process, as illustrated in Figure 11.
Without prior knowledge, constraint weights are uniformly set as w i = 1/k E . When the application context is known, di erent heuristics might also be applied. For instance, in case of images, we de ne the importance of a color as the relative coverage of its associated regions in the image, and set its constraint weights as the product of their color importance.
From the user perspective, these di erent initialization procedures can be classi ed in three design modes: • Content-driven mode. Assuming an input media de ning the input colors (e.g. color image), the system automatically computes the constraints' reference values and presets the graph for the optimization. This mode is fully automatic and does not require further user action or speci c user expertise. • Template-driven mode. In this mode no assumption is made on the input media. It is rather done on the color relations: the system needs to provide a catalog of color relations (e.g. level of shades, color harmonies) and expose parameters to the user (e.g. luminance spacing, template type). This mode follows the idea of existing palette design tools such as Paletton [Staníček 2015] and Adobe Kuler [Adobe 2015] (see Section 6.3 for practical use cases). • Expert mode. In this mode both colors and constraints' references are unknown. Two main approaches can be considered: in the rst, the user is able to provide the full set of reference values (e.g. desired contrasts), from which the system automatically infers a rst set of corresponding colors using stochastic search. In the second, colors and/or references are set using heuristics (random, uniform, preset), and the user re nes the parameters by exploration.

Use cases
According to the recent study by Jalal et al. [2015], users tend to manipulate "relationships among [the palette colors], but usually harmony (paletton) augmented harmony Fig. 9. Center: input pale e. Le : desaturation of a pale e with a color harmony (triadic, analogous) defined and edited in Pale on. Right: desaturation of a pale e with an augmented harmony using our approach, by desaturing the central color. Bo om: hue distribution of input and output images, with pale e colors shown as vertical bars.
in an ad hoc way, combining features from multiple tools or creating their own techniques". This section presents two practical systems with which users can easily create and modify color palettes while manipulating both the colors and their relations. The systems are tailored for color harmony relations, one extending interactive palette design tools [Adobe 2015] (Section 6.3.1), and the other following the work ow introduced by Cohen-Or et al.
6.3.1 Harmony-based interactive pale e design. Interactive color palette design tools such as Paletton [Staníček 2015] and Kuler [Adobe 2015] provide catalogs of harmony templates. In this catalog, the user selects a type of harmony (e.g. analogous, complementary) and a number of colors. Starting from colors initialized by the system, the user manipulates the palette by changing one of its colors at a time, and the palette parameters when available (e.g. the angular distance between complementary colors). In real-time, the system updates the other colors to conform the palette to the selected template. The main limitation of these systems is to be restricted to harmony templates expressed as geometric con gurations. Thus, they cannot generate palettes corresponding to compromises expressed by heterogeneous constraints.
This exploration metaphor directly translates in our formalism: the catalog of templates correspond to prede ned palettes composed of a unique color harmony group (see Section 6.1 for the de nition of its graph). From the user perspective, there is no visible change between the two approaches: starting from a template, the user edits the palette colors and some template parameters, and the result is provided in real-time.
In addition to providing the same functionalities as existing systems, our optimization-based exploration opens exciting opportunities to improve harmony-based palette design. For instance, a known limitation of harmony templates is the generation of unexpected contrast variations during the exploration. With our representation, we can model the perceptual distance between colors and nd the compromise between harmony and contrast preservation. This is done by adding a contrast constraint e c Lab between the colors. The advantage of this representation is to produce results comparable to harmonies in most of the cases, while preserving the readability in extreme cases, as shown in Figure 9 for a strong desaturation of a palette. 6.3.2 Color harmonization. This system takes as input a raster image and harmonizes its palette colors, following the ideas presented in the seminal work by Cohen-Or et al. [2006]. In practice, the system works as follows: the user composes the image palette using the palette extraction tool introduced by Chang et al. [2015]. Then, she provides a harmony template de ned by a set of colors and tolerance values along the hue axis. The right inset shows an example for the analogous template used in Figure 10. From an input template, the system automatically generates constraints as follows (see Figure 7 in supplementary materials): (1) Colors are constrained by anchors penalizing deviation in hue from the template colors (f a T with T the template colors).
(2) Colors are constrained by anchors in saturation and value.
(3) A complete graph of Lab contrast connects all the colors. (4) A global constraint is added to preserve the palette luminance. The constraint parameters (reference values and weights) are automatically computed from the image: contrast values are set from the initial contrasts, and the target anchor distances are set to 0.
As the anchors penalize the deviation from the template colors, the conformity of the initial palette is unlikely to be 0. Running our optimization provides a compromise between the palette color contrasts and distances to the harmony template. There is no explicit assignment between the palette colors and the individual template colors: our anchor formulation automatically choose the reference color that minimizes the distance.
Optimization results are illustrated in Figure 10. Our approach operates on the palette and not directly on the pixel colors as in the original approach, which explains the di erences in the resulting images and histograms. We link the pixel and the palette colors with Chang et al. [2015] image recolorization process. This produces wider picks in the hue histogram, while maintaining local contrasts in the image.

Implementation and timings
We use the Levenberg-Marquardt implementation provided by the Eigen library [Guennebaud et al. 2010], and we employ numerical di erentiation to compute the Jacobian matrix of the conformity error. We report in Tables 1 and 2 the timings required to solve  Equation 6 and Equation 7 respectively, for all the scenes shown in the paper and in supplementary material. All our experiments were done on single core, with a processor Intel(R) Core(TM) i7-950, with 12GB DDR3 memory.
Our rst exploration scenario takes as input a palette and optimizes it to reduce its conformity. The optimization is performed interactively when the user edits a color in the interface. The edited color is considered as locked, and so excluded from the optimization variables to conform to the user edit. Any color of the palette can be locked in the interface, so that it is not optimized but its constraints are still considered (except unary constraints). In most of the cases (see Table 1), the optimization is done in less than 40 milliseconds, enabling real time updates of the palette w.r.t. to user inputs. As a result, our approach requires a competitive time budget against existing interactive tools, while being able to handle complex constraints and large palettes.
The interpolation-based exploration of palettes requires to solve a much larger optimization problem, and requires more computation time. However, our implementation provides an interactive feedback to the user for most scenes, by estimating the paths in seconds, depending on the palette complexity. Note that the timings do not include the computation of intermediate palettes once the paths are known, since the evaluation of few Bézier curves requires a negligible amount of time.

Exploration
This section presents several use cases of our approach, with the description of the hypergraph setup.
7.2.1 Semantic-aware pale e design. The palette of the drawing shown in Figure 1 is composed of 19 colors with strong relations, e.g. contrasts, color shades, semantic meaning. The palette setup is illustrated in the accompanying video: we rst start by identifying 7 color groups containing from one to four colors. We consider two groups as static (e.g. gray-scales, eye colors) and do not consider Scene k #Constraints Timing #Iter. Unary Binary Global (ms) Interpo. Test (Fig. 3) 2 -1 1 170 110 Small Map (Fig. 4) 6 6 15 1 900 38 Task 2 6 2 7 -65 5 Small Map (Fig. 15) 6 -15 2 4302 147 Table 2. Statistics for the interpolation-based exploration of pale es. Timings are reported for paths optimization only.
them for the exploration. The second group has only one color, the background, which we constrain to pastel colors using two anchors e c L and e c sat . Each of the ve other groups represents a shade with an almost uniform hue: we encode these relations using complete graphs with e c L , e c a and e c b . Groups corresponding to hair and skin have a strong color meaning: we constrain them with anchors to restrict colors w.r.t. to tone reference charts. We ensure the readability of the palette by connecting the di erent groups with a complete graph of clamped contrasts between each group representative.
During the exploration, when the user moves one color, the other colors of its group follow by keeping an almost rigid structure in Lab, and thus preserve the ordering in luminance and the di erences in colors. When the representative colors of two groups come too close, the group moved by the user pushes the other so that the clamped contrast is still satis ed. We demonstrate in the additional video how the exploration of colors with semantic meaning is a ected by the lack of unary anchors, and the di erences between single color and color chart anchors. 7.2.2 Editing constraints parameters. We illustrate in Figure 11 the exploration of palettes by editing a constraint reference f r ef . The hypergraph setup de nes two shade groups (one by characters), and xed colors for the lips, tongue, black and white. The other colors are set to ensure contrast between each others and with the two shade groups. While the principal color of left character is locked, the user controls the expected contrast with principal color of the other character. When the contrast is reduced to zero, the two characters have merely the same color, while when the contrast is increased, the color of the second character reaches gamut limits to maximize the color di erence between the two characters. The color of the basket changes during the exploration to preserve contrasts with the colors of the second character.
7.2.3 Contrast enforcement for CVD. We illustrate our approach capabilities for CVD-aware palette editing by optimizing a CVD test image to make it readable by people with CVD ( Figure 12). Figure 13 the exploration of a palette with large cardinality (55 colors including all color shades) using a complete graph of contrast constraints (e c Lab , see graph topology in supplementary materials). This simple graph generation provides limited exploration capabilities compared to a ne palette tuning. It is however simple to generate, and can be further edited if required. 7.2.5 Photo editing. We also combined our approach with the recent work by Chang et al. [2015] on Palette-Based Photo Recoloring. We use their approach to extract a palette from an input Input image Fig. 13. Image samples from an interactive exploration session. The pale e constraint has been automatically computed from the input colors for contrast preservation (white is not optimized). The hypergraph has 55 colors and 1485 pairwise e c Lab constraints. Image courtesy Kelvinsong.

Large pale es. We illustrate in
Fig. 14. Le : input image and its pale e extracted with the method proposed by Chang et al. [2015] . Right: pale e obtained a er modifying the buildings and the sky colors (black borders) using our approach. The final image is recomposed with the method proposed by Chang et al. [2015]. Image courtesy Bernard Spragg NZ.
image, from which we design a constraint graph. Here we want to preserve the image coherence when the user modi es the palette colors. For the example shown in Figure 14, we rst grouped the colors corresponding to level of shades (3 groups), and constrained their colors by component-wise contrast constraints in L, a and b (see Section 6.1). Then, we constrained the groups to preserve readability using contrast edges (e c Lab ) between colors. We also add anchors to the sky and the window color (dark brown) to convey lighting semantics, and a global constraint to preserve the luminance of the photo. The palette generated by our approach is then given as input to Palette-Based Photo Recoloring to generate the nal output (we disabled the luminance correction originally applied on the input palette).

Interpolation
We consider in a rst example (see Figure 15) the case where we need palettes to be used in a GPS with an OLED screen, such that the palettes can adapt to varying display energy requirements. We start by a palette applied on a simple map, and modify it using our interactive exploration technique by lowering its display energy consumption. The interpolation between the two palettes (lower and higher energy requirements) ensures contrast preservation for all intermediate palettes.   Top-row: linear interpolation; Bo om row: Cartographic constraint-based optimized interpolation, using anchored colors, hue and value contrasts. Notice how our approach produces a more readable map, with increased contrast e.g. on the road network.
approach respect the cartographic constraints from the rst input, while mixing the colors from the second input.
Finally, these interpolation-based explorations show the expressiveness of our modeled constraints, which would allow a mapmaker to subtly mix and re ne salient colors in both map endings.

USER STUDY
We performed a user study to evaluate our palette exploration approaches. We recruited 30 people, all with normal vision (see statistics in additional materials). We divided the study in two tasks, a rst task on the exploration of an analogous triad under contrast constraints for normal and impaired vision, and a second task on the interpolation between two palettes. 8.1 Task 1: Augmented harmony and color blindness 8.1.1 Description. The goal of this task is to measure how our approach helps users to nd a trade-o between constraints. Users were asked to modify an existing palette to reach a speci ed target color tone, while preserving simultaneously three goals: (1) contrasts for people with normal vision, (2) contrasts for people with deuteranope vision, (3) color harmony. No formal de nition of these quantities was given to the user, however we provided real-time feedback during the exploration to help them evaluating their design (see supplementary materials for more details).
8.1.2 Scene setups. We designed three different constrained palettes, sharing the same graph topology (shown in the right inset) but with di erent constraints between the colors. Colors have been generated using a triadic analogous harmony template with the online tool Paletton 1 . We design the palettes from these colors as follows: (a) Harmony template: each pair of colors is constrained by contrast constraints e c hue , e c sat and e c val , respectively on their hue, saturation and value properties. This template aims at reproducing the expected behavior of common harmony exploration tools, and is considered as a baseline for comparisons.
1 See http://paletton.com/#uid=53z0t0keGvq5jR69VG8jjqCnvm8 (b) Extended harmony template: extends (a) by adding a contrast constraint e c Lab . (c) CVD-aware extended harmony template: extends (b) by adding a contrast constraint e c Lab on the colors after CVD simulation. These palettes are designed to preserve the harmony (a,b,c), contrasts for normal vision (b,c) and deuteranope vision (c).
8.1.3 Task. We asked the users to modify the palette to obtain one of the two colors shown in the right inset. This task was repeated 6 times in a random order, so that the user explored with the three templates (a-c) for the two target colors.
8.1.4 Results and discussion. Explorations are evaluated by reporting the residuals of the harmony and contrasts (normal vision and deuteranope) constraints, de ned as: For the harmony template (a), the palette de nition ensures the generation of color sets that all perfectly meet the harmony criterion. On the other hand, due to the non-linear mapping between the HSV and Lab spaces, users mostly fail at preserving contrasts between the colors. The introduction of a contrast constraint with the augmented harmony template (b) gives more chances to the user to obtain better contrasts, and in practice divide the mean contrast error by 2. A similar pattern is observable for the contrast preservation for deuteranope vision, with the introduction of the associated constraint in the CVD-aware extended harmony template. In the case of template (b) and (c), the contrast improvement is obtained at the cost of an increase of the harmony error. For template (c), the palette constraints encode the three measures the user has to minimize, so the trade-o is performed by the system while the user could focus on the target color tone.
In summary, this study con rms the ability of our approach to represent and nd palettes compromising between di erent constraints. With template (a), some user explored and found a palette that reached a good trade-o between the three measures, while most of them gave up with a relatively high residual error. From the user point of view, our approach hides the complexity of the constraints without limiting the exploration experience.
8.2 Task 2: Interpolation 8.2.1 Description. The goal of this task is to evaluate if palettes generated by interpolation are perceived as such by users.
8.2.2 Task. The study interface presents 3 palettes applied sideby-side on the same vector image, with from left to right: the source, the candidate and the target palettes. The 12 candidates are shown sequentially to each user with randomized order. No information is given to the users about the constraints or the method used to generate the images. Instructions are given as follow: Considering two images, called source and target, please indicate if a third image is a good intermediate between the source and target images. This comparison must be made only considering the image colors. It is up to you to decide, whether an image is a good intermediate, or if it is either too di erent or too similar to input palettes, considering the two examples (shown in Figure 17).

Images generation.
The source palette is composed of a group of 4 contrasted colors, i.e. constrained by a complete contrast graph with e c Lab pairwise constraints. The background (white) and shape borders (black) colors are kept in place using color anchors e a c r e f and a pairwise contrast distance e c . We designed the target palette by exploration from the source palette.
We generate a set of twelve palettes (see Figure 18) using 4 di erent methods: • Our: palettes obtained along the interpolation paths (α = {0.25, 0.5, 0.75}) optimized with our approach between source and target, • Linear: palettes obtained by linear interpolation between source and target (α = {0.25, 0.5, 0.75}), • Stochastic: palettes obtained by stochastic search from the source palette, with low residual error, • Stochastic_bad: palettes obtained by stochastic search from the source palette, with high residual error.
8.2.4 Results and discussion. Figure 19 presents the repartition of the labels assigned by the users to the di erent palettes. We propose to analyze the results by label: Too di erent As expected, this label is mostly assigned to the stochastic and stochastic_bad palettes, which are generated from the input source image only. Also notice that it is also assigned to 43% of the linear palette (α = 0.5), probably due to the contrast loss at this intermediate interpolation step.
Our Linear Stochastic Stochastic_bad source target Fig. 18. Target, source and candidate pale es used for the study, applied on a vector image. For Our and Linear, the interpolation parameter α is set from top to bo om at 0.25, 0.5 and 0.75.
Too similar Unsurprisingly, this label is mostly assigned to the palettes generated by the interpolation methods (our, linear) for α = {0.25, 0.75}. This con rms the ability of interpolation methods to produce smooth transitions to input palettes. Good intermediate Palettes obtained by our method are clearly the most recognized as good intermediates. We also observe a notable di erence between our approach (70%) against linear interpolation (36%) when α = 0.5.

LIMITATIONS
Scalability: In our representation, the larger the hypergraph, the lower the in uence of each constraint. Hence, moving a color in a large graph may have a small in uence on the optimization process, which would reduce the user control on the exploration. A solution could be the use of meta-vertices in the graph to group colors and their relations in a multi-level fashion. A similar hierarchical approach has been used successfully for grouping geometric relations between 3D primitives [Monszpart et al. 2015].
Numerical constraints: Our constraints are de ned as real-valued cost functions. While very e ective for representing and optimizing continuous constraints, it prevents the inclusion of labels and Boolean relations in our palettes. A solution could be to represent such constraints as probability distribution functions, which can be directly inserted in constrained palettes. Another would be to optimize the conformity and labels with mixed-integer programing.
Extreme contrasts: In Lab space, if a constrast constraint is set between colors with very strong constrasts (e.g. with highly saturated colors and di erent hues), the color variation space maintaining a low conformity will be restricted by the gamut boundaries. During exploration, this may produce very high conformity errors that would prevent the optimization to provide adequate color solutions.

CONCLUSION
We present a new de nition of constrained palettes based on an hypergraph representation. This de nition enables interactive optimization and is well suited for palette space explorations. We present two exploration strategies whose e ciency and robustness are illustrated on real world use cases, such as artistic/cartographic palette design, and photo recoloring. In addition to contrast constraints, our approach naturally handles context-based constraints, such as Color Vision De ciencies simulation and energy consumption optimization.
Our work opens several promising directions of investigation. For instance, multiple-palettes interpolation would enable the generation of palettes manifolds, following the same spirit of color manifolds [Nguyen et al. 2015]. We also believe that our approach can be applied to other parametric spaces than digital color: we plan to study the exploration of parametrized appearance properties, such as hatching patterns and procedural textures. Finally, we would like to consider the human perception system with higher accuracy in our constraint de nitions and optimizations.